Cloud + Machine-to-Machine = Disruption of Things: Part 1
This post is part of our ReadWriteCloud channel, which is dedicated to covering virtualization and cloud computing. The channel is sponsored by Intel and VMware. Read the white paper about how Intel Xeon processors help organizations get unprecedented levels of performance.
The use of cloud infrastructure and cloud services provides a low-cost means to create highly scalable applications. Even better, the cloud dramatically improves development speed and agility. Applications can be developed in much less time with much smaller teams. And as these benefits extend themselves in the machine-to-machine (M2M) space, companies creating M2M applications will see dramatic reduction in the cost of developing applications and provisioning services.
Articles on the Internet of Things (or Web of Things) are increasingly finding their way into mainstream news. Executives of large companies (such as the CEO of Sprint) and even government officials (such as the Chinese Premier) are speaking about the possibilities and opportunities of having ubiquitous sensors connected to the Internet.
Almost every major electronic device, vehicle, building component, and piece of equipment has the ability to become "smart" by connecting sensors to it. Most devices already do. The difference though is that moving data to the cloud and being able to process it in infinite combinations provides new capabilities in very low cost, transparent ways.
M2M Business Transformation
The case for what the Internet of Things might entail has been eloquently made here, here, andhere. When devices and machines can send data to the cloud and have dashboards and interfaces on Web browsers, HDTV wallboards, mobile phones, and ipads, the impact becomes large.
This potential will affect almost every industry - just as the Internet, email, websites, e-commerce, and now Web 2.0 are touching every industry and every business process. The impact will be most noticeable at non-Web companies.
The change here will be dramatic - from where every device is by itself or controlled through a local device to where every device can be accessed anywhere (by authenticated users), where data streams can be "followed," and interfaces and dashboards improved on the fly to provide new views and device control. Does the concept of "following" a jet engine or a pneumatic thermostat have appeal to equipment makers and airlines or building owners? You bet it does.
Equipment, automobile, and device manufacturers need to beginning positioning themselves to gather realtime data on the performance of each product and use cloud processing and data storage to do it. Using this approach, they'll be able to rapidly improve their products, build direct connections with customers, and get ahead of customer and product issues. They'll also be able to offer service offerings and develop new revenues sources. Services will become a part of every product. Some as ways to improve customer support and customer connections. Others as revenue sources in and among themselves.
Want a quick diagnosis on your transmission? Go to CloudAutoDiagnostics.com and check in with your car's data feed. It will compare your data from the transmission sensors against others with similar transmissions. Yup, there's a issue but nothing serious. Want a 10% coupon for the service shop around the corner?
Below is a short list where the Internet of Things and M2M in the cloud will matter although it really could be just a single line that says anywhere where there is a sensor, an electronic device, or a machine.
- Personal Health and Fitness
- Medical Devices
- Automobiles
- Shipping, and Transportation
- Smart Grid and Smart Buildings
- Retail
- Architecture
- Agriculture
- Mining
- Natural Resource Management
The use of the cloud - in combination with the advent of low-cost sensors and high-availability M2M data transmission - will transform old industries and modify many business models. As is the case in each disruptive tech cycle, new companies will arise, existing market share will be threatened, and the separation between industries and channels will become blurred. Those in the M2M space who take advantage of what the cloud offers will not only be able to anticipate these changes but will lead the way into these new opportunities.
Key M2M Cloud Patterns
The goal of this paper is not to convince readers of what the future will like or even go through what new devices might look like. ReadWriteWeb and other tech publications will do a far better job there. The goal here is to list out the advantages that cloud computing brings to M2M applications.
Separation of Data Collection, Processing, Interface, and Control
The use of cloud computing means that data collection, processing, interface, and control can be separated and distributed to the most appropriate resource and device. Current M2M implementations combine data collection, processing, interface, and control. Either chips in sensor bodies or an onsite laptop or desktop PC tied within a local mesh network perform the data processing and determine the control logic.
Once data collection and processing moves to the cloud, however, most current limitations disappear. One of the more obvious ones is that data limits go away. Storing data in the cloud means that the data buffers within devices (whether its 500 or even 5,000 data points) no longer matter. Cloud storage is near limitless and so historic data can be saved for as long as its deemed valuable.
Applications are also not restricted by tiny processors, low-power consumption, and special purpose programming languages. Processing in the cloud brings best-of-breed programming capabilities. These include widely popular programming languages and frameworks, flexible data structures, and extensive algorithm libraries.
Not only is there access to super-fast processors, if there are server or storage bottlenecks, these can be addressed by on-demand launching of more servers or horizontally scaling storage. Using the cloud and dynamic languages and frameworks, the cost of ownership goes way down and the limitations go away. There's also a huge increase in the speed of product development.
Lastly, interfaces can move to Web browsers, wallboards, mobile phones and tablets, eliminating the need for either having screens a part of every devices or local computers permanently a part of installations. Medical devices no longer have to come with their own monitors. Separating the data input from the processing from the screen readout not only means lower costs (less components to the devices) but also easier upgrading of diagnostics and far better visualization capabilities.
An MRI or sonogram sent to the cloud, digitally refined using the latest algorithms distributed across multiple machines, and presented on an iPad or an HDTV screen is going to look a lot better and be more insightful than if displayed on existing monitors, no matter how new the device is. Separating the components and putting the data and processing in the cloud allows devices to keep getting smarter while not necessarily becoming more complex.
Data Virtualization
Data storage is one of the biggest advantages of using the cloud for M2M applications. The cloud not only offers simple and virtual ways to run applications, it also offers simple and virtual ways to store data. Cloud infrastructure companies are increasingly offering simple-to-use services to provision and maintain databases. These services even extend to offering databases as a service - meaning offering expandable data storage at the end of an IP address all the while masking or eliminating the management of servers, disks, backup and other operational issues. Examples include Amazon's SimpleDB and RDS service and Salesforce's Database.com offering.
Once transmitted to the cloud, data can be stored, retrieved and processed without having to address many of the underlying computing resources and processes traditionally associated with databases. For M2M applications, this type of virtualized data storage service is ideal.
Being able to seamlessly handle continuous streams of structured data from sensor sources is one of the more fundamental requirements for any distributed M2M application. As an example, Appoxy processes the data streams from network adapters from Plaster Networks, a fast-growing leader in the IP-over-power line space. Status inputs are sent by the adapters continuously to Plaster which runs its applications on one of the top cloud infrastructure providers. Appoxy processes these for use with the user dashboard running on the Web.
This console provides insights to users on the status and performance of the adapters and their networks (allowing users to see whether they are getting optimal performance out of their devices and networks). The data also provides valuable diagnostic information to Plaster, dramatically reducing support calls and improving device usage and customer satisfaction. The information is also invaluable for product development. Data on the performance of new products and features can be assessed in real-time, providing insights that would otherwise be unattainable from devices in field.
This type of smart device marks the beginning of the trend. It's a fair bet that all network devices will become smart and cloud-aware. Followed by all vehicles, manufacturing equipment and almost all machines of any substance.
The types of data stores available include SQL, NoSQL and block or file storage. SQL is useful for many application needs but the massive, parallel and continuous streams of M2M data lends itself well to the use of NoSQL approaches. These data stores operate by using key-value associations which allows for a flatter non-relational form of association. NoSQL databases can work without fixed table schemes, which makes it easy to store different data formats as well as evolve and expand formats over time.
NoSQL databases are also easy to scale horizontally. Data is distributed across many servers and disks. Indexing is performed by keys that route the queries to the datastore for the range that serves that key. This means different clusters respond to requests independently from other clusters, greatly increasing throughput and response times. Growth can be accommodated by quickly adding new servers, database instances and disks and changing the ranges of keys.
The NoSQL approach play well in M2M applications. Data for sensors or groups of sensors can be clustered together and accessed by an ever-expanding set of processes without adversely affecting performance. If a datastore gets too large or has too many requests, it can be split into smaller chunks or "shards." If there are many requests on the same data, it can be replicated into multiple data sets, with each process hitting different shards lessening the likelihood of request collisions.
Cloud + Machine-to-Machine = Disruption of Things: Part 2
This post is part of our ReadWriteCloud channel, which is dedicated to covering virtualization and cloud computing. The channel is sponsored by Intel and VMware. Read the white paper about how Intel Xeon processors help organizations get unprecedented levels of performance.
Once data is in the cloud, it can be syndicated - made accessible to other processes - in very simple and transparent ways. The use of of REST APIs and JSON or XML data structures, combined with dynamic language data support, allows data to be accessed, processed and recombined in flexible and decentralized ways. A management console, for example, can set up specific processes to watch ranges of sensors and perform operations specific data sets. These processes can be launched and run on any server at any time, controlled via a set schedule or initiated in response to other signals.
Data can also propagate easily throughout a system so that it can be used by multiple processes and parties. Much like Twitter streams that make use simple subscription/asymmetric followapproach, M2M data streams can be exposed to credentialed entities by a similar loosely coupled and asymmetric subscription process.
The model is essentially a simple data bus containing flexible data formats, which other processes can access without need for a formal agreement. Data sets, alarms and triggers can move throughout the system much like status and other federated signals are flowing within Consumer Web 2.0.
Low-Cost Sensors and Reliable Data Transmission
The cloud benefits outlined here are predicated on low-cost sensors being available (and low-power for a number of uses) as well as inexpensive and high availability data transmission.
The first assumption is a pretty solid bet now and will only become more true in the next year or two. And with mobile companies looking at M2M as important sectors and wireless IP access extending out in both coverage and simplicity (and bluetooth to mobile for some instances), being able to reliably and pervasively send data to the Web is also becoming more of a certainty.
Flexible and Agile App Development
Another core benefit of the cloud is development speed and agility. Because screen interfaces can be separated from data collection, inexpensive and remote devices can be used as the interfaces. Separating views from data from application logic means that widely available languages and tools can be used, making it easier and faster to add new features and rapidly improve interfaces. Device monitoring and control can take on all the features, functions and capabilities that Web apps and browsers provide without having to have developers learn special languages or use obscure device SDKs.
Developing these applications can be done quickly by leveraging popular dynamic languages such as Ruby on Rails, Python and Java. Ruby on Rails (Ruby is the language; Rails is an application framework) offers many advantages when it comes to developing Web applications:
- simple dynamic object-oriented language
- built-in Web application framework
- transparent model-view-controller architecture
- simple connections between applications and databases
- large third-party code libraries
- vibrant developer community
Here's Mark Benioff, CEO of Salesforce explaining his purchase of Heroku, a Ruby on Rails cloud development platform from ComputerWeekly in December 2010:
"Ruby is the language of Cloud 2 [applications for real-time mobile and social platforms]. Developers love Ruby. It's a huge advancement. It offers rapid development, productive programming, mobile and social apps and massive scale. We could move the whole industry to Ruby on Rails."
Developing applications in the cloud provides added speed and agility because of reduced development cycles. Projects can be developed quickly with small teams. Cloud-based services and code libraries can be used to so that teams develop only what is core to their application.
Adding charts and graphs to an application is a matter of including a code library or signing up for third party service and connecting to it via REST APIs. Adding geolocation capabilities involves a similar process. In this way, new capabilities can be added quickly and current capabilities extended without having to develop entire stacks of functionality not core to a company's competencies.
Here's VMWare's CEO Paul Maritz on the advantages of programming frameworks:
"Developers are moving to Django and Rails. Developers like to focus on what's important to them. Open frameworks are the foundation for new enterprise application development going forward. By and large developers no longer write windows or Linux apps. Rails developers don't care about the OS - they're more interested in data models and how to construct the UI."
Monitoring and control dashboards and data visualization are critical to creating effective M2M applications. Having dynamic languages and frameworks that facilitate rapid development and rapid iteration means companies can move quickly to roll out new capabilities and respond rapidly to customer needs.
Big Data Processing
Having data stored and available in the cloud makes it far easier to analyze. Distributing data on a sensor-by-sensor basis means simpler access for individual nodes or collections of nodes. Data can be distributed quickly to different entities much like the way Facebook photos or Twitter posts are distributed to each friend or follower. Flatter data formats means applications can be simpler and take less to develop. But big challenges arise when trying to analyze a massively growing data streams.
Given the large amount of incoming data and the wide range of queries on the data, significant effort needs to go into parsing and processing it so that queries are as responsive as possible. This means use cases for managing data that include optimizing near-term storage for immediate queries, setting up frameworks for analytics across massive data sets and deciding on what data to archive and in what raw, processed and derivative forms.
With Plaster Networks, for example, Appoxy keeps near-term data available for queries on the current performance of adapters. Appoxy also structured the data model and data flow to quickly respond to queries based on common time frames - performance for the last day, week, month, for example - and node groupings (adapter to adapter and adapter to device). Detecting and flagging unusual performance within a network is also an inherent M2M pattern architecture.
In addition to near-term data handling challenges comes issues with analyzing large data sets. An example is running queries extending across an entire set of nodes in a system to obtain insight on aggregate behavior and device performance. There could be thousands of sensors each with hundred of thousands of data points. This is no unlike data processing queries within social network sites or other large consumer websites with millions and hundreds of millions of users.
Just as NoSQL storage options are largely derived from consumer Web needs, big-data data architectures and analytical methods are also making their way from these same efforts. Hadoop, for example, is a set of frameworks used for distributed data analysis. One of its projects, MapReduce, is a basis approach pioneered by Google to process data across many clusters and then propagate the findings up from the nodes.
Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.
A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.
Using these big data approaches, large sets of M2M data can be analyzed across an elastic and scalable cloud infrastructure. The virtual nature of the cloud along with distributed data models means that jobs can be queued up by the thousands and run in parallel if needed.
Once results have been reduced and then answers assembled, they'll need effective visualization tools in order to have meaning. This requires charts, graphs, tables and maps and might require a similar level of optimization as handling the analysis at the start of the process. The good part here is that the elastic nature of the cloud processing and the flexibility and agility of cloud development means that these interfaces can be created and extended and control options added in the same manner and with the same speed as other parts of applications.
Summary
M2M applications across many devices and industries will have many of the same patterns regarding data collection, processing, visualizing and control. Just as there are patterns to social applications and Web 2.0 cloud services, there are patterns to M2M applications. Smart devices for building efficiency have similar needs as do medical devices, mining and agriculture sensors, truck and automobile diagnostics and most other electronic devices and machines.
The particular types of sensors used, the data collected, the way it's transmitted to the cloud, the views and reports generated and the actions triggered may be different, but many core application needs, process flows and data approaches will be the same.
The advantages of the cloud -- in data storage and application development alone -- present a significant inflection point in the cost of operation and the speed of development for M2M applications. Sensor, device and equipment makers are only just beginning to leverage these capabilities but not near there levels where they could be.
And the opportunities aren't just limited to M2M applications surrounding devices. There are big opportunities to create new M2M platforms and platform services. Just as smart phone service companies and cloud industries have formed, overtaking established mobile and IT leaders, new M2M companies will arise to supply important cloud-based components to this emerging set of smart-enabled devices, vehicles and machines.
The mobile and cloud industries have made people acutely aware the power that open platforms in combination with rich development tools and vibrant support community have in regards to technology adoption (see Apple iPhone and Google Android vs. RIM, Palm, Symbian and Microsoft). These same insights - and the strategic value of creating and leveraging cloud-based platforms - can be employed to service makers of devices and machines.
Great products can no longer be great products in isolation. They'll need to be cloud aware in order to be viable in this changing ecosystem. An ecosystem where data matters and monitoring and control of devices can exist anywhere. Cloud computing and the way it impacts M2M applications will touch and transform, every industry in the same manner as has the Internet, email and the Web.
No comments:
Post a Comment