TALEND CONNECT 2018 : Get inspired by the movers and shakers in the big data world in NYC
Sensors, Environment and Internet of Things (IoT)
Sensors, Environment and Internet of Things (IoT)
According to Jacob Morgan, “Anything that can be connected, will be connected.” At one time, the terms cloud and big data were regarded as just ‘hype’, but now we’ve all witnessed the dramatic impact both of these key technologies have had on businesses across every industry around the globe. Now people are beginning to ask if the same is true of IoT—is this all just hype? In my opinion, hype is certainly not the word I would use to describe a trend or technology that has implications of profoundly changing the world as we know it today.
In 2006, I was doing research on the use of RFID and we introduced a way to organize chaotic office document with an automatic data organizer and reminder technology. I published a paper on this topic called, “Document Tracking and Collaboration Support using RFID”. That was my first interaction with sensors where we focused on M2M (machine-to-machine) and afterward integration with collaborative environments, hence pushing us in Subnet of Things. The idea emerging at that time like smart home, smart cars, smart cities are now realizing. According to IDC, there will be 28 billion sensors in use by 2020 with an economical worth of $1.7 trillion. Before we jump in some applied scenarios, let outline the scope of sensors communication and how they are organized into three groups.
Machine-to-machine is different from IoT in scope and domain. Normally the two are restricted in availability and come with pre-defined operational bindings based on data. The suitable example would be manufacturing units and their communication with different sensors built-in. A more localized example would be heating sensor in the car that has a very definitive purpose and is only limited to that car.
SoT (Subnets of Things) can be scoped at the organization or enterprise level. Like in the above example where you have RFIDs on each file or book, SoT can be located inside organizations using collaborative platforms. For example, a car sending data to measure quality and utilization of its components to deliver better operations and stable experience.
In 2013 the Global Standards Initiative on Internet of Things (IoT-GSI) defined the IoT as "the infrastructure of the information society. The Internet of Things (IoT) is a system of interrelated computing devices, mechanical and digital machines, objects, animals or people that are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction. IoT has evolved from the convergence of wireless technologies, micro-electromechanical systems (MEMS), micro-services and the Internet.
Sensor(s) Data Type and Network Challenges
During my travel to San Francisco, I met SAP’s R&D lead for IoT. It was very interesting to hear how big data is already shaping our daily lives. For example, he mentioned an airplane system where thousands of sensors are delivering data and need to make calculations up to 8 to 16 decimals in a fraction of a second. Thanks to sensors, we have so much data that our current technology infrastructure is the only gating factor to companies being able to harness the power of real-time, big data insight.
IT decision makers should create infrastructures and utilize solutions that can handle data which was kept due to unavailability of a network. Sometimes, you don’t have telecommunication network or sometimes surroundings do not allow you to connect due to a signal jammer, etc. so IT leaders need to focus on infrastructure which can hold and keep data until a network is available and data transfer is possible.
Sensor data is just like any other data coming from different sources that needs cleansing, analysis and governance. At the same time, it has some distinct properties. Normally, sensor data is a stream of information (values) juxtaposed against time. Not all data is meaningful but we cannot simply discard information as all data is destined to have some value, even if not known at time of collection. So you should not use any loose or questionable algorithms for data compression. There are few possibilities to overcome a situation like this:
· Send filtered data only if it is required
· Only transmit abnormal values
· Compress the data without using unfounded/untested algorithms
As my discussion with him goes on, he says, “just look around this place. It is generating so much data. If something happens or if any component fails, the data generated before that failure is very important. It’s a complete echo system. If we lose that data, then we will not be able to predict any such event in the future. So our only option is to compress the data using lossless compression to keep cost down (the only alternative is one we can’t afford). There are some situations where we have duplication of data or the frequency of data is overwhelming, which results in a lot of overhead. But again our back-end system should be capable of digesting all incoming information. At sensor level we can apply some curve-fitting techniques like Fast Fourier Transformation, so that we keep getting aggregated value. The sensor data are best stored in low-cost, scale-out systems called Data Lakes. Raw data normally doesn’t get pulled into the data warehouse as it lacks proven value but requires definitely governance like security etc.
To perform analysis and get value from data it should be stored in Data Warehouse and I would definitely recommend reading our Data Vault blog that explains how to store data effectively.
Sensors Impact on our Daily Lives
Sensors are impacting almost every aspect of our life. From movement sensors to manufacturing units, everything is going to get smarter day by day. Let’s walk through some very basic and brief examples of IoT that are going to impact us in the near future.
Manufacturing is a $12 trillion global industry a year. This is an industry where robots are transferring goods from one place to another and every action of metal-stamping machines in an auto-parts factory and assembly line is tracked by sensors. Otherwise known as Industrial Manufacturing, this sector has already shown remarkable growth rates. Data collected from different processes can help prevent unplanned downtime and/or predict supply needs based on Sales forecasts.
Automobile integration is being touted as the next great frontier in consumer electronics. According to Gartner, driverless vehicles will represent approximately 25 percent of passenger vehicles population in mature markets by 2030. Driverless cars may have a steering wheel just for legacy design and ‘machine override’ purposes, but ultimately there will be no manual controls in car at all. All traditional controls like brakes, speed or function indicators, and acceleration rate will be based on sensors, radar, GPS mapping, and a variety of artificial intelligence to enable self-driving, parking and the safest route to destination to avoid accidents. This impressive technology will focus not only on controlling cars, but on communication between other vehicles on the road based on that car’s road condition status or relation to the driver’s vehicle. Communication that require external networks will enable Internet-based services, like route calculation, vehicle status, e-call, and b-call usage based on insurance and backup of data.
Electric Power grids started operating in the 1890s, which were highly centralized and isolated. Later, the electricity network was extended and power houses got connected with load shifting technologies in-case of technical shutdown (backup and recovery). Many small power generating units like windmills and solar energy parks are now generating electricity at varying capacity depending on condition. Today, electrical distribution networks accommodate many more sources as households are also generating electricity, putting pressure on ‘the grid’ as well as making it difficult to manage a centralized system.
Based on system condition, utilities leaders will be able to decide which source of energy is cheapest for the moment and shut-down coal or fuel based energy sources to avoid unnecessary generation and cost. Similarly, smart meters are now billing clients based on usage, but in future, as they get more connected, those sensors might be capable of deciding which energy source is cheapest.
Smart Homes and Cities
The smart city is a very interesting possible application of the IoT. Smart homes and smart cities are connected to each other. Smart cities provide great infrastructure for communication and are an ideal candidate from which to extract benefits. There are already some projects in Chicago, Rio de Janeiro, and Stockholm where governments, in collaboration with private sector companies, are taking advantage of IoT to collect data from city street assets to determine whether or not they need repair, i.e. street lights. From school bus monitoring to garbage collection, IoT is changing the scope of how society functions.
"Smart Home" is the term commonly used to define a residence that has appliances, lighting, heating, air conditioning, TVs, computers, entertainment audio & video systems, security, and camera systems that are capable of communicating with one another and can be controlled remotely by a time schedule, from any room in the home, as well as remotely from any location in the world by phone or internet. The next iteration of today’s Smart Home will be capable of conducting inventory management on your fridge, where it automatically places an order for milk or eggs after determining your supplies of each are low. Or devices capable of recognizing interruption in electricity, water or even network connectivity and can inform service providers with needs for repair. Imagine smart trash bins that automatically notify garbage collectors that you’re your trashcans are full so they can pick it up.
I would summarize the whole article by stating that our data-driven world is now taking another shift towards Smart Systems. Our current infrastructure has empowered us to overcome major challenges from data generation, ingestion to analysis. In addition to above collective smart scenarios, personal body sensors like GPS chips, health and activity monitors has increased the living standard. I see a rapid growth in IOT in coming months to years.
Most Downloaded Resources
Browse our most popular resources - You can never just have one.