The Internet of Things (IoT) and big data are two expansive, complex elements that have become extremely valuable in the contemporary landscape. While being distinctive, both of these ideas are, to an extent, interrelated. While IoT comprises millions of devices that are used to collect and communicate information, big data is known to encompass a much greater landscape.
In a certain sense, IoT can be considered to be a series of rivers and creeks that feeds into a large ocean of big data. Discerning the collection of connected devices, sensors, and other “things” that represent the IoT makes a notable contribution to the volume of data collected by a company.
The use cases of IoT today span a number of industries and sectors, ranging from agriculture to construction. Sensors are especially used in several construction sites today for asset management and prompt alerts. Tools created for analytics and big data are extremely useful when it comes to corralling the influx of data streaming in from varying IoT devices.
Case study: Use of Big data and IoT in construction
The construction industry has developed considerably over the years. Nowadays, one can find a number of advanced techniques and tools being used at construction sites that help in streamlining the diverse tasks taking place there, while ensuring superior efficiency and productivity.
In the following case study, we would get to understand how one of the leading construction companies was able to optimize their asset utilization and monitoring processes by harnessing the power of Big Data and IoT.
Business problem: The majority of machinery used in the construction industry is quite expensive. Hence, companies usually rent them out, rather than purchasing. This makes the need to monitor their usage extremely important, and increases the need for proper metrics measurements and alerting of non-standard equipment operations.
Any type of non-standard operations like running earthmovers on high RPM and tower cranes picking up overload than suggested range can lead to machinery failure, which will impact both business operations and paying hefty fines. On-time alerting will aid the site managers to intervene in a timely manner, while Real Time dashboards can help optimize the utilization of resources.
Challenge: As per the current use case, construction sites have more than 32,000 assets, which emit about 20,000 events/s. These numbers additionally keep growing, making it a challenging task for the scaling compute clustersr to handle such load. The absence of hardware subsequently leads to a delay in processing and a backlog of events. As a result, real-time events get inordinately delayed, rendering them to be useless.
Solution: An auto-scalable stream processing pipeline with the usage of Spark Structured Streaming on Databricks was implemented. CosmosDB was additionally used as a lookup store for high throughput window calculations. This ultimately resulted in efficient and in-time processing of all events, subsequently generating all the alerts and real-time dashboards in a timely manner
Usage of IoT metrics aggregation and alerting: In the use case provided, construction-site machinery, earthmovers, and material transport, which emit IoT events are involved. The prime goal was to compute and aggregate various metrics on a near real-time basis and generate alerts. This was a complex issue as the events came in at a rate of 20,000 events/s. To solve the problem, and make asset utilization easier, CosmosDB was used as a fast lookup store in order to cache intermediate events and computations.
Business impact: On-time alert generation led to timely intervention at the construction site for reducing and correcting the non-standard operations and real-time dashboards for better utilization and allocation of resources. It adds direct benefits to the costs associated with operations.
IoT and Big Data: Delivering complementary solutions
As witnessed in the case study provided above, while Big data and IoT might be distinctive ideas, Big Data Technologies complements IOT for ultimate success in several cases. Both of these technology concepts focus on the need to convert data into tangible insights that can be acted upon.
Companies desiring to harness the power of data must carefully consider the devices they opt to deploy, as well as the type of information they ultimately collect. Gathering useful and applicable data, and using powerful applications and software to process in a sector-specific way shall make the process of analytics highly fruitful.