Streamlined IoT at Scale with iguazio
Yaron Haviv | October 10, 2016
iguazio announced its revolutionary Enterprise Data Cloud platform on September 27th, designed with a modern approach from the ground-up. iguazio's platform serves millions of requests per second, scaling to petabytes and managed as a PaaS with extreme simplicity. We built an end to end IoT application for monitoring and controlling smart connected cars to demonstrate the magnitude of innovation.
The demo spans a hybrid cloud environment with Iguazio running in a hosting facility (Equinix), Spark and Docker micro-services in Amazon AWS, mobile apps and connected model cars (running Raspberry Pi).
As illustrated below, cars and mobile devices interact with Iguazio directly through AWS-compatible, RESTful APIs (S3, Kinesis and DynamoDB). The cars generate real-time sensor data, produce live video feeds, read configurations and accept real-time operational commands:
Real-time events and statistics are immediately aggregated and sent to live AngularJS-based dashboards within less than a second from the event - without any intermediate services or micro-batch processing that may slow it down. Dashboards use advanced APIs to run real-time queries and generate geo live maps with summaries, real-time aggregations and real-time event enrichment that correlates the stream data with historical information from multiple tables.
At the same time, complex aggregations and machine-learning algorithms generate predictions using Spark, listening to the same data feeds in parallel. Data is accessed a high-performance data source API with analytics offload (predicate push-down and real-time enrichment of the stream with historical state). The resulting DataFrames are sent back to iguazio from Spark, with vector operations to update car, driver and geo data. Spark is also used to run OLAP queries on the stored data and show the results in the Dashboard, or interactively by using Apache Zeppelin.
Various triggers are handled in a set of real-time event-processing Docker micro-services. They listen on an elastic message queue (stream) generated by iguazio and process trigger events while combining them with state and configuration data stored as tables (using the real-time enrichment features). These microservices are able to update any of the data items in iguazio (car, driver, geo, etc.), or send commands directly to the cars through downstream message queues.
Building such an application using existing technologies would take months and require dozens of servers to run it in production. It took us just a couple of weeks with a few part time developers and only a few thousand lines of code in Java, Scala, Python and Angular. No need for a complex infrastructure setup, no performance tuning and with robust security that is managed through policy without a single line of code.
The demo ran live at our Strata+Hadoop NY booth.
