Audi Electronics Venture is a wholly-owned subsidiary of Audi AG with a primary focus on pre-development services for Audi AG.
Initial situation
AEV was looking for a partner to help them implement the SMACK fast data platform for high-performance data analysis for processing vehicle data using streams. The SMACK stack is a collection of different tools that can be integrated with each other to form a big data platform. The name SMACK is an acronym derived from its constituent tools: Apache Spark, Apache Mesos, Akka, Apache Cassandra and Apache Kafka.
Solution
DC/OS is ideal for installing the various SMACK components on the individual computers in a cluster. It is a distributed operating system for 4 data centers and is based on the Apache project Mesos and comes with a repository that makes it very easy to install components such as Spark, Cassandra and Kafka.
A fully operational DC/OS cluster was set up on AWS in collaboration with AEV in just a few weeks. The decision was made to fully automate the platform using Terraform, which is a tool that can be used to describe and manage the infrastructure or an entire data center in code. Terraform code can be executed to create a complete data center on AWS, including all components such as networks, instances, routing tables, and load balancers.
Result
The cluster creation can be started at the push of a button in a Jenkins. In addition, the Jenkins is used to audit who started which job and when. After the Jenkins job has been started, it takes about 30 minutes to install the complete SMACK stack, including all ingest and digest applications.
Log output is still an important source of information on applications. The Elastic Stack was therefore adopted as a monitoring tool to analyze the distributed system's log data. A SMACK cluster has various sources such as Spark, Mesos, Akka, Cassandra and Kafka, where log output can occur.
The Elastic Stack consists of tools such as Elasticsearch, Logstash and Kibana. Elasticsearch is used for storing log data. It is also ideal for full-text searches. Logstash can parse log data and read it from various sources such as files, but also messaging systems or TCP/IP ports, and writes it to Elasticsearch. Kibana can be used to search through data or generate graphical reports. The interplay of all these tools makes it easy to analyze log files from more than 50 computers in one central location.
codecentric AG provided all the Jenkins jobs needed to deploy the DC/OS SMACK cluster using Jenkins DSL. AEV employees were trained to use the cluster so that they would not be left alone with managing the cluster after the initial expert support from codecentric AG.
In addition, a real-time streaming use case was created in collaboration with AEV to process and prepare sensor data. Visualization is performed in a JavaScript application. Zeppelin was also introduced to provide further visualization. Zeppelin is a browser-integrated user interface with notebook-like capabilities for analysts and data scientists to interactively explore their data and perform advanced data analysis. Zeppelin is also used by Amazon Web Services, JuJu, Hortonworks and Twitter to analyze data.
Thanks to codecentric's outstanding expertise in the field of automation and big data technologies, it was possible to put a fast data stack into operation within a very short time. A new release of the resulting platform can be made available within a very short time thanks to the high level of automation.
David Schmitz
Lead Big Data Architect, Audi Electronics Venture GmbH
Any questions about the project?
Are you interested in a custom cloud-based solution for your business? Let's talk.
Further projects of codecentric AG
Find out about further successful projects that we have completed with our customers. Perhaps you will find inspiration for a use case in your company.
Fritz Gerhard