Informed Decisions Thanks to Efficient Data Orchestration
The “FurnSpin Configurator” helps Hettich customers configure furniture professionally – independently where possible and with expert support where necessary.
Tchibo GmbH is an international trading company based in Hamburg with over 10,000 employees worldwide. Founded in 1949, Tchibo started out as a mail-order coffee retailer and is now known for its unique business model: in addition to high-quality coffee specialties, Tchibo offers a weekly changing range of non-food products, from clothing and household items to electronics. With over 900 stores, international online shops, and island shops in supermarkets, Tchibo is represented in numerous countries and achieved a turnover of 3.2 billion euros in 2023.
The project at a glance
- Modernization: Migration of several hundred data pipelines to Apache Airflow to ensure future operation of the company-wide data warehouse
- Optimized operation: Airflow-specific configuration and monitoring functions ensure optimized and scalable operation of the pipelines.
- Cost savings: The migration has reduced operating and licensing costs.
- Foundation for future innovations: The expanded platform creates a reliable basis for the efficient implementation of new data-driven use cases.
Background
Tchibo already works in a data-driven manner in many areas and processes and has established a central, in-house big data analytics platform in the Google Cloud Platform (GCP) for this purpose. The data warehouse solution (DWH solution) contained therein is based on a scalable framework, the data vault approach, and BigQuery for data storage. This means that data from dozens of source systems and different technologies is available to analysts and data scientists in a uniform and centralized manner.
The data warehouse is connected to the heterogeneous system landscape via several hundred data pipelines. A subcomponent for integrating the pipelines has been implemented in SAP Data Intelligence (SAP-DI) to date. In order to continue to reliably provide data for analyses, reporting, or AI systems—e.g., for product recommendations and route optimization—in the future, another alternative needed to be implemented.
Solution
In addition to SAP Datasphere, Apache Airflow was selected for the target architecture as an additional component for integrating the pipelines. Airflow is an open-source workflow management platform and is seamlessly integrated into the GCP ecosystem under the name Google Cloud Composer. After a brief evaluation and planning phase, a “lift and shift” approach was favored and implementation began. This allowed existing workflows to be migrated to a new environment with minimal adjustments.
The implementation was carried out with the aim of setting up as many of the pipelines as generically as possible and reducing both manual intervention and pipeline downtime to a minimum. To this end, the existing framework was further expanded and effective use of the Airflow API was implemented. For example, the existing central configuration and orchestration approaches have been integrated into Airflow, enabling further steps towards seamless integration into the Airflow ecosystem.
At the same time, additional data sources were connected and the operation of the DWH was ensured during the migration phase. Intensive cooperation within the team and with other teams and stakeholders in the company was established to coordinate the migration.
Result
By successfully migrating numerous pipelines to Apache Airflow, Tchibo was able to maintain data warehouse operations, prepare the platform for future tasks and challenges, and reduce operating and licensing costs.
In addition, the Airflow-based solution now enables simplified management and more targeted scaling of data processes thanks to its advanced configuration and monitoring capabilities.
The expansion of the in-house framework and the integration of Airflow thus lay the foundation for further innovations in the data-driven environment. The ability to efficiently connect, manage, and monitor additional data sources enables Tchibo to master future data integration requirements, thereby strengthening Tchibo's position as a data-driven, customer-oriented omnichannel provider.
Andreas Mastel
Lead Data Engineering & Artificial Intelligence, Tchibo GmbH
Our collaboration with codecentric was professional and cooperative throughout. We particularly appreciated the open exchange of knowledge and our shared understanding of high-quality, code-centric data engineering solutions.
Andreas Mastel
Lead Data Engineering & Artificial Intelligence, Tchibo GmbH
Any questions about the project?
Are you facing the challenge of modernizing your data processes and making them more efficient? Feel free to contact me! We are happy to support you on your journey to becoming a data-driven organization.
Consultant Data & AI
Further reference projects
Find out about other successful projects that we have completed with our customers. Perhaps you will find inspiration for a use case in your company here.
Berthold Schulte
Consultant Data & AI