IT Consultant | Data Engineer

The data landscape in an organization often looks like this: teams gather and produce data everyday. Each team develops their own metadata models and documentation, if there is any at all. Governance policies exist in scattered documentation (spreadsheets, wikis, Confluence pages) and are most of the time outdated. Data quality enforcement happens manually. The usage of your data is monetized mostly manually, if it is monetized at all. More often you see one time exports in CSV files sent around. Service Level Agreements are often communicated in meetings that are not documented, so there are no machine-readable commitments that can be monitored.\nYou end up with datasets that cannot scale beyond their original team, cannot be discovered or evaluated by AI agents, cannot cross organizational boundaries without extensive manual coordination.\n\nFinding the Right Data Without Standards [Symbolic Image]\n\nStart Thinking in Data Products\nYou should start to think in data products. The previously described state of organization’s data landscape is mostly based on datasets in contrast to data products. Datasets are generated without a consumer in mind. They are the outcome of a team’s daily work and lack company wide intention. Data products on the other hand serve a purpose. They are business-oriented and solve specific problems. That said, the value of data products is much higher for an organization than the value of datasets.\nWhy Standards Could Help\nData products are most valuable if your whole organization has the same understanding of data products. Each team knows how to assess a data product if there is a universal language. The perfect situation to introduce a standard. Without a standard all teams invent their own format of data product metadata and document it in different places. Data products cannot be discovered by AI agents automatically. Manual discovery is also only possible if you have tribal knowledge.\n\nOrganizing Data Products With ODPS [Symbolic Image]\n\nOpen Data Product Specification (ODPS)\nThe \nOpen Data Product Specification (ODPS)\n is the standard to address this problem. It is a machine-readable YAML standard designed to describe data products in a consistent and interoperable way. Developed as a vendor-neutral, open-source initiative it provides a unified language that can be interpreted in an automatic way.\nODPS (v4.1) has been designed to provide information about \ntechnical, business, legal, and ethical\n aspects in ten core elements:\n\n\nTechnical Aspects (Infrastructure & Data Access)\n\nData Access\n\nHow to technically access the data (e.g. APIs, files, SQL, AI/MCP)\n\nData SLA\n\nTechnical service commitments (e.g. uptime, latency, response time)\n\nData Quality\n\nTechnical quality dimensions and validation rules\n\nData Contract\n\nTechnical expectations and schema definitions\nBusiness Aspects (Product Strategy, Pricing, and Plans)\n\nData Product Details\n\nBusiness metadata (e.g. name, description, value proposition, use cases)\n\nData Product Strategy\n\nBusiness objectives and KPI alignment\n\nData Pricing Plans\n\nPricing models and monetization strategy\n\nPayment Gateways\n\nTransaction processing for monetization\nLegal Aspects (Licensing & Intellectual Property Rights)\n\nData Licensing\n\nLegal terms, usage rights, geographical restrictions\n\nData Holder\n\nLegal entity responsible for the data product\nEthical Aspects (Data Privacy)\n\nData Licensing\n\nPrivacy terms, data subject rights, confidentiality\n\nData Access\n\nAccess control and authentication (who can access what)\n\nData Holder\n\nAccountability for ethical data handling\nEverything-as-Code\nThe key feature of ODPS is that everything is machine-readable code. Traditionally you have the problem that governance documentation exists – if at all – in Word documents, PDF files or the internal wiki. To enforce a policy, you manually translate documentation into code that can monitor the SLAs or your data quality. You need to do this every time for every product, while this process is error prone. \nODPS eliminates the downsides of manual translation since the specification is the source for automation.\nSLA:
  declarative:
    default:
      [...]
      dimensions:
        - dimension: uptime
          displaytitle:
            en: Uptime
          objective: 90
          unit: percent
  executable:
    - dimension: uptime
      type: prometheus
      reference: 'https://prometheus.io/docs/prometheus/latest/querying/basics/'
      spec: |-
        avg_over_time(
          (
            sum without() (up{job="prometheus"})
              or
            (0 * sum_over_time(up{job="prometheus"}[7d]))
          )[7d:5m]
        )\nWith the code example in the executable part you can automatically generate monitoring dashboards in Prometheus in your CI/CD pipeline. The metadata becomes executable infrastructure. As ODPS supports reusable components, you only have to change one YAML file to update production across 100 data products. This is infrastructure-as-code applied to data products.\nWhere You Could Start\nODPS standardizes data product metadata for automation. Developers get reusable components and tooling integration. Decision makers get faster scaling, reduced governance overhead, and vendor independence. The standard is production-ready, actively maintained, and backed by Linux Foundation.\nODPS makes sense for you, if you have multiple data products requiring consistent governance, want to monetize your data (internal or external), want to scale governance without headcount, have a data mesh architecture, or want to provide structured metadata to your AI agents to gain valuable insights from your data products\nTo see if the Open Data Product Specification provides value to your organization, I recommend the following: Pick one of your existing data products and map its metadata to the ODPS format to see what gaps the specification reveals in your current documentation.\nYou can simply start by managing ODPS files in git to streamline your documentation and automate your governance. In the future you could introduce an ODPS-supporting data marketplace like \nEntropy Data\n or \nAlation\n to gain even more value from your data products.\n\n\n⁠Sources: 
⁠https://unsplash.com/photos/office-table-with-pile-of-papers-6jA6eVsRJ6Q
⁠https://unsplash.com/photos/multi-story-white-painted-library-interior-RtQ1CTQMySk
⁠https://opendataproducts.org/v4.1/images/ODPS-design-971d4b0b.png

Discover how ODPS brings order to data chaos. Learn why thinking in data products and using standards transforms how organizations manage and monetize data.

ODPS: The Standard for Data Products

codecentric Blog

Head of Data

Building customer-facing analytics, e.g. directly built into the web application they are using, has always been tricky. The requirements are different from in-house BI: users expect sub-second responsiveness, queries can - to a certain degree - be unpredictable and it must be built without overloading operational systems. A few caveats:

- When building directly on your production infrastructure and database, there is a good chance you will overload those systems with the analytical workload. Also, those technologies often do not excel for analytical query patterns, but focus on classic transactional queries.
- Specialised analytical system like warehouse a often heavy-weighted: expensive, slow to provision and focusing on longer running queries
- Also serverless query engines like Athena are quite complex. They require setup, IAM and tuning that are overkill for simple embedded analytics.
- Overall, the client-server model is still challenging. Managing clusters and configurations add operational overhead.
- The user is not interested in any of this. They just want instant and quick UIs, fast and snappy dashboards, which can be easily filtered along different dimensions.
- Especially in SaaS scenarios you need to ensure tenant separation. Not only should each tenant only see their data. Often it is also required they are stored separately

The result: teams either overspend on heavyweight solutions or underdeliver on responsiveness. Often the solutions are built on the application infrastructure and technology, not made for those cases.

### How MotherDuck and DuckDB Help

MotherDuck extends DuckDB — the in-process analytical database — into the cloud. Together, they offer a lightweight yet powerful model that is well-suited for customer-facing use cases:

1. **Lightweight architecture**: No heavy OLAP cluster, no complex IAM, no server fleet to manage. DuckDB runs embedded in your app or service, and MotherDuck adds shared, cloud-scale persistence.
2. **Decoupled storage and compute**: Compute is only consumed, when a query is executed. The data itself can be stored in an inexpensive blob-storage - managed by yourself or Motherduck.
3. **Tenant separation made easy**: Next to common SQL-based strategies like separate schemas per tenant, with DuckDB you can go one step further and could easily use one database per tenant. Each database can be one separate file, stored in its own path. In Motherduck you can take this even further and assign every tenant a separate duckling - a compute instance that can be scaled independently. So every tenant gets the compute power it needs.
4. **Dual query execution**: You can query local data (e.g. Parquet, CSV, cached DuckDB tables), data stored in your cloud provider and remote data in MotherDuck’s cloud in the same SQL. Motherduck distributes the query workload, so that it’s “data-local”, meaning the parts of the query are executed where the data is stored, and network transfers are minimized.

The dual query execution can also be used for caching tables. Customer facing analytics is often on a specific dataset, where just filters or aggregations are changed. The underlying dataset keeps the same. With the dual query execution one could easily build a system where the first query reads the data from a cloud source like Motherduck, caches the data locally and uses the cached data for subsequent queries. This uses the DuckDB feature to create a local temporary table.

```sql
CREATE TABLE main.local_cache AS SELECT * FROM ({userQuery})
```

### Example: Local Caching + Cloud Data

Here’s a simple SQL example showing how to combine local caching and remote cloud data with MotherDuck:

```sql
ATTACH 'md:sample_data';

CREATE OR REPLACE TABLE main.local_cache AS
SELECT *
FROM
  (SELECT date_trunc('day', created_date) AS date,
          agency,
          count(*) AS request_count
   FROM sample_data.nyc.service_requests
   WHERE created_date &gt;= '2021-01-01'
     AND created_date &lt;= '2021-01-31'
   GROUP BY ALL
   ORDER BY 1 ASC);

EXPLAIN
SELECT *
FROM main.local_cache;

SELECT agency,
       sum(request_count)
FROM main.local_cache
WHERE date &gt; '2021-01-16'
GROUP BY ALL
ORDER BY 2 DESC
```

What’s Happening Here

1. We run a query over a small time slice on data that is stored remotely in motherduck. The result is cached locally as a temporary table.

2. The explain of a select on the cached table shows that it processed locally, indicated by (L) next to the operator name, here the SEQ_SCAN. *![query_plan_simple.jpg](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/cmfwwn0eyg84i07tb0ukd6bde)Query-Plan für ein einfaches lokales Select*

3. We further narrow down the data, additional group bys, more filtering and different sorting. All running locally.

As DuckDB can be used in all relevant programming languages (python, typescript, java, rust, go, php and more), this can be easily extended to a pattern where a starting data set is load, queries on this data will be executed locally and more data can be loaded from remote.

For the last part one could use the dual execution functionality of Motherduck, making it possible to combine local data and remote data:

```sql
SELECT *
FROM main.local_cache
UNION
SELECT date_trunc('day', created_date) AS date,
       agency,
       count(*) AS request_count
FROM sample_data.nyc.service_requests
WHERE created_date &gt;= '2021-01-02'
  AND created_date &lt;= '2021-03-31'
GROUP BY ALL
```

Running this query with EXPLAIN will show the Physical Plan. The “Download Source” Operator indicates that a part of the data is downloaded from Motherduck and then unioned with the local data. 
*![query_plan_union.jpg](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/cmfwwnmjhg77l07ukdgvxpfpe)Query-Plan für ein SELECT UNION mit lokalen und remote Daten*

### Takeaways
For customer facing analytics, the balance is always between speed, cost, and simplicity. MotherDuck + DuckDB hit a sweet spot:

- No heavyweight infrastructure to operate.
- No overloaded OLTP databases.
- No complex cluster tuning.
- Just fast, federated analytics, with explicit caching where it matters most.

If you’re building last-mile analytics into your SaaS product or customer portal, you don’t need to choose between an overloaded OLTP backend and an overengineered OLAP system. With MotherDuck and DuckDB, you can cache hot queries locally for instant dashboards and use cloud-compute power for more complex queries. The result: analytics is fast for the enduser, but also simple to build and maintain for the engineers.


Motherduck and duckdb help building customer facing data applications by providing easy to use compute, tenant separation and fast and responsive queries.

Customer-facing analytics with DuckDB and MotherDuck

DuckDB and MotherDuck for customer facing analytics

IT Consultant & Developer

Service Lead "Software Modernization" & People Lead

[Vibe Coding](https://www.codecentric.de/wissens-hub/blog/vibe-coding-was-nun) is a programming approach that delegates virtually every task involved in working with source code—from understanding to creation to modification—to a GenAI, placing almost complete trust in the output of these kinds of AI. Based on a recent [architecture and code review](https://www.codecentric.de/en/services/software-modernization#entry-offers) of a Space-as-a-Service platform (SPaaS platform) that was built from scratch with Vibe Coding, we provide an assessment of the suitability of this programming approach for the implementation of complex software products.

**Spoiler:** Without context-based structuring and orchestration of the used GenAI, the quality deficits of generated code outweigh supposed productivity gains.

## The lure of speed—Anyone can develop now(?)

The promise of Vibe Coding is huge: with simple instructions in natural language ([prompting](https://www.codecentric.de/en/knowledge-hub/blog/how-to-program-my-llm)), entire applications can be created in no time at all and without any knowledge of software development. This is made possible by increasingly powerful GenAIs based on [Large Language Models](https://www.codecentric.de/wissens-hub/blog/grosse-sprachmodelle) such as Claude, Gemini, or GPT. With manageable costs and a growing selection of agents, tools, and platforms, the question arises: Can anyone now implement productive software, and will developers have to look for new jobs sooner or later?

We were confronted with this question when a customer recently tasked us to review a prototype of an SPaaS platform created entirely with Vibe Coding. The platform was developed with no programming experience within a few weeks using a small number of prompts and a specialized Vibe Coding portal. In addition to its core functionality of offering and booking locations, the platform integrates other complex functions such as user management, chat, and payment in a modern UI. Despite its prototypical nature and relatively small codebase (approx. 50,000 lines of TypeScript), initial tests showed that the application is fundamentally executable.

The aim of the commissioned architecture and code review was to assess whether the source code generated by GenAI represents a high-quality basis for bringing the prototype of the SPaaS platform [to market maturity](https://www.codecentric.de/en/services/production-readiness-review) in the medium term.

In the following, we report on our findings from the review, and discuss the opportunities and risks of implementing software products with Vibe Coding, which still exist despite a large number of specialized providers.

## The analysis—What our review revealed

For the review of the AI-generated SPaaS platform, we drew on [established architectural evaluation methods](https://www.codecentric.de/en/knowledge-hub/blog/pull-off-architecture-reviews-at-light-speed-with-lasr) and used analysis tools for the Node.js ecosystem on which the application is ultimately based. The following paragraphs describe our findings for each quality deficit identified.

**Vendor lock-in:** For the review, we downloaded the source code of the SPaaS platform from the used Vibe Coding portal and first checked it for possible malicious code. We then attempted to run the application locally. To do this, we first followed the AI-generated setup documentation, but this proved to be insufficient. After minor adjustments, the platform could be started locally, but registration and login could not be executed without errors. In addition, it remained unclear how the database management system required could be connected correctly to the platform. The setup documentation lacks the necessary information. Since the Vibe Coding portal can operate the platform without errors and additional configuration, there is a significant risk of vendor lock-in with the portal.

**Risky dependencies:** For the vast majority of security-critical functions, such as user management and payment, the GenAI followed the best practice of integrating with proven external libraries and frameworks. However, some of these were redundant or competing, such as the use of different hash methods for the same purpose, which can lead to runtime problems that are difficult to locate. In other cases, the AI referenced versions of dependencies that are outdated or whose production readiness is unclear. In one particularly serious case, certain user management functions relied on an external dependency that had not undergone any significant further development since 2014\. This was only noticed because the version number appeared suspicious (less than 1.0) and we consequently checked the associated GitHub repository. The integration of outdated dependencies and the associated high risk of security vulnerabilities is particularly problematic for security-critical aspects of a web application that processes user sessions and customer data.

**Lack of architecture-related security:** While certain security features such as password hashing were successfully coded by the GenAI, we found them completely absent at the architectural level. For example, communication between the frontend and backend, which are essential components of the SPaaS platform's three-tier architecture, is not encrypted. All calls to the backend API by the frontend are made in plain text via HTTP. This means they can be relatively easily intercepted and manipulated, opening the door to man-in-the-middle attacks, session hijacking, and credential theft.

**Improvable code quality:** We used established static analysis tools to assess the quality of the AI-generated code. Unlike the security risks described above, the deficiencies identified in this process are initially less serious for a prototypical application such as the SPaaS platform under review. Nevertheless, they will become relevant at the latest after successful market validation, when the focus shifts to ensuring medium- to long-term software quality, especially maintainability. However, some of the code-related deficiencies may also come into play at an earlier stage if they cause runtime errors that affect platform operation in unforeseen ways. In fact, the reviewed source code has great potential for improvement, as a large number of smells were uncovered–ranging from high cognitive complexity of individual methods to unused or incorrect `import` statements and incorrectly implemented concurrency. In addition, we encountered strong coupling due to a lack of separation of concerns, a significant amount of unused code fragments, and a division of modules into folders that did not follow standard best practices. Taken together, all of the above findings suggest that the employed GenAI in conjunction with unstructured Vibe Coding exhibits weaknesses when it comes to extending the codebase: The integration of new features does not seem to be sufficiently accompanied by quality-oriented refactoring which includes, for example, purposeful modularization, establishing reusability, or the removal of obsolete code.

**Inadequate test coverage:** The risk of significantly reduced maintainability is increased by inadequate test coverage. Although the GenAI produced tests during Vibe Coding sessions even without explicit prompting, these tests do not validate business processes such as payment processing or the creation and booking of advertisements. Instead, they focus on the technical connection to the database by issuing `INSERT` statements and verifying the correct execution of these statements.

## The bottom line—Consequences of unstructured Vibe Coding

In summary, the use of Vibe Coding in the case of the reviewed SPaaS platform led to a number of quality deficits of varying criticality. Despite the prototype stage, the security issues identified should be resolved in the short term, and in particular before the application is validated on the market. If market validation is successful, vendor lock-in and poor maintainability should be addressed during further development, as both deficits represent technical debt that can lead to significant follow-up costs in the medium term and thus impair competitiveness.

The development of software systems is often compared to building a house. If we stick with this analogy, the use of unstructured Vibe Coding likely results in the construction of a house with fragile foundations: the façade may look good and some rooms may be usable, but others may not and the entire building may be at the risk of collapse after moving in. Initial time savings then lead to a mortgage with unacceptably high interest rates, so that in the worst case the costs of renovation exceed those of a redevelopment. Additionally, these costs are usually very high for supposedly finished houses or productive software applications–and in both cases experts are needed to gradually uncover and fix the problems. Consequently, new job profiles such as “[Vibe Coding Cleanup Specialist](https://t3n.de/news/vibe-coding-chaos-diese-menschen-reparieren-schlechte-ki-codes-1708009/)” currently emerge.

In conclusion, based on our review of the AI-generated SPaaS platform, we can say that Vibe Coding and specialized portals enable people with little to no experience in software development to quickly generate usable prototypes. However, without special precautions, these applications do not constitute sustainable, secure, and maintainable products. [The dream of the “Citizen Developer” will not automatically come true](https://www.codecentric.de/wissens-hub/blog/ersetzt-ki-die-softwareentwickler) in the AI age because it neglects the complexity of professional software development.

## Our recommendation—Competitive advantages through structured Vibe Coding

If we disregard our findings, the use of GenAI accelerates the development of software systems–at least in the short term. This raises the question of how this speed advantage can be sustained in the medium to long term by ensuring that generated code is and remains of high quality. The answer to this question lies in combining AI tools with human expertise and structured methods.

Instead of relying solely on the current vibe, GenAI should be given guidelines, for example with approaches such as [Product Requirements Prompts](https://github.com/Wirasm/PRPs-agentic-eng), [Context Engineering](https://github.com/coleam00/context-engineering-intro), or [BMAD](https://github.com/bmad-code-org/BMAD-METHOD) (Breakthrough Method of Agile AI-Driven Development). These structured methods have in common that the product idea is defined together with coarse-grained technical specifications, such as the desired architecture style, and precise requirements for technical implementation. Together, this structured information provides the context and a plan for development that GenAI and its agents can follow. This can be illustrated very well by the BMAD method and the following mnemonic:

* **B**ig Picture: The Big Picture explains the overall goal of the project to GenAI (*“We are building a web-based SPaaS platform for...”*).  
* **M**ethodology: The Methodology sets clear rules and guidelines (*“Use React in the frontend, Express.js in the backend, and write all code in TypeScript.”*).  
* **A**ction: The Action formulates precise, atomic instructions (*“Create an Express route in the `routes.ts` file that...”*).   
* **D**etails: The Details provide further information on implementing the action (*“Define the API endpoints using OpenAPI”*).

This information is usually stored in dedicated files in Markdown format in the repository of the application in question. The files follow a specific naming scheme so that GenAI can automatically identify them as context-relevant. This means that information about the application's intended use, its architecture, and the frameworks for its implementation does not have to be specified repeatedly for each code generation by the AI and is also subject to versioning when applying a revision control system such as Git. For prompts such as “Integrate a login button,” GenAI takes the provided context into account. This is not the case with unstructured Vibe Coding that lacks precise guidelines to limit the degrees of freedom of GenAI, so that the probability of producing deficient code is much higher than with a structured method.

The benefits of structured methods for AI-supported coding are obvious: with a clear target vision and unambiguous rules, GenAI can generate high-quality, secure, and maintainable code at high speed from natural language prompts, while technical debt remains relatively low. This is especially true when combined with deterministic approaches to ensuring high software quality, such as test automation and Infrastructure as Code, as well as the involvement of human expertise, which can assess, edit, and contextualize AI decisions in case of doubt.

## Getting ready for the AI-supported future of software development

What does this mean concretely for our SPaaS customer? In the next step, we empower them with our [AI-Assisted Coding Workshop](https://www.codecentric.de/en/services/generative-ai/ai-assisted-coding-workshop), in which we teach the practical basics of structured AI-supported software development so that they can independently increase the quality of their platform with GenAI and ensure its long-term success when entering the market. For those who are still unsure how to find [the right use cases for GenAI](https://www.codecentric.de/wissens-hub/blog/wie-finde-ich-die-richtigen-generative-ki-use-cases-5-learnings-aus-der-praxis), we recommend an [AI Use Case Workshop](https://www.codecentric.de/en/services/generative-ai/ai-use-case-workshop) to identify and strategically prioritize promising AI use cases.

## Conclusion

AI is not an autopilot that can generate complex, production-ready applications from simple instructions in natural language. Rather, in the right hands, AI is an extremely powerful co-pilot that can in fact accelerate software development. The Vibe Coding approach is useful for quick experiments such as clickable prototypes. However, building robust platform products that meet common security requirements and are maintainable in the long term requires more: a structured approach to the use of AI tools and trained developers. Therefore, it is important to invest not only in AI tools, but also in building knowledge to use them effectively. This makes the difference between an expensive experiment and a real competitive advantage.

Ein Roboter versucht aus Prompts eine Anwendung zu bauen. Die Anwendung sieht sehr instabil aus und droht einzustürzen.

A robot attempts to build an application from prompts. The application appears very unstable and threatens to crash.

A review of AI-generated code reveals the serious quality and security risks of unstructured Vibe Coding. Implement structured methods to leverage GenAI safely.

Vibe Coding: A Practical Review of GenAI-Generated Code

Where Vibe Coding helps—and where it doesn't: A field report

IT Consultant

I don’t think anyone will be surprised when I say that SQL is not the nicest language to work with. Some might even say that it has terrible ergonomics, especially for larger and more complex queries. Still, there are very good reasons why SQL is *the choice* for accessing and processing data. Fortunately, DuckDB’s friendly SQL dialect tackles a lot of those issues while retaining the familiar style of classic SQL.

The main issue with maintaining large and complex SQL queries boils down to the [syntactical order being very different from the semantic order](https://www.kdnuggets.com/the-essential-guide-to-sql-execution-order). This has to be kept in mind when writing queries, making it unintuitive and harder to understand unfamiliar queries: for example, the logical starting point of a query is usually buried somewhere in the middle, and adding [additional subqueries is necessary](https://www.cidrdb.org/cidr2024/papers/p48-neumann.pdf)<sup>PDF</sup> to use previous results. But it can also impair tool support, for example [autocompletion of column names](https://blog.jooq.org/a-beginners-guide-to-the-true-order-of-sql-operations/) before the `FROM` clause has been written.

To show that this can be improved significantly, I’ll start with a *somewhat realistic example* in PostgreSQL dialect as a stand-in for classic SQL. Then, I’ll go over some of DuckDB’s friendly SQL features and show how they make the same example easier to read and maintain. But before I go into the details, some notes for additional context:

---

* Do you prefer a hands-on approach?
    * Check out [my GitHub repository](https://github.com/LennartH/friendly-ducking) to run the examples locally.
    * Or use the web IDE from [MotherDuck](https://motherduck.com/) (free account required) and access my example data with this share: `ATTACH 'md:_share/duckdb-friendly-sql-example-data/0c374195-c236-4b6f-a8ce-02595d90cba6'`
* DuckDB is not the only dialect that improves on classic SQL, it just happens to be one I like a lot. So the shown features might exist elsewhere, too.
* The shown features are just a personal selection, ranging from subtle to fancy, and are by no means an exhaustive list. You might not like all of them (or my formatting), but since DuckDB is [largely compatible with PostgreSQL](https://duckdb.org/docs/stable/sql/dialect/postgresql_compatibility), you can cherry-pick to your liking.

---

## A *somewhat realistic example*: Rubber Duck KPIs

Minimal examples are great for showing how a feature works, but aren’t enough to showcase their impact on things like developer experience. So, I tried to create a *somewhat realistic example*: big enough to be meaningful, but not so large that it becomes unwieldy. This way, I can show how each improvement makes a difference.

My query of choice selects a specific product category, adds some extra properties and aggregates a few KPIs grouped by product and customer country.

```sql
WITH
  enriched AS (
    SELECT
      p.product_id                   AS duck_id,
      p.product_name                 AS duck_name,
      COALESCE(p.color, 'Unknown')   AS duck_color,
      COALESCE(p.material, 'Rubber') AS duck_material,
      c.country                      AS customer_country,

      oi.quantity,
      oi.unit_price,
      COALESCE(oi.discount_pct, 0)   AS discount_pct,
      p.base_cost,

      (
        COALESCE(oi.discount_pct, 0) &gt;= 0.20
        OR COALESCE(p.color, '') IN ('Gold', 'Platinum')
        OR COALESCE(p.material, '') ILIKE '%silicone%'
      )                              AS is_premium_duck
    FROM orders      AS o
    JOIN order_items AS oi USING (order_id)
    JOIN products    AS p  USING (product_id)
    JOIN customers   AS c  USING (customer_id)
    WHERE p.category = 'Rubber Duck'
  )

SELECT
  duck_id,
  format('%s (%s, %s)', duck_name, duck_color, duck_material) AS duck_description,
  customer_country,
  SUM(quantity * unit_price * (1 - discount_pct))                            AS net_revenue,
  SUM(quantity * base_cost)                                                  AS cogs,
  SUM((quantity * unit_price * (1 - discount_pct)) - (quantity * base_cost)) AS gross_margin
FROM enriched
WHERE is_premium_duck
GROUP BY duck_id, duck_description, customer_country
ORDER BY duck_id, customer_country;
```

Some points that make this kind of query harder to maintain and reduce developer experience:

* Lots of repeated, similar expressions. This makes it cumbersome to change and easy to introduce inconsistencies.
* Not easy to read from top to bottom. For me, the logical starting point would be the `FROM` clause of the `enriched` CTE.
* I would prefer if the calculation of the individual KPIs were not part of the aggregation in the final projection. This can be easily done by introducing another CTE or some subqueries, but this would add more repetitions ([more verbose example](https://github.com/LennartH/friendly-ducking/blob/main/00-standard-sql/00-standard-sql.sql)).
* The last one is a personal preference, but I do not like the default alias syntax. The column name can provide helpful context for understanding an expression. `expression AS alias` changes the text directionality to a half-baked right-to-left.

In the following sections, I’ll introduce the friendly SQL features I chose with minimal examples and what the *somewhat realistic example* looks like with them applied.

## Friendlier Syntax

The first features for a friendlier SQL are purely syntactical. Some are subtle, but others can take some getting used to. Especially if you have been working with SQL for a long time, the reading patterns you’ve formed might collide with the changed structure.

**FROM-first syntax**

This one simply lets you put the `FROM` and `JOIN` clauses before the `SELECT` statement.

```sql
FROM orders o
JOIN customers c USING (customer_id)
SELECT o.order_id, o.order_date, c.country;
```

This makes it a lot easier to read queries from top to bottom. For more details, check out the [official documentation](https://duckdb.org/docs/stable/sql/query_syntax/from.html#from-first-syntax).

**Trailing commas**

Many modern programming languages allow trailing commas in different places, and for good reason. Being able to reorder projections or comment out lines without thinking about it makes working on queries a lot smoother.

```sql
FROM products
SELECT
  product_id,
  product_name,
  base_cost,
;
```

It's something you don't really notice at first, but it is sorely missed once it's gone. Unfortunately, trailing commas aren’t supported everywhere. [Most notably in CTEs](https://github.com/duckdb/duckdb/discussions?discussions_q=trailing+comma), but `ORDER BY` doesn’t allow them either.

**Prefix aliases**

Putting the column alias before its expression is not a subtle change and is absolutely a matter of personal preference.

```sql
FROM products
SELECT
     duck_id:  product_id,
   duck_name:  product_name,
  duck_color:  color,
WHERE
  category = 'Rubber Duck';
```

I find it makes queries easier to read, but I can see how it can complicate formatting and might feel unnatural when you’re used to `expression AS alias`. Ultimately, it’s more important to pick a style and stick with it than which one is chosen. Like trailing commas, prefix aliases aren’t supported everywhere. For example, you can’t use them in the star and `COLUMNS` expressions (which I’ll cover later).

**A friendlier *somewhat realistic example***

Applying all three of these features to the example query is a solid first step toward a better developer experience.

```sql
WITH
  enriched AS (
    FROM  o: orders
    JOIN oi: order_items USING (order_id)
    JOIN  p: products    USING (product_id)
    JOIN  c: customers   USING (customer_id)
    SELECT
               duck_id:  p.product_id,
             duck_name:  p.product_name,
            duck_color:  COALESCE(p.color, 'Unknown'),
         duck_material:  COALESCE(p.material, 'Rubber'),
      customer_country:  c.country,
  
           oi.quantity,
         oi.unit_price,
          discount_pct:  COALESCE(oi.discount_pct, 0),
           p.base_cost,
  
       is_premium_duck:  (
         COALESCE(oi.discount_pct, 0) &gt;= 0.20
         OR COALESCE(p.color, '') IN ('Gold', 'Platinum')
         OR COALESCE(p.material, '') ILIKE '%silicone%'
       ),
    WHERE p.category = 'Rubber Duck'
  )

FROM enriched
SELECT
           duck_id,
  duck_description:  format('{0} ({1}, {2})', duck_name, duck_color, duck_material),
  customer_country,
       net_revenue:  SUM(quantity * unit_price * (1 - discount_pct)),
              cogs:  SUM(quantity * base_cost),
      gross_margin:  SUM((quantity * unit_price * (1 - discount_pct)) - (quantity * base_cost)),
WHERE is_premium_duck
GROUP BY duck_id, duck_description, customer_country
ORDER BY duck_id, customer_country;
```

## Reusable column aliases

In classic SQL, you can’t reuse aliased expressions within the same scope. This leads to lots of repeated code or extra subqueries and CTEs. Complex queries can quickly become hard to maintain and balancing its length and legibility is not an easy feat.

The way [DuckDB allows using aliases directly](https://duckdb.org/2022/05/04/friendlier-sql.html#column-aliases-in-where--group-by--having) is a major reason why I consider it a game changer. It enables us to think in a more direct fashion when constructing queries. Chaining calculations together in small increments feels very smooth and is easier to follow.

```sql
FROM order_items
SELECT
  discount_price:  unit_price * (1 - discount_pct),
     total_price:  discount_price * quantity,
     taxed_price:  total_price * (1 + tax_rate),
WHERE
  taxed_price - total_price &gt; 5;
```

But minimal examples like this undersell the difference this makes and even the *somewhat realistic example* is \- in my opinion \- too small to really show this. So I suggest to everyone to take their *SQL monster of the week* and see what changes when aliases can be reused.

```sql
WITH
  enriched AS (
    FROM  o: orders
    JOIN oi: order_items USING (order_id)
    JOIN  p: products    USING (product_id)
    JOIN  c: customers   USING (customer_id)
    SELECT
                duck_id:  p.product_id,
              duck_name:  p.product_name,
             duck_color:  COALESCE(p.color, 'Unknown'),
          duck_material:  COALESCE(p.material, 'Rubber'),
       customer_country:  c.country,

          duck_discount:  COALESCE(oi.discount_pct, 0),
       net_revenue_line:  oi.quantity * oi.unit_price * (1 - duck_discount),
              cogs_line:  oi.quantity * p.base_cost,
      gross_margin_line:  net_revenue_line - cogs_line,

        is_premium_duck:  duck_discount &gt;= 0.20 OR duck_color IN ('Gold', 'Platinum') OR duck_material ILIKE '%silicone%',
    WHERE p.category = 'Rubber Duck'
  )

FROM enriched
SELECT
           duck_id,
  duck_description:  format('{0} ({1}, {2})', duck_name, duck_color, duck_material),
  customer_country,
       net_revenue:  SUM(net_revenue_line),
              cogs:  SUM(cogs_line),
      gross_margin:  SUM(gross_margin_line),
WHERE is_premium_duck
GROUP BY duck_id, duck_description, customer_country
ORDER BY duck_id, customer_country;
```

A few differences I want to highlight:

* Reusing aliases in the same `SELECT` statement makes the `enriched` CTE much more concise.
    * Columns like `duck_color` and `duck_material` can be reused, making it feasible to put the `is_premium_duck` expression in a single line.
    * The KPIs can be calculated step by step, without needing an intermediate subquery.  
* Moving the KPIs to the `enriched` CTEs improves separation of concerns.
    * `enriched` becomes a collection of self-contained, pre-processed data points.
    * The final projection builds the target structure and calculates the desired aggregations.

## Star &amp; COLUMNS expression

The previous two categories were more on the low-key end of things, so I wanted my last pick to be a bit fancy. DuckDB’s star and `COLUMNS` expressions are powerful additions. But, as with any powerful tool, overusing them can make things worse rather than better. And, as mentioned earlier, prefix aliases do not work here, so the classic alias syntax has to be used.

**Extended star expression capabilities**

A shorthand to select every column of a table is useful, but what if you need every column except one or two? In classic SQL you would have to list all the columns you want. DuckDB makes this easier by letting you use `EXCLUDE`, `REPLACE`, and `RENAME` clauses with the star expression.

```sql
-- EXCLUDE
FROM products
SELECT * EXCLUDE(category)
WHERE category = 'Rubber Duck';

-- REPLACE
FROM order_items
SELECT * REPLACE(
  lpad(order_item_id::VARCHAR, 4, '0') AS order_item_id,
  lpad(product_id::VARCHAR, 4, '0') AS product_id,
);

-- RENAME
FROM customers
SELECT * RENAME(customer_name AS full_name);

-- Combination of all
FROM customers
SELECT 
  * EXCLUDE(country)
    REPLACE('000' || customer_id AS customer_id)
    RENAME(customer_name AS full_name),
;
```

This isn’t everything that has been added to the star expression. You can find all features in the [documentation](https://duckdb.org/docs/stable/sql/expressions/star).

**COLUMNS expression**

The extended star expression is already powerful, but the `COLUMNS` expression takes things even further. For example, you can apply the same expression on all matching columns, or even rename columns via regular expressions. The latter is something I would absolutely mark as *„danger territory“*.

```sql
FROM products
SELECT lower(
  COLUMNS(* EXCLUDE (product_id, base_cost))
);

FROM order_items
SELECT COLUMNS('(.+)_id') AS '\1';
```

There’s a lot more you can do with this. Check out the [documentation](https://duckdb.org/docs/stable/sql/expressions/star#columns-expression) for all the details.

**A fancy *somewhat realistic example***

Applying those features to the *somewhat realistic example* isn’t straightforward, so I had to introduce an additional CTE. This makes it somewhat more artificial, but it should be easy enough to imagine reasons why a query evolves this way over time.

```sql
WITH
  enriched AS (
    FROM  o: orders
    JOIN oi: order_items USING (order_id)
    JOIN  p: products    USING (product_id)
    JOIN  c: customers   USING (customer_id)
    SELECT
                duck_id:  p.product_id,
              duck_name:  p.product_name,
             duck_color:  COALESCE(p.color, 'Unknown'),
          duck_material:  COALESCE(p.material, 'Rubber'),
       customer_country:  c.country,

          duck_discount:  COALESCE(oi.discount_pct, 0),
       net_revenue_line:  oi.quantity * oi.unit_price * (1 - duck_discount),
    --  tax_collected_line:  net_revenue_line * COALESCE(oi.tax_rate, 0),
              cogs_line:  oi.quantity * p.base_cost,
      gross_margin_line:  net_revenue_line - cogs_line,

        is_premium_duck:  duck_discount &gt;= 0.20 OR duck_color IN ('Gold', 'Platinum') OR duck_material ILIKE '%silicone%',
    WHERE p.category = 'Rubber Duck'
  ),
  premium_with_description AS (
    FROM enriched
    SELECT
      * EXCLUDE (duck_name, duck_color, duck_material),
      duck_description: format('{0} ({1}, {2})', duck_name, duck_color, duck_material),
    WHERE is_premium_duck
  )

FROM premium_with_description
SELECT
  duck_id,
  duck_description,
  customer_country,
  SUM(COLUMNS('(.*)_line')) AS '\1',
GROUP BY duck_id, duck_description, customer_country
ORDER BY duck_id, customer_country;
```

Being able to pick’n’mix columns with less verbosity is nice to have, but adding a new KPI by simply adding a column named `_line` to `enriched` is very fancy. It’s also a bit scary, because it significantly increases the potential for side effects. In comparison, a [similar query with classic SQL](https://github.com/LennartH/friendly-ducking/blob/main/03-star-columns/03-standard-sql.sql) is about 15 lines (35%) longer and requires touching every query to add a new KPI.

## Wrapping up

I hope this has given you a sense of why I think DuckDB’s friendly SQL is a game changer for developers and data engineers once the code grows beyond a few lines. If you’re considering DuckDB for your next project, keep in mind that it is [explicitly scoped as a simple and fast DBMS](https://duckdb.org/why_duckdb). Something else is needed for orchestration, distributed workloads, scalability and other big-project needs. One option is [MotherDuck](https://motherduck.com/), which is the only tool I know of that has DuckDB at its core. So it might be the right one for you.

Also, this has been just a glimpse of the features DuckDB offers, ranging from nice things making SQL friendlier to having completely new possibilities. Here is an arbitrary list of things, with links to their documentation, that didn’t fit in the scope:

* [`QUALIFY` clause](https://duckdb.org/docs/stable/sql/query_syntax/qualify) for filtering immediately on window function results.
* [`GROUP BY ALL`](https://duckdb.org/docs/stable/sql/query_syntax/groupby.html#group-by-all) and [`ORDER BY ALL`](https://duckdb.org/docs/stable/sql/query_syntax/orderby.html#order-by-all) so you can omit the columns entirely.
* First class support for complex data structures like [lists](https://duckdb.org/docs/stable/sql/data_types/list), [structs](https://duckdb.org/docs/stable/sql/data_types/struct) &amp; [maps](https://duckdb.org/docs/stable/sql/data_types/map.html) including [list comprehension](https://duckdb.org/docs/stable/sql/functions/list.html#list-comprehension) and [slicing](https://duckdb.org/docs/stable/sql/functions/list.html#slicing).
* [`CREATE OR REPLACE`](https://duckdb.org/docs/stable/sql/statements/create_table.html#create-or-replace) as a shorthand for overwriting objects.
* [Lambda functions](https://duckdb.org/docs/stable/sql/functions/lambda) to turn SQL on its head.
* Additional join types like [`ASOF`](https://duckdb.org/docs/stable/sql/query_syntax/from.html#as-of-joins) and [`POSITIONAL`](https://duckdb.org/docs/stable/sql/query_syntax/from.html#positional-joins).
* [Recursive CTEs with `USING KEY`](https://duckdb.org/docs/stable/sql/query_syntax/with.html#recursive-ctes-with-using-key) allowing access to results beyond the previous iteration.
* [Dot operator for function chaining](https://duckdb.org/docs/stable/sql/functions/overview.html#function-chaining-via-the-dot-operator) so statements don’t have to end in five closing brackets.
* And everything else in [the friendly SQL docs](https://duckdb.org/docs/stable/sql/dialect/friendly_sql).

Is there anything on the list above you’d like to hear more about? Did I miss an obvious DuckDB feature that would improve the *somewhat realistic example*? Do you think my SQL formatting is awful and prefix aliases shouldn’t exist? Let me know and I’ll see what I can do.

Grober Vergleich der semantischen Reihenfolge von PostgreSQL und DUckDB

High level comparison between semantic order of PostgreSQL and DuckDB

This article shows how DuckDB's friendly SQL dialect can improve developer experience significantly without forfeiting the familiarity of classic SQL.

DuckDB’s friendly SQL is a developer experience game changer

DuckDB’s friendly SQL is a game changer for developer experience

MotherDuck, the cloud-native service built on DuckDB, fundamentally transforms how organizations interact with data stored in cloud blob storage. By eliminating the traditional ETL/ELT pipeline, MotherDuck enables direct SQL analytics on Parquet, JSON, and CSV files (among others) located in Amazon S3, Azure Blob Storage, or Google Cloud Storage without requiring data movement or preprocessing. This approach represents a fundamental shift from conventional data warehouse architectures where data must be ingested, transformed, and stored before analysis can begin.

The zero-ETL approach allows organizations to maintain their data in its original format and location while performing analytical queries. This eliminates data duplication, reduces storage costs, and removes the latency associated with traditional ETL processes. Most importantly, it enables data teams to query data immediately without waiting for batch jobs or complex pipeline orchestrations.

## How MotherDuck Accesses Cloud Data

### Direct File Access

When you execute a query against cloud storage, MotherDuck establishes direct connections to the storage service and reads only the necessary data segments. Consider this query:

```sql
SELECT customer_id, SUM(order_total) as revenue
FROM read_parquet('s3://analytics-bucket/orders/*.parquet')
WHERE order_date >= '2025-01-01'
GROUP BY customer_id;
```

MotherDuck doesn't copy these files to a staging area or intermediate storage. Instead, it leverages DuckDB's table functions that natively understand cloud storage protocols, enabling direct reads from AWS S3, Azure Blob Storage, or Google Cloud Storage endpoints. The system maintains persistent HTTP connections to these storage services, reusing them across multiple requests to minimize connection overhead and optimize for both latency and throughput. Prior authentication is required through IAM roles, access keys, or other cloud-native security mechanisms.

### Format-Native Processing

For Parquet files, MotherDuck takes advantage of the columnar format's structure to minimize data transfer. When querying a 10GB Parquet file containing 50 columns but selecting only three columns, MotherDuck reads only those specific column chunks directly from storage. This columnar reading happens at the storage layer through HTTP range requests, not after downloading the entire file. A query selecting three columns from a 50-column dataset drastically reduces data transfer volume.
For JSON files, MotherDuck employs stream parsing strategies that allow direct querying of nested structures:

```sql
SELECT 
    json_extract_string(data, '$.user.email') as email,
    CAST(json_extract(data, '$.purchase.amount') AS DECIMAL(10,2)) as amount
FROM read_json('s3://logs-bucket/events/*.json')
WHERE json_extract_string(data, '$.event_type') = 'purchase';
```

The JSON reader detects types and column names automatically and optimizes the conversion to vectors.

## Query Optimization Techniques

MotherDuck uses Predicate and Projection Pushdown to further enhance query performance. Both techniques will be illustrated with examples below.

### Predicate Pushdown

Predicate pushdown represents one of the most impactful optimizations in MotherDuck's execution engine. Instead of reading all data and filtering afterward, the system pushes filtering operations as close to the storage layer as possible. Consider this scenario with Parquet files:

```sql
SELECT COUNT(*) as premium_sales
FROM read_parquet('s3://sales-data/year=2024/month=*/day=*/*.parquet')
WHERE sale_amount > 1000 
    AND product_category = 'Premium';
```

MotherDuck applies multiple levels of filtering. First, partition elimination uses the directory structure to skip entire paths that don't match the query predicates. Second, row group elimination leverages Parquet metadata, specifically min/max statistics and Bloom filters when available, to skip row groups where the maximum sale_amount is below 1000. Third, late materialization defers reading non-essential columns until after filtering is complete.

This optimization can drastically reduce the transferred data volume and is even more effective with sorted data or when querying recent partitions in time-series datasets.

### Projection Pushdown

Projection pushdown ensures that only required columns are transferred from storage. MotherDuck's query planner identifies all columns needed for the entire query execution, including those required for filtering, joining, aggregation, and final projection:

```sql
SELECT customer_name, order_date, total_amount, shipping_address, payment_method
FROM read_parquet('s3://orders-archive/2024/*.parquet')
WHERE order_status = 'completed'
    AND total_amount > 100;
```

The optimizer determines that it needs to read six columns: the five in the SELECT clause plus order_status for filtering. It configures the Parquet reader to request only these specific column chunks via HTTP range requests, ignoring the remaining columns entirely, decreasing memory usage and network transfer.

## Hybrid Execution: Intelligent Query Routing

MotherDuck's hybrid architecture automatically determines optimal query execution location based on data locality and computational requirements. When connected to MotherDuck from a local DuckDB instance, the system routes operations intelligently:

```sql
SELECT 
    l.product_id,
    l.local_price,
    AVG(c.cloud_price) as avg_cloud_price
FROM read_csv('local_prices.csv') l
JOIN read_parquet('s3://pricing-data/historical/*.parquet') c
  ON l.product_id = c.product_id
GROUP BY l.product_id, l.local_price;
```

The execution engine performs S3 reads in MotherDuck's cloud environment. Local CSV operations execute on the client machine. The join strategy depends on relative data sizes: small local tables might be pushed to the cloud for processing, while aggregated cloud results might be pulled locally for final joining. This prevents unnecessary movement of large datasets while maintaining query performance.

## Multi-Format Query Federation

MotherDuck can join data across different formats and locations within a single query. The system can combine Parquet, JSON, and CSV sources seamlessly:

```sql
SELECT 
    p.customer_id,
    p.purchase_amount,
    json_extract_string(s.session_data, '$.duration') as session_duration
FROM read_parquet('s3://purchases/2025/*.parquet') p
JOIN read_json('s3://sessions/2025/*.json') s
  ON p.session_id = json_extract_string(s.session_data, '$.session_id')
WHERE p.purchase_date >= '2025-01-01';
```

Each format is processed using its optimal access pattern while joins are executed using the most efficient strategy based on data size and distribution.

## Limitations and Considerations

While MotherDuck's zero-ETL approach offers substantial benefits, certain scenarios require careful consideration. Extremely complex transformations involving multiple stages of windowing and aggregation may perform better with materialized intermediate results. Real-time streaming use cases requiring very low latency still benefit from dedicated streaming infrastructure. Regulatory requirements sometimes mandate data residency or transformation audit trails that zero-ETL approaches cannot provide.

File format considerations also matter. While Parquet offers the best optimization potential with its columnar format and rich metadata, JSON files lack these optimizations and may result in higher scan volumes. 
Organizations should also consider network bandwidth costs when frequently querying large datasets across regions, and evaluate whether certain frequently-accessed, heavily-transformed datasets might benefit from selective materialization alongside the zero-ETL approach for occasional ad-hoc queries.

## Conclusion

MotherDuck's zero-ETL capabilities represent an interesting alternative for cloud data analytics architecture. By treating cloud storage as a queryable data source with elegant pushdown optimizations, it eliminates the traditional boundaries between data storage and data processing. The combination of predicate pushdown, projection pushdown, and hybrid execution creates a system where large datasets become as accessible as local databases.

For organizations evaluating their data architecture, MotherDuck offers an alternative to traditional ETL pipelines. The ability to query data directly where it resides, in its native format, with full SQL capabilities and automatic optimizations, simplifies the entire analytics stack. This offers an option to rethink data architecture, making analytics more immediate, more flexible, and more accessible across the organization. As data volumes continue to grow and real-time insights become increasingly critical, the zero-ETL approach positions organizations to handle future scale without the complexity of traditional data pipeline architectures.


This article explains how MotherDuck runs SQL queries directly on files in cloud storage, eliminating the need for complex data pipelines.

Zero-ETL with MotherDuck: A technical Deep Dive

Zero-ETL with MotherDuck: A Technical Deep Dive

Senior Solution Architect

Using Cloud Native Buildpacks (CNB) in your GitLab CI pipelines without Docker and pack CLI is common practice. Especially if you leverage shared Kubernetes runners without access to the host's Docker daemon. In this case you need to use the CNB lifecycle directly. But what if you need to define environment variables for the build process? 

<h2>Cloud Native Buildpacks – blog series</h2>
<p>
Part 1: <a href="/en/knowledge-hub/blog/buildpacks-spring-boot"> Goodbye Dockerfile: Cloud Native Buildpacks with Paketo.io &amp; layered jars for Spring Boot</a> <br>
Part 2: <a href="/en/knowledge-hub/blog/gitlab-ci-paketo-buildpacks"> Cloud Native Buildpacks / Paketo.io in GitLab CI without Docker &amp; pack CLI</a> <br>
Part 3: Using Environment Variables with the Cloud Native Buildpacks lifecycle
</p>

I'm a big fan of Cloud Native Buildpacks (#CNB) since years 😍 I wrote about [how to say goodbye to your Dockerfiles already in 2020](https://www.codecentric.de/en/knowledge-hub/blog/buildpacks-spring-boot) and used them in nearly every project since then. But in modern CI/CD pipelines installments, you'd often don't have access to the host's Docker daemon (like it had been common using `priviledged` mode or Docker socket mounts). And this is a great thing taking security aspects into account and is implemented by our [codecentric Cloud Managed GitLab](https://www.codecentric.de/en/services/gitlab-solutions/managed-gitlab) per default. 

Using Buildpacks in such a scenario requires you to omit pack CLI and instead use the CNB lifecycle directly, as [I wrote about in this post](https://www.codecentric.de/en/knowledge-hub/blog/gitlab-ci-paketo-buildpacks). The [deprecated Kaniko](https://github.com/GoogleContainerTools/kaniko) adressed a similar issue, but you still had to maintain Dockerfiles.

![cloud-native-buildpacks-gitlab-without-docker-packcli.png](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/cmg805fzgjm0n07rsz7j3qzru)

## An updated .gitlab-ci.yml to use the CNB lifecycle without Docker

Since I wrote the last post in 2021 there was a slight change in the direct usage of the CNB lifecycle in GitLab CI, since it needs some variables to be defined before execution that weren't needed back then. In order to prevent errors like `failed to get platform API version; please set 'CNB_PLATFORM_API' to specify the desired platform API` we now need to define the variables `CNB_USER_ID`, `CNB_GROUP_ID` and `CNB_PLATFORM_API` like it is shown in the fully working  `.gitlab-ci.yml` here: 

```yaml
stages:
  - test
  - container-image

variables:
  # see usage of Namespaces at https://docs.gitlab.com/ee/user/group/#namespaces
  REGISTRY_GROUP_PROJECT: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME

maven-test:
  stage: test
  image: maven:3-eclipse-temurin-25
  script:
    - mvn test

build-publish-image:
  stage: container-image
  # For a detailed description on how to use Paketo.io Buildpacks in GitLab CI
  # with Kubernetes executor &amp; unprivileged Runners (without pack CLI &amp; docker)
  # see https://stackoverflow.com/questions/69569784/use-paketo-io-cloudnativebuildpacks-cnb-in-gitlab-ci-with-kubernetes-executo
  image: paketobuildpacks/builder-jammy-base

  # As we don't have docker available, we can't login to GitLab Container Registry as described in the docs https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#using-the-gitlab-container-registry
  # But we somehow need to access GitLab Container Registry with the Paketo lifecycle
  # So we simply create ~/.docker/config.json as stated in https://stackoverflow.com/a/46422186/4964553
  before_script:
    - mkdir ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" &gt;&gt; ~/.docker/config.json
    # As described in https://buildpacks.io/docs/for-platform-operators/tutorials/lifecycle/#set-environment-variables
    # we need to set some variables like the platform api version in order to prevent errors like this:
    # failed to get platform API version; please set 'CNB_PLATFORM_API' to specify the desired platform API version
    - export CNB_USER_ID=$(id -u) CNB_GROUP_ID=$(id -g) CNB_PLATFORM_API=0.14

  script:
    - /cnb/lifecycle/creator -app . -cache-image $REGISTRY_GROUP_PROJECT/paketo-build-cache:latest $REGISTRY_GROUP_PROJECT:latest
```

This pipeline is derived from the 2021 article and defines two stages. The `test` stage runs all tests using Maven with the `maven-test` job. The second stage `container-image` features the `build-publish-image` job that showcases the direct usage of the CNB lifecycle without Docker or pack CLI. So far so good.


## What's the issue with defining environment variables for the lifecycle?

Last week I had the requirement to use a simple environment variable with the CNB lifecycle, since the customer needed a JDK inside the run image (which is generally not a recommended thing to do, but the software needed to validate stuff using the JDK tooling). The [default Bellsoft Liberica buildpack](https://github.com/paketo-buildpacks/bellsoft-liberica) has a parameter `BP_JVM_TYPE` that you can set to `JDK`. This will configure the JVM type that is provided in the run image to use a JDK instead of the default JRE. So exactly what the customer needed. Would I be able to use pack CLI, the command would be simple and the environment variable could be defined using `--env` like this:

```shell
pack build your.gitlab.contaner.registry.com/yourgitlabproject/microservice-api-spring-boot:latest \
              --builder paketobuildpacks/builder-jammy-base \
              --path . \
              --env "BP_JVM_TYPE=JDK"
```

But simply running `export BP_JVM_TYPE=JDK` before the lifecycle execution doesn't work, as I expected it in the first place. The run image produced by the CNB build process still featured a JRE instead the needed JDK. 

To verify if a JDK is used in the CNB build run image, you can simply use a `docker run -it` and [override entrypoint of Spring Boot run image](https://buildpacks.io/docs/for-app-developers/how-to/build-outputs/specify-launch-process/#user-provided-shell-process):

```shell
docker run -it --rm  --entrypoint launcher your.gitlab.contaner.registry.com/yourgitlabproject/microservice-api-spring-boot:latest bash
```

You might need to [authenticate to your GitLab's registry](https://docs.gitlab.com/user/packages/container_registry/authenticate_with_container_registry/) for this to work. I got to know the GitLab CLI with `glab auth login` which I found quite handy here. 

Now inside the container you can verify, if it uses a JDK or JRE as the runtime by running

```shell
java -XshowSettings:properties -version | grep java.home
``` 

If the output contains the `java.home = /layers/paketo-buildpacks_bellsoft-liberica/jdk` the JDK is successfully configured (which was the goal for my customer). If the output contains `java.home = /layers/paketo-buildpacks_bellsoft-liberica/jre`, the `BP_JVM_TYPE` variable hasn't been correctly set (which was the case for me).


## The solution: Create the environment variables using files

I had a hard time finding a solution. But luckily [an accepted answer on stackoverflow](https://stackoverflow.com/a/73990313/4964553) from 2022 brought me near the solution. So right back into our `.gitlab-ci.yml` we need to create a different `platform` directory in the `cnb` user home:

```shell
mkdir -p ~/platform/env
```
This is because directly using `echo "JDK" &gt;&gt; platform/env/BP_JVM_TYPE` gives a `bash: /platform/env/BP_JVM_TYPE: Permission denied` error, because the directory `platform` is already created inside the builder image. I guess that the builder images somehow changed and now contains the folder `platform` already (and we don't have - and don't want - root permissions inside the container).

Having this directory in place we can now create the environment variables as described [in the buildpack platform spec documentation](https://github.com/buildpacks/spec/blob/main/platform.md#user-provided-variables) in the form of 

&gt; Each file SHALL define a single environment variable, where the file name defines the key and the file contents define the value.

So inside the `~/platform/env` directory we create a file `BP_JVM_TYPE` using the following command:

```shell
echo -n "JDK" &gt;&gt; ~/platform/env/BP_JVM_TYPE
```

Now you should see a file `BP_JVM_TYPE` with the content `JDK` inside `~/platform/env`.

&gt; **It is extremely important to use the `-n` parameter with `echo`, since it removes the trailing newline!** This was the thing that drove me nuts. [I ran into an odyssey](https://stackoverflow.com/q/79776035/4964553), since there is [a bug in the buildpack implementation tracked in this issue](https://github.com/paketo-buildpacks/libpak/issues/416), where the newline is causing parameters to **NOT WORK** with the lifecycle (many thanks to [Daniel Mikusa for providing the hint](https://stackoverflow.com/a/79777081/4964553)). I can only advise you to double check the contents of the files you create like `~/platform/env/BP_JVM_VERSION` in an editor:

wrong (containing newline):

![cloud-native-buildpacks-lifecycle-environment-variables-BP_JVM_VERSION_wrong.png](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/cmg6c72jjp5cq06ur8svt1pp0)

correct (NO newline):

![cloud-native-buildpacks-lifecycle-environment-variables-BP_JVM_VERSION.png](https://eu-central-1.graphassets.com/AiE4QoWSSiIQO3k152ugkz/cmg6c7i5sp6kt06urc6k2fuae)


## Run the CNB lifecycle using -platform

With the `BP_JVM_TYPE` file in place containing our value without a trailing newline, we are now ready to run the CNB lifecycle. Therefore execute the `/cnb/lifecycle/creator` command with an appended `-platform /home/cnb/platform`. But again be careful, since the directory needs to be passed **WITHOUT** the trailing `/env`!. Otherwise it won't use the environment configuration correctly:

```shell
/cnb/lifecycle/creator -app . -platform /home/cnb/platform $REGISTRY_GROUP_PROJECT:latest
```

That's it already! I would have loved to find such a blog post like this when I started out to "simply" create a run image featuring a JDK with Buildpacks :)

Here's the fully working `.gitlab-ci.yml` again, featuring the definition of the environment variable `BP_JVM_TYPE=JDK`: 

```yaml
stages:
  - test
  - container-image

variables:
  # see usage of Namespaces at https://docs.gitlab.com/ee/user/group/#namespaces
  REGISTRY_GROUP_PROJECT: $CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME

maven-test:
  stage: test
  image: maven:3-eclipse-temurin-24
  script:
    - mvn test

build-publish-image:
  stage: container-image
  image: paketobuildpacks/builder-jammy-base

  before_script:
    - mkdir ~/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_JOB_TOKEN\"}}}" &gt;&gt; ~/.docker/config.json
    - export CNB_USER_ID=$(id -u) CNB_GROUP_ID=$(id -g) CNB_PLATFORM_API=0.14
    # Configuring Environment variables for the paketo lifecycle is a bit more sophisticated
    # As described in https://stackoverflow.com/a/79766353/4964553 &amp; the lifecycle spec https://github.com/buildpacks/spec/blob/main/platform.md#user-provided-variables
    # we need to create a single file for each variable, where the file name defines the key and the file contents define the value
    # WE NEED TO USE -n with echo, otherwise we have a trailing newline in the environment variable configuration files, that breaks the parameters usage (https://github.com/paketo-buildpacks/libpak/issues/416)
    - mkdir -p ~/platform/env
    # Using a different JDK than default 21
    - echo -n "25" &gt;&gt; ~/platform/env/BP_JVM_VERSION
    # Using a JDK instead of a JRE at runtime (not recommended for default, but used inside the PMS currently for validation, that relies on Java Bytecode)
    - echo -n "JDK" &gt;&gt; ~/platform/env/BP_JVM_TYPE

  script:
    # We need to add the -platform /home/cnb/platform parameter in order to let the lifecycle use our defined environment variables
    - /cnb/lifecycle/creator -app . -platform /home/cnb/platform -cache-image $REGISTRY_GROUP_PROJECT/paketo-build-cache:latest $REGISTRY_GROUP_PROJECT:latest
```

Just to show the usage, i also added the `BP_JVM_VERSION` variable to change the JVM version.
 
The GitLab CI log output (cropped for this post) also looks good, since it finally contains the message `A JDK was specifically requested by the user, however a JRE is available.` which indicates, the environment variable is successfully picked up by the Bellsoft Liberica buildpack:

```shell
...
===&gt; BUILDING
target distro name/version labels not found, reading /etc/os-release file
Paketo Buildpack for CA Certificates 3.10.4
  https://github.com/paketo-buildpacks/ca-certificates
  Build Configuration:
    $BP_EMBED_CERTS                    false  Embed certificates into the image
    $BP_ENABLE_RUNTIME_CERT_BINDING    true   Deprecated: Enable/disable certificate helper layer to add certs at runtime
    $BP_RUNTIME_CERT_BINDING_DISABLED  false  Disable certificate helper layer to add certs at runtime
  Launch Helper: Reusing cached layer
Paketo Buildpack for BellSoft Liberica 11.4.0
  https://github.com/paketo-buildpacks/bellsoft-liberica
  Build Configuration:
    $BP_JVM_JLINK_ARGS           --no-man-pages --no-header-files --strip-debug --compress=1  configure custom link arguments (--output must be omitted)
    $BP_JVM_JLINK_ENABLED        false                                                        enables running jlink tool to generate custom JRE
    $BP_JVM_TYPE                 JDK                                                          the JVM type - JDK or JRE
    $BP_JVM_VERSION              25                                                           the Java version
  Launch Configuration:
    $BPL_DEBUG_ENABLED           false                                                        enables Java remote debugging support
    $BPL_DEBUG_PORT              8000                                                         configure the remote debugging port
    $BPL_DEBUG_SUSPEND           false                                                        configure whether to suspend execution until a debugger has attached
    $BPL_HEAP_DUMP_PATH                                                                       write heap dumps on error to this path
    $BPL_JAVA_NMT_ENABLED        true                                                         enables Java Native Memory Tracking (NMT)
    $BPL_JAVA_NMT_LEVEL          summary                                                      configure level of NMT, summary or detail
    $BPL_JFR_ARGS                                                                             configure custom Java Flight Recording (JFR) arguments
    $BPL_JFR_ENABLED             false                                                        enables Java Flight Recording (JFR)
    $BPL_JMX_ENABLED             false                                                        enables Java Management Extensions (JMX)
    $BPL_JMX_PORT                5000                                                         configure the JMX port
    $BPL_JVM_HEAD_ROOM           0                                                            the headroom in memory calculation
    $BPL_JVM_LOADED_CLASS_COUNT  35% of classes                                               the number of loaded classes in memory calculation
    $BPL_JVM_THREAD_COUNT        250                                                          the number of threads in memory calculation
    $JAVA_TOOL_OPTIONS                                                                        the JVM launch flags
    Using Java version 25 from BP_JVM_VERSION
  A JDK was specifically requested by the user, however a JRE is available. Using a JDK at runtime has security implications.
  BellSoft Liberica JDK 25.0.0: Reusing cached layer
...
```

Let's finally verify if the JDK is used the run image:

```shell
docker run -it --rm  --entrypoint launcher your.gitlab.contaner.registry.com/yourgitlabproject/microservice-api-spring-boot:latest bash
java -XshowSettings:properties -version | grep java.home
```

The output now contains the `java.home = /layers/paketo-buildpacks_bellsoft-liberica/jdk` which indicates that our pipeline worked the way we wanted it to! 


## Optional: Developing the solution locally 

If you're like me you might not like to create a lot of git commits in GitLab, wait for the pipelines to be triggered and run into errors there (the dreaded "staring on pipelines" problem).

But there's a simple solution. Clone the project you want to build to your local machine (in my case this is [a Spring Boot based project](https://github.com/jonashackt/microservice-api-spring-boot)), `cd` into it and use the paketo.io builder [paketobuildpacks/builder-jammy-base](https://github.com/paketo-buildpacks/builder-jammy-base) simply directly as a local Docker container:

```shell
docker run -it --rm -v "$PWD":/usr/src/app -w /usr/src/app paketobuildpacks/builder-jammy-base bash
``` 

This mimics the usage in GitLab, where your Java project is mounted into the container and is based on that exact image.

Now in order to make the CNB lifecycle run successfully inside your local container, you need to to the things we do in the GitLab CI pipeline also:

```shell
mkdir ~/.docker

echo "{\"auths\":{\"your.gitlab.contaner.registry.com\":{\"username\":\"yourUserName\",\"password\":\"your_gitlab_access_token\"}}}" &gt;&gt; ~/.docker/config.json

export CNB_USER_ID=$(id -u) CNB_GROUP_ID=$(id -g) CNB_PLATFORM_API=0.14
```

For the authorization data you might need to create a new access token in your GitLab profile.

You might also run into `permission denied` errors later while building your image inside the container that relate to your container not having the right permissions for the `target/classes` directory used in the Maven build. Solutions to that [are described in this great writeup](https://labex.io/tutorials/docker-how-to-resolve-permission-denied-error-when-mounting-volume-in-docker-417724#resolving-permission-denied-errors).


## Wrap up

Cloud Native Buildpacks still remain a fantastic solution to handle our container build requirements. This applies just as much for scenarios with shared (Kubernetes based) GitLab runners without Docker daemon access. Using the CNB lifecycle directly is slightly less documented compared to the default way using pack CLI and Docker, therefore I tried to shed some more light on this important scenario.

For more alternatives I can recommend [a great overview about options to replace the deprecated Kaniko](https://www.codecentric.de/en/knowledge-hub/blog/7-ways-to-replace-kaniko-in-your-container-image-builds) by   my colleague Felix Dreißig. I think we need to drink a beer in order to get Cloud Native Buildpacks on his list as option 8 :)  





If you need to use Cloud Native Buildpacks without Docker & pack CLI, you might also need to provide environment variables too. Here's how to do that.

Environment Variables with Cloud Native Buildpacks lifecycle

Using Environment Variables with the Cloud Native Buildpacks lifecycle

Principle Consultant + People Lead

In this post, we'll explore how [MotherDuck](https://motherduck.com), powered by [DuckDB](https://duckdb.org), revolutionizes the way you interact with your data, particularly when dealing with CSV files. You'll learn how to quickly parse and filter even large datasets directly from your local machine, using familiar SQL syntax, and without the need for a dedicated server or extensive configuration. Get ready to transform your raw CSV data into valuable insights in a matter of minutes!

CSV (Comma Separated Values) files remain a ubiquitous format for exchanging data, despite the emergence of newer, more structured formats like JSON and Parquet. Their simplicity and widespread compatibility ensure their continued relevance in various data workflows. However, working with large CSV files can sometimes be cumbersome, especially when you need to quickly extract insights without the overhead of complex setups.

## From Gigabytes to Insights: Parsing CSVs with DuckDB

### Setup

If you haven't set up the DuckDB CLI head over to [DuckDB Installation](https://duckdb.org/docs/stable/installation/index) and select your preferred option to install the DuckDB CLI.

### About the dataset
We are using the well-known [New York City Taxi dataset](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page), specifically the Yellow Taxi Trip data from 2023 which is available [here](https://data.cityofnewyork.us/Transportation/2023-Yellow-Taxi-Trip-Data/4b4i-vvec/about_data). This dataset contains approximately 38.3 million (38,310,226) records, and the CSV file size is 3.78 GB. This size is typically too large to manage effectively with tools like Microsoft Excel and we leverage the power of DuckDB and MotherDuck

> [!NOTE]  
In MotherDuck there is a `sample_data` database which contains the taxi data for December 2022 which is about 10% of the data volume we use in this blog post. You can also use this dataset if your upload link is limited.

### Loading the dataset
DuckDB supports two persistence modes, in-memory and persistent. To load the dataset into a permanent DuckDB database, begin by creating the `nyc_taxi.duckdb` database. Execute the following command:

```sh
duckdb nyc_taxi.duckdb  
```
Once the database is created, load the dataset by executing the subsequent command within the DuckDB CLI.

```sql
.timer on
CREATE TABLE nyc_yellow_taxi_trips AS FROM '2023_Yellow_Taxi_Trip_Data_20250903.csv';
```

Depending on your machine, the dataset should load in mere seconds. For instance, on a Mac M3 the entire 3.78GB dataset, containing over 38.3 million records, loads in approximately 3.5 seconds, highlighting DuckDB's impressive performance capabilities.

```sql
Run Time (s): real 3.540 user 38.143785 sys 1.770184
```

### Our first query - calculate the trip duration for each trip
Once the data is loaded, a process that happens incredibly quickly, we can run our first query. We'll start by calculating the trip duration, which is the difference between `tpep_dropoff_datetime` and `tpep_pickup_datetime`, using the commands below.


```sql
.timer on
SELECT *, (tpep_dropoff_datetime - tpep_pickup_datetime) AS trip_duration 
FROM nyc_yellow_taxi_trips;
```
You should see the following output

```sql
Run Time (s): real 1.304 user 1.268312 sys 2.229287
```

As you can see the trip_duration is calculated in just a second or all 38.3 million records which is pretty awesome.

### Enriching the data - store the trip duration as a new column

To facilitate further queries involving trip duration, we will store these values in a new column. This can be achieved by executing a simple command that creates a new column called trip_duration.

```sql
ALTER TABLE nyc_yellow_taxi_trips
ADD COLUMN trip_duration INTERVAL;
UPDATE nyc_yellow_taxi_trips
SET trip_duration = tpep_dropoff_datetime - tpep_pickup_datetime;
```

Now let's use the newly created column to calculate the average trip duration. For this execute the following command and you get the result again in milliseconds.

```sql
.timer on
SELECT AVG(trip_duration) FROM nyc_yellow_taxi_trips;
```

## Moving to MotherDuck

We've explored DuckDB locally using its CLI, witnessing its remarkable speed with large CSV datasets. Now, let's transition to MotherDuck to share your data with your team, enabling them to leverage the combined power of DuckDB and MotherDuck.

> [!NOTE]  
For this you need a MotherDuck account which you can easily create on [https://motherduck.com](https://motherduck.com)

To load the CSV data into MotherDuck, open a new DuckDB CLI session execute the following command:

```sh
duckdb
```

Afterwards run the following commands in the DuckDB CLI shell.

> [!NOTE]  
If you haven't set the environment variable `motherduck_token` as documented in the [MotherDuck documentation](https://motherduck.com/docs/key-tasks/authenticating-and-connecting-to-motherduck/authenticating-to-motherduck) a browser window will open where you need to confirm the access to MotherDuck from your DuckDB CLI shell.

```sql
ATTACH 'md:';
CREATE OR REPLACE DATABASE nyc_yellow_taxi_trips;
use nyc_yellow_taxi_trips;
create or replace table nyc_yellow_taxi_trips as from '2023_Yellow_Taxi_Trip_Data_20250903.csv';
```

This generates a new database `nyc_yellow_taxi_trips`, and populates a table of the same name with the loaded data. The duration of this step can range from two to three minutes, contingent on your internet upload speed.

> [!NOTE]  
While this post focuses on CSV file stored on a local storage, it's worth noting that DuckDB/MotherDuck also supports importing CSV data from various object storage systems, including AWS S3, Azure Blob Storage, and Google Cloud Storage.

Let's advance to more complex queries using MotherDuck. We'll be executing these queries within the MotherDuck UI.

### Average trip duration depending the number of passengers

To begin, navigate to [https://app.motherduck.com](https://app.motherduck.com) and create a new notebook titled `nyc_yellow_taxi_trips`. In the initial cell, insert the SQL statement shown in the block below and execute it by clicking the small triangle button.

```sql
SELECT passenger_count, AVG(tpep_dropoff_datetime - tpep_pickup_datetime) AS avg_trip_duration 
FROM nyc_yellow_taxi_trips 
GROUP BY passenger_count
ORDER BY passenger_count;
```

### Does the amount of tip depend on the trip distance?

For a taxi driver it would be interesting to know if there is a correlation between the amount of the tip and the length of the trip. For this we can execute the following query:

```sql
SELECT 
 CASE WHEN trip_distance BETWEEN 0 AND 4 THEN 'short'
      WHEN trip_distance BETWEEN 4 AND 9 THEN 'medium'
      WHEN trip_distance > 9 THEN 'long' 
      END AS trip_length,
 AVG(fare_amount) AS fare, 
 AVG(tip_amount) AS tip
 FROM nyc_yellow_taxi_trips
 GROUP BY trip_length
 ORDER BY tip DESC;
```

|trip_length|fare|tip|
|-----------|---:|--:|
|long|60.66741257482049|9.713919128310547|
|medium|29.65654459738106|5.005007720056084|
|short|12.775505292438215|2.5128599942576764|

As you can see from the results, the longer the trip the higher the amount of tip for the taxi driver. 

### Find the pickup locations which have the longest trip distance

As we have seen it is beneficial to look for longer trips as the amount of the tip is almost 4 times higher for longer trips compared to shorter trips.

Therefore let's find out which are the 20 most interesting pickup locations which have on average the longest trips and on average the highest tip. Only consider pickup locations with more than 1000 trips.

For this execute the following sql statement:

```sql
SELECT
    PULocationID,
    COUNT(*) AS trip_count,
    AVG(trip_distance) AS avg_trip_distance,
    AVG(total_amount) AS avg_total_amount,
    AVG(tip_amount) AS avg_tip_amount
FROM
    nyc_yellow_taxi_trips
GROUP BY
    PULocationID
HAVING
    trip_count > 1000
ORDER BY
    avg_tip_amount DESC
LIMIT 20;
```

In just milliseconds calculating over 38.3 million records we have the information and with this information a taxi driver can try to optimize his income by strategically positioning himself in the vicinity of the most profitable pickup locations.

### Are airport trips more profitable?

The last example we want to look at is if trips from the airport are more profitable than other trips.

We want to find out if the average tip is higher for trips from the airport compared to trips starting somewhere else. To simplify the example, we just take the average price without considering trip duration or other factors. Note that drop-off location may actually affect the amount of tip. The `PULocationID` for the three commercial airports are 1, 132, 138.

```sql
SELECT
    CASE
        WHEN PULocationID IN (1, 132, 138) THEN 'airport'
        ELSE 'not_airport'
    END AS pickup_type,
    AVG(tip_amount) AS avg_tip_amount,
    COUNT(*) AS trip_count
FROM
    nyc_yellow_taxi_trips
GROUP BY
    pickup_type;
```

Again, the result is returned in milliseconds, confirming that tips for airport trips are nearly three times higher.

## Conclusion
DuckDB and MotherDuck revolutionize data analysis by offering unparalleled speed and efficiency, especially when dealing with large datasets. The traditional hurdles associated with big data — such as the time-consuming loading processes and the need for complex infrastructure — are virtually eliminated.

### The DuckDB Advantage: Speed and Simplicity

With DuckDB, loading gigabytes of data is no longer a multi-hour ordeal but a matter of seconds. This incredible speed extends to query execution, which is _extremely fast_, allowing for real-time insights that were previously unattainable. This marks a significant departure from conventional CSV tools like Microsoft Excel, which buckle under the weight of large datasets, often becoming unresponsive or taking an eternity to process.

Traditional data warehouse solutions, while powerful, often demand a significant upfront investment in time and resources. They typically require intricate data pipeline mechanisms to ingest data, followed by the provisioning and scaling of compute resources before even the first query can be executed. This elaborate setup often translates to days or even weeks of preparation before any meaningful analysis can begin.

### MotherDuck: The Cloud-Native Companion

MotherDuck complements DuckDB by extending its capabilities to the cloud, offering a serverless experience that further simplifies data analysis. This combination means that users can leverage the power of DuckDB's in-process OLAP engine with the scalability and accessibility of a cloud platform, without the operational overhead.

### The "First Query" Race

The true power of DuckDB and MotherDuck becomes evident in what can be called the _first query_ race. In many scenarios, by the time other solutions are still being configured or are just beginning the data loading process, users of DuckDB and MotherDuck have already completed their analysis, extracted critical insights, and are ready to make data-driven decisions. This agility provides a significant competitive advantage, enabling faster iterations and more responsive business strategies.

In essence, DuckDB and MotherDuck are not just tools; they represent a paradigm shift in how data analysis is approached, making it more accessible, faster, and significantly less cumbersome for anyone looking to transform raw data into actionable intelligence.


In this post, we'll explore how MotherDuck, powered by DuckDB revolutionizes the way you interact with your data, particularly when dealing with CSV files. 

From CSV to Insights in 5 Minutes with DuckDB and MotherDuck

Your First Data Analysis with MotherDuck and DuckDB: From CSV to Insights in 5 Minutes

Service Lead Data & ML & AI

# 5 Reasons We’re Excited About MotherDuck’s Launch in AWS Frankfurt

For some time, a key challenge for European data teams has been balancing innovation with strict regulation. We’ve often seen powerful tools launch first in the US, while our need for GDPR compliance and data sovereignty requires a local presence.

This is why the official launch of [MotherDuck](https://motherduck.com/) in the AWS Frankfurt region on 24 September (eu-central-1) is a noteworthy development for the data community here [^1]. As an official launch partner, we evaluated the platform ahead of its European debut. Our perspective is clear: this is far more than just a new cloud location. It represents a new, more efficient philosophy for small- to medium-sized data analytics. 

Before we dig into the five key reasons for our positive assessment, let’s first address the fundamental question:

## What the Duck Is MotherDuck?

To fully appreciate [MotherDuck's](https://motherduck.com/) value proposition, one must first understand its foundation: [DuckDB](https://duckdb.org/).

In essence, DuckDB is to analytics what SQLite is to transactional databases – a powerful, serverless, and embedded data engine. It operates “in-process,” which eliminates the need for any complex client–server architecture. There is no server to install, no instance to configure. This simplicity, combined with its consistently high rankings in performance benchmarks like [ClickBench](https://benchmark.clickhouse.com/)[^2], has led to widespread adoption (over 30k GitHub stars) and a massive developer following.

While DuckDB excels on a local machine, enterprise environments require additional capabilities for collaboration, security, and scale. This is precisely where MotherDuck comes in. It extends DuckDB with a suite of production-ready, cloud-based features, including:
- Centralized authorization and user management
- Durable, persistent cloud storage
- Intelligent caching and query scaling

As MotherDuck’s founder, Jordan Tigani, often puts it: “While DuckDB turns your laptop into a personal analytics engine, MotherDuck scales your laptop into the cloud with dual execution.” In short, MotherDuck provides the collaborative and enterprise-grade layer that allows teams to harness the full power of DuckDB in a secure and scalable way. More on dual execution below.

The architectural approach of combining a powerful local engine with a lean, collaborative cloud layer is what makes the platform particularly compelling. Based on our evaluation, here are five key reasons why we are excited about this development:

## Your Data Most Likely Isn’t as Big as You Think

The data industry has been obsessed with “Big Data” for over a decade. But as MotherDuck’s founders argue, [Big Data Is Dead](https://motherduck.com/blog/big-data-is-dead/). The reality is often simpler: companies don’t operate at the petabyte scale of Google or Meta. Many analytics workloads run on datasets that are gigabytes or a few terabytes in size. This scale doesn’t always require massive, complex distributed systems.

Today’s laptops and single servers are incredibly powerful—or, as DuckDB creator Hannes Mühleisen puts it, we lived through a “[lost decade of small data](https://duckdb.org/2025/05/19/the-lost-decade-of-small-data.html).” MotherDuck and DuckDB are engineered for this small- to medium-sized data reality. They provide the power of a distributed system without the overhead, cost, and complexity, allowing you to process data efficiently where it makes the most sense.

## A Focus on Simplicity and Performance

One of the biggest barriers in data analytics is often the initial setup. Traditional systems require installing servers, configuring instances, and managing permissions before a single query can be run. DuckDB’s core mission is to eliminate this friction entirely.

There are no servers to manage. You simply install the library and start querying. This radical simplicity, combined with its high-performance query engine, changes the time-to-insight. An analyst can go from a 10 GB Parquet file to meaningful results in seconds, not hours (depending on hardware and data layout). MotherDuck extends this philosophy to team collaboration, providing the necessary infrastructure for production-ready data applications without reintroducing the complexity that DuckDB was designed to remove.

## Simplified and Secure Data Sharing

A common challenge in collaborative analytics is the manual exchange of data files, such as CSV exports. A practice that still kicks off many data science and AI projects. This approach creates data silos, raises security concerns, and leads to version-control issues with inconsistent datasets spread among different team members.

MotherDuck directly addresses this by changing the paradigm from sharing data to sharing access to data. Instead of creating data copies, teams can work from a single, consistent source of truth.

The sharing model, accessible via a web interface and controllable programmatically with standard SQL commands (e.g., GRANT/REVOKE semantics), eliminates the need for “nasty CSV exports” and provides a clean way to collaborate.

## Rich Partner Ecosystem

A data platform’s value is directly tied to its ability to integrate with the tools an organization already uses. MotherDuck embraces a “bring your own tools” philosophy, standing in contrast to more monolithic platforms.

The platform is supported by a rich and growing [ecosystem of partners](https://motherduck.com/ecosystem/), including numerous open-source projects and commercial SaaS solutions. This commitment to interoperability allows organizations to pick and choose a best-of-breed data stack tailored to their specific needs, rather than being forced into a one-size-fits-all solution.

## Dual Execution: Local Speed and Cloud Scale

Perhaps the most technically innovative feature of the platform is its approach to dual execution. MotherDuck’s intelligent query planner can distribute a single query plan between a local DuckDB instance and the MotherDuck cloud service.

This model leverages the best of both environments: the interactive speed and low latency of local processing with the scalability and durable storage of the cloud. By executing parts of a query on the user’s machine, this architecture minimizes data transfer by design, which in turn reduces both latency and cloud compute costs.

This powerful dual execution model even extends to modern web applications, where DuckDB can run directly in the browser (via WebAssembly) while being powered by MotherDuck’s backend for larger, shared datasets.

# Conclusion

SQLite revolutionized the world of transactional databases. By providing a powerful, serverless, file-based database, it became an integral part of nearly every application and device on the planet – for examples, see [here](https://www.sqlite.org/famous.html).

We believe DuckDB is poised to do for analytics what SQLite did for transactions. It brings the same principles of simplicity and high performance to a domain that has long been burdened by complexity. MotherDuck is the catalyst that scales this potential, transforming DuckDB from a personal powerhouse into a viable, collaborative platform for modern data teams.

The official launch in Frankfurt is an exciting milestone for data analytics in Europe. We are proud to be a launch partner on this journey and look forward to seeing the innovative solutions our clients will build with this technology.

What’s Next? If you are exploring ways to modernize your data stack or want to see how this new approach could benefit your team, we are here to help. Contact us for a no-obligation consultation to discuss your specific use case.

[^1]: As with any cloud service, customers should verify whether any control‑plane metadata (e.g., service logs, metrics, authentication/usage telemetry) leaves the region. Separately, each organization must assess the data‑protection compliance of hosting personal data with MotherDuck in AWS Frankfurt (eu-central-1).
[^2]: Please take performance benchmarks with the usual grain of salt, as the complexities are thoroughly explained [here](https://hannes.muehleisen.org/publications/DBTEST2018-performance-testing.pdf) by DuckDB co-creator Hannes Mühleisen: On Performance-Testing in Database Systems.

Eine Gummiente mit der Skyline von Frankfurt im Hintergrund. 

A rubber duck with the Frankfurt skyline in the background.

We're an official MotherDuck launch partner in DACH! Discover the 5 key features that make this hybrid data platform a game-changer for modern analytics.


5 Reasons Why We’re Excited About MotherDuck Launch

5 Reasons Why We’re Excited About MotherDuck Launch in AWS Frankfurt

Foto von Felix Dreißig, einem jungen Mann in einem gelben Pullover, der aus einem grünen Rahmen herausschaut

Information Security Specialist

Looking for a Kaniko alternative in GitLab CI? Explore 7 options from sticking with Kaniko to redesigning your Runner infrastructure, incl. tools like Buildah.

Seven Ways to Replace Kaniko in your Container Image Builds

Machine Learning Specialist

The evolution of AI-assisted development has reached an interesting inflection point. While tools like Claude Code offer powerful coding assistance, the traditional workflow still requires constant human intervention—approving each action, managing permissions, and dealing with the security implications of giving AI broad system access. What if we could change this paradigm entirely?

I've been experimenting with a containerized approach that transforms Claude Code from an interactive assistant into an autonomous developer, capable of working independently while maintaining security and quality standards. My proposed result is a development workflow that feels almost magical in its simplicity: write a GitHub issue, let Claude Code implement the solution completely, then review the finished pull request and even give feedback.

## The Problem with Current AI Development Workflows

Anyone who has used Claude Code extensively knows the friction points. Every file operation requires approval. Each command execution needs permission. The constant interruptions break the flow of development, and the security implications of granting broad system access make many organizations hesitant to adopt AI development tools at scale.

More fundamentally, the interactive nature of current AI development tools doesn't always match how we actually want to work with AI. In most cases we don't want to micro-manage an AI assistant—we want to delegate entire tasks and receive completed, high-quality work.

## A New Workflow Paradigm

The Claude Code Development Container enables a fundamentally different approach. Instead of interactive sessions filled with permission requests, the workflow becomes elegantly simple:

You write a detailed GitHub issue describing what needs to be built, fixed, or improved. Claude receives this issue within a secure, pre-configured container environment and works completely independently. It analyzes requirements, checks for existing pull requests, implements solutions following best practices, creates comprehensive tests, and submits a well-structured pull request. You then review the completed work and either merge it or provide feedback for iteration.

This three-step process—issue, autonomous implementation, review—maximizes both productivity and quality control. Developers can focus on architecture and design decisions while Claude handles the routine but important work of implementation, testing, and documentation.

## Technical Architecture

The solution leverages containerization to create isolated, secure environments where Claude Code can operate without restrictions while being completely separated from the host system. Built on Python 3.11 with a carefully curated toolchain including Git, GitHub CLI, and the UV package manager, the container provides everything needed for modern Python development.

Security remains paramount. The container runs with a non-root user and operates with limited permissions, ensuring Claude cannot accidentally affect the host system while maintaining the autonomy needed for effective development work. Authentication is handled through secure token-based systems, eliminating the complexity of SSH key management.

The real innovation lies in the custom command system which is part of claude code. A sophisticated `issue` command intelligently detects whether it's working on a new implementation or improving an existing pull request based on review feedback. This command encapsulates years of DevOps best practices, automatically handling everything from semantic commit messages to comprehensive testing and PR template generation.

## Beyond Traditional Development Automation

What makes this approach particularly powerful is how it handles the nuanced aspects of software development that simple automation typically misses. When working on an existing pull request, Claude analyzes review comments, categorizes feedback by type, creates improvement checklists, and systematically addresses each point. When implementing new features, it follows test-driven development practices, creates proper documentation, and ensures comprehensive test coverage.

The container environment ensures reproducibility across different systems and team members. Whether running on a developer's laptop or in a CI/CD pipeline, the behavior remains consistent and predictable. This eliminates the common problem of environment-specific issues and makes the AI development process truly portable.

## Real-World Impact

The practical implications of this approach are significant. Development teams can transform their GitHub issue backlogs from growing lists of technical debt into actively addressed items. Issues that might languish for weeks in traditional workflows can be implemented, tested, and submitted for review within hours.

The quality of work remains high because Claude Code operates within well-defined constraints and follows established best practices. The container includes pre-commit hooks, linting tools, and testing frameworks that ensure code quality standards are maintained automatically. The review process provides human oversight while maintaining the efficiency gains of autonomous implementation.

Perhaps most importantly, this approach changes how developers think about AI assistance. Instead of viewing AI as a tool that needs constant guidance, it becomes a capable team member that can be trusted with complete tasks. This mental shift opens up new possibilities for how development teams structure their work and allocate their time.

## Looking Forward

The containerized approach to AI development represents just the beginning of what's possible when we rethink the relationship between human developers and AI assistants. As the technology matures, we can envision more sophisticated delegation models, better integration with existing development workflows, and expanded support for different programming languages and frameworks.

The open architecture makes extension and customization straightforward. Teams can develop their own commands, modify container configurations, and integrate with their specific toolchains and processes. The potential for community-driven improvements and shared best practices is substantial.

This evolution from interactive assistance to autonomous development doesn't replace human developers—it amplifies their capabilities. By handling routine implementation tasks with the same attention to quality that experienced developers demand, AI enables teams to focus on the creative and strategic aspects of software development that truly require human insight.

The future of AI-assisted development isn't about better autocomplete or smarter suggestions. It's about transforming how we think about delegation, automation, and the division of labor between humans and machines in the creative process of building software.

---

*Interested in trying this approach? Check out the [Claude Code Development Container](https://github.com/dstalzjohn/agents-cli-sandbox) for implementation details and setup instructions.*



Go beyond interactive AI. A secure container turns Claude into an autonomous developer that takes a GitHub issue and delivers a complete, tested pull request.

Claude Code: From AI Assistant to Autonomous Developer

From Interactive Assistant to Autonomous Developer: Containerizing Claude Code

Recently, codecentric's security experts were tasked with testing the IT infrastructure security of a company with several hundred employees. The clients believed they were secure: The systems were running on the latest version of Windows 11 and Windows Server 2022, and a commercial antivirus product was in use.

Nevertheless: After two days, codecentric's security experts had become domain administrators with full access to all systems without the client noticing. An attacker in this position could have exfiltrated business data and encrypted or deleted all systems. How could this happen?

## Summary of the Attack

The attack was carried out as part of a so-called "assumed-breach" scenario, meaning we were provided with a normal user account without special permissions, similar to what an attacker might obtain through a phishing attack.

First, we conducted an enumeration of the domain, examining what information and data our user could access. During this process, a shared file storage was discovered through which, due to incorrect configuration, **personal files of other users could be accessed**. Many passwords were found in these files, including an old backup of the IT admin's password manager. A list was created from these passwords, which was then automatically tested against the available systems.

**This allowed access to one of the systems on which a domain administrator was logged in.** The goal was now to read the memory of the process in which the domain administrator's credentials were temporarily stored. However, the execution of malicious code was prevented by Windows Defender and the antivirus product. But **using a self-developed bypass of the AMSI (Antimalware Scan Interface), this defense mechanism was also circumvented**. Subsequently, the domain administrator's credentials could be extracted and finally used to log in to the domain controller.

## Preventative Measures

The attack described above is a good demonstration that defending IT infrastructure on a single level is insufficient. For example, virus protection is at most a temporary obstacle for experienced attackers. A multi-layered approach is necessary, where various products and procedures are used at multiple levels. This concept is also known as [Defense in Depth](https://www.cloudflare.com/de-de/learning/security/glossary/what-is-defense-in-depth/).

Here are some measures that would have made the attack described above more difficult:

- Hardening of the Active Directory domain so that, for example, a normal user can only query the most necessary information
- Introduction of appropriate password management, for example via [LAPS](https://learn.microsoft.com/de-de/windows-server/identity/laps/laps-overview), so that local administrator passwords are not used on multiple systems
- Use of [Credential Guard](https://learn.microsoft.com/de-de/windows/security/identity-protection/credential-guard/), so that credentials cannot be easily extracted from the LSASS process

The measure that would have had the greatest impact on an attacker's chances of success in this case would be the use of tools for logging, monitoring, and alerting, also known as "[Security Information and Event Management](https://de.wikipedia.org/wiki/Security_Information_and_Event_Management)" (SIEM). The idea is to centrally collect log data from various systems, analyze them for potential attacks, and trigger alarms when detected.

This measure creates visibility into the processes on client systems and in the network and **allows responding to an attack before it's too late**. **It makes the attackers' lives significantly more difficult**, because they suddenly have to be careful not to trigger warnings or alarms. They can no longer try things arbitrarily until a security measure is bypassed. The attack described above could have been quickly noticed this way.

## Technical Details of the Attack

### Step 1: Enumerating the Domain

With the provided normally privileged user account, various tools were used to gather more information about the domain:

- [SharpHound + Bloodhound](https://github.com/SpecterOps/BloodHound)
- [adPEAS](https://github.com/61106960/adPEAS)
- Various PowerShell commands

Using these tools, a detailed overview of users, computers, groups, and potential privileges was obtained.

Potential countermeasures:

- [Deny AD users read access to certain objects](https://learn.microsoft.com/en-us/answers/questions/929688/best-practice-to-prevent-active-directory-enumerat)
- Logging & Alerting (SharpHound is easy to detect)
- Make access to PowerShell in Citrix more difficult (it should be noted that completely preventing command-line calls is usually not feasible)

### Step 2: Enumerating Shared Directories

The provided user had access to many shared directories. In these directories, we specifically searched for files that potentially contained credentials. Among other things, a plaintext backup of the IT admin's password manager and credentials of a user with RDP rights were found.

Potential countermeasures:

- Restrict access to shared directories
- Do not share private user data such as Desktop or Documents. The respective user directories should only be accessible to the users themselves.
- Sensitize users so that no password lists or password manager backups are stored in plaintext → Password management
- Logging & Alerting (Automated scanning is easy to detect)

### Step 3: Local Administrator on the Admin Jumphost

Thanks to the domain enumeration, it was known that a computer existed on which domain administrators had valid and active sessions. Using the found credentials, authentication via the Remote Desktop Protocol was possible on this computer. The goal now was to expand the available permissions to read the credentials of the logged-in domain administrators with local administrator privileges.

For this purpose, a list of all known local administrator usernames and another list of all already identified passwords were created. Using the tool [crackmapexec](https://github.com/byt3bl33d3r/CrackMapExec), all combinations of these usernames and passwords were then automatically tried against all computers in the domain. This was deliberately done only against local users and not against domain users to avoid locking domain users and thus restricting availability. Through this method, valid login credentials for an administrator jumphost server were identified, among others. Thus, a login with a local system administrator was possible here.

Countermeasures:

- Introduce appropriate password management, for example via [LAPS](https://learn.microsoft.com/de-de/windows-server/identity/laps/laps-overview)
- Implement appropriate group policies to prevent domain administrators from logging into third-party systems ([Protecting Domain Administrator Groups](https://learn.microsoft.com/de-de/windows-server/identity/ad-ds/plan/security-best-practices/appendix-f--securing-domain-admins-groups-in-active-directory))
- Restrict remote desktop permissions for normal users to a necessary minimum

### Step 4: Reading the LSASS Process

The hashed credentials of active users are stored in Windows in the so-called [LSASS process](https://learn.microsoft.com/en-us/windows-server/security/windows-authentication/credentials-processes-in-windows-authentication). A local administrator has the permission to read this data. However, in this case, this was prevented by the activated virus protection. Using a so-called [AMSI bypass](https://pentestlaboratories.com/2021/05/17/amsi-bypass-methods/), the virus protection could be circumvented, and the process memory could still be read using the [Out-Minidump](https://github.com/PowerShellMafia/PowerSploit/blob/master/Exfiltration/Out-Minidump.ps1) script. This dump was then transferred to the attacker systems. To read the hashes contained therein, the tool [Mimikatz](https://github.com/gentilkiwi/mimikatz) was used on the attacker systems.

Countermeasures:

- Use of [Credential Guard](https://learn.microsoft.com/de-de/windows/security/identity-protection/credential-guard/). However, there are also [bypasses](https://research.ifcr.dk/pass-the-challenge-defeating-windows-defender-credential-guard-31a892eee22)
- Avoid active sessions and logins with domain administrators outside of domain controllers. In general, domain administrator accounts should only be used to administer the domain itself.
- Implement an appropriate [Privileged Access Model](https://learn.microsoft.com/de-de/security/privileged-access-workstations/privileged-access-access-model) for administrators

### Step 5: Generating a Kerberos Ticket

Using the extracted hash, a so-called Pass-The-Key attack could be executed. For this, the [getTGT.py](http://gettgt.py/) script from Impacket was used to obtain a Kerberos ticket with the hash.

With this ticket, the permissions of a domain administrator and thus the maximum possible privileges in the network were obtained.

Countermeasures:

- Deactivate RC4 key type in Kerberos
- Logging & Alerting

### Step 6: Post Exploitation

With these permissions, any actions in the domain environment of the network were now possible. Using impacket-secretsdump, we conducted a so-called DCSync attack, which provided the hashed credentials of all domain users as well as a plaintext password of the domain administrator.

Additionally, our own domain administrator with all privileges was created. From this position, an attacker with malicious intentions could have, for example, triggered encryption or deletion of all company data.

Countermeasures:

- Logging & Alerting

## About Us

At codecentric, we offer extensive consulting and testing in the field of IT security. Specifically in the area of IT infrastructure, we can assist you with assessments for inventory, penetration tests for verification, or Red/Purple team exercises for training a Blue team. More information can be found [here](https://www.codecentric.de/it-security).

Full control despite virus protection and modern systems – IT infrastructure can only be truly protected through a multi-layered approach.

How to truly protect IT-infrastructure

Full control despite virus protection and modern systems – How to truly protect IT-infrastructure

Recruiterin

Recruiter

In 2024, I embarked on the journey to become a recruiter for an IT Security Consulting team. I thought, “How hard can it be?” I had already been a recruiter for over 10 years, focusing predominantly on software developers, and I imagined my new task would be fairly similar. However, I quickly realized that my established methods wouldn't be as effective in this domain.

**Why?**

IT Security professionals (and particularly DFIR colleagues) are notoriously hard to find, much harder than, let’s say, Kotlin developers. They keep a very low profile on business platforms and the internet in general. In many cases, even the most sophisticated keyword searches fail to uncover their profiles, unless they actively desire to be found. So, what’s the cause?

Firstly, these professionals are acutely aware of the risks associated with personal information exposure and exploitation, given the nature of their work. Secondly, they often distrust the data collection practices of large corporations, fearing the misuse of their information for marketing or political purposes. Finally, publicly displaying their professional information online can inadvertently make them targets for “the bad guys”: hackers, attackers and malicious actors of all sorts.

So, where does this leave the eager recruiter? How do you “catch the good guys”?
###  My learnings

**Proactive Engagement:**
Even with minimal profiles, I initiated contact to gain insights within the industry. While my approach may not have always been highly tailored, I emphasized engaging with wit and curiosity. This allowed me to generate responses from fantastic future candidates and gather valuable industry insights. As much as I appreciate speaking with potential candidates, I am equally thrilled to connect with professionals who may not be looking for a new role but are happy to chat. Over the past year, this strategy has significantly enhanced my understanding of the cybersecurity community in Germany.

**Community Immersion:**
Speaking of the community and their members who often fly under the radar of classic recruitment methods, I found it to be essential to go where they are. Our team members regularly attend and speak at meetups and conferences, which I joined as often as possible to mingle with the crowd. We also exhibit at IT:SA, which has proven to be an outstanding networking opportunity. It goes without saying that this is a relatively indirect and long-term approach. 
While active recruiting efforts are generally unwelcome at dedicated industry events, my presence has allowed me to network organically and build relationships. I’ve also successfully scheduled meetings with prospective candidates at events we both attended. Added benefit: I had colleagues from the IT Security team around to help me make “the pitch”.

**Online Visibility:**
As the saying goes, "The best place to hide a dead body is on the second page of Google search results." Since IT security professionals are so difficult to locate, we wanted to make sure that they could find us. This was a collaborative effort involving leadership, marketing, the IT Security team, and HR.

We launched a dedicated sub-brand homepage for IT security, the team started publishing more blog articles, and I optimized my recruiter profile with relevant keywords. This stronger online presence has been crucial in attracting passive candidates who were quietly looking for their next professional opportunity.

**Employee Referrals:**
People referred by existing employees are far more likely to be hired than applicants coming through any other channels (direct search, job boards, events, etc.). Good people know good people. While they will still take part in the official recruitment process, these candidates have automatically undergone an “informal vetting process”. Team members wouldn’t recommend a new colleague who they don’t believe to have a suitable skillset or be a team fit, right? This approach is even more important for IT security professionals whose networks are often not openly accessible. And yes, we have been successful in hiring new colleagues through team members’ referrals over the last year. 

**Niche Platforms:**
IT Security professionals often have a deep passion for their work that extends far beyond their daily tasks. You'll find them actively involved in organizations like the Chaos Computer Club (CCC) and honing their skills on online platforms such as TryHackMe or Hack The Box – these are essentially online hacking playgrounds and vibrant cybersecurity communities.

Interestingly, some of these platforms allow you to post job roles directly and even anonymously approach members who've indicated they're "open for opportunities." While you won't know the individual's name you're reaching out to, their scores on these platforms can offer valuable insights into their technical skills and areas of expertise. (I haven't personally tried this approach yet, so if you have, I'd love to hear about your experiences!)

These are the most successful strategies I've used over the past year to grow our IT Security team, along with an outlook of what I'm still eager to try in the future. While the core approaches aren't revolutionary, adapting them for the unique challenge of finding IT security specialists was absolutely key.

Do you have any other ideas on how to find and recruit "the good guys"? I'd love to hear them!

Hiring for cybersecurity roles presents unique challenges. Here's why it's so difficult and strategies to overcome the hurdles.

My Learnings on Recruiting IT Security Professionals

How to Catch the Good Guys: My Learnings on Recruiting IT Security Professionals One Year In

IT expertise from developers for developers

CODECENTRIC BLOG

Angular 17 – Eine echte Renaissance?

Für mehr Nachhaltigkeit – CO2 sparen mit Rust

Produktstrategie als Erfolgsgrundlage: Warum du nicht mit deiner Vision starten solltest

How to gain visibility as a software developer?

Wie als Software-Entwickler sichtbar werden?

Charge your APIs Volume 26: Choosing the Right API Development Strategy: A Guide to API Design First vs. Code First Approaches

Cloud

Data

data protection

Digital Sovereignty

Generative AI

GitLab

Input-Management

IT-Security

Sustainability

Software Modernization

Here at codecentric, we embrace an open culture and enjoy sharing our knowledge. Keep diving into our world.

WISSENS-HUB

Der Softwerker Spezial Generative KI

This issue is all about the benefits and potential, but also the risks and limitations, of generative AI in software development & corporate IT.

KNOWLEDGE HUB

The Softwerker Special Generative AI

Drei Personen sitzen am Tisch in einem Meetingraum mit Laptop und besprechen sich.

Services

codecentric weiterhin unter den Top 25 der Lünendonk-Liste 2025

Every year, the renowned B2B market research company Lünendonk & Hossenfelder publishes the list “Leading medium-sized IT consultancies in Germany” - in 2025, codecentric is listed again in the top 25.

In 2025 ist codecentric erneut unter den Top 25 mittelständischen IT-Beratungsunternehmen in der Lünendonk-Liste

NEWS

codecentric remains in the top 25 of the Lünendonk List 2025

Logo SoftwerkerCast - der IT-Podcast von Techies für Techies

Community

Software development

Culture

SoftwerkerCast

The SoftwerkerCast presents Softwerker knowledge for you to listen to and review.

Events – codecentric mittendrin

Meetups, regulars' tables, hackathons, user groups: codecentric is far more than the sum of our employees and projects.

Events - codecentric in the middle of it all

Webinar: CompanyGPT mit und ohne OpenAI™

Service Lead GenAI

WEBINAR on demand

Machine Learning

CompanyGPT mit und ohne OpenAI™: Umsetzung & Herausforderungen

DISCOVER CODECENTRIC

Logo der codecentric AG, einem in Deutschland führenden IT-Consulting Unternehmen

Logo der codecentric AG einem in Deutschland führenden IT-Consulting Unternehmen

Data Effectiveness

Digital Products & Solutions

Digital sovereignty

IT Security

Artificial intelligence

Services from A to Z ...

How we work

References

Locations

Blog

Knowledge Hub

Careers

About us

News

Contact us

Do you have any questions or would you like to exchange ideas? Happy to help – just enter your contact details.

[ENG] Contact Us | Sub-Navi-Kontakt oder Zero Search Hits

Kontakt aufnehmen

Marketing & Community Management

Konntest du keine Einträge zu deinem Thema finden?  

The codecentric blog provides expert knowledge on current IT topics: Agile methods, software architecture, AI, testing and much more. ► Discover now!

Blog: IT knowledge from developers for developers

Aktuelles

Thank you for your request. I will get back to you.

Thank you for your registration. A confirmation email is on its way to you.

This page is a preview. \nClick here\n to exit preview mode.

Sorry, we did not find any matching blog articles to your search. 

Do you still have questions? Just send me a message.

You have questions or want to exchange ideas with me? Gladly - just enter your contact details.

I have read the \nprivacy statement\n of codecentric AG and confirm this by submitting the form.*

Discover exciting further topics and let the codecentric world inspire you.

I hereby consent to the collection and processing of the above data for the purpose of receiving the newsletter of codecentric AG by e-mail. You can revoke your consent at any time by sending an e-mail to datenschutz@codecentric.de, by clicking on the link in the information e-mail itself, or by sending an e-mail to the contact details listed in the \nimprint\n. I have taken note of the \nprivacy policy\n of codecentric AG and confirm this by sending the form.*

Job application form field placeholder 10

Thank you for taking part in our free online IT security check 🙏.

To receive your result, we still need your email address

Karriere Podcast Page Info CTA Paragraph Mobile

Would you like to hear more quotes from our colleagues? What is it like to work in IT consulting at codecentric? Click here for an overview of all culture & career podcasts.

Softwerkercast Podcast Page InfoCTA Paragraph Mobile

Softwerkercast Podcast Page InfoCTA Paragraph

Softwerkercast Podcast Page InfoCTA Title

Softwerkercast Podcast Page InfoCTA Button

Sorry, we couldn't find any results for your search query.

Maybe the popular search entries will help you:

Apply now as [HEADING_TITLE] at codecentric