Reliable by Design: Building an Evaluation Harness for Databricks Genie

The first two parts of this series established the principles. A semantic layer gives your organization a governed vocabulary that both humans and AI can trust. A Data Flywheel gives your team the operational discipline to keep GenAI systems improving over time rather than quietly degrading. This third part is where those principles meet working code.

The subject is Databricks Genie, the natural-language "Ask Your Data" interface built into the Databricks platform. Genie lets business users ask questions in plain English and receive SQL-backed answers against their actual data. It is, in many ways, the direct application of everything the first two articles argued for: a data assistant built on governed foundations that closes the gap between business questions and trusted data. But deploying Genie without an evaluation framework is what Part 2 called a hope-driven deployment. This article describes how to make it an engineering-driven one, and how to close the Learning Loop that transforms a deployed Genie Agent into a system that measurably improves over time.

The Problem Genie Solves, and the Problem It Creates

Genie is compelling precisely because it is accessible. A finance analyst can ask "What were total sales in Japan last quarter?" without writing a line of SQL. The answer comes back in seconds, backed by the same tables the data team governs. For organizations that have invested in a semantic layer, this is the payoff: the semantic model does its job, and the end user never has to know it exists.

The problem is that accessibility creates a trust gap. When a business user gets an answer from a dashboard, they can inspect the underlying query. When they get an answer from Genie, they see a result. If that result is wrong, if Genie joined the wrong tables, misinterpreted "last quarter," or counted transactions when it should have summed revenue, the user may not know. They may not find out until the number appears in a board presentation.

The specific danger is that the answer does not look wrong. Genie presents results with the same formatting and apparent authority regardless of whether the underlying SQL is correct. There is no confidence indicator, no ranked alternative, no asterisk. An assistant that is occasionally wrong but always confident is more dangerous than one that fails visibly, because a visible failure can be caught. The evaluation harness described here exists to make that silent failure detectable.

What the Harness Does

The harness is a Databricks notebook that implements the evaluation leg of the Data Flywheel for Genie Agents specifically. It does four things:

Submits questions programmatically. Rather than testing Genie manually through the UI, the harness drives it via the Genie Conversation API, submitting natural-language questions, polling until the answer is ready, and extracting the generated SQL and query results from the response.

Compares against ground truth. Each test case carries an expected SQL query and, where the answer is a known value, an expected result. The harness checks whether the generated SQL matches the expected pattern and whether the result contains what it should.

Scores semantic SQL correctness. Exact string matching is too brittle for SQL evaluation, two queries can be syntactically different but semantically identical. The harness uses an LLM-as-a-judge to assess whether the generated SQL would produce the same result set as the expected SQL, even when the phrasing differs.

Tracks improvement over time in MLflow. Every evaluation run is logged as an MLflow experiment, creating a time series of accuracy scores. This is what turns the flywheel: you can see, run by run, whether a change to Genie's instructions or example SQL improved or regressed the system's performance.

The full notebook is available in the companion repository. It runs against samples.bakehouse, a Databricks sample dataset available in every workspace, so no data setup is required. To adapt it to your own Genie Agent, set SPACE_ID in the Configuration cell and replace the test cases with questions relevant to your data model.

Native Benchmarks

Databricks of course offers benchmarking capabilities for Genie out of the box. These benchmarks split cleanly into two modes, and neither one replaces what's happening here. Chat mode assesses accuracy by running a provided SQL answer and comparing the result set against what Genie generated — it doesn't reason about the query itself, just whether the output rows match. Agent mode does use an LLM judge, but it's grading Agent Mode's narrative report, not comparing two SQL queries for logical equivalence. So the specific thing this approach does is reading both queries and reasoning about whether WHERE country = 'Japan' and WHERE f.country = 'Japan' mean the same thing — and has no built-in equivalent. It also catches a different kind of failure than result-set matching does: a query that's structurally wrong but coincidentally returns the right rows on today's sample data won't get flagged by Chat mode's comparison, but a judge reading the SQL directly will usually catch it.

Designing the Test Suite

The test suite is the most important artifact the harness produces, and it deserves more care than it usually receives. A test suite that only covers easy cases gives false confidence. A good one is a deliberate cross-section of the questions real users actually ask.

The harness organizes test cases along two axes: category and difficulty. Categories reflect the SQL complexity involved — simple aggregations, filtered queries, GROUP BY, single-table joins, multi-table joins, time-based filters, and ambiguous phrasings where the correct interpretation is not obvious from the question alone. Difficulty runs from easy to hard, with hard cases typically involving multi-hop joins across three or more tables.

A few design principles are worth making explicit:

1. Include ambiguous questions deliberately. "What's the most popular product?" is genuinely ambiguous — does "popular" mean most transactions or highest revenue? How Genie resolves that ambiguity is a property of the system worth measuring.
2. For cases where the result is a known value, pin it. If the correct answer to "How many transactions were paid with Visa?" is 1,083, put that number in the test case. Exact-match checks on results are cheap and catch regressions that the SQL judge might miss.
3. Expect the test suite to grow. An initial set of eight questions, as in the companion evaluation harness notebook, covers the obvious cases. Production data will surface questions you did not anticipate, and those should be added to the suite as they appear.

The Polling Pattern

Genie processes questions asynchronously. When you submit a question via the Conversation API, you get back a conversation ID and a message ID immediately, but the answer is not ready yet. Genie is still fetching metadata, planning the query, generating SQL, and executing it against the warehouse. The harness polls the message endpoint on a configurable interval until the status reaches a terminal state.

1terminal_statuses = {"COMPLETED", "FAILED", "CANCELLED", "QUERY_RESULT_EXPIRED"}
2
3for attempt in range(MAX_POLL_ATTEMPTS):
4    msg_response = w.api_client.do(
5        method="GET",
6        path=f"/api/2.0/genie/spaces/{space_id}/conversations/{conversation_id}/messages/{message_id}"
7    )
8    status = msg_response.get("status", "UNKNOWN")
9    if status in terminal_statuses:
10        return {"status": status, "message": msg_response, ...}
11    time.sleep(POLL_INTERVAL_SECONDS)

Extracting SQL from the Response

Genie returns its output in a structured attachments array on the message object. The generated SQL lives at attachments[].query.query — notably, the field is named query, not sql.

1def extract_sql(message: dict) -> str:
2    attachments = message.get("message", message).get("attachments", []) or []
3    for attachment in attachments:
4        if "query" in attachment:
5            query_info = attachment["query"]
6            if "query" in query_info:
7                return query_info["query"]
8            if "sql" in query_info:   # fallback for older API versions
9                return query_info["sql"]
10    return ""

The same attachment structure carries query results and Genie's narrative text response. Separating these cleanly — SQL, result rows, and explanatory text — matters for evaluation, because each requires a different assessment strategy.

The SQL Judge

Exact SQL matching fails for a straightforward reason: WHERE country = 'Japan' and WHERE f.country = 'Japan' are the same filter. JOIN ... ON t.franchiseID = f.franchiseID and JOIN ... USING (franchiseID) are the same join. A judge that requires character-level equivalence will flag correct answers as failures.

The harness uses MLflow's make_judge to define a custom LLM evaluator that reasons about semantic equivalence rather than syntactic identity. The judge receives the original question, the generated SQL, and the expected SQL, and returns a boolean: would these two queries produce the same result set?

1SQL_JUDGE_INSTRUCTIONS = """
2You are an expert SQL evaluator for Databricks SQL. Determine whether the
3generated SQL is semantically equivalent to the expected SQL, meaning they
4would produce the same result set.
5
6Acceptable differences (still rate True):
7- Different column aliases, join syntax, whitespace, equivalent date functions
8
9Must match (rate False if different):
10- Different tables, different aggregation functions, missing/extra filters,
11  different GROUP BY columns, different LIMIT values
12"""
13
14sql_correctness_judge = make_judge(
15    name="sql_semantic_correctness",
16    instructions=SQL_JUDGE_INSTRUCTIONS,
17    model="databricks:/databricks-claude-sonnet-4",
18    feedback_value_type=bool,
19)

The binary feedback type is a deliberate choice that echoes the principle from Part 2: binary metrics are much easier to align and reason about than graded ones. A SQL query either produces the right result set or it doesn't. Asking the judge to score on a scale of one to five introduces ambiguity that makes it harder to detect regressions and harder to explain results to stakeholders.

Reading the Results

The harness produces two outputs after each run.

The first is an immediate breakdown in the notebook: completion rate overall, per-category accuracy, per-difficulty accuracy, and a list of any questions that failed entirely. The per-category view is the most actionable — it tells you not just that something is wrong, but what kind of question Genie is struggling with.

This can of course be easily visualized.

The second output is an MLflow run, logged automatically. This is what enables the flywheel. Each run records the space ID, the number of test cases, the evaluation timestamp, and the aggregate accuracy score from the SQL judge.

Here we can also see the detailed feedback of the judges to individual queries.

Comparing runs over time is as simple as querying the experiment:

1runs = mlflow.search_runs(experiment_names=[EXPERIMENT_NAME])
2display(runs[["run_id", "start_time", "metrics.sql_semantic_correctness/rating/average"]])

A rising accuracy score means your instructions and example SQL are improving. A sudden drop means a regression, either in your configuration or in the underlying model, which, as Part 2 noted, is a service that can change beneath you without warning.

The Improvement Loop in Practice

The evaluation harness is not a one-time exercise. It is the instrumentation that makes the Data Flywheel operational for Genie Agents specifically. The loop looks like this in practice:

1. Run the harness and identify weak categories. If join queries are failing at 60% accuracy, examine the specific failures. Is Genie picking the wrong join key? Does it not know that franchiseID connects the transactions table to the franchises table? The fix is usually one of four things: a text instruction that defines the relationship in natural language, an example SQL pair that teaches Genie the expected join pattern, a column configuration that exposes example values for ambiguous filter fields, or a join specification that makes the relationship explicit in the Genie Ontology.
2. Apply the fix, re-run the harness, and compare the MLflow runs. If the score in the failing category went up and nothing else regressed, the fix is good. If accuracy in another category dropped, the instruction conflicted with something else — a reminder that prompt complexity, as discussed in Part 2, is its own form of debt.

Over time, the test suite itself becomes a valuable artifact. It encodes the organization's expectations for how Genie should behave, in the form of executable specifications. New team members can run it immediately to understand the system's current state. Stakeholders can be shown accuracy scores rather than demos. And when the underlying data model changes — a table renamed, a column added, a join key updated — the harness will catch the regression before it reaches a user.

Conclusion

This article turned the principles from Parts 01 and 02 into working code: a harness that submits questions to Genie, judges SQL correctness semantically, and logs accuracy over time in MLflow. What it doesn't tell you is what to do once a category scores low. That's the subject of Part 04: how to read the harness output, diagnose failure patterns, and iterate without introducing regressions.

With a harness in place, a Genie Agent becomes something you can inspect, improve, and stand behind. Building reliable AI isn't about getting everything right on the first attempt — it's about building the feedback infrastructure that makes getting it right, eventually and continuously, a realistic expectation.