Closing the Loop: A Prompt Iteration Workflow for Databricks Genie

Part 03 covered the improvement loop in a single sentence: identify weak categories, examine the failures, apply the fix, re-run, compare. That was accurate as far as it went, but "apply the fix" is doing a lot of hidden work. In practice, it means choosing between four configuration levers, each one scoped differently, each with a different blast radius. Pick the wrong one and you can fix the failure you were looking at while quietly breaking three queries you weren't.

This article works through that choice systematically, so improvements are auditable, regressions are traceable back to a cause, and the configuration ends up looking like a deliberate specification rather than a pile of accumulated guesses.

The Four Levers

A Genie Agent's SQL generation is shaped by four kinds of configuration. Column configuration, join specifications, and example SQL live in the agent's Knowledge Store; instructions are set separately, as free-text directives. Before changing any of them, it helps to know exactly how far each one reaches.

Column configuration is the narrowest lever. Descriptions, synonyms, and example values attached to a single column only affect queries that touch that column, which makes it the right first stop when Genie is misreading a field name, picking the wrong literal for a filter, or missing a synonym a user actually typed.

Join specifications sit at roughly the same scope. Spell out how two tables relate, which columns, which direction, and you fix multi-table join failures without touching anything else. Most broken joins trace back to a relationship Genie was never told about.

Example SQL works one level up, at the pattern rather than the column. A question paired with its correct query teaches Genie the shape of an entire class of questions, without dictating anything about queries outside that class. Reach for this when Genie already picks the right table but gets the aggregation, the GROUP BY, or the join approach wrong.

Instructions are the odd one out: free text, applied globally, to every query the agent handles. They're also the most powerful lever by a wide margin, one well-placed instruction defining a time convention or naming which table owns a concept can fix a dozen failures at once, and the most dangerous, because that same reach means it can quietly rewrite behavior for queries that had nothing to do with the problem you were solving. Save instructions for last.

It rarely feels that way at the moment. Instructions are the lever people reach for first, because they read like plain English and feel like the fastest fix available. They're usually the slowest, once you count the time spent tracking down what they broke.

Reading the Harness Output

Before touching any configuration, read the per-category breakdown. The aggregate score tells you how things are going overall; the category view tells you where to look; the individual failures tell you why.

Each failure traces back to exactly one of the four levers, and finding that mapping before writing anything is really the whole job. Get it wrong and you'll likely solve one problem while creating another without noticing.

Failure pattern	Root cause	Start here
Wrong table selected	Genie doesn't know which table owns a concept	Instruction naming the table's purpose
Wrong filter value or literal	No example values exposed for that column	Column configuration: add example values
Correct table, correct structure, wrong aggregation	Expected pattern not demonstrated	Example SQL pair
Time range misinterpreted	No convention defined for terms like "last quarter" or "YTD"	Instruction defining your time conventions
Ambiguous term resolved incorrectly	Multiple valid interpretations, no explicit disambiguation	Column configuration synonym or targeted instruction
Multi-table join missing or wrong	Join key unknown or relationship not explicit	Join specification
Correct SQL, wrong result	The test case's expected value may be stale	Update the test case, not Genie

That last row is easy to skip past, but it's worth checking first. Before blaming Genie for anything, confirm the test case itself still reflects reality, tables get renamed, columns get added, values shift after a reload. Thirty seconds spent ruling this out saves a lot of time chasing a fix for a problem that doesn't exist.

It's also worth reading across cases, not just within them. One failed test in a category is a data point. Four out of five failed multi-table join tests is a pattern, and almost certainly means one missing join specification rather than four unrelated bugs, diagnose at that level, not test case by test case. Difficulty adds a second axis: failures clustered among the hard, multi-hop cases usually point to missing join specs, while failures spread evenly across easy and medium questions suggest something more basic is wrong, a table description that's missing, or a term Genie is misreading before it even starts writing SQL.

A different signal altogether is accuracy dropping across several categories at once after a change goes in. That's not four new problems, it's one change that conflicted with something else. Roll it back, confirm the baseline returns, and go at the original failure with a narrower lever. Adding more instructions on top won't fix a cross-category regression; it tends to be how the next one gets started.

The Regression Problem

Instructions don't stay isolated. They compose with whatever else is already in the agent's configuration, and that composition can produce behavior nobody actually asked for.

Here's how that plays out in practice. A Genie Agent built on a franchise data model has a transactions table and a franchises table, joined on franchiseID. Early harness runs show multi-table joins failing consistently, Genie keeps querying transactions alone instead of joining to franchises whenever a question touches franchise-level analysis. The obvious fix is an instruction: "when analyzing revenue by franchise, join the transactions table to the franchises table on franchiseID." It works. Join accuracy in the multi-table category climbs.

Then, on the next full run, something else breaks. Simple aggregations that used to hit the transactions table alone, queries with nothing to do with franchise-level breakdowns, now carry an unnecessary join. Genie generalized the instruction further than intended, and the blast radius reached queries that were fine before.

That's the regression problem, and it isn't an edge case, it's the ordinary cost of adding broad configuration to a system where instructions interact with each other. A harness with coverage across both query types will catch it immediately. One built only around the category that was failing will show the improvement and miss the damage entirely.

The defense is to ask, before writing anything, what the narrowest lever is that still solves the problem:

• Column configuration or join spec, touches only that column or relationship, no cross-category risk
• Example SQL, affects queries structurally similar to the example, limited blast radius
• Scoped instruction, targeted to a specific context, e.g. "when the user asks about franchise-level breakdown…"
• Broad instruction, applies everywhere; use it only once nothing narrower works, and run the full suite right after For the franchise example, that means skipping the broad instruction in favor of a join specification that names the franchiseID relationship directly. It gives Genie the join key without telling it when to use it, the decision of whether a join belongs stays with Genie, based on the actual question, instead of being forced by an instruction that can't tell the two query types apart.

The Iteration Protocol

Three rules follow directly from the regression problem above.

One change per iteration. Don't fix two failing categories in the same run. If both changes help, you won't know which one did it; if accuracy drops, you won't know which one to blame. It's slower per iteration and faster overall, because every result, good or bad, is attributable to something specific.

Run the full suite every time, not just the category you're working on. Cross-category damage is invisible without full coverage: a change that gains ten points on joins while quietly losing eight on aggregations isn't a win, and a partial run will never show you the loss.

Compare runs in MLflow before deciding anything. The rule is binary, not a judgment call: did the target category improve, and did every other category hold steady or improve too? If yes to both, keep the change. If anything else dropped, something conflicted, and you're back to a narrower fix.

1runs = mlflow.search_runs(experiment_names=[EXPERIMENT_NAME])
2comparison = runs[[
3    "run_id",
4    "start_time",
5    "params.change_description",
6    "metrics.sql_semantic_correctness/rating/average",
7    "metrics.category_joins/accuracy",
8    "metrics.category_aggregations/accuracy",
9]]
10display(comparison.sort_values("start_time", ascending=False))

A change_description parameter on each run is a small addition that pays for itself quickly. Without it, run IDs are meaningless and you're relying on memory to know what each one tested. With it, the table doubles as a decision log, every change, what it did to each category, whether it held up.

Do this consistently and the effect compounds in a good way. A series of small, targeted fixes that each lift one category without disturbing the others adds up cleanly over time. A series of broad instructions that each solve one thing while quietly undermining another produces the opposite: a configuration nobody can fully explain, and one that's nearly impossible to safely simplify later.

When to Expand the Test Suite

The suite you start with only covers what you thought to anticipate. Production will always surface more.

Two moments call for adding a test case right away. The first is any user-reported wrong answer, turn it into a test case immediately, with the correct SQL as the expected result, so the harness catches that exact failure mode from then on. The second is a fix that works but feels shaky: add a variant of the question that probes the same behavior from a slightly different angle. A sound fix handles the variant too; a fragile one won't, and you'd rather find that out now than in production.

There's also a slower-burning source: Genie's own conversation history. Questions that come up often but aren't in the suite are worth adding. So are questions users immediately rephrase, a sign the first answer didn't land. Follow-up questions are useful too, since they often expose an assumption about conventions the suite never tested in the first place.

Keep the suite versioned alongside the agent's configuration itself. When something in the underlying data changes, a rename, a dropped column, an updated key, a current suite is what catches the mismatch before a user does.

Configuration as Code

Run this discipline long enough and the agent's configuration stops looking like an accumulation of one-off fixes. It becomes a specification, one where every instruction, every example query, every join spec, every deliberate column description traces back to a test case that justified adding it. Configuration and test suite become two halves of the same artifact: one says what the system does, the other says what it's supposed to do.

That pairing buys you two things you don't get from configuration alone.

The first is safe deletion. An instruction with no test case tied to it can be removed without much risk, either it was never necessary, or its effect was never actually verified, and either way there's nothing to lose. Even an instruction that does have a linked test case is safe to remove, in a different sense: if it was actually load-bearing, the harness will tell you immediately.

The second is safe onboarding. Someone new to the project can run the suite to see where things stand today, then read the configuration to understand how it got there. Together they act as an executable specification, not a document describing intent, but something that actually verifies behavior, the same value unit tests bring to ordinary software engineering, applied here to how an AI agent is supposed to behave.

The companion repository shows this concretely: a genie-ontology/ folder with instructions.md, example_queries.sql, and join_specs.md, one file per lever, versioned right alongside the test suite. A Git diff on any of them tells you what changed, when, and, because the linked test case is right there, why.

It's the same semantics-as-code idea from Part 01, applied one level up. There, the artifact under version control was a metric definition. Here it's a behavioral contract: what Genie does when someone asks it a question. Treating that as code, with the same review and version-control discipline, is what makes it something you can actually trust.

Conclusion

Across four parts, this series has made one argument in stages. Part 01 was the foundation: a semantic layer that encodes business meaning so humans and AI systems work from the same definitions. Part 02 named the disciplines that keep a GenAI system from quietly rotting on top of that foundation, lean tooling, focused prompts, observable pipelines, a feedback loop that actually closes. Part 03 turned that feedback loop into working code: a harness that questions Genie, judges the SQL, and logs the results. This article is the other half, what to do once you have those results in front of you.

The rule underneath all of it is simple, even if applying it takes discipline: reach for the narrowest lever that solves the problem, in order, column configuration, then join specifications, then example SQL, then instructions, and only then the last resort. One change at a time. Full suite, every time. Follow that and the resulting configuration isn't just more accurate; it's the kind of thing someone else can open a year from now and actually understand.

An unevaluated Genie Agent runs on hope. One with a harness and a disciplined way of acting on its output runs on evidence instead, not because it stops making mistakes, but because every mistake gets caught, every fix gets measured, and the whole trajectory is written down somewhere rather than remembered.