RAG on Structured Data: Stop Treating Tables Like Documents

June 4, 2026 (1mo ago)

—

19 min read

Most RAG systems are built around unstructured data. You split text into chunks, embed those chunks, retrieve the nearest neighbors, and ask an LLM to answer with the retrieved context. This works surprisingly well for documents, PDFs, manuals, tickets, and long-form text, even though it still has familiar problems like chunk boundaries, missing context, and semantic drift.

Structured data is different.

A table is not a document that happens to have rows. A column is not a paragraph. A feature name is not always semantically close to the question it helps answer. This is where many "RAG over tabular data" systems go wrong: they embed column names, feature descriptions, sample rows, or generated summaries, then hope the retriever finds the right features. That works when the user query and feature description share obvious language. It breaks when the useful feature is conceptually related but lexically far away.

Consider a credit-risk dataset with columns like:

  • revolving_utilization
  • days_since_last_payment
  • num_hardship_plan_enrollments
  • minimum_payment_ratio
  • statement_balance_volatility
  • recent_limit_decrease_flag

If the user asks, "Which customers are most likely to become financially stressed after a rate increase?", a semantic retriever might look for "rate", "stress", or "delinquency". But some of the most useful features may never mention those words. minimum_payment_ratio can indicate whether a borrower is only barely keeping up. statement_balance_volatility may reveal unstable cash flow. recent_limit_decrease_flag may encode lender-side risk intervention. These are not direct synonyms of the query, but a domain-aware reasoning system should know they matter.

That is why structured-data RAG should usually be framed less like document retrieval and more like text-to-query.

By text-to-query, I do not mean we must always generate SQL. Sometimes the "query" is SQL. Sometimes it is a dataframe operation. Sometimes it is just selecting the right feature names so another deterministic system can fetch values. The important shift is this: the model should reason over the schema or feature catalog, produce an explicit access plan, then fetch data through controlled tools.

This framing is already visible in serious text-to-SQL systems. Take LangChain's SQL agent flow or Vercel's text-2-sql, for example, fetches schemas, decides relevant tables, generates a query, checks it, executes it, and repairs errors surfaced by the database engine. LlamaIndex's text-to-SQL guide similarly notes that putting all schemas into a prompt can overflow context, so it supports query-time table, row, and column retrieval.

The technique I propose is simple:

Maintain a feature catalog, not a vector index as the primary decision-maker. At the core, each feature only needs a name and a short description.

[
  {
    "name": "minimum_payment_ratio",
    "description": "Share of the statement balance covered by the customer's last payment."
  },
  {
    "name": "cash_advance_frequency_90d",
    "description": "Number of cash advances taken by the customer in the last 90 days."
  },
  {
    "name": "statement_balance_volatility",
    "description": "How much the customer's statement balance has fluctuated over recent billing cycles."
  }
]

You can add more context when it helps, such as a few example values, a time-window note, or a leakage warning. But that is production hardening, not the essence of the method. The important part is that the feature catalog gives the LLM enough language to reason about what the feature means.

At query time, give the user question and a slice of the feature catalog to an LLM. Ask it to return structured output:

{
  "selected_features": [
    {
      "name": "minimum_payment_ratio",
      "reason": "Indicates whether the customer is only making minimum payments, a proxy for financial stress.",
      "confidence": 0.86
    },
    {
      "name": "statement_balance_volatility",
      "reason": "Unstable balances may indicate inconsistent cash flow or changing repayment capacity.",
      "confidence": 0.78
    }
  ],
  "missing_information": [],
  "filters_or_grouping_needed": ["customer_id", "current_apr"]
}

Then fetch those columns deterministically. The LLM is not trusted to invent data. It is trusted to choose the right handles into the data system.

For small catalogs, one model call can inspect the full schema. For large catalogs, shard the feature list and run multiple LLM calls in parallel. One model sees features 1-100, another sees 101-200, another sees 201-300, and so on. Each returns candidate features with reasons. Since the features are already distinct handles, we do not need a reducer model as a core part of the system. The selected feature names can simply be unioned, validated, and passed to the deterministic fetch layer.

This resembles the idea behind Recursive Language Models: instead of stuffing a huge prompt into one context window, the model treats the long input as an external environment, decomposes it, and recursively calls models over smaller snippets. The RLM paper reports that this strategy can process inputs far beyond normal context windows while remaining comparable in cost. For structured RAG, the same principle becomes a practical schema-routing pattern: decompose the feature universe, reason locally, then union the selected handles globally.

Vector retrieval can still be useful, especially when the feature universe becomes enormous. If you have 100,000 features, you may not want to send all of them through LLM routers. In that case, embeddings can act as a broad prefilter. They might reduce 100,000 features to a few thousand candidates that are at least plausibly related. The key is not to squeeze too hard. If the vector step narrows the pool down to 10 features, it may delete the exact proxy features the LLM would have reasoned its way toward.

The key advantage is that we move from similarity to judgment.

Embedding-based retrieval starts to break down when the query and the feature description are not semantically close, even though the feature is technically useful for answering the question. LLM-based schema routing asks a different question: which features would help answer this, even if they use different words? That distinction matters most in business, finance, healthcare, operations, and risk systems, where the useful variables are often proxies.

There are still risks. The model can over-select features, miss subtle leakage, misunderstand grain, or choose columns that are unavailable for the target entity. So the system needs guardrails: validated feature names only, read-only execution, schema checks, leakage flags, confidence thresholds, and evaluation sets built from real user questions. The LLM should produce a plan, but the database or dataframe layer should enforce reality.

The best version of structured-data RAG is not "embed every column and pray." It is a layered system:

First, use the LLM as a schema and feature reasoning engine.

Second, use deterministic tools to fetch the selected data.

Third, use validators to check names, joins, grain, permissions, and leakage.

Fourth, use the LLM again to explain the result in natural language.

That is a more honest architecture for tables. Tables are already structured. The retrieval system should respect that structure instead of flattening it into prose.

References