Beyond Code Generation: VaultIQ Cognition™ Solves 66.62% of BIRD Benchmark Questions

From generating code to executing intelligent analysis

Large language models have made it easier to generate SQL queries and Python programs from natural-language instructions. Enterprise analytics requires more than code generation.

A user asking a question about business data expects an answer. Producing that answer requires understanding the data, selecting appropriate operations, executing them, handling errors, and validating the results.

VaultIQ Cognition™ brings these capabilities together in an AI-driven analytical execution system.

Cognition combines language-model reasoning with computational tools and an execution workflow. It plans analytical operations, generates and executes code, examines results, validates outputs, and supports corrective actions.

The system integrates with VaultIQ's enterprise knowledge platform, enabling AI-driven analysis of databases, structured files, and other authorized information sources.

Putting Cognition to the test

We evaluated VaultIQ Cognition™ against BIRD, a recognized benchmark for natural-language database querying.

BIRD challenges AI systems to answer questions over realistic databases. Successful execution requires understanding database schemas, identifying relevant tables and columns, interpreting business terminology, constructing SQL, and returning correct results.

The broader benchmark contains more than 12,700 question-SQL pairs across 95 databases and over 37 professional domains.

Our evaluation covered 1,534 questions from the BIRD development benchmark.

We ran the benchmark against our existing Cognition system without making BIRD-specific optimizations or tuning Cognition for the benchmark. This allowed us to measure its general analytical capabilities against an external set of database problems.

66.62% of benchmark questions solved

VaultIQ Cognition™ successfully solved 1,022 of 1,534 BIRD questions across benchmark testing, achieving a 66.62% question solve rate.

The evaluation used BIRD's execution-based scoring methodology to determine whether the generated SQL returned the expected results.

Cognition operated through its analytical execution workflow, combining AI reasoning with actual computation rather than relying on SQL generation alone.

The evaluated questions covered database joins, filtering, aggregation, calculations, and domain-specific interpretation.

One benchmark question required identifying ZIP codes associated with charter schools in a California education district. Cognition identified the relevant datasets, generated SQL joining the appropriate tables, applied the requested conditions, and produced the correct result.

What makes Cognition different?

AI development frameworks provide tools for building agents that generate code, execute programs, and coordinate workflows.

VaultIQ Cognition™ brings analytical execution and result verification together as an integrated enterprise capability.

Its architecture connects several essential operations:

• Analytical planning: Determine the operations needed to answer a question.
• Code generation and execution: Translate those operations into executable SQL or Python and run them against authorized data.
• Result verification: Check computed outputs against execution evidence and validation rules.
• Error recovery: Use execution feedback to identify problems and attempt corrections.
• Enterprise integration: Make analytical results available within VaultIQ's broader information workflows.

Cognition goes beyond generating instructions for another system to execute. It performs computational work and processes the resulting evidence as part of answering the user's question.

Toward a new generation of enterprise analytics

The BIRD evaluation demonstrates VaultIQ Cognition's ability to answer complex natural-language questions through database reasoning and executable analysis.

The same architectural approach extends beyond relational databases. Cognition can work with generated Python programs and analytical libraries to process structured files and support technical and scientific workflows.

The results are particularly encouraging because they were achieved without optimizing Cognition for BIRD. They provide a baseline for measuring improvements to the system's general analytical capabilities.

Our ongoing work focuses on expanding the range of supported data sources, improving analytical accuracy, strengthening validation, and reducing execution time.

VaultIQ Cognition™ is part of our broader effort to make enterprise information directly usable by AI systems that can reason, compute, and deliver evidence-backed answers.

Learn more at vaultiq.ai.