Next platform deadline: Amazon EKS Nov 26, 2026  

All 14 deadlines

AI & Automation

From GenAI pilot to production: a checklist for RAG and AI agents

Many GenAI pilots never reach production. Research from Gartner and MIT points to the same causes: unclear value, cost and weak controls. A practical checklist for getting a RAG assistant or agent over the line.

By the SRASH team · · 7 min read

Key facts

  • Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing cost, unclear business value and inadequate risk controls.
  • MIT Project NANDA's 2025 report found only about 5% of custom enterprise AI tools reached production with measurable value (a directional, preliminary study).
  • Gartner forecasts worldwide AI spending to grow 49.5% in 2026.
  • The common gaps are evaluation, access control, cost control and a clear business owner.

The gap between demo and production

Building a convincing GenAI demo now takes days. Getting it approved for real users takes much longer, and many never make it. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027 because of escalating costs, unclear business value or inadequate risk controls. MIT's Project NANDA, in a preliminary 2025 study, reported that only around 5% of custom enterprise AI tools reached production and delivered measurable value.

The model is rarely the problem. The missing pieces are measurement, permissions, cost control and ownership.

The checklist

1. Value and ownership

  • A named business owner and one metric the system must move, such as handle time, deflection or hours saved.
  • A baseline measured before launch.

2. Data and retrieval

  • Sources inventoried, with owners and refresh schedules.
  • Chunking, embeddings and hybrid search chosen by testing on your content, not by default.
  • Document-level permissions enforced at retrieval time.

3. Evaluation

  • A golden set of real questions or tasks with expected outcomes.
  • Automated scoring for correctness, groundedness and safety on every prompt, model or data change.
  • Regular human review of a sample of live answers.

4. Guardrails and control

  • Input and output filtering, PII handling and refusal when sources are missing.
  • For agents: narrowly scoped tools, approval steps for actions that change data or spend money, and full action logs.

5. Cost and performance

  • Cost per request and per user tracked from day one.
  • Caching, model routing and token budgets with alerts.

6. Operations

  • Tracing of prompts, retrievals, tool calls and latency.
  • Versioned prompts and models with staged rollout and rollback.
  • A feedback loop from users into the evaluation set.

Start narrow

The projects that reach production usually start with one well-defined workflow, one data domain and one metric, then expand. A narrow scope makes evaluation possible, which makes approval possible.

Start with a free 30-minute review.

Databases, cloud or AI: talk to a senior engineer about what you run and what is coming. We reply within one business day and sign your NDA on request.