AMI Insight
A data analysis platform that gathers ad and viral performance data from several countries in one place and answers plain-language questions with queries the server validates. A product of the AMI project.
Time to check a metric
30m → 10s
one metric, from question to answer
Answer latency
2.7s avg
about $0.003 per turn
What I did
- Built Insight, a platform for asking data questions in plain language.
- Kept improving it based on feedback from the people who used it.
Problem
A question about content performance took one line, but answering it meant collecting and processing data scattered across many analytics tools and sheets. Without writing queries yourself, it was hard to get an answer.
What I built
Ad and viral performance sheets from several countries sync into Supabase every day and show up as dashboards, an Explore view and an AI chatbot. There are two ways to ask in plain language:
- Explore AI: Gemini writes a query definition (JSON), the server checks it against an allow-list, and a Postgres RPC runs the aggregation.
- Chatbot: Gemini function calling makes up to three tool calls per turn (data lookup, period comparison, weekly report) and streams the answer.
Trial and error
- A model that copies examples: Putting 26 reference Q&As into the prompt made the light model copy their shape. Fixed reports moved to code, analysis stayed with the LLM, and the 26 examples became an eval set.
- A model that gets arithmetic wrong: It miscounted list differences and sums, so totals, period comparisons and amount formatting moved to server code, and the model only transcribes the results.
- Filters the model added: Filters the model invented on its own shrank the results a lot. The server now rejects such filters, and questions with no period, with a reason, and the model tries again.
- Made-up numbers: Some answers had numbers but no query behind them. A grounding guard throws those answers away and forces a tool call.
- Silently wrong totals: A
COALESCEapplied after aggregation dropped some countries’ revenue rows, and one country’s euro payments were summed as won. Row-level calculation and currency conversion fixed both. - Gemini 3 multi-tool errors: Several tool calls in a row returned HTTP 400. Keeping the
thoughtSignaturefixed it. - Cost and abuse: Four unauthenticated POST routes could run up Gemini costs. A session check and a usage log now guard them, and the weekly insight is cached per week.
Quality and operations
- Evals: A 48-question golden set, a 100-question accuracy runner and a 35-question deep-analysis set check the answers. The two questions the accuracy runner caught were fixed with the guards above.
- Tests and deployment: There are 174 tests (Vitest), and GitHub Actions deploys to EC2 with a health check only after lint, type checks, build and tests pass. An old CI setting that hid failures was removed.
- Speed and cost: Chatbot answers take 2.7 seconds on average (p95 5.1 seconds) and cost about $0.003 per turn.
- Sync: All 984 sheet syncs over 90 days succeeded.
- Database: It uses Supabase Postgres RPCs, RLS, pg_cron and materialized views.
Results
- Checking a metric went from 30 minutes to 10 seconds.