How We Built a Scanner That Scores 150 Companies in One Session

The Short Answer

The monthly AI Visibility Benchmark runs on a single Python orchestrator called run_monthly.py. It coordinates 4 API calls per company, handles deduplication across editions, resumes interrupted scans, and auto-generates analysis. 150 companies in one session, ~$30 per run. No external frameworks beyond requests and playwright.

The Problem: Research That Scales Monthly

In March, we published the first AI Visibility Benchmark with 50 companies. Scanning was semi-manual. Each company took a few minutes of API calls, CSV wrangling, and manual review.

That works at 50. It does not work at 150. And the plan is to add roughly 100 companies per month, rescanning previous cohorts alongside new additions. By August, the benchmark will cover 500+ companies.

The question was straightforward: how do you build a pipeline that handles growing scale without growing cost or time proportionally?

What I Built

run_monthly.py is the orchestrator. It manages the full scan pipeline from company selection to final analysis output.

The architecture has four layers:

Layer 1: Company Registry

A master file — all-companies.json — holds every company ever scanned. Each entry includes domain, sector, first scan date, and a list of edition appearances. When new companies are added via --new-companies, the registry deduplicates by domain before accepting them. No company gets scanned twice in the same edition.

Layer 2: Edition Management

Each monthly run creates a directory: editions/march-2026/, editions/april-2026/, and so on. The --edition flag sets the target. Inside each edition directory: raw scan results, a CSV export, a timestamped snapshot, and analysis.md with summary statistics.

The --rescan-cohort flag pulls all companies from a previous edition into the current one. March's 50 companies appear in April's 150 automatically. This is how we track changes over time without rebuilding anything.

Layer 3: The Scanner

Each company is scored across 4 dimensions using 4 separate APIs:

→ OpenAI gpt-4o-mini — Citation Presence. "Recommend a company for [category]." Does it name the company?

→ Google Gemini 2.0 Flash — Citation Presence cross-validation. Same prompt, different model. Two independent citation checks reduce false positives.

→ Brave Search API — Entity Recognition and Citation Breadth. How widely is the company referenced across independent sources?

→ Tavily — Content Structure. Can AI extract structured information from the company's website? Tests for FAQ schema, clear headings, and parseable content.

The scanner saves results after each company. If the session crashes at company 87, restarting picks up at 88. No repeated API calls. No lost data.

Layer 4: Analysis Generation

After scanning completes, the pipeline auto-generates analysis.md. This includes:

→ Overall averages and medians → Sector breakdowns → Score distribution (how many companies in each range) → Edition-over-edition comparison (if previous editions exist) → Bottom 10 and top 10 lists

The analysis file feeds directly into the research page content. Most of the April benchmark statistics come from this automated output.

The Flags

The orchestrator uses a clean CLI interface:

→ --edition april-2026 — Target edition → --new-companies companies-to-add.json — Add new companies to the registry and this edition → --rescan-cohort march-2026 — Pull all companies from a previous edition into the current run → --dry-run — Show what would happen without calling any APIs

A typical monthly run looks like:

python run_monthly.py --edition april-2026 --new-companies new-100.json --rescan-cohort march-2026

That single command adds 100 new companies, pulls in the 50 from March, deduplicates, scans all 150, and generates the analysis.

What It Cost

The April edition — 150 companies, 4 API calls each, 600 total API requests — cost approximately $30.

The breakdown:

→ OpenAI gpt-4o-mini: ~$8 (cheapest per-call cost) → Google Gemini 2.0 Flash: ~$6 → Brave Search API: ~$10 → Tavily: ~$6

At this rate, a 500-company benchmark in August costs roughly $100. That is a full market study for the price of a team lunch.

What I Learned Building It

Resume capability is not optional. API calls fail. Rate limits hit. Sessions disconnect. The scanner saves after every single company. This turned a fragile process into a reliable one. The April scan required three restarts due to rate limiting. Total data loss: zero companies.

Deduplication prevents silent errors. Without domain-level deduplication, companies get double-counted when merging cohorts. The registry catches this before any API call is made. In April, 4 companies appeared in both the original 50 and the new 100 list. The registry flagged and skipped them.

Cross-model validation matters. Using two independent models for Citation Presence (OpenAI and Gemini) catches cases where one model hallucinates a recommendation. If only one model cites the company, the score reflects that ambiguity. Both models agreeing produces a stronger signal.

The analysis layer saves hours. Before auto-generation, I spent 2-3 hours per edition building summary tables manually. Now it takes 30 seconds and the output is more consistent.

What Is Next

The May edition will add approximately 100 UK accountancy firms — the first vertical-specific expansion beyond law firms. The scanner handles this without modification. Add the companies, set the edition, run the command.

By Q3, the goal is automated scheduling: a cron job triggers the monthly run, generates the analysis, and drafts the research page update. The human step becomes review and publish, not build and calculate.

The infrastructure is the product. The benchmark is just what it produces.