# Method — AI answer-engine citation audit

Run on 2 September 2026. Everything below was executed by the scripts in this
directory; every figure quoted in the report is a key in `numbers.json`, which
`analyze.py` regenerates from the raw files.

## 1. Question set (`gen_questions.py` → `questions.json`)

210 companies, chosen as well-known technology firms across infrastructure and
developer tools, AI, SaaS and productivity, security and data, and commerce and
fintech. Ten question templates, each a fact a buyer or analyst would look up:
founding, most recent funding round, headcount, entry price of the cheapest paid
plan, headquarters, most recent revenue or ARR, disclosed security incidents,
current CEO, acquisitions, and the paid-tier uptime SLA.

Every company received one question on a randomly chosen template; 100 companies
drawn at random received a second question on a different template. 310
questions in total. Seed 20260902, so the set is reproducible.

## 2. Answer collection (`ask.py`, `retry.py`)

Each of the 310 questions was put to three engines through OpenRouter, at
temperature 0, with the suffix "Answer concisely with specific facts and cite
your sources":

- `perplexity/sonar`
- `perplexity/sonar-pro`
- `openai/gpt-4.1:online` (GPT-4.1 with OpenRouter's web plugin)

Citations were taken from the response's `citations` array, or from the
message's `annotations` where the array was absent. 86 of the 930 calls failed
on OpenRouter's in-flight credit limit and were re-issued by `retry.py` at lower
concurrency; all 310 questions were answered by all three engines. No question
was dropped.

## 3. Fetching every cited URL (`fetch.py`, `refetch.py`, `proxy_refetch.py`)

Every unique cited URL was requested once. Concurrency 20, 10-second timeout,
`HausResearchBot/1.0` in the User-Agent with a contact address and a link to the
research page.

Three passes, each of which can only improve a URL's classification:

1. Direct, 10-second timeout.
2. Anything that timed out or failed to connect, retried at 30 seconds, so a
   slow host is not recorded as broken.
3. Anything still not openable, retried through a rotating proxy gateway
   (a fresh exit IP per request, one retry on a proxy-side error), so a page is
   not recorded as blocked merely because one datacentre IP was unwelcome. This
   pass moved 192 of `perplexity/sonar`'s URLs, 178 of `sonar-pro`'s and 5 of
   `gpt-4.1`'s into a better class; it can never move one into a worse class.
   Pages that opened on pass 1 were left as first-pass results. The proxy
   credentials are supplied in the `HAUS_PROXY` environment variable and are not
   in any published file.

Classes are mutually exclusive and applied in this order:

| Class | Definition |
|---|---|
| `dead` | HTTP 404, 410, 451 or 5xx; DNS failure; TLS failure; or a 200 whose first 1,200 characters of visible text match an explicit "page not found" pattern (soft 404) |
| `gated` | HTTP 401, 403, 429 or 999 (LinkedIn's bot block); a bot interstitial; or subscription, paywall or login wording in the first 6,000 characters |
| `empty` | HTTP 200 with fewer than 250 characters of visible text — client-rendered shells we cannot read without a browser |
| `unreachable` | still timing out or refusing connections after all three passes |
| `live` | everything else |

`empty` exists because an earlier study of ours misread client-rendered pages as
parked domains. Nothing in that class is reported as dead.

## 4. Binding claims to citations (`pairs.py`)

Both Perplexity models mark claims inline as `[n]`, where `n` indexes the
citation array. Answers were split into sentences; for each sentence, each
distinct `[n]` produced one claim–citation pair against the exact URL cited for
it. Sentences under 25 characters were dropped. No marker in either model's
output pointed past the end of its citation list, so there were no phantom
indices. `openai/gpt-4.1:online` emits no inline markers, so it contributes to
the URL-level results only and to no claim-level result.

## 5. The deterministic check — the headline metric

From each claim we extracted its "specifics": money amounts, percentages,
magnitudes written as millions or billions, four-digit years, and any run of
three or more digits. A pair counts as **passing** when the cited page is `live`
and its visible text contains **at least one** of those figures, after
normalising commas, spaces and `million`/`billion` against their numeric
expansions (so `$185 million`, `$185M` and `185000000` all match).

The pass criterion is deliberately generous. One matching figure is enough, and
a bare year is enough. The failure rate is therefore a floor, not an estimate:
every pair we count as failing is one where the cited page does not contain a
single number from the sentence it was cited for.

Pairs whose claim contains no figure at all are outside the denominator, because
there is nothing deterministic to check.

## 6. The LLM judgment — reported separately

As a secondary, clearly-labelled metric, `google/gemini-2.5-flash` at
temperature 0 was shown each claim with the text of its cited page and asked for
SUPPORTED, PARTIAL or UNSUPPORTED. A random sample of 400 readable pairs per
Perplexity model was judged (seed 11). Where a page exceeded 9,000 characters,
the judge saw the first 3,000 characters plus the 6,000-character window with
the highest density of the claim's own terms.

This metric is not the headline. A judge model checking another model's work is
the weakest part of the design. It is also a stricter test: the deterministic
check passes on one matching figure, while the judge is asked about the whole
assertion. End to end the two differ by twenty-six points (65.9% against
40.0% for `perplexity/sonar`), which is the honest size of the uncertainty a
judge introduces.

## 7. Archivability (`wayback.py`)

A random sample of 1,500 of `perplexity/sonar`'s cited URLs was looked up in the
Wayback Machine's CDX index through the same proxy, recording whether the URL
has ever been captured and the first and last capture timestamps. 1,432 of the
1,500 lookups resolved within three attempts; the other 68 are excluded from the
denominator rather than assumed either way.

## 8. Own-site and directory shares (`analyze.py`)

A citation counts as pointing at the company's own site when the registrable
domain's first label is exactly the company's normalised name, with a hand-written
alias table for the cases where it is not (`wandb` for Weights & Biases,
`getdbt` for dbt Labs, and so on). The directory list is a hand-written set of
hosts whose company pages are generated from a database rather than written:
B2B contact and funding directories, revenue estimators and lead lists. Both
lists are in `analyze.py` and can be audited and changed.

## 9. Confidence

All proportions are reported with 95% Wilson intervals, computed in
`analyze.py`. Where two intervals overlap the report says the difference was not
resolved rather than naming a winner.

## 10. Exclusions and known limits

- One snapshot, one day. This measures the state of the citations on
  2 September 2026, not a decay rate.
- English-language questions about technology companies only.
- The `gated` class means "not readable without an account, a subscription or a
  friendlier IP". It is never reported as dead, and a human subscriber may well
  find the claim supported there.
- We fetch HTML without executing JavaScript. The `empty` class isolates the
  pages where that matters; it is 2.5% of `perplexity/sonar`'s URLs.
- The judge sample is 400 pairs per model, not the full set, which is why its
  interval is roughly ±5 points against roughly ±3 for the deterministic metric.
- The unit is the claim–citation pair, not the claim. Two thirds of the numeric
  claims carry more than one marker; scored at claim level, where one of the
  sentence's cited pages carrying a figure is enough, the failure rate is 14.4%
  instead of 34.7% (`pooled.claim_level_any_fail_pct`).
- Answers were obtained through OpenRouter, not from the Perplexity consumer
  product, and fetches were made as an identified bot from datacentre addresses.
- The 210 companies were picked by hand, not sampled from a defined universe.

## 11. Cost

OpenRouter spend for the whole run, from the account's own credit endpoint:
$674.240702 before, $702.913974 after settlement — **$28.67**. No other paid API
was used.

The endpoint lags: readings taken immediately after each stage understated the
committed spend by several dollars and kept rising for about twenty minutes
after the last call returned. Anyone reproducing this should budget from the
settled figure, not the live one.