How we verify every citation.

Large language models hallucinate citations. They produce DOI strings that look plausible, journal names that exist, author surnames pulled from training data, and they assemble them into references that look rightand are not real. In November 2023 a Stanford study estimated that 69-88% of references generated by general-purpose LLMs were fabricated or misattributed. For working researchers this is worse than useless: the cited paper does not exist, but the argument that depended on it has been written down.

The fix is not "use a smarter model." The fix is to never let an unverified reference reach the user. Everything below is how humanovo enforces that promise.

Every claim that lands on your screen has been through this pipeline. There are no short-circuits.

for claim in hypothesis.claims:
    candidates = extract_doi_candidates(claim)
        # regex over claim text + LLM-proposed refs;
        # produces a list of (doi, confidence) pairs.

    if not candidates:
        # Nothing to verify means nothing to display.
        # The claim is preserved but flagged as "unsupported"
        # in the hypothesis trace — the user sees this.
        claim.mark("unsupported")
        continue

    for doi, conf in sorted(candidates, by=conf, desc=True):
        meta = crossref.fetch(doi, timeout=8s)
        if meta is None:
            # CrossRef has no record of this DOI; either fabricated
            # or pre-2000 + not yet indexed. Either way: not safe.
            audit.log("crossref_miss", doi=doi)
            continue

        # DOI exists. Now: does the cited paper SAY what we claim?
        if not_titlematch(meta.title, claim.attributed_title):
            audit.log("title_mismatch", doi=doi,
                      attributed=claim.attributed_title,
                      actual=meta.title)
            continue

        # Cross-check NCBI for retraction + corrigenda + EoC.
        retracted = retraction_watch.lookup(doi)
        if retracted:
            audit.log("retracted", doi=doi, source=retracted.source)
            claim.mark("retracted_source")
            break  # we will not cite a retracted paper

        # Verified. Attach the canonical metadata and proceed.
        claim.attach(verified_citation(meta))
        audit.log("verified", doi=doi)
        break
    else:
        # No candidate verified. Drop the citation rather than
        # display an unverifiable one — and the claim it supported
        # downgrades to "unsupported."
        claim.mark("unsupported")

The pipeline is designed around the failure modes a sceptical reader would test for first. Each of these is logged to the audit trail with a structured event and surfaces in the per-hypothesis trace UI.

Fabricated DOI

Generated DOI string that does not resolve via CrossRef. Caught at the crossref.fetch step. Logged as crossref_miss.

Title mismatch

DOI resolves but the actual paper has a different title than the LLM attributed. Caught via fuzzy title comparison (Levenshtein + token-set ratio). Logged as title_mismatch.

Retracted paper

DOI resolves and titles match, but the paper has been retracted. Cross-referenced against the Retraction Watch database (mirrored hourly). Logged as retracted; the citation is dropped and the claim degrades to unsupported.

Predatory journal

DOI resolves to a journal flagged on the curated mirror of Beall's list / DOAJ-removed registry. Logged as predatory_source; surfaces a warning chip in the UI.

Self-citation echo chamber

When >40% of citations on a hypothesis are by overlapping author sets, the cluster is logged as cohort_concentrated. Doesn't block — but the trace shows it so the user can weight accordingly.

AI-generated paper masquerade

Heuristic against the rising tide of LLM-generated 'papers' uploaded to lower-tier indexes. Not perfect; we catch the obvious ones (no Methods section, fabricated authorship, suspicious citation graph) and log as ai_generated_suspect.

Every event from the algorithm above is written to a tamper-evident audit log. The log is per-hypothesis, in JSON-Lines, with a Merkle-root commit hashed every hour to immutable storage (S3 with object-lock, MFA-delete). A graduate student in 2126 should be able to verify that the chain of evidence we attached to a 2026 hypothesis was the chain we actually generated, byte-for-byte.

{"ts":"2026-05-07T10:14:02.117Z","ev":"verified","doi":"10.1038/s41586-021-03819-2","stage":"grounding","model":"claude-sonnet-4.6","conf":0.94}
{"ts":"2026-05-07T10:14:02.482Z","ev":"crossref_miss","doi":"10.1234/fake.ref.2024","stage":"grounding","attempt":1}
{"ts":"2026-05-07T10:14:02.951Z","ev":"title_mismatch","doi":"10.1093/brain/awz189","attributed":"Lehmann 1991","actual":"Tau pathology in early-onset AD","stage":"grounding","conf":0.41}
{"ts":"2026-05-07T10:14:03.300Z","ev":"retracted","doi":"10.1126/science.adi2317","source":"retraction_watch","stage":"grounding"}
{"ts":"2026-05-07T11:00:00.000Z","ev":"merkle_commit","root":"7c8f2a…b41e","prior":"3a91ff…22c9","s3":"s3://humanovo-audit/h_abc123/2026-05-07T11.merkle"}

Anything that ever rendered to a user is in the log. Anything in the log can be replayed against the same CrossRef / NCBI / Retraction-Watch snapshots we used at decision time (we pin snapshots per hypothesis so "the literature changed" is never an excuse).

A user asks for hypotheses about tau aggregation in early-onset Alzheimer’s. The pipeline generates a candidate hypothesis whose mechanism cites three papers. The trace below is the actual decision log we’d show in the per-hypothesis trace UI.

Citation 1 — claimed: "Lehmann 2001, J. Neurochem"
  → extract DOI candidates       → 1 candidate (conf 0.62)
  → crossref.fetch                → resolved
  → title fuzzy-match             → 0.91, accept
  → retraction_watch              → clean
  → predatory_source              → no
  → VERIFIED  (final: doi:10.1046/j.1471-4159.2001.00407.x)

Citation 2 — claimed: "Park & Ng 2024, Nat. Med."
  → extract DOI candidates       → 1 candidate (conf 0.78)
  → crossref.fetch                → 404 — not found
  → DROPPED  (logged crossref_miss; claim degrades to unsupported)

Citation 3 — claimed: "Yamada et al. 2019, Brain"
  → extract DOI candidates       → 2 candidates
  → crossref.fetch (#1)           → resolved
  → title fuzzy-match (#1)        → 0.43, REJECT (title_mismatch)
  → crossref.fetch (#2)           → resolved
  → title fuzzy-match (#2)        → 0.87, accept
  → retraction_watch              → clean
  → VERIFIED  (final: doi:10.1093/brain/awz189)

Hypothesis trace (rendered to user):
  ✓ 2 of 3 citations verified
  ⚠ 1 claim degraded to "unsupported"
  Audit log: /audit/h_abc123.jsonl  (Merkle root: 7c8f2a…b41e)

The user sees a hypothesis with two verified citations and one explicit gap. The fabricated reference never reached them.

Honesty about scope is part of the methodology. The things we don’t yet verify, but plan to:

Quote-level grounding

Right now we verify the citation exists and the title matches. We don't yet verify that the cited PARAGRAPH says what we claim it says. Next quarter: extract the cited passage from the paper PDF / OA fulltext and check the claim against it.

Image-claim verification

If the hypothesis mentions a figure ('the Western blot in Lehmann fig. 3 shows…'), we currently don't fetch and verify the figure. On the roadmap.

Author-disambiguation against ORCID

We match author names but don't yet resolve to ORCID IDs. ORCID match is a stronger signal for papers with common surnames.

Automated peer-review flag

Whether a paper has been independently replicated, has corrigenda, or is part of a contested debate. Currently we surface the citation; we don't yet annotate the cohort context.