Modern AI fake news detectors are not magic filters. They are engineered systems that combine machine learning, information retrieval, graph analytics and human judgement. When we built NoMoreFakeNews at ON TIME TECHNOLOGY, we quickly discovered that any single technique used in isolation fails at scale.
What works is a layered architecture: classification pipelines for speed, source-credibility graphs for structure, retrieval-augmented verification for depth, and human-in-the-loop controls for accountability. This article unpacks how that looks in practice, for engineering teams who need to build or evaluate similar systems.
From raw content to structured signals: the classification pipeline
Every fake news detector starts as a text (or multimodal) classification problem. The goal of the initial pipeline is not to make an irreversible verdict, but to transform unstructured content into a rich set of signals.
A typical end-to-end pipeline in NoMoreFakeNews resembles:
1.
Ingestion and normalisation
- Fetch article, post or video transcript via crawlers, APIs or user submissions. - Normalise character encoding, remove boilerplate (navigation menus, ads), and detect language. - De-duplicate near-identical content using locality-sensitive hashing to avoid re-verifying clones.
We combine classic NLP and deep representations: - Tokenisation and sentence segmentation. - Named Entity Recognition (people, organisations, locations, dates, numerical quantities). - Dependency and constituency parsing to understand relationships between claims and entities. - Transformer-based embeddings (e.g. domain-tuned LLM or sentence encoder) for semantic representation. - Metadata features: domain, author handle, posting time, typical audience, propagation velocity.
1.
Claim detection and segmentation
Not all sentences are equal. A modern detector isolates atomic claims: - Use sequence labelling models to identify claim-like spans (e.g. "X said Y", statistics, concrete predictions, historical facts). - Cluster similar sentences into canonicalised claims (e.g. "5G causes COVID" variants). - Represent each claim as a structured object: subject, predicate, object, qualifiers, temporal context.
1.
Multi-head classification
We avoid a single binary label. Instead, the system runs multiple specialised classifiers, such as: - Verifiability classifier – can this claim be checked against external data at all? - Topical classifier – health, geopolitics, finance, climate, etc. - Narrative pattern classifier – matches against known disinformation templates. - Harm potential classifier – indicative of potential impact if false (e.g. medical advice vs celebrity gossip).
Each classifier outputs calibrated probabilities and intermediate features. These are stored, not discarded, because they feed later verification and graph analysis.
This pipeline is architected as a set of microservices in NoMoreFakeNews, with strict versioning of models and feature schemas, so we can audit which model produced which decision at any point.
Modelling reputation: source-credibility graphs
Pure content analysis is brittle. Attackers adapt language faster than models can be retrained. To counter this, we maintain a dynamic source-credibility graph.
Graph structure
At a high level, the graph contains:
•
Nodes: domains, accounts, authors, organisations, IP ranges, and sometimes specific content clusters.
•
Edges: relationships such as hyperlinking, co-posting, quoting, co-appearance in stories, ad-tech ties, and shared infrastructure.
Edges are typed and weighted. For example:
•
A frequently cited peer-reviewed journal linking to a site creates a positive edge.
•
A domain cluster sharing the same registrar, name server and tracking scripts as known disinformation hubs creates a suspicious edge.
•
Rapid cross-posting of identical content across a subnet of accounts may indicate a coordinated network.
Credibility scoring
Each node has a credibility score that evolves over time. This is not a moral judgement; it is a probabilistic estimate of how often content from that node is later found accurate or inaccurate, given context.
We blend:
•
Historical performance: fraction of claims later verified as true, false, misleading or unsubstantiated.
•
Expert curation: whitelists (e.g. official statistical offices, regulatory agencies) and blacklists (sanctioned propaganda outlets).
•
Network effects: graph algorithms (e.g. personalised PageRank, random walks) to infer credibility from neighbours, attenuated to avoid echo-chamber reinforcement.
•
Freshness: recent behaviour is weighted more heavily than distant history.
The credibility score becomes a prior in subsequent reasoning. When two otherwise identical claims emerge, one from a historically reliable medical journal and one from a known conspiracy outlet, the system will direct its verification resources differently and apply stricter evidence thresholds to the latter.
Critically, the graph is explainable. For regulated use cases, we can present a path of evidence: which past misinforming stories, which network relations, and which expert annotations contributed to a low-credibility assessment.
Retrieval-augmented verification: grounding claims in reality
Detecting fake news is ultimately about what is true in the world, not about how something is written. That requires retrieving and reasoning over external, trustworthy evidence.
NoMoreFakeNews uses retrieval-augmented verification in three stages:
- Normalise and paraphrase the claim into a compact, unambiguous query (e.g. convert "the government increased inflation by 20%" into a structured query about CPI change for a specific country and period). - Remove rhetorical framing and focus on verifiable content.
1.
Evidence retrieval across heterogeneous sources
We query multiple backends, each with its own trust and freshness profile: - Curated fact-checking databases maintained by reputable organisations. - Knowledge graphs (e.g. Wikidata-like structures) for stable facts: biographical data, historic events, company registrations. - Official data portals and APIs (statistics bureaus, central banks, health authorities). - High-quality news archives and academic repositories.
Retrieval is hybrid: lexical search (BM25, keyword filters) plus dense vector search over embeddings, with domain-specific rerankers. The result is a ranked list of candidate evidence snippets, each annotated with source credibility and timestamp.
An LLM-style verifier or dedicated reasoning model evaluates the relationship between the claim and retrieved evidence: - Is the evidence directly supporting, contradicting, partially related, or irrelevant? - Are there temporal mismatches (outdated statistics, pre-policy-change facts)? - Is the claim mixing accurate numbers with incorrect causal statements?
The model outputs: - A verdict distribution (true, false, misleading, unsubstantiated, out-of-scope). - A confidence score based on evidence coverage and consistency. - A rationale: a natural-language explanation citing specific data points and sources.
Retrieval-augmented verification is resource-intensive, so we use the earlier pipeline stages and credibility graph to prioritise which claims warrant deep verification. Low-impact, low-spread content may only receive lightweight checks.
Human-in-the-loop: governance, not decoration
Any serious fake news detection platform must assume that models will be wrong, sometimes systematically so. Human-in-the-loop review is therefore a design principle, not an afterthought.
In NoMoreFakeNews, humans intervene at several points:
- Random sampling of outputs to assess false positives/negatives and detect drift. - Targeted sampling of low-confidence or high-impact claims (e.g. elections, public health) for priority review.
1.
Expert panels for sensitive domains
For high-risk topics, we route contentious claims to domain specialists: - Medical professionals for clinical claims. - Economists or policy analysts for macroeconomic and regulatory claims. - Regional experts for geopolitics and conflict zones.
Their decisions are recorded as ground truth signals for retraining and calibrating models.
1.
Feedback loops and overrides
- Analysts can override automated verdicts with documented justifications. - Overrides feed into model retraining pipelines after quality control, preventing the system from perpetuating known errors. - We maintain lineage: which human decision influenced which subsequent model versions.
1.
Ethical and legal governance
- Policy teams define what kinds of content trigger strict review (e.g. political speech during elections) and what requires minimal intervention. - Legal advisors review borderline cases where classification could have regulatory or defamation implications.
For engineering teams, the implication is clear: build interfaces, data models and logging from day one with human review in mind. Without it, compliance, explainability and continual improvement are impossible.
Engineering lessons from NoMoreFakeNews
Several practical lessons emerged while building and operating NoMoreFakeNews:
•
Modularise aggressively: ingestion, NLP, graphs, retrieval and verification should be separately deployable and replaceable. This allows rapid upgrades (e.g. swapping in a new retriever) without re-architecting the entire system.
•
Log everything, but structure it: store feature vectors, intermediate scores, evidence snippets and graph states with strict schemas and version tags to enable debugging and regulatory audits.
•
Treat labels as evolving, not absolute: early human labels may later be revised as more information appears. The training pipeline must cope with label revisions and temporal validity.
•
Design for adversaries: simulate attacks (prompt injection into quoted sources, coordinated domain farms, adversarial paraphrasing) and bake in detection heuristics plus retraining procedures.
•
Separate risk tiers: the system should not treat a meme the same way it treats a viral vaccine rumour. Risk-based triage is essential for both performance and responsibility.
The road ahead: towards trustworthy information ecosystems
Fake news detection is moving from monolithic classifiers to integrated information-trust platforms. We expect three trends to accelerate:
1.
Richer, cross-platform graphs linking domains, accounts and narratives across the open web, messaging apps and emerging networks.
2.
Stronger formal verification for numerical and statistical claims, combining symbolic reasoning with neural models.
3.
User-facing transparency primitives such as machine-readable provenance, per-claim evidence cards and standardised credibility signals.
Ultimately, AI fake news detectors will be judged less on their clever models and more on their ability to provide defensible, auditable and adaptive judgements in real time. At ON TIME TECHNOLOGY, we see NoMoreFakeNews as a foundational component of that future: a rigorous, engineering-first approach to building information systems that deserve to be trusted.