19 Building Event Databases with AI
19.1 Orientation
Event databases convert dispersed accounts into comparable records. A researcher may begin with a question about protest, repression, political violence, institutional reform, migration, or disaster response. The relevant traces may appear across newspapers, archival catalogues, government reports, organizational records, and born-digital collections. Before generative AI, much of the work required repetitive searching, copying, normalization, and initial classification. These tasks consumed time that researchers needed for interpretation.
AI changes the feasible scale of this work. A system can expand queries, search an authorized collection, extract candidate fields, compare reports, and flag records that may describe the same occurrence. It can apply a preliminary codebook to thousands of passages and identify dimensions the original codebook omitted. Studies of text annotation show that language models can perform some bounded classification tasks quickly and inexpensively. Performance, however, varies substantially across tasks and datasets, which makes local validation indispensable.1 A useful distinction separates tasks by whether the coding material is supplied and by how much interpretation the coder must contribute. Extracting a date or place from a specified report is a different automation problem from rating a regime by drawing on unspecified material and tacit political knowledge.2
The resulting speed creates a methodological danger. A fluent extraction can make an uncertain mention look like an observed event. Several articles can make one wire report look like independent confirmation. A model can produce a precise code for an ambiguous passage or a causal probability without defining the comparison that gives the number meaning. Faster processing does not remove these problems. It can reproduce them at scale.
This chapter presents an auditable alternative. AI compresses clerical stages while the researcher preserves a visible chain from source to record, record to event, event to variable, and written evidence to causal inference. Every transition produces an artifact that can be reviewed, corrected, and versioned.
The event record begins after a document has been located and read. Manuscripts, legal records, and other historically situated sources require a finer upstream chain from source witness to transcription and source-bound observation. The next chapter, Historical Sources as Evidence, supplies that chain and extends it into the audit of historical claims.
19.2 Learn
19.2.1 Begin with the unit, not the model
An event database requires a declared unit of observation. The unit might be an attack, demonstration, arrest, policy decision, organizational founding, public statement, or interaction between named actors. Its boundary must state what counts as one event, when two reports refer to the same event, and when one report contains several events.
Write the boundary before retrieval. Record inclusion and exclusion rules, temporal and geographic scope, actor and action definitions, and the minimum evidence required for a verified record. The first codebook should follow the research question and hypotheses. It should identify the variables needed to describe the event, evaluate rival explanations, and document the reporting process.
This order matters because a model will readily propose whatever categories are easy to extract. Extractability is not theoretical importance. If the system defines the unit through its output, the database may record what the model recognizes instead of what the research design requires.
19.2.2 Build a source frame
Manual newspaper searching often proceeds title by title and query by query. A digital source frame allows the researcher to define a collection before searching it. The frame records each newspaper, archive, database, date range, language, access route, known gap, and relevant restriction. Open news infrastructures can support large-scale retrieval, while library archives can expose searchable historical newspapers and catalogue records.3 The frame should also include local and oppositional sources that a prominent aggregator may not index.
Digitized collections are not transparent windows onto the past. Selection, survival, digitization, OCR quality, licensing, and search interfaces shape what becomes discoverable. Studies of newspaper digitization show why collection composition must be examined before researchers interpret search results as historical coverage.4 A failed query may indicate absence, vocabulary mismatch, OCR error, unavailable dates, or a source omitted from the platform. Record which explanation remains plausible.
AI can accelerate work inside the source frame. It can translate query concepts, generate historical names, adapt syntax across archives, and rank retrieved documents for review. A search-connected system may also query an approved interface directly. These operations remain collection procedures. The search log should preserve the source collection, query, date, filters, result count, model or script, and review decision. Access must comply with archive rules, licenses, privacy obligations, and research ethics.
19.2.3 Separate discovery from verification
The first extraction produces a candidate record, not an event. A candidate record reports that a particular source span may describe an event within scope. It should preserve the document identifier, publication date, page or URL, exact source span, retrieval route, proposed event date, actors, action, target, location, and model output. The preserved span allows a reviewer to return to the evidence without trusting the summary.
A candidate becomes a verified record only after the relevant passage and metadata have been checked. Verification asks whether the source exists, whether the passage supports the proposed fields, whether uncertainty has been preserved, and whether the item satisfies the inclusion rule. The decision may be accepted, rejected, or marked unresolved. It must name the reviewer and date.
This separation allows wide retrieval without lowering the evidentiary threshold. The system can favor recall during discovery because false positives remain outside the verified dataset. Reviewers can then favor precision when deciding what enters the database. A model should have an abstention route for passages that do not support a stable classification.
19.2.4 Resolve reports into events
News collections rarely provide one document per event. The same account may be syndicated, translated, updated, quoted by another outlet, or repeated in a retrospective report. Conversely, one article may describe several occurrences. The database therefore needs an event-resolution stage between verified records and analytical events.
AI can compare names, dates, places, actions, and source spans to generate possible matches. Event extraction research shows, however, that simple zero-shot prompting can perform poorly when reports contain ambiguity, hypothetical language, or several interacting actors.5 A match score should therefore open a review task instead of merging records automatically.
The event resolution log records every merge, split, and non-match. It links source-record identifiers to a stable event identifier and gives the rule, evidence, reviewer, and confidence category. It also records dependence. Five newspapers repeating one wire dispatch represent broad circulation, not five independent observations. Preserving that distinction prevents source abundance from becoming false corroboration.
19.2.5 Let the codebook learn without moving silently
An initial codebook should express the research hypotheses, but it should not be treated as complete. Early records may reveal unanticipated actors, mechanisms, event sequences, source effects, or confounders. AI can scan verified records and propose additional variables that distinguish rival explanations or capture recurrent variation.
Dynamic coding requires controlled change. The codebook should contain a stable core and an experimental extension layer. Core variables remain fixed during a declared release. Proposed variables enter the extension layer with a definition, rationale, allowed values, inclusion and exclusion rules, supporting examples, and an account of the hypothesis or comparison they serve. The codebook version must accompany every coded record.
A proposed variable should be tested on a validation sample before promotion. Reviewers ask whether it can be observed consistently, whether it duplicates an existing field, whether it encodes an inference as a fact, and whether it reflects reporting practice instead of the event itself. If promoted, the team either recodes earlier events or records that prior values are missing by design. Silent addition would make change over time indistinguishable from a change in measurement.
This workflow turns AI into a critic of the original design. The model may notice a missing variable, but it does not decide that the variable belongs in the database. That decision remains theoretical, empirical, and comparative.
19.2.6 Validate before scaling
Automated coding should be evaluated against researcher-coded material drawn from the intended corpus. The validation sample should include different periods, sources, languages, common classes, rare classes, and difficult boundary cases. A temporal holdout helps expose drift. A source holdout tests whether the system has learned one outlet’s style instead of the event concept.
Validation must test absolute agreement, not only association. Weidmann, Faulborn, and García asked GPT-4o and Llama-3.1-70B to code 53 expert-rated V-Dem indicators for 171 countries in 2023. The zero-shot ratings correlated positively with the aggregated expert scores, but the models shifted the scales in opposite directions: GPT-4o was generally more pessimistic and Llama-3.1 more optimistic. Both diverged more as disagreement among human coders increased.6 The study calls these values differences, not errors, because the expert aggregate is not ground truth. Its design nevertheless demonstrates why a system can reproduce relative ordering while remaining unsuitable for absolute classification.
Report performance by variable and class. Overall accuracy can conceal failure on rare but substantively important events. Precision, recall, and error types usually provide more useful guidance. Double-code a consequential or ambiguous subset, adjudicate disagreements, and record whether each error arose from retrieval, source quality, event resolution, the codebook, or the model. Contemporary event-data research treats this as a measurement problem, not merely a software benchmark.7
Validation has a scope. A model that performs well on English national newspapers from one decade has not been validated for local-language archives or another period. Record the tested population, prompt, model, codebook version, and decision threshold. Revalidate after a material change to any of them.
19.2.7 Use Bayesian updating as a declared argument
Written evidence can change the relative plausibility of rival causal explanations. Bayesian reasoning makes this change explicit. Suppose (H_1) and (H_0) are rival explanations and (E) is a verified item of evidence. The update can be expressed as:
\[ \frac{P(H_1 \mid E)}{P(H_0 \mid E)} = \frac{P(H_1)}{P(H_0)} \times \frac{P(E \mid H_1)}{P(E \mid H_0)} \]
The first ratio represents prior odds. The second is the likelihood ratio: how much more expected the evidence would be under one explanation than the other. Explicit Bayesian process tracing can discipline comparison among explanations, but priors and likelihoods are not mechanically supplied by a document.8
AI can help prepare the reasoning. It can extract passages bearing on each hypothesis, identify temporal sequence, propose rival explanations, detect duplicated evidence, and ask what each hypothesis predicts. It can calculate posterior odds from declared inputs and run a sensitivity analysis across plausible likelihood ranges.
The researcher assigns the prior and likelihood judgments. The researcher also decides whether a report is independent, authentic, contemporaneous, and diagnostic. A percentage generated from model confidence is not a likelihood ratio. Research on confidence elicitation shows that a language model’s stated certainty requires separate calibration and cannot substitute for a causal model.9
The Bayesian evidence table should therefore record the hypothesis pair, evidence identifier, provenance, predicted observation under each hypothesis, dependence group, likelihood range, rationale, reviewer, update, and sensitivity result. The model may draft alternatives for review. It may not convert fluent interpretation into causal probability.
19.3 Worked Example
Consider a project examining whether a campaign of coordinated attacks followed a central directive or emerged from local imitation. The source frame includes national and local newspapers, organizational statements, judicial records, and a digitized archive. Retrieval returns 8,000 documents. An AI-assisted classifier identifies 730 candidate records, while human verification accepts 412 and leaves 28 unresolved.
The event-resolution pass finds that many accepted records repeat the same wire stories. It also finds articles that combine several incidents in one retrospective account. Reviewers link the 412 verified records to 167 resolved events. The database retains every source relationship, but analytical counts operate on event identifiers instead of article counts.
The original codebook records date, location, actor, target, tactic, casualties, and claimed responsibility. During a pilot, the model repeatedly proposes three additional fields: prior local confrontation, evidence of tactical imitation, and evidence of communication before the event. The first two can be coded from defined written traces. The third is too inferential as phrased. The team revises it into separate observable fields for a documented meeting, a message, and an attributed statement. After a stratified validation sample, the new fields enter codebook version 1.1 and earlier events are recoded.
For causal analysis, similar tactics across several events initially appear to favor coordination. The evidence table reveals that newspapers copied descriptions from one source and that imitation predicts similarity as well. The likelihood ratio stays close to one. An authenticated message sent before several events would be more diagnostic if its content distinguishes a directive from general encouragement. The model helps compare these implications and calculate sensitivity across researcher-assigned ranges. It does not determine which mechanism caused the campaign.
The final database is smaller than the first extraction and stronger than a manually assembled spreadsheet. AI shortened retrieval, comparison, and preliminary coding. The audit chain preserved the difference between speed and evidence.
19.4 Try It
Choose a research question that could be represented as events. Write one paragraph defining the unit, the event boundary, the source frame, and the minimum evidence required for a verified record. Then create ten candidate records from a small, rights-compliant sample. Preserve a source span for every proposed field.
Resolve the ten records into events without using automatic merges. Record one merge, split, or non-match decision for each plausible pair. Identify shared wire copy, quotation, translation, or retrospective dependence.
Draft codebook version 0.1 from the research hypotheses. Ask an AI system to propose up to five additional variables that might distinguish rival explanations. For each proposal, require an observable definition, exclusion rule, two positive examples, one difficult boundary case, and the decision it would support. Reject at least one proposal and record why.
Finally, select two rival hypotheses and one verified item of written evidence. State what each hypothesis predicts, assign a transparent likelihood range, and calculate the posterior odds across that range. Repeat the calculation after treating two apparently separate reports as dependent. Explain why the answer changed or remained stable.
19.5 Guided AI Workflow
Use three bounded passes. In the extraction pass, provide the event definition, codebook version, required output schema, and rights-cleared text. Require the model to return candidate records with exact source spans, uncertainty markers, and an abstention option. In the challenge pass, provide de-identified candidate records and ask for likely duplicates, boundary violations, missing variables, and rival interpretations. In the causal pass, provide only verified evidence identifiers, hypothesis predictions, and researcher-declared likelihood ranges. Ask for arithmetic and sensitivity checks, not a causal verdict.
A usable codebook prompt is:
Compare these verified records with codebook version 0.1. Propose no more than five additional variables that could distinguish the stated rival hypotheses. For each, provide an observable definition, allowed values, an exclusion rule, supporting record IDs, one boundary case, and the decision the variable would inform. Do not code the full corpus and do not treat an inference as an observed fact.
Permitted input: Rights-cleared source text, public records, verified metadata, redacted event descriptions, declared hypotheses, codebook versions, and stable record identifiers.
Do not provide: Restricted archives, personal data, confidential allegations, protected participant material, or documents whose licenses prohibit external processing.
Verify: Check every source span, metadata field, event match, promoted variable, dependence judgment, and causal evidence assignment against the underlying record. Validate automated codes on a stratified researcher-coded sample.
Record: Save collection and query details, model and version, prompt, codebook version, candidate output, reviewer disposition, event resolution log, validation results, and accepted or rejected codebook changes.
The model should return structured candidates, not silently edit the database. A human-controlled import step provides the boundary between proposal and record.
19.6 Integrity Checkpoint
Speed can redistribute visibility. Large searchable newspapers may dominate because they are technically accessible, while local publications, minority languages, ephemeral media, and undigitized collections disappear. Audit the source frame before interpreting the event distribution.
Source reports can contain allegations, state narratives, strategic claims, and copied errors. Preserve attribution. Do not transform “officials alleged” into an event fact. Distinguish event variables from variables about the reporting process.
An event database can expose people who never consented to systematic aggregation. Public availability does not remove privacy, security, or ethical risk. Minimize personal data, document the lawful and ethical basis for processing, and restrict or aggregate sensitive fields when release could cause harm.
Dynamic coding can also produce retrospective confirmation. A variable proposed after seeing the outcome may be analytically useful, but its origin must be disclosed. Mark exploratory variables and test them against new material where possible.
19.7 Save the Artifact
Save six linked artifacts: the source frame and search log; the candidate record table with source spans; the verification register; the event resolution log; the versioned codebook with its change log; and the validation report. For causal work, add a Bayesian evidence table that preserves hypothesis predictions, dependence, likelihood ranges, rationales, and sensitivity results.
Each analytical export should name the source-frame version, codebook version, model-assisted workflow version, and validation report. Preserve stable identifiers so a correction to one source record can propagate to the resolved event and every downstream analysis.
Release only what rights, privacy, security, and ethics permit. A public methodological register can describe restricted evidence without exposing it.
19.8 Advanced Practice
A mature pipeline separates retrieval, extraction, verification, resolution, coding, and inference into distinct operations. Each operation can be rerun without silently overwriting the preceding layer. This structure supports multilingual retrieval, active learning, model comparison, and targeted human review.
Consider a model cascade. A low-cost system can rank documents, a stronger system can extract difficult candidates, and a researcher can adjudicate the consequential cases. Route rare classes, low-confidence outputs, conflicting source spans, and causal evidence to expert review. Measure how the routing rule changes both error and labor.
Test model and source drift. Recode a fixed benchmark after provider updates. Sample new records by period and outlet. Compare human disagreement with model error without treating one as a substitute for the other.
Multiple models can be informative, but averaging is not a substitute for calibration. In the V-Dem experiment, averaging a pessimistic and an optimistic model reduced the mean difference from expert coding. The authors still caution that the direction and strength of each model’s tendency must be measured beforehand and checked on difficult cases.10 A useful ensemble benchmark therefore preserves model-specific outputs, tests each against the same frozen human-coded sample, and records when apparent balance merely cancels opposing errors.
For causal inference, compare several defensible prior and likelihood specifications. Run the analysis with and without dependent reports, contested documents, and exploratory variables. The aim is not to manufacture a precise posterior. It is to show which evidence changes the argument, which assumptions carry the result, and what new observation would alter the conclusion.
AI improves an event database when it makes the pipeline faster and the transformations more visible. It weakens the database when convenience conceals where evidence ended and inference began.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli, “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks,” Proceedings of the National Academy of Sciences 120, no. 30 (2023): e2305016120, https://doi.org/10.1073/pnas.2305016120; Nicholas Pangakis, Samuel Wolken, and Sergio Fasching, “Automated Annotation with Generative AI Requires Validation,” arXiv, 2023, https://doi.org/10.48550/arXiv.2306.00176; Caleb Ziems et al., “Can Large Language Models Transform Computational Social Science?” Computational Linguistics 50, no. 1 (2024): 237–91, https://doi.org/10.1162/coli_a_00502.↩︎
Nils B. Weidmann, Mats Faulborn, and David García, “Large Language Models Are Democracy Coders with Attitudes,” PS: Political Science & Politics 59, no. 1 (2026): 18–19, https://doi.org/10.1017/S1049096525101248.↩︎
Hal Roberts et al., “Media Cloud: Massive Open Source Collection of Global News on the Open Web,” Proceedings of the International AAAI Conference on Web and Social Media 15, no. 1 (2021): 1034–45, https://doi.org/10.1609/icwsm.v15i1.18127.↩︎
Kaspar Beelen et al., “Bias and Representativeness in Digitized Newspaper Collections: Introducing the Environmental Scan,” Digital Scholarship in the Humanities 38, no. 1 (2023): 1–22, https://doi.org/10.1093/llc/fqac037.↩︎
Erica Cai and Brendan O’Connor, “A Monte Carlo Language Model Pipeline for Zero-Shot Sociopolitical Event Extraction,” in NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023, https://doi.org/10.48550/arXiv.2305.15051.↩︎
Weidmann, Faulborn, and García, “Large Language Models Are Democracy Coders with Attitudes,” 19–21.↩︎
Patrick T. Brandt and Marcus Sianan, “Measurement of Event Data from Text,” Frontiers in Political Science 6 (2025): 1453640, https://doi.org/10.3389/fpos.2024.1453640.↩︎
Tasha Fairfield and Andrew E. Charman, “Explicit Bayesian Analysis for Process Tracing: Guidelines, Opportunities, and Caveats,” Political Analysis 25, no. 3 (2017): 363–80, https://doi.org/10.1017/pan.2017.14; Sherry Zaks, “Updating Bayesian(s): A Critical Evaluation of Bayesian Process Tracing,” Political Analysis 29, no. 1 (2021): 58–74, https://doi.org/10.1017/pan.2020.10.↩︎
Miao Xiong et al., “Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs,” in The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=gjeQKFxFpZ.↩︎
Weidmann, Faulborn, and García, “Large Language Models Are Democracy Coders with Attitudes,” 21–23.↩︎