8 Stage 5: Evaluate Evidence
8.1 Orientation
Collection answers “What did we find?” Evaluation asks “What can this record support?” The two questions should not collapse into one. A source can be relevant and unreliable for a specific claim. It can be accurate and still unrepresentative. It can be biased and indispensable.
8.2 Learn
Evaluate the record in relation to a claim. Authority alone is insufficient. A ministry may be authoritative about its own procedure and self-interested about its success. A participant may recall experience with precision and reconstruct chronology poorly. A database may standardize variables carefully while inheriting uneven news coverage.
Seven judgments belong in the evidence register. Relevance connects the source to a dimension of the question. Proximity describes its relation to the event or process. Provenance records who created and transformed it. Authority concerns competence and institutional position. Dependence identifies shared underlying sources. Representation asks whose experience enters the record. Preservation risk concerns whether the record can be checked later.
These judgments should remain separate. Combining them into one quality score hides the reason a source matters.
8.3 Worked Example
The Global Terrorism Database documents its methodology, inclusion criteria, and variables.1 That documentation makes evaluation possible. It does not make each observation equally certain. Event records depend on available sources, definitions, collection periods, and coding judgments.
Research that separates domestic and transnational incidents shows how analytical purpose can require transforming a published event universe.2 The transformed data may answer a new question. The transformation must remain visible.
One evidence-register entry for a single GTD event record shows the seven judgments.
| Judgment | Assessment for one GTD event record |
|---|---|
| Relevance | Bears on the divergent-accounts dimension: shows how one event is coded |
| Proximity | Retrospective; coded from published reports, not observed directly |
| Provenance | Created by GTD coders from news sources; transformed through the codebook |
| Authority | Authoritative on GTD coding rules, not on the ground truth of the event |
| Dependence | May share an originating wire report with database and news entries |
| Representation | Reflects events that reached English-language reporting; local silences likely |
| Preservation risk | Stable while START hosts the release; version and access date recorded |
The entry keeps the judgments separate rather than collapsing them into one score.
8.4 Try It
Select one source that appears strong and one that appears weak. Write what each can establish, what it cannot establish, and which other source family could test it. Then identify whether the two sources depend on the same originating report.
8.5 Guided AI Workflow
Provide a source description and the intended claim, not confidential full text. Ask the model for questions about provenance, dependence, missing actors, and temporal distance. Treat the output as an evaluation checklist. Do not ask for a verdict on credibility. A usable prompt: “Given this source description and the claim I want to make, list questions about provenance, dependence, missing actors, and temporal distance. Do not judge credibility.”
Permitted input: A source description and the intended claim, with no confidential full text or participant identities.
Do not provide: Restricted full text, participant-identifying material, or protected collection details.
Verify: Answer each question against the source itself and the source-family inventory. The model raises questions; it does not rate credibility.
Record: Log which questions changed the evaluation, and the model and date.
8.6 Integrity Checkpoint
Evaluation can reproduce status hierarchies. Official sources should not receive an automatic presumption of truth, and participant testimony should not be romanticized as unmediated access. Confirm before saving:
No automatic authority: no source is trusted by rank alone; each is tied to a claim.
Position stated: the source’s institutional position is recorded beside the use made of it.
Judgments kept separate: the seven judgments are not collapsed into a single quality score.
8.7 Save the Artifact
Save evidence-register-v0.1.xlsx with one entry per record or coherent collection. Required fields are the seven judgments (relevance, proximity, provenance, authority, dependence, representation, preservation risk), plus links to the source-family inventory, the search cycle, the relevant concept, and any restriction on use.
8.8 Advanced Practice
Teams should calibrate judgments through difficult examples rather than pursue artificial agreement on every record. Preserve reasoned disagreement when it affects interpretation. Consensus can erase uncertainty that belongs in the final analysis.
See National Consortium for the Study of Terrorism and Responses to Terrorism, “Global Terrorism Database Codebook: Methodology Inclusion Criteria and Variables,” 2024, https://www.start.umd.edu/gtd/downloads/Codebook.pdf.↩︎
See Walter Enders, Todd Sandler, and Khusrav Gaibulloev, “Domestic Versus Transnational Terrorism: Data Decomposition and Dynamics,” Journal of Peace Research 48, no. 3 (2011): 319–37, https://doi.org/10.1177/0022343311398926.↩︎