Evidence-based Document Reasoning
Use AI to analyze documents and explain each finding.
Find the relevant facts across documents and compare their dates, subjects and conditions. Show what needs checking, why it matters and the source passages behind each finding.
Jupgo
Explain findings and open questions
What is document reasoning
Find what becomes clear when documents are read together.
Document reasoning uses AI to read several documents and compare the facts needed to answer a question. If the first source is insufficient, it looks for related material. The result distinguishes what the sources say from the conclusions drawn from them.
- Question and documentsFind the relevant facts
- AI connectsRelations across documents
- Compare conditionsDates, rules, conflicts
- Grounded resultConfirmed and unverified
Why it matters
Relationships across records
Align subjects, dates and conditions to find connections a single document can leave unclear.
From a finding to its source
Retain the information used in a finding so the reasoning can be inspected.
Visible unknowns
Keep missing and conflicting information visible for follow-up.
The value of step-by-step evidence
What changes when retrieval and reasoning work together?
IRCoT compared a single retrieval pass with retrieval interleaved with intermediate reasoning.
GPT3 code-davinci-002 · CoT reader in both systems · 500 evaluation questions per dataset · mean ± SD over three demonstration sets.
HotpotQA rose from 53.6 to 60.7; IIRC from 49.8 to 49.9. Gains vary with the task and evidence.
Answer F1 measures token overlap with reference answers; it is not percent accuracy. 2Wiki denotes 2WikiMultihopQA.
View all values and standard deviations
| Dataset | OneR | IRCoT |
|---|---|---|
| HotpotQA | 53.6 ± 0.7 | 60.7 ± 1.1 |
| 2Wiki | 54.8 ± 2.1 | 68.0 ± 1.5 |
| MuSiQue | 29.4 ± 0.8 | 36.5 ± 1.2 |
| IIRC | 49.8 ± 2.3 | 49.9 ± 1.1 |
Trivedi et al. · IRCoT · ACL 2023 · Appendix Table 4
MuSiQue and IIRC use answerable subsets. Calls, cost and latency were not held equal.
External Wikipedia QA research, not Jupgo performance or legal-decision accuracy.
Jupgo combines AI document analysis with auction-specific rules.
BLUEURBAN studies how AI can extract relevant facts and explain the evidence behind a finding. In Jupgo, AI extracts dates, rights and tenancy information from auction records; the system then checks those facts using auction-analysis rules.
Research behind the technology
Research and application at BLUEURBAN
Jupgo: use AI to analyze auction documents and flag rights and conditions to check.
Jupgo uses AI to extract dates, rights and tenancy details from auction documents and organize them by property. The system checks them against defined auction-analysis rules and shows what to verify, with the supporting documents. Missing or conflicting information is marked separately.
“No issue found” and “no evidence available” are different results.
Property sale statements, occupancy reports and related records
AI
- Extract dates, rights and tenancy details from the documents.
- Connect the relevant facts to their passages in the source.
Jupgo
- Compare the extracted facts with defined auction-analysis rules.
- Keep missing records, conflicts and applied assumptions explicit.
Results with distinct evidence states
- Confirmed
- The fact and its source were found
- Unverified
- The required record is unavailable
- Conflicting
- Documents disagree; further checking is needed
- Extract facts needed for a finding
- Extract rights and tenancy information from auction documents as structured records. ‘Confirmed absent’ and ‘unknown for lack of records’ are kept as different states.
- Compare dates and conditions
- The system compares the extracted dates, rights and conditions using defined auction-analysis rules. AI interpretation of document language is separate from the assessment made under those rules.
- Show what remains unverified
- Retain missing or conflicting information and applied assumptions. Insufficient evidence does not become a clean finding.
Frequently asked questions
Q1How does this differ from summarization?
It compares subjects, dates and conditions across documents to inspect relationships relevant to a finding.
Q2What happens when information is missing?
It stays unresolved, with any applied assumptions recorded. Areas needing more evidence remain visible.
Q3Does the chart measure auction-decision accuracy?
No. It reports Answer F1 in Wikipedia question answering. Jupgo’s domain-specific decisions require their own validation.
Papers and resources
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
ACL 2023
- MuSiQue: Multihop Questions via Single-hop Question Composition
TACL 2022
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
COLING 2020
- IRCoT code, data and prompts
Author repository
From document facts to findings you can check.
Review the documents, business rules and treatment of missing evidence.
Discuss document analysis