AI pre-submission quality assurance for investment designs and evaluations
60% of DFAT evaluation plans scored less than adequate in the 2024–25 annual quality review (Bluebird Consultants, December 2025). Staff reported least confidence in reviewing plans against quality standards. Quality has not improved since 2017. The same pattern holds for investment designs: scores below 4/6 trigger mandatory revision, re-circulation to all appraisers, and weeks of delay.
The current QA process is manual and expensive. A $100M+ investment design requires three independent appraisals (2–5 days each at $1–2K/day), a peer review, and a Development Policy Committee review. Standard turnaround: 10 working days. Total appraiser cost: $6–30K per design.
Nobody checks the document before this process begins.
DFAT uses the same 9 quality criteria and 6-point scoring scale for both investment design QA and evaluation QA. This makes it possible to build one platform that does three things:
Upload a draft investment design document. The AI checks it against all 9 criteria using the official scoring matrix, identifies gaps, estimates likely appraiser scores, and flags mandatory requirements that are missing (gender outcome, disability analysis, climate consideration). Available instantly, not in 10 working days.
The same for evaluation plans and reports. Checks coverage, methodology, evidence quality, cross-cutting themes, and the specificity of recommendations. Identifies the gaps that cause formal reviewers to return the document.
The distinctive insight nobody else offers: a design’s MEL framework determines what questions the evaluation can answer years later. If the design has no quantitative targets, the evaluation has no baseline. Design QA catching this before submission saves the future evaluation from being inconclusive. One platform connects the design to its future evaluation and back.
The three prototypes span two document types (evaluation, design), three sectors (trade, humanitarian, infrastructure), three scales (single district to 15 countries), and three budget levels ($2M to $120M). The format generalises. Methodology is consistently the weakest criterion. Specific gaps are sector-appropriate.
DFAT’s own 2024–25 Review of Evaluation Quality found only 40% of plans rated adequate or better. Quality has not improved since 2017.
Landscape scan of AI tools for development evaluation: OpenEval (Finland) does evidence retrieval. UN SWEO does mapping. NVivo, Transana and DevResults help with work inside evaluations. Nobody checks evaluation or design quality before formal QA.
Scoring experiment using DFAT’s 9-criteria rubric produces scores that vary meaningfully across documents and criteria. Methodology is consistently weakest; context consistently strongest. Scores track what human reviewers would flag.
Two evaluation QA prototypes — one multi-country trade programme, one single-district humanitarian review — produce different specific findings on the same rubric. The tool discriminates by evaluation type, not just by quality level.
~86 Pacific evaluations planned over 3 fiscal years (~42 in 2025–26). Median evaluation cost AUD 89K. QA at 3–5% = AUD 225–620K/year for evaluation QA alone. Design QA adds AUD 40–200K/year.
The design QA prototype applies the same 9 criteria to an investment design document. Different focus areas activate (budget precision, delivery model, governance), but the rubric, scoring scale and output format are identical.
Buyer 1: DFAT and other bilateral donors. DFAT commissions ~37 evaluations and ~40 designs per year at thresholds that require formal QA. Each QA cycle costs $6–30K in appraiser fees and 10+ working days. A pre-submission check at $1–5K saves revision cycles and appraiser time.
Buyer 2: Implementing partners. DT Global, Abt, Palladium and Tetra Tech hold ~66% of DFAT’s ~$1.67B/year contracting portfolio. Each firm prepares dozens of design submissions and evaluation reports per year. Higher willingness to pay for competitive advantage in tender quality.
The cross-cycle feedback loop. Existing QA is document-by-document. This platform connects a design’s quality to its future evaluation’s quality — and vice versa. A design with no MEL targets guarantees an inconclusive evaluation. An evaluation with weak methodology guarantees unreliable lessons for the next design. Nobody else closes this loop.
The rubric is public; the assessment quality is the product. DFAT’s scoring matrix is freely available. What matters is whether the AI assessment is good enough that a design team would change their document after reading it. Three prototypes suggest it is: each identified specific, actionable gaps that would affect appraiser scores.
Built: Three prototype assessments across two document types and three sectors. The 9-criteria rubric encoded. The scoring matrix parsed. The evaluation pipeline quantified. The competitive landscape mapped.
Not built: An automated upload-and-assess workflow. A user interface. A pricing model. Integration with DFAT’s existing QA process. Testing with actual design teams. Any of these requires access and feedback from real users.
The concept is validated; the product is not. What would move it forward is one design team using a pre-submission check on a real document and reporting whether they changed anything as a result.
Cross-evaluation synthesis — 86% utility score, 7 cross-cutting patterns across multiple evaluations. Nobody synthesises across evaluations; this is part of the cross-cycle learning capability.
Field Notes — public reasoning summaries documenting this investigation as it developed.