Product prototype #2 — Evaluation Quality Assurance

Pre-Submission Quality Check

An AI assessment of the Independent Review of the South Fly Resilience Plan (Roselyne Kenneth & James Lloyd (QTAG / Oxford Policy Management), June 2022, 40 pages) against DFAT's 9-criteria quality framework.

Second prototype. The first prototype assessed a multi-country trade programme evaluation (PACER Plus, 62 pages, 8 countries). This second prototype tests whether the same QA format works on a very different evaluation: a single-district resilience and humanitarian review in remote Papua New Guinea. Same rubric, same format, different sector, different scale, different evaluation type.

Summary

3Critical gaps

9Important gaps

10Minor gaps

4.0Average rating (of 6)

22 gaps identified across 9 criteria. 1 criterion passes without significant issues. The weakest criterion is Methodology (3/6) — the same pattern found in the first prototype and in DFAT's own sector-wide quality review. Addressing the 3 critical and 9 important gaps before submission would materially improve the quality rating.

Cross-evaluation comparison

	PACER Plus (Prototype 1)	South Fly (Prototype 2)
Type	Multi-country trade programme	Single-district resilience/humanitarian
Pages	62	40
Countries	8 Pacific states	1 (PNG, South Fly District)
Total gaps	19	22
Critical	3	3
Average rating	4.2/6	4.0/6
Weakest criterion	Methodology (3/6)	Methodology (3/6)
Strongest criterion	Context (5/6), Eval Questions (5/6)	Context (5/6)

Pattern confirmed: Methodology is the weakest criterion in both evaluations — consistent with DFAT's finding that only 65% of evaluations were adequate on methodology in 2022. Both have strong context and weaker recommendations. The specific gaps differ by evaluation type: PACER Plus lacks sampling strategy for multi-country coverage; South Fly lacks ethical framework for vulnerable community research. The QA format produces sector-specific, actionable findings rather than generic templates.

Assessment by criterion

1. Purpose of evaluation · 2. Scope of evaluation · 3. Appropriateness of methodology and use of sources · 4. Adequacy and use of M&E data · 5. Context of the initiative · 6. Evaluation questions · 7. Credibility of evidence and analysis · 8. Recommendations · 9. Executive summary

Criterion 1: Purpose of evaluation 4/6 — Needs attention

Purpose is clearly stated: to communicate lessons and justify recommendations for adapting food and water security programming, and to inform the Western Province Partnership design. Terms of reference are appended (Annex 1), which is good practice. However, primary users are not identified by specific role, and no sensitive information or publication protocols are described.

[Important] Primary users not identified by title or role The review is submitted to 'James Marshall – Australian High Commission, Papua New Guinea' but does not identify the broader set of intended users by role. The ToR mentions 'AHC', implementing partners, and the incoming managing contractor, but does not specify which roles within those organisations will use the findings — e.g., DFAT First Secretary, Programme Manager, Post Activity Manager. The DFAT standard asks for identification by title.

Suggested fix: Add a paragraph identifying the specific roles who will use the review: e.g., 'DFAT First Secretary (Post), Western Province Programme Manager, APSP Team Leader, District Adviser, incoming Western Province Partnership managing contractor.' This helps the review team tailor findings and recommendations.

Reference: p.1 and Annex 1

[Minor] No publication or sensitive information protocols The report discusses sensitive findings (community dissatisfaction, gender-based violence prevalence, implementing partner performance) but does not describe how sensitive information is handled — anonymisation, aggregation, or publication intent. Given the small number of implementing partners and identifiable community locations, this is a meaningful omission.

Suggested fix: Add a statement on how sensitive findings are communicated (anonymised community identities, aggregated partner performance), and whether the report is intended for publication on dfat.gov.au.

Reference: p.1 / Annex 1

[Minor] Relationship to prior evaluations not described The ToR references a prior QTAG Analytical Review Paper (October 2021) and prior Building Resilience in Treaty Villages programming (since 2015), including a 2019 programme evaluation by Butler et al. However, the purpose section does not describe what those prior reviews found or whether their recommendations were implemented.

Suggested fix: Add a brief paragraph summarising the 2019 Treaty Villages evaluation findings and the October 2021 analytical review, noting which prior recommendations were actioned. This establishes the review's value-add beyond prior assessments.

Reference: Section 1.4, pp.1–4

Criterion 2: Scope of evaluation 4/6 — Needs attention

Scope is clearly defined: effectiveness, efficiency, coherence, sustainability, and gender equality. The ToR includes 15 guiding questions across five criteria. Timeline is described (4 phases, March–June 2022). However, team composition in the main report is minimal — only two names — and person-day allocation is absent.

[Important] Team roles and contributions not described in main report The report identifies two authors (Roselyne Kenneth and James Lloyd) but does not describe their respective expertise, field roles, or how they divided responsibilities. The ToR specifies that the team should include an international team leader, a PNG-based deputy, a Western Province coordinator, and a panel of experts — but the main report does not confirm which roles were filled or how the panel contributed.

Suggested fix: Add a paragraph in the methodology section describing team composition: who led the review, who had local knowledge, who conducted community consultations (and in what language), who provided quality assurance. Given the cross-cultural context, this is important for assessing the credibility of community-level findings.

Reference: Section 1.3, p.1

[Minor] No person-day allocation described The ToR describes a four-phase timeline but the main report does not state how many consultant days were used, how time was split between Port Moresby, Daru, and community visits, or how many communities were visited. The report mentions a 'two-week field trip' (p.1) but does not quantify the effort.

Suggested fix: Add a brief table: total person-days, days in Daru and communities, number of community visits, and time spent on analysis and drafting. This helps readers assess whether the scope was realistic for the time and budget.

Reference: Section 1.3, p.1

Criterion 3: Appropriateness of methodology and use of sources 3/6 — Critical gap

The weakest criterion, matching the sector-wide pattern DFAT's quality review found. The methodology description is four sentences. No evaluation matrix. No data analysis methods. No sampling strategy despite acknowledged notification failures. No ethical framework despite working with vulnerable remote communities, observing gender-based violence, and encountering power dynamics between implementing partners and communities.

[Critical] Methodology section is four sentences Section 1.3 (Methodology) contains only: (1) timing (late April and May 2022), (2) 'employed elements of theory-based approaches', (3) pre-mission reading followed by consultations, (4) Aide Memoire and feedback, (5) note on notification failures. There is no evaluation matrix linking the 15 guiding questions to specific data collection methods. No description of how interview data, document analysis, and field observations were combined. The report draws substantial conclusions about programme effectiveness, sustainability, and gender — but the reader cannot assess the methodological basis for those conclusions.

Suggested fix: Expand Section 1.3 to include: (a) an evaluation matrix linking each guiding question to specific methods and data sources; (b) description of consultation approaches (semi-structured interviews, focus groups, community meetings — and in what language); (c) document analysis approach; (d) how qualitative data was analysed (thematic analysis, framework analysis); (e) how findings from different sources and stakeholder types were triangulated.

Reference: Section 1.3, p.1

[Critical] No sampling strategy despite acknowledged notification failures The report acknowledges 'some communities did not feel they were adequately notified of the visit' and that this 'impacted the tenor of responses' (p.1). But there is no description of how communities were selected for visits, how many of the 52 SFRP villages were visited, how stakeholders were selected within communities, or how the team ensured representation across genders, wards, and implementing partner areas. Annex 3 lists organisations consulted but not the number of individuals, community members, or the selection rationale.

Suggested fix: Add a sampling section: which communities were visited and why (purposive sampling criteria), target and achieved sample by stakeholder type (government, implementer, community men, community women), and how the team addressed the acknowledged selection and notification bias.

Reference: Section 1.3, p.1; Annex 3, pp.31–32

[Critical] No ethical framework despite working with highly vulnerable communities The review describes: (a) communities being told they would 'not be further assisted' if they hadn't prepared appropriate pits (p.8); (b) women concerned about lack of progress; (c) very high rates of intimate partner violence; (d) disabled people present at meetings but not participating; (e) community alarm when hydrogeological surveys were conducted without adequate explanation. Despite documenting these serious power dynamics and vulnerabilities, the report describes no informed consent procedures, no confidentiality protocols, and no ethical framework for community-level research in remote PNG.

Suggested fix: Add an ethics section covering: informed consent process (particularly important given language and literacy barriers), confidentiality assurances for community members who criticised implementing partners, how data from vulnerable individuals is stored, whether DFAT or institutional ethics approval was sought, and specific measures for consulting women separately on gender-based violence.

Reference: Section 1.3; Sections 2.4 and 6

[Important] No data analysis methods described The report presents findings organised by DAC criteria with community examples and budget figures interspersed. But there is no description of the analytical approach. How were field notes or interview records processed? How were findings validated across communities? How were conflicting views (e.g., implementer vs community perspectives on progress) reconciled? The phrase 'theory-based approaches' is not explained or operationalised.

Suggested fix: Describe the analytical framework: what 'theory-based approaches' meant in practice, how the theory of change was used to structure analysis, how themes were identified, how contradictory evidence was weighed, and how the team moved from data to the seven recommendations.

Reference: Section 1.3, p.1

[Minor] Data collection tools not appended The ToR specifies guiding questions, but the actual interview guides or community consultation protocols used in the field are not provided. Given the report's emphasis on the need for culturally sensitive engagement, the tools used for that engagement are relevant evidence of methodological quality.

Suggested fix: Append interview guides and community consultation protocols, including any adaptations for Tok Pisin or local languages and any visual aids used in community meetings.

Reference: Annexes

Criterion 4: Adequacy and use of M&E data 4/6 — Needs attention

The review engages substantively with the M&E framework (MEF), including its late delivery (one year after programme launch) and its limitations. Output summary data is cited. Monthly food and water security reports are referenced. The results framework (Annex 2) is included. However, the review does not systematically describe what M&E data was available and how it was used as an evaluative input.

[Important] No systematic description of M&E data used as evaluative input The review discusses the MEF as a finding (its assumptions, timing, limitations) but does not describe, as a methodological input, what monitoring data the review team had access to and how it informed their assessment. For example: were the monthly food and water security reports analysed systematically? Were output tracking records compared against work plans? Were the Ranger utilisation data or Ward Profile quality assessed? The output summary (April 2022) is referenced but not appended or described.

Suggested fix: Add a paragraph in the methodology listing the specific M&E data sources the review team accessed (MEF, monthly situation reports, output tracking records, Ranger logs, financial reports, Ward Profiles) and how each informed the findings. If M&E data was inadequate, explain what the team used instead.

Reference: Sections 1.3 and 2.1

[Minor] Output data not disaggregated by gender or disability in the report itself The review recommends (p.11) that output-level results 'should be disaggregated by gender and disability' but does not itself report disaggregated data. For example, the 40 recipients of financial literacy training are described as '20 men and 20 women' in one place but other outputs (water wheels, Ward Profile workshops) are not disaggregated.

Suggested fix: Present all available output data disaggregated by gender and, where available, disability status. Where disaggregated data was not available, note this as a limitation.

Reference: Sections 2.2 and 2.3

Criterion 5: Context of the initiative 5/6 — Pass

Strong performance. The review provides excellent contextual depth: population densities, multidimensional poverty data, COVID-19 impacts, geopolitical dynamics (Indonesia border, Torres Strait), climate events (droughts, floods, king tides), disease burden (TB, HIV). The programme history from Building Resilience in Treaty Villages (2015) through COVID-19 response (2020) to the SFRP is clearly traced. The CSEP alignment is articulated. Implementing partner roles and budget allocations are tabled. The challenging operating environment is conveyed through specific examples rather than generalisation.

No significant gaps identified. This criterion meets the standard.

Criterion 6: Evaluation questions 4/6 — Needs attention

The ToR (Annex 1) contains 15 well-structured guiding questions across five DAC criteria. The report is organised around these criteria and addresses the questions substantively. However, the questions are in the annex rather than prominently in the main text, and the report does not explicitly map findings back to specific numbered questions.

[Important] Questions not explicitly linked to findings in the main report The ToR includes detailed guiding questions (pp.27–29) with cross-references to where each is addressed. But the main report sections do not reference the specific question numbers they are answering. A reader must cross-reference between the ToR and the report to know which question is being addressed. The ToR's cross-referencing helps, but the main report should be self-standing.

Suggested fix: At the start of each section (Effectiveness, Efficiency, Coherence, Sustainability, Gender), list the specific guiding questions being addressed, numbered consistently with the ToR. This makes the report navigable as a standalone document.

Reference: Sections 2–6; Annex 1 pp.27–29

Criterion 7: Credibility of evidence and analysis 4/6 — Needs attention

Findings draw on multiple sources: document analysis, stakeholder consultations, community visits, budget data, and output records. Specific examples from communities add credibility (the flooded pit latrine incident, the hydrogeological survey alarm, Rangers' declining utilisation). Financial data is included. However, the review relies heavily on hedging language, and contribution analysis is absent despite the complex multi-actor delivery model.

[Important] No contribution analysis despite complex delivery model The SFRP involves four implementing entities (APSP/Abt, RRRC/INLOC, WVI, shipping), multiple government levels (District, Provincial, National), and prior programming. The review assesses whether progress is being made but does not explicitly analyse which actors' contributions were most significant, or whether observed changes (e.g., WASH governance improvements) are attributable to WVI's approach or to broader governance trends. The standard requires that 'attribution and/or contribution to results are explained.'

Suggested fix: Add contribution analysis for each component: what would likely have happened without the SFRP intervention (counterfactual), which implementing partner's approach was most effective and why, and where results are attributable to factors outside the programme (e.g., existing Ranger programme, government momentum).

Reference: Sections 2–5

[Important] Evaluator's position obscured by hedging language The review frequently uses qualified language: 'there seems a need to' (used 5 times), 'value will be found in', 'it will be useful to', 'more can be done', 'challenges could arise'. While appropriate caution in cross-cultural evaluation, this hedging means the reader cannot always tell whether the evaluators view performance as adequate or inadequate. For example, 'progress against outputs is behind schedule' (p.10) — is this a serious concern or an expected result after 12 months?

Suggested fix: For each major finding area, add a clear evaluative judgment: 'Progress on [component] was [adequate / less than adequate / inadequate] because [key evidence]. This reflects [expected / concerning] performance given the programme's stage and operating environment.'

Reference: Sections 2–6

[Minor] Community voices not systematically presented The report includes powerful specific examples (women digging pits, alarm at surveys, lack of progress communication) but these appear as illustrative anecdotes rather than as systematically collected evidence. The reader cannot tell how many communities expressed similar views or whether the examples represent widespread or isolated experiences.

Suggested fix: When presenting community findings, indicate the breadth of the evidence: 'In X of Y communities visited, stakeholders expressed...' or 'This view was echoed in multiple communities across [area].' This strengthens the evidence base without requiring statistical precision.

Reference: Sections 2.4 and 6

Criterion 8: Recommendations 4/6 — Needs attention

Seven numbered recommendations, consolidated in Section 7.1 and placed at relevant points throughout the text. Each flows logically from the analysis. Good thematic coverage: engagement, resourcing, Rangers, coordination, sustainability, gender, GESI function. However, recommendations are framed as areas for improvement rather than specific actions, with no resource estimates, timelines, or responsible parties.

[Important] Recommendations are directional rather than specific and actionable Recommendations use language like 'consider how best to improve engagement' (Rec 1), 'an increase in resourcing is justified' (Rec 2), 'should be more actively engaged' (Rec 3), 'identify pathways to sustainability' (Rec 5). The standard asks that recommendations be 'clear, specific, relevant, targeted and actionable.' For example, Recommendation 2 says resourcing should increase but does not say by how much, in which budget lines, or what additional staff should be hired.

Suggested fix: For each recommendation, add: (a) a specific action with a defined deliverable; (b) a realistic timeline (e.g., 'within the next quarter', 'before the Western Province Partnership design is finalised'); (c) enough specificity that a programme manager could act without further interpretation.

Reference: Section 7.1, pp.24–25

[Important] No resource implications estimated Recommendation 2 explicitly calls for increased resourcing but provides no estimate of the additional cost. Recommendation 7 calls for resourcing a GESI function and conducting gender and disability analyses — these have clear cost implications. The standard requires that 'if recommendations imply human, financial or material costs, these are estimated.'

Suggested fix: Add indicative resource requirements: additional staff (FTE or short-term technical assistance days), estimated cost for gender and disability analyses, additional programme funds for accelerating WASH infrastructure. Even order-of-magnitude estimates help decision-makers prioritise.

Reference: Section 7.1, pp.24–25

[Minor] No priority ranking or sequencing of recommendations The seven recommendations are presented as a list without priority ranking. Some (Rec 1: improve engagement) are foundational and should precede others (Rec 5: sustainability pathways). The review does not indicate which recommendations should be implemented first or which are most critical.

Suggested fix: Add a brief prioritisation: which recommendations are immediate (this quarter), which are medium-term (next design phase), and which are ongoing. Alternatively, mark each as 'critical', 'important', or 'desirable'.

Reference: Section 7.1, pp.24–25

[Minor] Responsible parties not specified Recommendations are addressed to 'implementing partners and DFAT' (Rec 1) or stated passively. The standard asks that 'individuals have been allocated responsibility for responding to recommendations.' Given the complex delivery structure (APSP, RRRC/INLOC, WVI, AHC, District authorities), clarity on who leads each recommendation is important.

Suggested fix: For each recommendation, specify the lead responsible party: e.g., 'APSP Programme Manager', 'AHC First Secretary', 'RRRC Team Leader', 'WVI Country Director', 'District Administrator.' Where joint responsibility exists, name the coordinator.

Reference: Section 7.1, pp.24–25

Criterion 9: Executive summary 4/6 — Needs attention

Comprehensive executive summary covering all major themes: progress in both components, socio-cultural sensitivity, efficiency concerns, coherence, sustainability challenges, and gender. All seven recommendations are included. Good standalone document. However, it lacks an overall performance judgment and does not summarise the resource implications gap from the recommendations.

[Minor] No overall performance assessment The executive summary describes progress as 'evident, but there are risks and challenges' and notes that evidence of contribution to resilience is 'very limited.' However, it does not provide an overall assessment of whether the programme is on track, partially on track, or off track — information that DFAT decision-makers need when deciding on the Western Province Partnership design.

Suggested fix: Add a one-paragraph overall assessment: 'After 12 months, the SFRP has made [adequate/less than adequate] progress toward its objectives. WASH governance is the strongest area; FLAG outputs and sustainability are the weakest. The programme is [on/off] track for its end-of-programme outcomes given the operating context and time elapsed.'

Reference: Executive Summary, pp.iii–v

[Minor] Executive summary is long relative to report length The executive summary runs approximately 3 pages for a 24-page main report (excluding annexes). The standard recommends the executive summary be proportionate. At roughly 12% of the report, it could be tightened without losing substance.

Suggested fix: Tighten to 1.5–2 pages by removing detail that is covered in the main text and focusing on key judgments and recommendations. Consider a summary table of findings by criterion.

Reference: pp.iii–v

Priority improvement checklist

These 12 items would most improve the quality rating if addressed before submission:

Criterion 1: Primary users not identified by title or role
Criterion 2: Team roles and contributions not described in main report
Criterion 3: Methodology section is four sentences
Criterion 3: No sampling strategy despite acknowledged notification failures
Criterion 3: No ethical framework despite working with highly vulnerable communities
Criterion 3: No data analysis methods described
Criterion 4: No systematic description of M&E data used as evaluative input
Criterion 6: Questions not explicitly linked to findings in the main report
Criterion 7: No contribution analysis despite complex delivery model
Criterion 7: Evaluator's position obscured by hedging language
Criterion 8: Recommendations are directional rather than specific and actionable
Criterion 8: No resource implications estimated

About this assessment

Evaluation assessed	Independent Review of the South Fly Resilience Plan
Authors	Roselyne Kenneth & James Lloyd (QTAG / Oxford Policy Management)
Date	June 2022
Pages	40
Funder	DFAT
Location	Papua New Guinea (South Fly District, Western Province)
Programme value	AUD ~12.2M (combined FLAG and WASH components, FY20/21–21/22)
Quality framework	DFAT 9-criteria quality standard (Quality Review of Australia's 2022 Development Evaluations, Annex 3)
Assessment date	24 September 2026
Assessed by	AI (Asa) — not a substitute for expert review

What this second prototype demonstrates. The same QA tool, applied to a very different type of evaluation, produces different specific findings tailored to the evaluation's sector and context. The ethical framework gap in South Fly (critical for remote community research) did not appear in PACER Plus (a desk-based trade programme review); the multi-country sampling gap in PACER Plus does not apply here. This is not a template — it reads the evaluation and identifies what this report is missing.

See also: First prototype (PACER Plus) · Field Notes