Multilingual AI Reliability in Dari and Pashto
A comparative study of factual accuracy, translation quality, citation reliability, and response consistency in languages that receive limited attention in mainstream AI evaluation.
Reliability and safety across languages
Our first project measures factual accuracy, translation quality, citation reliability, and response consistency in Dari and Pashto, compared with matched English tasks. It also tests whether supplied evidence and structured verification reduce common errors.
Primary research question How reliably do current general-purpose large language models answer factual questions, interpret short passages, and translate text in Dari and Pashto compared with matched English tasks?
A documented, comparative evaluation
The study combines matched multilingual tasks, independent human review, repeated model runs, and source-controlled evidence. The project will report negative and inconclusive findings alongside positive results.
60 factual questions, 60 reading-comprehension items, and 60 translation items.
Dari, Pashto, and matched English material where comparison is valid.
Expanded after pilot testing, quality review, and protocol refinement.
What the initiative will produce
The work begins with model evaluation, benchmark construction, error analysis, and mitigation experiments. The initial phase does not train a foundation model or involve human research participants.
-
01
Evaluation benchmark
A documented set of Dari, Pashto, and English tasks with item-level provenance and licensing records.
-
02
Reproducible experiment runner
Versioned prompts, model settings, access dates, raw outputs, scoring decisions, and exclusions.
-
03
Error taxonomy and analysis
Human-reviewed findings on correctness, unsupported claims, citation validity, translation quality, and consistency.
-
04
Public research package
A report or preprint, benchmark documentation, scoring rubric, code, aggregate results, and limitations statement.
Eight-month research plan
Each phase creates dated evidence of progress, from the protocol and source inventory through the final analysis and public release.
Protocol and pilot
Complete the literature review, source inventory, scoring rubrics, experiment logging, and first benchmark items.
Review and experiments
Finish the 180-item pilot, conduct language review, run controlled experiments, and document amendments.
Full evaluation
Lock the 600-item benchmark, complete model runs, and preserve outputs and rating records.
Analysis and release
Run statistical and replication checks, prepare the report, and publish the permitted research materials.
Research integrity and responsible release
The initiative operates within OTED’s administrative, financial, and organizational controls. The Research Lead / PI-equivalent sets the questions, approves the protocol, supervises experiments, reviews results, and accepts responsibility for scientific integrity.
Dari and Pashto outputs are assessed under written rubrics by reviewers who do not score material they authored.
The initial phase uses public, licensed, or researcher-authored material and excludes personal data.
Model versions, prompts, settings, source records, scoring decisions, and exclusions are preserved with dated history.
OTED will distinguish measured findings from interpretation and release datasets only when redistribution is permitted.
Read the full research protocol
The program charter describes the research questions, methods, evaluation measures, safeguards, governance, and planned public outputs.
Turkish 
English 
Arabic