Applications for Cohort 1 of the KAISH Introduction Program close 15/10/2026, 23:59 Erbil time.
ApplyHome / Reading library
The papers and open curricula we teach from, with notes on why each one matters and how to read it. Use them in our programs or on your own. Everything here is free to access.
Method
We use a three-pass method, adapted from S. Keshav’s How to Read a Paper (2007).
Bring three things to every discussion: one point you’d defend, one you doubt, and one question.
Core papers · 20
4 papers
What it showsFrames five practical research problems (side effects, reward hacking, scalable supervision, safe exploration, distributional shift) with simple machine-learning examples.
How to read itRead sections 1 and 2 and the one problem that interests you most. Ask which of the five has seen real progress since 2016.
What it showsArgues how models trained with today’s methods could come to pursue goals that conflict with their developers’ intentions.
How to read itRead in full. Mark each claim as supported by evidence or argued from theory.
What it showsMaps safety research into robustness, monitoring, alignment and systemic safety.
How to read itSkim it as a map of the field, then go deep on one area.
What it showsThe second edition of the government-backed synthesis of evidence on the capabilities, risks and safeguards of general-purpose AI, written by over 100 experts.
How to read itRead the executive summary first, then the chapter on risk management.
7 papers
What it showsAn agent can learn complex behaviour from a reward model trained on human comparisons of short clips, with feedback on a small fraction of its experience.
How to read itFocus on the reward-model setup. Ask what happens if raters can be fooled.
What it showsThe InstructGPT pipeline: supervised fine-tuning, a reward model, then RLHF. Raters preferred a much smaller fine-tuned model over the original GPT-3.
How to read itRead the methods and the main results figures. Note whose preferences are being optimised.
What it showsTrains a harmless assistant from a written set of principles and AI-generated feedback in place of human harm labels.
How to read itCompare it step by step with InstructGPT. Ask who writes the constitution.
What it showsAgents can keep their skills out of distribution while pursuing the wrong goal, such as heading for the end of a level instead of the coin.
How to read itRead the introduction and the CoinRun experiment. Ask how you’d detect this before deployment.
What it showsTests whether a weak model’s labels can draw out a stronger model’s abilities, as an analogy for humans supervising more capable systems.
How to read itUnderstand the “performance gap recovered” metric. Ask where the analogy breaks.
What it showsDeliberately backdoored behaviour in language models can survive standard safety training.
How to read itSeparate what is shown (an inserted behaviour persists) from what is not (that such behaviour arises on its own).
What it showsEvaluates protocols for using a possibly untrustworthy model safely, with a weaker trusted model and limited human review.
How to read itFocus on the red-team and blue-team setup. Ask which assumptions make the problem tractable.
6 papers
What it showsArgues that networks can be understood as features connected into circuits, with evidence from vision models.
How to read itRead in full. List its three central claims and the evidence offered for each.
What it showsDecomposes small attention-only transformers into interpretable paths and introduces the residual stream view.
How to read itWork through it with a notebook open. Reread the residual stream section until you can explain it aloud.
What it showsPresents evidence that induction heads drive much of in-context learning and form during a sudden phase change in training.
How to read itRate the strength of each line of evidence separately.
What it showsReverse-engineers the circuit GPT-2 small uses to complete sentences such as “Mary and John went to the store; John gave a drink to …”
How to read itFollow the activation-patching figures. Note the faithfulness, completeness and minimality tests.
What it showsSmall networks store more features than they have dimensions when features are sparse, which explains polysemantic neurons.
How to read itRead the introduction and key results, then run the accompanying notebook.
What it showsSparse autoencoders extract features from a small transformer that are far more interpretable than its neurons.
How to read itBrowse the interactive feature pages. Ask how you would know a feature’s label is correct.
3 papers
What it showsArgues for evaluating dangerous capabilities and alignment, and for feeding the results into training, deployment and security decisions.
How to read itRead the framework section. Ask who should run evaluations, and who checks them.
What it showsProposes building blocks for regulating the most capable models: standards, registration and reporting, and compliance mechanisms.
How to read itCompare its proposals with what governments have done since. Ask which could apply in Iraq.
What it showsArgues compute is unusually governable, since it is detectable, excludable, quantifiable and concentrated, and weighs the policy options.
How to read itRead the summary and one policy section. Ask what compute governance means for countries that import all their hardware.
Open syllabi
Everything we teach is published here under a Creative Commons Attribution licence (CC BY 4.0). Use it, adapt it, and credit the Hub.
| Syllabus | Format | Available |
|---|---|---|
| KAISH Introduction Program, Cohort 1 | Web page | Now |
| Program reading list | Spreadsheet (CSV) | 16/10/2026 |
| Week 0 prep pack | Colab notebooks | 16/10/2026 |
| Reading group guides | One page per paper | From Oct-2026 |
Free, well-maintained material from established programs. We are not affiliated with these organizations.
ARENA, London Initiative for Safe AI
Exercise notebooks from PyTorch fundamentals to transformers, interpretability, reinforcement learning, evaluations and alignment science.
Self-study of the skills in Weeks 1 to 9
BlueDot Impact
A free, cohort-based online course of about 30 hours on current safety techniques and open problems.
A structured overview with a discussion group
AI Safety Atlas
An open textbook that distills the field chapter by chapter.
Conceptual background for Weeks 4 to 6 and 10
Center for AI Safety
A free textbook with lecture videos, largely non-technical.
Societal risk and governance
Andrej Karpathy
Video lectures that build neural networks from scratch up to GPT.
Prerequisites, Weeks 1 and 3
A step-by-step guide with success criteria for each stage.
The interpretability path
Open-source maintainers
A Python library for interpretability research on language models.
Weeks 7 to 9 drills
UK AI Security Institute
An open-source framework for language model evaluations.
Week 10 drill
Study paths
For self-learners. Each path has a time estimate and a clear finish line.
Path 0 · 6 to 8 weeks · about 5 h a week
Done when you pass the self-check on the program page.
Path A · 12 to 16 weeks · about 10 h a week
Done when you’ve replicated one result from a paper in this library, with a public repository and a two-page write-up.
Path B · 10 to 12 weeks · about 8 h a week
Done when you can present a behaviour in GPT-2 small and the patching evidence for the circuit behind it.
Path C · 8 to 10 weeks · about 5 h a week
Done when you’ve written a two-page brief on one AI governance question facing Iraq or the Kurdistan Region.
Know a paper or course that belongs here? Email kurdistanaisafetyhub@gmail.com with the link and one sentence on why.
Occasional emails about application rounds, reading groups and public talks. A few messages a term. We never share your address.