Resources

Notes from the annotation desk.

Engineering writeups, methodology posts, and project breakdowns from the people doing the work. New post every two weeks.

Why Cohen's kappa beats accuracy for annotation QA

Accuracy lies when classes are imbalanced. We walk through three real failure modes — and why we publish kappa with every batch instead.

The infrastructure choices that let a three-person team turn around a 100-item pilot in under a day, including the parts we got wrong twice.

Most RLHF playbooks assume frontier-scale compute. Below 7B parameters, the rules — and the labelers you need — are different.

Anonymized walkthrough: what we found, what we fixed, and what the model accuracy looked like before vs. after the audit pass.

Self-check, peer review, lead-engineer audit — and why doing them in parallel rather than sequence cut our cycle time by 38%.

The single change that dropped our calibration round from four iterations to one. Hint: it's about edge-case enumeration.