← Fine Tuning From First Principles

From Real Tasks to Labelling Schemas

Define annotation semantics, uncertainty and adjudication while separating stable incident attributes from policy-dependent routing.

Two careful annotators read “The account is inaccessible after a suspicious login.” One chooses access because the user cannot authenticate. The other chooses security because compromise may explain the failure. Both can produce valid JSON. A training system receiving those records will see disagreement about the desired behaviour, not a malformed file that a parser can remove.

The first question is what the annotation unit means. Are we assigning one category to the complete incident, marking every issue mentioned, or selecting an operational escalation? These are different tasks. A single-label dataset cannot quietly mix them and then blame the model for uncertainty.

Write the Task Contract Before Training already defines our output vocabulary and abstention relationships. We use that authority rather than constructing a fresh taxonomy for the data loader. The present work makes the route from an incident to a reference answer inspectable. Its labels are synthetic authored constructions; nobody’s participation in an independent human annotation exercise is implied.

A label has a meaning beyond its spelling

The unit is one incident under one identified policy revision. Category describes the problem; severity describes its impact and urgency under the declared rubric. Route describes the action required by the applicable policy, while the citation identifies that authority. A record identifier merely locates the example. It must not become a hidden classifier.

The contract uses mutually exclusive category outputs. If an application wanted multiple simultaneous categories, it would need a new output schema, annotation instructions and evaluation definition. A hierarchy such as security/credentials could preserve both a broad category and a subtype, but that is also a different ontology. Adding increasingly specific strings to the current enum would not be a harmless formatting change.

The following boundary guide expands the existing rubric without claiming a universal support taxonomy:

Label Positive evidence Counterexample or boundary
access Credential or permission failure A credible active compromise requires security review rather than choosing the first credential word
service Availability or degradation A historical outage that has ended does not establish a current outage
security Credible suspected compromise An explicitly denied or resolved historical compromise is not automatically active
billing Charge or invoice issue Mentioning an invoice in an unrelated access incident does not settle its category
unknown Insufficient, contradictory or unresolved evidence It is not a convenient fifth ordinary department
high Ongoing outage or credible compromise The word “urgent” alone supplies neither fact
medium Multiple-user degradation Several mentions of one affected user do not establish multiple users
low Single-user non-security issue A single affected account can still involve a credible compromise

The distinction between text and trusted intake metadata matters. The current contract supplies information_status and reported_impact; the validator does not infer those values from text. An annotation may expose disagreement with the metadata, but cannot silently replace it. If real intake cannot supply that information reliably, revise the task definition before using these fixtures as a production blueprint.

Refuse confident detail when evidence is absent

An insufficient report should not receive a confident category just because the database requires a string. Under our contract, abstention has a coordinated representation: unknown category and severity, human_review, and abstain=true. The citation still names the escalation policy. Supplying unknown category alongside a confident automatic route is inconsistent.

This is distinct from malformed input. A valid but incomplete report can require human review. A record missing the required policy_version fails the input interface. Converting every malformed record into a valid-looking abstention would hide an upstream integration failure and change the evaluation denominator.

Nor is abstention the same as low annotator confidence. Someone can confidently identify insufficient evidence. Someone else can be uncertain about a case that the rubric eventually resolves. Confidence describes the annotator’s judgement; correctness depends on evidence and the agreed rule. A numeric confidence field cannot repair a missing rationale.

For overlapping issues, the instructions must say whether a priority rule resolves them or adjudication is required. Our conservative authored cases escalate unresolved multi-intent and contradictory reports. This sacrifices automation coverage. The cost is visible and preferable to teaching arbitrary confident answers under an undocumented tie-break.

Separate incident facts from the policy lookup

Write the output as $(c,s,a,r,q)$: category, severity, abstention, route and citation. For sufficient evidence $x$, the intended factorisation is

$$ (c,s,a)=f(x),\qquad (r,q)=g(c,a,P), $$
where $P$ is the identified routing policy. This expresses the task design, not a claim that an existing model has learned either function. It explains why a routing label can become obsolete without the incident itself changing.

Our paired example keeps the exact incident text and intake attributes. Policy p1 routes access incidents to access_team. The explicitly derived synthetic p2 routes them to service_ops. Category remains access, severity remains low, and the citation changes from triage-policy:p1 to triage-policy:p2.

Same incident Category Severity Route Citation
Under p1 access low access_team triage-policy:p1
Under p2 access low service_ops triage-policy:p2

FineTuneLab creates the p2 contract using the existing revision interface: a new task version, parent contract digest and exact policy-content digest. The canonical p1 draft remains unchanged. Inventing a different route string inside an annotation would fail the existing semantic validator. A matching citation string with a different policy table would also fail identity validation.

This pair is particularly useful against a misleading dataset shortcut. If the text alone determines every route in training and evaluation, a model can appear to understand policy while merely reproducing an old mapping. A policy-shift test must supply the active policy and evaluate its consequences. It cannot ask a model to infer which hidden policy the evaluator intended.

Record how an answer came to exist

An annotation binds the record digest, task-contract digest, annotation-schema revision, policy digest and revision, label values, supporting facts, missing-information state, abstention rationale, origin, disagreements and revision history. Each dimension answers a different question. A stable incident semantic ID can survive a policy change, while the annotation identity must change because the correct route and citation have changed.

Evidence should identify what supports the answer, not merely repeat the answer in another field. Our tiny fixtures attach explicit authored facts. Real annotation might cite character spans, intake evidence or a documented adjudication decision. A label supplied by a model should record its model/source identity and remain a proposal until the required review occurs. Asking the same generator twice does not create two independent judges.

The current implementation distinguishes synthetic rules, authored fixtures and AI proposals. AI_PROPOSED is not admissible merely because its JSON passes rule checks: it needs ADJUDICATED provenance and externally verified review evidence bound to the annotation, record, task contract and policy. The delivered dataset uses SYNTHETIC_RULE labels; no genuine human adjudication is claimed. It does not accept a fabricated HUMAN_GOLD origin. RULE_CHECKED_NOT_HUMAN describes what happened. A schema and route check establish consistency; they cannot prove that an arbitrary natural-language report was interpreted correctly. That remaining boundary is central to dataset QA.

Disagreement is information to preserve

Agreement rates are often tempting because they look like objective measures of quality. For two raters, observed agreement is

$$ A_o=\frac{1}{n}\sum_{i=1}^n\mathbf1[y_i^{(1)}=y_i^{(2)}]. $$
That statistic does not correct for label prevalence or establish that both raters are right. An all-access annotation rule can agree perfectly with itself. Chance-corrected measures add assumptions about the annotation setting; [Artstein and Poesio's survey](https://aclanthology.org/J08-4004/) explains why the choice of agreement measure needs care. We report no human agreement statistic because no such study was conducted here.

Our deterministic review workflow exercises agreement, insufficient evidence, policy conflict, disagreement requiring adjudication, and an explicitly authored adjudication decision used as a mechanism fixture. That fixture is not an authenticated human decision. The decision retains who or what supplied it and its rationale; it does not erase the proposals. Even an adjudicated fixture remains a constructed reference, not independently established operational truth.

In an actual annotation project, independent proposals should precede discussion. Otherwise the second rater can be anchored by the first answer. Sample difficult boundaries as well as frequent easy cases. Track systematic disagreements by category and guideline revision. A revision that resolves an ambiguity can invalidate old labels, so it needs a migration decision and renewed downstream checks.

Annotation lifecycle: disagreement is preserved

The diagram shows the review states an independent proposal can reach and the two promotions the gate refuses. Notice that even the adjudicated outcome remains a constructed reference, with its proposals and rationale retained rather than replaced by a gold stamp.

Annotation economics follows from this structure. If an extra review costs $c_r$ per example, reviewing every example costs $nc_r$. Risk-based sampling can reduce that expense but introduces assumptions about which failures the sample can reveal. Reviewing only high-confidence generated examples is likely to conceal precisely the boundaries needing attention. The pipeline records review status so a later study can state which subset received which treatment.

Check the boundary the validator actually understands

The executable tests reject a rehashed wrong route, a p2 annotation paired with p1 authority, a label bound to a different record and a blocking disagreement. They accept the valid policy pair and the required abstention case. The three downstream training representations all retain the same annotation digest.

These tests establish software behaviour on authored examples. They do not make the free-text labels independent of their generator. Later QA therefore adds separately written expectations and counterexamples, including negation, historical statements and malicious instructions. When uncertainty survives those checks, keeping the example quarantined is a valid outcome.

Once we have an eligible source and an explicit answer, another choice remains: what does the model actually see? A schema containing the correct label is not yet a training example. The next chapter follows that label through completion, instruction and conversation formats, showing how a seemingly cosmetic conversion can change the problem being learned.

Further Reading

  1. Ron Artstein and Massimo Poesio, “Inter-Coder Agreement for Computational Linguistics” (2008). Read the source. Details agreement coefficients and their assumptions, explaining why agreement alone cannot establish correctness.
  2. Rion Snow et al., “Cheap and Fast — But is it Good?” (2008). Read the source. Evaluates non-expert annotation quality across NLP tasks, informing the chapter’s review economics and sampling cautions.
  3. Curtis G. Northcutt, Lu Jiang and Isaac L. Chuang, “Confident Learning” (2021). Read the source. Estimates label errors from noisy labels, offering a QA check that is distinct from inter-annotator agreement.