Embeddings From First Principles
Chapter 17 of 26

Versioning the Space

By AI, Embeddings

Concepts

CHAPTER 17 — VERSIONING THE SPACE

PART V — EMBEDDING SPACES ARE NOT UNIVERSAL

PURPOSE

Turn the “spaces are separate universes” principle into engineering: define the space identity and exact space_hash, work the model-upgrade decision tree, and show the silent recall penalty of a mixed-space index.

CENTRAL QUESTION

What identifies an embedding space, when are two vectors in the same space, and what has to happen when the model changes?

UNIQUE CLAIM

A matching space_hash establishes DECLARED IDENTITY (same pipeline, same config bytes) — not identical geometry, and never “safe to mix”; compatibility is empirical (measured, Ch 16/21) and usability is a scoped policy call (Ch 20). Identity / compatibility / usability are three layers, never collapsed. A model upgrade is a self-inflicted model swap; mixing v1 and v2 vectors in one index costs recall silently, by an amount that grows with the Ch16 overlap (row 3.2: <1 point for near-aligned spaces; a BOOK HYPOTHESIS for the divergent case - RELATE v0.1 has no divergent same-dim pair); every threshold/calibration/eval is bound to space_hash; and a DERIVED space (PCA, whitening, bridge output, Matryoshka prefix) is a new space with its own hash.

THE OBJECT

Spaces — identity and lifecycle. Demonstration MEASURED (Wave 3 row 3.2, PARTIAL): the only same-dim pair (bge-large/mxbai, overlap 0.88) is too aligned - naive mixed-index penalty 0.008 nDCG, calibrated merge removes it. Magnitude for a divergent v1/v2 pair is a BOOK HYPOTHESIS (penalty-vs-overlap curve, not one number).

CONCEPTS INTRODUCED

space_identity fields (weights-artifact hash + revision + tokenizer-artifact hash, not just a release tag; query and document instruction prefixes SEPARATELY; max_sequence_length + truncation_policy; dtype/quantization; post_processing); space_hash as exact canonical digest; identity vs compatibility vs usability (three layers); the identity/compatibility/usability table; a matching hash permits an op, a mismatch requires a MEASURED bridge; a derived space is a new registered space (parent hash + derivation); upgrade decision tree (coexist only if never compared; full re-embed for one geometry; cross-version search needs a bridge; dual-write migration with calibrated merge).

CONCEPTS DEVELOPED / REUSED

Cross-space bans (Ch16) operationalized; calibration bound to conditions (Ch14) now bound to space_hash; evaluation card (Ch13) per space; space-comparison report (Ch16) used to verify “backward compatible” claims; bridge forward-refs Ch20.

PREREQUISITES

Ch1–16. Hashing, index partitioning, migration patterns.

LOCAL INVARIANTS

Tag every vector with space_hash at write time; a matching hash means declared identity only — never infer compatibility from it; treat a model-version bump as a migration not a deploy; verify compatibility claims with neighborhood overlap; register every derivation (PCA/whitening/bridge/prefix) as a new space and re-derive its thresholds and evals; never put a single threshold over a mixed-space index.

FAILURE MODES

Untagged vectors; silent client-library model bumps; trusting “backward compatible” notes; reusing v1 thresholds/evals for v2; mixed-space index with one similarity threshold.

DIAGNOSTIC METHOD

  1. Compute space_hash per configuration; confirm they differ. 2. Recall@10 for all-v1, all-v2, mixed-naive, mixed-calibrated-merge. 3. Estimate re-embed cost (tokens × price, wall-clock). 4. Run the Ch16 space-comparison between v1 and v2.

RESEARCH-DERIVED IDEAS

Backward-compatible representation learning (Shen et al. “towards backward-compatible representation learning”); model-version drift in production embeddings; Matryoshka prefix-dimension compatibility (Kusupati et al.); dual-write / expand-contract migration patterns from data engineering. Named without citation metadata; the mixed-index magnitude is a BOOK HYPOTHESIS (row 3.2) - RELATE v0.1 has no divergent v1/v2 pair.

EXPERIMENT / LAB

Lab 17 (PROPOSED): space_hash per config; Recall@10 across all-v1 / all-v2 / mixed-naive / mixed-calibrated-merge; re-embed cost estimate; Ch16 comparison report v1 vs v2. Deliverable: a migration plan (re-embed now vs dual-space until coverage X%) with cost and recall cost of each.

COMPANION COMPONENT

space_registry: {space_hash: space_identity}, derived_from {space_hash: parent + derivation}, write-time tagging, cross-space ops denied without a MEASURED bridge, per-space {calibration_record, evaluation_card, index} (re-derived per derived space); migration {status, v2_coverage, merge_policy}. Observatory enforces tagging, registers every derivation as a new space, and tracks coverage.

READER OUTCOME

Reader can define a space identity, keep identity/compatibility/usability separate, detect when a “minor” change or a derivation created a new space, and produce a costed migration plan.

DEPENDENCIES

Ch1–16.

FORWARD BRIDGE

Ch18 “Can One Embedding Space Be Translated Into Another?” — Part VI’s constructive question: learn a map between universes and define what success means.

ANTI-CLAIMS / LIMITS

A matching hash does not prove identical geometry, only declared identity; a non-matching hash does not prove incompatibility, only that it must be measured; immediate re-embed is not always required (calibrated dual-space migration is valid); some model families do offer real version compatibility; the mixed-index penalty is overlap-dependent - measured <1 point for near-aligned spaces (row 3.2), a BOOK HYPOTHESIS for divergent spaces. Claim: space identity must be explicit, and mixing spaces without a measured bridge is a bug.

Explain this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Apply this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Part V — Embedding Spaces Are Not Universal

The upgrade that broke search quietly

Production runs embedding model v1. A better v2 ships. Someone updates the client library. New documents get v2 vectors; the 10 million existing vectors are still v1. Nothing errors. Queries are embedded with v2 and compared against a mix of v1 and v2 vectors.

Recall drops a few points. Nobody notices for a month, because the system still returns ten results and they still look plausible. The v1 and v2 vectors are in different universes (Chapter 16), and half the index is now in the wrong one.

What identifies an embedding space, when are two vectors in “the same” space, and what has to happen when the model changes?

The space identity

A vector is only meaningful relative to the space that produced it. That space is identified by everything in the pipeline that can change the geometry:

space_identity:
  weights_artifact_hash:   <hash of the model weights file — not just a release tag>
  model_revision:          <commit / revision id>
  tokenizer_artifact_hash: <hash of the tokenizer files>
  dimension:               <int, output>
  pooling:                 <cls | mean | last | ...>
  normalization:           <none | l2 | whitened(params_hash) | mean_centered(mean_hash)>
  instruction_prefix_query:    <the query-side instruction string, verbatim, or none>
  instruction_prefix_document: <the document-side instruction string, verbatim, or none>
  max_sequence_length:     <int>
  truncation_policy:       <head | tail | middle | none>
  precision:               <fp32 | fp16 | bf16 | int8 — dtype/quantization shifts vectors>
  post_processing:         <PCA(matrix_hash) | whitening(params_hash) | none>
  space_hash:              SHA256 of all of the above, canonicalized

Query-side and document-side instruction prefixes are listed separately because an instruction-tuned model produces a different geometry for each; max_sequence_length and truncation_policy are listed because they silently change the vector of any document longer than the window.

A matching space_hash establishes declared space identity — same pipeline, same configuration bytes — and nothing more. It is not a claim that the geometry is identical (two independently serialized configs could produce operationally equivalent vectors with different hashes), and it is never a claim that two spaces are safe to mix. Identity is exact and cheap to check. Compatibility is empirical — measured with the Chapter 16 comparison and the Chapter 21 preservation profile. Usability is a third thing again: a scoped policy decision (usable_for, Chapter 20). The three layers stay separate:

Layer Nature Artifact Established in
Space identity exact, configuration-derived space_hash this chapter
Compatibility empirical, task-dependent measured preservation Ch 16, 21
Usability a scoped policy decision usable_for(scope, op) Ch 20

A non-matching hash therefore means exactly one thing: do not assume compatibility — measure it.

A derived space is a new space. PCA truncation, whitening, a bridge’s output, a Matryoshka prefix — each produces vectors with a different geometry, so each gets its own space_identity (with post_processing naming the derivation and the parent space_hash) and its own space_hash. A truncated index is not “the same space, smaller”; it is a new space that needs its own calibration and its own preservation record.

The upgrade decision tree

When model_version changes:

    flowchart TD
    V["model_version changes: v1 -> v2"] --> Q1{"want one consistent geometry now?"}
    Q1 -->|yes| RE["re-embed everything — the default correct answer (compute cost + migration window)"]
    Q1 -->|"not yet"| CO["v1 and v2 vectors coexist ONLY if you never compare across them — partition the index by space_hash"]
    CO --> DM["migrate: dual-write new docs to both spaces, query both, merge by per-space calibrated scores (Ch14), cut over at full v2 coverage"]
    RE --> TH["re-derive every threshold, calibration, and eval — all bound to space_hash (Ch13, Ch14)"]
    DM --> TH
    V --> X["cross-version search (v1 vector vs v2 query, or the reverse) is not meaningful without a bridge (Ch20)"]

Compatibility is a measured claim, not a version-number courtesy

“v2 is backward compatible” is meaningless for embeddings unless it means “we trained v2 with a constraint that keeps it aligned to v1’s coordinate system” — which is rare and must be stated and verified. Some model families do offer this (aligned successor versions, or dimensions that are prefixes of each other). Most do not. Verify with the space-comparison report (Chapter 16): if neighborhood overlap between v1 and v2 on your corpus is 0.5, they are not compatible no matter what the release notes say.

Demonstration: the mixed-index penalty

PARTLY MEASURED — Wave 3 row 3.2 (experiments/embeddings-from-first-principles/wave3/artifacts/mixed-index-penalty-curve.json). The magnitude below is a BOOK HYPOTHESIS: RELATE v0.1 ships no genuinely divergent v1→v2 pair of one model, so the measured penalty is a lower bound, not the curve.

RELATE corpus. Index half the items in space A, half in space B, query in B — the mixed-index scenario an upgrade creates.

A / B pair                              A–B 10-NN overlap   naive mixed-index penalty (nDCG@10)
bge-large + mxbai (same dim, aligned)         0.88                    0.008
divergent v1 -> v2 (BOOK HYPOTHESIS)          low                     large (grows as overlap falls)

MEASURED (bounded): the only same-dimension pair the locally-available models offer — BGE-large and mxbai — is too aligned (overlap 0.88, linear CKA 0.99, Chapter 16) for a mixed index to hurt: the naive penalty is under one point of nDCG@10, and a per-space calibrated offset removes even that. The book’s claim is that the penalty grows as the two spaces diverge (as the Chapter 16 neighborhood overlap falls); demonstrating the magnitude needs a real v1→v2 model pair with low overlap, which RELATE v0.1 cannot supply. What is confirmed: when the two spaces are close, mixing is nearly free and a calibrated merge closes the gap — exactly the “migration bridge” this chapter describes.

What this chapter establishes and what it does not

Establishes: the space identity and the exact space_hash; that a matching hash establishes declared identity only, while compatibility is empirical (Ch 16, 21) and usability is a scoped policy call (Ch 20); that a derived space (PCA, whitening, bridge output, Matryoshka prefix) is a new space with its own hash; the upgrade decision tree; mixed-space indexes carry a silent recall penalty; all thresholds, calibrations, and evals are bound to space_hash.

Does not establish: that you must always re-embed immediately (a calibrated dual-space migration is valid), or that no model family offers real version compatibility (some do). It establishes that space identity must be explicit metadata and that mixing spaces without a bridge is a bug.

Lab 17: quantify your migration cost

PROPOSED, not executed.

Setup. Two versions of a model (or two models you might migrate between). One corpus, labeled queries.

Task.

  1. Compute space_hash for each configuration. Confirm they differ.
  2. Recall@10 for: all-v1/v1-query, all-v2/v2-query, mixed/v2-query naive, mixed/v2-query with per-space calibrated merge.
  3. Estimate re-embedding cost: tokens × price, wall-clock at your throughput.
  4. Run the Chapter 16 space-comparison report between v1 and v2.
Config Recall@10 notes
all v1 … baseline
all v2 … target
mixed, naive … the penalty
mixed, calibrated merge … migration bridge

Success criterion. A migration plan: re-embed now, or dual-space with a calibrated merge until v2 coverage hits X%, with the cost and the recall cost of each option stated.

Companion component: the space registry

space_registry:
  spaces:  { space_hash: space_identity }        # includes DERIVED spaces
  derived_from: { space_hash: parent_space_hash + derivation }   # PCA / whitening / bridge / prefix
  vectors_are_tagged_with: space_hash            # enforced at write time
  cross_space_ops: DENIED unless bridge(space_hash_a, space_hash_b) exists
                  # a matching hash permits the op; a mismatch requires a MEASURED bridge, never an assumption
  per_space: { calibration_record, evaluation_card, index }       # re-derived per derived space too
migration:
  status:      <single | dual_write | cutting_over>
  v2_coverage: <fraction of corpus embedded in v2>
  merge_policy: <per-space calibrated | none>

The Observatory tags every stored vector with its space_hash, treats every derivation (PCA, whitening, bridge output, prefix truncation) as a new registered space, refuses to compare across hashes without a measured bridge, and tracks migration coverage.

Failure modes

  • Untagged vectors. If a vector does not carry its space_hash, you cannot tell which universe it is in.
  • Silent model-version bumps. A client library update that changes the model is a space change; it needs a migration, not a deploy.
  • Trusting “backward compatible” release notes. Verify with neighborhood overlap on your corpus.
  • Reusing v1 thresholds and evals for v2. They are bound to the old space_hash.
  • Mixed-space index with a single similarity threshold. Half your comparisons are cross-universe.

What this chapter established

  • The space identity — weights hash, revision, tokenizer hash, dimension, pooling, normalization, query and document instruction prefixes (separately), max length, truncation policy, precision, post-processing — hashed to a space_hash.
  • Identity / compatibility / usability are three layers: a matching hash means “same declared pipeline,” not “same geometry” and never “safe to mix”; compatibility is measured (Ch 16, 21); usability is scoped (Ch 20).
  • A derived space (PCA, whitening, bridge output, Matryoshka prefix) is a new space with its own space_hash and its own calibration and preservation record.
  • The upgrade decision tree: coexistence needs partitioning; a consistent geometry needs a full re-embed; cross-version search needs a bridge.
  • Compatibility is a measured structural claim, verified with the space-comparison report.
  • Mixed-space indexes carry a silent recall penalty that grows as the two spaces diverge (row 3.2 — negligible for near-aligned spaces, a BOOK HYPOTHESIS for the divergent case); a per-space calibrated merge is a valid migration bridge.
  • The space registry: vectors tagged by hash, cross-space ops denied without a bridge, migration coverage tracked.

Next

Chapters 16 and 17 established that spaces are separate universes and that crossing them naively fails. Part VI asks the constructive question: can one embedding space be translated into another — and what would it mean for such a translation to succeed?