Part V — Embedding Spaces Are Not Universal

The same sentence, three universes

Embed one sentence with three models:

model A (384-d):  [ ... ]
model B (768-d):  [ ... ]
model C (768-d):  [ ... ]

A and B differ in length, so nobody expects to compare them coordinate-wise. But B and C are both 768-dimensional. Line up their vectors and compute the cosine between B’s vector and C’s vector for the same sentence:

cos(B_sentence, C_sentence) ≈ 0.02

Essentially orthogonal. The two 768-dimensional encoders placed the same sentence in unrelated positions. There is no shared origin, no shared axes, no shared scale. Dimension 12 of model B and dimension 12 of model C have nothing to do with each other.

When two models embed the same text, what — if anything — do their outputs have in common, and how do you measure it without assuming a shared coordinate system?

Why equal dimension is not shared space

Each model’s space is defined by its own training: its data, objective, architecture, initialization, and random seed. The output dimension is just how many numbers it emits. Two models at d = 768:

• have independently-chosen (and rotation-arbitrary, Chapter 5) bases;
• have different anisotropy, effective rank, and score scales (Chapter 8);
• encode different notions of “similar” (Chapter 1);
• were never trained with any constraint linking their coordinates.

The only things that could be shared are relational: which items are near which other items. And even that is only partly shared.

How to compare spaces without shared coordinates

You compare the structure, not the coordinates:

• Neighborhood overlap / rank agreement. For each item, take its top-k neighbors in space A and in space B; measure Jaccard overlap or rank correlation. “Do the two models agree on what is near what?”
• Mutual k-NN consistency. Fraction of pairs that are mutual neighbors in both spaces.
• CKA (Centered Kernel Alignment). A scalar in [0, 1] measuring how similar two representations are up to linear transformation and rotation. Widely used to compare neural representations.
• RSA (Representational Similarity Analysis). Compare the two full pairwise-distance matrices (correlate the distance matrices).
• Procrustes residual. Best orthogonal alignment error (Chapter 19) — how well can one space be rotated onto the other?
• Retrieval agreement. Run the same queries through both; measure overlap in returned sets and rank correlation of scores.

These give a graded answer: two models can have 0.8 neighborhood overlap on common topics and 0.3 on rare ones; high CKA globally and low agreement on hard negatives.

What is usually shared, and what is not

Demonstration: three models on RELATE

MEASURED on RELATE v0.1, Wave 3 row 3.1 — artifact experiments/embeddings-from-first-principles/wave3/artifacts/space-comparison.json.

model pair                          10-NN overlap   linear CKA   top-1 retrieval agree   hard-neg decision agree
minilm-l6 (384) vs mpnet (768)          0.71           0.81              0.75                    0.94
mpnet (768) vs bge-large (1024)         0.71           0.85              0.67                    0.91
bge-large (1024) vs mxbai (1024)        0.88           0.99              0.86                    0.96

MEASURED: equal dimension is not the axis of agreement — shared training regime is. BGE-large and mxbai (same width, both retrieval-tuned, different creators) are nearly the same space: linear CKA 0.99, 88% neighborhood overlap. The cross-family pairs sit at CKA 0.81–0.85 and ~70% neighborhood overlap, and disagree on a third of top-1 retrievals. Two things to hold apart: the spaces are linearly close (high CKA) but their decisions still diverge — and on RELATE v0.1 the hard-negative decision agreement is high (0.91–0.96) only because its near-restatement queries let every model rank the correct answer on top; Wave 3’s per-relation preservation (Chapter 21, row 3.7) shows the polarity distinction actually inverting under a fitted bridge. “Both are 768-dimensional” predicts nothing about coordinate compatibility.

The practical consequences: you cannot average vectors from two models, cannot use one model’s threshold with another, cannot search model A’s index with a model B query, and cannot A/B test two models by comparing raw scores — only by comparing task outcomes.

What this chapter establishes and what it does not

Establishes: two models’ spaces share no coordinate system regardless of dimension; comparison must be structural (neighborhood overlap, CKA, RSA, Procrustes, retrieval agreement); coarse topical structure is usually shared and fine/hard/rare/calibration structure usually is not.

Does not establish: that models are incomparable (structural agreement is real and measurable), or that no map between them exists (Part VI builds one). It establishes the principle: equal dimensions do not imply compatible representation.

Lab 16: measure how much two models agree

PROPOSED, not executed.

Setup. One corpus (1,000+ items). Two or three embedding models. Same items.

Task.

1. Per item, compute top-10 neighbors in each space; report mean Jaccard overlap and mean rank correlation.
2. Compute CKA between the two representation matrices.
3. Run 200 queries through each; report retrieval-set overlap@10 and score rank correlation.
4. Restrict step 3 to a hard-negative subset; recompute.
5. Take 20 items and manually inspect where the two models’ neighborhoods disagree.

Success criterion. One number for “how much do these two models agree on my corpus” (pick the metric matching your use) and a list of where they diverge most — the regions where a model swap would change results.

Companion component: the space-comparison report

space_comparison(space_A, space_B, corpus):
  neighborhood_overlap:  {k: value}
  cka:                   float
  rsa_correlation:       float
  procrustes_residual:   float          (from Ch19)
  retrieval_agreement:   {all: .., hard_negatives: ..}
  divergence_regions:    [cluster ids where overlap is lowest]
  verdict:               "structurally close | partly aligned | unrelated coordinates"

The Observatory computes this whenever two spaces are registered over the same corpus, and refuses cross-space operations (shared threshold, shared index, averaged vectors) unless a bridge exists (Chapter 20).

Failure modes

• “Both are 768-d, so they’re comparable.” Dimension is not a coordinate system.
• Averaging or concatenating vectors from two models without alignment. The result has no coherent geometry.
• Reusing a threshold across models. Score scales differ (Chapter 14).
• A/B testing by raw score. Compare task outcomes, not similarity numbers.
• Assuming high global CKA means agreement everywhere. Check the hard and rare regions.

What this chapter established

• Two models’ embedding spaces share no origin, axes, or scale — regardless of output dimension.
• This is the compatibility layer of the identity / compatibility / usability separation (Chapter 17): identity (the space_hash) is exact and per-model; whether two models are compatible for a task is measured here, never inferred from matching dimension or a version label.
• Comparison is structural: neighborhood overlap, CKA, RSA, Procrustes residual, retrieval agreement.
• Coarse topical structure is usually shared; fine, hard-negative, rare-item, and calibration structure usually are not.
• On RELATE (row 3.1) 10-NN overlap between encoders runs ~0.70 for differently-trained pairs (linear CKA 0.81–0.85) and 0.88 for two similarly-trained 1024-d models (CKA 0.99) — equal dimension predicts nothing; shared training regime predicts a lot. High CKA does not mean shared decisions: the same pairs disagree on a third of top-1 retrievals, and a fitted bridge inverts the polarity distinction (Ch 21, row 3.7).
• The space-comparison report, and the Observatory’s refusal of cross-space operations without a bridge.

Next

If a model swap changes results, then a model upgrade — v1 to v2 — is a model swap you did to yourself, over a corpus you have already stored. The next chapter makes the engineering consequences explicit: space identity, coexistence, and re-embedding.