Alignment

Mon, 07 Sep 2026 14:10:00 +0000

Part VI — Crossing Embedding Spaces

A family of maps, ordered by how much they can bend the space

The null map (T(x) = x). Do nothing. Always run this first. If it already scores well, your two spaces were nearly the same space to begin with (this happens for models of the same size, backbone, and objective) and any “translation” you fit is fitting noise. vec2vec’s own baseline is exactly this, and for their near-identical model pairs it reaches top-1 ≈ 1.0 — the real work is only on pairs where the null map collapses.
Orthogonal Procrustes. T is constrained to a rotation/reflection: min ‖X_A R − X_B‖² subject to RᵀR = I. Solution: R = UVᵀ from the SVD of X_Aᵀ X_B. Preserves all distances and angles within space A — it only re-orients. Cannot fix scale or shear. Best when the two spaces are “the same shape, different pose.”
Linear regression (least squares). Chapter 18. T is any linear map: rotation + scaling + shear + projection. More expressive; can overfit; does not preserve within-A geometry.
CCA (Canonical Correlation Analysis). Finds paired directions in A and B that are maximally correlated; aligns in that shared subspace. Naturally handles different dimensions and discards directions with no counterpart. Good when the two spaces share a subspace but also have private structure. Note it maximizes correlation, which is not the same as preserving retrieval rank — highly correlated shared directions can still reorder near-ties.
Relative representations (“re-express, don’t map”). Instead of fitting a map at all, replace every vector by its list of cosine similarities to a fixed set of shared anchor items (Moschella et al., 2023). That representation is invariant to rotation and rescaling by construction, so two spaces expressed this way become directly comparable — no T to fit. The cost: it needs a shared anchor set (texts embedded in both spaces — paired data by another name), and the invariance holds “under the same data and modeling choices.”
Learned nonlinear map (small MLP). T is a 2–3 layer network. Most expressive; needs the most anchors; highest overfitting risk; hardest to reason about. Use only if the linear family measurably plateaus below your target.

Map	What it can do	Overfitting risk	Best when
Null `T(x) = x`	nothing — run it first	none	the two spaces were already nearly one
Orthogonal Procrustes	rotation / reflection only; preserves within-A distances and angles	low (closed form)	“same shape, different pose”
Linear regression	rotation + scale + shear + projection	medium; can overfit	scale or shear differ; anchors plentiful
CCA	align a maximally-correlated shared subspace; drop private directions	medium	the spaces share a subspace but also have private structure
Relative representations	re-express every vector as cosines to shared anchors — no `T` fitted	n/a	you have a shared anchor set and want rotation/scale invariance for free
Nonlinear MLP	an arbitrary map	highest (~10⁶ params)	the linear family measurably plateaus below target (or unpaired optimization needs the flexibility)

The ordering is deliberate: try the most constrained map that meets your preservation target. A rotation that gets you to 0.85 retrieval agreement is better than an MLP that gets you to 0.87, because the rotation has fewer ways to be wrong on data you have not seen. The recent cross-encoder alignment work reinforces this: vec2vec’s nonlinear translator and mini-vec2vec’s linear one reach comparable coarse quality, and the linear one is far cheaper and more stable (Dar, 2025). Where nonlinearity earns its keep is not representational capacity but optimization robustness in the unpaired regime — when there are no anchors to regress on, the extra flexibility helps the alignment converge, not the final map express more.

Retrieval Is Not Geometry

Wed, 09 Sep 2026 01:50:00 +0000

Research Coda — What Alignment Really Preserves

The result that looks like success

Suppose a bridge translates vectors from space A into space B. On a held-out set, the true B-space counterpart appears in the translated vector’s top ten 99.7% of the time.

That sounds finished.

Now ask a different question: does the translated vector have the same ten neighbors as the native B-space vector?

The answer can be barely 59%.

What Should a Translation Preserve?

Wed, 09 Sep 2026 02:00:00 +0000

Research Coda — What Alignment Really Preserves

“Preserve the geometry” is not a complete instruction

A bridge translates vectors from space A into space B.

The natural instinct is to say:

Preserve the source geometry while you translate.

That sounds obviously correct. If two source points are close, keep them close. If two source points are far apart, keep them far apart. Preserve pairwise cosine, distances, neighborhoods.