Alignment

Mon, 07 Sep 2026 14:10:00 +0000

Part VI — Crossing Embedding Spaces

A family of maps, ordered by how much they can bend the space

The null map (T(x) = x). Do nothing. Always run this first. If it already scores well, your two spaces were nearly the same space to begin with (this happens for models of the same size, backbone, and objective) and any “translation” you fit is fitting noise. vec2vec’s own baseline is exactly this, and for their near-identical model pairs it reaches top-1 ≈ 1.0 — the real work is only on pairs where the null map collapses.
Orthogonal Procrustes. T is constrained to a rotation/reflection: min ‖X_A R − X_B‖² subject to RᵀR = I. Solution: R = UVᵀ from the SVD of X_Aᵀ X_B. Preserves all distances and angles within space A — it only re-orients. Cannot fix scale or shear. Best when the two spaces are “the same shape, different pose.”
Linear regression (least squares). Chapter 18. T is any linear map: rotation + scaling + shear + projection. More expressive; can overfit; does not preserve within-A geometry.
CCA (Canonical Correlation Analysis). Finds paired directions in A and B that are maximally correlated; aligns in that shared subspace. Naturally handles different dimensions and discards directions with no counterpart. Good when the two spaces share a subspace but also have private structure. Note it maximizes correlation, which is not the same as preserving retrieval rank — highly correlated shared directions can still reorder near-ties.
Relative representations (“re-express, don’t map”). Instead of fitting a map at all, replace every vector by its list of cosine similarities to a fixed set of shared anchor items (Moschella et al., 2023). That representation is invariant to rotation and rescaling by construction, so two spaces expressed this way become directly comparable — no T to fit. The cost: it needs a shared anchor set (texts embedded in both spaces — paired data by another name), and the invariance holds “under the same data and modeling choices.”
Learned nonlinear map (small MLP). T is a 2–3 layer network. Most expressive; needs the most anchors; highest overfitting risk; hardest to reason about. Use only if the linear family measurably plateaus below your target.

Map	What it can do	Overfitting risk	Best when
Null `T(x) = x`	nothing — run it first	none	the two spaces were already nearly one
Orthogonal Procrustes	rotation / reflection only; preserves within-A distances and angles	low (closed form)	“same shape, different pose”
Linear regression	rotation + scale + shear + projection	medium; can overfit	scale or shear differ; anchors plentiful
CCA	align a maximally-correlated shared subspace; drop private directions	medium	the spaces share a subspace but also have private structure
Relative representations	re-express every vector as cosines to shared anchors — no `T` fitted	n/a	you have a shared anchor set and want rotation/scale invariance for free
Nonlinear MLP	an arbitrary map	highest (~10⁶ params)	the linear family measurably plateaus below target (or unpaired optimization needs the flexibility)

The ordering is deliberate: try the most constrained map that meets your preservation target. A rotation that gets you to 0.85 retrieval agreement is better than an MLP that gets you to 0.87, because the rotation has fewer ways to be wrong on data you have not seen. The recent cross-encoder alignment work reinforces this: vec2vec’s nonlinear translator and mini-vec2vec’s linear one reach comparable coarse quality, and the linear one is far cheaper and more stable (Dar, 2025). Where nonlinearity earns its keep is not representational capacity but optimization robustness in the unpaired regime — when there are no anchors to regress on, the extra flexibility helps the alignment converge, not the final map express more.

Procrustes on Programmer.ie: Modern AI programming

Alignment

A family of maps, ordered by how much they can bend the space