The Shape of an Embedding Space

Mon, 07 Sep 2026 10:30:00 +0000

Part II — Inside the Space

Same corpus, three shapes

Embed the same 1,173 RELATE items with five models. Do not look at task scores yet. Look at the shape (measured — Wave 1 row 1.4, experiments/embeddings-from-first-principles/wave1/artifacts/anisotropy.json):

                          MiniLM-L6   mpnet-base   mxbai-large   bge-large   bge-small
dimension                     384         768         1024          1024        384
mean random-pair cosine       0.06        0.08         0.34          0.40        0.45
top singular-value share      0.07        0.08         0.09          0.09        0.10
effective rank                 259         387          425           434         271
distance concentration*       0.08        0.09         0.09          0.11        0.10

*ratio of standard deviation to mean of pairwise distances — smaller means “everything is roughly equidistant.”

Whitening on Programmer.ie: Modern AI programming

The Shape of an Embedding Space

Same corpus, three shapes