Part I — A Vector Is Not Meaning

A space you can draw

Give six words two coordinates each, by hand:

              (x = royalty/power,  y = gender: -1 female … +1 male)
king      ( 0.9,  0.8)
queen     ( 0.9, -0.8)
man       ( 0.1,  0.9)
woman     ( 0.1, -0.9)
apple     (-0.8,  0.0)
orange    (-0.8,  0.1)

Now every semantic question has a geometric form:

• Are king and queen related? → they are close (distance 1.6, mostly along y).
• What distinguishes king from queen? → a direction: king − queen ≈ (0, 1.6), the “gender axis.”
• Is apple like king? → the angle between them is wide; they point different ways.
• Which words form a group? → {apple, orange} cluster tightly, far from the royalty cluster.

This is the whole promise of embeddings in one picture:

Once information becomes vectors, semantic questions become geometric questions.

The chapter’s job is to make that sentence precise — and then immediately qualify it.

When is a geometric answer a trustworthy answer to the semantic question?

The geometric primitives

Five quantities do almost all the work in this book.

• Coordinates. The d numbers. Individually near-meaningless (Chapter 5); collectively a position.
• Distance. How far apart two points are. Euclidean ‖x − y‖ is the default; there are others (Chapter 4).
• Direction. x − y as a vector. The claim behind analogies: the same semantic change is the same direction anywhere in the space. Sometimes true, often only locally.
• Angle. The angle between x and y, measured by cosine. Insensitive to magnitude; sensitive to orientation.
• Magnitude. ‖x‖. In many trained spaces this correlates with frequency, confidence, or specificity rather than meaning — which is why cosine, ignoring it, is often preferred.

Two derived ideas:

• Neighborhood. The set of points near a given point. “What does this vector mean?” is usually answered operationally as “what is it near?”
• Cluster. A region of higher density, loosely corresponding to a topic or category.

Where the translation leaks

The hand-built space above was constructed to make the geometry clean. Learned spaces are messier, in specific ways.

The axes are not given. We labeled x “royalty” and y “gender.” A learned model gives you d numbers with no labels, and the useful directions are usually diagonal combinations of coordinates, not the coordinates themselves (Chapter 5).

Directions are not globally constant. king − queen might equal man − woman in our toy space. In a real space, the “gender direction” near royalty words can differ from the “gender direction” near occupation words. Analogy arithmetic works on average, locally, and the famous examples are partly curated.

Distance is not calibrated. A Euclidean distance of 1.6 means nothing until you know the distribution of distances in this space. In high dimensions, most pairs are roughly equidistant (Chapter 8), so the interesting signal is a small deviation from a large baseline.

The map is lossy. Two genuinely different concepts can collapse to nearly the same point if the objective never had reason to separate them — as capital and not capital did in Chapter 1.

Demonstration: RELATE in 2D

MEASURED on RELATE v0.1 — 250 items, bge-small-en-v1.5 (384-d), PCA to 2D. The UMAP contrast is the well-documented qualitative behaviour of that method, not run here.

Embed 250 RELATE items and project to 2D with PCA:

variance kept by the top 2 principal components      15%
correlation of PCA axis 1 with sentence length       0.64   ← the dominant axis is largely a nuisance
10-NN neighbours preserved from full space to 2D      24%   ← three quarters of every neighbourhood is wrong

PCA at least reports its 15% honestly. UMAP does not: it produces six crisp, well-separated islands that look obviously correct, but check any three sentences it placed on the same island against the raw 384-dimensional vectors and you will routinely find them closer to items on a different island.

MEASURED: a 2D projection of an embedding space keeps ~15% of the variance and ~24% of the neighbourhood structure. The layout invents separations the real geometry does not have and hides adjacencies that it does. The picture is a transformation of a transformation — use it to generate hypotheses, never to verify one.

This is a recurring warning, stated once here and reused throughout:

A visualization of an embedding space is itself another lossy transformation. Verify structural claims against the space you actually query, not the picture.

What this chapter establishes and what it does not

Establishes: the correspondence between semantic and geometric questions (relatedness↔distance, contrast↔direction, orientation↔angle); the five primitives; four specific ways the correspondence leaks in learned spaces; the visualization warning.

Does not establish: that analogy arithmetic works or fails in general, that any particular projection method is best, or that clusters in a plot correspond to categories. Each is measured later, per space.

Lab 2: build one, then break one

PROPOSED, not executed.

Part A — build. Hand-assign 2D coordinates to 12 words across 3 categories. Verify that within-category distance is smaller than across-category distance, and that one consistent “direction” separates two of the categories. Write the rule you used.

Part B — break. Embed the same 12 words with a real model. Project to 2D three ways (PCA, t-SNE, UMAP). For each projection, record: which pairs are nearest, and whether that matches the full-dimensional nearest pairs.

Success criterion. Name one pair whose apparent relationship changes depending on the projection, and state which answer you would trust and why.

Companion component: the geometry probe

Add to the Observatory a function that, given a space and a set of items, reports the raw geometric facts before any plotting:

geometry_probe(space, items) -> {
  pairwise_distance:   matrix (euclidean, cosine)
  nearest_neighbors:   top-k per item, in the FULL space
  direction_samples:   (a-b) for labeled contrast pairs
  distance_distribution: histogram of all pairwise distances
}

Rule: any 2D plot shown to a user carries a note saying which of its adjacencies survive in the full space.

Failure modes

• Reading the plot as the space. The single most common embedding mistake after forgetting normalization.
• Assuming a global “meaning axis.” Directions drift across regions of the space.
• Uncalibrated distance. Reporting a raw distance without the distribution it lives in.
• Confusing magnitude with importance. In many spaces ‖x‖ tracks frequency, not salience.

What this chapter established

• Semantic questions have geometric forms: relatedness→distance, contrast→direction, orientation→angle, category→cluster.
• Five primitives (coordinates, distance, direction, angle, magnitude) and two derived notions (neighborhood, cluster).
• Four leaks: unlabeled axes, non-constant directions, uncalibrated distance, lossy collapse.
• The visualization warning, with a demonstration where 2D invents a separation and hides an adjacency.
• The geometry probe: raw geometric facts, computed in the full space, before any picture.

Next

We have been handed a space and asked to trust it. The next chapter refuses the handout and learns a small space from scratch — counts, then a tiny predictive task — so the reader sees that embedding structure is learned compression, not semantic coordinates handed down by a model.