Explore what it means for information to become geometry, when that geometry can be trusted, and how representation, similarity, retrieval, calibration, cross-space alignment, and compression combine into an embedding runtime that knows its own limits.
A vector is a list of numbers.
Somewhere between that list of numbers and a working search system, a retrieval-augmented model, or a semantic memory, we start talking about meaning. We say two documents are “close.” We say a query “matches.” We say a space “understands” a distinction.
This book is about that gap.
Not the API call that turns text into a vector — that part is easy. The gap is everything we assume once the vector exists: that distance tracks meaning, that similarity implies equivalence, that a high score implies a correct answer, that two models with the same output dimension produce comparable coordinates, that a smaller representation preserves a larger one.
Each of those assumptions is sometimes true, under conditions, and the conditions are measurable.
What does it actually mean for information to become geometry — and when can we trust that geometry?
That is the question the book works through, one mechanism at a time.
The simple picture of an embedding system looks like this:
graph LR
T[text] --> E[embedding model] --> V[vector] --> S[similarity] --> R[result]
It is a useful starting point and a poor foundation.
In practice, each arrow hides a decision, and each decision has a failure mode:
which transformation produced the vector, under which objective
what geometry that transformation actually imposes on the corpus
how many degrees of freedom the representation really uses
whether the metric matches the distinction you care about
whether a near neighbor is a correct neighbor
whether the operating threshold transfers to new data
whether a second model's vectors mean anything next to the first model's
whether a compressed or translated vector preserved what mattered
These are not the same question, and no single similarity score answers them.
The book therefore expands the picture into a process:
graph TD
D[define the relation that matters] --> EM[embed: record the space]
EM --> G[measure the geometry]
G --> ST[stress it: hard cases, perturbations, model swaps]
ST --> C[calibrate: distributions and operating points]
C --> P[preserve: what survives compression and translation]
P --> GV[govern: turn measurement into policy]
The shift is from asking “are these two things similar?” to asking “under this representation and this metric, what does their geometric relationship establish, and what does it not?”
The book runs one continuous experiment rather than disconnected notebooks.
We start with a designed set of about 1,200 short text items — call it the RELATE corpus — where the relationship between pairs is known and labeled:
equivalent paraphrase entailment
topic-related entity-related partial support
contradiction negation temporal mismatch
unrelated hard negative
Every few chapters, we learn something new about the same corpus:
Ch 4 cosine looks excellent — on the easy relations
Ch 6 local geometry is stranger than the global picture suggests
Ch 7 a 1,536-dimensional vector does not use 1,536 dimensions
Ch 10 the nearest neighbor is confidently, geometrically wrong
Ch 11 hard negatives collapse the margin that easy benchmarks show
Ch 14 the threshold that worked yesterday does not transfer
Ch 15 one scalar cannot carry what the geometry contains
Ch 16 a second model builds a different universe from the same text
Ch 18 we try to translate one universe into the other
Ch 21 the translation preserves retrieval and destroys something else
Ch 22 we compress a document and measure what the vector forgot
Ch 23 a semantic edit is an operator with a complexity, not a magic direction
Ch 24 we build a runtime that carries all of these limits as metadata
Ch 25 a bridge retrieves the paired target almost perfectly while reproducing only part of the target neighborhood
Ch 26 preserving source-space cosine can actively fight the target geometry — so a translation must name what it is trying to preserve
Chapters 25–26 use a second, frozen 5,264-sentence cross-space benchmark to sharpen the alignment results: counterpart recovery, target-neighborhood fidelity, and the geometry named by the training objective are measured separately.
That discovery arc is the structure of the book.
The book is a sequence of distinctions that are easy to collapse and expensive to collapse:
A vector is not meaning. It is a representation produced by a learned transformation under a particular objective.
Similarity is not equivalence. “Close” is a property of the representation and the metric, not of the two texts.
Proximity is not truth. A false statement can sit next to its correction.
Retrieval is not verification. Finding a passage is not confirming a claim.
Dimension is not information capacity. Nominal dimension and effective dimension are different numbers.
Equal dimensions do not imply compatible spaces. Two 768-dimensional encoders do not share a coordinate system.
A bridge is not compatibility until preservation is measured. A map between spaces can keep rankings while losing calibration, or keep clusters while losing rank order.
Counterpart recovery is not geometry preservation. A bridge can place the true target in the top ten almost every time while replacing several of the target model’s native neighbors. “Found the right item” and “recreated the target space” are different claims.
“Preserve the geometry” is incomplete until the reference is named. Source-isometry, target-neighborhood fidelity, and downstream task preservation can disagree. A loss can faithfully preserve the wrong geometry.
Identity is exact; compatibility is empirical; usability is a policy decision. These are three separate layers:
SPACE IDENTITY exact, configuration-derived → space_hash COMPATIBILITY empirical, task-dependent → measured preservation / evaluation USABILITY a policy decision → usable_for(scope, operating_point)A matching
space_hashestablishes declared identity, nothing more. Whether two spaces are compatible for a task is measured, not inferred from the hash. Whether a system may use that compatibility is a scoped policy call.
Every transformation of an embedding creates an obligation to measure what was preserved. Truncation, whitening, a cross-space bridge, document compression, a semantic edit — each returns different vectors, and none is equivalent to its input until a preservation profile says so.
Coarse geometric preservation is systematically compatible with fine semantic failure. The book’s experiments show this three independent ways: an encoder places a sentence and its negation in nearly the same spot (Ch 1, 10); a cross-space bridge preserves retrieval to within a few points while inverting the paraphrase-vs-negation distinction (Ch 21 — measured, +0.033 → −0.107); and a document compression holds its position in semantic space to three decimal places while silently reversing which company acquired which (Ch 22 — whole-document embedding drift detected 0% of every controlled corruption). A representation can look preserved and be wrong about the one thing that matters. Only a claim-level check finds it.
These distinctions converge on one idea, which the book earns rather than asserts:
Geometry is evidence about a representation, not permission to use it.
They are stated early and paid off with experiments.
By working through the chapters, you should be able to:
usable_for scopes and preservation metrics rather than an assumed universal compatibility;The objective is not a magic similarity score. It is to understand the representation layer well enough to know what a number means, where it breaks, and what a system should do with it.
The From First Principles books share a pattern. Each one refuses a comfortable shortcut:
Models a model is a learned function, not a knower of facts
Hallucination a model's output is not automatically evidence
Agents a model call is not automatically an agent
Context more information is not automatically better context
Embeddings a vector is not meaning
This book adds the representation layer underneath all of them. Retrieval systems, agent memory, RAG pipelines, deduplication, clustering, recommendation, and semantic caching all run on embeddings, and all inherit the geometry’s limits whether or not anyone measured them.
The book deliberately spends little time on vector databases. FAISS, Qdrant, and pgvector are implementations of one primitive operation. The subject here is larger and outlives them: representation, geometry, measurement, retrieval, failure, calibration, alignment, translation, compression, and infrastructure.
By the end of Embeddings From First Principles, you should not see an embedding as a black-box array that “captures meaning.”
You should see a conditional, lossy, model-specific transformation whose geometry can be measured, stress-tested, calibrated, versioned, translated, and compressed — and whose trustworthiness for a given job is a number you can compute rather than a hope you carry. And you should treat every operation that transforms that geometry — a truncation, a bridge, a compression, a semantic edit — as something that owes you a measurement before you trust its output.
The book’s capstone is a runtime that carries those limits as metadata. The final research coda then returns to the bridge and sharpens two questions the runtime must ultimately answer: did the translation merely recover the paired item or recreate the destination structure, and what geometry was the translator actually trained to preserve?
The book begins with three words and a list of numbers.
It ends with a system — and a reader — that can ask not only whether a transformed representation looks close, but which property survived, against which reference, and whether that is the property the destination actually needs.
Start from the list of numbers. Separate identifiers, features, vectors, representations, and embeddings, and establish the load-bearing claim of the book: an embedding does not contain meaning — it is a representation produced by a learned transformation under a particular objective.
Read chapter →Build a tiny embedding space by hand. Watch semantic questions turn into geometric ones — distance, direction, angle, magnitude, neighborhood — and mark exactly where that translation is lossy and conditional.
Read chapter →Learn a small embedding space from scratch — co-occurrence counts, then factorization, then a tiny predictive task — so the reader sees embedding structure as learned compression rather than semantic coordinates handed down by a model.
Read chapter →Derive cosine, dot product, and Euclidean and Manhattan distance, and show through experiment that 'similar' is not an intrinsic property of two texts — it is a joint choice of representation and metric.
Read chapter →Ask what dimension 173 means and find the answer is usually nothing on its own. Distributed representation, basis dependence, rotation invariance, superposition, and anisotropy — and the reframing that relationships between coordinates, not coordinates, carry the structure.
Read chapter →Examine local structure directly — nearest neighbors, local density, clusters, boundaries, hubs, and outliers — and find it uneven: some points are everyone's neighbor, some regions are dense and some sparse, and the 2D picture hides all of it.
Read chapter →Separate nominal dimension from the number of degrees of freedom a representation actually uses. Intrinsic dimension, rank, effective rank, entropy rank, participation ratio, and the singular-value spectrum — and what a 1,536-dimensional vector really occupies.
Read chapter →Measure the shape of how a space uses its dimensions — anisotropy, concentration of distances, density, covariance structure, dominant directions — and compare the geometry several real models impose on the identical corpus.
Read chapter →Implement retrieval from first principles — embed, score, sort — with no vector database, so the primitive operation is completely visible. Then add top-k, thresholds, and approximate nearest-neighbor search only once the fundamentals exist.
Read chapter →Build the cases where the nearest neighbor is fluent, on-topic, geometrically closest, and wrong: paraphrases that are not equivalent, negations that sit next to assertions, lexical traps, and same-topic-different-claim pairs. Similarity is not equivalence; retrieval is not verification.
Read chapter →Mine the hardest negatives automatically — topically, lexically, and entity-matched, plus contradictory, temporally wrong, and relation-swapped — and watch the ranking margin that easy benchmarks report collapse. Easy negatives measure separation; hard negatives measure the distinction you care about.
Read chapter →Retrieval is not query-to-nearest-vectors. It is a chain: representation, candidate generation, similarity, threshold, ranking, top-k, filters, context budget. Changing any link changes the memory the system ends up with. Make the chain explicit and testable.
Read chapter →Introduce Recall@k, precision, MRR, nDCG, and rank correlation — and then draw the line the metrics hide: representation quality versus application quality. A model can top a generic similarity leaderboard and be wrong for your retrieval problem.
Read chapter →A cosine of 0.81 means nothing by itself. Build positive and negative score distributions, watch them overlap, and derive operating points from false-acceptance and false-rejection rates, ambiguity bands, and deterministic thresholds — not from eyeballing a number.
Read chapter →A similarity score of 0.78 collapses the geometry to one number. Pull several independent signals from the same pair — margin, local density, participation ratio, neighborhood stability, alignment, sensitivity — and ask whether the phenomenon is one scalar or a vector.
Read chapter →Embed identical text with three models and compare. The vectors are not merely different lengths — they inhabit different coordinate systems. Even two 768-dimensional encoders do not produce mutually meaningful coordinates. Equal dimensionality does not imply compatible representation.
Read chapter →You upgrade the embedding model from v1 to v2. Ten million stored vectors are now legacy representations. Can they coexist? Must everything be re-embedded? What metadata identifies a space, and what does compatibility even mean? Introduce the space identity and its exact hash.
Read chapter →Suppose two models represent the same objects. Can we learn a map T such that T(E_A(x)) ≈ E_B(x)? Start with the simplest case — a linear projection trained on paired embeddings — and test it on points the map never saw.
Read chapter →Widen the alignment toolkit — orthogonal Procrustes, linear regression, CCA, learned nonlinear maps — and handle dimension mismatch and normalization. Then sharpen the central distinction: coordinate reconstruction is not semantic preservation, and the second is what matters.
Read chapter →Assemble alignment into an explicit artifact: a bridge with exact source and target space hashes, a method, a status, a checkpoint, calibration, preservation metrics, and a scoped usable_for list — so a transformation never silently claims universal compatibility.
Read chapter →The empirical investigation behind the bridge's preservation metrics: cosine preservation, top-1 and rank preservation, Recall@1/5/10, MRR, nDCG@10, order preservation, and cluster preservation — computed, compared, and interpreted for what each certifies and what it does not.
Read chapter →Embed a full document; embed a compressed summary or distilled cartridge of it; measure the discrepancy. How much of the original document's representational geometry survived compression — and can that discrepancy serve as a semantic checksum, a quality gate, or a summary evaluation?
Read chapter →The difference between two vectors is the simplest way to represent a transformation, and usually the wrong one. Put it on a ladder — identity, delta, rank-1, low-rank, affine, local, nonlinear, conditioned — and ask the real question: what is the simplest operator class that represents this transformation out of sample while leaving the geometry it should not touch alone?
Read chapter →The capstone: assemble every artifact the book produced into one system — an Embedding Observatory that ingests text under multiple models and answers what space produced a vector, whether two spaces are compatible, how stable a neighborhood is, what a translation preserves, and whether a corpus should be re-embedded.
Read chapter →A bridge can recover the correct counterpart almost every time while preserving barely half of the target neighborhood. Separate point recovery from structural fidelity, learn why near-perfect Recall@10 can coexist with reordered neighbors, and make geometry preservation an explicit requirement rather than an assumption.
Read chapter →A translation objective is incomplete until it names the geometry it is trying to preserve. Source-isometry, point reconstruction, target-neighborhood fidelity, and downstream task preservation can disagree. Learn why preserving source-space cosine structure can actively damage a useful map, and how to choose the invariant from the destination task.
Read chapter →