Learning an Embedding Space

Mon, 07 Sep 2026 09:20:00 +0000

Part I — A Vector Is Not Meaning

Counting words in a tiny corpus

Six sentences:

The cat drinks milk
The dog drinks water
The kitten drinks milk
The puppy drinks water
The cat chases the mouse
The dog chases the cat

Build a co-occurrence matrix: for each word, count how often each other word appears within one step of it.

          cat  dog  kitten  puppy  milk  water  drinks  chases  mouse
cat        -    1     0      0      0     0      1       2       0
dog        1    -     0      0      0     0      1       1       0
kitten     0    0     -      0      1     0      1       0       0
puppy      0    0     0      -      0     1      1       0       0
milk       0    0     1      0      -     0      2       0       0
water      0    0     0      1      0     -      2       0       0

Each row is already a crude embedding: kitten and milk co-occur; puppy and water co-occur. The rows for cat and dog look alike because both chase and drink.

Co-Occurrence on Programmer.ie: Modern AI programming

Learning an Embedding Space

Counting words in a tiny corpus