Cross-Modal Cognitive Mapping: A Technical Overview

Page content

Cross-Modal Cognitive Mapping

A Technical Overview of System Design and Implementation

Author: Ernan Hughes
Published: April 2025

Abstract

Cross-Modal Cognitive Mapping is a new framework designed to extend traditional text-based cognition modeling into multimodal representations.
This system combines text prompts, visual generation, human selection behavior, and semantic memory retrieval to better understand and track human conceptual architectures.

This post presents a technical overview of the core architecture, database design, embedding workflows, search functionality, and resonance mapping built during the initial research phase.

The full whitepaper is available here:
📄 Download Whitepaper (PDF)

All supporting code and development history is published on GitHub:
🔗 GitHub Repository

Motivation

While current AI systems excel at modeling language, true human cognition encompasses much more —
including imagery, emotion, intuition, and subjective internal visualization.

By observing how users visualize and select representations of their thoughts,
we can build a deeper, richer map of cognitive processes beyond words.

Cross-Modal Cognitive Mapping aims to provide the infrastructure for that deeper modeling.

Architecture Overview

The core Version 1 system implements three main modules:

  1. Memory Insertion

    • Capture user text prompts.
    • Generate or obtain embeddings.
    • Insert embeddings into a PostgreSQL database with pgvector support.

  2. Semantic Memory Search

    • Query the cognitive memory store using new prompts.
    • Retrieve closest conceptual matches based on embedding distance.

  3. Resonance Graph Construction

    • Calculate pairwise memory similarities.
    • Build a graph connecting closely resonant memories.
    • Visualize conceptual clusters and divergences.

System Pipeline

[User Prompt]
     ↓
[Embedding Generation]
     ↓
[Memory Storage (PostgreSQL + pgvector)]
     ↓
[Semantic Retrieval] ← [Query Text]
     ↓
[Resonance Mapping]

What We Built (In Simple Terms)

At the heart of this project is a simple but powerful idea:

Human thought isn’t just made of words.
It’s also made of images, emotions, and mental pictures.

Today, AI systems mostly understand people by looking at their words.
But humans think deeper than just text.

So we built a system that does something new:

  • First, when a person writes something — like a journal entry, an idea, or a reflection —
    we capture it not just as text, but also as a kind of “mental fingerprint” (an embedding).

  • Then, we store these thought-fingerprints in a memory database, like a digital mind map.

  • Later, when you search or think again, the system finds the memories that feel closest, even if you use different words.

  • Finally, we built a way to map how your thoughts are connected —
    by drawing a graph that shows how close different memories and ideas are inside your mind.

In short:

  • We aren’t just storing text.
  • We aren’t just searching words.
  • We are starting to map the shape of your thinking itself.

This is the first version of a system that could one day:

  • Build visual diaries of your thoughts,
  • Track how your mind changes over time,
  • Connect like-minded people through deeper patterns,
  • Help AI understand people not just by what they say — but by what they mean.

This is the beginning of a new kind of cognitive technology:
One that sees thought, not just language.

Database Design

PostgreSQL with pgvector extension is used for scalable vector storage and similarity search.

Example table schema:

CREATE TABLE memories (
    id SERIAL PRIMARY KEY,
    user_id INTEGER,
    text_prompt TEXT,
    embedding VECTOR(1536)  -- 1536 dimensions (OpenAI ADA embedding model)
);
  • Embeddings stored as native vector types.
  • Nearest-neighbor search via <-> distance operator.

Embedding Workflow

Text prompts are embedded using:

Ollama mxbai-embed-large

Scripts are available for easy insertion of prompts and retrieval of embeddings.

Given a query prompt, the system retrieves the top-K nearest memories using cosine similarity:

SELECT id, text_prompt
FROM memories
ORDER BY embedding <-> query_embedding
LIMIT 5;

This allows exploration of a user’s conceptual history through semantic similarity, even across different phrasing or languages.

Resonance Graphs

Memories are connected into a resonance graph if their embedding similarity exceeds a threshold.

Example visualization (using NetworkX + Matplotlib):

Nodes: individual memories (text prompts).

Edges: strong semantic resonance (> 0.75 cosine similarity).

This allows visual clustering of a user’s thought space. (Sample graph outputs included in the GitHub repo.)