# Communication Was Never Just Words
# Book: Language
# Chapter URL: https://programmer.ie/books/language/03-chapter/
# Prose fingerprint (sha256, normalized): 0ceb74abe50c1e10bd2c7b8c0144ec7534ae773ffb059fdb9a45462e1d7e42ec
# Snapshot generated at build time by Hugo. This is the complete rendered prose
# of the chapter; it is not a transcript of the source file.

A junior engineer is stuck on a distributed-systems bug: two services deadlock during a batch process that runs on Tuesdays. A senior colleague begins with a paragraph about lock ordering. The junior nods, but cannot say which service is waiting for which lock. The explanation has reached them; the relationship they need has not become clear.
The colleague draws two timelines on a whiteboard. Service A holds lock one and waits for lock two; service B holds lock two and waits for lock one. The circular wait becomes visible. Then the colleague uses two coffee cups as the locks and acts out the sequence. Now the junior can explain why neither service can move.
Did the change of form help? It may have, but this illustration cannot isolate its effect. The diagram followed an earlier explanation; the demonstration followed two. Each gave the senior another chance to respond to confusion. What the example does expose is a design question: which relationships did each representation make available, and what could the receiver reconstruct and use?
Prose can express lock order exactly. A timeline can make simultaneous waiting easier to inspect. The facts need not change for the work of reconstruction to change. This chapter asks when a different or additional form helps a particular receiver perform a particular task. Pictures always beating words, or each person having one true medium, would be convenient answers. Neither is adequate.
Three terms help us ask the question precisely. A channel is a route through which communication reaches someone. A modality concerns how it is perceived, such as hearing or sight. A representation is the organisation of what is conveyed, such as a timeline, map or paragraph. A written paragraph and a diagram may both be visual, while speech can combine words, prosody and gesture.
This chapter sometimes uses channel in the ordinary broad sense; the engineering decision ahead usually concerns representation. Choosing a form requires us to ask what it must preserve, what benefit it should produce, and how we would detect failure. A receiver’s accessibility requirements constrain that choice. Sorting people into fixed visual, auditory or kinaesthetic types is a separate claim, requiring separate evidence.
The oldest interface is multimodal A warning can sound urgent because of its pitch as well as its wording. Long before writing, communication combined spoken words with prosody, pace, gesture, gaze, posture and shared surroundings. Repeated rhythms and formulas could help a story remain recognisable through retelling. The words were already part of a richer arrangement of cues and practices.
Walter Ong’s account of oral traditions examines the demands of communication without a fixed written record. Preservation did not depend on one listener’s memory alone. Communities also kept knowledge in shared practices, performance, material arrangements and repeated social exchange. The distinction concerns the means available for preserving and correcting knowledge; it says nothing about an absence of sophistication.
A written document can travel without the speaker’s voice, gaze or immediate surroundings. It offers revision, spatial arrangement, punctuation, headings, lists, tables, diagrams and marginal notes. Writing did not invent word order, and neither writing nor print guaranteed reliable meaning. They changed what could be held still, compared and revisited. Print widened the reach of typography, pagination, indexes and standardised layouts.
A timetable, circuit schematic, subway map or knitting pattern shows how arrangement can itself carry information. Prose can describe a route through the London Underground; a map can expose alternative paths and connections for comparison at once. Communication draws on verbal, pictorial, spatial, gestural and typographic resources because they offer different ways to recover what matters.
Sometimes two forms describe the same fact. Sometimes they contribute complementary information. Replacing prose with a diagram and placing a diagram beside prose are therefore different interventions. Each has attention and preservation costs. To judge either, we need to identify what the receiver can recover from it, and what the change may obscure.
Why form changes reconstruction Dual coding. A picture of a lock may invite both a visual representation and the verbal label lock. Allan Paivio’s theory (1971, 1986) proposes interacting verbal and nonverbal representational systems. This gives us a possible explanation for why a labelled diagram might help. It does not guarantee two memory traces for every image.
Classic experiments found a picture advantage under certain recognition conditions. Shepard (1967) compared recognition for words, sentences and pictures. Standing, Conezio and Haber (1970) studied recognition after exposure to 2,500 visual stimuli. These findings concern memory for studied items. They do not establish a general advantage in explanation, causal reasoning or transfer. Retrieval demands and distinctiveness, discussed below, further narrow what we can infer.
Cognitive load and multimedia learning. A diagram beside on-screen prose gives the reader two forms to inspect, both demanding visual attention. Richard Mayer’s cognitive theory of multimedia learning distinguishes partly separate processing channels, limited capacity and active learning: learners select, organise and integrate material. The sensory distinction between seeing and hearing differs from the distinction between pictorial and verbal content.
The theory motivates several design principles. Coherence excludes extraneous material; signalling makes organisation visible. Spatial contiguity places related words and pictures together, while temporal contiguity coordinates narration with animation. These principles address different processing problems. Their effects also depend on the learner: Richter, Scheiter and Eitel’s 2016 meta-analysis of text–picture signalling found that benefits depended in part on prior knowledge.
Cromley and Chen’s 2025 meta-analysis of Mayer’s research programme covered 181 studies and 591 effects. It reported a pooled Hedges’ g of 0.37, with substantial moderation by principle, medium, age, domain and outcome. Some active-learning interventions had stronger effects than presentation changes alone. A pooled benefit therefore gives us reason to investigate particular designs, not to treat multimedia as a universal upgrade.
Adding material can work against these principles. Coherence may favour removing seductive details. Redundancy warns that on-screen text and identical simultaneous narration may compete rather than reinforce. A diagram may expose a causal relationship, while an unnecessary animation divides attention or makes the reader wait for information a static figure shows immediately. These are design hypotheses to test against the task and observable performance.
Signalling and contiguity as attention instruments. Two principles become practical controls in the system built later. Headings, arrows, highlighting or spoken stress can signal where to attend. Contiguity can reduce the work of connecting a claim with the relevant part of a figure or source. Both concern how easily the reader can find and connect the material they need.
A Semantic Browser might place a summary assertion beside the exact passage from which it was derived, highlighting that passage on request. The arrangement can make checking easier. Placement alone establishes neither the assertion’s truth nor the passage’s support for it; it cannot show that the reader checked either. Source support and successful reconstruction still require separate tests.
Conceptual comparison. Multimodality is not a quantity of channels; it is a question about which evidence the task needs. Open full-size diagram. Where pictures fail The same lock diagram that clarifies a circular wait could conceal it with reversed arrows or indistinguishable boxes. To choose a representation responsibly, we need to know how it might fail. Research on retrieval and distinctiveness supplies two boundaries. A third comes from the distinctions the task itself requires. Each becomes a design constraint later.
Retrieval demands change the advantage. Weldon and Roediger (1987) tested memory for pictures and words in different ways. Pictures did better in free recall. Studied words produced more priming on word-fragment completion, and the balance changed again with degraded-picture identification. Performance depended partly on the match between operations during study and those required at test.
Test format alone need not explain every memory difference. The practical lesson is to name the outcome before judging a representation’s success. Immediate recall, causal explanation, delayed retention, task completion and transfer can legitimately disagree. An experiment that measures one cannot award the representation an unqualified victory across the others.
Distinctiveness can matter more than pictureness. Ensor, Surprenant and Neath (2019) varied words’ font, size, colour and capitalisation across experiments. In one experiment they also reduced pictorial distinctiveness, using black-and-white images with similar orientations. Under their recognition-test conditions, making words more distinctive could eliminate the familiar picture advantage. This challenges an explanation that credits the advantage to image format alone.
The finding gives us a candidate mechanism to isolate, rather than a command to make everything look different. It does not establish that unusual typography generally improves learning, or that uniformly styled diagrams generally fail. The distinction matters for a generated interface: changing appearance is only useful if it helps with the task under examination.
Labels can conceal the distinction the task requires. Suppose a reader must distinguish two similar components both labelled gateway, or tell whether service A waits for service B rather than the reverse. Identical boxes may reveal the category while concealing the decisive relation. An unfamiliar legend can create a similar problem. A tidy diagram can still leave the reader unable to answer.
This is a design risk, not a separately established picture-superiority result from the cited studies. For the Semantic Browser, it creates a checking obligation. Entity identity, direction, order, uncertainty and exceptions must be correct against the source. A familiar icon or tidy layout cannot supply that evidence, however easy it makes the view to inspect.
Together these boundaries turn “pictures help” into the working hypothesis the book needs:
An alternative or additional representation helps when it preserves the required claims and relationships, makes a task-relevant distinction less costly to recover, and can be understood by this receiver under these conditions.
The rule is a proposal; no experiment in this book has established it. It asks three separate questions: was the transformation faithful to its source, could the receiver use it, and did it improve on a simpler alternative? A form can pass any one while failing the others. The next chapters will need all three answers before promoting a generated representation.
Preference is not performance A reader may prefer a polished illustration, animation or interactive view while performing no better on the intended task. The reverse can also occur. Ease of reading, confidence in understanding, immediate recall and successful transfer are different outcomes. This chapter does not establish how often they diverge. A preference rating alone cannot tell us whether a representation helped.
Evaluate preference and task performance separately, including time and errors, and preserve disagreement between the measures. A view readers love but cannot use needs revision. A demanding view that improves transfer may be worth retaining, while still needing work on accessibility and effort. Averaging the measures into one impression would hide precisely the difference a designer needs to understand.
Nor does a preferred form establish a fixed learning style. Clinton-Lisell and Litzinger’s 2024 meta-analysis examined 21 eligible studies and 101 effect sizes. It found a small pooled matched-instruction advantage (g = 0.31). Only 26% of learning-outcome measures, however, showed the crossover pattern needed to support the stronger claim: that different self-identified groups learn best from their own matched form.
The studies were generally of low quality, and the authors judged the benefits too inconsistent to justify widespread matching. This limits the case for turning fixed style labels into instruction policy; it does not show that representation choice never matters. A system should attend to the task, material, goals and observed needs rather than assign supposed person types.
Accessibility requirements remain constraints to meet, rather than optional gains to trade against cost. Where feasible, readers should be able to choose a form, request an alternative and override an inferred preference. One successful interaction gives the system information about that encounter. It should not become a permanent personal classification.
An experiment the Semantic Browser can run To test whether the diagram helped, we would need to separate format from repeated explanation. The opening illustration cannot do that because its explanations were sequential. A proposed comparison would begin with one frozen source describing the deadlock: which service holds and waits for each lock, why the Tuesday batch exposes the problem, and the correction being considered.
Declare those facts and relations as preservation obligations before generating any view. Prepare three source-linked conditions: the original prose alone, a diagram with the same facts and explicit lock-direction arrows, and prose plus diagram together. A decorative diagram offering no additional useful structure could serve as a negative control, provided it too is checked for faithfulness.
An independent reviewer would check every generated view for omitted obligations, reversed arrows and unsupported additions before readers see it. Passing that review makes a view eligible for comparison. Whether it helps remains to be tested with equivalent, counterbalanced tasks, so that exposure order and repeated explanation do not decide the result.
Predetermine a primary outcome, such as correctly diagnosing which service must release or reorder a lock. Record error type, response time, confidence, preference and later transfer to a different two-resource deadlock as secondary outcomes. Record prior domain knowledge and accessibility requirements without assigning learning-style types. Compare each arm against the unchanged source, report uncertainty, and allow no gain or harm as outcomes.
This protocol has not been run here. It specifies the evidence that “the diagram helped” would require, keeping source fidelity separate from receiver performance. A faithful diagram might still confuse the reader or offer no benefit over prose. The comparison must be able to reveal either possibility before the system treats the extra representation as useful.
What a second channel costs A diagram must be drawn or generated, then checked for reversed arrows, omitted exceptions and invented causal links. Cheap generation does not make faithful generation free. The receiver also pays in attention and time: decoding a legend, switching between prose and diagram, or waiting through an animation. Making the relationship visible may help enough to justify that work, but the work belongs in the comparison.
Dynamic rendering consumes computation and may add latency. Screen space and reading time spent on one form are unavailable for another; sometimes the unchanged source is enough. These operational and opportunity costs sit alongside production, checking and attention. A view can help with a task while still demanding more resources than its benefit warrants.
Some obligations cannot be traded against those costs. If a transformation drops a required safety condition or makes the material inaccessible, speed or popularity cannot compensate. It fails an eligibility test before competing on usefulness. The decision rule for the rest of the book therefore has two stages:
First, refuse representations that violate the declared preservation obligations or the receiver’s access requirements. Then, among admissible choices, compare task benefit with generation, checking, attention and operating cost, keeping the unchanged source as a baseline.
This is a design rule to evaluate, not a proven universal utility function. Gain retains its Chapter 2 meaning: the underlying information becomes easier to recover, compare or act on. Making a relation visible, improving an immediate answer, being remembered a day later and transferring to a new case earn four different claims. Neither a similarity score nor a preference rating can combine them into one.
When the evidence cannot distinguish alternatives, the system may retain the existing representation or ask what the receiver needs. A fixed page offers a meaningful baseline: an author has already chosen, arranged and reviewed it, and the resulting artifact is stable enough to inspect. The reader need not pay for a fresh rendering decision on every encounter.
That page is neither costless nor infallible. Its prior review still matters when we judge a generated alternative. A dynamic representation should earn its place against the fixed page for the declared task and with the same underlying source. Looking newer or more elaborate supplies no evidence that the reader can recover more from it.
Documented trade-offs, stated as a balance rather than a verdict. The chapter does not license matching readers to representational types. Open full-size diagram. Where this leaves us The junior engineer needed to recover a relationship: two services each waiting for what the other held. Words, timelines and acted demonstrations made different evidence available for that reconstruction. Dual coding and multimedia-learning research suggest why some combinations may help. Retrieval demands, distinctiveness and task design explain why we must judge the benefit under particular conditions.
For a source-linked system, the resulting engineering test is clear: preserve what matters, compare what the receiver can do, measure the added cost, and leave the unchanged source available. Preference cannot stand in for performance, nor representation choice for a diagnosis of personality. The proposed deadlock comparison would test the approach; this chapter reports no completed experiment.
So far, an author, teacher or colleague has chosen the forms in advance. Choosing or revising a representation at use time, for a stated task, brings provenance, preservation, receiver choice and the option to keep the original into the interface. A new rendering remains a view of its source, not a new source of truth.
Later chapters will ask when the system may use personal context to make that choice. First we need to examine the change in timing itself: what becomes possible, and what must remain accountable, when the supposedly fixed page becomes dynamic?
References Paivio, A. (1971). Imagery and Verbal Processes. New York: Holt, Rinehart and Winston. Paivio, A. (1986). Mental Representations: A Dual Coding Approach. Oxford: Oxford University Press. Shepard, R.N. (1967). Recognition memory for words, sentences, and pictures. Journal of Verbal Learning and Verbal Behavior, 6(1), 156–163. Standing, L., Conezio, J. & Haber, R.N. (1970). Perception and memory for pictures: Single-trial learning of 2500 visual stimuli. Psychonomic Science, 19(2), 73–74. Mayer, R.E. (2021). Multimedia Learning (3rd ed.). Cambridge: Cambridge University Press. Cited for the theoretical assumptions and design principles. Richter, J., Scheiter, K. & Eitel, A. (2016). Signaling text–picture relations in multimedia learning: A comprehensive meta-analysis. Educational Research Review, 17, 19–36. Cromley, J.G. & Chen, R. (2025). A meta-analysis of Richard Mayer’s multimedia learning research: Searching for boundary conditions of design principles across multiple media types. Educational Research Review, 49, 100730. DOI 10.1016/j.edurev.2025.100730. Weldon, M.S. & Roediger, H.L. (1987). Altering retrieval demands reverses the picture superiority effect. Memory & Cognition, 15(4), 269–280. Ensor, T.M., Surprenant, A.M. & Neath, I. (2019). Increasing word distinctiveness eliminates the picture superiority effect in recognition: Evidence for the physical-distinctiveness account. Memory & Cognition, 47, 182–193. Clinton-Lisell, V. & Litzinger, C. (2024). Is it really a neuromyth? A meta-analysis of the learning styles matching hypothesis. Frontiers in Psychology, 15, 1428732. DOI 10.3389/fpsyg.2024.1428732. Pashler, H., McDaniel, M., Rohrer, D. & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105–119. Ong, W.J. (1982). Orality and Literacy: The Technologizing of the Word. London: Methuen.