Article body

Introduction

What if media was nothing but an extension of a cognitive function ? The historically decisive contribution of writing would then be represented as the mere extension of working memory. Writing is a technology by which one can provide an external structuration of mental objects, each loaded in working memory, thus creating a stream of intentional mental objects that is coded outside the body. Writing, in essence, is but a partition to the human mind.

Taking Husserl’s definitions of noems (objects of intentional thought), could the learned association of phonemes, graphemes and noems of writing be generalised through other associations ? We know for example that writing is essentially a “ventrolateral’ media, in that it taps into both the ventral stream of vision, that which cognitive sciences may simplify as the path of the “what’, and the left hemisphere. Since writing is a very specialised yet, from an evolutionary point of view, relatively artificial form of association, could there be other such forms ? In particular, could one establish a “dorsal-bilateral’ form of writing, that would tap into the dorsal stream of vision, and be space-based in nature ? Could one engineer such a writing with the aim of externalising not only working memory, but some aspects of proprioception, and episodic memory ?

Therefore, could it be possible to engineer de novo, exotic forms of writings beyond the grapheme-phoneme association, but, for example, a loceme-noem association, namely a writing that would consist of the systematic association of space and thought, and be endowed with a natural grammar based on the transitive inclusion of places in natural space ? The speculation into such a form of writing is the purpose of this article. It provides the prototype of a form of hyperwriting, a zoomable writing based on the method of loci, that is, the art of memorising large collections by mentally mapping them onto space, also known as “memory palaces’ or “memory galeries’. In the same manner that writing systematised and externalised the strategy of learning by heart, hyperwriting should systematise and externalise the strategy of building memory palaces. It would be the art of writing with space, and attributing meaning to space, so as to form a working language. This language, however, would be a silent one : it could not be articulated yet, or spoken, since it would not integrate phonemes. It would remain a structured language still, and a language of the mind, as it would integrate noems. Its main shortcoming in terms of communication would simply be that it could not yet be shared orally but only visually. The purpose of this paper is to discuss the interests of such a media, exhibit a possible early form of it, and consider its continuity with Vannevar Bush’s early conception of the hypertext.

On the method of loci

The method of loci is a mnemotechnic tool consisting of mapping mental objects onto a mental space. Usually, this mental space is a familiar place, thus easing its storing by episodic and autobiographic memory. The general idea behind this intuitive “technology of memory’ is to use memories of the environment, which are much more resistant to decay and intrinsically non-verbal, to structure the stream of consciousness and in particular that of working memory. The method of loci may be considered a structured dialogue between episodic and working memory, one with a very large long-term storage capability, and the other with an access to what Baars, Dehaene and others call the “global cognitive workspace’. In a way, during the particular episodic retrieval of a memory palace, one could consider episodic memory a Read-Only Memory, and working memory the Random Access Memory of the brain.

Pesenti et al. (2001) have demonstrated that prodigy calculator Rudiger Gamm was extensively tapping into episodic memory networks to achieve his high-performance mental calculations, such as computing divisions of prime numbers down to the sixtieth decimal for example. Mathematical skills at large tap into many visuomotor networks, including cerebellar networks, as can now be easily reviewed with such integrative platforms as Mesmoudi and Burnod’s linkrbrain (https://iscpif.fr/blog/projects/linkrbrain-2) automatic literature-crawler and brain-mapper.

In general, mnemonists tend to develop interesting mental routines to memorise longer lists of elements than the average limit of working memory can allow for. One of them is very close to the creation of a language, for example in memorising large alphanumeric lists, in that it associates seemingly random series of characters an ad-hoc meaning to facilitate its memorisation. Combined with the method of loci, such ad hoc, pseudo-languages are the most efficient way of memorising large collections for professional mnemonists. It is for example the one used by such memory athletes as Joshua Foer, Dominic O’Brien or Nelson Dellis.

If any media is but the externalisation of a mental strategy, then one could externalise the method of loci into a novel form of artificial writing. The interest of such a media could be immense, in that it would transcend existing writing, and, from a historical point of view, may be just as significant as the introduction of the latter. From a neuroscientific point of view as well, the study of artificial writings could pose fascinating questions as to the “potential wells’ of media when coevolving with the cerebral cortex. We know for example that the neural correlates of reading have a very low level of polymorphism in the human population : writing tends to route itself across the left hemisphere, from the early visual areas to the temporal and then frontal lobe in most people. Mathematical skills, being more fuzzy in their definition and practical purpose (from the professional mathematician to the architect etc) tend to recruit more diversified populations of neurons, but almost always tap into the intraparietal sulcus, which is critical to exact arithmetic in humans (Piazza et al. 2004). The fascinating question of course is : would the neural correlates of de novo and ad hoc writings be always as stable as those of regular writing ? Could there be reproducible bifurcations ? Is there a general topology of the interaction between neural populations and artificial media ? Can one establish a clear landscape of it, just as one can achieve some levels of predictability in hydrology, namely the interaction between geological and hydrographic landscapes ? If we consider human mental routines rivers (and fasciculi may indeed make the metaphor not too far-fetched) forming in their easiest possible course through the “geology’ of the human brain, then indeed the interaction between artificial media and their neural correlates could be, to some extent, studies with comparable tools as those of the complex systems of geology, including catastrophe theory. The extensive research of Bach-y-Rita in sensory substitution is also seminal to the question. Here I attempt to extend it into not sensory but mental substitution, or mental re-routing, by introducing a prototype for an hyperwriting, a writing with the method of loci.

“Curvy A’ : a prototype hyperwriting

The core concept of hyperwriting is loceme-noeme association. Locemes we know are critically correlated to the entorhinal cortex and hippocampus, and in particular with the so-called “place cells’ and “grid cells’. Them being so well correlated to very precise brain structures, just as the Broca and Wernicke areas correlate with phonemes and the visual word form area with graphemes, makes them very interesting areas to target in neowriting engineering, and thus, one could say that they were not picked at random, either by memory athletes, nor for this very study, in that there seems to be a natural level of neuroergonomics to loceme-noeme association, just as there was a certain level of neuroergonomics to grapheme-phoneme association.

In homage to the undeciphered Minoan script “Linear A’, I named the first prototype hyperwriting “Curvy A’. It consists of a simple way to script landmarks and places, just as writing is a simple way to script graphemes. Dehaene has reminded that the letter A as a grapheme in itself may very well have come from the inversion of the early script of a bullhead. Considering the evolution of regular writing as the simplification of figurative scripts, why not begin with a figurative metaphor to structure locemes as well, and then simplify it ? Writing is correlated with civilization ; one of the most easily recognised symbol of locemes in the collective mind of most civilisations are rivers. Mesopotamia, after all, simply means “the land between the two rivers’, the Nile has structured the entire Egyptian civilisation, just as the Indus in the Indian subcontinent, etc. Let us then use simplified rivers as the most basic element of hyperwriting’s first loceme-loceme association (ie. a place that maps other places onto space, themselves later mapping noems). The structure of hyperwriting indeed can be simplified as

grapheme —> loceme —> … —> loceme —> noem (1)

The multiscale dimension of hyperwriting comes from that, as the built environment, it is finitely zoomable. However, all the scales of hyperwriting preceding that of noems, the final scale, are locate scales, namely, scales that represent places either containing or being contained by other places. Regular writing, in contrast, as a different descendent complexity, which, for example, may be represented as

book—> chapter —> section —> paragraph —> sentence —> word —> letter (2)

Both Hyperwriting and regular writing are finitely zoomable, their difference lies specifically in the brain areas they are targeting. A more complex grammar could still be associated with the various levels of complexity of hyperwriting, and this is a possibility that Curvy A already explores.

The fundamental symbol of Curvy A is a stylised river shape, curved, hence the name, which is essentially a Bézier curve (fig. 1). As it turns out, a more complex, asymmetric and packaged curve (that is, with more meanders) is intuitively more memorable, and allows for the mapping of more content in a single view (fig. 2). This river shape is a strand of loci (fig. 3), each being another strand for other loci (fig. 4) which are groups of noems of various shapes so as to facilitate their memorisation. The purpose of Curvy A is to capture some of the aspects of the interaction between long term and short term memory in the method of loci, and to make it writable in a procedural manner. Hence, being an externalisation of the method of loci, it is a media, albeit fundamentally different from writing. Among other things, it is a procedural writing.

Fig. 1

Fig.1 Curvy A.1, a Bézier curve is figuring the highest loceme, itself mapping other quasi-self-similar locemes. The lowest level locemes are black dots, which in three dimensions could figure a stylised rock, just like the Bézier curves are stylised rivers. The lowest levels locemes are directly mapping noems, unlike the higher-level ones.

-> See the list of figures

Fig. 2

Fig.2 Curvy A.2, first step: the curve is made more complex so as to map more locemes in the same space. As a stylised river, it has more asymmetric meanders, making it slightly more memorable.

-> See the list of figures

Fig. 3

Fig.3 Curvy A.2, second step: other quasi-self-similar locemes are added at the level below the second ones.

-> See the list of figures

Fig. 4

Fig.4 Curvy A.2, third step : half of the locemes have been populated by the lowest level locemes (the dots, stylised rocks) each clustered in groups, and each associated with a single noem. Here the entire script is mapping about 630 noems and could be continued.

-> See the list of figures

Conclusion

Reading, for the human mind, is essentially a stream of noems, mental images, which are multimodal in nature. Mental phonemes are of this kind, for example, and essentially monomodal. Phonemes in themselves, can be very good intermediates to noems, although they do not capture all of them, such as pure mental images, smells and pictures, qualia in general, which are not verbal by definition yet remain highly correlated to their linguistic counterpart. The entire interest of hyperwriting is to propose not a substitute for writing, but a higher-level, synergistic form of writing to supplement and enrich it. The practical purpose of hyperwriting is to enrich regular writing and allow for a mapping of it onto space.

It could be interesting to further the development of hyperwriting by developing ad hoc pseudo languages to specifically map locemes and phonemes, ie. to give names to the natural grammar of space that would not be logical constructs (eg. “A is in B’) but single words. Another function of hyperwriting, that of the method of loci in general, is to allow the reader and writer not only to ask themselves “what was the keyword of this thought again ?’ but also, more importantly and more mnemotechnically “where did I leave this thought again ?’. This, in essence, is an early form of both dorsal and episodic process, in that hyperwriting intrinsically suggests to develop affordances for thought, namely the art of reaching them, and of placing them in a mental space so as to give meaning to their reaching. This could further be developed into a proper grammar, but a dorsal grammar, based on affordances.

Curvy A and hyperwriting in general naturally provide for a grammar of inclusion as well. There are levels of complexity in regular writing, as we have seen, and the next step to the development of hyperwriting would be to seamlessly entangle them with the natural levels of complexity of the mental space. Here my interest has been to suggest that an artificial writing based on a loceme-noem association could recruit very precise areas of the brain, act either in synergy with existing writing or independently from it.

The epistemologic contribution of hypertext has been to create both a grapheme-grapheme and eventually, a noem-noem association. This has enriched the way one can explore, and understand large collections of words, especially by giving a lateral, intertextual dimension to them. One epistemologic contribution of hyperwriting is to facilitate the mental grasping of large collections of contents, its memorising, and at the same time the capturing of it in a writable form. Another interest of de novo writing is to pose the question : has man domesticated writing or has writing domesticated man ? The wilful domestication of both our dorsal visual stream and the neural correlates of our episodic memory could lead to the development of fascinating new media, of which “Curvy A’ is but a very early suggestion.