Between touch and sound. A synaesthetic code for the colour perception

Abstract

This article presents the design of a combined tactile and auditory code for encoding colours for blind individuals, intended for the reproduction of paintings, particularly non-figurative ones. The aim of the proposed system is to evoke a sensation related to colour in blind users, not as a forced association with the sighted concept of colour, but rather as a means to develop an abstract and personal notion of colour through tactile and auditory assistance. The system is based on studies on chromatic synaesthesia conducted by artists and musicians throughout the twentieth century and is primarily designed for the enjoyment of abstract paintings from that period.

1. Introduction and State of the Art

The issue of accessibility in cultural heritage includes the creation of specific pathways designed for individuals with different abilities. This not only refers to ensuring access for people with reduced mobility through the removal of physical entry barriers but also to the development of multisensory routes that cater to the needs of blind and visually impaired individuals, allowing them to experience museums or heritage sites through senses other than those typically employed in such contexts.

Since 1995, the Italian Union of the Blind and Visually Impaired has emphasised the necessity of recognising the right of blind individuals to engage in tourism and enjoy cultural heritage (Bellini, 2000). Since then, there has been growing attention towards making artworks accessible to the blind, leading to the creation of tactile routes for experiencing sculptures, architectural pathways featuring scaled tactile reproductions, and raised-relief reproductions of paintings. However, accessibility remains limited to a few major exhibitions and a small selection of works within them. For this reason, organisations continue to call for greater attention from museums and the art sector. While physical accessibility issues can be relatively easily addressed through tactile pathway indicators embedded in handrails or flooring - using a system that is now widely implemented in public transit spaces - developing a means of communication that enables blind individuals to perceive artworks through alternative sensory channels remains more complex. Touch is the primary alternative, but it may also be combined with hearing and even smell (Handa et al., 2010).

Very few works of art can be easily perceived without vision, the sense through which sighted individuals receive most of the information about their surroundings. Therefore, it is crucial to develop communication methods that enable the broadest possible perceptual experience (Mesquita & Carneiro, 2016). Due to its intrinsic nature, architecture already lends itself to a multisensory perception: a real, immersive visit provides an entirely different experience compared to merely observing a photograph. Integrating a real visit with a scaled tactile model of the architectural work allows blind individuals to fully grasp its characteristics, granting them a holistic understanding of the structure (Karbowski, 2020).

Sculpture is the art form best suited to tactile exploration, offering additional insights even to sighted individuals when they are permitted to touch the work. It can therefore be considered both a visual and tactile art. Tactile perception complements visual perception in that it is more precise and requires a longer engagement period. However, the goal in both cases is an aesthetic experience.

The most challenging art form to perceive multi-sensorially is painting, as it is inherently visual. Translating it into a tactile communication form requires converting the pictorial content into a bas-relief. In this way, blind individuals can construct a mental image corresponding to what is more immediately perceived through sight (Grassini, 2016). The selection of artworks to be reproduced in relief within exhibition spaces that offer these tactile pathways generally favours paintings whose details lend themselves well to being highlighted through volume enhancement. For a tactile image to be readable, it must be created with appropriately thick elements and easily distinguishable graphic forms, allowing figures to be identified by touch (Ballarin et al., 2018). Sensory accessibility can be achieved through audio guides, braille texts and programs with sign language interpretation.

All paintings adapted for tactile perception exclude the notion of colour entirely. It is impossible to reproduce the chromatic experience through touch alone, as no other sense can fully substitute vision in this regard. Individuals who have lost their sight retain memories of light and colour sensations, which can be evoked through associations with other sensory stimuli. However, those with absolute or congenital blindness cannot mentally represent a sensation they have never experienced, making colour-related terms meaningless to them (Shepard & Cooper, 1992). The tactile perception of a painting reproduced in relief within a museum pathway is therefore always incomplete, as it includes formal components while excluding chromatic ones, that are very important to offer a more holistic understanding of the work.

2. Chromatic Synaesthesia

Synaesthesia is the ability to perceive the same object simultaneously through different senses. Neurophysiological studies on synaesthetic individuals have demonstrated that, during synaesthetic experiences, the brain activates different sensory areas simultaneously, as confirmed by modern functional neuroimaging techniques (Mazzeo, 2005). In addition to true synaesthesia, which is observed in a very limited percentage of the population, it has been noted that the tendency towards multisensory perception is highly developed in individuals who cannot rely on one of their senses. Consequently, they become accustomed to perceiving reality through their remaining senses, whether for daily life activities or for experiencing a work of art. This multisensory perception is also naturally and spontaneously present in children, who explore their surroundings through multiple senses: they do not merely look at an object but also touch or taste it to experience it fully. "It thus seems that children possess a general capacity, which we may call a-modal perception, to receive information in one sensory modality and translate it in some way into another sensory modality." (Stern, 1987, p. 66).

Through multisensory experiences, it is therefore possible to attempt to induce in blind individuals a sensation akin to that received through the visual perception of colour, which is, in any case, highly subjective. It is worth noting that chromatic synaesthesia is one of the most recorded and studied forms, particularly by artists and musicians, who have identified a link between the perception of music and colour (Baroncini, 1989). In this context, we can refer to an association between sight and hearing, whereby music induces the visualisation of colours, or a painting evokes a melody.

The correspondence between colours and musical notes had already been observed by the physicist Isaac Newton, who directly linked optical and acoustic phenomena. In his seminal work Opticks, Newton noted that the geometry of the visible spectrum coincided with the metric geometry of a specific sequence of musical tones. "Where the Ring is successively made by the limits of the five principal Colours (red, yellow, green, blue, violet) in order (that is, by the extreme red, by the limit of red and yellow in the middle of the orange, by the limit of yellow and green, by the limit of green and blue, by the limit of blue and violet in the middle of the indigo, and by the extreme violet) are to one another very nearly as the fix lengths of a Chord which found the notes in a fifth Major, sol, la, mi, fa, sol, la." (Newton, 1706, p. 8). By analogy with the seven musical notes, he then expanded the spectrum of colours to seven, further observing its geometric correspondence with a particular musical sequence: "Red, orange, yellow, green, blue, indigo, violet in order, are to one another as the Cube roots of the Squares of the eight lengths of a Chord, which found the notes in an eighth, sol, la, fa, sol, la, mi, fa, sol." (Newton, 1706, p. 8). Following Newton’s theory, musical instruments capable of associating colours with notes were developed in subsequent years, including the Colour Organ, patented by Bainbridge Bishop in 1877 (fig. 1). This instrument featured small windows with different coloured glass panes, each equipped with a shutter. When a key on the organ was pressed, the corresponding shutter opened, revealing the colour of its light (Bishop, 1893).

Figure 1. Relation between the colours and the harmonic series of music (Bishop, 1893).

At the beginning of the 20th century, various experimental and theoretical investigations were conducted in Europe by both musicians, who sought to transform sound into colour using light, and painters, who aimed to incorporate the temporal dimension typical of music into their works. In this regard, Kandinsky, describing his experience of listening to Lohengrin by Richard Wagner, wrote: "I saw all my colours in my mind; they were before my eyes. Wild, almost insane lines were sketched in front of me." (Kandinsky, 1974, p. 158). Following this revelation, Kandinsky conducted extensive studies and experiments, collaborating with musicians to express musical sensations within his paintings. Several experimental associations between music and colour date back to this period: for example, the Russian composer Scriabin created a table of correspondences between musical notes and colours, which he used for the composition of Prometheus, while Kandinsky established a relationship between chromatic tones and the timbres of musical instruments.

In more recent times, various prototypes have been developed to associate sound and colour to induce sensations comparable to chromatic perception in blind individuals (Riccò, 2005). However, none of these systems has been widely adopted or implemented in museum spaces for the interpretation of paintings.

Other synaesthetic associations with colour can be established through touch. Many Bauhaus artists demonstrated great interest in colour, contributing to perceptual experimentation through different approaches. It was within this cultural milieu that the most well-known association between touch and vision concerning colour emerged -namely, the distinction between "warm colours" and "cool colours". This concept, originally theorised by Johannes Itten (Itten, 1961) but now widely accepted and part of common language, effectively acknowledges the existence of colour attributes that extend beyond mere visual perception, confirming the presence of a synaesthetic relationship between touch and sight.

It is precisely through the combination of the tactile and auditory systems that the present study explores the development of a code for chromatic perception in blind individuals.

3. Design of a code for the synesthetic perception of colour

Starting from the observation of the difficulties and incompleteness in the enjoyment of pictorial works by visually impaired individuals, this research has addressed the theme of synesthetic perception of colour, aiming to develop a multisensory communication system suitable for blind users. The analysis of existing exhibition pathways has highlighted that reproductions primarily concern figurative paintings, where the perception of object shapes - more easily rendered in relief - prevails over colour perception for understanding the artwork.

Conversely, in many abstract art pieces, colour interpretation is predominant and essential for comprehending the painting. Consider, for example, Mondrian’s work, with its geometric compositions of primary colours, the result of his research into balance and formal perfection. In such cases, while it is relatively simple to reproduce the form in relief, this alone is insufficient for interpretation unless accompanied by the chromatic component. Based on these considerations, a combined tactile and auditory system for encoding colours has been developed for use by blind individuals. This system is particularly suited for the reproduction of pictorial works - especially non-figurative ones - where comprehension is even more directly and exclusively linked to the ability to perceive colour. Moreover, the abstract art of the 20th century, to which this chromatic code is primarily addressed, is closely connected to the studies on colour perception mentioned earlier, making it particularly suitable for recreating perceptual correspondence through senses other than vision.

The objective of this system is to evoke a sensation linked to colour in visually impaired individuals, not as an imposed association with the conventional concept of colour perceived by sighted individuals - of which, as previously stated, they cannot have direct cognition - but rather as a means of inducing a new abstract and personal concept of colour through tactile and auditory stimuli. In this way, the artwork can fulfil its communicative function as an artistic artefact. Colour is represented through different tactile textures that correspond to the nature of each colour, each accompanied by an audio track.

Primary and secondary colours have been encoded using tactile textures, each associated with an audio track that reproduces the sound of the instrument most closely matching the colour’s character, based on Kandinsky’s studies (Kandinsky, 2005).

From a tactile perspective, the initial objective was to recreate the distinction between the most associated non-visual characteristics of colours - warmth and coolness. Due to difficulties encountered in an initial experimental phase, where temperature differentiation was attempted using different materials, a decision was made to attribute a rough background to textures associated with warm colours and a smooth background to those linked to cool colours. This choice was guided by the common tactile experience of associating roughness with warmth and smoothness with coolness.

Secondly, for each colour, a texture was designed (featuring raised elements easily readable by individuals familiar with the Braille system) to reflect its character, once again drawing on Kandinsky’s associations (fig. 2).

Figure 2. Association between texture and colours.

For instance, regarding primary colours, the texture for red is dense and intense, conveying its strong and assertive nature. The background is rough (as with other warm colours), with raised dots, and is associated with the deep sound of a drum. The yellow texture, also with a rough background, features grooves rather than raised dots, as it is more delicate than red - still intense and vibrant, but, as Kandinsky described, lacking in depth. It is associated with the high-pitched notes of a trumpet. The texture for blue is sinuous, soft, and undulating, evoking the movement of water, conveying a sense of smoothness and fluidity to the touch. Its background is smooth, like other cool colours, and it is paired with the deep, solemn sound of a double bass. The textures of secondary colours result from the fusion of the textures of the primary colours from which they derive, maintaining a rough background for warm colours (orange) and a smooth background for cool colours (green and violet). For example, the orange texture combines the characteristics of red and yellow, featuring an alternation of raised and recessed areas on a rough background. The associated instruments are the flute for orange, the violin in the middle register for green, and the English horn for violet.

These coded textures were first applied to a hexagonal representation of primary and secondary colours based on Itten’s colour wheel (fig. 3), deemed the most suitable system for illustrating the formation of secondary colours from primary ones (Itten, 1961). This representation can also serve an educational purpose, such as supporting blind children when learning about colour characteristics and the division between primary and secondary colours, typically taught during the 1st cycle of basic education.

Figure 3. Representation of primary and secondary colours.

Alongside the colours, the same panel features representations of neutral tones - white, black, and grey - encoded using the same logic. Black is associated with silence, as it is the colour poorest in sound; white is paired with the delicate notes of a xylophone interspersed with small pauses. The idea is to structure an exhibition pathway beginning with this panel, accompanied by Braille captions or an audio guide, allowing users to familiarise themselves with the coding system before engaging with the artworks. Ideally, if this system will be standardised like Braille, users would need to learn it only once and would then encounter the same reading system in all museums.

The proposed technology for producing these panels is thermoforming, which relies on heat deformation of a thermoplastic sheet. Simple to manufacture at a low cost, this is currently the most widely used technique for creating tactile reproductions. Strategically placed sensors on the tactile panel could activate audio through commercially available devices. The duration of each sound associated with a colour should correspond to the quantity of colour used - larger colour fields should proportionally trigger a longer audio playback, considering the additional time required for tactile exploration of that area.

As an example, artworks by Mondrian (fig. 4) and Malevich (fig. 5) have been reproduced using the described textures. Furthermore, the most appropriate approach to tactile reading has been studied, which could be guided by museum staff or an audio guide that pauses when the colour-related audio is activated. To this end, for each selected artwork, a formal analysis was first conducted to identify key characteristics to be highlighted for tactile interpretation. The most suitable exploration pathway was then determined, outlining the order of reading and sequence to be followed.

Figure 4. Piet Mondrian, Composition in Red, Blue, Yellow. Oil on canvas, 1930 (left). Formal analysis (centre) and tactile reproduction with sensor placement (right).

Figure 5. Kasimir Malevich, Blue Triangle and Black Rectangle, Oil on canvas, 1915 (left). Formal analysis (centre) and tactile reproduction with sensor placement (right).

4. Conclusions

The recognition of recurring intersensory associations between colours, sounds, or tactile sensations is of particular significance in the design of artefacts that can induce similar sensations in individuals. The interconnection can enrich the artistic experience and facilitate a deeper understanding of the works. The research conducted has led to the development of prototypes, which, at present, have been tested on too small a sample of blind individuals (only 3) to provide conclusive feedback on their validity. The initial responses have been positive, but it is anticipated that the experimentation will be expanded by involving associations for the blind and museums, in order to gain more substantial feedback for further refinement or enhancement of the code’s definition.