This is an open access article distributed under the terms of the Creative Commons CC-BY-NC 4.0 License, permitting all non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.
In her 1994 article “The New Basics: Learning to Read in a Multi-Media World,” Margaret Mackey states that “to talk about children’s literature, in the normal restricted sense of children's novels, poems and picture-books, is to ignore the multi-media expertise of our children” (17). Almost 30 years have passed since Mackey’s statement, but the relationship between children’s literature and digital media remains an important topic of discussion. Today’s children grow up in a media environment of various digital cultural products, many of them a result of the development of mobile technology. This development also influences readers’ engagement with literature (Stougaard Pedersen et al.).
Children’s literature has put the possibilities of the mobile platform to the test. After the launch of the first iPad in 2010, various children’s book products – such as interactive picturebook applications, book-based mobile games, and picturebooks that combine a printed book with augmented reality (AR) – have appeared on the market. An AR picturebook usually consists of a printed picturebook and a marker-based AR app that enriches the illustrations of the printed book with computer-generated and interactive content. The reading of an AR picturebook requires a mobile device.
This study focuses on reader engagement with the Finnish picturebook
Many interactive features of book apps emerge from the world of digital and mobile games for children (Järvenpää). Lisa Nagel notes that previous studies (Al-Yaqout and Nikolajeva; Schwebs; Stichnothe; Turrión) conceptualise interactivity through its aesthetic value and, most importantly, the degree of influence the reader has over the narrative. This point of view values co-creation over reading. As Nagel argues, “such an approach to interactivity is problematic in the sense that it leaves the reader and the reading experience out of the equation” (2). At the same time, such approaches disregard the rich print tradition of interactive children’s literature, such as pop-up and lift-the-flap books. In this tradition, interactive features do not need to serve other purposes than engaging readers and playing with the book medium.
Following Nagel’s reasoning, this study is interested in child and adult readers’ playful engagement with digitally enhanced literature. More precisely, I take a look at three families with children between the ages of 4 and 6 and examine their shared reading experience of
Instead of studying, for instance, reading comprehension or literacy, I aspire to observe how readers engage with the book and the app. The
Interactive picturebook apps differ from standard e-books by their multimodal nature. For instance, apps can add audio-visual effects to storytelling or invite users to physically interact with the app (Søyland and Gulliksen). Typically, picturebook apps contain “hotspots” that can trigger, for example, animation, sound effects, or music. In addition, the picturebook app provides an attractive platform for aesthetic experimentation. Ayoe Quist Henkel, who emphasises the materiality of book apps, describes their essence as follows: “As digital artefacts, literary apps for children have their own ways of being texts […]” (339). Many previous studies have focused on two main dimensions of book apps: the features and aesthetics of book apps as well as children’s digital literacy skills and literary education (e.g., Manresa and Real).
Although there are many different children’s book apps available, some literary works experiment with the possibilities of the digital medium more than others. Works of this kind seem to form part of the “border area” where, according to Ghada Al-Yaqout and Maria Nikolajeva, “interesting experimental, hybrid texts emerge” (3). The multimodal narratives and aesthetics of these texts have been the subject of Nordic picturebook studies for several years. For instance, the Danish app
Previous studies on picturebook apps and their aesthetics have focused on two characteristics: the interactive features of book apps and the number of user options provided by them (Schwebs; Stichnothe). For instance, Lovise Søyland and Marte S. Gulliksen emphasise user autonomy and describe the co-creative aspects of picturebook apps as “potentially crucial” for sense-making (4). Søyland and Gulliksen further describe the boundaries between interactivity and user autonomy as “akin to following someone else’s path, without the possibility of finding one’s own path and creating new solutions” (9). In other words, interactivity does not afford opportunities for direct involvement in the digital narrative, and interactive hotspots may even interfere with reading comprehension.
Studying children’s digital reading skills and comprehension is a major field of research in itself. Some practical observations of reader behaviour seem to contradict the arguments of studies focusing on the narrative or aesthetics of book apps. For instance, a larger quantity of interactive, though congruent, hotspots in book apps was related to children’s better learning outcomes in a study conducted by Tanya Christ and colleagues in the context of early education. In addition, the findings of Cristina Aliagas and Ana María Margallo’s reader-response study of shared digital reading implicate that interactive features increase child readers’ autonomy.
Studying reader response to digital children’s literature needs to include the perspective of child-computer interaction that is often not considered when looking at interactive books for children. CCI is interested in the relationship between children and computers as a pedagogical issue, but also in children’s activities and behaviour when using computers (Read and Bekker). The CCI point of view includes the child readers and their reading behaviour in the studies of digital children’s literature: children’s engagement with picturebook apps is as interesting as the aesthetic or pedagogical potential of such texts. Since children’s picturebooks often are read aloud by an adult co-reader, adults naturally play a role in the reading of digital picturebooks. In addition, the authors and illustrators of children’s literature take the adult co-reader or co-receiver into consideration (Nikolajeva).
The ways in which child and adult readers navigate and negotiate the shared reading of picturebooks, especially AR picturebooks, are as significant as the multimodal analysis of the medium. In their study on shared AR picturebook reading, Kun-Hung Cheng and Chin-Chung Tsai propose four patterns of AR reading behaviours: “parent as dominator,” “child as dominator,” “communicative child-parent pair,” and “low communicative child-parent pair” (309). Based on the level of cognitive attainment, Cheng and Tsai suggest that a communicative reading strategy that enables a higher level of child agency is the most efficient in relation to reading comprehension and learning. This means that supporting children’s agency as readers is likely to support their literacy.
Digital books and book apps presuppose considerable technical skills from adult co-readers if they wish to support children’s reading and interaction with electronic texts at the same time. In their study on parent-child dialogue and electronic reading, Julia Parish-Morris and colleagues show that parents speak more about the child’s behaviour than about the content of the book when reading electronic books. Drew Cingel and Anne Marie Piper make the same observation when comparing parents’ expressive behaviour during regular e-book and interactive e-book reading sessions. In other words, adults may focus more on operating the book as an object than on reading the interactive work.
Print culture for children has tested the limits of technology and the book medium long before digital literature. This tradition, positioned somewhere between children’s books and playthings, is connected with the tradition of children’s literature in general: toy books, movable books, pop-up books, carousel books, and so on emerged as early as in the Victorian era (Field; Reid-Walsh). Still to this day, babies and toddlers get introduced to literature by engaging with early (toy) books that Marilyn Apseloff calls “learning toys” (63). Although their ontological category is somewhere between books and playthings, Bettina Kümmerling-Meibauer and Jörg Meibauer note that toy books are pre-literature (340). Early-concept books familiarise young children with the book as an object (i.e., how books work), whereas adults have a role in teaching children the “rules of book behaviour” (Lewis 135; see also Kümmerling-Meibauer and Meibauer 340). In this sense, children’s play with books is related to the aesthetics of children’s literature and the pedagogical aims of parents and educators.
Many material enhancements in printed picturebooks seem to serve no other purpose than being interesting and engaging on their own. My suggestion is that the previous critique of the lack of co-creative possibilities in children’s book apps stems from overlooking the specific aesthetics of children’s interactive literature (print and digital). This argument is supported by Lucas Ramada Prieto, who notes that many previous studies have analysed the digital picturebook as a mere evolutionary step-up from the traditional picturebook. This point of view does not consider the multidisciplinary prospects already handed down from the world of digital games (Ramada Prieto; see also Järvenpää).
Reader-response studies have also suggested that the relationship between digital literature and games needs more careful exploration in practice. For example, in a study by Mireia Manresa, the young readers who were more comfortable with digital literature were also the most experienced with video games. These observations highlight the importance of studying the intersection of children’s digital play and digital reading practices. Additionally, as Birgitte Stougaard Pedersen and colleagues argue, “reading with the ears and reading through touch and movement are competences that can create intense multisensory reading experiences that need to be refined and developed” (286). Studying readers’ playful interaction with books is one way to refine how we look at different modes of reading.
However, readers’ navigation between reading a book and playing a game is not straightforward. In Manresa’s study, the participants, ranging from 9 to 15 years old, perceived tension between the story and the interactive elements or between reading and playing. Manresa connects this to the readers’ age: the readers probably had reasonably solid reading habits due to their age. In this sense, digital children’s literature challenges young readers’ reading strategies. Manresa’s participants made a distinction between reading a (printed) book and engaging with a digital book. The following section of this article examines similar dynamics between reading and playing in the case of younger readers and shared AR picturebook reading.
Joann Swann and Daniel Allington make a distinction between two ways of studying readers and reading: experimental and naturalistic. The two types of studies have paradigmatic differences that influence the research design of studies on literary reading. While experimental studies tend to place readers in controlled and artificial environments, naturalistic studies observe participants in their usual environments. In addition, the two types have other distinct characteristics, for instance whether participants engage in atypical (experimental) or typical (naturalistic) reading behaviour.
The research design of this study is a mix of both approaches: while the reading took place in the participants’ homes, I asked them to engage in reading behaviour formerly unfamiliar to them. The study is essentially qualitative and the “reading experiments” were not controlled. The aim of the sessions was to observe the reading activities and collect audio-visual data that capture aspects of the readers’ behaviours, interpretations, or evaluations (Swann and Allington 247; Whiteley and Canning 72) in response to the
Each reading session took place separately in the early fall of 2021. At the beginning of the sessions, I conducted a semi-structured interview with the participants and mapped their reading and gaming habits. Next, I instructed the participants in the joint use of the book and mobile application, after which I observed and recorded the reading, usually lasting approximately 25–30 minutes. The adult participants read the book and the children handled the tablet containing the app. The reading sessions were loosely structured; the participants had the freedom to proceed with the reading and playing as they wished. After reading, we discussed the experience for as long as the children wanted. Finally, about a month after the initial session, I conducted a brief follow-up interview with the parents.
It is important to note that using this kind of method with children requires specific ethical considerations before, during, and after the study. I informed the families about the research design in advance so that the child participants were able to receive sufficient information about the purpose and structure of the study. The children gave their informed and voluntary consent of participation before the study. Throughout the reading sessions, I made sure that the participants, especially the children, were comfortable with the situation. In addition, the research data was collected and processed following the General Data Protection Regulation (GDPR) to ensure the privacy of the participants and their personal information.
Three voluntary families were recruited for the study. The child participants were 4–6 years old and fit the target audience of the book and the app. The children did not have siblings around the same age. One family had an infant participant who was also present in the reading session with their mother. Another family included a grandparent in the session, as he was a regular co-reader in the child participant’s daily life. To ensure the families’ anonymity, I will use the fictive names Alice, Bertha, and Daniel to refer to the children in this article.
The members of each family and their gaming frequencies are presented in
Family profiles.
The collected data includes the audio-visual data of the interviews, discussions, and reading sessions. I carried out the follow-up discussions either in person or via email. Afterwards, I processed the material and analysed the content of the data in a data-driven manner. The analysis focused on the audio-visual data of the reading and playing activities; the main purpose of the interviews was to provide background information and support to the case analyses.
As Sara Whiteley and Patricia Canning point out, studies of reading have moved towards empirical observation instead of the theoretical formulation of the “reader” within classic reader-response and reception theories (see e.g. Culler; Fish; Iser). The impact of the reader-centred approach to literary analysis still influences the empirical study of reading. This study is no exception: the analysis discusses the empirical observations in relation to the affordances (i.e., the hypothetical possibilities of reader/user action) provided by the
After the data collection, I analysed the recordings by focusing on several aspects of the audio-visual data. For instance, the movement and different gestures of the participants were considered in addition to textual data (speech). Based on previous studies on children’s digital play, picturebook reading, and the affordances of book apps (Arizpe and Styles; Frederico; Järvenpää; Koivula and Mustola; van Oers; Serafini; Songer and Miyata), three different themes representing the participants’ activities emerged in the analysis:
The first category of reader engagement with
Interpretative activities include different parts or levels of the
Reading the book and looking at the pictures were natural to all the participants, who interpreted the interplay between the text and the illustrations quite effortlessly. For example, Bertha’s grandfather told me before reading that he always explains the meaning of pictures to Bertha when reading. During the shared AR reading, he described the printed and AR illustrations to Bertha in the same manner.
The technical use of AR with a picturebook was unknown to the readers prior to the study. I had to assist all of them several times during the reading sessions, as the app lacked detailed instructions and turned out to be physically challenging to manage – a topic that all adults brought up after the reading sessions. It is unclear whether the participants would have been able to interpret the user interface sufficiently without my presence in the sessions. For instance, every family got stuck in the same part of the AR storyline, and they did not recognise all the interactive hotspots and possibilities of the app.
As described earlier, the interactive features of the app differ based on the type of AR content. All child participants were more attracted to the panoramic landscapes than the interactive illustrations, even though the 3D scenes were more difficult to navigate. The children quickly learned how marker-based AR works. Finding the marker (a bullfinch) in the printed book quickly became a game of hide-and-seek in itself. The children also recognised the story world locations in AR even when they did not understand what to do with the app. In addition, the child participants expressed their observations and interpretations in a playful manner when using the app:
Alice: “Miksi mä lennän? Eiku mä oon puussa!” (Why am I flying? No, I’m up in a tree!)
Daniel: “Mikä tää on? Onko tää… eikö tää ole peli vai on? Onko tää peli vai ei!” (What is this? Is this… isn’t this a game or is this? Is this a game or not!)
Rules and their negotiation are characteristic of children’s digital play (Koivula and Mustola). These rules may concern, for example, the social or technical dimensions of playing (Koivula and Mustola; van Oers). Although I did not give any instructions regarding the reading session, it started similarly with each family: the adult reader took the book and the child the tablet. In family 2, the grandfather was responsible for reading and the mother helped Bertha with the tablet. As the session progressed, the participants negotiated the rules of reading and playing both verbally and non-verbally. The primary areas of negotiation related to the social, technical, and aesthetic dynamics of shared AR reading. Some frequent questions seemed to emerge as themes of negotiation throughout the data:
Who uses the app?
Who directs the reading?
How is the app used?
How and when to proceed with reading/playing?
What is the meaning of [text, pictures, AR content]?
The question of negotiation settled finally around three subjects: child agency, co-operation, and adult guidance. For instance, Alice and her mother negotiated reading and playing both non-verbally and verbally. They took turns turning the pages, reading aloud, or using the tablet, and they discussed these matters throughout the session. Alice assisted her mother with the app and interpreted the gamified elements of AR to her. In this respect, Alice showed strong agency, but she also relied repeatedly on her mother when trying to understand the content of the app. The mother, on the other hand, noted several times that her daughter knows digital tools better and understands the app more easily.
Similar to Cheng and Tsai’s study, the participants of this study represented different forms of communication and authority. However, I would describe the different parental strategies of negotiation as
Bertha’s family had more participants directly involved in the study and their session was particularly intergenerational. During the reading, the grandfather gave instructions to his daughter (Bertha’s mother) who, in turn, instructed and guided both Bertha and the grandfather. Using the app and reading the book were divided between the two adults and, therefore, navigating between the two modes required more mediation. This family’s shared reading session did not involve as much responsive communication as the other ones and the grandfather adopted an authoritarian role throughout the session. The presence of three generations in the session complicated all areas of negotiation.
As I described earlier, picturebook apps draw aesthetically and technically on children’s digital and mobile games. Many games for younger children involve interactive proposals that may seem unimportant. Games of this kind are more like digital toys: tools for digital play. Therefore, I suggest that focusing only on the narrative importance of apps’ interactive features is reductive. Based on my observations, the shared reading of AR picturebooks includes rather sophisticated forms of playful activities that enhance the reading experience.
The
The first element refers to playful exploration and experimentation with the book and the app. Although the participants did not take advantage of all the interactive affordances, they explored the app’s interface and AR content closely. One part of the app’s storyline turned out to be particularly interesting to the readers. In the scene, the app places its user inside the den where the bears are hibernating. This perspective plays with the aesthetics and narrative of the book. It also engages the readers in immersive interaction with the app. Nagel associates the physical activity required to get a full grasp of a panoramic view with a playful reading manner (9). This reading manner engages the entire body and adds the pleasure of physical play to the reading experience. In this study, Daniel and Bertha were especially intrigued by the immersive scene: Daniel explored the den with his father whereas Bertha came up with her own version of peek-a-boo with the bear characters.
The
All of the children used the tablet or app in ways other than those originally intended. Alice scanned her toes with the app. Daniel engaged his mother, who was not directly participating, in play with the app by directing the bullfinch towards her in AR. Daniel also repeatedly tapped the hotspot that took him back to the home screen of the app. Bertha, who played digital games most often, was the most active creator of new ways of playing with the app. For instance, she interacted with the story characters by playing with them in AR or engaging with them verbally. She was especially keen on the bullfinch, the mediator between the world of the book, augmented reality, and the readers:
Bertha: “Senkin lintu! Mä haluan sinut minun linnuksi!” (You bird! I want you to become my bird!)
Bertha’s engagement with the bullfinch reminds me of Clementine Beauvais’ suggestion that children are better “gap-fillers” than adults when it comes to interpreting the readerly or didactic gap between the text and the pictures of picturebooks, and how “[i]n picturebooks, therefore, there is an elsewhere beyond text and pictures […]” (1). There was something fascinating in the children’s play with AR and the book. Based on the findings of this study, it seems that augmented reality may work as an element of the “elsewhere.”
Based on the observations made in this study, children’s technical competency does not necessarily mean the ability to interpret and navigate a multimodal and interactive text. The child readers were often more familiar with the playful or gamified mechanics of augmented reality, but could not fully connect the different, yet complementary storylines of the book and the app. One of the children, Bertha, was not very interested in discussing such things and focused much rather on playing. The dynamics of the reading sessions indicated interplay between child agency and parental support, in addition to navigation between reading and playing.
The child readers showed active agency in the reading sessions. The question of child agency challenges both the pedagogical and aesthetic goals of children’s literature because child readers have their own preferences and aims that may or may not resonate with the aims of authors, parents, and educators. In this study, the children interacted with the book and the app in creative ways that probably were not initially intended by the authors and designers. This is a remarkable observation in relation to the
The operation of the app turned out to be challenging for both children and adults. Despite the technical challenges, the child readers enjoyed using the AR app throughout the session, which is in line with the findings of previous studies on AR reading (e.g., Yilmaz et al.). However, the readers would have benefited from clearer rules of “book behaviour” regarding augmented reality literature. As previous studies have shown, co-reading parents may engage less in creative and expressive practices when reading interactive digital works. Based on the findings of this study, adults’ technical and literary skills are an advantage in shared digital reading.
The study provides one starting point to studying readers’ playful and shared engagement with hybrid literary products. The findings suggest that studies in children’s digital picturebook reading benefit from understanding children’s digital play. In the future, it would be interesting to explore if, for instance, children’s and parents’ shared gaming experiences support playful co-reading practices. In addition, examining the repeated co-reading of digital picturebooks is desirable, as the participants of this study faced clear obstacles when encountering an AR picturebook for the first time.
Based on the findings of this study, I conclude that the shared reading of digital works presupposes considerable aesthetic, pedagogical, and technical competency from the adult reader and that this competency includes the understanding of children’s digital play and games. This is especially important when dealing with complex multimodal works such as the
The author of this article is thankful to Tammi publishers (Bonnier Books) for donating the books used in the study. In addition, I would like to thank Kaisa Happonen and Anne Vasko for their insights regarding my research.