Immersive Intelligent Aesthetics as Conduit for Digital and Public Humanities Research

2.1 The Advanced Visualisation & Interaction Environment (AVIE)

At the heart of iCinema’s experimental creative-arts-led research lies the Advanced Visualisation & Interaction Environment (AVIE), a bespoke mobile 360° 3D interactive cinematic-scale environment developed by Del Favero and Shaw.¹ It affords groups of up to 30 users the capability to manipulate dynamic 3D datascapes in real time. Its 4.5 metre-high cylindrical screen with stereoscopic visualisation affords a full-body immersive experience that foregrounds the corporeal experience of physical space while retaining social interaction with other users in the environment – contrary to head-mounted-display solutions that cut off users from social interaction by enveloping their visual sense at the expense of kinaesthetic engagement (Thurow 2017, 246). In opposition to conventional desktop computer screens that only provide a small window into a virtual space that remains largely cordoned off from the user’s embodied experience, the virtual world here dominates the user’s sensory apparatus. Its aesthetic experience aims to establish a continuity between the user and representational space, reorganising perception so that the distance between subject and object becomes highly condensed. AVIE fully immerses users through multi-viewpoint projection and spatialised sound in the informational space by enveloping peripheral vision and auditory system. Its carefully calibrated motion capture and tracking system allows exploration not just via voice and tablet but also through gestural and kinaesthetic input. The entire body is thus activated as an interface, facilitating an intimate and intuitive engagement with data across the entire circumference of the projection space which opens up a range of dimensions for holistic enquiry and insight. For example, stage designers may collaboratively review and adapt a simulated set design for a theatre play at one-to-one scale or a group of students may embark on a virtual field trip to ancient temple ruins in Vijayanagar (India), each garnering comprehensive information through the digital experience that would be inaccessible through a small-screen or an analogue engagement. The interactive functions of the environment additionally provide users with a range of capabilities to deepen their exploration of the displayed data, for example, rotating, zooming in and out as well as collating datasets – actions that significantly augment and provide an edge over conventional information processing by allowing new narratives to emerge from the data (Brown, Barker, Del Favero 2011, 214).

The AVIE environment has been commercialised to 15 partners across Australia, Europe, the US and Asia, enhancing scope for networked research collaboration. To cater to stakeholder needs arising from various disciplines, budgets and objectives, it has been adapted into a range of complementary platforms that supplement iCinema’s infrastructural assets. For example, AVIE_SC and iDOME are mobile 160° and 180° versions, respectively, of AVIE, with reduced circumferences that accommodate up to five simultaneous users. Their size, configuration and lower prize-point make them ideal for temporary location at end-user sites, readily extending audience and user groups beyond the circle of immediate collaboration partners. The systems’ focus on kinaesthetic audio-visual engagement streamlines its technical architecture, keeping its mobile deployment feasible. Tactile and olfactory engagement lies outside of AVIE’s current scope, yet for the purposes of its applications thus far this has been a negligible limitation. Similarly, floor and ceiling projection are aspects that could heighten AVIE’s immersive experience – as seen in the popular works of Japanese collective teamLAB –, yet which have so far not proven cost-benefit efficient to warrant investment. Instead in 2017, UNSW set up the Expanded Perception & Interaction Centre (EPICentre), a novel ultra-high-resolution VR visualisation facility, whose fixed-location 340° 120M pixel EPICylinder environment set new benchmarks in clarity, brightness and contrast for interactive computer-automated visualisation environments (so called ‘CAVEs’) (Paterson, Simpson-Williams, Cordner 2020, 166). While its tiled display, comprising 56 separate LCD screens, slightly detracts from the immersive effect especially in video-reliant creative arts applications, its strength for analysis of scientific data has been extensively exhausted in various bioinformatics, anatomy and neuro-imaging projects (167).

2.2 Foundational Interactive Immersive Visualisation Projects

All projects developed within iCinema’s research context have been conceptualised through a lens that prioritises creative arts methodologies, harnessing the productivity that arises from the reformulation of basic aesthetic parameters of received disciplinary approaches. For example, in the early 2000s when museological displays largely followed non-interactive, small-screen standards strongly privileging the visual sense, Shaw and Del Favero assembled a collaborative team (including digital museology pioneer Sarah Kenderdine, interactive art curator Peter Weibel and AI expert Maurice Pagnucco) to explore the capabilities of 3D animation, 360° video and multi-sensory interpellation for deeply engaging museum audiences in heritage settings that bespeak their ongoing dynamic evolution. For the ARC-funded PLACE Hampi project (2006-10; LP0669163), they designed a highly responsive experience that overlaid cinematography of Vijayanagar’s ancient temple ruins with animated computer graphic scenarios re-enacting key scenes of mythological Hindu texts. Triggered by user gaze and movement in the AVIE space, the animation progresses along alternate pathways, entangling users and characters in basic co-evolutionary narratives. These are enabled through symbolic logic and high-level cognitive programming, which endows digital characters with a limited set of pre-programmed goals – such as seeking to protect or to socialise (Kenderdine 2007, 68). These form the basis on which they are able to enact a limited behavioural vocabulary in response to user positioning, escalating chains of interaction. Through responsive coupling of sounds, picked up from within the AVIE space to subtle visualisation changes, the overall responsive impression for visitors is additionally reinforced. By complementing kinaesthetic visual and auditory immersion with projected textual information that affords insight into Hampi’s layered socio-cultural, historical and political dimensions, the PLACE Hampi project supplied an early proof-of-concept for advanced museological applications that target the entire human sensorimotor apparatus for data exploration (58). Conceptually, this research drew on Bruno Latour’s spatial and Maurice Merleau-Ponty’s phenomenological theories that conceive of the body as closely entangled with the objects of the world, formulating any meaning making as contingent on dynamic emplacement and sensory engagement (62-3). These theories were deployed to provide a point of articulation for the Hindu concept of darshan, i.e. engaging with a deity through sensory impressions, within the PLACE Hampi experience (67). Deploying immersive interactive aesthetics allows translating these concepts into an exploration of new museology approaches that seek to reinstitute the body as an integrated vehicle for information processing. It recognises cultural artefacts as dynamically evolving sites of encounter that speak of past significance as well as of present and future imaginations (69).

Shaw and Kenderdine developed these capabilities further in the subsequent Pure Land project (2012-16) that brought to life the murals of Cave 220 of the Buddhist Mogao Grottoes at Dunhuang (China). Using high-resolution photography and laser-scanned models enhanced through composited animation, pictorial recolouring, digital enlargement and a complex sound design, they created a virtual one-to-one-scale facsimile of the 1378-year-old images of the Medicine Buddha. Entering AVIE, visitors are able to closely investigate colourful reconstructions of the Cave’s images and sculptures, taking in the aesthetic, narrative and spiritual dimensions of these vulnerable cultural artefacts. Conceptually, the project refined the premises of PLACE Hampi, seeking to imbue the rendition of this historic site with an ‘aliveness’ that exploits the capability of digital platforms to augment visitor experience via new functionalities that are inaccessible in the physical domain. Examples are a magnifying function that allows selecting panel sections – even of those near the cave’s ceiling – and enlarging them by a factor of up to 10; a retouching tool that re-enlivens the colour palette of the paintings, overlaying century-old faded designs with their original pigments that had been reconstructed at the collaborating Dunhuang Academy; and animations of paintings via superimposed cinematic sequences that performatively extend the activities depicted on the cave’s walls. These elements contribute to generating a heightened sense of presence in the virtual cave, enabling a reconfigured relation to historical artefacts as giving rise to a cosmos of epistemological and hermeneutic practice. Pure Land was praised for its innovation of conceptual, technological and operational paradigms in the field of digital preservation, cultural heritage interpretation and embodied museography – especially in light of the solutions it offered for engaging visitors with delicate physical artefacts, such as the original cave structure, which prohibit extensive public display (Kennicott quoted in Greuter, Kenderdine, Shaw 2016, 82).

The keys to functional immersive interactive visualisation projects are database architectures and programming that allow flexible access to and dynamic assembly of datasets. This vouchsafes that animations can be correlated with user interaction, so that the imagery responds in conclusive and consistent ways to a given user input. While users of works like PLACE Hampi and Pure Land might not be fully aware of such ‘backend’ engineering, other iCinema applications have specifically foregrounded database navigation and interlinking of assets, calling attention to this ‘underbelly’ of information processing and its aesthetic effect on scopic regimes. For example, the mARChive project, delivers a 360° museological data browser for industry partner Museums Victoria (2014-16; LP100100466), which affords users the means to view, browse, select, compare and analyse over 30,000 assets from nineteen collections, experimentally applying a re-compositional narrative framework. Layered according to thematic fields and stratified by chronology, digitised assets can be displayed within AVIE’s full circumference, collectable via tablet so that users may access and analyse them beyond the visualisation environment at their convenience. Standard solutions at the time had been constrained through comparably flat interfaces, inability to handle large amounts of heterogeneous data and paucity in their consistent assembly due to application of narrow analytical systems. Conceptually, mARChive explored the questions raised by co-Investigator Lev Manovich in his work on info-aesthetics, investigating how the super-human scale of our present-day information structures may be translated to the scale of human perception and cognition (Manovich, forthcoming). This project successfully addressed Museums Victoria’s challenge of interrogating, managing and curating from its vast distributed collections while also providing sustained public access to those parts of collections not on current display. This represents a widespread concern of many institutional agents in the field, such as of ZKM (Germany), which commissioned iCinema to adapt the mARChive solution to cater to the needs of its extensive media arts collection.

The methodologies for effectively assembling and curating video content in AVIE’s full-body immersive space have been supplied through sustained exploration as part of iCinema’s flagship T_Visionarium project (2003-05; DP0345547). This study investigated the potential of interactive narrative as an aesthetic framework for reassembling televisual information within a cinematic space. It provided a user with the means to simultaneously display, browse, curate and recombine 20,000 video clips of free-to-air Australian television into new storylines. To enable this, the data was processed and tagged in line with a set of categories, such as dominant colour scheme, mood, gender of actors, activity, etc. The programming design incorporates low-level AI algorithms that analyse input and data in real time, supporting a user’s search by proffering 125 potentially relevant clips in their immediate field of vision while displaying an additional 125 oppositional datasets at a 180° angle. This allows users to field the database in effective ways, with each collection of assets resulting in a dynamic interactive response by the AI system that seeks to forecast evolving user interest through aggregating new data in light of preceding user choices. While the AI system’s response at times digresses into high-level pattern recognition that is no longer intelligible to human users, the project prototyped an intriguing human-machine dialogue that probes the limits of reciprocal understanding. Doing so, T_Visionarium reformulates the televisual experience into the cinematic space by stripping it of its conventional aesthetic, conceptual and semantic parameters. It dissolves them into a novel immersive spatial experience of inhabiting an all-encompassing mediascape that explicates the televisual dispositive by revealing and engaging users with its underlying formal properties, casting them as active producers rather than passive consumers (Bennet 2008, 39).

A similar deconstructive lens was applied to the genre of cinematic narration in the Scenario project (2011-15; DP0556659), which explores the concept of co-evolutionary narrative through the interaction of human users and AI-enabled autonomous characters. Rather than enacting pre-scripted sequences in response to a defined trigger as in PLACE Hampi, these characters are furnished with basic sensing and interpretive capabilities. This allows them to respond to dynamic user movement in the AVIE space and to escalate interaction through a carefully calibrated decision-making range stored on a knowledge database. For example, they can either prompt users to collaborate or actively antagonise their actions (Scheer 2011, 29). Rather than exposing viewers to a string of pre-sampled cinematic events, the filmic narrative in Scenario is contingent on active intervention by users and their reckoning with the characters’ behaviours. Semantic determinacy and formal closure, as hallmarks of cinematic storytelling, are hence impeded in favour of an emergent interplay of multiple agencies who each contribute to morphing the direction and outcome of the cinematic narrative. Rather than unfolding a preconceived meaning defined by the director, interaction generates semantic structures, constantly in flux with unpredictable outcomes. As such, Scenario translates the basic tenets of postmodern interactive art into the cinematic genre (Colebrook 2014, 142), shifting the focus from auctorial exposition to dynamic process as the key site for cinematic meaning making. Meaning hence is no longer something to be unfolded but an ephemeral malleable product that emerges from transactions between multiple agents. The exploration of such ambient agential networks was refined in the Intra Space project – a collaboration with Academy of Fine Arts, Vienna, funded by the Austrian Science Fund (2016-17; AR299-G21). It applied to human-machine interaction the concept of ‘agential realism’ as developed by theoretical physicist Karen Barad, which stipulates that entities constitute each other and maintain their identities only through processes of interaction (Barad 2007, 123). The project team, including Del Favero and architect Wolfgang Tschapeller, honed the interactive coupling of user and digital character to investigate the potential of advanced AI programming for dynamic spatial exploration. Equipping the digital character with both its own distinct behaviours as well as the capability to sense, mimic and splice a user’s physical behaviour into its choreographic sequences, the project delivered valuable methodologies that support the exploration of reciprocal stimulus and response transactions. This helps pave the way for new forms of creative encounters with technology that let users experience themselves as part of a wider network of agencies, finetuning their awareness of ecological embeddedness that is crucial for developing sustainable ways of being and acting in the world (Del Favero, Thurow, Frohne, forthcoming).

2.3 Elaboration SPACE of SPACE Intelligent SPACE Aesthetic SPACE Frameworks

The latest generation of research at iCinema focuses on upscaling the intelligent capabilities of digital agents and leveraging these for reformulating basic processes in creative arts as well as museological practice. With the long-term objective to unlock new creative and institutional potential with the help of AI, this research focuses on integrating human and machinic capabilities in such a way that technology may augment and extend human creativity. The aim hereby is to institute ‘parallel autonomy’ for digital systems rather than to strive toward a ‘serial autonomy’ that would replace human agents in a digitised process (D’Inverno, McCormack 2015, 2440). In flanking and supporting human agency with digital systems, the experience takes on a distributed quality, honing collaborative and explorative skills by extending the means available for research, production and reflection. Working from the premise that technological advances condition certain ways of practice and vice versa (Delanda 1994, 264), iCinema’s current research investigates the foundational concepts that inform a given practice, taking stock of standard methodologies and analysing how these have enabled progress yet, also specifying how they may inhibit further advances in a field. Currently, the Centre administers three large-scale research projects under the direction of Del Favero that seek to deliver tailored solutions for the museological and performing arts sectors, leveraging AI-methodologies for upscaling collection management, curation as well as design and rehearsal practice.

The netARCHIVE project (2019-22; LP180100126) is undertaken in collaboration with the Sydney-based Museum of Applied Arts & Sciences (MAAS), addressing the challenge of providing compelling user-led digital approaches that enable coherent exploration and management of dispersed collections and their environs. Institutionally, MAAS encompasses three venues located in different parts of Greater Sydney, each holding distinct collections that extend to particular subject areas and geographic locales. To achieve their experiential and curatorial integration, netARChive investigates an embodied, distributed storytelling framework that is underpinned by an artificially intelligent database. It is constituted through a cluster of networked platforms, including the AVIE, a mobile application and a website portal to be run on desktop computers. From each of these platforms, users are able to access and interrogate MAAS’ database. Yet, rather than accessing it through their conventional EMu interface, user searches are funnelled through a customised application programming interface (API) that provides additional AI-supported capabilities. On the one hand, the application works as a real-time recommender system, proffering data from the collection that concurs with a user’s query. It dynamically refines its search outcomes as the user continues to select and save their preferred digitised collection items to a personal account. On the other hand, the system logs each user interaction, processing the bundles of assets generated by visitors as new data on their preferences which is fed into the EMu database.

For example, a visitor to MAAS’ Powerhouse Museum will be able to enter the AVIE and encounter a one-to-one scale panoramic projection of an animated urban scene in Sydney’s Central Business District anno 1850. Being accosted by a range of digital characters, the visitor may choose to engage through gesture and voice with a merchant who sells astronomical instruments. Adopting this character through the app run on their personal mobile device, the user will experience the museum’s physical exhibition with this digital character as a personal guide and interpreter. As users point their device at an artefact, the app will activate augmented content, revealing aural histories, collection links, maps, multimedia content, 3D animations and a web browser. The user will be able to virtually collect and store artefacts on their mobile device, with the system analysing their choices according to emerging fields of interests. Having dynamically mapped these preferences, the system will generate personalised recommendations for further exploration, suggesting both sites in the immediate vicinity of the museum as well as items of interests on display at MAAS’ other venues. To deepen engagement with these knowledge pathways, users may utilise the desktop application to extend their searches into dedicated fora and external partner websites, offering enhanced browsing and social networking functionalities.

The hypothesis is that, when provided with such a framework, users will be empowered to experience heterogeneous artefacts and data in lucid ways and to compose them as an expanding network of distributed yet semantically connected episodes. Doing so will enable to release the concept of museum experience from a solely curator-directed event playing out in a restricted physical space, to advance instead a concept in which visitors and AI systems are involved in co-creating meaning by actively shaping the experience and contributing behind the scenes to data management and elaboration (Macalik, Fraser, McKinley 2015, 2). This will furnish MAAS with greater clarity on their visitors’ interests, enabling them to closely cater to their various visitor demographics as well as to enhance their curatorial pipelines through AI-assisted database management.

A similar objective drives iCinema’s memorySCAPE project (2019-22; LP180100080), which investigates the application of an intelligent database narrative framework to develop the commemoration of war, working with the Australian War Memorial (AWM) in Canberra to integrate their steadily growing heterogeneous collection of artefacts and documents pertaining to the Afghanistan conflict. Since a large portion of this collection represents digitally native material in various formats, the AWM’s challenge resides in processing these for a compelling visitor experience that integrates with physical artefacts while operating within the restricted protocols governing military commemoration. To address this challenge, the project team is currently working on a solution that enables memorial audiences and staff, assisted by an AI database, to compose museological experiences by interlinking and contextualising diverse data across a range of interactive digital platforms. This will deliver a new intelligent form of commemorative database and display, transforming the way in which the experiences and recollections of personnel can be interactively captured, organised and explored. This will allow more effectively addressing the social nature of memorialisation and the multi-dimensional quality of defence personnel experiences than currently practised (Phillips, Reyes 2011, 14). While adopting a similar cross-platform approach as in netARChive, the key AI deliverable for this project is to achieve a coherent translation of a large amount of heterogeneous, mostly unscripted and unprocessed data into compelling narrative episodes that users may assemble into informative storylines. For example, the AWM’s audio-visual archive of accounts by Afghanistan veterans currently comprises 20,000 hours of uncatalogued data that in its present state and volume provides little curatorial value to AWM staff, let alone their visitors. The project team leverages natural language processing algorithms to automatically transcribe this data, comprehensively auto-tagging the personnel accounts with narrative-based keywords. This will enable the digital agent to identify and play back selected veteran accounts in response to conversational user queries about certain facets of the Afghanistan war experience. Hence, the digital agent here significantly improves accessibility and intelligibility of the data, opening it up for analysis and engagement that would otherwise devour a substantial amount of the AWM’s time and resources. Parallel autonomy here articulates as a complementary interactive relationship between human user and digital agent that draws on their respective strengths in quantitative and qualitative information processing. Same as in netARChive, the digital agent programming will enable to identify, analyse and dynamically respond to user queries, supporting or challenging their assumptions with a selection of related data that provides expanded perspectives on the content in question. The digital agent hence becomes a conduit for deepening the engagement with an exhibition. It provides avenues for reaching beyond the physical limitations of artefacts and their on-site discursive framing by inviting the user onto unanticipated informational trajectories. These may reveal unexpected relational patterns among the AWM’s data that can shape new perspectives on experiences of deployment and warfare, their history as well as their impact on communities in Australia and abroad.

These AI capabilities also find application in iCinema’s current performing arts-based applications, which seek to enhance design and rehearsal practices via full-body immersion. The iDesign project (2018-21; LP170100471) thereby targets a reconceptualisation of set design aesthetics. It does so by supplying an interactive digital framework that embodies the entire modelling pipeline, eliminating the costly need for alternating between digital and physical platforms in the creative process. Conventionally, set designers create scaled set models using a mix of analogue tools and desktop applications. Once drafted, the scale model provides the blueprint for a physical trial-extrusion on stage. In this so-called ‘Bauprobe’, the functionalities and aesthetic properties of a proposed design are evaluated and subsequently adjusted (Brejzek, Wallen 2018, 20). Due to growing resource constraints and sustainability concerns in the performing arts sector, alterations at this stage in the production process tend to focus on improving a design’s operation rather than radically revising its foundational aesthetics. By immersing designers in AVIE’s one-to-one scale visualisation space, iDesign affords the means to collaboratively prototype set models from the outset at life size on a virtually twinned theatre stage. Its full-body immersion enables a sense of inhabitable space from the beginning, allowing an impromptu evaluation and critical iteration of design ideas without the obligation to commit costly resources to a physical Bauprobe. With a customisable industry-tailored asset library as well as free-drawing and sophisticated lighting capabilities, the platform enables designers to introduce significant efficiencies into their workflow that open up scope for expanded creative exploration. The platform’s AI functionalities thereby support practice in three ways: firstly, the system monitors a design’s effect on audience sightlines, calculating optimal dimensions and placement of assets on stage to improve the viewer experience while keeping negative impact on aesthetic quality at a minimum. Secondly, it tracks occupational health and safety compliance, evaluating a design’s potential hazards and suggesting a range of feasible adjustments. Thirdly, on the grounds of its steadily growing design archive, the AI is being trained to creatively collaborate with the user, for example by identifying a designer’s intentions and recommending solutions to problems on the grounds of previously realised designs. This form of parallel autonomy for the AI agent is carefully integrated into the modelling process so as to not disrupt the creative flow of ideas or unduly overdetermine experimentation. Responding to desiderata identified through industry focus groups, the AI system can be fore- or backgrounded in the application interface, ensuring that it will analyse input throughout, yet only offering suggestions to the creative team when prompted.

Other research at iCinema currently adapts and extends the AI capabilities developed thus far in its research context, applying them to support the entire performing arts production design process. It develops an intelligent cyber-physical spatial aesthetic that affords the means to virtually ideate, compose, integrate and test the entire palette of design components flowing into a stage production within an interactive 3D modelling environment. This represents a significant step change for performing arts rehearsal design as the current standard relies on partitioned design pipelines that draw on a range of individual digital tools. While the latter have introduced significant efficiencies especially into the creative ideation phase – proving powerful in refining emergent design ideas –, communicating these ideas across the creative team and testing their feasibility vis-à-vis other components is currently still markedly constrained. This is due to the lack of a platform that is capable of visualising and flexibly adapting the interaction of all activity streams, such as set, costume, lighting and multimedia design. Thus far, comprehensive testing, evaluation and refinement is only undertaken in costly physical stage rehearsals that require gathering the entire team of directors, designers, choreographers, cast, musicians and technicians in the theatre space, which during this time is unavailable to other performers. Consequently, rehearsal periods tend to be timed meticulously, condensed into very short periods of time that place contradictory demands of efficiency and creative affluence on the team to communicate and fuse their separate visions into a coherent whole. The new iCinema platform will address this limitation by developing a fully immersive shared rehearsal environment that facilitates collaborative decision making by distributed creative teams. Using a dynamic cross-platform visualisation pipeline (comprising AVIE, HTC Vive, tablets and desktop computers that will be networked using the HEVS software; Bourke, Bednarz 2019, 65), the creative team will be able to upload their specific design components into a shared modelling space where they can virtually validate ideas before testing their interaction on a physical stage. Component data may extend to costume and choreographic sketches, musical scores, set models and stage plans, indicative lighting states, mechanical tracking notes, performance cues, screen content, robotic props, stage configurations and truss plans. The modelling system will allow real-time team composition, assembly and manipulation of components – both in orthographic (i.e. diagrammatic) as well as stereoscopic (i.e. 3D) view. Each interaction on the platform will be recorded, enabling playback and archiving of component operation, both individually and in combination, in time with the musical score, to afford exhaustive testing capabilities. The AI deliverables for this platform once again target the streamlining and optimisation of testing and iterative refinement. Its AI data analysis, documentation and recommendation system tracks user action on the platform, monitoring the impact of component integration on line-of-sight angles as well as flagging potential collisions or obstructions resulting from component dynamics. At any time, creatives may prompt the system to generate alternative solutions to their current vision, with capability to ‘accept and integrate’ or ‘reject and return’ to their original design. Leveraging the agility and computing power of the AI system in this way will allow the team to swiftly revert their focus back to aesthetic conceptualisation and refinement rather than getting caught up in lengthy assessments and calculations. The system will hence support creative flow, collaborating with users on solving integration issues as well as supplying expanded resources for further ideation that may impact future aesthetic direction. Accordingly, the AI system will supply both the framework for interaction as well as act as a proactive partner throughout the design process, configuring it as an evolving matrix of co-determining material, human and technological dynamics (Latour 2017b, 174).