Immersive Intelligent Aesthetics as Conduit for Digital and Public Humanities Research

Susanne Thurow

iCinema Centre, UNSW Sydney, Australia

Dennis Del Favero

iCinema Centre, UNSW Sydney, Australia

Ursula Frohne

Westfälische Wilhelms-Universität Münster, Deutschland

Abstract The article outlines conceptual frameworks and experimental studies developed at the University of New South Wales, surveying a selection of creative arts industry research projects. These deliver fully immersive interactive visualisation systems that enable new forms of embodied engagement with artistic, cultural and performance datasets, deploying Artificial Intelligence to analyse and augment user navigation for improving data identification, access and processing. Sketching the developmental trajectory of intelligent aesthetics across these projects, the paper closes with a critical consideration of effects and implications of human-machine co-agency in this new context of cultural reflection.

Keywords Artificial Intelligence. Collaborative Design. Collection Management. Digital Museology. Immersive Visualisation. Interactive Aesthetics. User-led Experiences.

1 Introduction

Digital and public humanities research methodologies have generated robust capabilities for deep artistic and cultural engagement in the Digital Age. By leveraging the scale and speed of digital platforms, new pathways have been generated to explore cultural heritage, discourse formation and creative practice. As these technologies themselves have facilitated new forms of being and doing across societies, their deployment for exploring the human condition, and its manifold ecologies, provides suitable means to reflect transformations in a highly malleable medium. Digital platforms and their interfaces can be configured to cater to multiple dimensions of human perception, simultaneously activating multiple senses so users can comprehensively generate knowledge in highly contextualised ways. Flexible data assembly and dynamic analysis afford capability to amplify the potential for knowledge creation, allowing exploration of both the datasets themselves as well as their usage over time. This integration of artefact and engagement data supplies compelling avenues especially for creative arts and humanities research as it streamlines access to heterogeneous information and enables coherent curation and processing. A key challenge is the translation of material artefacts and practices, which have long since dominated historical documentation and memorialisation, into the virtual domain. While 3D-rendered virtual twins and supplementary digitally native material are powerful tools that enable addressing this challenge, the underpinning aesthetics of implementation play a crucial role in achieving engaging and productive representations in virtual space.

For the past two decades, the iCinema Research Centre at the University of New South Wales (UNSW) in Sydney (Australia) has been home to dedicated research into interactive immersive aesthetics, seeking to reconceptualise access and engagement with cultural datasets via digital platforms to enable new forms of experience, exploration and insight. Experimenting with cinematic-scale motion-tracked environments networked to desktop and mobile applications, the research has focused on leveraging intuitive interface design and interactive capabilities to translate visual, textual and auditory data into expansive 3D multimedia assemblages that embody new perspectives on cultural, industrial and pedagogical data (Langdon 2014, 75-6). Collaborating across disciplines with a wide range of researchers and industry partners, the fusion of artistic and technological approaches has over the years progressively yielded solutions that enable ever more sophisticated simulations and animations of complex real-world scenarios at one-to-one scale. Through cross-platform functionalities, these can be simultaneously engaged with by multiple users. In step with contemporary philosophical currents that highlight agency as the result of a transactional web maintained by interrelated entities (Latour 2017a, 58; Barad 2007, 123) and technological advances in the field of computational knowledge representation and reasoning (e.g. Schwering, Lakemeyer, Pagnucco 2017), new research at iCinema has been exploring the integration of immersive visualisation and Artificial Intelligence (AI) programming to leverage the Centre’s existing capabilities for unlocking the benefits of latest advances for creative arts and humanities research. Doing so promises to deliver a step change for both creative practice as well as academic reflection since it aims to couple the speed and scale of computational processing with the subtlety and agility of human creativity in the production, curation and analysis of cultural artefacts.

The article surveys a selection of milestones of iCinema’s research across the past two decades, outlining the technological and conceptual frameworks that guide the development of its digital methodologies. With this broader research context mapped, the paper critically reviews the adaptation of iCinema’s established approaches across three current industry research projects. These projects, ranging from museological to performing arts application, each develop artificially intelligent aesthetics for augmenting user engagement and insight. They deliver interactive collaborative full-body immersive visualisation platforms that enable new forms of embodied engagement with cultural datasets, while also intelligently analysing user navigation to improve identification, access and processing of relevant data. Having sketched the developmental trajectory of intelligent aesthetics across these projects, the paper closes with a critical consideration of effects and implications of human-machine co-agency in this new context of cultural expression and reflection.

2 The iCinema Research Centre

In 2002, research artists Dennis Del Favero and Jeffrey Shaw, along with design researcher Neil Brown, co-founded the iCinema Research Centre as UNSW’s first interdisciplinary cross-faculty research centre (Leggett 2003, 76), initially spanning the Faculties of Art and Engineering, and from 2009 the Faculties of Arts & Social Sciences and Science, as well as the National Institute of Dramatic Art (NIDA) – Australia’s major performing arts training institutes. This structure has guaranteed access to and creative exchange with key research centres, groups and facilities across the university, such as with, for example, the Autonomous Systems and Knowledge Representation Groups, the EPICentre VR-facility and the National Facility for Human Robotic Interactions. Drawing on the available interdisciplinary expertise and regularly recruiting new partners from within academia as well as industry, the centre has grown into one of Australia’s most successful research hubs focused on arts-led visualisation, having garnered funding and commissions in excess of AUD $40M from a range of national as well as international sources. Realised projects have thereby spanned the arts, as well as the broadcast, construction, environment, heritage and museum sectors. They have been presented at first-tier outlets, such as ArtLAB ÉPFL (Switzerland), Avignon Festival (France), EMPAC (USA), Melbourne Museum (Australia), Shenyang Research Institute (China), Smithsonian Institution (USA), STRP Festival (Netherlands) and ZKM | Centre for Art & Media (Germany).

2.1 The Advanced Visualisation & Interaction Environment (AVIE)

At the heart of iCinema’s experimental creative-arts-led research lies the Advanced Visualisation & Interaction Environment (AVIE), a bespoke mobile 360° 3D interactive cinematic-scale environment developed by Del Favero and Shaw.1 It affords groups of up to 30 users the capability to manipulate dynamic 3D datascapes in real time. Its 4.5 metre-high cylindrical screen with stereoscopic visualisation affords a full-body immersive experience that foregrounds the corporeal experience of physical space while retaining social interaction with other users in the environment – contrary to head-mounted-display solutions that cut off users from social interaction by enveloping their visual sense at the expense of kinaesthetic engagement (Thurow 2017, 246). In opposition to conventional desktop computer screens that only provide a small window into a virtual space that remains largely cordoned off from the user’s embodied experience, the virtual world here dominates the user’s sensory apparatus. Its aesthetic experience aims to establish a continuity between the user and representational space, reorganising perception so that the distance between subject and object becomes highly condensed. AVIE fully immerses users through multi-viewpoint projection and spatialised sound in the informational space by enveloping peripheral vision and auditory system. Its carefully calibrated motion capture and tracking system allows exploration not just via voice and tablet but also through gestural and kinaesthetic input. The entire body is thus activated as an interface, facilitating an intimate and intuitive engagement with data across the entire circumference of the projection space which opens up a range of dimensions for holistic enquiry and insight. For example, stage designers may collaboratively review and adapt a simulated set design for a theatre play at one-to-one scale or a group of students may embark on a virtual field trip to ancient temple ruins in Vijayanagar (India), each garnering comprehensive information through the digital experience that would be inaccessible through a small-screen or an analogue engagement. The interactive functions of the environment additionally provide users with a range of capabilities to deepen their exploration of the displayed data, for example, rotating, zooming in and out as well as collating datasets – actions that significantly augment and provide an edge over conventional information processing by allowing new narratives to emerge from the data (Brown, Barker, Del Favero 2011, 214).

1 With funding supplied by the Australian Research Council through the grants DP0345547, DP0556659 and LE0453517.

The AVIE environment has been commercialised to 15 partners across Australia, Europe, the US and Asia, enhancing scope for networked research collaboration. To cater to stakeholder needs arising from various disciplines, budgets and objectives, it has been adapted into a range of complementary platforms that supplement iCinema’s infrastructural assets. For example, AVIE_SC and iDOME are mobile 160° and 180° versions, respectively, of AVIE, with reduced circumferences that accommodate up to five simultaneous users. Their size, configuration and lower prize-point make them ideal for temporary location at end-user sites, readily extending audience and user groups beyond the circle of immediate collaboration partners. The systems’ focus on kinaesthetic audio-visual engagement streamlines its technical architecture, keeping its mobile deployment feasible. Tactile and olfactory engagement lies outside of AVIE’s current scope, yet for the purposes of its applications thus far this has been a negligible limitation. Similarly, floor and ceiling projection are aspects that could heighten AVIE’s immersive experience – as seen in the popular works of Japanese collective teamLAB –, yet which have so far not proven cost-benefit efficient to warrant investment. Instead in 2017, UNSW set up the Expanded Perception & Interaction Centre (EPICentre), a novel ultra-high-resolution VR visualisation facility, whose fixed-location 340° 120M pixel EPICylinder environment set new benchmarks in clarity, brightness and contrast for interactive computer-automated visualisation environments (so called ‘CAVEs’) (Paterson, Simpson-Williams, Cordner 2020, 166). While its tiled display, comprising 56 separate LCD screens, slightly detracts from the immersive effect especially in video-reliant creative arts applications, its strength for analysis of scientific data has been extensively exhausted in various bioinformatics, anatomy and neuro-imaging projects (167).

2.2 Foundational Interactive Immersive Visualisation Projects

All projects developed within iCinema’s research context have been conceptualised through a lens that prioritises creative arts methodologies, harnessing the productivity that arises from the reformulation of basic aesthetic parameters of received disciplinary approaches. For example, in the early 2000s when museological displays largely followed non-interactive, small-screen standards strongly privileging the visual sense, Shaw and Del Favero assembled a collaborative team (including digital museology pioneer Sarah Kenderdine, interactive art curator Peter Weibel and AI expert Maurice Pagnucco) to explore the capabilities of 3D animation, 360° video and multi-sensory interpellation for deeply engaging museum audiences in heritage settings that bespeak their ongoing dynamic evolution. For the ARC-funded PLACE Hampi project (2006-10; LP0669163), they designed a highly responsive experience that overlaid cinematography of Vijayanagar’s ancient temple ruins with animated computer graphic scenarios re-enacting key scenes of mythological Hindu texts. Triggered by user gaze and movement in the AVIE space, the animation progresses along alternate pathways, entangling users and characters in basic co-evolutionary narratives. These are enabled through symbolic logic and high-level cognitive programming, which endows digital characters with a limited set of pre-programmed goals – such as seeking to protect or to socialise (Kenderdine 2007, 68). These form the basis on which they are able to enact a limited behavioural vocabulary in response to user positioning, escalating chains of interaction. Through responsive coupling of sounds, picked up from within the AVIE space to subtle visualisation changes, the overall responsive impression for visitors is additionally reinforced. By complementing kinaesthetic visual and auditory immersion with projected textual information that affords insight into Hampi’s layered socio-cultural, historical and political dimensions, the PLACE Hampi project supplied an early proof-of-concept for advanced museological applications that target the entire human sensorimotor apparatus for data exploration (58). Conceptually, this research drew on Bruno Latour’s spatial and Maurice Merleau-Ponty’s phenomenological theories that conceive of the body as closely entangled with the objects of the world, formulating any meaning making as contingent on dynamic emplacement and sensory engagement (62-3). These theories were deployed to provide a point of articulation for the Hindu concept of darshan, i.e. engaging with a deity through sensory impressions, within the PLACE Hampi experience (67). Deploying immersive interactive aesthetics allows translating these concepts into an exploration of new museology approaches that seek to reinstitute the body as an integrated vehicle for information processing. It recognises cultural artefacts as dynamically evolving sites of encounter that speak of past significance as well as of present and future imaginations (69).

Shaw and Kenderdine developed these capabilities further in the subsequent Pure Land project (2012-16) that brought to life the murals of Cave 220 of the Buddhist Mogao Grottoes at Dunhuang (China). Using high-resolution photography and laser-scanned models enhanced through composited animation, pictorial recolouring, digital enlargement and a complex sound design, they created a virtual one-to-one-scale facsimile of the 1378-year-old images of the Medicine Buddha. Entering AVIE, visitors are able to closely investigate colourful reconstructions of the Cave’s images and sculptures, taking in the aesthetic, narrative and spiritual dimensions of these vulnerable cultural artefacts. Conceptually, the project refined the premises of PLACE Hampi, seeking to imbue the rendition of this historic site with an ‘aliveness’ that exploits the capability of digital platforms to augment visitor experience via new functionalities that are inaccessible in the physical domain. Examples are a magnifying function that allows selecting panel sections – even of those near the cave’s ceiling – and enlarging them by a factor of up to 10; a retouching tool that re-enlivens the colour palette of the paintings, overlaying century-old faded designs with their original pigments that had been reconstructed at the collaborating Dunhuang Academy; and animations of paintings via superimposed cinematic sequences that performatively extend the activities depicted on the cave’s walls. These elements contribute to generating a heightened sense of presence in the virtual cave, enabling a reconfigured relation to historical artefacts as giving rise to a cosmos of epistemological and hermeneutic practice. Pure Land was praised for its innovation of conceptual, technological and operational paradigms in the field of digital preservation, cultural heritage interpretation and embodied museography – especially in light of the solutions it offered for engaging visitors with delicate physical artefacts, such as the original cave structure, which prohibit extensive public display (Kennicott quoted in Greuter, Kenderdine, Shaw 2016, 82).

The keys to functional immersive interactive visualisation projects are database architectures and programming that allow flexible access to and dynamic assembly of datasets. This vouchsafes that animations can be correlated with user interaction, so that the imagery responds in conclusive and consistent ways to a given user input. While users of works like PLACE Hampi and Pure Land might not be fully aware of such ‘backend’ engineering, other iCinema applications have specifically foregrounded database navigation and interlinking of assets, calling attention to this ‘underbelly’ of information processing and its aesthetic effect on scopic regimes. For example, the mARChive project, delivers a 360° museological data browser for industry partner Museums Victoria (2014-16; LP100100466), which affords users the means to view, browse, select, compare and analyse over 30,000 assets from nineteen collections, experimentally applying a re-compositional narrative framework. Layered according to thematic fields and stratified by chronology, digitised assets can be displayed within AVIE’s full circumference, collectable via tablet so that users may access and analyse them beyond the visualisation environment at their convenience. Standard solutions at the time had been constrained through comparably flat interfaces, inability to handle large amounts of heterogeneous data and paucity in their consistent assembly due to application of narrow analytical systems. Conceptually, mARChive explored the questions raised by co-Investigator Lev Manovich in his work on info-aesthetics, investigating how the super-human scale of our present-day information structures may be translated to the scale of human perception and cognition (Manovich, forthcoming). This project successfully addressed Museums Victoria’s challenge of interrogating, managing and curating from its vast distributed collections while also providing sustained public access to those parts of collections not on current display. This represents a widespread concern of many institutional agents in the field, such as of ZKM (Germany), which commissioned iCinema to adapt the mARChive solution to cater to the needs of its extensive media arts collection.

The methodologies for effectively assembling and curating video content in AVIE’s full-body immersive space have been supplied through sustained exploration as part of iCinema’s flagship T_Visionarium project (2003-05; DP0345547). This study investigated the potential of interactive narrative as an aesthetic framework for reassembling televisual information within a cinematic space. It provided a user with the means to simultaneously display, browse, curate and recombine 20,000 video clips of free-to-air Australian television into new storylines. To enable this, the data was processed and tagged in line with a set of categories, such as dominant colour scheme, mood, gender of actors, activity, etc. The programming design incorporates low-level AI algorithms that analyse input and data in real time, supporting a user’s search by proffering 125 potentially relevant clips in their immediate field of vision while displaying an additional 125 oppositional datasets at a 180° angle. This allows users to field the database in effective ways, with each collection of assets resulting in a dynamic interactive response by the AI system that seeks to forecast evolving user interest through aggregating new data in light of preceding user choices. While the AI system’s response at times digresses into high-level pattern recognition that is no longer intelligible to human users, the project prototyped an intriguing human-machine dialogue that probes the limits of reciprocal understanding. Doing so, T_Visionarium reformulates the televisual experience into the cinematic space by stripping it of its conventional aesthetic, conceptual and semantic parameters. It dissolves them into a novel immersive spatial experience of inhabiting an all-encompassing mediascape that explicates the televisual dispositive by revealing and engaging users with its underlying formal properties, casting them as active producers rather than passive consumers (Bennet 2008, 39).

A similar deconstructive lens was applied to the genre of cinematic narration in the Scenario project (2011-15; DP0556659), which explores the concept of co-evolutionary narrative through the interaction of human users and AI-enabled autonomous characters. Rather than enacting pre-scripted sequences in response to a defined trigger as in PLACE Hampi, these characters are furnished with basic sensing and interpretive capabilities. This allows them to respond to dynamic user movement in the AVIE space and to escalate interaction through a carefully calibrated decision-making range stored on a knowledge database. For example, they can either prompt users to collaborate or actively antagonise their actions (Scheer 2011, 29). Rather than exposing viewers to a string of pre-sampled cinematic events, the filmic narrative in Scenario is contingent on active intervention by users and their reckoning with the characters’ behaviours. Semantic determinacy and formal closure, as hallmarks of cinematic storytelling, are hence impeded in favour of an emergent interplay of multiple agencies who each contribute to morphing the direction and outcome of the cinematic narrative. Rather than unfolding a preconceived meaning defined by the director, interaction generates semantic structures, constantly in flux with unpredictable outcomes. As such, Scenario translates the basic tenets of postmodern interactive art into the cinematic genre (Colebrook 2014, 142), shifting the focus from auctorial exposition to dynamic process as the key site for cinematic meaning making. Meaning hence is no longer something to be unfolded but an ephemeral malleable product that emerges from transactions between multiple agents. The exploration of such ambient agential networks was refined in the Intra Space project – a collaboration with Academy of Fine Arts, Vienna, funded by the Austrian Science Fund (2016-17; AR299-G21). It applied to human-machine interaction the concept of ‘agential realism’ as developed by theoretical physicist Karen Barad, which stipulates that entities constitute each other and maintain their identities only through processes of interaction (Barad 2007, 123). The project team, including Del Favero and architect Wolfgang Tschapeller, honed the interactive coupling of user and digital character to investigate the potential of advanced AI programming for dynamic spatial exploration. Equipping the digital character with both its own distinct behaviours as well as the capability to sense, mimic and splice a user’s physical behaviour into its choreographic sequences, the project delivered valuable methodologies that support the exploration of reciprocal stimulus and response transactions. This helps pave the way for new forms of creative encounters with technology that let users experience themselves as part of a wider network of agencies, finetuning their awareness of ecological embeddedness that is crucial for developing sustainable ways of being and acting in the world (Del Favero, Thurow, Frohne, forthcoming).

2.3 Elaboration SPACE of SPACE Intelligent SPACE Aesthetic SPACE Frameworks

The latest generation of research at iCinema focuses on upscaling the intelligent capabilities of digital agents and leveraging these for reformulating basic processes in creative arts as well as museological practice. With the long-term objective to unlock new creative and institutional potential with the help of AI, this research focuses on integrating human and machinic capabilities in such a way that technology may augment and extend human creativity. The aim hereby is to institute ‘parallel autonomy’ for digital systems rather than to strive toward a ‘serial autonomy’ that would replace human agents in a digitised process (D’Inverno, McCormack 2015, 2440). In flanking and supporting human agency with digital systems, the experience takes on a distributed quality, honing collaborative and explorative skills by extending the means available for research, production and reflection. Working from the premise that technological advances condition certain ways of practice and vice versa (Delanda 1994, 264), iCinema’s current research investigates the foundational concepts that inform a given practice, taking stock of standard methodologies and analysing how these have enabled progress yet, also specifying how they may inhibit further advances in a field. Currently, the Centre administers three large-scale research projects under the direction of Del Favero that seek to deliver tailored solutions for the museological and performing arts sectors, leveraging AI-methodologies for upscaling collection management, curation as well as design and rehearsal practice.

The netARCHIVE project (2019-22; LP180100126) is undertaken in collaboration with the Sydney-based Museum of Applied Arts & Sciences (MAAS), addressing the challenge of providing compelling user-led digital approaches that enable coherent exploration and management of dispersed collections and their environs. Institutionally, MAAS encompasses three venues located in different parts of Greater Sydney, each holding distinct collections that extend to particular subject areas and geographic locales. To achieve their experiential and curatorial integration, netARChive investigates an embodied, distributed storytelling framework that is underpinned by an artificially intelligent database. It is constituted through a cluster of networked platforms, including the AVIE, a mobile application and a website portal to be run on desktop computers. From each of these platforms, users are able to access and interrogate MAAS’ database. Yet, rather than accessing it through their conventional EMu interface, user searches are funnelled through a customised application programming interface (API) that provides additional AI-supported capabilities. On the one hand, the application works as a real-time recommender system, proffering data from the collection that concurs with a user’s query. It dynamically refines its search outcomes as the user continues to select and save their preferred digitised collection items to a personal account. On the other hand, the system logs each user interaction, processing the bundles of assets generated by visitors as new data on their preferences which is fed into the EMu database.

For example, a visitor to MAAS’ Powerhouse Museum will be able to enter the AVIE and encounter a one-to-one scale panoramic projection of an animated urban scene in Sydney’s Central Business District anno 1850. Being accosted by a range of digital characters, the visitor may choose to engage through gesture and voice with a merchant who sells astronomical instruments. Adopting this character through the app run on their personal mobile device, the user will experience the museum’s physical exhibition with this digital character as a personal guide and interpreter. As users point their device at an artefact, the app will activate augmented content, revealing aural histories, collection links, maps, multimedia content, 3D animations and a web browser. The user will be able to virtually collect and store artefacts on their mobile device, with the system analysing their choices according to emerging fields of interests. Having dynamically mapped these preferences, the system will generate personalised recommendations for further exploration, suggesting both sites in the immediate vicinity of the museum as well as items of interests on display at MAAS’ other venues. To deepen engagement with these knowledge pathways, users may utilise the desktop application to extend their searches into dedicated fora and external partner websites, offering enhanced browsing and social networking functionalities.

The hypothesis is that, when provided with such a framework, users will be empowered to experience heterogeneous artefacts and data in lucid ways and to compose them as an expanding network of distributed yet semantically connected episodes. Doing so will enable to release the concept of museum experience from a solely curator-directed event playing out in a restricted physical space, to advance instead a concept in which visitors and AI systems are involved in co-creating meaning by actively shaping the experience and contributing behind the scenes to data management and elaboration (Macalik, Fraser, McKinley 2015, 2). This will furnish MAAS with greater clarity on their visitors’ interests, enabling them to closely cater to their various visitor demographics as well as to enhance their curatorial pipelines through AI-assisted database management.

A similar objective drives iCinema’s memorySCAPE project (2019-22; LP180100080), which investigates the application of an intelligent database narrative framework to develop the commemoration of war, working with the Australian War Memorial (AWM) in Canberra to integrate their steadily growing heterogeneous collection of artefacts and documents pertaining to the Afghanistan conflict. Since a large portion of this collection represents digitally native material in various formats, the AWM’s challenge resides in processing these for a compelling visitor experience that integrates with physical artefacts while operating within the restricted protocols governing military commemoration. To address this challenge, the project team is currently working on a solution that enables memorial audiences and staff, assisted by an AI database, to compose museological experiences by interlinking and contextualising diverse data across a range of interactive digital platforms. This will deliver a new intelligent form of commemorative database and display, transforming the way in which the experiences and recollections of personnel can be interactively captured, organised and explored. This will allow more effectively addressing the social nature of memorialisation and the multi-dimensional quality of defence personnel experiences than currently practised (Phillips, Reyes 2011, 14). While adopting a similar cross-platform approach as in netARChive, the key AI deliverable for this project is to achieve a coherent translation of a large amount of heterogeneous, mostly unscripted and unprocessed data into compelling narrative episodes that users may assemble into informative storylines. For example, the AWM’s audio-visual archive of accounts by Afghanistan veterans currently comprises 20,000 hours of uncatalogued data that in its present state and volume provides little curatorial value to AWM staff, let alone their visitors. The project team leverages natural language processing algorithms to automatically transcribe this data, comprehensively auto-tagging the personnel accounts with narrative-based keywords. This will enable the digital agent to identify and play back selected veteran accounts in response to conversational user queries about certain facets of the Afghanistan war experience. Hence, the digital agent here significantly improves accessibility and intelligibility of the data, opening it up for analysis and engagement that would otherwise devour a substantial amount of the AWM’s time and resources. Parallel autonomy here articulates as a complementary interactive relationship between human user and digital agent that draws on their respective strengths in quantitative and qualitative information processing. Same as in netARChive, the digital agent programming will enable to identify, analyse and dynamically respond to user queries, supporting or challenging their assumptions with a selection of related data that provides expanded perspectives on the content in question. The digital agent hence becomes a conduit for deepening the engagement with an exhibition. It provides avenues for reaching beyond the physical limitations of artefacts and their on-site discursive framing by inviting the user onto unanticipated informational trajectories. These may reveal unexpected relational patterns among the AWM’s data that can shape new perspectives on experiences of deployment and warfare, their history as well as their impact on communities in Australia and abroad.

These AI capabilities also find application in iCinema’s current performing arts-based applications, which seek to enhance design and rehearsal practices via full-body immersion. The iDesign project (2018-21; LP170100471) thereby targets a reconceptualisation of set design aesthetics. It does so by supplying an interactive digital framework that embodies the entire modelling pipeline, eliminating the costly need for alternating between digital and physical platforms in the creative process. Conventionally, set designers create scaled set models using a mix of analogue tools and desktop applications. Once drafted, the scale model provides the blueprint for a physical trial-extrusion on stage. In this so-called ‘Bauprobe’, the functionalities and aesthetic properties of a proposed design are evaluated and subsequently adjusted (Brejzek, Wallen 2018, 20). Due to growing resource constraints and sustainability concerns in the performing arts sector, alterations at this stage in the production process tend to focus on improving a design’s operation rather than radically revising its foundational aesthetics. By immersing designers in AVIE’s one-to-one scale visualisation space, iDesign affords the means to collaboratively prototype set models from the outset at life size on a virtually twinned theatre stage. Its full-body immersion enables a sense of inhabitable space from the beginning, allowing an impromptu evaluation and critical iteration of design ideas without the obligation to commit costly resources to a physical Bauprobe. With a customisable industry-tailored asset library as well as free-drawing and sophisticated lighting capabilities, the platform enables designers to introduce significant efficiencies into their workflow that open up scope for expanded creative exploration. The platform’s AI functionalities thereby support practice in three ways: firstly, the system monitors a design’s effect on audience sightlines, calculating optimal dimensions and placement of assets on stage to improve the viewer experience while keeping negative impact on aesthetic quality at a minimum. Secondly, it tracks occupational health and safety compliance, evaluating a design’s potential hazards and suggesting a range of feasible adjustments. Thirdly, on the grounds of its steadily growing design archive, the AI is being trained to creatively collaborate with the user, for example by identifying a designer’s intentions and recommending solutions to problems on the grounds of previously realised designs. This form of parallel autonomy for the AI agent is carefully integrated into the modelling process so as to not disrupt the creative flow of ideas or unduly overdetermine experimentation. Responding to desiderata identified through industry focus groups, the AI system can be fore- or backgrounded in the application interface, ensuring that it will analyse input throughout, yet only offering suggestions to the creative team when prompted.

Other research at iCinema currently adapts and extends the AI capabilities developed thus far in its research context, applying them to support the entire performing arts production design process. It develops an intelligent cyber-physical spatial aesthetic that affords the means to virtually ideate, compose, integrate and test the entire palette of design components flowing into a stage production within an interactive 3D modelling environment. This represents a significant step change for performing arts rehearsal design as the current standard relies on partitioned design pipelines that draw on a range of individual digital tools. While the latter have introduced significant efficiencies especially into the creative ideation phase – proving powerful in refining emergent design ideas –, communicating these ideas across the creative team and testing their feasibility vis-à-vis other components is currently still markedly constrained. This is due to the lack of a platform that is capable of visualising and flexibly adapting the interaction of all activity streams, such as set, costume, lighting and multimedia design. Thus far, comprehensive testing, evaluation and refinement is only undertaken in costly physical stage rehearsals that require gathering the entire team of directors, designers, choreographers, cast, musicians and technicians in the theatre space, which during this time is unavailable to other performers. Consequently, rehearsal periods tend to be timed meticulously, condensed into very short periods of time that place contradictory demands of efficiency and creative affluence on the team to communicate and fuse their separate visions into a coherent whole. The new iCinema platform will address this limitation by developing a fully immersive shared rehearsal environment that facilitates collaborative decision making by distributed creative teams. Using a dynamic cross-platform visualisation pipeline (comprising AVIE, HTC Vive, tablets and desktop computers that will be networked using the HEVS software; Bourke, Bednarz 2019, 65), the creative team will be able to upload their specific design components into a shared modelling space where they can virtually validate ideas before testing their interaction on a physical stage. Component data may extend to costume and choreographic sketches, musical scores, set models and stage plans, indicative lighting states, mechanical tracking notes, performance cues, screen content, robotic props, stage configurations and truss plans. The modelling system will allow real-time team composition, assembly and manipulation of components – both in orthographic (i.e. diagrammatic) as well as stereoscopic (i.e. 3D) view. Each interaction on the platform will be recorded, enabling playback and archiving of component operation, both individually and in combination, in time with the musical score, to afford exhaustive testing capabilities. The AI deliverables for this platform once again target the streamlining and optimisation of testing and iterative refinement. Its AI data analysis, documentation and recommendation system tracks user action on the platform, monitoring the impact of component integration on line-of-sight angles as well as flagging potential collisions or obstructions resulting from component dynamics. At any time, creatives may prompt the system to generate alternative solutions to their current vision, with capability to ‘accept and integrate’ or ‘reject and return’ to their original design. Leveraging the agility and computing power of the AI system in this way will allow the team to swiftly revert their focus back to aesthetic conceptualisation and refinement rather than getting caught up in lengthy assessments and calculations. The system will hence support creative flow, collaborating with users on solving integration issues as well as supplying expanded resources for further ideation that may impact future aesthetic direction. Accordingly, the AI system will supply both the framework for interaction as well as act as a proactive partner throughout the design process, configuring it as an evolving matrix of co-determining material, human and technological dynamics (Latour 2017b, 174).

3 Conclusion

The research conducted at the iCinema Centre has addressed many challenges that have constrained advancements in creative arts and museological design. Its most recent focus on AI probes the current frontiers of collaborative human-machine paradigms, seeking to carry across latest advances in the field of Computer Science into the digital and public humanities to exploit their capabilities for supporting new cultural insight and expression. Dominant practical challenges that circumscribe this work and its translation into broader domains relate to data availability, interaction modalities, pipeline integration and aesthetic innovation. In order to deliver workable digital solutions to industry partners’ requirements, a representative amount of data has to be available and of sufficient quality so as to furnish the AI system with an informational foundation to draw on. This means that, for example, a museum has to first implement a digitisation strategy and provide a data collection that can be streamlined through computation and tagging. Such preliminary steps can become a costly endeavour and need to be resolved prior to commencing any research activity. As digitisation has entered the list of priorities of many institutional agents, the quality and cohesiveness of the records created can be decisive for delivering compelling experiences. A higher degree of completeness and homogeneity within a database enables more streamlined discovery and informative interlinking of assets that result in dynamic human-machine transactions. How these exchanges are best conducted, i.e. which interface will suit users the most, will thereby depend on the kind and length of activity they seek to perform and what form of experience will result in novel insight. For example, while gestural input within AVIE during set design ideation can powerfully support the development of ideas, allowing embodied exploration of inhabitable space, carrying this input modality across the entire refinement phase would prove physically too strenuous and would potentially disrupt the creative flow when adjusting intricate details of a design sketch. Consequently, a combination of input modalities can enhance the usability of a platform, flexibly adapting to the needs of a user. A new platform’s user-friendliness is key also because it needs to as effortlessly as possible integrate with existing institutional pipelines to ensure uptake by staff and visitors. If an interface design proves unintuitive and too complex, it will not successfully compete with established systems as users will likely revert back to proven and familiar solutions once the ‘gadget novelty effect’ wears off. Providing compatibility with software suites that are firmly established on the market and ensuring seamless platform exchanges can help, making the costly investment that accompanies any new digital ecology a worthwhile consideration within long-term strategic institutional planning. An important consideration for implementing such strategies is that the digital experience needs to add genuine value to an institution’s engagement, addressing an experiential dimension that the physical alone cannot supply.

iCinema’s research explores how such add-on can be generated by abandoning the conventional approach of simply providing a digital 3D twin of a physical artefact that can be explored on a desktop screen. Instead, it reconceptualises the aesthetics that underpin the display of data as well as the status of the digital system as an agent in its own right. Imbuing the digital system with autonomous decision-making capability and enlisting this to support human exploration and reflection of cultural datasets configure it as a shared practice, centralising collaboration and joint agency at its heart. Instead of re-inscribing modernist conceptions of the human subject as central to meaning-making activities, this recalibration articulates agency as a rhizomatic structure that intertwines human and non-human entities in dynamic collaboration. It accounts not just for the human gaze onto the world but introduces machinic reasoning into the epistemological process that complements and expands the user’s schemes. iCinema’s research is thus indebted to theories that incepted a non-human turn in the humanities, conceiving of users as part of a wider network comprising organic and synthetic agencies (e.g. Barad, Delanda, Latour). This body of thought pushes us to redefine our place in and our relation to the world, providing us with the tools to investigate what forms ‘knowledge’ and ‘understanding’ in the Digital Age may take. Fusing human and machinic agencies in research that is conceptualised through a humanities-driven lens provides an ideal conduit for such exploration, given its concern for the ways in which we perceive, process and understand information.


Barad, K. (2007). Meeting the Universe Halfway. Quantum Physics and the Entanglement of Matter and Meaning. Durham: Duke University Press.

Bennett, J. (2008). T_Visionarium: A User’s Guide. Karlsruhe: ZKM.

Bourke, C.; Bednarz, T. (2019). “Multi-Modal High-End Visualisation System (HEVS)”. Jorge, J.; Kim, J. (eds), Proceedings VRCAI. New York: ACM, 65-6.

Brejzek, T.; Wallen, L. (2018). The Model as Performance: Staging Space in Theatre and Architecture. London: Bloomsbury.

Brown, N.; Barker, T.; Del Favero, D. (2011). “Performing Digital Aesthetics: The Framework for a Theory of the Formation of Interactive Narratives”. Leonardo, 44(3), 212-19.

Colebrook, C. (2014). “Not Kant, Not Now: Another Sublime”. Askin, R.; Ennis, P.; Hägler, A.; Schweighauser, P. (eds), Speculations V: Aesthetics in the 21st Century. Brooklyn: Punctum, 127-57.

Delanda, M. (1994). “Virtual Environments and the Emergence of Synthetic Reason”. Mark D. (ed.), Flame Wars: The Discourse of Cyberculture. London: Duke University Press, 263-86.

Del Favero, D., Thurow, S.; Frohne, U. (forthcoming). “Artificial Intelligence and a New Dialogical Commons?”. Tschapeller, W.; Jauernik, C. (eds), Towards an Intra Space. Berlin: Sternberg.

D’Inverno, M.; McCormack, J. (2015). “Heroic vs Collaborative AI”. Yang, Q.; Wooldridge, M. (eds), Proceedings of the 24th International Joint Conference on Artificial Intelligence. Palo Alto: AAAI, 2438-44.

Greuter, S.; Kenderdine, S.; Shaw, J. (2016). “Pure Land UNWIRED: New Approaches to Virtual Reality for Heritage at Risk”. Connor, A.; Marks, S. (eds), Creative Technologies for Multidisciplinary Applications. Hershey PA: IGI Global, 76-98. Advances in Media, Entertainment, and the Arts Series. https//

Kenderdine, S. (2007). “The Irreducible Ensemble: Place-Hampi”. Wyeld, T.; Kenderdine, S.; Docherty, M. (eds), Virtual Systems and Multimedia, VSMM 2007. Berlin; Heidelberg: Springer, 58-72. Lecture Notes in Computer Science 4820.

Langdon, M. (2014). The Work of Art in a Digital Age: Art, Technology and Globalisation. New York: Springer.

Latour, B. (2017a). Facing Gaia. Eight Lectures on the New Climatic Regime. Cambridge: Polity.

Latour, B. (2017b). “On Actor-Network Theory. A Few Clarifications Plus More Than a Few Complications”. Philosophical Literary Journal Logos, 27(1), 173-97.

Leggett, M. (2003). “iCinema’s Big Picture”. RealTime, 57, October-November, 76.

Macalik, J.; Fraser, J.; McKinley, K. (2015). “Discursive Space”. Curator, 58(1), 1-3.

Manovich, L. (forthcoming). Info-Aesthetics. London: Bloomsbury.

Paterson, E.; Simpson-Williams, T.; Cordner, W. (2020). “Interview with Tomasz Bednarz”. Paterson, E.; Simpson-Williams, T.; Cordner, W. (eds), Once Upon a Pixel: Storytelling and Worldbuilding in Video Games. Boca Raton: CRC, 163-73.

Phillips, K.; Reyes, M. (eds) (2011). Global Memoryscapes: Contesting Rememberance in a Transnational Age. Tuscalossa: Uni of Alabama Press.

Scheer, E. (2011). Scenario. Karlsruhe: ZKM.

Schwering, C.; Lakemeyer, G.; Pagnucco, M. (2017). “Belief Revision and Projection in the Epistemic Situation Calculus”. Artificial Intelligence, 251, 62-97.

Thurow, S. (2017). “Response to the Metamaterial Turn: Performative Digital Methodologies for Creative Practice and Analytical Documentation in the Arts”. Australian & New Zealand Journal of Art, 17(2), 238-50.