Geonsun Lee

Effective presentations blend gestures, speech, and slides contents harmoniously. However, current tools for embedding presenters into slides often demand substantial manual effort. We introduce TalkDirector, a novel system that dynamically renders the presenter’s video feed within slides, optimizing placement for content relevance, gestures, and layout. Our pipeline leverages automatic speech recognition, real-time segmentation, gesture detection, and large multi-modal models to infer the video feed’s placement and size within the slides. We conducted two workshops (n=5) to understand motivations and decision-making for video placements in presentations. Further, we collected VTalk-68, a multimodal dataset of 68 presentations with 20 presenters to gain insights into user preferences for video placements in various presentation contexts. A user evaluation against a baseline prototype (n=12) showed that TalkDirector significantly reduces cognitive load and enhances presentation effectiveness, underscoring the potential of dynamic video integration to improve online presentations.

Maintaining engagement in immersive meetings is challenging, particularly when users must catch up on missed content after disruptions. Traditional transcription methods, like tabletop panels, distract users from the group, diminishing social presence, while always-on avatar-fixed interfaces fail to provide past context. We developed a context-aware avatar-fixed transcription panel that adapts based on user engagement, offering live transcriptions and summaries to enhance catching up while preserving social presence. A 12-participant formative study in a live VR meeting guided our design, leading to two user studies with small (3 avatars) and mid-sized (7 avatars) groups. Results showed our method significantly improved social presence (p < .05) over baseline and information recall (p < .05) over always-on interfaces, with even greater improvements in mid-sized groups (p < .001). Participants ranked our method as the most preferred. These findings suggest that context-aware transcriptions effectively maintain engagement and support re-engagement, particularly in larger group settings.

In this paper, we present a novel multi-modal attention guidance method designed to address the challenges of turn-taking dynamics in meetings and enhance group conversations within virtual reality (VR) environments. Recognizing the difficulties posed by a confined field of view and the absence of detailed gesture tracking in VR, our proposed method aims to mitigate the challenges of noticing new speakers attempting to join the conversation. This approach tailors attention guidance, providing a nuanced experience for highly engaged participants while offering subtler cues for those less engaged, thereby enriching the overall meeting dynamics. Through group interview studies, we gathered insights to guide our design, resulting in a prototype that employs light as a diegetic guidance mechanism, complemented by spatial audio. The combination creates an intuitive and immersive meeting environment, effectively directing users’ attention to new speakers. An evaluation study, comparing our method to state-of-the-art attention guidance approaches, demonstrated significantly faster response times (p<0.001), heightened perceived conversation satisfaction (p<0.001), and preference (p<0.001) for our method. Our findings contribute to the understanding of design implications for VR social attention guidance, opening avenues for future research and development.

Reading monolithic instructional documents in VR is often challenging, especially when tasks are collaborative. Here we present DocuBits, a novel method for transforming monolithic documents into small, interactive instructional elements. Our approach allows users to:(i) create instructional elements (ii) position them within VR and (iii) use them to monitor and share progress in a multi-user VR learning environment. We describe our design methodology as well as two user studies evaluating how both individual users and pairs of users interact with DocuBits compared to monolithic documents while performing a chemistry lab task. Our analysis shows that, for both studies, DocuBits had substantially higher usability, while decreasing perceived workload (p<0.001). Our collaborative study showed that participants perceived higher social presence, collaborator awareness as well as immersion and presence (p<0.001). We discuss our insights for using text-based instructions to support enhanced collaboration in VR environments.

Virtual reality (VR) offers the promise of an infinite office and remote collaboration, however, existing interactions in VR do not strongly support one of the most essential tasks for most knowledge workers, reading. This paper presents VRDoc, a set of gaze-based interaction methods designed to improve the reading experience in VR. We introduce three key components: Gaze Select-and-Snap for document selection, Gaze MagGlass for enhanced text legibility, and Gaze Scroll for ease of document traversal. We implemented each of these tools using a commodity VR headset with eye-tracking. In a series of user studies with 13 participants, we show that VRDoc makes VR reading both more efficient (p < 0.01) and less demanding (p < 0.01), and when given a choice, users preferred to use our tools over the current VR reading methods.

In a one-to-many mixed reality collaboration environment, where multiple local users wearing AR headsets are supervised by a remote expert wearing a VR HMD, we evaluated three view-sharing techniques: 2D video, 360 video, and 3D model augmented with 2D video. Through a pilot test, the weaknesses of the techniques were identified, and additional features were integrated into them. Then, their performances were compared in two different collaboration scenarios based on search and assembling. In the first scenario, a local user performed both search and assembling. In the second scenario, two local users had dedicated roles, one for search and the other for assembling. The experiment results showed that the 3D model augmented with 2D video was time-efficient, usable, less demanding and most preferred in one-to-many mixed reality collaborations.