Maintaining engagement in immersive meetings is challenging, particularly when users must catch up on missed content after disruptions. Traditional transcription methods, like tabletop panels, distract users from the group, diminishing social presence, while always-on avatar-fixed interfaces fail to provide past context. We developed a context-aware avatar-fixed transcription panel that adapts based on user engagement, offering live transcriptions and summaries to enhance catching up while preserving social presence. A 12-participant formative study in a live VR meeting guided our design, leading to two user studies with small (3 avatars) and mid-sized (7 avatars) groups. Results showed our method significantly improved social presence (p < .05) over baseline and information recall (p < .05) over always-on interfaces, with even greater improvements in mid-sized groups (p < .001). Participants ranked our method as the most preferred. These findings suggest that context-aware transcriptions effectively maintain engagement and support re-engagement, particularly in larger group settings.
Effective presentations blend gestures, speech, and slides contents harmoniously. However, current tools for embedding presenters into slides often demand substantial manual effort. We introduce TalkDirector, a novel system that dynamically renders the presenter’s video feed within slides, optimizing placement for content relevance, gestures, and layout. Our pipeline leverages automatic speech recognition, real-time segmentation, gesture detection, and large multi-modal models to infer the video feed’s placement and size within the slides. We conducted two workshops (n=5) to understand motivations and decision-making for video placements in presentations. Further, we collected VTalk-68, a multimodal dataset of 68 presentations with 20 presenters to gain insights into user preferences for video placements in various presentation contexts. A user evaluation against a baseline prototype (n=12) showed that TalkDirector significantly reduces cognitive load and enhances presentation effectiveness, underscoring the potential of dynamic video integration to improve online presentations.
TVCG
“May I Speak?”: Multi-modal Attention Guidance in Social VR Group Conversations
Geonsun
Lee, Dae Yeol
Lee, Guan-Ming
Su, and Dinesh
Manocha
IEEE Transactions on Visualization and Computer Graphics, 2024
In this paper, we present a novel multi-modal attention guidance method designed to address the challenges of turn-taking dynamics in meetings and enhance group conversations within virtual reality (VR) environments. Recognizing the difficulties posed by a confined field of view and the absence of detailed gesture tracking in VR, our proposed method aims to mitigate the challenges of noticing new speakers attempting to join the conversation. This approach tailors attention guidance, providing a nuanced experience for highly engaged participants while offering subtler cues for those less engaged, thereby enriching the overall meeting dynamics. Through group interview studies, we gathered insights to guide our design, resulting in a prototype that employs light as a diegetic guidance mechanism, complemented by spatial audio. The combination creates an intuitive and immersive meeting environment, effectively directing users’ attention to new speakers. An evaluation study, comparing our method to state-of-the-art attention guidance approaches, demonstrated significantly faster response times (p<0.001), heightened perceived conversation satisfaction (p<0.001), and preference (p<0.001) for our method. Our findings contribute to the understanding of design implications for VR social attention guidance, opening avenues for future research and development.
IEEE VR
DocuBits: VR Document Decomposition for Procedural Task Completion
Geonsun
Lee, Jennifer
Healey, and Dinesh
Manocha
In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), 2024
Reading monolithic instructional documents in VR is often challenging, especially when tasks are collaborative. Here we present DocuBits, a novel method for transforming monolithic documents into small, interactive instructional elements. Our approach allows users to:(i) create instructional elements (ii) position them within VR and (iii) use them to monitor and share progress in a multi-user VR learning environment. We describe our design methodology as well as two user studies evaluating how both individual users and pairs of users interact with DocuBits compared to monolithic documents while performing a chemistry lab task. Our analysis shows that, for both studies, DocuBits had substantially higher usability, while decreasing perceived workload (p<0.001). Our collaborative study showed that participants perceived higher social presence, collaborator awareness as well as immersion and presence (p<0.001). We discuss our insights for using text-based instructions to support enhanced collaboration in VR environments.
Virtual reality (VR) offers the promise of an infinite office and remote collaboration, however, existing interactions in VR do not strongly support one of the most essential tasks for most knowledge workers, reading. This paper presents VRDoc, a set of gaze-based interaction methods designed to improve the reading experience in VR. We introduce three key components: Gaze Select-and-Snap for document selection, Gaze MagGlass for enhanced text legibility, and Gaze Scroll for ease of document traversal. We implemented each of these tools using a commodity VR headset with eye-tracking. In a series of user studies with 13 participants, we show that VRDoc makes VR reading both more efficient (p < 0.01) and less demanding (p < 0.01), and when given a choice, users preferred to use our tools over the current VR reading methods.
2020
IEEE VR
A user study on view-sharing techniques for one-to-many mixed reality collaborations
Geonsun
Lee, HyeongYeop
Kang, JongMin
Lee, and JungHyun
Han
In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2020
In a one-to-many mixed reality collaboration environment, where multiple local users wearing AR headsets are supervised by a remote expert wearing a VR HMD, we evaluated three view-sharing techniques: 2D video, 360 video, and 3D model augmented with 2D video. Through a pilot test, the weaknesses of the techniques were identified, and additional features were integrated into them. Then, their performances were compared in two different collaboration scenarios based on search and assembling. In the first scenario, a local user performed both search and assembling. In the second scenario, two local users had dedicated roles, one for search and the other for assembling. The experiment results showed that the 3D model augmented with 2D video was time-efficient, usable, less demanding and most preferred in one-to-many mixed reality collaborations.
2019
ISMAR
SafeAR: AR alert system assisting obstacle avoidance for pedestrians
HyeongYeop
Kang, Geonsun
Lee, and JungHyun
Han
In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2019
This paper presents an obstacle detection and alert system for the pedestrians who use smartphone AR applications. The system analyzes the input camera image to extract feature points and determines whether the feature points come from obstacles ahead in the path. With the obstacle detector, two experiments were made. The first investigated the obstacle alert interfaces, and the second investigated the orientation guide interfaces that instruct users to hold their smartphones with some angles/orientations appropriate to capture the environment. Then, the best interfaces identified from the experiments were integrated and tested to examine their usability and user experiences.
IEEE VR
Visual manipulation for underwater drag force perception in immersive virtual environments
HyeongYeop
Kang, Geonsun
Lee, and JungHyun
Han
In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019
In this paper, we propose to reproduce drag forces in a virtual underwater environment. To this end, we first compute the drag forces to be exerted on human limbs in a physically correct way. Adopting a pseudo-haptic approach that generates visual discrepancies between the real and virtual limb motions, we compute the extent of drag forces that are applied to the virtual limbs and can be naturally perceived. Through two tests, our drag force simulation method is compared with others. The results show that our method is effective in reproducing the sense of being immersed in water. Our study can be utilized for various types of virtual underwater applications such as scuba diving training and aquatic therapy.
IEEE VR
Jumping further: Forward jumps in a gravity-reduced immersive virtual environment
Hyeong Yeop
Kang, Geonsun
Lee, Dae Seok
Kang, Ohung
Kwon, and
3 more authors
In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019
In a cable-driven suspension system developed to simulate the reduced gravity of lunar or Martian surfaces, we propose to manipu-late/reduce the physical cues of forward jumps so as to overcome the limited workspace problem. The physical cues should be manipulated in a way that the discrepancy from the visual cues provided through the HMD is not noticeable by users. We identified the extent to which forward jumps can be manipulated naturally. We combined it with visual gains, which can scale visual cues without being noticed by users. The test results obtained in a prototype application show that we can use both trajectory manipulation and visual gains to overcome the spatial limit. We also investigated the user experiences when making significantly high and far jumps. The results will be helpful in designing astronaut-training systems and various VR entertainment content.
2018
CHI
Flotation Simulation in a Cable-driven Virtual Environment–A Study with Parasailing
HyeongYeop
Kang, Geonsun
Lee, Seongsu
Kwon, Ohung
Kwon, and
2 more authors
In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018
This paper presents flotation simulation in a cable-driven virtual environment. For this, a virtual parasailing system was developed, where the visual stimulus was provided through a VR headset and the physical stimulus was given by wires. In order to prevent the user from moving out of the limited workspace of the cable-driven system, the visual acceleration was washout-filtered to produce the physical acceleration. In the parasailing trajectory, we focused on the stages of vertical acceleration/deceleration and conducted an experiment to identify how much gain can be applied to the visual acceleration, which makes the user feel the natural self-motion when integrated with physical stimulus. Then, the results were tested using several types of full-course virtual parasailing. The results showed that fairly large differences between visual and physical stimuli would be accepted and different gains could be assigned depending on the user’s altitudes.
SafeAR: AR alert system assisting obstacle avoidance for pedestriansIn 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2019
