Publication

Virtual reality (VR) has become widely adopted for training, education, and collaboration applications. The construction industry, known for its complex work processes and numerous personnel involved, relies heavily on effective collaboration. Many existing VR research platforms are tailored to specific tasks and offer limited variability. They often demand specialized tools, programming languages, or expertise to enable configurability, thereby restricting accessibility and flexibility. Furthermore, researchers often require diverse settings for conducting user studies, and trainees generally benefit from varied scenarios to enhance training effectiveness. To address these challenges and needs, we introduce Col-Con, a VR platform designed to investigate collaborative behaviors in construction. Col-Con is a configurable, multi-user platform that enables researchers to set up simulations through intuitive and human-readable YAML configuration files. It provides immersive and realistic simulated construction environments, where real-time voice communication, synchronized transformations, animations, sounds, and interactions enhance collaborative experiences. To validate Col-Con's capabilities and investigate user behaviors in a collaborative context, we implemented a realistic and configurable pipe installation task as a case study. A user study demonstrated that pipe installation on Col-Con is highly feasible, with participants reporting a strong sense of immersion and collaboration, making it well-suited for observing collaboration. Thanks to its modular architecture, Col-Con can be easily extended to support additional construction tasks. We envision that Col-Con will facilitate research on exploring virtual reality-based collaborative behaviors in construction and that Col-Con’s architecture and implementation can serve as a reference for designing future VR-based multiplayer research platforms. Col-Con is open-sourced at https://github.com/gmu-dcxr/VRC2-System.

Paper Video Code

Paper Video

Paper Video

Paper Video

The rapid advancement of Vision-Language-Action models has created an urgent need for large-scale, high-quality robot demonstration datasets. Although teleoperation is the predominant method for data collection, current approaches suffer from limited scalability, complex setup procedures, and suboptimal data quality. This paper presents XRoboToolkit, a cross-platform framework for extended reality based robot teleoperation built on the OpenXR standard. The system features low-latency stereoscopic visual feedback, optimization-based inverse kinematics, and support for diverse tracking modalities including head, controller, hand, and auxiliary motion trackers. XRoboToolkit's modular architecture enables seamless integration across robotic platforms and simulation environments, spanning precision manipulators, mobile robots, and dexterous hands. We demonstrate the framework's effectiveness through precision manipulation tasks and validate data quality by training VLA models that exhibit robust autonomous performance.

Paper Video Code

The integration of AI has significant opportunities for enhancing human-machine collaboration, particularly in dynamic environments like the construction industry, where excessive information affects decision-making and coordination. This study investigates how visual attention distribution relates to SA development under information overload by addressing three research questions: (1) How does visual allocation relate to individual SA under information overload? (2) How does visual allocation influence TSA formation? (3) Do high-TSA teams exhibit different visual allocation patterns compared to low-TSA teams? To answer these questions, a multi-sensor virtual reality (VR) construction environment is created as testbed that includes realistic task simulations involving both human teammates and AI-powered robots (e.g., drones and robotic dog). Participants completed a pipe installation task when navigating construction hazards like falls, trips, and collisions, while experiencing varying degrees of information overload. Team situation awareness (TSA)—the shared understanding of tasks and environmental conditions—was assessed using the situation awareness global assessment technique (SAGAT) and eye movements were tracked using Meta Quest Pro. The relationship between eye-tracking metrics and SA/TSA scores is analyzed using linear mixed-effects models (LMMs) and a two-sample t-test compared visual allocation patterns between high- and low-TSA teams. Results indicate that eye tracking metrics can predict SA’s levels, an individual’s SA may also be enhanced through dyadic communication with team members, allowing participants to acquire updates without directly seeing the changes. Furthermore, high TSA teams significantly allocated more attention to environment-related objects and exhibited a more balanced visual allocation pattern (run count and dwell time) on task- and environment-related objects. In contrast, low TSA teams were more task-focused, potentially reducing their awareness of broader situational risks. These findings helps to identify at-risk workers using their psychophysiological responses. This research contributes to developing safer and more effective human-AI collaboration in construction and other high-risk industries by prioritizing TSA and AI-driven personalized feedback.

Paper

Augmented reality (AR) games, particularly those designed for headsets, have become increasingly prevalent with advancements in both hardware and software. However, the majority of AR games still rely on pre-scanned or static scenes, and interaction mechanisms are often limited to controllers or hand-tracking. Additionally, the presence of identical objects in AR games poses challenges for conventional object tracking techniques, which often struggle to differentiate between identical objects or necessitate the installation of fixed cameras for global object movement tracking. In response to these limitations, we present a novel approach to address the tracking of identical objects in an AR scene to enrich physical-virtual interaction. Our method leverages partial scene observations captured by an AR headset, utilizing the perspective and spatial data provided by this technology. Object identities within the scene are determined through the solution of a label assignment problem using integer programming. To enhance computational efficiency, we incorporate a Voronoi diagram-based pruning method into our approach. Our implementation of this approach in a farm-to-table AR game demonstrates its satisfactory performance and robustness. Furthermore, we showcase the versatility and practicality of our method through applications in AR storytelling and a simulated gaming robot. Our video demo is available at: https://youtu.be/rPGkLYuKvCQ.

Paper Video Code

Peggy Smedley and Laura Black talk to Craig Yu, associate professor, computer science, college of engineering and computing, George Mason, Brenda Bannan, professor, division of learning technologies, college of education and human development, George Mason University, and Liuchuan Yu, third-year computer science PHD student, George Mason University. They discuss a project they are doing that uses VR (virtual reality) simulations to study the work and collaboration patterns of people with ADHD in construction.

Advancements in extended reality (XR) technology have spurred research into XR-based training and collaboration. On the other hand, mixed reality (MR) fuses the real and the virtual world in real time and provides interaction, which brings the possibility of completing real-world tasks collaboratively through MR headsets. We present HoloCook, a novel real-time remote cooking tutoring system utilizing HoloLens 2. HoloCook is a lightweight system that doesn't require any additional devices. HoloCook can not only synchronize the coach's action with the trainee in real time but also provide the trainee with animations and 3D annotations to aid in tutoring process. HoloCook supports tutoring two recipes: pancakes and cocktails. Our user evaluation with one coach and four trainees establishes HoloCook as a feasible and usable remote cooking tutoring system in mixed-reality environments.

Paper Video Code

We present a novel AAC application, HoloAAC, based on mixed reality that helps people with expressive language difficulties communicate in grocery shopping scenarios via a mixed reality device. A user, who has difficulty in speaking, can easily convey their intention by pressing a few buttons. Our application uses computer vision techniques to automatically detect grocery items, helping the user quickly locate the items of interest. In addition, our application uses natural language processing techniques to categorize the sentences to help the user quickly find the desired sentence. We evaluate our mixed reality-based application on AAC users and compare its efficacy with traditional AAC applications. HoloAAC contributed to the early exploration of context-aware AR-based AAC applications and provided insights for future research.

Paper Video Code

We synthesized virtual reality fire evacuation training drills in a shared virtual space to explore people's collaboration behavior. We formulate the authoring process of the fire evacuation training drill in a total cost function, which we later solve with a Markov Chain Monte Carlo (MCMC) optimization-based method. The users' assigned task in the synthesized training drill is to help virtual agents evacuate the building as quickly as possible using predefined interaction mechanisms. The users can join the training drill from different physical locations and collaborate and communicate in a shared virtual space to finish the task. We conducted a user study to collect both in-game measurements and subjective ratings to evaluate whether the synthesized training drills would affect how the participants collaborated.

Paper Video

We discuss the existing facilities at the Design Computing and Extended Reality (DCXR) Lab at George Mason University, which comprise mostly commercial-off-the-shelf computing and extended reality devices, for conducting research on virtual reality-based training. We also share thoughts on extending the facilities for conducting more sophisticated virtual reality (VR) training research in the future, which features more advanced functionalities such as remote VR training, adaptive training, and co-training in VR. In particular, we discuss a remote VR training platform to be established between George Mason University and Purdue University.

Paper