Contextual toolbars are toolbars that are spatially connected near the object they control, providing immediate and easy access to relevant functionality. In 3D mixed reality, however, issues related to their placement, including occlusion and reachability introduce new challenges that remain underexplored and can lead to usability problems. To address this gap, we conduct a series of studies, finding that the two leading approaches, object- and user-referenced toolbars, only work well in specific, opposing task scenarios. To address this, we introduce a novel hybrid toolbar that combines the features of the object- and user-referenced approaches, allowing the toolbar to be dynamically anchored to objects or detached from them to follow the user. Our hybrid approach balances efficiency and spatial context, remaining performant across different task scenarios. Our results contribute a new flexible hybrid interaction technique and new design guidelines for toolbars in mixed reality.

mixed reality, augmented reality, toolbar, contextual toolbar, interface design, human-computer interaction, spatial user interfaces

This paper presents a systematic evaluation of five controller-free pointing techniques for 2D target selection in AR, using ISO 9241-411. We compared them across multiple depths (2 m, 6 m, 10 m) in terms of movement time, accuracy, throughput, and workload (NASA TLX). Head- and eye-based pointing significantly outperformed the hand-based methods (Finger, Wrist, and Arm); Head input was the most accurate and remained the most consistent across depth. Depth significantly impacted performance, with complex interactions with target size and distance. Our results offer a comprehensive empirical basis for selecting appropriate controller-free techniques in depth-varying AR tasks.

Pointing, Augmented Reality, Virtual Reality, Fitts’ law

Mobile virtual reality (MVR) facilitates financially accessible VR by incorporating a smartphone into a head-mounted viewer but lacks effective input mechanisms to trigger events. We propose a low-cost unpowered button detected via acoustic sensing and integrated into low-fi pointing devices to support interaction in MVR. The smartphone’s built-in microphone passively listens to environmental audio to identify a unique click sound issued by the button. We compared button performance to 300 ms dwell time with both head-gaze pointing and an optically tracked cardboard controller. The button offered pointing throughput similar to ray-based selection techniques reported in prior studies at 2.22 bps with head-gaze, and 1.81 bps with the controller. Overall, the button accuracy rate was 92.5% with 7.85% of clicks registered as false negatives. Our results suggest the button is a viable input mechanism for MVR that can support 3D pointing techniques.

Virtual reality, Sound-Based Input, Pointing Devices, Interaction Technique, Prototyping

Immersive analytics in mixed reality (MR) often requires users to adjust many virtual parameters, yet only a small number of physical controls can be comfortably worn on the body at once. This creates a key interaction challenge for wearable tangibles: how can a limited set of on-body controls be dynamically rebound to a larger virtual parameter space? We present TactiFlow, a modular forearm-worn tangible interface developed to investigate this rebinding problem. Its slider-and-button modules snap onto a wearable band and can be rebound to different virtual parameters during use. Using TactiFlow, we investigate three embodied linking techniques: Head-Tilt, which uses subtle head motion to target a parameter before module-button confirmation; Gesture-Select, which combines hand-tracked ray selection with tangible confirmation; and Physical-Alignment, which uses computer-vision-assisted spatial alignment between the forearm-mounted module and a virtual control. We evaluated these techniques with 18 participants in an immersive analytics task involving repeated filter adjustment. A preliminary manipulation-validation block verified that the wearable modules provide an effective control substrate relative to in-air manipulation. Our main rebinding comparison revealed clear trade-offs: Head-Tilt enabled the fastest and lowest-effort linking overall, Gesture-Select offered a familiar but moderately slower alternative, and Physical-Alignment, while intuitively grounded in spatial coupling, imposed higher workload and slower relinking under seated use. These findings highlight the importance of rebinding in wearable tangibles for immersive analytics in MR and contribute guidance for future on-body interfaces that must support many virtual parameters using a limited number of reconfigurable controls.

Mixed reality, wearable tangibles, on-body interaction, dynamic rebinding, parameter control, immersive analytics

Timelines are effective ways to tell historical and personal stories. However, most timeline visualization tools impose an inflexible model of time prioritizing chronological clarity. On the other hand, unconstrained representations can better capture the irregular and contextual nature of lived time, but often at the cost of interpretability. In this work, we explore this continuum with a study of how historical and personal timelines could manifest in physical spaces. We conducted a formative study (N=12) in which participants freely arranged events within a physical environment. We observed a diversity of strategies reflecting the personal and context-dependent nature of temporal mental models. We also invited participants to consider how others could move through their timelines. Our analysis led to a choreographic approach to timeline creation, as well as a proof-of-concept tablet-based augmented reality (AR) application that supports spatial timeline drawing and viewing. Finally, we reflect on the design implications of encoding chronology, pacing, and spatial context in immersive timeline stories.

Visualization, timelines, formative study, augmented reality

Recent advances in 3D reconstruction and view synthesis enable visually rich models from casually captured images and videos, making reconstructed environments increasingly useful for tasks such as visual inspection and spatial search. However, these reconstructions remain imperfect. They can be noisy or incomplete, smooth away fine detail, and introduce artifacts that limit reliable verification. As a result, users face a trade-off between pixel-accurate 2D images that lack spatial context and navigable 3D models that may be visually compelling but unreliable for confirming details. We present SpatialSync, an interactive system that tightly couples side-by-side 2D and 3D workspaces to reduce the cost of cross-modal validation. SpatialSync establishes bidirectional correspondence between modalities, allowing users to synchronize viewpoints, map 2D pixels to precise 3D locations, retrieve 2D evidence from 3D selections, and create persistent cross-modal annotations anchored in both representations. The system also supports multiple 3D representations, including dense point clouds and 3D Gaussian Splatting, while preserving correspondence. In a user study (N=11), participants reported that bidirectional transitions and linked annotations improved contextual grounding and reduced re-orientation effort when verifying fine details in reconstructions.

Spatial Interaction, 2D-3D Interaction, Cross-Modal Interaction

When someone is in a state of flow while reading, they are totally immersed, losing track of the physical act of reading. Flow in reading is desirable because it is strongly associated with enjoyment and interest in the content being read. It is unclear, however, whether mixed reality devices—which can be heavy, uncomfortable, and lack display clarity—can support the flow experience as well as conventional reading technologies. In this work, we compare a virtual reality headset (VR HMD) and a see-through augmented reality headset (AR HMD) to a tablet (with an LCD screen) and an eReader (with an e-ink display) in a controlled reading study. Critically, we find that both the VR and AR HMDs better support readers' experience of flow than the eReader and LCD tablet in long-form texts. Our findings provide valuable new insights into the study and understanding of reading experiences in mixed reality.

Mixed Reality, User Experience, Flow

UI/UX designers work with both paper-based and digital artifacts but lack tools that seamlessly integrate the two. Mixed Reality (MR) offers under-explored opportunities to combine the strengths of both design environments. To examine these opportunities, we first conducted interviews with 19 professional UI/UX designers to understand their current experiences using paper and digital artifacts. Motivated and informed by the interview insights, we organized nine conceptual-probe user study sessions in which designers engaged with a MR-probe that combined paper and digital prototyping processes and brainstormed MR's potential in UI/UX design. We found that participants valued MR for enabling continuous hybrid design workflows, reducing manual reconstruction, supporting spatially anchored workspaces, and facilitating real-time cross-medium collaboration. They also envisioned future MR tools with AI assistance, richer interactive and dynamic content, and the ability to manage diverse design artifacts within a unified environment. From these findings, we derive four design dimensions for future MR systems that could enable more fluid, creative, and collaborative design practices.

User Interface Design; Mixed Reality; Design Artifacts; Physical and Digital Artifacts; Design Support

Generative Artificial Intelligence (GenAI) tools are rapidly being used to support user interface (UI) development. When building UIs, AI chatbots are highly effective at instantiating and designing projects; however, the quality of the AI response is highly dependent on the user's ability to translate their ideas into well-structured and defined prompts. Factors like varying AI literacy, can lead to vague inputs, inefficient iteration, and suboptimal outputs. Furthermore, there has been little innovation in how users engage with AI chatbots, as most chatbots rely on free-form inputs from the user. We investigate prompting format as an interaction design problem by comparing a free-form format (FFF) with a question-guided format (QGF) that helps users structure their initial requests before code generation. A within-subjects study with 12 participants completing two comparable UI-building tasks found that the QGF enabled significantly fewer iterations and revisions, illustrating that answering guided questions helps create a more satisfactory initial prompt. Interview results also revealed that the QGF encouraged more deliberate reflection on requirements, but also introduced extra time and a sense of constraint for some participants. Our findings highlight a fundamental trade-off between speed and structured guidance, informing the design of future GenAI interfaces for development.

Generative AI, Prompt Engineering, Human–AI Interaction, User Interface Development, Developer Tools

Writing longer prompts for an AI assistant to generate a story increases psychological ownership, a user's feeling that the writing belongs to them. To encourage users to write longer prompts, we evaluated two interaction techniques that modify the prompt entry interface of chat-based generative AI assistants: pressing and holding the prompt submission button, and continuously moving a slider up and down when submitting a short prompt. A within-subjects experiment investigated the effects of such techniques on prompt length and psychological ownership, and results showed that these techniques increased prompt length and led to higher psychological ownership than baseline techniques. A second experiment further augmented these techniques by showing AI-generated suggestions for how the prompts could be expanded. This further increased prompt length, but did not lead to improvements in psychological ownership. Our results show that simple interface modifications like these can elicit more writing from users and improve psychological ownership.

generative AI, large language models, interaction techniques, controlled experiments

Tutorial videos are widely used for learning feature-rich software, yet following screencast tutorials often breaks down in practice. Through a survey and contextual inquiry, we found that learners frequently rewind or get stuck because critical input information, especially mouse actions and keyboard-modified operations, is often implicit or missing in tutorials without input metadata. To address this problem, we present AutoCue, a multimodal LLM-assisted, human-in-the-loop tutorial augmentation pipeline for externalizing implicit inputs as instructional visual cues. AutoCue integrates frame-to-frame visual changes, narration signals, and operation guidance from official software manuals to infer likely mouse and key-modifier actions, then produces aligned cue layers and editable artifacts for human refinement. Grounded in multimedia learning and cognitive load theory, we further develop a visual cue grammar for representing mouse, keyboard, and combined inputs in software-learning tutorials. We instantiate and evaluate AutoCue in Autodesk Maya, focusing automatic inference on selected UI-mediated interactions with observable visual or textual feedback while supporting more ambiguous state changes through editable authoring artifacts. In a between-subjects study with 24 participants, the AutoCue-augmented tutorial reduced task completion time and interaction breakdowns and showed improved learner-reported experience.

screencast tutorials; software learning; instructional visual cues; Multimodal Large Language Models

Qualitative data analysis is increasingly performed on large datasets, which poses challenges for traditional methods of interpretation. Artificial Intelligence systems can assist with data analysis in the initial stages, but questions remain about control and accountability. We present the Systematic Qualitative Information Diagrammer (SQuID), a mixed-initiative system that uses large language models (LLMs) to propose provisional groupings and hierarchies for affinity diagramming, which researchers can review. We evaluated SQuID in a laboratory study with 13 human-computer-interaction affinity diagrammers completing short, individual analysis tasks. We examined how participants interacted with SQuID during early-stage affinity diagramming, particularly with regard to accepting or rejecting AI-suggestions. Participants described reduced effort during repetitive tasks common in early structuring, alongside recurring concerns about how AI assistance could be justified to others. From these observations, we provide design implications for qualitative analysis tools that support qualitative data analysis while maintaining transparency and opportunities for human judgment.

Interactive Systems, Qualitative Analysis, Affinity Diagramming, Human-AI Collaboration

Thematic analysis is widely used in qualitative research but can be difficult to scale because of its iterative, interpretive demands. We introduce DeTAILS, a toolkit that integrates large language model (LLM) assistance into a workflow inspired by Braun and Clarke’s thematic analysis framework. DeTAILS supports researchers in generating and refining codes, reviewing clusters, and synthesizing themes through interactive feedback loops designed to preserve analytic agency. We evaluated the system with 18 qualitative researchers analyzing Reddit data. Quantitative results showed strong alignment between LLM-supported outputs and participants’ refinements, alongside reduced workload and high perceived usefulness. Qualitatively, participants reported that DeTAILS accelerated analysis, fostered trust through transparency and control, but primarily supported verification-oriented reflexivity, falling short of deeper positional reflexivity. We contribute: (1) an interactive human–LLM workflow for large-scale qualitative analysis, (2) empirical evidence of its feasibility and researcher experience, and (3) design implications for trustworthy AI-assisted qualitative research.

Machine Learning ; Social Media/Online Communities ; Artifact or System ; Interview ; Qualitative Methods ; Quantitative Methods ; Usability Study

Qualitative researchers increasingly rely on computational tools to analyze large text corpora, creating opportunities for scale but also raising concerns about loss of control, transparency, and interpretive authority. In this work, we examine how personal agency can serve as a design lens for qualitative data visualization tools. Using a design science research approach, we conducted interviews with five qualitative researchers to identify agency-related breakdowns in existing tools, developed a design probe embodying agency-driven interaction commitments, and gathered follow-up feedback using the probe as a concrete design artifact. Our findings show that researchers value tools that preserve interpretive control while providing structured guidance and transparent delegation of analytic tasks. Rather than proposing a full AI system, we contribute design knowledge about how interaction structure can support mastery of action, choice, narrative, and space in computationally supported qualitative analysis. These insights inform the design of future qualitative visualization tools that aim to scale analysis without diminishing researcher agency.

Design Methods ; Visualization ; Artifact or System ; Empirical study that tells us about people ; Design Research Methods ; Interview ; Quantitative Methods

Modern knowledge work is increasingly collaborative and fragmented across space and time. The traces from virtual collaboration platforms could be used to support effective collaboration by measuring group-level teamwork characteristics. We call these proposed systems group informatics, which analyze and allow teams to reflect on collaboration dynamics. When systems consolidate and present data from a variety of stakeholders, team members are likely to disagree on what successful collaboration looks like. In this work, we present an exploratory study of (mis)alignment in group informatics systems, focusing on human-human (H-H) alignment, human-algorithm (H-A) alignment, and algorithm-algorithm (A-A) alignment. Using a parallel mixed-methods study across two populations, we collect qualitative perceptions of alignment using a functional technology probe that measures shared understanding through four time-series graphs derived from team Slack messages, in an in-situ study. We complement this with a larger-scale quantitative investigation. Across two populations, we found little evidence of alignment of any type. We find that these teamwork constructs are complex—algorithmic measures capture different facets of the construct, H-H alignment presents a barrier to H-A alignment, and visual presentation of the algorithms influences alignment. We further find that alignment is influenced by extreme or unexpected events. We discuss implications for the design of holistic, co-constructed measures in group informatics systems, the appropriate granularity of algorithmic feedback, and whether alignment is achievable or necessary.

Group Informatics, Collaboration, Alignment, Digital Trace Data, Shared Understanding

This work builds on brushing and linking from data visualization and extends it to knowledge work. Knowledge workers frequently repurpose content across artifacts (e.g., creating a one‑page summary of a report and visuals for its presentation), yet the conceptual relationships between these representations are often implicit, placing a cognitive burden on knowledge workers to maintain alignment. We introduce interpretive linking, an approach that leverages generative AI to identify and highlight semantically related content across representations (brushing) and to interpret and propagate changes between them (linking). Rather than preserving strict correspondence, interpretive linking aims to maintain semantic coherence across formats and modalities. Using a research‑through‑design approach, we built a technology probe leveraging multiple models across textual and visual content and experimented with different strategies for managing semantic coherence. We then conducted a qualitative study with six knowledge workers, who used the probe to reflect on potential workflows. Participants generated 22 application concepts, ranging from responsive documents adapting to device form factor to collaborative tools supporting parallel work across modalities. Together, these findings surface design opportunities and open challenges for interpretive linking in knowledge work.

content transformation, generative AI, knowledge work

Meetings allow sharing knowledge, forming connections, and making decisions. However, we might miss important meetings due to conflicting schedules, and options for catching up are limited and often lack active involvement. To support re-engagement with missed meetings beyond linear consumption of recordings and text summaries, we designed and studied two complementary approaches in our system ReliveMeet: visual storyboards and a chat-based guide. The comic-inspired interactive visual storyboards use frames concisely representing key moments, add relevant quotes, and use panel composition techniques which we describe in a design space. The chat-based guide complements this experience, for asking direct questions in a personalized chat about past events. Frames and chat are interactive, with direct navigation to key moments in the video. We studied both approaches (12 participants), with findings about their distinct affordances, complementary patterns of use, context-dependent strengths (storyboards for complex conversations, chat for informative meetings), and importance of on-demand details expanding on concise summaries.

video-conferencing, interactive meeting recap, storyboards, generative AI

Many visualizations use the X and Y dimensions to define a primary spatial layout, but also need to display additional attributes associated with individual data elements –- e.g., temperatures over time for different locations on a map. Additional visualizations can be embedded within the primary spatial representation, but little is known about how to design these displays to maintain the goals of both visualizations. We created a design-space framework to set out visual and interaction dimensions for embedded time-series visualizations. The framework helps designers understand different methods for visually embedding a time-series, as well as constraints imposed by the primary spatial representation. We then carried out a crowdsourced study to test the effects of three of the framework’s visual factors: shape (spiral vs. rectangle), size (areas of 1000, 2700, 5200 pixels), and encoding (dot, colour, and dot+colour) on user performance when interpreting embedded time-series in three different primary spatial representations.

time-series visualization, embedded visualizations

Natural Language Interfaces (NLIs) are emerging as an alternative to or supplement for dashboard visualization interfaces, allowing users to formulate queries using conversational input rather than structured commands, and have responses provided as visualizations tailored to their query. However, despite growing interest in both natural language and dashboard-based visualization systems, there is limited empirical evidence from controlled user studies that directly compare their usability, cognitive workload, and efficiency on realistic analytical tasks in applied, real-world monitoring contexts. In a controlled study (N=24) we compare a chatbot-driven visualization interface and a visualization dashboard for a set of realistic analytics tasks in a specific applied domain (monitoring fish farm data). While we did not find statistically significant differences in System Usability Scale (SUS) scores or task accuracy scores between interface conditions, NASA TLX scores show significant higher temporal demand and higher mental demand when using the chatbot vs. the dashboard, and the chatbot yielded significantly higher task times overall. Participant feedback indicated complementary strengths, however: overall, the chatbot was praised for simplicity and adaptability, and the dashboard for precision and clarity. We consider ways of integrating natural language and traditional interfaces to enhance data exploration and decision-making.

Data Visualization, Chatbots, Natural Language Interface, Dashboard, Large Language Models, Human-Computer Interaction

Although Virtual Reality (VR) is increasingly used for information work, there is a limited understanding of how data should be encoded in immersive visualizations to support accurate interpretation. It remains unclear whether established two-dimensional graphical perception guidelines generalize to VR settings with different viewing conditions. To address this gap, we conducted two immersive experiments in which participants performed ratio-estimation tasks, with mean absolute error as the metric. In Experiment 1, we replicated a foundational graphical perception study on a virtual tabletop display and found that accuracy followed the established ranking of visual variables, with the lowest mean absolute error for bar charts, followed by length and area encodings. In Experiment 2, we examined the effects of chart orientation and viewing distance on a virtual wall display, extending an established large-display paradigm to VR, and found that horizontal bar charts produced higher error and greater between-participant variability than vertical bar charts, with accuracy degrading more sharply with viewing distance for horizontal than vertical orientations. These findings indicate that while classic perceptual rankings largely carry over to VR, viewing conditions interact with chart design in ways that affect accuracy, informing recommendations for optimal encoding of 2D visualizations for immersive environments.

Graphical Perception, Virtual Reality, Visualization, Human-Computer Interaction, Immersive Analytics

The expressivity of hand-drawn and design-oriented visualizations stems from the rich composition, hierarchical organization, and controlled variation of glyphs. However, existing visualization authoring tools offer limited support for constructing such glyphs: they treat glyphs as flat assemblies of marks, support limited parametrization, and rely on procedural workflows that hinder exploration. We present Glyphaestus, an interactive system for designing parametric, drawing-based glyphs with hierarchical composition and structured variability. Glyphs are compositions in which designers define reusable components, expose visual parameters, and specify constraints across multiple levels. This enables designers to move from graphical construction to data semantics prior to data binding. We report on a preliminary user study exploring how Glyphaestus expands the expressive design space of glyph-based visualization and supports the transition from graphical representations to data-driven concepts.

Visualization authoring systems, expressive data visualization, parametric design, glyph-design tools, graphical constraints

Distance field rendering is an effective technique to render low resolution vector-like raster assets with sharp detail. In 2D, this technique is commonly used to render simplified mono-colour elements such as text or glyphs alongside 3D objects. We extend signed distance fields for improved rendering of crisp outlines and rendering interior multi-colour regions. Outlines are rendered through a nonlinear distance metric that prevents detail loss during downsampling. Interior regions are represented through a fixed number of four distance field channels, plus four channels to store colour indices, inspired by the four-colour mapping problem. Our method enables cartoon-like sprites, images, and icons to be rendered at high quality with reduced storage and memory requirements for real-time rendering.

Distance Fields, Real-Time Rendering, Image-Based Rendering

Image mosaics have traditionally been automated by employing a search method to match source images to locations in a target layout. However, search can be replaced by a generative process that generates custom tiles. The challenge is twofold: the generated images must match a target color layout, while maintaining the image quality of unconditional generation. Here, we use a diffusion model within a training-free, optimization-based framework. Users select a theme that comprises a dataset of text prompts, which guide the generation of each mosaic tile. Rather than fine-tuning, we introduce a custom loss function that balances two competing objectives: ensuring that the local color of the generated tile matches that of the target, while preserving sufficient visual diversity and detail. Our approach requires no model training or large image databases, making it broadly accessible and adaptable to arbitrary themes and target images.

Image Mosaics, Photomosaics, Image Generation, Diffusion Models

Characterized by a deliberately limited resolution and colour palette, pixel art is an exercise in the conveyance of visual information with a limited number of samples. We present the first computational solution to a novel problem in computer graphics: the rendering of images in the pixel art style on arbitrary pixel shapes – restricted only by their ability to tile the plane – while simultaneously respecting image features. We formulate the non-square (or any-shape) pixel art rendering task as an energy minimization problem over tile-shaped filter supports, given a conventional raster image and geometric tiling data as input. We then demonstrate that our method produces images with superior qualitative and quantitative properties in comparison with naive methods. Finally, we offer some stylization and artist interaction procedures for the final render, such as colour palette generation and simple adjustment of the filter vertices. This method has the potential to be useful in several artistic contexts, such as the creation of highly stylized portraiture and landscapes, and authoring of image and video for real hardware displays that use non-square pixels.

Pixel Art, Stylized Rendering, Tilings, Image Abstraction, Gradient Descent

Materials used in real clothing exhibit remarkable complexity and spatial variation due to common processes such as stitching, hemming, dyeing, printing, padding, and bonding. Simulating these materials, for instance using finite element methods, is often computationally demanding and slow. Worse, such methods can suffer from numerical artifacts called “membrane locking'' that makes cloth appear artificially stiff. Here we propose a general framework, called SpringTime, for learning a simple yet efficient surrogate model that captures the effects of these complex materials using only motion observations. The cloth is discretized into a mass-spring network with unknown material parameters that are learned directly from the motion data, using a novel force-and-impulse loss function. Our approach demonstrates the ability to accurately model spatially varying material properties from a variety of data sources, and resistance to membrane locking which plagues FEM-based simulations. Compared to graph-based networks and neural ODE-based architectures, our method achieves significantly faster training times, higher reconstruction accuracy, and improved generalization to novel dynamic scenarios. Codebase for the paper can be found at \url{https://github.com/ericchen321/springtime}.

neural surrogate, neural constitutive model, neural cloth

We present a method for increasing the accuracy of multigrid solvers by optimizing the entries in the smoother matrices. We focus on real-time applications where the wall clock time restrictions require the multigrid solver to run for a small, predetermined number of iterations. We additionally focus on physical systems that have a constant, linear system matrix. In doing so, we are able to modify the entries in the smoother matrices directly, which makes our method very easy to incorporate into existing multigrid solvers. We demonstrate our method using two examples: the heat equation and statics with linear elasticity. For the heat equation, we observe up to an 11.7x decrease in the average error for a single solve. For statics with linear elasticity, we observe up to a 41.7x decrease in the average error for a single solve.

multigrid, PDEs, optimization

Users frequently need to enter both text and commands on mobile devices, and these commands can be selected in several ways. Previous research suggests that keyboard shortcuts are substantially faster for command selection than GUI components such as menus or toolbars, but these studies tested desktop rather than mobile environments. It is possible that some of the factors that lead to the advantage of keyboard shortcuts in desktop settings do not have as large an effect in mobile interfaces. To better understand the performance of these mechanisms in mobile typing tasks, we carried out a study (N=120) that compared four command-selection methods: a chording technique, a hotkey technique, and two types of onscreen toolbar (one that required navigation actions, and one in which all commands were visible). Results showed that in both a training game and a phrase-typing task, the toolbar with visible commands was substantially faster than the keyboard-based techniques; however, the toolbar that required navigation actions was slower than all other methods. Our study shows that the performance advantages of keyboard shortcuts do not necessarily translate to the mobile context, but also shows that toolbar performance is strongly determined by the number of navigation actions.

Mobile Devices, Command Selection, Keyboard Shortcuts, Chording, Text Entry

Hall effect keyboards use magnetic fields to detect the degree of pressing for each key, enabling continuous and precise sensing of traversal distance compared to a single key switch. If users can control the depth of their key presses, many new interaction opportunities become possible. We explored this richer interaction space of the Hall effect keyboard with a controlled experiment that examined user ability to perform multiple levels of pressing distance with different fingers. Results showed that most participants can accurately press keys with two levels, with the index fingers preferred for multi-level keypresses. We demonstrate the interaction potential of Hall effect keyboards by identifying user abilities and strategies, summarizing design considerations, and discussing practical applications.

interaction techniques, keyboards, controlled experiments

Subtle finger movements, or microgestures, enable low-fatigue and discreet interactions. However, prior work often relies on custom hardware or recognizes isolated gestures, with limited exploration of directional input on commodity devices. We introduce Mudra, a microgesture recognition system that leverages inertial sensors in commercial off-the-shelf (COTS) smartwatches to support a broad range of interactions. Mudra recognizes four primary microgestures: Finger Tap and Surface Tap for selection, and Swipe and Pinch-and-Drag for targeting. Through a user study with 30 participants, we demonstrate that Mudra generalizes effectively across users, achieving an average microgesture classification accuracy of 0.93 ± 0.04. Incorporating directional variants of Swipe and Pinch-and-Drag, we design and evaluate gesture sets for 1D and 2D navigation, achieving user-independent accuracies of 0.93 ± 0.06 and 0.86 ± 0.07, respectively. By enabling subtle, directional input on commodity devices, Mudra paves the way for complete gesture control for smartwatches, media players, and extended reality interfaces.

microgestures, smartwatch, IMU, tapping, swiping

Despite their growing popularity, in-ear Earable / Hearable devices (i.e., ear-mounted wearables) face interaction challenges due to limited input space and compact form factors. To enhance interaction capabilities, researchers are exploring off-device hand-based input spaces above the neck using midair and onskin gestures. However, existing literature primarily focuses on axial swipes (i.e., horizontal and vertical), leaving nonaxial swipes (i.e., unidirectional swipes with varied orientations) and angular swipes (e.g., L, U, or V) largely underexplored despite their potential interaction advantages. To address this gap, we conducted a within-subject gesture motion analysis study with 24 participants, analyzing 5,568 swipes of varying shape, orientation, and complexity. Our results revealed preferred starting and ending regions for different unidirectional and angular swipe shapes, as well as intuitive swipe shapes within the off-device, above-neck manual interaction space. We further examine off-device swipe characteristics, discuss the feasibility of recognizing these earable gestures with current sensing technologies, and highlight their potential application in various scenarios. These findings broaden the understanding of off-device earable gestures and provide design insights for integrating suitable nonaxial and angular swipes alongside traditional axial gestures to enhance interaction with in-ear earable devices.

Embodied Interaction, Input Techniques, Unimanual Interaction, Hand-to-face Gestures, Earables

We explored the technical feasibility of wireless smart wooden panels that integrate conventional sensors and electronics. Prior work has explored either embedding wired, off-the-shelf electronic components in wood or enabling wireless energy transfer, but supporting both simultaneously using existing technology has remained uncertain. Addressing this gap is important for moving smart wooden panels from conceptual visions to practical applications. Following a validation-through-implementation approach, we developed a prototype that leverages a commodity RF-based wireless power transfer (WPT) system to continuously power off-the-shelf sensors (e.g., accelerometers) and electronic components (e.g., MCUs, BLE modules) within wooden furniture panels. We evaluate its performance across three dimensions: panel scalability (how different-sized panels reliably operate embedded sensors), power infrastructure requirements (the number and placement of RF transmitters needed for continuous operation), and activity recognition. Our results demonstrate that wireless smart wooden panels are not merely a conceptual idea, but a viable approach to realizing smart environments.

Smart Environment, Computational Material, Battery-Free

As haptics and tangible interaction are increasingly adopted, designers face challenges when coming to a shared understanding of capabilities and materiality with collaborators. Existing tools either impose or limit the available technologies, reducing the ability for non-technical team members to make decisions. We present the Tickle Trunk: an open-source, hardware-only toolkit for rapid, playful exploration of diverse physical modalities. The toolkit consists of plug-and-play widgets that can be combined in seconds without programming. We report on the design of the Tickle Trunk within a larger co-design project, and on a 14-person user study that found the toolkit to be highly usable, supportive of creative exploration, and easily extensible by designers, with participants adding new components in less than four minutes. By prioritizing speed of iteration, a broad multimodal palette, and open extensibility, the Tickle Trunk facilitates early-stage mutual-learning and ideation to democratize the design of haptic and tangible interactions for any team.

Haptics; Tangible Interaction; Design; Protyping; Multimodal Interaction

We investigate sketch-like pen input as an alternative way to support execution control in interactive debugging. In our interface, programmers draw lightweight marks to set breakpoints, use symbolic strokes to control execution, and extend strokes into spirals to repeat traversal actions. The prototype combines gesture recognition with Python execution tracing in a conventional editor interface. In a controlled study with 24 programmers, we compared the sketch interface with conventional mouse-and-keyboard input on debugging tasks that required breakpoint placement, step-wise execution, and runtime state inspection. The results show that sketch-like input can support these execution-control tasks, while also introducing challenges in precision, recognition, and gesture recall. Our findings suggest that pen input is most promising where debugger interactions benefit from spatial grounding or continuous movement, rather than as a wholesale replacement for conventional debugging controls.

interactive debugging, sketch-based interaction, pen input, interaction techniques, controlled experiments

Multimodal output presents information through multiple sensory channels, and has been shown to improve performance. However, previous work has not considered how expertise in one modality could affect the usefulness of multimodal output. Expertise could potentially reduce the effectiveness of adding another modality (e.g., by interfering with or distracting from the first). To investigate how expertise affects the value of multimodal output, we conducted a study where participants with three levels of audio experience interpreted differences between two values presented as visual, audio, or audio+visual information. We assessed the improvement in time and errors when moving from unimodal to multimodal presentation. Results showed substantial effects of audio expertise: experts improved much less than novices when visual information was added to audio, and improved much more than novices when audio information was added to visual. Our study shows that designers should consider user expertise when designing multimodal output.

Multimodal Feedback, Expertise, Audio, Visual

Effective command selection in graphical user interfaces (GUIs) depends on users' ability to remember command locations. While spatial memory-based interfaces can aid learning, the large number of commands in desktop applications complicates this process. Adding elements like coloured regions or background images can help, but they may also clutter interfaces and hinder learning. We investigate whether variation in icon size — an inherent GUI feature — can aid spatial learning. Therefore, we conducted two studies with varying icon sizes (small, large, and mixed) in interfaces with over 130 commands with meaningful and abstract icons, respectively. Results suggested that size variation, particularly enlarged icons, can improve performance and reduce errors. We show that variation in icon size can serve as an intrinsic spatial cue, thereby improving the design of spatial memory-based interfaces.

Command selection, landmark, menu design, spatial memory

In eye-tracking research, visualizations of eye movements are com- monly rendered as “scanpaths” with movement patterns reduced to fixations and saccades. Although the eyes are crucial for sensory in- put, some eye-tracking applications use the eyes to both sense and control a user interface, with “eye typing” a typical example. Used in this way, the eyes perform “double-duty,” so to speak. We advance the idea that more information is available in double-duty mode since the location of selections (i.e., fixations) is known in advance. This is not so when the eyes are only a sensory input channel, since the eye movements are more casual, even serendipitous. Using data from an eye-tracking experiment using dwell-time target selection, we demonstrate that this added information provides a novel re- source to adapt and fine-tune the algorithm that converts raw data to fixations and saccades. The result is a “gazepath” visualization.

Eye tracking, scanpath, fixation, saccade, sensitivity analysis

Distractors have traditionally been regarded as problematic elements in graphical user interfaces. Interestingly, however, in interactive games, distractors are often deliberately employed as design elements to create novel and engaging scenarios. Among the various characteristics of distractors, prior research has primarily focused on their spatial aspects. In contrast, studies on their temporal effects remain limited. This paper investigates how distractors influence user behavior and performance in temporal target selection, defined as selecting a target within a limited time window. Such interactions are commonly encountered in game scenarios, such as shooting games involving moving targets. We empirically studied the impact of the temporal distractor — including temporal distance, width, and departure position — on temporal target selection. Additionally, we derived a predictive model for user performance and evaluated it in a controlled and uncontrolled experiment. We believe our research findings will assist interactive game designers and developers in testing their designs.

Temporal pointing, distractor effects, endpoint distribution, user behavior

Complex visual information spaces require a considerable amount of time and effort to visually search for items. To help inexperienced users locate items faster, some systems predict and visually emphasize the item the user is looking for. However, while designers of these navigation aids rely on the rehearsal principle to help users acquire spatial knowledge of item locations, in situations where the prediction is unavailable or incorrect, it is unclear whether performance will suffer due to users being overly reliant on the assistance. To investigate the effect of predictive navigation assistance on location learning, we ran a study to compare three navigation techniques: Automatic Assistance highlighted the target item with perfect accuracy, Search Assistance allowed users to filter items by name, and No Assistance provided no help to find items. Results showed that while Search Assistance had comparable memory performance to the unassisted baseline condition, Automatic Assistance had substantially reduced location learning. Our work contributes new understanding of the spatial-memory effects of prediction and rehearsal, and provides designers with new opportunities to support location learning in visual workspaces.

location learning, prediction, navigation assistance, visual workspaces, rehearsal interfaces

Spatial overviews allow fast navigation in static documents by allowing users to develop spatial memory of page locations. However, little is known about whether spatial overviews support navigation tasks that involve inter-document navigation, and whether memory-based navigation is compromised when document sets change. We evaluated these two issues in the context of a real-world software project. Our first study looked at inter-document navigation, and compared a spatial overview to standard IDE tools. Across several finding and re-finding tasks, participants were able to revisit files faster with the spatial overview –- an advantage that increased as experience grew. A second study explored the effects of project evolution on performance with the spatial overview, and found that spatial memory was resilient to local changes, with participants quickly returning to expert-level performance after a change. Our work reinforces the value of spatial overviews for realistic tasks and usage scenarios in software development environments.

Spatial overviews, spatial navigation, space-filling thumbnails

Situational visual impairments (SVIs) often make mobile screen con- tent harder to distinguish in bright real-world settings. Re-analysis of an earlier large-scale dataset and our subsequent GI24 study together suggested that incident light on the display does not fully account for these effects, and that the luminance of the surround- ing visual field is a likely contributor. In this paper, we investigate how surround luminance affects colour differentiation on a mobile display. We conducted a controlled study under three surround luminance conditions (35, 550, and 2000 𝑐𝑑/𝑚2) chosen using out- door luminance measurements. Thirty-six participants completed just-noticeable-difference (JND) tests for 19 stimuli: 7 achromatic and 12 chromatic colours. Brighter surrounds produced larger JNDs across both achromatic and chromatic stimuli, with the effect con- centrated at high L* — JNDs at L*=95 grew by approximately 60% from the dimmest to the brightest surround. A pooled cross-study comparison with our earlier work indicates that JNDs grew approx- imately three times faster per tenfold increase in illuminance when surround was directly manipulated than when only incident illumi- nation on the screen was varied. Spectroradiometer measurements showed that reflected ambient light made only a small contribution to the luminance reaching the observer, supporting a perceptual rather than a purely physical interpretation. These findings argue for surround-aware extensions to mobile-display contrast guide- lines, particularly for designs relying on subtle distinctions among light colours.

HCI, accessibility, mobile computing, colour perception

While the association between colour and emotion is well-documented and the significance of colour in cinema is widely acknowledged, there remains a lack of functional tools for retrieving or recommending movies based on their colour palettes. This paper explores the integration of colour palettes as a feature in movie recommendation systems, moving beyond traditional metadata. We present four core contributions: first, a comprehensive literature review synthesizing colour palette applications, generation techniques, and colour-emotion associations. Second, we evaluate a prototype application that utilizes palettes for movie recommendations; a USE (Usefulness, Satisfaction, and Ease of Use) questionnaire confirms a statistically significant trend toward the tool being perceived as useful, simple, and exciting. Third, we conduct a comparative study between Netflix, Amazon Prime Video, and our palette-based recommendation algorithm on iconic colour-graded movies. While results underscore the market dominance of Netflix, our method produces better movie recommendations than Prime Video in 50% of our study trials, and an analysis of user justifications reveals a clear consideration for colour. Moreover, a statistically significant majority of participants expressed a direct interest in colour-based recommendation features. Finally, we compare a palette-based refinement of TMDb movie recommendations against standard TMDb recommendations. Statistical analysis demonstrates a significant user preference for recommendations narrowed by colour-matching. Together, these findings validate colour as a viable and desired metric for enhancing digital media retrieval.

digital image processing, image retrieval, video retrieval, colour palettes, human-computer interaction

Spatialized document layouts are widely used for exploratory analysis of text corpora, but interpreting the spatial organization of documents and the relationships between regions remains challenging. Existing approaches primarily summarize document content or explain how layouts are generated, providing limited support for understanding spatial relationships within the layout itself. We present CAPE, a context-aware explanation framework that generates natural-language explanations grounded in both document semantics and layout-derived spatial context. CAPE identifies salient spatial patterns (e.g., clusters, subgroups, outliers, and bridging documents) and constructs multi-level contextual representations to guide LLM-based explanation generation. It supports both AI-guided overview and user-driven exploration, with explanations available at multiple levels of detail. We demonstrate CAPE on news and scholarly document layouts and evaluate it in a controlled user study against keyword-based and content-only LLM baselines. Our results suggest that spatially grounded explanations are perceived as more helpful than content-only baselines for interpreting the spatial organization of document layouts.

Spatialized document layouts; Spatial interpretation; Natural language explanations; Visual analytics

We present an investigation of using the web-based 3D virtual world platform, Spot Virtual, for exhibiting design work. We built an exhibition gallery, recruited interior design graduate students, and installed their work in the gallery. This included images, videos, and 3D scanned objects. We then held an online public opening event for the exhibition. We analyzed data from interviews with student participants, survey responses from gallery visitors, and curator reflections. Our findings show that the exhibition was positively experienced by student participants and visitors, and supported dynamic engagement, both socially and around the student work. Students particularly appreciated the scale, fidelity, and amount of their work that could be exhibited, as well as the ability of visitors to join from anywhere in the world. However, both visitors and participants noted some navigational difficulties, and we describe some other interaction features that are missing or lacking. We provide recommendations for enhanced features for web-based 3D worlds that would better support visitor interaction with image, video, and 3D design artifacts.

web-based virtual worlds, social VR, creativity support, virtual exhibitions, virtual galleries

Intimate partner violence (IPV) is a pervasive global health issue. While HCI research often examines how technology facilitates violence (e.g., surveillance, harassment), far less is known about its role in prevention. We conducted a PRISMA-ScR scoping review of 226 articles from the ACM Digital Library, IEEE Xplore, PubMed, and USENIX to survey technology-based IPV prevention. We found that most work focuses on reactive, secondary prevention, with minimal attention to upstream or culturally specific approaches. We synthesize this landscape and outline five recommendations for HCI: (1) establish clear definitions of IPV; (2) increased attention to primary preventions that addresses root causes of IPV; (3) engage stakeholders and end-users; (4) conduct real-world testing and implementation; and (5) tailor interventions to regional and demographic contexts. Without addressing these gaps, HCI risks producing unsustainable or even harmful technologies. Our review calls for a reorientation toward proactive, community-centred, and culturally grounded design of IPV prevention technologies.

Intimate Partner Violence, Violence Prevention, Gender Based Violence, Human-Computer Interaction, Scoping Review

Digital mental health research has been dominated by a data-centric paradigm that treats fragmented sensor signals as objective ground truth, prioritizing predictive accuracy over meaningful user support. This paper challenges that paradigm through a critical perspective grounded in a purposive synthesis of 24 recent studies at the intersection of interactive systems and digital health. We argue that computational models systematically overlook the subjective, context-dependent nature of psychological experience. This misalignment is not a technical shortcoming but a structural consequence of how the problem has been framed. To address this, we introduce two original constructs. First, the complementary evidence chain reconceptualizes mental health assessment as the integration of subjective, behavioral, and physiological data, in which no single modality serves as ground truth and cross-modal divergence is treated as a meaningful signal rather than noise. Second, a user-agency-centered architecture operationalizes this reconceptualization through two mechanisms: algorithmic veto power, which positions users as the governing authority over their own data inferences, and graceful degradation of inference, which preserves system support when users restrict data access. Together, these constructs articulate a system-level transition from passive emotion recognition toward interpretable, participatory, and privacy-preserving mental health support. We conclude with a set of concrete design implications and open research directions for the interactive systems community.

digital mental health; user agency; complementary evidence chain; algorithmic veto power; affective computing; self-management

This paper explores the design space for one-minute digital interventions that prompt immediate action without onboarding or sensing. By embracing Fogg’s Behavior Model and four design principles informed by literature, the goal of these interventions was to provide triggers that encourage actions so simple that even people with low motivation would be willing to complete them. We examined the utility of these prompts by conducting a 14-day study with 22 participants interested in making small lifestyle improvements in at least one of three domains: physical activity, healthy eating, and mental well-being. Although we found evidence supporting the intent and design of our one-minute interventions, we also observed that participants encountered four types of barriers that hindered follow-through: temporal, physical, resource, and cognitive friction. When combined with insights drawn from participants’ rewrites of our prompts, our findings suggest that intentional personalization through co-authorship could be a lightweight personalization mechanism that balances relevance with low friction.

Behavioral Interventions Design, Behavior Change, Human-Computer Interaction.

As conversational AI becomes ubiquitous, Virtual Home Assistants (VHAs) are increasingly deployed to support aging-in-place. While the functional utility of these devices is well-documented, their potential to mitigate loneliness among older adults remains contested. This paper presents findings from an eight-week qualitative study examining the domestication of Amazon Echo Dots by older adults living alone. Through post-deployment interviews, we investigate how participants negotiate the device’s utility, social presence, and role in their daily lives. Our findings reveal a complex trajectory of adoption: while participants successfully integrated the VHA as a critical tool for Instrumental Activities of Daily Living (IADLs) and safety (e.g., environmental control for fall prevention), they largely rejected it as a direct remedy for loneliness. Instead of functioning as a companion, the device acted as an asocial proxy, facilitating connections to the outside world while failing to sustain the reciprocal, multi-turn dialogue required for genuine social bonding. We identify a learning runway heavily dependent on intergenerational scaffolding and highlight an interaction gap where the lack of conversational context fractures the illusion of social presence. We contribute a nuanced understanding of how older adults perceive AI companionship, arguing that future designs must prioritize phatic communication and proactive engagement to bridge the gap between transactional utility and emotional support. These insights offer critical design implications for developing age-inclusive conversational interfaces that move beyond command-and-control models to support the psycho-social well-being of older adults.

Virtual Home Assistant, Older Adults, Voice Command, Loneliness, Technology Adoption

While researchers project that social robots may one day be prevalent in the lives of older adults, privacy issues remain underexamined. Social robots can present unique and significant threats to their users' privacy, especially within scenarios involving interpersonal communication. We conducted two exploratory co-design sessions with six older adults to investigate the design of rule-based access control systems for social robots, involving tasks such as 'narrative prototyping.' We analyzed participants' artefacts and discussions from these sessions to identify themes pertaining to: (1) their desire to be active users of robot privacy mechanisms; (2) challenges and opportunities for 'translating' privacy desires into rule-based robot policies; and (3) normative concerns regarding the integration of a robot within the shifting and interpersonal privacy dynamics of old age. We outline high-level design implications following from our analysis; our findings contribute to a growing, relational understanding of what it means to design privacy-conscious social robots for older adults.

social robots, older adults, privacy, access control, co-design

As car cabins evolve with the integration of diverse sensors, traditional car cabins are transforming into smart environments. This shift raises important questions about how privacy is understood and managed in such spaces. In this work, we investigate privacy perceptions from the perspectives of both vehicle owners (i.e., people who purchase and own cars) and non-owners (i.e., people who temporarily use cars, such as family members, friends, or renters). Through semi-structured interviews with eighteen participants, we identified key factors that influence these groups’ views on privacy. Our findings reveal factors that commonly influence privacy preferences for both owners and non-owners, as well as factors that have a stronger impact on one group over the other. Drawing on these insights, we discuss design implications for future designs to better support and balance the diverse privacy needs of multiple stakeholders in smart car cabins.

Privacy, Interview, Smart Vehicle Cabins

Aphasia is a communication disorder that affects millions worldwide, but those affected by aphasia have limited access to in-person therapy. They compensate for this with at-home practice, but existing tools are ineffective. We present Pragmatica, a VR platform that enables people with aphasia to practice their communication skills independently at home through immersive, context-rich activities. In an eight-week case study, we compared Pragmatica with traditional therapy (4 participants per group). While we detected no difference in Quick Aphasia Battery (QAB) scores, our qualitative results show that VR participants engaged in substantial practice (31 hours, 366 activities) and described the VR experience as engaging, fun, and motivating, but had a limited variety of relevant and unique activities. Our study contributes empirical evidence of VR’s feasibility for autonomous language practice, as well as design insights and considerations for accessible, aphasia-friendly VR systems (flexible controls, multi-modal prompts and inputs).

Virtual Reality, Aphasia, Language Therapy, HCI, Accessibility

Virtual reality (VR) offers user experience (UX) practitioners new ways to empathize with users by enabling them to adopt others’ perspectives within immersive environments. To date, prior work has only studied fixed perspectives within a single user group. We created an interaction technique for switching perspectives and compared it with a single-perspective mode across multiple user groups in a simulated hospital emergency room. With a between-participant study (n = 30), we compared three conditions: single-perspective for 1) medical staff, 2) patients, and 3) switching between the two. Our results show that while all conditions fostered affective and cognitive empathy, the switching perspective method demonstrated broad strength in understanding both user groups and the environmental factors influencing user experience. In comparison, both single-perspective methods fostered an in-depth understanding of the respective user groups. Additional outcomes include the importance of visual cues, personal connection, and recommendations for using different perspective-taking methods.

Virtual Reality, User Experience, Perspectives, Design Thinking, Empathizing

As virtual reality hardware becomes more widely available, there is a critical need for pedagogical models that move beyond technical software training and toward better ways to teach human-computer interaction and accessible design for creating virtual experiences. This paper presents a longitudinal study of an undergraduate design studio in which students developed multi-platform, multi-user virtual reality prototypes over the course of a semester-long project. Across two years (the winter semesters of 2024 and 2025), we analyzed the development process of 45 student participants (N2024=26; N2025=19), creating 13 immersive learning experiences for clients in a museum (N2024=7) and new media arts (N2025=6) context. Through reflexive thematic analysis of the first year's 239 developer reflections, researcher observations, and a systematic codebook analysis of over 160 interaction instances, we identify five pedagogical challenges that hinder novice VR development. We subsequently validate these findings against data from the second year participants (N=19) to ensure longitudinal consistency. Primary among these is Platform Bias, a tendency for developers to prioritize desktop-based interfaces over immersive head-mounted display interactions due to development friction, and the Accessibility Gap, where inclusive design is often treated as a secondary optimization rather than a core requirement. We illustrate these findings using two case studies from the 2025 cohort written by the students who developed them: The Journey, which used abstract aesthetics to overcome mobile hardware constraints, and Echoes of the Mind, which implemented novel multisensory brainwave visualizations. To address these pedagogical challenges, we propose five Teaching Considerations that prioritize creator experience and the transparency of the development frameworks. This work contributes to the limited literature on VR pedagogy by providing a data-driven foundation for training the next generation of developers to create usable and inclusive virtual worlds.

virtual reality, human-computer interaction, teaching, WebXR

Computational notebooks are popular for the iterative, exploratory, and narrative work of data science, but their one-dimensional (1D), top-down organization may clash with nonlinearity prevalent in data science work. While evaluations of rigid two-dimensional (2D) notebooks suggest enabling structured nonlinearity has benefits, users may prefer to organize cells in 2D without rigid constraints; freeform 2D notebooks thus warrant research. This work contributes a sanity check by comparing freeform 2D (SAGECells) to standard 1D (Jupyter) on coding and comparing semantically similar sections of code (Branching Code Paths). Freeform 2D with nonlinear cell organizations saved users approximately 30% in time taken to revisit their analysis, and users preferred freeform 2D for all surveyed data science tasks. The results suggest that further research investigating freeform 2D notebooks is warranted.

computational notebooks, data science, design, evaluation, 2D

Text entry in virtual reality (VR) remains challenging due to the limited field of view and the inability to see and reliably locate external physical devices. Existing methods often rely on external hardware or sustained physical actions, limiting accessibility. We present WinkSwype, a novel hands-free, wink-based shape-writing technique where users trace the letters of a word using single-eye gaze. To support this, we developed a single-eye calibration model and a hybrid gesture recognition algorithm optimized for single-eye gaze input. We evaluated WinkSwype against two established techniques: controller-based raycasting and gaze-assisted controller input. Participants retained 58-71% of their text entry speed with WinkSwype while achieving comparable error rates and usability ratings. Subjective feedback indicated no significant increase in workload or visual fatigue during short-term use. These findings suggest its potential as a hands-free text entry method for VR.

Text entry, Gaze pointing, Virtual reality, Gesture typing, Shape-writing

Inattentional blindness occurs when individuals fail to perceive visible elements in a scene because their attention is focused elsewhere. This phenomenon is particularly relevant in Virtual Reality (VR), where subtle scene changes can go unnoticed during task engagement, enabling techniques such as redirected walking and seamless environment manipulation. We investigated how multisensory engagement with a moving object, either by passively following it with gaze alone or by combining gaze with active hand movement, affects users’ ability to detect rapid background changes in a VR scene. A shape identification task in each trial was used to confirm that participants maintained fixation on the moving object throughout. Our results show that participants were generally less sensitive to subtle scene shifts when performing an object displacement task, regardless of sensory engagement modality. However, participants using only gaze were significantly better at detecting larger shifts than those using both hand and gaze. These findings inform the design of immersive VR systems that aim to mask visual changes during task engagement.

Inattentional Blindness, Virtual Reality, Haptics, Change Blindness, Multimodal Interaction