BACKGROUND

Craft learning has expanded from traditional in-person instruction to various forms of remote and self-directed learning and practice, allowing a broader audience to explore craft at different skill levels.

However, achieving mastery remains difficult due to the challenge of acquiring tacit knowledge:

complex and nuanced coordination of body, material, and tools (somatic);
reflection on ”critical incidents” in complex craft processes characterized by “workmanship of risk” (relational);
the social dynamics of instructor-learner models in studio environments (collective).

RESEARCH QUESTIONS

What does the interplay between AI-augmented Mixed Reality systems and embodied craft practices reveal about the evolving roles of systems, learners, and practitioners?

(RQ 1) How does the interplay between AI-augmented MR technology and the embodied nature of craft learning influence the design of instructions?
(RQ 2) How do novice and experienced ceramicists perceive the system’s ability to support craft learning, and how do they envision its role in future practice?
(RQ 3) What broader impacts emerge as AI-MR systems and craft practices co-evolve to challenge existing roles and processes in craft practice?

METHOD

Research-through-Design. — To explore our research questions, we adopted a Research-through-Design approach comprising three phases: formative study, system design, and user study.

Formative Study

On-site study of the ceramic learning process. — On-site study of the ceramic learning process: (a) Outcomes from our immersive ceramic learning in the studio; (b) Interaction between learners and instructors during an ethnographic study session.

Schematic diagram of each step in wheel-throwing. — Schematic diagram of each step in wheel-throwing: The entire process is divided into the following steps: (1) Setup: Anchoring the clay. (2) Centering: Ensuring the clay is balanced on the wheel. (3) Opening: Creating the initial cavity. (4) Pulling Up: Raising the walls to the desired height. (5) Shaping: Creating specific shapes, such as the neck for vases. (6) Finishing: Removing the completed piece from the wheel.

System Design

The ceramics guiding system comprises two main components. — The ceramics guiding system comprises two main components: hardware and application. The hardware includes a pottery wheel, a webcam, and a Quest 3 headset as the display device. The software modules consist of a Python script for processing the detected shape and generating instructions via the OpenAI API, a piggybacked XRHand package for gesture recognition and guidance, and C# scripts for managing the learning process, including backend logic and the frontend user interface.

Novice system’s UI and functionality. Left: UI diagram for novices: (1) Instruction panel displaying all text-based instructions for the current step. (2) Hand and clay holograms for reference and imitation. (3) A progress bar to track learning progress. (4-6) Optional panels for video playback, tips, and voice command listings. Right: In-situ demonstration of system functionality in headset: (a) Gesture imitation with text-based instructions. (b) Video and tips-based guidance. (c) Rule-based correction using hands and tools. (d) Summary with scores and suggestions for the next session.

Experienced system’s UI and functionality. Left: UI diagram for experienced users: (1) Instruction panel displaying all text-based instructions. (2) Optional hand and shape holograms for skill refreshment. (3) Optional shape comparison panel to track the current clay shape. (4) Optional shape score bar indicating progress. (5) Optional panel displaying all available voice commands. Right: In-situ demonstration of system functionality in headset: (a) Practice goal and reference panels; (b) Recalled gesture hologram for skill review; (c) Multimodal suggestions with text, audio, and holograms; (d) Color-coded shape guidance.

FINDINGS

Theme	Sub-theme	Description
Tensions Between Virtual Instruction and Physical Environment	Video and Hologram as a Compound View for Embodied Craft Learning in MR	Gesture holograms supported understanding of hand actions. Shape holograms helped with goal identification and progress tracking. Holograms lacked detail and felt overly mechanical. Videos provided richer detail and tacit cues for novices and skill refresh for experienced users. Cross-referencing videos and holograms supported spatial understanding and error correction.
	MR as Both Assistance and Obstacle in Physical Environments	MR enabled direct engagement with materials beyond screen-based instruction. Novices felt urged to interact with holograms, sometimes damaging in-progress work. Participants struggled to perform precise actions while interpreting spatial instructions.
System Workflow’s Impact on Craft Knowledge Transfer	Immersion Is Beneficial but Constrained by Skill Discrepancies	Step-by-step workflow was immersive but fragile to real-world mismatches. Fixed prompts and criteria were difficult to adapt to individual progress.
	Autonomy for Craft Knowledge Acquisition	Autonomy allowed participants to customize progress. Novices gained freedom for trial and error but risked skipping critical steps.
Need for In-Situ and Personalized Instructions Beyond Shape-Based Feedback		Instructions lacked personalization and contextual grounding. Feedback focused too narrowly on shape-based evaluation. Participants desired more proactive guidance.
Perception on System’s Roles and Use Scenarios in Craft Learning	When and How to Use the System	Valuable for early-stage learning and skill development. Well suited for hybrid learning and post-instruction practice. Enabled scalable, personalized instruction for instructors. Acted as a mediator between learners and instructors. Potential uses include social activities, professional training, and production assistance.
	Comparison Between the System and Human Instructors	Excelled as a knowledge repository supporting asynchronous instruction. Lacked ability to convey tacit knowledge, physical intervention, emotional support, and adaptive responses.
Improvisation Is Limited by the Nature of Wheel Throwing and Skill Levels		System offered limited space for improvisation across skill levels. Novices improvised unconsciously when struggling with steps. Experienced users felt constrained by task rigidity, habits, perfectionism, and physical limits.

Theme	Sub-theme	Description
Insights from System Design: Effective Strategies and Opportunities for Refinement	Designing for Craft Learning with Immersion in MR: Restoration, Reconstruction, and Augmentation	Restoration: Interprets and organizes real-world instructions. Reconstruction: Supports user-controlled replay. Augmentation: Enables compound views and gamification.
	Designing Instructions in MR: Detail and Hierarchy	Provide detailed, context-aware instructions. Organize instructions hierarchically, drawing on active and passive studio workflows. Opportunities include real-time feedback at critical moments, metric visualizations, collaborative learner input, and intent-based adaptive AI support.
	Designing More Effective Communication Features in AI-MR: Modalities and Emotion Support	Use metaphor-based language, coherent multimodal guidance, and emotional feedback. Opportunities include AI-generated context-specific instructions, diverse gesture datasets, and systematic evaluation of emotional feedback in AI-MR interactions.
Insights Beyond Design Expectations: Emerging Patterns and Implications for Future Design	Designing Spatial and Motor Experience in MR: Instruction Distribution and Body Engagement	Hologram placement interfered with practice, while limited field of view and physical constraints hindered instruction. Integrate haptic feedback and structured spatial design to support crafts involving full-body movement.
	Designing for Personalized Craft Learning in AI-MR: Bridging Skill Differences and Growth	Balance adaptive personalization with standardized processes. Leverage AI to address horizontal skill gaps and support vertical development in tacit knowledge and design judgment.
AI-MR Systems Beyond Learning: Creation and Collaboration in Craft Making	Towards a Creative Craft Support System	Support the shift from unconscious to conscious improvisation to foster practitioner growth. Use AI to detect and encourage meaningful deviations as guided experimentation for creation.
	Evolving Roles and Speculative Applications of AI-MR in Craft Practice	Shift from human–human instruction to human–agent collaboration as AI empowers MR systems with greater agency. Position the system as a collaborator with variable agency across educational, professional, and leisure contexts.

Reshaping Craft Learning: Insights from Designing an AI-Augmented MR System for Wheel-Throwing