Lego XR

Can tactile real-world play experiences feel right in XR?

(2020) I firmly believe that playful creative tools are among the most promising applications of XR. This space has already been thoroughly explored, but I wanted to extend the discussion to creative experiences for children. It's well known that even toddlers intuitively understand swipe gestures on mobile devices - will we see similarly rapid intuitive understanding of XR interaction?

"LEGO Anywhere" could enable children to build with virtual bricks in any context. Eventually, I envision an AR experience. VR with hand tracking provided by a Leap Motion (opens new window) controller proved sufficient for prototyping.

My question was about translation, rather than replication. Translation meant compensating for the shortcomings of XR, but also harnessing it's opportunities. I compensated by introducing brick auto-snapping and a rotating build plate. The opportunities were exciting; generating infinite similar bricks of any colour by cloning. As a child I wished for that ability.

Turntable

Prototyping with 3D primitives is the closest analogy to UI/UX wireframing. In this case I'm experimenting with a turntable to enable users to rotate the building base.

Dial

I drew inspiration from the physical interfaces designed by Dieter Rams for Braun record players, where the dial is a common control.

Building system

The red line is drawn between the closest points on the bricks. The cyan line is drawn between the closest LEGO grid positions.

Iterative sketching to find the right spatial arrangement and interaction affordances

Build assist

When released, bricks snap to valid grid positions and orientations. I discovered quickly that assistance was required to augment the precision of hand tracking technology.

Colour dial

The colour dial is used to select the active colour. When a brick is hovered near the sphere it initiates a countdown. When complete the active colour is applied.

Clone sphere

The clone sphere has similar behaviour, cloning the held brick when the countdown completes. While prototyping this behaviour I accidentally spawned bricks for hundreds of frames

HoloLens

(2021) My time at Microsoft's Lighthouse Mixed Reality Studio in London has taught me numerous valuable lessons in XR design. Chief among them is the importance of storyboarding when developing complex spatial experiences. They facilitate stakeholder alignment, offer a concrete foundation for discussion and a jumping off point for physical roleplaying and prototyping.

Personally, I have relished the opportunity to dust off my sketching skills, which at one time I used daily, designing similarly tactile play experiences for children.

AirLine

What if learning to code could be fun from the first minute, physically energising and available as a mobile app?

(2022) Parents and educators want a learn-to-code product that intrinsically motivates children aged 8-12. But the incumbents are expensive, often with a daunting learning curve. They typically lack depth, longevity and stickiness.

Turtle Graphics, built in the 1960s, still offers many children their first introduction to coding. I'm investigating whether a modern mobile format, visual block coding, AR and multiplayer can update this proposition and make it compelling for new audiences. Children are gradually introduced to the principles of automation as they explore three levels of interaction:

1. Free draw

Pilot the paper plane (the turtle) in AR by making it follow your finger on the screen

2. Waypoints

Drop 'waypoints' for the plane to follow in sequence

3. Automate

Build a simple computer program that controls its movement

One of the key aspects of the experience is active outdoor play. Piloting the turtle and dropping waypoints both require physical movement - this is not a stand and point experience, the more space to move the better.

My intention is that these simple foundations enable emergent sandbox play, like waypoint hide and seek or maze making. I hope children's curiosity will lead them to play in increasingly sophisticated ways so that unlocking the power of automation becomes a natural next step.

Taking advantage of the 3D context, there is a new turtle command Dive() for rotating in the third axis. In the future I envision pairing a web-based editor with the mobile device, to make it easier for children to write more sophisticated programs using a keyboard. This details the user journey and gameplay loop.

This is a work in progress and more updates will follow.

// Simple turtle program

for (int i = 0; i < 120; i++)
{
    Move(0.5);
    Turn(80);
    SetColor(0, i, i / 2);
    SetSize(i % 3);
    Dive(-40);
}