Massless: Backstory
2010–11: An unshakable idea
On Friday nights the Theoretical Physics department often descended on one of the nearby historic Oxford pubs. Inevitability, I’d ask:
“Wouldn’t you rather use a computer in 3D with your hands? Like in Minority Report?”
Something unusual was happening. They agreed. I took the question to a wider audience: some thought this already existed, others thought it was impossible, they all were intrigued. I got the feeling I was on to something for the first time…
2012–13: A life in focus, elements & the world in motion
I was researching the creation of microscopic machines and methods to sequence DNA under the exceptional Physicist Prof. Ramin Golestanian [Thesis]. I wrote simulations from scratch, ran them on a cluster of hundreds of computers and analysed the data using custom analysis and visualization software that was graphical, animated, interactive and three dimensional. Visitors to the department would see my visualizations and exclaim “You should work for Pixar!”.
Working in my small apartment outside of the city center, I’d wake up, open the laptop or notepad and continue writing code or mathematics for my thesis. This would continue all day, then I’d sleep, and repeat. This went on for around 6 months and was an intense version of a practice I had performed for most of my life.
I felt extreme frustration in the process of interacting with the digital world. I had everything in my mind, knew how to do it, but the process of expressing it from my mind to the screen was too slow!
This legacy device with a nonsensical arrangement of letters, pushing around a single point within a small rectangular frame housing a cascade of too many windows. I don’t want to be chained to a desk living a sedentary existence, I want to move when I work, I want the entire space around me to be a digital medium to create. I want to minimise the process involved in sharing the three-dimensional mental visualizations I had with others.
There has to be a better way. What would that look like?
Three elements for three-dimensional computing
I came up with three elements needed for spatial computing:
- Stereoscopic head-mounted displays that are low cost and high quality.
- Three-dimensional input that is precise, and reliable.
- A software development framework for persistent and collaborative spatial applications
I came to a sudden realization. Given my skills, interests, and experience so far the best way I can serve the world is not purely through physics, but through creating a better way to work in the digital world. To create tools to enhance creativity and problem-solving. It felt like my life had come into focus.
I wasn’t the only one intoxicated by this idea…
In late 2012, Oculus launched its Kickstarter. I remember the rush of adrenaline when I heard — we have one of the first pieces! In 2013 Leap Motion launched its Kickstarter. “This is it!” I thought. “We are close…”
The Oculus DK1 blew me away, the potential was clear. Leap Motion is incredibly impressive, but it suffers from the line-of-sight limitation that all camera-only based hand tracking based systems have. It is very common for a single hand to block parts of itself from the camera or for one hand to block the other. This limits its use cases to broad gestures but not precise, reliable interaction across the entire interaction phase space. The only way you can overcome this is if you image through the body, for instance with a Terahertz Camera seen at airport security, or to place a motion sensor on the fingertips to pick up the tracking when the cameras fail.
2013–14: Birds, fish, particles, prototypes, name & money
At this point, in 2013, I was studying in a field of physics called active matter. This field applies the principles of physics to living systems such as the flocking of birds and the schooling of fish. I had previously read a paper of a research group that tracked 1000s of Starlings flocking over Rome using two cameras at either side of a building. So, I knew about stereo camera tracking, calibration, and reconstructing 3D positions. I had also taken over a project to track the schooling of fish in a tank using an infrared background and cameras, so I knew about infrared tracking, image processing, and computer vision. My thesis involved tracking the path of microscopic particles suspended in a fluid to understand the statistics of their motion, so I knew how to track particles very precisely.
It always delights me how the tools of physics and mathematics can be applied to such seemingly irrelevant areas with great success. This a field of its own know as Complexity Science and I was working under Prof. Matthew Turner who is a master of this art.
I got two USB cameras, modified them to detect infrared, 3D printed some mounts and screwed them to an aluminum beam. I soldered an IR LED to an IMU motion sensor, connected it to an Arduino with Bluetooth communication and mounted that to a 3D printed support structure to be worn on your hand. I wrote the image processing and computer vision code needed to track in 3D, and for the first time, I wrote my name in three dimensions. I had to move very slowly because the cameras were not synchronized in time and the data was post-processed so there was no real-time display — but it worked. I wrote in 3D!
What’s in a name?
To make it real we need a name that captures everything this represents. I was talking with a good friend of mine from Oxford, Irwin Zaid. I was thinking of “non-zero mass” because the massless things we are creating in VR are made by physical devices that have mass. He said “That’s too complicated, no one will understand. Call it Massless”.
Here are three reasons why Massless is the perfect name:
- The things we create in VR look real but aren’t made of matter, they have no weight or mass. Content is Massless.
- Information itself has no mass. You may store lots of information in your RAM or hard drive. But regardless of how much you store the mass doesn’t change. Information is Massless.
- The way that VR/AR works is by displaying photons (light) to your eyes. Photons are massless particles. VR/AR is Massless.
Funding
I did what comes naturally to academics and in May 2014 I submitted a grant proposal to Innovate UK for the Massless Hands. They rejected my proposal later that year on the basis that there is no market for hand tracking. I started to question whether starting with Massless Hands was a good idea.
Read “Massless: Five years of spatial computing” for what happens next…
