Leap motion had the right idea when they recently released a mount for the Oculus Rift. The idea was that you can see your hands from the front of your face and give you a cool way to put your hands inside VR. It was a good idea but due to the limited range of the device fundamentally flawed. Tracking beyond 30cm is just too far away for the LM to handle reliably. Still, it’s a step in the right direction. Where else can you find a solution to track fingers and hands with excellent accuracy for less than $25 (second hand)?
Hand tracking really is key to a more immersive VR experience, a problem that the people at Perception neuron used to garner a sweet $570k on kickstarter recently. But gloves? In 2014? It’s a cool set of kit, but so many parts to break, wires to cut, parts to snag and you look somewhat foolish wearing the full setup. And $200 minimum investment? Ouch!
So what do we really need, at a bare minimum? Being able to see our dominant hand (preferably both) in VR space reliably, if we can track our lower arm accurately we can pretty much track the rest of the arm too. Your elbow is a pretty simple joint, approximating the upper arm isn’t hard if you know where your wrist is and where it’s pointing. The VR guns that are appearing more and more don’t give us that information, but guns are easy, fingers are where it’s hard. Showing your hand and fingers, which 99% of the world automatically look for when they first don a Rift, is really what we’re always going to wanting to do.
The main issue is cost, it’s mostly a solved problem if you have the money, but unless you can do it cheaply no one will adopt it and you’ll end up with early adopter blues. Nintendo powerglove anyone?
So, who wouldn’t want to be able to see their hand and fingers move accurately in front of their face while in the Oculus Rift. Everyone right? Now who wants that for less than $50?
It’s easy if you know what to hack together 🙂
Once again, it’s a PROTOTYPE! It’s not perfect, nothing ever is, but show me how to track your hand in VR, move around and shoot for less than $50 and I (and everyone else) will be very happy! I threw this together in a couple of days, if you like it, tell me! If you don’t like it… well, go make your own bloody controller!! :p
And if Oculus is watching, please give us access to the ‘skeleton’ of the camera in the SDK. We know you’re working on your own controller, but you never know, someone else might come up with a better solution and it will die on the vine because it’s impossible to support easily.
Here is a quick video, some pics, a break down of the hardware and how you can hack the software together to make it work.
Note: The MPU isn’t attached, and the nunchuck isn’t plugged in.
Hardware
The hand tracker is actually quite simple. I’m going to break down each part, what they do and why they’re needed.
Cuff
This is the base of the prototype, everything hangs on the wrist cuff. In its current incarnation it’s a little rough and could be more comfortable. A later version will have foam padding for comfort and a quick release buckle, or velcro, for a snug fit. It will also need a box for the arduino and gyro. Ideally it would also contain a small USB3 hub so the leap motion and the arduino can communicate to the the PC over only one lead. This raises the price a little for convenience. We also have the option of adding a few more buttons to the cuff which can be activated by the off hand.
(the tubes on the side are for support, printed plastic can be quite weak when printed like this, the tubes allow us to insert a 3mm length of filament for strength)
Wii Nunchuck
Why reinvent the wheel? The nunchuck can be found for as little as $3 online, it has a joystick and two trigger buttons as well as a 3 axis accelerometer for simple motion detection. It’s cheap, reliable and easy to replace. It also has a convenient arduino library just waiting to be used.
Ideally we would be able to connect and disconnect two nunchucks, for use in both hands, although using only one is perfectly fine. The cable is long enough to allow use in the ‘off-hand’ so the user could move with one hand and aim/’finger shoot’ with the other. it has a slightly inconvenient proprietary plug but this is easily adapted with a $1 gizmo from ebay. 
Arduino
A fully formed computer for $3? How can we refuse. This forms the brains of the gyro sensors and interprets the nunchuck signals. Using a pro-micro we can also emulate a joystick with no drivers. Handy!
Gyrometer/accelerometer/magnetometer
This tiny miracle on a chip provides a mass of information a thousand times a second. With this we can accurately measure where your wrist is pointing since it’s attached to the Cuff. They’re also only $8 each.
Female Arm
A simple printed arm is attached to the cuff. This provides some simple cable management too. Requires a couple of bolts to attach to the cuff.
Male arm.
Another printed part that can be used to adjust the total distance of the arms. A bolt locks it to the female arm and allows adjustment.
The leap holder.
The fourth and final printed part. This hold the leap motion sensor which will be pointing at the hand and providing constant hand and finger tracking. It also has some holes for wires and needs two bolts to attach to the arm.
Leap Motion
Another miracle in a small package. You can find them second hand online now for $25. It has fairly mature drivers and a SDK for use with games. By mounting it to the wrist we can over come it’s problems with range and free it from the desk. The leap gives a better experience than putting on sweaty gloves for a fraction of the price. It also has zero moving parts, so there is nothing to break.
Ping Pong Ball
This is attached to the end of the leap holder and has a hole inside it. Illuminated from the inside it gives a cheap way to give us positional information. 25c
LED and wires.
This gives the eleventh and last component something to look at in the darkness. <$1.
PS3 Camera (or equivalent).
The camera tells the PC where your arm is in space, just like the Sony Move works. We can use open source software to track it quite easily and if we use an IR filter and Infrared LED inside the ball we can cut a lot of the tracking processing. Available on ebay for $5.
3D Printing Alternatives:
If you don’t have access to a 3D printer you could always make a leap holder from some wood, or even better use friendly plastic. You melt the pellets in boiling water and mould them into shape, this stuff is perfect to make a project like this and it’s very cheap (and reusable).
Bill of Materials
- Printed Cuff
- Printed female arm
- Printed male arm
- Printed Leap Holder
- Arduino $3 (new)
- MPU-9150 $8 (new)
- Wii Nunchuck $3 (new)
- Leap Motion $25 (second hand)
- PS3 Eye camera $5 (second hand)
- Ping Pong ball 25c
- LED + 40cm wire + 10 oh resistor $1
- 5 bolts and washers ~$1
- Extra cuff buttons 25c each (optional)
- Wii nunchuck adapter $1 (optional)
- Cuff Velcro $1 (optional)
- Cuff comfort foam $1 (optional)
- Short pieces of wire to connect the MPU and arduino
- USB lead for arduino (on hand)
- 5 meter USB3.0 extender for the Leap Motion $7
- 5v motor for cuff vibration $1 (optional)
Total : $46.25
or $58 with optional extras.
notes : The Cuff is ideally printed using Nylon since this is more flexible than ABS or PLA and should last longer. The leap can be position to be facing the palm or the back of the hand, although the latter, while more convenient, might be less accurate. It’s also possible that if an optimal position can be found we can reduce the required printed parts to a complete arm and cuff, removing the need for the bolts, and improving the appearance. A USB 2.0 lead can be used for the Leap Motion but this lowers the data speed and may affect accuracy.
Software
This is a work in progress… but we can break it down into 4 distinct areas, all of which have open source software available.
Finger Tracking : Sign up with Leap here for their SDK
Position Tracking : Choose from two open computer vision projects, SimpleCV or OpenCV
Rotation Tracking : It’s a work in progress over at Arduino
Nunchuck libraries : Tim Teatros or check the arduino Site
Some creative people took this idea and turned it around to create LED strips that project an average light reading and display it on the wall behind your TV. Ambilight and lightpack are of limited use, but a cool idea. It didn’t take long for someone to suggest putting one inside an Oculus Rift to increase the perception of a wide FOV but HMD’s are quite a bit too small for this to work well. It also seems Apple has a Patent for this idea but my following suggestions are not the same.
Each of the eight corners are only displaying black. Instead we could cover these areas with translucent plastic that bounces these pixels light out to the edges of the screen in a similar way that fiber optics can relocate light. Instead of wasting this potential light we can add it to the experience and a 110° FOV could be perceived as perhaps 130°, bringing us one step closer to a more immersion.
