Less material is better when you’re waving your arms around, so I did a quick one piece redesign on the Leap holder. Since I’m actually using the USB lead to hold the leap in place it makes sense to remove the sides of the holder, although it would be a much better idea to go ‘belts and braces’ and use some velcro to ensure the Leap isn’t going anywhere when you’re moving around….
The new design has the disadvantage of losing it’s adjust-ability but it looks so much better, should be lighter and can’t move out position. A new cuff connector will also be needed. It would also be great to completely hide any wires although this will require placing some tricky holes in strategic places….
Leap motion had the right idea when they recently released a mount for the Oculus Rift. The idea was that you can see your hands from the front of your face and give you a cool way to put your hands inside VR. It was a good idea but due to the limited range of the device fundamentally flawed. Tracking beyond 30cm is just too far away for the LM to handle reliably. Still, it’s a step in the right direction. Where else can you find a solution to track fingers and hands with excellent accuracy for less than $25 (second hand)?
Hand tracking really is key to a more immersive VR experience, a problem that the people at Perception neuron used to garner a sweet $570k on kickstarter recently. But gloves? In 2014? It’s a cool set of kit, but so many parts to break, wires to cut, parts to snag and you look somewhat foolish wearing the full setup. And $200 minimum investment? Ouch!
So what do we really need, at a bare minimum? Being able to see our dominant hand (preferably both) in VR space reliably, if we can track our lower arm accurately we can pretty much track the rest of the arm too. Your elbow is a pretty simple joint, approximating the upper arm isn’t hard if you know where your wrist is and where it’s pointing. The VR guns that are appearing more and more don’t give us that information, but guns are easy, fingers are where it’s hard. Showing your hand and fingers, which 99% of the world automatically look for when they first don a Rift, is really what we’re always going to wanting to do.
The main issue is cost, it’s mostly a solved problem if you have the money, but unless you can do it cheaply no one will adopt it and you’ll end up with early adopter blues. Nintendo powerglove anyone?
So, who wouldn’t want to be able to see their hand and fingers move accurately in front of their face while in the Oculus Rift. Everyone right? Now who wants that for less than $50?
It’s easy if you know what to hack together 🙂
Once again, it’s a PROTOTYPE! It’s not perfect, nothing ever is, but show me how to track your hand in VR, move around and shoot for less than $50 and I (and everyone else) will be very happy! I threw this together in a couple of days, if you like it, tell me! If you don’t like it… well, go make your own bloody controller!! :p
And if Oculus is watching, please give us access to the ‘skeleton’ of the camera in the SDK. We know you’re working on your own controller, but you never know, someone else might come up with a better solution and it will die on the vine because it’s impossible to support easily.
Here is a quick video, some pics, a break down of the hardware and how you can hack the software together to make it work.
Note: The MPU isn’t attached, and the nunchuck isn’t plugged in.
The hand tracker is actually quite simple. I’m going to break down each part, what they do and why they’re needed.
This is the base of the prototype, everything hangs on the wrist cuff. In its current incarnation it’s a little rough and could be more comfortable. A later version will have foam padding for comfort and a quick release buckle, or velcro, for a snug fit. It will also need a box for the arduino and gyro. Ideally it would also contain a small USB3 hub so the leap motion and the arduino can communicate to the the PC over only one lead. This raises the price a little for convenience. We also have the option of adding a few more buttons to the cuff which can be activated by the off hand.
(the tubes on the side are for support, printed plastic can be quite weak when printed like this, the tubes allow us to insert a 3mm length of filament for strength)
Why reinvent the wheel? The nunchuck can be found for as little as $3 online, it has a joystick and two trigger buttons as well as a 3 axis accelerometer for simple motion detection. It’s cheap, reliable and easy to replace. It also has a convenient arduino library just waiting to be used.
Ideally we would be able to connect and disconnect two nunchucks, for use in both hands, although using only one is perfectly fine. The cable is long enough to allow use in the ‘off-hand’ so the user could move with one hand and aim/’finger shoot’ with the other. it has a slightly inconvenient proprietary plug but this is easily adapted with a $1 gizmo from ebay.
A fully formed computer for $3? How can we refuse. This forms the brains of the gyro sensors and interprets the nunchuck signals. Using a pro-micro we can also emulate a joystick with no drivers. Handy!
This tiny miracle on a chip provides a mass of information a thousand times a second. With this we can accurately measure where your wrist is pointing since it’s attached to the Cuff. They’re also only $8 each.
A simple printed arm is attached to the cuff. This provides some simple cable management too. Requires a couple of bolts to attach to the cuff.
Another printed part that can be used to adjust the total distance of the arms. A bolt locks it to the female arm and allows adjustment.
The leap holder.
The fourth and final printed part. This hold the leap motion sensor which will be pointing at the hand and providing constant hand and finger tracking. It also has some holes for wires and needs two bolts to attach to the arm.
Another miracle in a small package. You can find them second hand online now for $25. It has fairly mature drivers and a SDK for use with games. By mounting it to the wrist we can over come it’s problems with range and free it from the desk. The leap gives a better experience than putting on sweaty gloves for a fraction of the price. It also has zero moving parts, so there is nothing to break.
Ping Pong Ball
This is attached to the end of the leap holder and has a hole inside it. Illuminated from the inside it gives a cheap way to give us positional information. 25c
LED and wires.
This gives the eleventh and last component something to look at in the darkness. <$1.
PS3 Camera (or equivalent).
The camera tells the PC where your arm is in space, just like the Sony Move works. We can use open source software to track it quite easily and if we use an IR filter and Infrared LED inside the ball we can cut a lot of the tracking processing. Available on ebay for $5.
3D Printing Alternatives:
If you don’t have access to a 3D printer you could always make a leap holder from some wood, or even better use friendly plastic. You melt the pellets in boiling water and mould them into shape, this stuff is perfect to make a project like this and it’s very cheap (and reusable).
Bill of Materials
- Printed Cuff
- Printed female arm
- Printed male arm
- Printed Leap Holder
- Arduino $3 (new)
- MPU-9150 $8 (new)
- Wii Nunchuck $3 (new)
- Leap Motion $25 (second hand)
- PS3 Eye camera $5 (second hand)
- Ping Pong ball 25c
- LED + 40cm wire + 10 oh resistor $1
- 5 bolts and washers ~$1
- Extra cuff buttons 25c each (optional)
- Wii nunchuck adapter $1 (optional)
- Cuff Velcro $1 (optional)
- Cuff comfort foam $1 (optional)
- Short pieces of wire to connect the MPU and arduino
- USB lead for arduino (on hand)
- 5 meter USB3.0 extender for the Leap Motion $7
- 5v motor for cuff vibration $1 (optional)
Total : $46.25
or $58 with optional extras.
notes : The Cuff is ideally printed using Nylon since this is more flexible than ABS or PLA and should last longer. The leap can be position to be facing the palm or the back of the hand, although the latter, while more convenient, might be less accurate. It’s also possible that if an optimal position can be found we can reduce the required printed parts to a complete arm and cuff, removing the need for the bolts, and improving the appearance. A USB 2.0 lead can be used for the Leap Motion but this lowers the data speed and may affect accuracy.
This is a work in progress… but we can break it down into 4 distinct areas, all of which have open source software available.
Finger Tracking : Sign up with Leap here for their SDK
Rotation Tracking : It’s a work in progress over at Arduino
While I’m putting together the prototype I thought I would share some of the potential details.
This can easily be used on both hands, although dominant hand for weapons/world manipulation and left hand for movement and utility would be perhaps more practical.
Full dominant arm tracking in six degrees of freedom.
Full finger and hand tracking, sub millimetre. 6DOF. No glove, no moving parts.
Analogue joystick for off hand, two buttons. 3DOF gyro included.
Extra utility buttons available.
Buildable with a 3D printer in a couple of hours, at the moment I’m on 4 printed parts. Assembly should be under an hour.
I will probably release it under a non commercial open source licence, on github, software and hardware, which gives me the option of kickstarting the hardware for people without 3d Printers.
All the electronics are readily available and available now, I will start figuring out the software side if people show some enthusiasm.
Total cost (without the 3D printed parts) : Under $50 (about $40 is doable if you find the right auctions).
I’m sure a few clever people can figure out what I’m doing 🙂
Remember, Palmer didn’t technically invent anything with the Rift, he put together existing hardware, added some code and reinvented the world. This is the same (on a lesser scale), finger/hand/arm/weapon tracking in 6DOF for $40? What’s not to like?
I’ll post some pictures and a video soon.
To track my fingers I need to first track a circle…. Seek and ye shall find
15 lines of code in SimpleCV (computer vision)….
I wonder if this is how Trinity Magnum got started…. I think I better get a PS3 camera from ebay, otherwise I have everything I need already. The PS3 camera is really cheap and has a superb frame rate for very fast tracking, it’s also easy to filter for IR so it makes seeing IR illuminated objects so much easier and more accurate.
Oculus Connect was pretty good but I was really hoping for a input device reveal. I think a lot of other people were too since we’re in danger of some really expensive solutions becoming the ‘standard’. Stem and PrioVR are pretty cool, but not only expensive but massive overkill or underkill for VR input. The solution that will win will be the one that lets you see your hands and tracks fingers. Being able to hold a ‘gun’ would also be very useful. Anyway, I have a few ideas to achieve this, and cheaply, which is always good.
The seed was planted long ago 🙂
Just like the Rift the best way to do something new is to put together existing technology, open source it and let people play with the idea. I’m printing parts of a ‘proof of concept’ now, which also means it should be pretty easy for people to try the idea and improve it.
Who wouldn’t want to print 4 simple parts, put together some simple electronics, install a program and have hand/finger tracking for less than $100?
Can anyone point me in the direction of a IR camera tracking solution on github….?