Exoskeleton Glove for Data Collection

Why is this important?

As the world looks at making general purpose robots, we need to collect copious amounts of high quality data to teach the robot how to do things. There’s a variety of different data types out there, where quality and quantity seem inversely proportional. On one end of the spectrum, there’s a lot of videos out there of humans doing tasks, but a human hand differs wildly from a robot gripper contributing to the vast embodiment gap. On the other end of the spectrum is teleoperating the robot, no embodiment gap but prohibitively expensive to collect. Exoskeleton gloves that constrain the human hand to act like the robot gripper sits in the middle of the spectrum, minimizing the embodiment gap while being able to scale data collection more quickly and easily compared to tele-op.

The data recipe problem is largely unsolved in the world of robotics, so exoskeleton gloves as a viable data type is a hypothesis the team is trying to prove out. Other startups (Sunday Robotics, Generalist AI) and research papers (UMI, DexUMI, DexOP) out there have similar hypotheses.

Minimum Viable Product

• Match the kinematics of the chosen OTS 3-finger gripper (degrees of freedom, ranges of motion)

• For each demonstration, collect:

  • Gripper pose (joint angles of each of the degrees of freedom)

  • Wrist camera stream

  • Wrist end effector positioning

• Preserve the inner surfaces of the gripper as much as possible, so the surfaces the data collector is contacting the world with is the same as the gripper

• Ergonomics optimization - data collector should not be hurt in any way or sore because of this

Mechanical Design Iterations

V0 Iteration - Prove Ergonomics / Wearability

• Mental model, try things quickly (design, 3D print, learn, revise, repeat)

• Applied learnings from the DexUMI paper (exoskeleton glove for an off-the-shelf 5 finger gripper) to constrain the human hand to match the kinematics of a common OTS 3-finger gripper

• Created more than 50 design iterations in CREO using 3D printing best practices and design for assembly to refine the ergonomics of the glove

• DexUMI shown below. I’m unable to share the detailed design of the exoskeleton glove I’ve designed, but this is a similar style exoskeleton glove.

DexUMI Exoskeleton

V1 Iteration - Integrating Joint Encoders

• Integrated DexUMI’s joint potentiometers (rotary resistive encoders) but improved on the DexUMI integration by leveraging custom parts that allowed the assembly per joint to be consistent across exoskeleton glove units. I designed a custom steel shoulder screw had a D-shaped screw head that fit into a D-shaped hole in the 3D printed part. This eliminated the per-unit calibration effort that would have been needed as we began scaling data collection.

V2 Iteration - Integrating Magnetic Joint Encoders

• The team began collaborating with a startup, General Intelligence Labs, to finalize the hardware design, synchronize the data streams, and begin data collection. Per General Intelligence Labs’ suggestion, they recommended switching to a magnetic hall effect joint encoder that they had success with in the past, reporting a significant improvement in sensing accuracy. This was a late breaking request so I brainstormed a variety of ways to incorporate the encoder with the intent to minimize thickness stack-up impact. Switching to magnetic joint encoders also meant that my neighboring hardware needed to be non-magnetic.

Hard Problems I Solved Along the Way

Thumb Reconciliation

• Deep dive: Specifically with this gripper design, one of the thumb’s DoFs was being able to rotate about its central axis. Based on where the gripper thumb needed to be, the human thumb would cause the gripper thumb to rotate about its central axis when trying to apply compressive force. 

  • Experiment 1: One way to reduce torque is to get rid of the moment arm (fix the human thumb as close to the axis of rotation as possible). This was uncomfortable to use especially since the human thumb is the shortest of all the fingers.

  • Experiment 2: Physically prevent the human thumb from applying torque and capitalize off the fact that the human thumb cannot rotate in the same way. Create a cavity for the human thumb to sit in to inherently prevent rotation. This worked better and I ended up going this route. 

Nut Loosening Issue

• I learned about proper bolted joint design to ensure the nuts in my assembly joint wouldn’t loosen with time. With the updated design, I’m able to apply enough pre-load on the metal shoulder as opposed to 3D printed plastic, previously. 

• The DexUMI author mentioned that tightening the joints was part of the process for using DexUMI, but I wanted to see if I could find a way to eliminate the inaccuracies and frustrations that come with that problem.

  • Deep Dive: I originally fastened my joints using a custom machined shoulder screw and a locknut. Though, through moderate usage of the glove, the locknut would loosen through vibration. This would introduce more unwanted play into the joint and risk part interference. 

  • Fundamental Problems:

    • A. Screw and nut cannot move relative to each other

    • B. Linkage needs to move freely around the shaft

    • C. Minimize axial misalignment to reduce rattle between finger linkages

  • Ways to Solve A

    • Superglue the screw head

    • Custom screw head and nut retainer part

    • Very strong loctite for nut

    • Double nut strategy

  • Ways to Solve B

    • Sloppy clearance hole

    • Use bushings

    • Use bearings

    • Drill out the 3D print to get a more accurate slip fit

  • I downselected to the following three main strategies:

    • Tighten clearance hole + double nut

    • Insert spacer to ensure that the nut is tightening on metal instead of plastic

    • Custom screw with tightly controlled shoulder length

  • I evaluated these strategies on the how easy it is to assembly, disassembly, and source these materials as well as the effects on the thickness stack-up. After conducting a few experiments by printing the cross section of interest and ordering off-the-shelf parts, I was able to validate the spacer idea. The critical lesson learned was: when the nut clamped down on plastic, it wasn’t able to apply enough preload to prevent vibrations from loosening it with time. The vibrations caused the plastic parts to creep, undoing the bolt preload and backing the nut out with time. This was solved when the nut tightened against metal. Ultimately, the exoskeleton glove is meant to be heavily used to collect data — not needing to re-tighten the joints further reduces friction to collect data

Results

• Preliminary results for training a VLA using 300 demonstrations of exoskeleton glove data shows promising results of being able to pick up a water bottle. Further work underway for how to combine the egocentric data and wrist data in an elegant way. Our team is planning on publishing a paper later this year, stay tuned for it! 

• Exoskeleton gloves provide a very visceral way of understanding the capabilities and limitations of a gripper because you’re controlling it with perfect controls and perception. It is a useful tool to evaluate gripper features, degrees of freedom, geometry prior to having a full hardware assembly ready.

Learnings for Future Designs

• Top down design

  • Setting up coordinate systems well initially will save headache in the future, base coordinate system for end effector, camera coordinate system, so as the camera positioning changes (since that is more iterative), the CAD follows

  • Once sandbox CAD moves to more production CAD, it’s worth it to redesign the assembly with top down design so skeletons control key features

• Validating the data collection stack in reality is paramount especially as you scale data collection.

  • When data quality is critical, teleoperating the gripper with the exoskeleton glove and ideally, replaying the demonstration on the robot are two checks that can validate the collected data’s usefulness. 

  • Having multiple people try collecting data for longer periods of time is also a vital piece of the puzzle in evaluating ergonomics. This helps add features in the right places to maximize comfort and evaluate how much mass is comfortable enough for the data collector to wield.

• Accommodating different hand sizes and closing the embodiment gap seem to be at conflicting ends of the spectrum.

  • I lucked out where my hand size happened to match the gripper embodiment decently well, but people have vastly different hand sizes. Creating an exoskeleton glove that accommodates a majority of hand sizes while minimizing the embodiment gap is certainly a tough problem and one that I’ve been thinking a lot about lately.