All Posts
Astute readers of our first blog post, where we introduced the TR01 hand, will have noticed the strange device that our fingertips are mounted on. Many of them will have probably also guessed its role: yes, it’s a touch sensor.
Many papers have been written about how the human sense of touch works and how important it is to our own dexterity. Just as many debate whether touch sensing is equally important to dexterous robots, and, if so, how to achieve it. Without claiming to have settled these debates, here are our own answers: yes, we believe that touch sensing will be massively beneficial to achieving skilled manipulation, and we need the sensors to be sensitive, rich in information, robust, practical to integrate and use, and to not break the bank.
Introducing Popcorn, Our First Touch Sensor
The Popcorn is the first touch sensor we have integrated into our robot hands. It is designed to sit at the base of the fingertip and provide information about the net contact forces that the fingertip experiences. In that sense, we refer to it not as a traditional “skin” but rather as a touch sensor. It has the following key characteristics:
- Highly sensitive: in realistic settings, subject to noise, wear-and-tear, hysteresis, and sensor-to-sensor variability, the Popcorn can still reliably detect forces as low as 0.1 N.
- Rich in information: the Popcorn provides a set of 14 raw signals that can map onto the full combination of force and torque experienced by the fingertip.
- Robust: we have observed the Popcorn to be highly robust to overload forces, which are easily experienced while learning to manipulate. The Popcorn also separates sensing electronics from the part of the finger that makes contact with the world, allowing the latter to be built from any durable materials we choose, and easily replaced.
- Easy to integrate in hands: the Popcorn is compact and easy to use, providing a direct digital readout with no external amplification electronics. It also places no restrictions on the shape of the fingertip, which can thus be adapted to the task as needed.
- Low latency, high frequency: we read the Popcorn at 250 Hz, and without the latency typically associated with camera-based tactile sensors.
- Easy to manufacture and low-cost: the Popcorn only uses commonly available and low-cost components, and can be manufactured without any complex equipment, based on externally sourced PCBs and flexures.
Basically, while transduction methods are interesting in and of themselves, we are manipulation people: we care about all the unsung aspects that make a touch sensor actually practical for integration into a hand and usable at scale for manipulation. The Popcorn checks all those boxes for us. In many manipulation labs, tactile sensors are precious, delicate things that are a pain to build into your hands and that you think twice before using lest they break. In contrast, we believe that touch sensors should be as easily available to builders of robot hands as, well, popcorn.
How it Works: Light as a Touch Transducer
The first fundamental choice we made was to use light sensors inside a robot finger to sense displacement of a soft element due to contact, and then equate that displacement to a touch signal. We do this because light is a phenomenal transduction mechanism: it provides exceptional signal-to-noise ratio with much simpler manufacturing and cheaper components than alternative methods.
The next key design choice, leading directly to the Popcorn, was that the first embodiment of this technology we would integrate in our hands would not be a “skin” covering the entire fingertip, but rather a sensor sitting at the base of the fingertip, giving us the overall contact forces applied by it.
This approach has obvious advantages. Fully sensorized fingertips tend to be bulky as they need to accommodate the transduction mechanism. Furthermore, once a fully sensorized fingertip is designed, its shape is hard to change. In contrast, the Popcorn can take any size and shape fingertip, including slender ones for manipulation of small parts. A damaged fingertip costs pennies to replace, since it contains no electronics.
There are clear parallels here to a different class of sensor that has been quietly doing the job for decades in industrial robotics: the good old six-axis force/torque sensor. In fact, there is previous work in robotic manipulation using F/T sensors embedded at the base of the fingertip. However, unlike industrial F/T sensors, which work extremely well but are designed for factory arms — large, expensive, with bulky amplification electronics — ours is meant for robot hands: compact, low-cost, robust to overload, with direct digital readout and no external amplification.
The optical signals from our sensor are not strictly forces, but a representation of the induced displacement. A learned model, trained on labeled data collected with a reference force/torque instrument, can map these raw readings to a full 6-axis force and torque reading, or to contact location and normal force — representations that are physically grounded, easy to reason about, and natively compatible with any modern physics simulator. (That last point matters: when training learned manipulation policies, the consistency between simulated and real sensor data can have a large effect on how well sim-to-real transfer works.) However, we still expect to train end-to-end visuomotor policies on the raw sensor data, not a learned intermediate representation.
This does not mean we have given up on the concept of tactile skin with full fingertip coverage. In the future, we might still experiment with fully sensorized fingertips, which do provide their own advantages over F/T sensors sitting at the fingertip base, such as information on contact area or contact pressure distribution. That said, the Popcorn gives us a lot of what we need to work with, for a fraction of the cost and integration effort.
What’s Next: Autonomous Visuotactile Manipulation
Force sensing alone does not make a robot hand dexterous, but we believe it is an important piece of the puzzle. Modern visuotactile policies, which combine visual perception with contact force information from each finger, are beginning to unlock manipulation capabilities that were utterly out of reach just a few years ago. This is one of those cases where getting the hardware and sensors right quietly determines how far the software can go.
We are collecting teleoperation data of complete manipulation tasks with touch sensing included. Stay tuned for upcoming posts on how this data will help close the loop on autonomous visuotactile manipulation.
Further Reading
Background and History
The Popcorn is based on an optical transduction approach for tactile sensing first developed at Columbia University, where members of our team built a robotic fingertip capable of sub-millimeter contact localization over complex, multicurved surfaces, and then showed, for the first time, that it enables in-hand object manipulation without any cameras, based exclusively on touch and proprioception. Our academic prototype of a dexterous hand integrating this technology was named one of Time Magazine’s Best Inventions of 2023. It then led directly to the very first Popcorn, which applied this transduction method for net contact force detection at the base of the fingertip. While the current version of the sensor is improved in many ways, a lot of the original DNA remains.
Technical Details
The Popcorn is an optical sensor consisting of two rigid plates — a top and a bottom — separated by a small gap and connected with carefully designed flexures, each carrying a custom PCB populated with infrared LEDs. Some LEDs act as emitters; others, on the opposite plate, act as receivers. At rest, each emitter illuminates its facing receivers with a known baseline intensity. When a contact force is applied to the finger (nominally connected to the top plate), the two plates shift relative to one another, changing the light intensity each receiver sees. The resulting sensor outputs 14 raw optical signals, streaming continuously at 250 Hz.
A key design choice — and a somewhat surprising one — is using LEDs as both emitters and receivers. LEDs used as photodetectors exhibit significantly higher signal-to-noise ratio than conventional photodiodes in this application, and their narrower viewing angle turns out to be a feature rather than a limitation: it makes the signals more sensitive to small lateral displacements, exactly what we want. In ideal conditions, this allows the sensor to detect relative plate displacements on the order of 10 microns.
We chose LEDs that work in the IR spectrum in order to minimize interference from ambient light. (LEDs are exceptional photodetectors only in the same band that they emit in.) Modern indoors lighting has virtually no IR component, and, indeed, we have observed very little interference. However, daylight can still have IR components, and small amounts of the light emitted by the sensor itself can also bounce back off the environment and show up as unwanted signal. To avoid both these phenomena, we wrap the sensor in a flexible outer shell made of TPU plastic that we have observed to have excellent IR absorption. While the Popcorn still works well without it, this outer shell virtually eliminates drift on larger timescales.
The mechanical interface between the two plates is critical. We need this interface to exhibit just the right level of compliance: too soft and it will introduce parasitic effects in high-speed manipulation, too stiff and it will decrease sensitivity. We also want as little hysteresis as possible, at both short time scales (seconds, touch-to-touch) and long time scales (hours or days). In fact, hysteresis at various time scales is often the secret weakness of many touch sensors, and we put a lot of effort into getting a handle on it. The result is a mechanical interface based on precision Titanium flexures, which significantly simplifies fabrication, reduces unit-to-unit variability, and produces more consistent sensor behavior.
As the sensor sits at the base of the fingertip structure rather than within the contact surface itself, it does not constrain fingertip design. The material, geometry, and stiffness of the fingertip can be chosen freely for different use cases — a compliant, high-friction surface for delicate grasps, or a stiffer, more geometrically precise tip for tasks that require it. For now, we went with a multi-material fingertip, with a 3D-printed TPU “skeleton” covered in a molded silicone rubber “flesh”. Adding a second layer of skin-level tactile sensing — for contact localization or pressure mapping — remains an option that this architecture leaves open.
Limitations
The Popcorn checks many of our boxes in terms of both performance and practicality for manipulation, and we are counting on it for our first examples of dexterity. However, areas of improvement remain. The “skin” approach has its own advantages, in terms of being able to identify complex contact areas or pressure distributions, which the Popcorn cannot do. The Popcorn is certainly compact enough for integration into human-size fingers, but it still has some size, and smaller is always better in this context. Finally, while our flexures do a good job of increasing repeatability, some amount of load/unload hysteresis and baseline shifts (especially after applying large forces) can still be observed; we have found that an auto-taring routine is effective at re-zeroing the sensor after contact.
