Research

How can ordinary photos become a 3D scene you can explore from new angles?

Using Gaussian splatting and multilayer image representations, we reconstruct 3D scenes from images and render them from viewpoints that were never captured. One of our methods builds these layers from focal stacks (sets of images focused at different depths) to reduce the effect of noise in individual images. Through inverse rendering, we also decompose a scene into its geometry, materials, and lighting. This works even under strong shadows, so the scene can be relit. Because reconstruction quality depends on the input images, we develop tools that guide users during capture by showing where additional views are needed and where reconstruction errors are likely to occur. We also apply our reconstructions to free-viewpoint visual inspection in industrial settings.

How do people move, and where will they go next?

We estimate human motion using cameras and wearable sensors such as inertial measurement units (IMUs) and pressure-sensing insoles. We incorporate physical constraints so that the estimated motion remains plausible. From first-person (egocentric) video, we estimate the wearer's pose and actions and localize the wearer on a map of the environment. We also forecast pedestrian trajectories, developing methods that cope with noisy detections, limited labeled data, and unfamiliar environments. These techniques can support applications ranging from understanding everyday activities to autonomous systems, such as robots that navigate among people.

What can we see with a camera that captures only changes in brightness?

Event cameras record per-pixel changes in brightness, offering microsecond-level temporal resolution, high dynamic range, and virtually no motion blur. We use event data to estimate depth, deblur images, and reconstruct 3D scenes and human body shapes, in some cases from events alone. We also visualize high-speed phenomena that conventional cameras struggle to capture, revealing details through slow-motion playback. Some of our work combines event data with 3D reconstruction techniques such as Gaussian splatting.

What can cameras tell us about a game and its players?

Sports are fast-paced and crowded, with players constantly blocking one another from view. This makes them a challenging testbed for computer vision. We track players across multiple cameras to maintain their identities despite occlusions, reconstruct players' 3D body poses from footage captured by moving cameras, and estimate ball spin using an event camera. We also reconstruct American football plays in 3D from handheld video. In volleyball, we use an event camera to track the ball and classify player actions in real time, identifying key moments for automatic replay generation. These techniques support performance analysis, training, and broadcasting.

How can computer vision support surgeons and life scientists?

Using cameras built into shadowless surgical lights, we record open surgery and generate occlusion-free views and 4D scenes (3D scenes that change over time) that can be explored from any angle. We have also built a dataset of open-surgery videos recorded with head-mounted cameras, which we use to recognize surgical phases. Beyond the operating room, we estimate skeletal landmarks used in orthodontic diagnosis from facial surface geometry or a single lateral face photograph, aiming to reduce reliance on X-ray imaging. We also detect protein particles in cryo-electron microscopy (cryo-EM) images to support structural biology. Together, these projects apply computer vision to surgical analysis, medical education, orthodontics, and life-science research.

How can machines understand real objects and changing environments?

We study how machines recognize objects through vision and touch, and how they identify parts of 3D objects suited to actions such as grasping or pressing. We also explore how robots can learn to handle fragile foods by practicing on reusable breakable objects instead of real food. Another line of work teaches robots to scoop a target weight of soft materials, such as wet soil or paste. Finally, we detect changes in real-world environments so that 3D maps can be kept up to date without rebuilding them from scratch.