publications

2024

1. EgoVLA

   EGOVLA: Learning Vision-Language-Action Models from Egocentric Human Videos

   CVPR 2025. Under Review , Nov 2024

   Bib

   @article{egovla,
     title = {EGOVLA: Learning Vision-Language-Action Models from Egocentric Human Videos},
     journal = {CVPR 2025},
     year = {2024},
     month = nov,
   }

2. Orbit-Surgical

   

   ORBIT-Surgical: An Open-Simulation Framework for Accelerated Learning Environments in Surgical Autonomy

   IEEE ICRA 2024. More Information can be found here , Apr 2024

   Abs Bib

   

   Physics-based simulations have accelerated progress in robot learning for driving, manipulation, and locomotion. Yet, a fast, accurate, and robust surgical simulation environment remains a challenge. In this paper, we present ORBIT-Surgical, a physics-based surgical robot simulation framework with photorealistic rendering in NVIDIA Omniverse. We provide 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR) which represent common subtasks in surgical training. ORBIT-Surgical leverages GPU parallelization to train reinforcement learning and imitation learning algorithms to facilitate study of robot learning to augment human surgical skills. ORBIT-Surgical also facilitates realistic synthetic data generation for active perception tasks. We demonstrate ORBIT-Surgical sim-to-real transfer of learned policies onto a physical dVRK robot.

   @article{orbit-surgical,
     title = { ORBIT-Surgical: An Open-Simulation Framework for Accelerated Learning Environments in Surgical Autonomy},
     journal = {IEEE ICRA 2024},
     year = {2024},
     month = apr,
     url = { https://orbit-surgical.github.io/ },
   }

2023

1. 

   Robot-Assisted Vascular Shunt Insertion with the dVRK Surgical Robot

   JMRR 2023. More Information can be found here , Nov 2023

   Abs

   

   Vascular shunt insertion is a common surgical procedure performed to restore blood flow to damaged tissues temporarily. It usually requires a surgeon and a surgical assistant. We consider three scenarios: (1) a surgeon is available locally; (2) a remote surgeon is available via teleoperation; (3) no surgeon is available. In each scenario, a minimally invasive surgical-assistant da Vinci robot operates in a different mode either by teleoperation or automation. Robotic assistance for this procedure is challenging due to precision and control uncertainty. The role of the robot in this task depends on the availability of a human surgeon. We propose a trimodal framework for vascular shunt insertion assisted by a da Vinci Research Kit (dVRK) robotic surgical assistant (RSA). To help further study for the community, we also present a physics-based simulated environment for shunt insertion built on top of the NVIDIA Isaac …

2. IsaacLab

   

   ORBIT: A Unified Simulation Framework for Interactive Robot Learning Environments

   IEEE RA-L 2023. More Information can be found here , Apr 2023

   Abs Bib

   

   We present ORBIT, a unified and modular framework for robotics and robot learning, powered by NVIDIA Isaac Sim. It offers a modular design to easily and efficiently create robotic environments with photo-realistic scenes, and fast and accurate rigid and soft body simulation. With ORBIT, we provide a suite of benchmark tasks of varying difficulty- from single-stage cabinet opening and cloth folding to multi-stage tasks such as room reorganization. The tasks include variations in objects’ physical properties and placements, material textures, and scene lighting. To support working with diverse observations and actions spaces, we include various fixed-arm and mobile manipulators with different controller implementations and physics-based sensors. ORBIT allows training reinforcement learning policies and collecting large demonstration datasets from hand-crafted or expert solutions in a matter of minutes by leveraging GPU-based parallelization. In summary, we offer fourteen robot articulations, three different physics-based sensors, twenty learning environments, wrappers to four different learning frameworks and interfaces to help connect to a real robot. With this framework, we aim to support various research areas, including representation learning, reinforcement learning, imitation learning, and motion planning. We hope it helps establish interdisciplinary collaborations between these communities and its modularity makes it easily extensible for more tasks and applications in the future.

   @article{orbit,
     title = {ORBIT: A Unified Simulation Framework for Interactive Robot Learning Environments},
     journal = {IEEE RA-L 2023},
     year = {2023},
     month = apr,
     url = {https://isaac-orbit.github.io/},
   }