Robotics and Autonomous Systems

Projects focus on developing socially assistive robot (SAR) systems based on AI/machine learning and capable of perceiving and understanding user activity, goals, and affect in real time, from multimodal and heterogeneous data (video, audio, physiologic sensor wearables), aiding the user through social interactions that combine monitoring, coaching, motivation, and companionship, and personalizing and adapting to the user within an interaction, between interactions, and over time in order to support health, education, and training contexts.

Sample Project: Developing SAR software: The student will work with PhD students involved in developing a socially assistive robot for use by university students in college dorms, to facilitate their practice of emotion-regulation skills such as cognitive behavioral therapy, mindfulness, etc., in order to mitigate anxiety and depression. The student will aid in gathering relevant literature both in robotics and in social science and behavioral science, aid in the development of software for the user interfaces (robot, tangibles, screens, web) and behavioral interventions, as well as multimodal data analysis and model learning. This applies to various projects in the PI’s lab, with different user populations: children with autism, children with CP, K-12 students, university students, adolescents/young adults, stroke patients, elderly with dementia, and healthy / typical users.

Expected Background: The student needs some level of Python or other programming experience.

Relevant projects focus on developing networked multi-robot systems capable of communicating among themselves and unattended, immobile, wireless sensors.

Sample Project: Developing software for aerial robots: The student will work with PhD students involved in developing teams of autonomous aerial robots. The student will work on gathering and studying relevant literature and aid in the development of visualization, control, planning, and navigation software for these systems.

Expected Background: The student must have software development experience at the level of an introductory CS programming class. All other training will be part of the 10-week REU.

This project focuses on developing new data-augmentation methods to improve learning-based robot manipulation, with an emphasis on generating diverse, physically consistent examples for challenging dual-arm tasks. The student will work with state-of-the-art generative models, such as diffusion models for synthesizing novel robot poses or video-conditioned models for creating new demonstrations, to expand limited real-world datasets without additional data collection. Once these augmented datasets are created, the student will test how they improve performance on bimanual nonprehensile manipulation tasks where robots must coordinate two arms to push, slide, lift, or stabilize objects through contact rather than grasping. Depending on interest, the student may explore viewpoint augmentation, RGB-D augmentation, cross-embodiment transfer, or contact-aware augmentation techniques that preserve task feasibility. The goal is to understand how synthetic data can enable robots to generalize across new objects, configurations, and embodiments, ultimately making complex dual-arm manipulation more robust and scalable.

Prerequisites: basic machine learning and computer vision knowledge, solid Python programming skills.

We aim to make post-deployment robot learning accessible, efficient, and practical for everyday users by enabling robots to learn from natural human feedback rather than large collections of expert demonstrations. The project develops algorithms that allow generalist robotic models to adapt to new tasks using intuitive feedback, such as natural language, eye gaze, human demonstrations performed without the robot, and real-time user interventions or corrections. To this end, this project involves research directions around (1) learning from natural, low-burden feedback by grounding robot behavior in shared representations with language and human actions; (2) modeling human teaching behavior to improve data-efficiency; and (3) developing continual learning methods that treat human interventions and corrections as rich probabilistic signals for continual adaptation. Together, these contributions will enable robots to be teachable and safe in real-world, human-centered environments, significantly lowering barriers to robot use.

Prerequisites: hands-on experience with basic machine learning methods, solid Python programming skills.

Our group is developing integrated humanoid Vision-Language-Action (VLA) systems that enable robots to understand and interact with the physical world through perception, reasoning, and embodied control. This research spans the complete lifecycle of model development—from pretraining large-scale multimodal representations, to post-training and fine-tuning through reinforcement and human feedback, to deployment on real humanoid platforms. At the pretraining stage, we focus on aligning 3D perception, language, and motor skills using massive embodied datasets and simulation environments. Post-training efforts emphasize adaptive learning and safe policy refinement for complex manipulation and locomotion tasks. Finally, deployment studies examine real-world transfer, robustness, and generalization across diverse humanoid systems such as Unitree and Dexmate. This line of work bridges machine learning, robotics, and geometric vision, aiming to create scalable embodied intelligence capable of grounded understanding and purposeful action. It forms a central pillar of our lab’s broader mission—building physically grounded, data-driven digital twins and humanoid agents that learn from experience and operate safely and autonomously in dynamic environments.

Prerequisites: basic machine learning, computer vision, and robotic knowledge.

How can a robot understand requests such as “bring me my cup” when several similar cups are nearby? This project explores how smart-glasses recordings of everyday interactions can provide the personal context robots need to identify the intended object. In this REU project, students will investigate methods for converting first-person video into a searchable memory of objects, interactions, and locations. Building on an existing research prototype, they will develop and evaluate a focused component, such as interaction-aware video sampling, selection of informative object images, or retrieval across recording sessions. The goal is to preserve useful visual evidence while reducing redundant observations and incorrect associations between similar objects. Students will gain hands-on experience with computer vision, vision-language models, multimodal retrieval, and robotics research. They will design controlled experiments, compare their approach with a simple memory baseline, and analyze when the system retrieves the correct object with supporting evidence. Depending on project progress and hardware availability, they may also test how recovered context supports personalized manipulation with a frozen robot policy. The project will culminate in a reproducible software prototype, an experimental report, and a presentation.

Basic Python programming is expected; prior experience with machine learning or robotics is helpful but not required.

Physical environments can provide a variety of interaction opportunities for robots to exploit towards their locomotion goals. However, it is unclear how to even extract information about – much less exploit – these opportunities from physical properties (e.g., shape, size, distribution) of the environment. This project integrates mechanical engineering, electrical engineering, computer science, and physics, to discover the general principles governing the interactions between bio-inspired robots and their locomotion environments, and uses these principles to create novel control, sensing, and navigation strategies for robots to effectively move through non-flat, non-rigid, complex terrains to support applications in Earth and planetary explorations.

Selected candidates will assist in development of robotic platforms (Arduino, Raspberry Pi, SolidWorks, 3D Printing), experiment data collection and analysis (motion capture tracking, force measurements), modelling and simulation (MATLAB, Python, C++, ROS).

Recent results on VLA models and WAMs show a lot of promise in learning robot policies that directly map sensor observations to robot actions. However, these policies typically do not provide any guarantees on the safety or performance of the robot. In this project, we will explore how a robot’s task and safety objectives could be encoded using logical formalisms and how neuro-symbolic methods could be used to train the robot’s policies in a way that guarantees the performance or safety of the robot. The project could also involve learning formal specifications from demonstrations and videos.  Some exposure to logic would be ideal. The intern should have some knowledge of using and training VLAs or other kinds of multi-modal foundation models.