RSS SemRob 2025

About
Semantic understanding of the world is essential for robots to make safe and informed decisions, to adapt to changing environmental conditions, and to enable efficient interactions with other agents. In pursuit of semantic understanding, agents must be able to (i) interpret and represent high-level goals, agnostic of their physical morphology and despite irrelevant aspects of their environments; they must be able to (ii) reason, i.e., to extract abstract concepts from observations in the real-world, logically manipulate these concepts, then leverage the results for inference on downstream tasks; and they must be able to (iii) execute morphology-, environment-, and socially-appropriate behaviors towards those high-level goals.

Despite substantial recent advancements in the use of pre-trained, large-capacity models (i.e., foundation models) for difficult robotics problems, methods still struggle in the face of several practical challenges that relate to real-world deployment, e.g., cross-domain generalization, adaptation to dynamic and human-shared environments, and lifelong operation in open-world contexts. This workshop intends to sponsor discussion of new hybrid methodologies—those that combine representations from foundation models with modeling mechanisms that may prove useful for semantic reasoning and abstract goal understanding, including neural memory mechanisms, procedural modules (e.g., cognitive architectures), neuro-symbolic representations (e.g., knowledge/scene graph embeddings), chain-of-thought reasoning mechanisms, robot skill primitives and their composition, 3D scene representations (e.g., 3DGS), etc.

Intended audience. We aim to bring together engineers, researchers, and practitioners from different communities to enable avenues for interdisciplinary research on methods that could facilitate the deployment of semantics-aware and generalizable embodied agents in unstructured and dynamic real world environments. In addition to the organizers, the presenters, panelists, and technical program committee are drawn from the following communities: Robot Learning, Embodied AI, Planning + Controls, Cognitive Robotics, Neuro-Symbolism, Natural Language Processing, Multimodal Machine Learning, Computer Vision, and Digital Twins. We likewise intend to attract an audience from these diverse sub-communities to contribute to compelling discussions.

Schedule

Time
08:30		Organizers Introductory Remarks
08:40		Keynote 1: Jesse Thomason Embracing Language as Grounded Communication Abstract Language is not text data, it is a human medium for communication. The larger part of the natural language processing (NLP) community has doubled down on treating digital text as a sufficient approximation of language, scaling datasets and corresponding models to fit that text. I have argued that experience in the world grounds language, tying it to objects, actions, and concepts. In fact, I believe that language carries meaning only when considered alongside that world, and that the zeitgeist in NLP research currently misses the mark on truly interesting questions at the intersection of human language and machine computation. In this talk, I’ll highlight some of the ways my lab enables agents and robots to better understand and respond to human communication by considering the grounded context in which that communication occurs, including neurosymbolic multimodal reasoning, natural language dialogue and interaction for lifelong learning, and utilizing NLP technologies on non-text communication.
09:00		Keynote 2: Yonatan Bisk Semantics? Reasoning? Can define either of those terms? Abstract In this talk I'll discuss some recent work on language conditioned robotics, but I might also choose to spend time questioning the basic assumptions of all of our work, and if we're all misguided about the important questions in robotics.
09:20		Keynote 3: Dorsa Sadigh Human-Aligned Robot Learning: manipulation policies via preferences, RLHF, and VLM feedback Abstract Abstract TBD
09:40		Spotlight Talks.
10:00		Coffee Break, Socializing, Posters
10:40		Keynote 4: Ted Xiao Full-stack Robotics Foundation Models: From Embodied Reasoning to Dexterity Abstract Abstract TBD
11:00		Keynote 5: Manolis Savva Towards Realistic & Interactive 3D Simulation for Embodied AI Abstract 3D simulators are increasingly being used to develop and evaluate "embodied AI" (agents perceiving and acting in realistic environments). Much of the prior work in this space has treated simulators as "black boxes" within which learning algorithms are to be deployed. However, the system characteristics of the simulation platforms themselves and the datasets that are used with these platforms both greatly impact the feasibility and the outcomes of experiments involving simulation. In this talk, I will describe several recent projects that outline emerging challenges and opportunities in the development of 3D simulation for embodied AI. Bio: Manolis Savva is an Associate Professor at Simon Fraser University, and a Canada Research Chair in Computer Graphics. His research focuses on analysis, organization and generation of 3D content. Prior to his current position he was a visiting researcher at Facebook AI Research, and a postdoctoral researcher at Princeton University. He received his Ph.D. from Stanford University under the supervision of Pat Hanrahan. His work has been recognized through several awards including an ACM UIST notable paper award (ReVision), an ICCV best paper nomination (Habitat), two SGP dataset awards (ShapeNet, SGP 2018; ScanNet, SGP 2020), the 2022 Graphics Interface early career researcher award, and an ICLR 2023 outstanding paper award (Emergence of Maps).
11:20		Keynote 6: Lerrel Pinto On Building General-Purpose Home Robots Abstract The concept of a "generalist machine" in homes — a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective — has long been a goal in robotics that has been steadily pursued for decades. In this talk, I will present our recent efforts towards building such capable home robots. First, I will discuss how large, pretrained vision-language models can induce strong priors for mobile manipulation tasks like pick-and-drop. But pretrained models can only take us so far. To scale beyond basic picking, we will need systems and algorithms to rapidly learn new skills. This requires creating new tools to collect data, improving representations of the visual world, and enabling trial-and-error learning during deployment. While much of the work presented focuses on two-fingered hands, I will briefly introduce learning approaches for multi-fingered hands which support more dexterous behaviors and rich touch sensing combined with vision. Finally, I will outline unsolved problems that were not obvious initially, which, when solved, will bring us closer to general-purpose home robots.
11:40		Debate: Implicit/Data-emergent Reasoning Capabilities versus Explicit Reasoning Mechanisms? Panelists: Jesse Thomason, Dorsa Sadigh, Ted Xiao, Manolis Savva, Lerrel Pinto, Yonatan Bisk
12:30		Organizers Closing Remarks

Call for Papers

Targeted Topics

In addition to the RSS subject areas, we especially invite paper submissions on various topics, including (but not limited to):

Neural architectures leveraging demonstrations as prompts
Goal understanding through few-shot demonstrations
Novel abstractions, representations, and mechanisms for few-shot learning
Retrieval-augmentation mechanisms used in learning and task-execution
Agentic frameworks for failure reasoning, self-guidance, test-time adaptation, etc.
LLM-based action models for robot control; large action models
Semantic representations for generalizable policy learning
World models used for reasoning and optimization

Submission Guidelines

RSS SemRob 2025 suggests 4+N or 8+N paper length formats — i.e., 4 or 8 pages of main content with unlimited additional pages for references, appendices, etc. However, like RSS 2025, we impose no strict page length requirements on submissions; we trust that authors will recognize that respecting reviewers’ time is helpful to the evaluation of their work.

Submissions are handled through CMT: https://cmt3.research.microsoft.com/SEMROB2025

(Required acknowledgement: the Microsoft CMT service was used for managing the peer-reviewing process for this conference. This service was provided for free by Microsoft and they bore all expenses, including costs for Azure cloud services as well as for software development and support.)

We will accept the official LaTeX or Word paper templates, provided by RSS 2025.

Our review process will be double-blind, following the RSS paper submission policy for Science/Systems papers.

All accepted papers will be invited for poster presentations; the highest-rated papers, according to the Technical Program Committee, will be given spotlight presentations. Accepted papers will be made available online on this workshop website as non-archival reports, allowing authors to also submit their works to future conferences or journals. We will highlight the Best Paper Award during the closing remarks at the workshop event.

Important Dates

Submission deadline: 1 June 2025, 23:59 AOE.
Author Notifications: 10 June 2025.
Camera Ready: 15 June 2025, 23:59 AOE.
Workshop: 21 June 2025, 14:00-18:30 CET