This website is under construction. More updates are on the way.

Sanjay OrugantiResearch Scientist · Rensselaer Polytechnic Institute

← Research

Research project · cognitive robotics

HARMONIC

Human-AI Robotic Team Member Operating with Natural Intelligence and Communication

When
2023 – present
My part
Designer and technical lead
Where
LEIA Lab, RPI, with UT Austin
Supported by
U.S. Office of Naval Research

Overview

HARMONIC joins the OntoAgent cognitive architecture to robot control, so a robot can reason about plans and goals, talk with its teammates, and explain its decisions. In the demonstration above, a ground robot, a drone, and a human operator search an apartment together.

My role

OntoAgent, the cognitive layer of HARMONIC, was developed by Sergei Nirenburg, Marjorie McShane, Jesse English, myself and many students at the LEIA Lab. I added the physical AI dimension to it. I designed HARMONIC and lead its development and its integration on robots. My part is to turn the work of faculty, research staff, and doctoral students at RPI and UT Austin into one system that runs on a robot. We have run it on several robots, including a sim-to-real transfer to a humanoid.

Architecture

Diagram of the HARMONIC framework: a strategic cognitive layer (System 2, OntoAgent) above a tactical control layer (System 1, behavior trees), connected by perception and action data APIs.

System 2 · Strategic layer

OntoAgent

  • Attention management and perception interpretation
  • Utility-based and analogical decision-making with metacognition
  • Natural language communication and explaining decisions

System 1 · Tactical layer

Behavior trees

  • Low-level planners and control algorithms
  • Plans motor actions for high-level commands such as “pick up that screwdriver”
  • Reactive responses such as collision avoidance

Inside the tactical layer

Interactive demos

Papers

Sanjay Oruganti with two co-authors in front of the 'Why Cognitive Robotics Matters' poster at IROS 2026.

IROS 2026

Why cognitive robotics matters: Lessons from OntoAgent and LLM deployment in HARMONIC for safety-critical robot teaming

S. Oruganti, S. Nirenburg, M. McShane, J. English, M. Roberts, C. Arndt, R. Parasuraman, and L. Sentis

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), Pittsburgh, PA, USA (2026).

AAAI FSS 2026

Metacognition and cognitive transparency for trustworthy human-robot teams

S. Oruganti, S. Nirenburg, M. McShane, and J. English

Proceedings of the AAAI Fall Symposium on AI Trustworthiness and Risk Assessment for Challenged Contexts (ATRACC 2026), vol. 10, Arlington, VA, USA, pp. 377–386 (2026).

HARMONIC: A content-centric cognitive robotic architecture

S. Oruganti, S. Nirenburg, M. McShane, J. English, M. K. Roberts, C. Arndt, C. Gonzalez, M. Seo, and L. Sentis

arXiv preprint arXiv:2509.13279 [cs.RO] (2025).

HARMONIC: Cognitive and control collaboration in human-robotic teams

S. Oruganti, S. Nirenburg, M. McShane, J. English, M. K. Roberts, C. Arndt, S. Kamireddy, C. Gonzalez, M. Seo, and L. Sentis

arXiv preprint arXiv:2409.18047 [cs.RO] (2024).

ICRA@40

HARMONIC: A framework for explanatory cognitive robots

S. Oruganti, S. Nirenburg, M. McShane, J. English, M. K. Roberts, and C. Arndt

IEEE Conference on Robotics and Automation – 40 Years (ICRA@40) (2024).

Explore further

Support

This research was supported in part by grant #N00014-23-1-2060 from the U.S. Office of Naval Research. Any opinions or findings expressed in this material are those of the authors and do not necessarily reflect the views of the Office of Naval Research. The team also took part in the NSF I-Corps programs (2024 – 2026), where I served as Entrepreneurial Lead.

For more on OntoAgent, see Linguistics for the Age of AI (MIT Press, 2021) and Agents in the Long Game of AI (MIT Press, 2024).