Sanjay OrugantiResearch Scientist · Rensselaer Polytechnic Institute
Research project · cognitive robotics
HARMONIC
Human-AI Robotic Team Member Operating with Natural Intelligence and Communication
- When
- 2023 – present
- My part
- Designer and technical lead
- Where
- LEIA Lab, RPI, with UT Austin
- Supported by
- U.S. Office of Naval Research
Overview
HARMONIC joins the OntoAgent cognitive architecture to robot control, so a robot can reason about plans and goals, talk with its teammates, and explain its decisions. In the demonstration above, a ground robot, a drone, and a human operator search an apartment together.
My role
OntoAgent, the cognitive layer of HARMONIC, was developed by Sergei Nirenburg, Marjorie McShane, Jesse English, myself and many students at the LEIA Lab. I added the physical AI dimension to it. I designed HARMONIC and lead its development and its integration on robots. My part is to turn the work of faculty, research staff, and doctoral students at RPI and UT Austin into one system that runs on a robot. We have run it on several robots, including a sim-to-real transfer to a humanoid.
Architecture
System 2 · Strategic layer
OntoAgent
- Attention management and perception interpretation
- Utility-based and analogical decision-making with metacognition
- Natural language communication and explaining decisions
System 1 · Tactical layer
Behavior trees
- Low-level planners and control algorithms
- Plans motor actions for high-level commands such as “pick up that screwdriver”
- Reactive responses such as collision avoidance
Inside the tactical layer
Interactive demos
-
Demo 01
Shipboard maintenance
A technician reports an overheating engine. The robot diagnoses the issue, asks clarifying questions, and retrieves the replacement part.
-
Demo 02
Multi-robot search and retrieve
A ground robot and a drone work together to find a lost object, sharing plans and reports as they go.
-
Demo 03
Teaching through instruction
A person teaches the robot a new procedure in plain language, and the robot carries it out safely when asked.
Papers

IROS 2026
Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), Pittsburgh, PA, USA (2026).
BibTeX
@inproceedings{oruganti2026cognitive,
author = {Oruganti, S. and Nirenburg, S. and McShane, M. and English, J. and Roberts, M. and Arndt, C. and Parasuraman, R. and Sentis, L.},
title = {Why cognitive robotics matters: Lessons from OntoAgent and LLM deployment in HARMONIC for safety-critical robot teaming},
booktitle = {Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026)},
address = {Pittsburgh, PA, USA},
year = {2026}
}AAAI FSS 2026
Metacognition and cognitive transparency for trustworthy human-robot teams
Proceedings of the AAAI Fall Symposium on AI Trustworthiness and Risk Assessment for Challenged Contexts (ATRACC 2026), vol. 10, Arlington, VA, USA, pp. 377–386 (2026).
BibTeX
@inproceedings{oruganti2026metacognition,
author = {Oruganti, S. and Nirenburg, S. and McShane, M. and English, J.},
title = {Metacognition and cognitive transparency for trustworthy human-robot teams},
booktitle = {Proceedings of the AAAI Fall Symposium on AI Trustworthiness and Risk Assessment for Challenged Contexts (ATRACC 2026)},
volume = {10},
pages = {377--386},
address = {Arlington, VA, USA},
year = {2026}
}HARMONIC: A content-centric cognitive robotic architecture
arXiv preprint arXiv:2509.13279 [cs.RO] (2025).
BibTeX
@misc{oruganti2025harmonic,
author = {Oruganti, S. and Nirenburg, S. and McShane, M. and English, J. and Roberts, M. K. and Arndt, C. and Gonzalez, C. and Seo, M. and Sentis, L.},
title = {HARMONIC: A content-centric cognitive robotic architecture},
eprint = {2509.13279},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
year = {2025}
}HARMONIC: Cognitive and control collaboration in human-robotic teams
arXiv preprint arXiv:2409.18047 [cs.RO] (2024).
BibTeX
@misc{oruganti2024harmonic,
author = {Oruganti, S. and Nirenburg, S. and McShane, M. and English, J. and Roberts, M. K. and Arndt, C. and Kamireddy, S. and Gonzalez, C. and Seo, M. and Sentis, L.},
title = {HARMONIC: Cognitive and control collaboration in human-robotic teams},
eprint = {2409.18047},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
year = {2024}
}ICRA@40
HARMONIC: A framework for explanatory cognitive robots
IEEE Conference on Robotics and Automation – 40 Years (ICRA@40) (2024).
BibTeX
@inproceedings{oruganti2024harmonic,
author = {Oruganti, S. and Nirenburg, S. and McShane, M. and English, J. and Roberts, M. K. and Arndt, C.},
title = {HARMONIC: A framework for explanatory cognitive robots},
booktitle = {IEEE Conference on Robotics and Automation -- 40 Years (ICRA@40)},
year = {2024}
}Explore further
On the LEIA Lab site
- Website HARMONIC at the lab The project’s home, with all three interactive demos. leia-lab.com ↗
- Website Architecture walkthrough The strategic and tactical layers, one step at a time. leia-lab.com ↗
- Video Videos The overview and the teaching-through-instruction demonstration. leia-lab.com ↗
- Lab OntoAgent The cognitive architecture that HARMONIC puts on robots. leia-lab.com ↗
Companion pages
- Website Why Cognitive Robotics Matters Can a language model stand in for a cognitive architecture when a robot team’s task is safety-critical? IROS 2026. rpi-leia.github.io ↗
- Website Metacognition and transparency Robots that know the limits of what they know and show their reasoning to teammates. AAAI Fall Symposium 2026. leia-lab.com ↗
- Website LLMs as knowledge workers Companion page for the AAAI-MAKE 2026 paper. leia-lab.com ↗
Support
This research was supported in part by grant #N00014-23-1-2060 from the U.S. Office of Naval Research. Any opinions or findings expressed in this material are those of the authors and do not necessarily reflect the views of the Office of Naval Research. The team also took part in the NSF I-Corps programs (2024 – 2026), where I served as Entrepreneurial Lead.
For more on OntoAgent, see Linguistics for the Age of AI (MIT Press, 2021) and Agents in the Long Game of AI (MIT Press, 2024).