Building Rational Robots: Prof. Leslie Pack Kaelbling

General Purpose Robot Development Philosophy
📌 The goal is to create completely general-purpose robots as capable as humans across diverse, everyday situations, acknowledging the enormous variability and huge state space involved.
🤖 The current paradigm of relying solely on large-scale simulators and generalized learning algorithms running for long periods might not be scalable for achieving general intelligence.
💡 The proposed contrarian approach favors building systems with built-in fundamental learning and inference algorithms, leveraging offline pre-trained massive models, and enabling the robot to learn continually from its own lifetime experiences about how the world works.
✅ This engineering strategy aims for systems that are easier for humans to understand and potentially more aligned with human operational methods.

Perception and World Modeling
👁️ Modern computer vision allows systems to process a single RGBD image and generate a usable 3D model of the world, segmented with object shapes, which aids in better planning.
🧠 A rational robot controller should incorporate a mental model of the world detailing object properties (shape, mass, friction) derived from perception.
🗣️ This mental model, combined with human goals (expressed via natural language), feeds into a planning algorithm that sequences actions, executing them sequentially while continuously re-evaluating based on new observations.
🧩 Objects do not need to be known or recognizable beforehand; the system can reason about and manipulate unknown "stuff on the table."

Learning Causal Models and Generalization
🛠️ The focus in learning should be on neuro-symbolic world models that mix discrete (symbolic, object abstraction) and continuous components (perception, physical changes).
📈 These models allow for efficient and incremental learning, ensuring new skills (like soldering) do not degrade pre-existing capabilities (like dancing).
👩‍🏫 Demonstrations allow learning causal models of operations; generalization is achieved because the system operates at a suitable symbolic level of abstraction, applying learned rules to vastly different physical circumstances (e.g., using the juicer in a new setup).

Learning through Planning and Practice
🧠 Learning can make planning more efficient (e.g., automating learned routines like using a turn signal), while planning can improve learning.
🏠 Robots can autonomously practice skills that are important for tasks assigned but where performance is currently low, using their knowledge of world dynamics to set up the practicing environment autonomously.
🔄 The ability to plan, observe, and replan when outcomes deviate from expectation (due to unexpected dynamics like slippage or external interference) is crucial for adapting to novelty.
💡 Knowing what you don't know is critical; robots should use their resources to gather information (e.g., through targeted actions) to reduce uncertainty in world models or skill execution parameters.

Key Points & Insights
➡️ The pursuit of general-purpose robots relies on a hybrid AI approach, integrating fundamental algorithms with massive pre-trained models and continuous, compositional learning.
➡️ Mental models of the world that include object properties and causal relationships are superior to purely reactive policies for robust reasoning and planning.
➡️ Robots must incorporate memory and handle uncertainty; memory allows inferring invisible states (like wind) from observed effects, and uncertainty drives information-gathering actions.
➡️ Autonomous practice driven by identified skill gaps, coupled with planning capabilities, allows robots to tune their generative models for specific environmental dynamics (like sweep strength).

📸 Video summarized with SummaryTube.com on Dec 11, 2025, 02:20 UTC