Connecting Robots and Large Language Models, Google’s SayCan

The convergence of robotics and large language models is a new frontier in the field of artificial intelligence, and it holds immense promise for the future.

You can read more about recent advances in robotics and Large Language Models in my previous blog post.

Today’s blog post revisit a recent research Paper that went unnoticed for some reason. As a matter of fact, with the development of robots like SayCan by Google, we are now seeing the first glimpses of a new generation of robots that can interact naturally with humans and understand high-level, temporally extended instructions.

• Paper Title: “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”
• Source link: https://say-can.github.io/

If you like this topic, please consider supporting us: 🔔 clap & follow 🔔

Connecting Language Models to the Physical World!

One of the main challenges in integrating robotics with large language models is providing the necessary contextual grounding that allows the language model to make informed decisions.

The robot acts as the “hands and eyes” of the language model, providing the necessary physical context, while the language model provides high-level semantic knowledge about the task.

In other words: Based on its available sensors, the language model should be able to assess the situation, ask the right questions, and act towards resolving any task at hand!

For this experiment, Google relied on a 540-billion parameter language model called PaLM to develop a one of kind butler robot!

Task Oriented Algorithm: Input Analysis and Real Time Decision Making

So what’s new? Can we simply connect outside sensors to an existing language model?

The answer is obviously no. We need to bring together language models and the physical world. And that’s SayCan’s mission!

As a matter if fact, the algorithm SayCan bridges the gap between large language models (LLMs) and robotic systems by providing a way to connect the high-level semantic knowledge represented in LLMs to real-world tasks that can be executed by robots.

This is achieved by combining

• the outputs of the LLM
• with the output of value functions, which provide a measure of how likely a particular task will succeed from the current state.

Thus, SayCan brings several advantages over traditional language models.

• Firstly, it provides a way to perform high-level natural language instructions in a real-world environment. Traditional language models can interpret and generate natural language text, but they lack the ability to execute those instructions in the real-world. SayCan enables this by using the output of the LLM to inform decision-making and guide the execution of tasks by a robot.
• Secondly, SayCan allows robots to operate in a more autonomous and flexible manner. By combining the output of the LLM with value functions, SayCan can select tasks that are both feasible and contextually appropriate, allowing the robot to operate in a more dynamic and adaptive manner.
• Finally, SayCan provides a way to make the decision-making process of robots more interpretable and transparent. The visualizations of the decision-making process in SayCan, which show the contribution of both the LLM and the value functions, highlight the interpretability of the approach and help to build trust in autonomous systems.

Experimental Results

In benchmarking experiments, SayCan has demonstrated impressive results in both

• an office kitchen,
• and a mock office kitchen,

where it successfully planned and executed complex tasks specified by natural language instructions.

For example, when given the task “I spilled my coke, can you bring me something to clean it up?” SayCan planned and executed the steps to find a sponge, pick it up, bring it to the user, and complete the task.

Similarly, when given the slightly different task “I spilled my coke, can you bring me a replacement,” SayCan successfully planned and executed the steps to find a coke can, pick it up, bring it to the user, and complete the task.

To be fair, the robot is still slow but for a first experiment, it does look promising!

This is still an early experiment and you can follow its updates here.

Conclusion

The convergence of robotics and large language models is a key development in the field of artificial intelligence, and it has the potential to revolutionize the way robots interact with humans.

With algorithms like SayCan, we are now seeing the first glimpses of a new generation of robots that can understand high-level, temporally extended instructions and complete complex tasks in real-world environments.

The future of robotics and AI is exciting, and it will be fascinating to see what other breakthroughs lie ahead.

That being said, its range of applications are very wide, and can potentially be dangerous. It’s important that we think about safe guards limiting robots’ ability to harm humans.

Read more about robots and LLMs as well as potential safe guards to protect humans:

If you liked this post, please consider supporting us: 🔔 clap & follow 🔔