Gemini uses Model Context Protocol to book Uber and...

If there's a Model Context Protocol (MCP) integration--the open source universal language that lets LLMs talk to third-party apps--then Gemini can run the task in the backend. (In this instance, you wouldn't see the whole process play out; you'd just see the final checkout step appear after you make the request.) There are also "App Functions" developers can build that allows Gemini to interface with it in a structured way. But if neither exist, then that's when Gemini can open the app itself and navigate through the buttons, text boxes, and menus to complete tasks.

"This is the first time we're doing this on Android with applications, and so getting this right is really important," Samat says. "I think it's an exciting step forward in technology. We sort of view this as the beginning of a new era of mobile intelligence, and Android is where we think you see the future first." Privacy concerns abound when it comes to granting Gemini access to your apps.

Samat says that's why Google hasn't included any overly sensitive apps in this first batch for task automation. He says this data is not used for advertising, and that users can delete the data that Gemini sees. "We do think it's really important that people have trust in the system, and that comes from having control and transparency of what it's doing." While the smartphone screen is still required at the moment to complete the task, Samat envisions a future where you can start these tasks through other devices--say, a pair of smart glasses, an AI pendant, or even a car.

(There will be several new Android XR-powered smart glasses launching this year.) He says the company is looking at other ways to make the final authentication capable on these other devices.