What is Project Astra? Google’s futuristic universal assistant explained

Almost everyone in tech is investing heavily in artificial intelligence right now, and Google is among those most committed to an AI future. Project Astra, unveiled at Google I/O 2024, is a big part of that – and it could end up being one of Google's most important AI tools.

Astra is being billed as “a universal AI agent that is helpful in everyday life”. It's essentially something like a blending of Google Assistant and Google Gemini, with added features and supercharged capabilities for a natural, conversational experience.

Here, we're going to explain everything you need to know about Project Astra – how it works, what it can do, when you can get it, and how it might shape the future. 

What is Project Astra?

In some ways, Project Astra isn't any different to the AI chatbots we've already got: you ask a question about what's in a picture, or about how to do something, or request some creative text to be generated, and Astra gets on with it.

What elevates this particular AI project is its multimodal functionality (the way text, images, video, and audio can all be combined), the speed that the bot works at, and how conversational it is. Google's aim, as we've already mentioned, is to create “a universal AI agent” that can do anything and understand everything.

Google IO 2024

Project Astra in action (Image credit: Google)

Think about the Hal 9000 bot in Kubrick's 2001: A Space Odyssey, or the Samantha assistant in the movie Her: talking to them is like talking to a human being, and there isn't much they can't do. (Both those AIs eventually got too big for their creators to control, but let's ignore that for the time being.)

Project Astra has been built to understand context and to take actions, to be able to work in real time, and to remember conversations from the past. From the demos we've seen so far, it works on phones and on smart glasses, and is powered by the Google Gemini AI models – so it may eventually be part of the Gemini app, rather than something that's separate and standalone.

When is Project Astra coming out?

Project Astra is in its early stages: this isn't something that's going to be available to the masses for a few months at least. That said, Google says that “some of these agent capabilities will come to Google products like the Gemini app later this year”, so it looks as though elements of Astra will appear gradually in Google's apps as we go through 2024.

When we were given some hands-on time with Project Astra at I/O 2024, these sessions were limited to four minutes each – so that gives you some idea of how far away this is from being something that anyone, anywhere can make use of. What's more, the Astra kit didn't look particularly portable, and the Google reps were careful to refer to it as a prototype.

Project Astra demonstration room at Google I/O showing large display and toys

We’ve already tried Project Astra (Image credit: Philip Berne / Future)

Taking all that together, we get the impression that some of the Project Astra tricks we've seen demoed might appear in the Google Gemini app sooner rather than later. At the same time, the full Astra experience – perhaps involving some dedicated hardware – is probably not going to be rolling out until 2025 at the earliest.

Now that Google has shared what Project Astra is and what it's capable of, it's likely that we're going to hear a whole lot more about it in the months ahead. Bear in mind that ChatGPT and Dall-E developer OpenAI is busy pushing out major upgrades of its own, and Google isn't going to want to be left behind.

What can I do with Project Astra?

One of Google's demos shows Astra running on a phone, using its camera input and talking naturally to a user: it's asked to flag up something in view that can play sounds, and correctly identifies a speaker. When an arrow is drawn on screen, Astra then recognizes and talks about the speaker component highlighted by the arrow.

In another demo, we see Astra correctly identifying world landmarks from drawings in a sketchbook. It's also able to remember the order of objects in a list, identify a neighborhood from an image, understand the purpose of sections of code that are shown to it, and solve math problems that are written out.

There's a lot of emphasis on recognizing objects, drawings, text, and more through a camera system – while at the same time understanding human speech and generating appropriate responses. This is the multimodal part of Project Astra in action, which makes it a step up from what we already have – with improvements in caching, recording, and processing key to the real time responsiveness.

In our hands-on time with Project Astra, we were able to get it to tell a story based on objects that we showed to the camera – and adapt the story as we went on. Further down the line, it's not difficult to imagine Astra applying these smarts as you explore a city on vacation, or solve a physics problem on a whiteboard, or provide detailed information about what's being shown in a sports game.

Which devices will include Project Astra?

In the demonstrations of Project Astra that Google has shown off so far, the AI is running on an unidentified smartphone and an unidentified pair of smart glasses – suggesting that we might not have heard the last of Google Glass yet.

Google has also hinted that Project Astra is going to be coming to devices with other form factors. We've already mentioned the Her movie, and it's well within the realms of possibility that we might eventually see the Astra bot built into wireless earbuds (assuming they have a strong enough Wi-Fi connection).

Google Pixel 8 Pro back in porcelain in front of animal print

Expect to see Project Astra turn up on Pixel phones, eventually (Image credit: Future / Philip Berne)

In the hands-on area that was set up at Google I/O 2024, Astra was powered through a large camera, and could only work with a specific set of objects as props. Clearly, any device that runs Astra's impressive features is going to need a lot of on-board processing power, or a very quick connection to the cloud, in order to keep up the real-time conversation that's core to the AI.

As time goes on and technology improves, though, these limitations should slowly begin to be overcome. The next time we hear something major about Project Astra could be around the time of the launch of the Google Pixel 9 in the last few months of 2024; Google will no doubt want to make this the most AI-capable smartphone yet.

You might also like

TechRadar – All the latest technology news

Read More

Google’s Project Astra could supercharge the Pixel 9 – and help Google Glass make a comeback

I didn't expect Google Glass to make a minor comeback at Google I/O 2024, but it did thanks to Project Astra. 

That's Google's name for a new prototype of AI agents, underpinned by the Gemini multimodal AI, that can make sense of video and speech inputs, and smartly react to what a person is effectively looking at and answer queries about it. 

Described as a “universal AI” that can be “truly helpful in everyday life”, Project Astra is designed to be proactive, teachable, and able to understand natural language. And in a video ,Google demonstrated this with a person using what looked like a Pixel 8 Pro with the Astra AI running on it. 

By pointing the phone's camera at room, the person was able to ask Astra to “tell me when you see something that makes sound”, to which the AI will flagged a speaker it can see within the camera's viewfinder. From there the person was able to ask what a certain part of the speaker was, with the AI replying that the part in question is a tweeter and handles high frequencies. 

But Astra does a lot more: it can identify code on a monitor and explain what it does, and it can work out where someone is in a city and provide a description of that area. Heck, when promoted, it can even make an alliterative sentence around a set of crayons in a fashion that's a tad Dr Zeus-like.

It can can even recall where the user has left a pair of glasses, as the AI remembers where it saw them last. It was able to do the latter as AI is designed to encode video frames of what it's seen, combine that video with speech inputs and put it all together in a timeline of events, caching that information so it can recall it later at speed. 

Then flipping over to a person wearing the Google Glass 'smart glasses', Astra could see that the person was looking at a diagram of a system on a whiteboard, and figure out where optimizations could be made when asked about them. 

Such capabilities suddenly make Glass seem genuinely useful, rather than the slightly creepy and arguably dud device it was a handful of years ago; maybe we'll see Google return to the smart glasses arena after this. 

Project Astra can do all of this thanks to using multimodal AI, which in simple terms is a mix of neural network models that can process data and inputs from multiple sources; think mixing information from cameras and microphones with knowledge the AI has already been trained on.

Google didn't say when Project Astra will make it into products, or even into the hands of developers, but Google's DeepMind CEO Demis Hassabis said that “some of these capabilities are coming to Google products, like the Gemini app, later this year.” I'd be very surprised if that doesn't mean the Google Pixel 9, which we're expecting to arrive later this year.

Now it's worth bearing in mind that Project Astra was shown off in a very slick video, and the reality of such onboard AI agents is they can suffer from latency. But it's a promising look at how Google will likely integrate actually useful AI tools into its future products.

You might also like

TechRadar – All the latest technology news

Read More