Amazon’s Panos Panay on ‘unlearning’ Alexa speak: Why your next Echo won’t just wait for commands

Panos: I think we are at the very early stages… when you talk about hardware, I think we haven’t quite seen the form factors that are yet to come with devices and the opportunities that AI can bring to them. We are starting to — there’s some incredible stuff out there, and some of the stuff that we’re building at Amazon is awesome — but we’re very early.

So how is Amazon looking at this entire ecosystem? You have started pushing the AI game, but do you think the experience is going to change a lot? The Echo devices have a certain history of how they sit in the living room or the bedroom. So how do you bring in AI? You have to think maybe a little bit out of the box here.

Panos: I think there are a couple of things. The first is, when you think about what Alexa has been, it’s been a lot of what I would say is command and control — meaning I say one thing, I get a very specifically structured response. I think we are now moving to a generation of a more fluid experience. It’s conversational, it’s more natural. It’s not about command and control. It’s about what am I thinking, how do I say it, and then how does the product respond.

So it’s changing in the way that it’s becoming more proactive. Alexa+ hasn’t made it to India yetbut when it gets here, the fundamentals of the product — understanding you better, holding context, holding memory as the customer chooses — all of a sudden you have interactions that have transformed. Alexa gets things done, makes recommendations, updates your calendar, updates your documents. When you get to that point, it really starts to feel like a natural assistant.

With AI, you are expecting things to go to an autonomous level at some stage. People have been using Alexa devices for over a decade now, so there’s a lot of muscle memory on how these devices are supposed to work. Is there going to be a level of discomfort?

Panos: What a perfect way to think about it. We’ve almost been teaching people how to speak to Alexa for 10 years. We call it “Alexa speak” sometimes — it’s its own language. It’s very simple, it’s short, it’s very directive, because maybe not all the time it understands what you’re trying to say, so you have to be very particular or use very short sentences. How you unteach that is a big challenge — getting somebody to the natural state. But once you discover it, it’s priceless. But how you get the customer to translate from “play this song” to “help me think about what the next song is” is very tricky.

Alexa devices have evolved over the years — it now has a screen, it’s maybe in your car. But is AI bringing in an entirely new interface in a way? How are you looking at that? Is that also a big requirement on the device side?

Panos: For many people, this will be the first ambient AI experience they have using Alexa — meaning they can just talk, and it’s there, it’s in the room with them.

Evolving the UI to have a screen is really important, because ultimately it’s going to be giving you information that will be very useful to you. It will dynamically update itself, using AI to construct what it is that you need right now and put it in front of you. We call this being proactive. There are a lot of moments where Alexa can be proactive — trying to determine, hey, what are you thinking you might want to do next — and put some visuals on the screen that make you think and ask a question. Or even as you’re having a discussion, it’ll bring you images that help enhance the discussion or bring visual language to you.

These are nuances that are really important, because when you think about Alexa being that ambient, AI-first experience, it’ll be the first time many users experience AI in ways outside of putting text into a prompt box — outside of what feels like a search box. This will be the time where natural conversation comes to full fruition. And then, if you’re calling on a service and you wanted something delivered to your house and you didn’t want to use your hands, it has to be able to dynamically update itself, prompting you to think about what you need next.

I think that’s part of the translation of the future of what AI can bring. Amazon’s context with its users, especially on the device side, is very different from maybe how all the other tech companies see it.

Is that arc going to be different from maybe how a smartphone company is looking at integrating AI into their devices?

Panos: It might be. At the end of the day, it’s really about what the customer needs. Will it be different? I think so. Understanding what the customer needs next is really important for an assistant. Just think about it like a real-life assistant — the better it knows you, the better your assistant would be. And that’s what we need here… I always think AI is best defined as somebody who’s with you — like a colleague, a co-pilot — but it really needs to understand you.