Apple, Siri, and Dealing With Failure

A report on Siri came out from the Information yesterday (Paywall) and got quite a bit of coverage and commentary from other news publications and talking heads on Twitter.

Reports like this that largely site ex-employees are always hard to know how much you can believe, however, the issues around management decisions regarding Siri and the shifting of blame stated in the article were quite concerning.

Many of you probably read the article, or read the other media reports on it, and while I’m sure some skepticism is warranted Siri is yet to live up to its potential and understanding why is an important part of this story and Apple’s future.

Siri was first to market, but in a way, being first may have hurt Apple more than helped. If you follow Apple closely, you observe that Apple is rarely first to market with anything. If anything they are usually “late” by people’s standards. I’ve always thought Apple timed the market just right releasing products and features when the market was ready not before and not after. Siri is the exception, and in retrospect, I’m not sure anyone at Apple predicted just how important voice assistants would likely be as a future platform. Many of the stories from the early Siri days, and the vision behind was certainly that Siri would be a helper but the extent humans may rely on these helpers in the future was likely not predicted back then. If Apple, management, and the Siri teams knew how important Siri as a platform and future operating system would be back then, I think decisions may have been made differently.

While I don’t agree with all of the claims and assumptions made in the Information article, it is important to recognize where Siri is today and more importantly some things to watch for on how Apple moves forward.

Apple the Post Steve Jobs Era
I hate to use this cliche term, but I do think Siri’s development is one area where we need to watch closely from Apple in the post-Jobs era of Apple. When Steve Jobs was at Apple, he was the ultimate arbiter when it came to Apple products. The final decision often rested with him, but more importantly, he had a keen eye for what makes a great product. If it didn’t work well enough, he sent the teams back to work and held people accountable for their actions. While it is unclear how current Apple management handles these issues today, the in-fighting, lack of confidence and support of the vision, and blame shifting within the Siri team from the Information report is concerning. While under Steve Jobs, a culture of innovation thrived at Apple, but what we may not have fully grasped before was how important a culture of accountability was in achieving this thanks to Steve Jobs.

I remember many conversations with investors and pundits about Apple the month’s after Steve Jobs died. There was a major concern that Apple would not be same, and their run on innovation would end. That certainly has not been the case, however, what I’m not sure we have seen is how Apple in the post Jobs era deals with failure or potentially impactful struggles to the companies future. Siri and Apple services largely a key question and area to observe here.

Apple is correct in their emphasis on measuring customer satisfaction as their bar. However, Apple tends to only state customer satisfaction around their hardware which is pretty stellar. How do Apple services like Apple Music, Siri, Maps, etc., rank with customer satisfaction? This is something I want to focus on with our research, but we did include Siri customer satisfaction in our Voice 2.0 study last year. I’m sharing this chart for the first time to assist in my broader point in this analysis.

Note the Siri, and Google Assistant satisfaction usage is from people using it on their smartphones where Alexa satisfaction is from people using it on Echo’s. In each result, the answers are narrowed to only users of each voice assistant. That’s an important distinction but doesn’t change the key interpretation of the results. Siri is an area where Apple has lower satisfaction than its competitors and may be the lowest customer satisfaction of any Apple product or service. Most companies would be happy with a ~70% customer satisfaction, and that is certainly not terrible. However, it’s not Apple level customer sat.

This is an area where being good enough won’t cut it for Apple, and as we track consumers experiences with Siri, we will see how sentiment changes.

Making Siri Better
As I have discussed the issues consumers have with Siri, I think it boils down to two key areas.

Inconsistent: A constant complaint I hear is the inconsistency of Siri. People get frustrated not because they can’t eventually get Siri to answer them correctly, but rather that it takes some tries and re-tries to get it right. When people have posted similar Siri Fail’s on Twitter, folks will comment that when they tried the same query, Siri worked for them. They use that to essentially say “Not a problem Siri works fine for me!” Which completely misses the point. It’s that is works sometimes, not every time, and is inconsistent.

The other element of inconsistency is how Siri functions a bit differently on different devices. For example, it can do certain things on my phone but not on my Apple Watch or HomePod. This may not be an issue most people have if they only have a small number of Apple devices, but you uncover this issue the more you try to use Siri on the many different devices you own. You come to expect Siri to work a certain way and when you try it on other devices and it doesn’t work that way it is not just confusing but also frustrating.

Incorrect: The other main Siri complaint is when Siri provides a result but its entirely incorrect. We have successfully made the #sirifail trend on Twitter, and I found these two examples, which are my current favorites, of Siri’s providing an incorrect answer.

What drives the user crazy is how Apple’s natural language processing works perfectly. In these cases, and many others, it appears that Siri understands what you said. Meaning it converted your voice to text accurately but then provided incorrect results. Here is where a significant new model for Siri needs to be developed by Apple. Something I loosely call context for failure. While Apple is certainly getting quite a bit of data, approximately 2 billion a week from their public record, I’d bet Apple is not getting data about contextual failures. Meaning if an incorrect result is given I don’t think Apple knows that Siri did fail and create a record as to why to train the system to not fail in the future. To the last statement in the title of my article, how does Siri deal with failure is a central question necessary to train the model to be correct more often. This is valuable data that I’m not sure Apple has any context for. I wonder if it would help that Apple makes it so Siri follows up her answer with a question like “is this what you were looking for?” Then will listen to if the user says yes or no. That would at least help Apple acquire data to the failure, incorrect result rate of Siri, and give them data to examine what went wrong and how to fix it.

The other option is for Apple not to have Siri give a result at all to a query or request if it is not X% certain it can deliver the right example. I feel Siri too often tries to get me a result even if it may not be correct just to provide a result vs. not provide a result. In this scenario, it would be up to Apple to decide a reasonable benchmark for confidence in a result and then deliver a different type of response if confidence is below that percentage.

We can only hope, that Siri has moved up the priority list for Apple in recent years. From some conversations I’ve had with folks at Apple, I do think this is the case. However, the proof is in the pudding as the saying goes. We may not see the advancements we desire immediately but I think we should, and hopefully will continue to see Siri make positive progress.

Published by

Ben Bajarin

Ben Bajarin is a Principal Analyst and the head of primary research at Creative Strategies, Inc - An industry analysis, market intelligence and research firm located in Silicon Valley. His primary focus is consumer technology and market trend research and he is responsible for studying over 30 countries. Full Bio

One thought on “Apple, Siri, and Dealing With Failure”

Leave a Reply

Your email address will not be published. Required fields are marked *