Why AI's Make Certain Kinds of Mistakes

There are so many who rave about AI, and its many forms like ChatGPT. Yes, the output of many of these AI’s is very impressive. And yes, they hallucinate too (cook up facts). All that’s well known.

 

What’s less well known is that many AI’s make mistakes with these two simple questions. The first one: Which is bigger? 9.2 or 9.11? The other one is just as simple: How many r’s does the word “strawberry” have?

 

Believe it or not, a lot of AI’s get those two questions wrong! What is going on? As you know, the AI’s can (on many topics) explain their reasoning. So they were asked to explain how they come to the wrong answers.

 

On the 9.2 v/s 9.11 question, the AI’s say there are multiple interpretational patterns to evaluate the question. The maths way is just one of them. The maths way leads to the obvious answer (9.2 is the same as 9.20; and so 9.20 or 9.2 is bigger). But there are other ways to look at the question. One such way is to read them as “2” and “11” respectively. In that way of doing things, 11 is bigger than 2 and so 9.11 is bigger than 9.2. Which interpretation should the AI pick? That last step (which interpretation decides?) is where the AI error arises:

“It’s not that “I can’t do the math” – it’s that the mathematical interpretation is one pattern among many, and not always the dominant one.”

This is almost certainly why most kids get it wrong in the beginning, at least until they get enough practice. And even then, only in the context of maths classes and tests! Ask them again in real life, with no context, and many will again be wrong.

 

Onto the r’s in “strawberry” question. This error, as the AI explains its “thought” process, is rooted in the fact that AI’s don’t break words into letters – why would they? Meanings lie in the entire word or sentence, not individual letters. So the word “strawberry” is treated as a “unified semantic unit”, and “not naturally decomposed into individual letters”. Therefore, when asked the number of r’s, the AI’s architecture doesn’t have any way to know the answer. Instead, it turns to what it “thinks” are the best or most relevant indirect ways within its architecture. And it turns out none of those indirect ways lead to the right answer!

 

When I saw the reasons, I realize the AI’s are becoming like us humans, are thinking like us and are making the kinds of mistakes humans could make with those questions. Should we be worried? How often do they make these kinds of mistakes? Would we even notice? On the other hand, if they make errors like this, are they really good enough to take over so many jobs?

Comments

Post a Comment

Popular posts from this blog

The Thrill of the Chase

So many people complain about the difficulty in making decisions while shopping. Do you buy Brand X that gives 20% extra free? Or Brand Y that has 10% off? Or Brand Z that gives something else free with it? Do you buy that flat screen TV today or wait for the sale at Diwali? Now since the customer is king, shouldn’t companies be simplifying pricing choices? Like maybe offering consistently low prices all the time? Not so fast, as the US store, JC Penney discovered to its cost. Turns out that while people do complain about pricing choices, they are irrational. Also, they enjoy the thrill of the chase. Huh? The word “sale” acts like a siren call for many people: it lures them in to make that purchase. Consistently low prices don’t do the same. Besides, if a competitor lowers their prices just one time of the year, and Poof! Suddenly that low price claim becomes suspect in shoppers’ eyes. Then there are those sites that offer coupons: have lunch at Restaurant A on Wedne...
Read more

Chess is too Boring

“Chess has problems at the grandmaster level...They're not getting a great game”, wrote Christian Donlan . Amen to that: Anand and his challenger, Magnus Carlsen drew their first game in just 16 moves. It gets worse: games between the top 20 end in draws 60% of the time. And Donlan adds that “games often end in resignation when an actual victory starts to loom on the distant horizon.” That often means that the average player (or fan) often has no clue why one guy won (since he can’t see that far ahead)! Then there’s all that memorization (of openings, of end games and everything in between). All that memorization is sort of against the spirit of the game, says Zachary Burns, “We want it to be a game of creativity and positional play.” Many have tried to change the rules of the game in the past for these exact reasons, including Bobby Fischer with his Chess960 variant. More recently, David Sirlin tried to make things interesting via his variant, Chess 2. So what are his c...
Read more

Europe #3 - Innsbruck

The last city we visited in Europe was Innsbruck in the Austrian Alps. After the cold and windy experience (at times) of Paris and Netherlands, Innsbruck was a wonderful change – it didn’t feel that cold, it was not windy all of which made it easy to walk around. Something absolutely necessary given how scenic the place is.   Since the hotel check-in time was much later in the day, we left the baggage and went to the Swarovski HQ-cum-museum about an hour by bus from Innsbruck. The crystal exhibits were brilliant – colorful, bright, shiny. Jewellery aside, it was a display of what one can do with crystals. While my wife and daughter loved the jewellery, I liked the superhero themed exhibits of Batman, Spiderman, the Hulk, and Star Wars. We didn’t buy anything there, though we did buy stuff in the Swarovski store in Innsbruck – jewellery, not the superhero stuff.   Innsbruck is a major tourist attraction during the skiing season but now in summer, we saw many groups of ...
Read more