I don’t think the strawberry thing is an hallucination. Here’s why:

If my daughter came to me and said “how many “r”s are there in the word “strawberry” I would assume that she’s asking me whether it’s spelled “strawbery“ or “strawberry.” In this context, I would tell her that there are two “r”s in strawberry, probably without ever even considering the first “r”. Understanding my answer, she would then go and spell it correctly.

The term “non-autistic” comes up when discussing ideal AI behaviour. While the answer “3” is more technically accurate than “2”, it shows a level of contextual interpretation which I believe we want LLMs to exhibit. Of course, we may want a technically accurate answer in some cases, and we can get that answer by phrasing the question technically, or in a way that eliminates the more conversationally rational but less mathematical interpretations (pictured).