Run the code, not the answer
ChatGPT got a simple count wrong, then wrote a function that got it right. The lesson I took from it.
I was using ChatGPT to help debug some code. I sent it a 9×9 matrix and asked how many times each number appeared. The answer looked off, so I counted by hand. It was completely wrong.
After a lot of back and forth trying better prompts, I gave up on asking it to count. I asked it instead to write a function that takes a matrix and returns how many times each number repeats.
I gave it the same matrix and asked it to use the function. It still returned the wrong output. But the function itself was correct. When I ran it on my machine, I got the right result.
The lesson I took was to distrust the model's answers and trust code I can run. A model that can't count reliably can still write a correct counting function, and the function can be checked.
Ask for the code, then run it yourself.