what do you mean by spell fine? They’re just emitting the tokens for the words. Like, it’s not writing “strawberry,” it’s writing tokens <302, 1618, 19772>, which correspond to st, raw, and berry respectively. If you ask it to put a space between each letter, that will disrupt the tokenization mechanism, and it’s going to be quite liable to making mistakes.
I don’t think it’s really fair to say that the lookup 19772 -> berry counts as the LLM being able to spell, since the LLM isn’t operating at that layer. It doesn’t really emit letters directly. I would argue its inability to reliably spell words when you force it to go letter-by-letter or answer queries about how words are spelled is indicative of its poor ability to spell.
what do you mean by spell fine? They’re just emitting the tokens for the words. Like, it’s not writing “strawberry,” it’s writing tokens <302, 1618, 19772>, which correspond to st, raw, and berry respectively. If you ask it to put a space between each letter, that will disrupt the tokenization mechanism, and it’s going to be quite liable to making mistakes.
I don’t think it’s really fair to say that the lookup 19772 -> berry counts as the LLM being able to spell, since the LLM isn’t operating at that layer. It doesn’t really emit letters directly. I would argue its inability to reliably spell words when you force it to go letter-by-letter or answer queries about how words are spelled is indicative of its poor ability to spell.
I mean that when you ask them to spell a word they can list every character one at a time.
Well that’s a recent improvement. GPT3 was very bad at that, and GPT4 still makes mistakes.