15-11-2025, 10:13 AM
(15-11-2025, 09:48 AM)nablator wrote: You are not allowed to view links. Register or Login to view.(14-11-2025, 11:41 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.If the spaces were decorative or had no function, the model should not be able to recover them with this level of accuracy.
On the contrary, the more predictable they are, the less information they can carry. 100% predictable = 100% useless.
It wasn't easy at all to find a model that detects morphemes that match spacing. As I explained, I started training models with natural languages, so I could check their accuracy. I must say that at the beginning it was almost impossible to get good results. The models tend to reduce the morphemes to a single char and then it makes nonsense. Think that the model starts with the text in a single line... I really think it is not that evident.