03-07-2025, 08:01 AM
More thoughts on this- decoder-only Transformers generally do well on many nonlinguistic seq2seq tasks (arithmetic, DNA to protein mapping, etc) when trained totally from scratch- if you're using a pretrained model, though, what it's pretrained on matters a lot.
I STRONGLY suspect that what's happening here is that, if you're using a pretrained model (which GPT-nano is), the non-Voynich texts were of the same language as many others in the training corpus, so the model effectively "already knew Latin" to some extent. If it's never seen EVA, though, the model is going to need a lot of fine tuning to learn it. That doesn't necessarily mean there's no linguistic structure present, though- just that EVA is likely dissimilar to the pretraining data.
Given that Voynich has much lower entropy than language generally, I strongly suspect a small model could learn it very well if trained from scratch- I'd try that next, if you've got the compute for it.
I STRONGLY suspect that what's happening here is that, if you're using a pretrained model (which GPT-nano is), the non-Voynich texts were of the same language as many others in the training corpus, so the model effectively "already knew Latin" to some extent. If it's never seen EVA, though, the model is going to need a lot of fine tuning to learn it. That doesn't necessarily mean there's no linguistic structure present, though- just that EVA is likely dissimilar to the pretraining data.
Given that Voynich has much lower entropy than language generally, I strongly suspect a small model could learn it very well if trained from scratch- I'd try that next, if you've got the compute for it.
![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)