Linguistic Patterns Before Decipherment: A Key to Understanding Unknown Texts

[Urtx13]

First guess: are you removing lines that start with #  (full comment lines)?

I had already removed the hash and other artifacts. But I just realized the real issue: some words from the control text were accidentally being included in the total token count.

IVTT is truly impressive. In my case, it’s just a bit of healthy stubbornness, like someone trying to crack a hard riddle for fun. I want to build a pipeline that helps me figure it out on my own.

Now, after correcting everything, I’m getting exactly:
-Total tokens: 39,876
-Unique tokens: 7,886