A mathematical approach to double words, conditional logic and the missing pages

[Vuk88]

The Logistic Regression doesn't use an n-gram model, it uses a Bag-of-Words approach at the page level. So it's not looking at physical adjacency, but rather page-level co-occurrence. It found that words like shedy and chees heavily turn on or off on the exact same pages where the doubles appear.

Thanks, it's clearer now.

On the sample size issue with the 36 chees vs 298 doubles, I had the exact same concern about statistical chance. That's precisely why I ran a Monte Carlo Permutation Test with 1000 iterations. The empirical p-value came out to p=0.0010, and the mathematical distance between the permuted random distribution and the true manuscript data is what confirms this isn't just a random fluke.

Question: did the MonteCarlo permutation reshuffle all the words, or did it keep the original doublets and reshuffled the rest? And, did it keep the page structure of the original (the number of words on each page)? I think I would be convinced your findings are not a statistical fluke with a permutation test which:

- keeps the same doublets as the original in the same positions (that is to say, on the same page as the original, given your algorithm works page by page)
- keeps the same page structure (number of words on each page) as the original
- avoids creating new doublets when reshuffling words
- and finally reshuffles words section by section (Balneo, Herbal A, Herbal B..). I say this because there are big differences in vocabulary between the sections and mixing everything together will tend by itself to dilute any statistical signal.

In any case, nice work 

Hi Mauro,
To answer your question directly...No, I strictly avoided physically reshuffling the words in the manuscript. Diluting the Currier sections or creating random doublets by chance are exactly the structural pitfalls I wanted to avoid. I invite you to read my recent reply to Andrew regarding the general standardization methodology, but here is how I bypassed the structural problem in the permutation test.
As you likely know, to avoid altering the manuscript's physical structure, I relied on Y-Scrambling (Target Permutation) rather than a physical text scramble. The text of the Voynich, its structure, and the exact positions of the words were left 100% untouched. 
I simply assigned a mathematical label ('1' or '0') to the text blocks depending on whether they actually contained a doublet or not. 
During the 1,000 permutations, the algorithm randomly shuffled only these mathematical labels, forcing the model to look for a syntactic rule on fake targets while the underlying text remained perfectly intact.
Thanks to this approach, the Currier vocabularies were never mixed, and the physical page structure was never altered.
I hope this clears up your doubts about the structural integrity of the test.

Thanks Mauro for the help you are giving me. 
What I really need right now is exactly this: to stress-test the algorithms as much as possible before moving on to the next steps of the analysis.
Alfredo