A mathematical approach to double words, conditional logic and the missing pages

[Vuk88]

(I apologize that I have not fully digested other people's comments and your replies. From what I read there, it seemed clearer if I responded directly to your initial post and your zenodo paper only.)
What you have done is very interesting and possibly valuable work, but I am unclear on several aspects of it.

Hi Andrew,
First of all, thanks for your careful reading and for giving me the opportunity to clarify the methodology, since the brevity of the initial post didn't really allow me to explain everything systematically.
I'll try to answer your questions.

First, if I understand correctly, your first chart shows the words with the highest correlation — that is, the unique words (aka word types) most likely to occur as doublet sequences — and the words with the lowest correlation — that is, the words least likely to occur that way. [...]
If so, then the complete bar graph would presumably show a continuous trend of words, with correlations decreasing gradually from 0.933 for “sy” down to 0.665 for “chees.” In other words, your chart has clipped out the long middle range of a continuous trend related to how often unique words occur in doublet form -- no indication of clustering per se.

To begin with, as you correctly pointed out, the logistic regression produces a continuous probability spectrum (ranging from a high probability of generating doublets to a high probability of inhibition). The bar chart deliberately clips the middle range to highlight the extremes, namely the highest positive and negative coefficients. Actually, the goal of this chart wasn't to prove "clustering" per se, but rather to show that the doublet phenomenon is not uniformly distributed across the vocabulary. Specific words act as heavy "syntactic wrappers" (red), while others act as strong "inhibitors" (blue).

The coloring of red and blue in that cluster plot is presumably assigned simply by the sign of the z-direction value. [...]
So is this chart not just showing the positive vs negative sign of an unidentifiable metric in the low-dimension reduction from the t-SNE algorithm?
Or did you instead use a seperate clustering algorithm on the points in the t-SNE output space in order to define two clusters?

Regarding how the clusters were defined and what the colors represent, I can confirm that the red and blue colors aren't the result of a clustering algorithm applied to the t-SNE coordinates. They are predefined labels imported directly from the logistic regression (Red = Main Trigger Words; Blue = Main Inhibitors). The t-SNE algorithm was used exclusively to reduce a high-dimensional contextual vector space (Word2Vec/TF-IDF) into a 3D space. The fact that the pre-labeled Red and Blue words naturally separate into distinct regions in the t-SNE space demonstrates that the words triggering doublets and those inhibiting them do not share the same contextual environments. They belong to different syntactic and semantic classes.

With the Monte Carlo simulations and the violin plot, again, if I understand your description correctly, the predictive accuracy concerns whether one can predict whether a given word belongs to one cluster rather than another. Is that correct?
[...]
But again, I am not clear how you conclude that this structure has anything to do specifically with word doublets, as opposed to other structuring effects, (such as the self-citation patterns described by Timm or the differing vocabularies found in Currier languages A and B).

As for the tests conducted, I can tell you that in each of the 1,000 permutations, the order of the words in the manuscript was completely randomized. However, I strictly preserved the exact global vocabulary, the frequencies of every single word, and the word lengths, while also standardizing the page lengths. I then ran the exact same predictive pipeline (Logistic Regression -> Accuracy Score) on the shuffled text just to see if the model could still predict the appearance of doublets based on the surrounding vocabulary. The predictive accuracy collapsed to about 50%, so pure randomness.
This proves that the doublet phenomenon is not an artifact of Zipf's Law, global word frequency, or macro-scale vocabulary differences like Currier A and B. It is a strictly local, non-random syntactic rule based on word adjacency. Timm's "self-citation" hypothesis would struggle to explain why specific "trigger" words manage to reliably predict the onset of a doublet sequence across the entire corpus.
To eliminate any doubt that these results might be polluted by transcription noise, I should also clarify the preprocessing methodology. Before running any algorithm, I applied a strict purification filter to the entire dataset. Any word containing uncertainty markers, non-alphabetic characters, or anomalous punctuation in the Takahashi transcription (e.g., `dar!o!m`) was cleaned to extract only its pure root (`darom`). So the mathematical models worked exclusively on the real linguistic data, eliminating at the source the risk of statistical "hallucinations" caused by transcription noise.
Additionally, returning to your point about the Currier A and B dialects, I verified that the syntactic rule I discovered ("Trigger -> Doublet") fits perfectly across both dialects. Even though the Herbal section (Currier A) uses a completely different set of nouns compared to the Recipes section (Currier B), the exact same "Trigger" words are used to close data blocks in both sections. It is undeniable that this observation raises serious doubts about the "language simulation device" hypothesis, since it's hard to imagine how a random generator or a medieval hoaxer could maintain such a rigid mathematical rule intact...

As I understand it, you have plotted the density of word doublets per folio page and compared that to the thematic assignments that have been observed through folio sequence of the manuscript. And you have found that five particular folios stand out by having four to five times as many word doublets as all the other folios. Those five pages each happen to either be the terminal page of a thematic section, or (at least in the last of the five cases) to precede a section of missing folios. Is that correct?
[...]
(By the way, the work about to be published by Colin Layfield and Lisa Fagin-Davis might be worth reviewing when it becomes available soon, to see if it affects the folio sequence positioning part of your finding.)

Finally, you noticed that the last analysis (the massive spike of doublets on terminal folios, like f108) seems a bit disconnected from the previous machine-learning charts. You are absolutely right, it is indeed a structural observation of the manuscript as a whole. However, it perfectly closes the circle of this research.
While the machine-learning models explained how the mathematical rule works at the single-line level (the "trigger" words generating doublets), the analysis of the terminal folios explains why the author (or authors) created it. The fact that this specific mathematical formula is so heavily concentrated precisely at the end of thematic sections proves that these double words act as actual "End of File" or "End of Database" markers, used to close out blocks of information.
Thanks for pointing out the upcoming work by Colin Layfield and Lisa Fagin-Davis. Since the codicological sequence of the quires is highly debated, any new data on the original binding order will be crucial to see if these syntactic "spikes" align even more perfectly with the original transitions between the fascicles.
Thanks for your meticulous observations.
Alfredo