+- The Voynich Ninja (https://www.voynich.ninja)
+-- Forum: Voynich Research (https://www.voynich.ninja/forum-27.html)
+--- Forum: Analysis of the text (https://www.voynich.ninja/forum-41.html)
+--- Thread: Measuring Long-Range Structure in the Voynich Manuscript (/thread-5380.html)
RE: Measuring Long-Range Structure in the Voynich Manuscript - quimqu - 19-02-2026
(19-02-2026, 12:28 PM)nablator wrote: You are not allowed to view links. Register or Login to view.(19-02-2026, 12:17 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.It is this formula. It is essentially the mutual information between characters separated by exactly d positions, averaged over all such pairs in the corpus.
That's what worries me (potentially, I don't know, but I suspect a possible issue there): how do you average the MI of all the pairs at distance d? A simple mean?
For each distance d, I do not average individual MI values. I treat all character pairs at that distance as samples of two random variables: the character at position t and the character at position t+d. I then compute the mutual information between those two variables in the standard way, using their empirical joint and marginal distributions. So there is only one MI(d) per distance, not an average of many MIs. This is the python code for the calculation:
Code:
# =========================
# MI(d) estimator (sparse joint, smoothed marginals)
# =========================
def mutual_information_lag_sparse(ids, lag, alpha=0.5):
x = ids[:-lag]
y = ids[lag:]
n = len(x)
if n <= 20:
return np.nan
V = int(max(ids)) + 1
cx = np.bincount(x, minlength=V).astype(np.float64)
cy = np.bincount(y, minlength=V).astype(np.float64)
px = (cx + alpha) / (n + alpha * V)
py = (cy + alpha) / (n + alpha * V)
joint = defaultdict(int)
for a, b in zip(x, y):
joint[(int(a), int(b))] += 1
mi = 0.0
inv_n = 1.0 / n
for (a, b), cnt in joint.items():
pxy = cnt * inv_n
mi += pxy * (math.log(pxy) - math.log(px[a]) - math.log(py[b]))
return float(mi)RE: Measuring Long-Range Structure in the Voynich Manuscript - nablator - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - quimqu - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - Fontanellean - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - Mauro - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - quimqu - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - quimqu - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - nablator - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - quimqu - 19-02-2026
RE: Measuring Long-Range Structure in the Voynich Manuscript - nablator - 19-02-2026