27-10-2026, 06:22 PM
(This post was last modified: 27-10-2026, 06:24 PM by Jorge_Stolfi.)
(26-10-2026, 05:00 PM)npcompl33t wrote: You are not allowed to view links. Register or Login to view....But let me say that I think that you are making your task harder than necessary by using your Golf split.
1) Whenever you split a section in two you are adding one more plot to create and analyze, and another bunch of stuff that you must comment about and explain. You may also need to add one more degree of freedom to your models, which may create spurious artifacts even far away from that split section. Like going from quadratic to cubic polynomial fitting as go from three points to four.
In particular, the split of Herbal-A into Golf 1a, 1b, and 1.5a seems unjustified. Or the splitting of Herbal-B into Golf 3a and 3b. Or the splitting of Stars into 3c and 4c. In the plots I have seen, there is no clear separation between those sub-sections in each section.
Even if these sub-sections were scribed by different Scribes (a claim that is still contested!), it is unlikely that a mere change of Scribe would go together with a detectable change in the contents or any significant text statistics. The two scribes would be copying from the same draft. Even if there are differences between the two Scribes that result in difference text statistics (like different rates of errors, of missing and bogus spaces, and of use of abbreviations), at this time these differences are better ignored than analyzed.
2) Moreover, if you split a section X into two sections X1,X2 with no other justification than some statistical criterion C, such as projection onto some axis or curve, you may trick yourself into believing that there is a "gradient" or "evolution" from one to the other, "demonstrated" by them having different values of C. Even if all the points of X were independent drawn from the same Gaussian distribution in R^n. That seems to be the case for Golf 1a, 1b, and 1.5a.
3) If, on the other hand, you merge two separate sets of pages X1, X1 into a single section X, just because they have similar statistics, you are asking for trouble. There will probably be unexplainable anomalies in the statistics of X. If you are looking for an order, temporal or otherwise, it may be that X1 and X2 were actually miles apart in that order.
That is the case of Golf 2a, "Late A/Pharma". It is damn obvious that the Pharma pages are unlike any Herbal pages -- in format, depth, style, focus, even topic (in the strict sense). If there is a section with no justification for joining or splitting, it is that one...
4) As for the oddball pages (with no illustrations, or with unclassifiable illustrations like You are not allowed to view links. Register or Login to view. or f85r2), they should be just left out. Including them in the analysis will only add clutter and distraction. Later, if any clear conclusion is reached about the major sections, those pages can be re-examined -- individually -- to see how they fit into it.
5) And, finally, consider seriously using word statistics rather than character and digraph statistics. The former are directly related to meaning. The latter are complicated projections of word frequencies that mix together hundreds of unrelated words. The frequency of word-initial "he" in English the sum of the frequencies of the words "he", "her", "herb", "hen", "help", "head", "heat", "heal", "here", "heart", "hearth", "held", ..., and "hemorrhoids". Why would that sum be a better indicator of semantic similarity, language evolution, etc, than the frequency of "herb" alone? Or the joint frequency of {"heart", "liver", "stomach', "spleen"}?
All the best, --stolfi