The evolution of P/F vs T/K raises important questions

[Jorge_Stolfi]

It's hard to look at these charts and come to the conclusion that P/F words are the same as K/T words.

But they are not expected to be.  Not in Stars, at least.

Even if one ignores the SPS=SBJ claim, Stars is fairly likely to be a materia medica, a list of remedies with applications.  Each entry may refer to any kind of substance: vegetal, animal, mineral, artificial (maybe even to a procedure like bloodletting or warm packs); with emphasis on its uses, maybe preparation and dosage, but with little or no description.

Therefore, in Stars, Herbal-A, and Herbal-B the first word of each parag (and possibly also the next few words) are very likely to be the name of the thing which the parag is about. As such, each of those words is very unlikely to be used again in the same parag.  It may be used a few times in other parags: like, there may be an entry for "black cherry" and another one for "blue cherry", or "true silphium" and "Nigerian silphium"; and a common word like "black" or "fat" may be used in the titles and/or bodies of several entries.

On the other hand, after a certain number N1 of tokens, we timidly expect that the word frequencies in the head lines will be similar to those in the rest of the parag, once the gallows are collapsed.

Again, based on your previous plots I would say that, in these three sections, there are three uses of puffs: (1) the first glyph of a parag is very often puffed if it is puffable.  Then (2) 4-8% of the glyphs of the rest of the head line are puffed -- possibly randomly, and only some fraction of the puffable ones.  Then, (3) in Herbal-A and Herbal-B -- but not in Stars -- about 2% of the words in the body get puffed-up.

For Bio, on the other hand, the previous plots seem to suggest that puffs are not used for (2), only for (1) and (3) -- and the latter applies to the head line as well as the body lines.  And Pharma  maybe has only type (3) puffs.

To do this, I did something like your most recent suggestion. I replaced all characters through the first gallows with G, and i removed e's.

In my current understanding of the structure of the words, only the first "e" after a gallows or bench is a modifier for that letter.  A double "ee" is counted as separate letter, in the same class as "ch" and "sh". A triple "eee" is ambiguous; it may be an "e"-modifier and an "ee", or an "ee" with its "e" modifier. (I arbitrarily assume the former, but this should be investigated more.)  Thus "chedy" in my model is {che}{dy} while {cheedy} is {ch}{ee}{dy} and cheeedy" is {che}{ee}{dy}. 

In fact I have the vague suspicion that "ee" is a "ch" where the ligature was omitted by the Scribe or faded away; or, conversely, "ch" is "ee" with a ligature usually added for clarity.  Anyway, deleting all "e" is a rather drastic hack, that may map words that could be the same (like "keedy" and "kchdy") to different skeletons ("Gdy" and "Gchdy")

On the other hand, I agree that collapsing k/t with ke/te is necessary, because of the hypothesis that the puffs may be fancy versions of either of those four.

By the way, for any word-based analyses, I would first correct/neutralize some possible errors (by the Scribe or the transcribers) and expand some likely abbreviations, as follows:

• map [A-Z] to [a-z]
  
• map "ih" to "ch"
  
• map "i[ktpf]h" to "c[ktpf]h"
  
• map token-final "i[rs]" to "iin"
  
• map token-final "m" to "iin"
  
• delete token-initial "qo" (or split it off as a separate token)
  

Then, for the analysis you did above, I would collapse the gallows a bit differently:

• map "eee" to "eEe"
  
• map "ee" to "Ee"
  
• map "hh" to "he"
  
• map "c[pfkt]h[e]?" to "H"
  
• map "[pfkt][e]?" to "G"
  
• map back "Ee" to "ee" (or maybe "ch")
  

I would also consider deleting the "circles" [aoy] because they seem to be somewhat "floating" and not strongly attached to the words.

I know you don't like the golf groups, but I think the stars split is significant, and that we see a register shift in terms of P/F usage between these two stars sections.

I can accept that there may be separable sub-clusters of pages in the Stars section.  Have you seen my post about the sub-section structure of the Shennong Bencao?  Independently of the claim SPS=SBJ, the Stars section may be similarly organized into subsections with very different classes of remedies, each with its own vocabulary (and hence with distinct digraph distributions). 

In fact,  the sub-clusters may be more evident if the data points are single parags rather than pages, since the sub-section boundaries may fall in the middle of pages.  Or even the subsections may have been scrambled, as happened in at least one digital edition of the SBJ.

All the best, --stolfi