Chronological Path Stability/Uncertainty Analysis

[Jorge_Stolfi]

The clusters as you have split them are not organized by section.

Not perfectly, indeed.  But even the approximate match says that the clustering of the points is significant, and not an artifact of the MDS algorithm or random clumping.

So there is not clean section splits here at all, and in fact divides some groups in ways you previously rejected (like splitting stars).

Here is my analysis of those plots, in detail.

When projected onto the MDS1xMDS2 plane, the data points (page statistics vectors in R^n = R^256) are definitely split into two main clusters A+B and C+D.  There is a broad band between them, about 0.1 MDS units wide, where the density of dots is much lower than the maximum density in the A+B and C+D subsets -- indeed, practically zero.   

Thus an adequate model for that set of projected points would be a process that, at each step, first chooses one side j, 1 (A+B) or 2 (B+C), and then generates random point according to some lumpy but smooth two-dimensional probability distribution Hj, which has almost all of its "mass" on side j of that band, and is negligible on that band and on the other side of it. 

On the other hand, the working hypothesis about the creation of the VMS is that each page was meant by the Author to belong to one specific section Sk, which had a specific mean digraph prob vector Ck, a point of data space R^n.   However, the text on each page p in that section is finite and says something different, so its actual digraph freq vector F(p) is not Ck but a random sample of an n-dimensional  probability distribution Gk on R^n, with mean Ck.

It is encouraging that the two clusters A+B and C+D that can be seen on that 2D plot mostly match the traditional division of the VMS into sections by figure topic.  Namely, the cluster A+B seems to comprise almost all and only the projections of your groups 1a+1b+1.5a+2a ("macro-group Golf-A").  I suppose they are practically the same as HerbalA+Pharma+AstrCosmo ("macro-section VMS-A").  Likewise the dot clusters C+D seem to comprise almost all and only the projections of your groups  5a+6+3a+3b+3c+4c+4a ("macro-group Golf-B"); which, IIUC, are practically the same as HerbalB+Zodiac+Bio+Stars ("macro-section VMS-B).  Is this correct?

The mostly-clear band between the two visual clusters projected on the MDS1xMDS2 plane lets us pick a linear separator on that plane, a line L such that most of the A+B cluster projects to the left of L and most of the C+D cluster projects to the right.  Then, given any page p, we can infer the index j in that model above by projecting it onto that plane and checking which half-plane of L contains the point.  

The classification implied by that line L that I picked agrees quite well with the split into the two Golf macro-groups and the two VMS macro-sections above.  There are about 4 pages of Golf-B that lie into the left half-plane.  Those points belong to Golf groups 5a or 3a (can't tell, the colors are too similar). Are those pages indeed in  VMS-B?  

However, I picked that line by eye, and it obviously not optimal.  I could have moved it a little and maybe reduce the "errors" to one point of each Golf macro-group on the "wrong" side.  Ditto for the VMS macro-groups, if they do not coincide with the Golf macro-groups.  Like this:
   
Makes sense so far?

(to be continued)

All the best, --stolfi