Chronological Path Stability/Uncertainty Analysis

[Jorge_Stolfi]

@JorgeStolfi I checked, euclidean embedding changes almost nothing about the numbers, like 0.74% difference

Thanks again for doing all this work!

Notably, none of the above particularly changes anything. Note the parabola / arch is visible in MDS1 vs 2 in every chart, including the one with just 4 transition rows.

Sorry, but I still don't see a parabola.  Nor any good justification for splitting the sections as you have done.  So I will ignore the colors of the dots too.

What I see on those plots are four clouds with roundish shape, which can be fairly well separated by linear combinations of MDS1 and MDS2.  Something like this:
   
The fact that these four clusters are mostly correlated with the traditional sections is a strong indication that they are significant.

The separating planes shown in yellow are not perfect. The scatter of points in each cloud implies that there are points on one side of the separator that should belong to the other cloud.  But we cannot tell which are those "errors" from the plot. (Again, I am ignoring the dot colors.)

These four clouds are obviously not aligned in any of the plots.  Three of the clouds -- A,B,C -- are arranged roughly as the vertices of an equilateral triangle.  We can fit not one, but three parabolas through the centroids of those three clouds, in three different orders (apart from reversals): A-B-C, B-C-A, or C-A-B.   The first one may seem "right", but only because in that projection the A-C distance in slightly greater than A-B and B-C.

Cloud D is more compact and close to cloud C,  in a direction C-D that is almost but not quite collinear with A-C.  It clearly does not lie on any of those three parabolas.  If one adjusts either of three three parabolas to go though the midpoint of C-D, the fit to C gets worse but that of D does not get much better.  To thread though all four centroids one would need a curve of degree 3 -- but then one can fit such a curve perfectly on any four points in any space.

So, forget the parabolas.

It looks like each of these 4 clouds can be adequately modeled as samples from a single n-dimensional Gaussian distribution.   On the MDS1-MDS2 plot, clouds A, C, and D look somewhat elongated. A is elongated in the A-D direction, while C and D are stretched in the MDS2 direction.  On the MDS1-MDS3

In my opinion, the next step should be to better identify and describe these four clusters (still ignoring the point colors).  The result should be the means and principal directions of four Gaussian  distributions GA,GB,GC,GD in R^256-space that are most compatible with the thesis that the data points are samples of the distribution GA+GB+GC+GD.  I bet that there are algorithms and softwares out there that will do this, but I don't have any experience with them.  Do you?

MDS1 and MDS2 are "good enough" as projection axes because they clearly show the four clusters and their linear separation. But there may be better axes.  MDS1xMDS3 is worse than MDS1xMDS2, but maybe there are directions in the MDS1xMDS2xMDS3 space (or in the MDS1x...xMDS6 space) that make the clustering even more clear.  

Then, with those four Gaussian models, one would be better prepared to investigate whether each cloud is indeed a single cluster, or it itself consists of multiple sub-clusters.

Makes sense?

All the best,--stolfi

[npcompl33t]

Register or login to view the content

[Jorge_Stolfi]

Register or login to view the content