Here is the cross-check with a shuffled corpus (words and lines):
Median group size: 15.0
Average similarity in groups: 0.633
Min similarity in groups: 0.600 (Threshold)
Max similarity in groups: 0.816
Exact matches (Actual == Predicted): 93 / 3885
It can be observed that the candidates are distributed much more evenly across the corpus, but the number of candidates in a group decreases dramatically. The number of exact matches is virtually unchanged (even slightly increased).
Conclusion: It appears that certain sections of the VMS bear a greater resemblance to other lines than others. This is not a coincidence, but indicates a structure within the text.
Median group size: 15.0
Average similarity in groups: 0.633
Min similarity in groups: 0.600 (Threshold)
Max similarity in groups: 0.816
Exact matches (Actual == Predicted): 93 / 3885
It can be observed that the candidates are distributed much more evenly across the corpus, but the number of candidates in a group decreases dramatically. The number of exact matches is virtually unchanged (even slightly increased).
Conclusion: It appears that certain sections of the VMS bear a greater resemblance to other lines than others. This is not a coincidence, but indicates a structure within the text.
