Confirmation that spaces are really spaces

[asteckley]

What I did was apply a Byte Pair Encoding (BPE) segmentation model. This model does not need to know anything about the language. It simply takes the entire text as a sequence of characters with no spaces or punctuation, identifies which pairs of characters are most frequent, and merges them. After a certain number of merges, the model produces a set of segments that can be interpreted as statistical morphemes. With these segments, you can try to reconstruct the original words by segmenting them into the most likely units, and then compare whether these boundaries match the original spaces.

Using BPE in this way is a useful and interesting approach and your conclusions seem reasonable -- providing they aren't interpreted for more than what they are. I think you have to be careful with using the word "morpheme" though, since it suggests that each one carries some linguistic meaning or grammatical function, and by extension that we are dealing with words in a linguistic sense. BPE doesn't guarantee that at all.

You did initially qualify it as "statistical morphemes"  (Not sure what that exactly is, but at least it distinguishes it from actual morphemes.) but after that you seem to treat them more like actual morphemes. Maybe it would be clearer and more accurate to just call them “segments”, or something like that, to avoid implying full linguistic status unless (until?) there is separate evidence for that. They could be (as an example) simply atomic components of some structured cipher system.