03-07-2019, 04:15 PM (This post was last modified: 03-07-2019, 04:17 PM by Koen G.)
(03-07-2019, 03:16 PM)Koen G wrote: You are not allowed to view links. Register or Login to view.
I collected some numbers already, you can see them here (final sheet xMATTR): You are not allowed to view links. Register or Login to view.
Of the texts I checked so far, here's the percentage that have no repeating strings of the following lengths:
2: 1% 3: 6% 4: 20% 5: 32% 6: 48%
So about half of these texts never repeat a string of 6 words exactly (taking into account the 1000-word window I always used, which seems reasonably large). What may be more interesting is that 9 of my txt files never even repeat a 4-word string.
Additionally, one file (attached) does apparently not even repeat 3-word strings. It's only 1762 words, which might explain a bit.
Still, I would seriously question the claim that natural languages cannot be without repeating "phrases". If the phrase is 4 words long, I have 9 counterexamples. Make the phrase 5 words long and 20% of checked texts don't repeat any.
Reply
The following 1 user says Thank You to Koen G for this post:1 user says Thank You to Koen G for this post • nablator