09-07-2026, 06:32 PM
Hi @JoeyB,
That's exactly the risk I was trying to avoid. Both instances of the duplicate are stripped before the classifier ever sees the text — the model is completely blind to them.
The logic in the code provided with the paper is straightforward. When the function detects a repeated word, it flags the page as "infected" but doesn't add the word to the cleaned list. The next iteration catches the second instance too, so both get dropped:
if i < len(words) - 1 and words == words[i+1]:
is_infected = True # flag the page, but don't add the word
else:
if i == 0 or words != words[i-1]:
cleaned_words.append(words) # continue with next word
With your example qokedy qokedy chol daiin, both tokens are gone. The model only sees chol daiin and predicts the page label from the surrounding 30 word context alone.
The fact that it still reaches p < 0.05 against 1,000 shuffled baselines is what makes the result interesting … the duplicates clearly co-occur with a specific syntactic environment, not at random.
Thanks for pushing on this, it's a fair point.
Alfredo
That's exactly the risk I was trying to avoid. Both instances of the duplicate are stripped before the classifier ever sees the text — the model is completely blind to them.
The logic in the code provided with the paper is straightforward. When the function detects a repeated word, it flags the page as "infected" but doesn't add the word to the cleaned list. The next iteration catches the second instance too, so both get dropped:
if i < len(words) - 1 and words == words[i+1]:
is_infected = True # flag the page, but don't add the word
else:
if i == 0 or words != words[i-1]:
cleaned_words.append(words) # continue with next word
With your example qokedy qokedy chol daiin, both tokens are gone. The model only sees chol daiin and predicts the page label from the surrounding 30 word context alone.
The fact that it still reaches p < 0.05 against 1,000 shuffled baselines is what makes the result interesting … the duplicates clearly co-occur with a specific syntactic environment, not at random.
Thanks for pushing on this, it's a fair point.
Alfredo