Word-Frequency Analysis: Spot Repetition in Your Writing

You reach for the same three verbs without noticing. Every writer has crutch words, and the problem is structural: you can't hear the repetition because you wrote it. A word-frequency pass fixes that by turning your prose into a ranked list of counts, so the word you leaned on eleven times stops hiding inside sentences you already believe are fine. This is a mechanical check that catches what re-reading never will.

#TL;DR

• Word frequency counts every word and sorts by how often it appears — repetition becomes a number, not a hunch.
• Strip stop words like "the" and "and" first; only content words reveal real repetition.
• Zipf's law says a few words dominate any text; watch for a content word climbing too high.
• Type-token ratio and lexical density quantify how varied your vocabulary is.
• iKit's Word & Character Counter runs the whole analysis in your browser, no upload.

#Why repeated words slip past you

#The "invisible to the author" problem

When you read your own draft, you reconstruct meaning from memory rather than parsing each word fresh. Your eye slides over "leverage" for the fourth time because you know what you meant. Repetition is invisible from the inside, which is exactly why editors exist — and why a frequency count is the cheapest editor you'll ever run. It doesn't understand your argument, and that's the point: it treats every word as a token to be tallied.

#How word frequency exposes it

A frequency analysis reduces a document to pairs of (word, count). Sort that list descending and the top is always the same: articles, prepositions, conjunctions. Scroll past those and you hit the first real word — the most-used noun or verb in the piece. If that word is "utilize" and it appears fourteen times in 900 words, you've found a revision target no amount of re-reading surfaced.

#Content words vs function words

Linguists split vocabulary into two buckets. Function words — pronouns, prepositions, conjunctions, articles — carry grammar, not meaning, and they repeat constantly by design. Content words — nouns, most verbs, adjectives, most adverbs — carry the meaning. Only content-word repetition damages prose. Separating the two is the single most useful move in the whole workflow, and it's what stop-word filtering does automatically.

#How to find repeated words in your writing

#Step 1: Count every word

Normalize case, split on non-letter boundaries, and tally. The raw count of every token is the foundation; everything else is filtering and sorting on top of it. Do this before you make any judgment about what's "too much" — you want the full distribution first.

#How to count word frequency in JavaScript

The core is a few lines. Lowercase the text, extract word-like runs with a regex, and accumulate counts in a Map:

function wordFrequency(text) {
  const words = text
    .toLowerCase()
    .match(/[a-z']+/g) || [];
  const counts = new Map();
  for (const w of words) {
    counts.set(w, (counts.get(w) || 0) + 1);
  }
  return [...counts.entries()]
    .sort((a, b) => b[1] - a[1]);
}

That returns an array sorted from most to least frequent. If you want to test the regex against edge cases like hyphenated or apostrophe words before trusting it, a quick pass through a regex tester saves you from silently dropping "don't" or "state-of-the-art."

#Step 2: Strip the stop words

The top of any frequency list is noise. Filter out a stop list so content words rise to the surface. There is no single official stop list — the Wikipedia entry on stop words notes practice has ranged from 200–300-term lists down to as few as 7–12 terms — so a small, sensible set is fine for editing:

const STOP = new Set([
  "the", "a", "an", "and", "or", "but",
  "of", "to", "in", "on", "is", "it",
  "that", "this", "for", "with", "as",
]);

const meaningful = wordFrequency(text)
  .filter(([word]) => !STOP.has(word));

#Step 3: Sort by frequency and scan the top

With stop words gone, read the top 15 entries. These are your candidates. Not every one is a problem — if you're writing about databases, "database" should appear a lot. The judgment call is whether a word repeats because the topic demands it or because your phrasing got lazy. Watch especially for interchangeable verbs and adjectives: "important," "really," "just," "very," and their friends.

#What Zipf's law tells you about your draft

#The rank-frequency curve

Word frequencies aren't random. Across natural language, the frequency of a word is roughly inversely proportional to its rank: the most common word appears about twice as often as the second, three times as often as the third, and so on. This is Zipf's law, and it holds statistically across languages and authors. Plotted on log-log axes, your document's rank-frequency data falls close to a straight line.

#When a content word climbs too high

Zipf's law is a baseline expectation. When a content word muscles into the ranks normally held by function words, that's the anomaly worth noticing. "The" belonging at rank 1 is normal. Your pet adjective sitting at rank 6, above most prepositions, is a signal that you've overused it relative to how language usually distributes. You don't need to compute the curve — the intuition is enough: a meaning-bearing word should not be competing with "of" and "to" for the top slots.

#Reading the long tail

The bottom of the list matters too. A healthy draft has a long tail of words used exactly once. That tail is where your specific, precise vocabulary lives. A draft that's all high-frequency words and a thin tail tends to read as vague and padded — which is the opposite of what Strunk demanded when he wrote:

Omit needless words. Vigorous writing is concise.

— The Elements of Style, William Strunk Jr.

#Measuring lexical diversity: type-token ratio and lexical density

Frequency lists are qualitative. Two numbers turn "this feels repetitive" into something you can track across drafts.

#How to calculate type-token ratio

Type-token ratio (TTR) is the number of unique words (types) divided by the total number of words (tokens). Per the TRUNAJOD documentation, it's bounded between 0 and 1: no repetition at all gives 1, and heavy repetition pushes it toward 0. A passage of 200 words with 140 unique words has a TTR of 0.70 — fairly varied.

function typeTokenRatio(text) {
  const tokens = text.toLowerCase()
    .match(/[a-z']+/g) || [];
  const types = new Set(tokens);
  return types.size / tokens.length;
}

#Why TTR punishes long documents

TTR has a well-known flaw: it drops as texts get longer, because the more you write, the more you inevitably reuse common words. Comparing the TTR of a 300-word abstract to a 3,000-word article is meaningless — the longer piece will always score lower. To correct for this, researchers use the Moving-Average Type-Token Ratio (MATTR), which slides a fixed window (often 500 tokens) across the text and averages the TTR within each window, removing the length bias.

#Lexical density: the content-word percentage

Lexical density is a different lens: the percentage of content words to total words. The measure traces to Ure (1971), summarized in the Wikipedia entry on lexical density, who found most spoken texts sit under 40% while most written texts land at 40% or higher. Low density often means your sentences are padded with grammatical scaffolding; high density can mean they're dense to the point of being hard to read. Here's how the three metrics compare:

#A practical word-frequency workflow for tighter writing

#Build a frequency list in the browser

You don't need to write code every time. Paste your draft into iKit's Word & Character Counter and read the per-word counts directly. Because it runs entirely client-side, your unpublished draft never touches a server — worth caring about for anything under embargo, NDA, or just not-ready-yet.

#Set thresholds and revise

Turn the list into a checklist. A workable rule of thumb for content words:

• Appears 3+ times in one paragraph — almost always worth varying or cutting.
• Top-5 content word and not your core topic — find a synonym or restructure.
• A filler word ("very", "really", "just") in the top 10 — delete most instances outright.
• Every sentence starts with the same word — vary your openings.

You're not chasing zero repetition. Deliberate repetition is a rhetorical tool. You're chasing accidental repetition, the kind that signals a tired first draft.

#Cross-check with a diff before and after

After a revision pass, run the before and after through a diff checker to see exactly what you changed. It's a fast sanity check that you tightened prose without dropping a clause or mangling a sentence — and it keeps you honest that the edit actually reduced the counts rather than just moving words around. Pair the frequency list with your target length and you can trim toward a word budget without padding.

#References

• Zipf's law — rank-frequency relationship and its statistical (not exact) nature.
• Type Token Ratios — TRUNAJOD documentation — TTR definition, 0–1 bounds, and the MATTR length correction.
• Lexical density — content-vs-function-word ratio and Ure's (1971) 40% spoken/written finding.
• Stop word — why there's no universal stop list and how list sizes vary.
• The Elements of Style — Strunk's "omit needless words" rule on concision.

#Related on iKit

• Paste any text and read live word, character, and reading-time counts — the flagship guide to the counter that also powers this frequency workflow.
• Hit a 1,000-word target without padding it out — the natural next step once frequency analysis flags your filler.
• Get your SEO title and meta description to the right length — tight, repetition-free copy matters most where every character is counted.
• Pack more meaning into X's 280-character limit — cutting repeated words is how you fit a full thought in one tweet.
• Count Chinese, Japanese, and Korean words correctly — frequency analysis depends on tokenizing words right, which CJK scripts complicate.
• Estimate a talk's runtime from reading vs speaking time — another place raw word counts turn a gut feeling into a number.