Capstone B: the tokenizer
Project B is a search index, and it starts at the bottom: a function that turns text into the words an index can hold. Everything above it depends on this one decision, because the words of the index are the words of the query, and a tokenizer that is wrong in one direction finds nothing while a tokenizer that is wrong in the other one finds everything twice.
The tokenizer of this project is four lines and three rules: lower-case the text, split on everything that is not a letter, and drop a small set of words that carry no meaning. The lesson builds the folder, writes the function and runs a check program that prints the words of a sentence, the count, the distinct count and the two edge cases.
The folder of project B
The folder is named search and starts like the folder of project A: a manifest with the two scripts, a lint config with the two Node.js globals, and a src folder. The manifest has no main field yet, because the library has no single entry until the module of lesson 133 gives it one.
mkdir search
cd search
mkdir src
The file package.json:
vi package.json
{
"name": "search",
"version": "1.0.0",
"description": "An inverted index over a few documents",
"type": "commonjs",
"scripts": {
"test": "node --test --test-reporter=dot",
"lint": "eslint ."
},
"devDependencies": {
"@eslint/js": "^10.0.1",
"eslint": "^10.14.0"
}
}
The file eslint.config.js:
vi eslint.config.js
const js = require('@eslint/js');
module.exports = [
js.configs.recommended,
{
languageOptions: {
ecmaVersion: 'latest',
sourceType: 'commonjs',
globals: {
console: 'readonly',
process: 'readonly',
},
},
},
];
The two files are the ones that project A ended with, with the name changed. A second project that starts from the same two files is the cheapest proof that they are a template rather than a decision about one program.
Case, letters and the split
The split takes a regular expression: one or more characters that are not an a to z. Because the text was lower-cased first, the classes do not need the capital letters, and the function is the same for a query and for a document, which is the property the whole index rests on.
The split leaves an empty string at the start of a text that begins with punctuation and at the end of one that ends with it, which is why the filter drops the empty strings. A comma, a full stop, a digit and a newline are all separators, so 12 apples gives one word and don't gives two.
The file src/tokenizer.js:
vi src/tokenizer.js
const STOP_WORDS = new Set(['a', 'an', 'and', 'are', 'for', 'in', 'is', 'of', 'on', 'the', 'to']);
function tokenize(text) {
const words = text.toLowerCase().split(/[^a-z]+/);
return words.filter((word) => word !== '' && !STOP_WORDS.has(word));
}
module.exports = { STOP_WORDS, tokenize };
There is no stemming and no accent folding here: cats and cat are two terms, and a word with an accent becomes two words or none. Both are real choices with a cost, and the exercises ask what the cost is. The lesson on regular expressions covers the character class the split uses.
The stop words
A stop word is a word that every document holds and that no query is about: the article, the preposition and the conjunction. Keeping them would give every document a point for the word the, and a query for the fox would rank the document that says "the" most often. Dropping them is the cheapest relevance decision a small index can make.
The set is exported, so the check prints it and the tests of lesson 133 loop over it rather than copying the list. A Set answers has in one step whatever its size, which the lesson on sets explains.
Eleven words is a small list on purpose. A large list is a language-specific artefact that has to be maintained, and a search over three documents does not need one; the exercise at the end asks what the list would look like for a real corpus.
The check
The check program prints the words of a sentence, how many there are, how many are distinct, the stop list, an empty text and a text of numbers and punctuation. It is the same shape as the check of project A: a few calls and one line per property, so a reader can compare the page with a run.
The file check.js:
vi check.js
const { STOP_WORDS, tokenize } = require('./src/tokenizer');
const text = 'The quick brown fox jumps over the lazy dog. The dog sleeps!';
const words = tokenize(text);
console.log('the words: ' + words.join(' '));
console.log('the count: ' + words.length);
console.log('the distinct words: ' + new Set(words).size);
console.log('the stop words: ' + [...STOP_WORDS].join(' '));
console.log('an empty text: ' + tokenize('').length + ' words');
console.log('numbers and punctuation: ' + tokenize('12 apples, 3 pears; 4 plums.').join(' '));
console.log('the case does not matter: ' + tokenize('Dog DOG dog').join(' '));
node check.js
the words: quick brown fox jumps over lazy dog dog sleeps
the count: 9
the distinct words: 8
the stop words: a an and are for in is of on the to
an empty text: 0 words
numbers and punctuation: apples pears plums
the case does not matter: dog dog dog
Nine words and eight distinct ones: dog appears twice. The sentence held the word the three times and it is gone, while over stayed, because it is not on the list. That is the list deciding the output, and it is why the list is data rather than a condition inside the function.
The last two lines are the edge cases. An empty text gives no words rather than one empty word, and a text of digits and punctuation gives the three words between them, which is the behaviour a document of prices needs.
Exercises
- Add over to the stop words and run the check. Which two numbers change, and which line of the output shows it?
- Tokenize 'don't stop' and 'café noir'. How many words come out, and why is the second answer worse than the first?
- Keep the case and search for Dog. Which part of the function would every caller have to repeat?
- Replace the filter with a loop that builds the array. Which version is easier to read, and which one allocates fewer arrays?
- Add a rule that drops words shorter than three letters. Which word of the sentence disappears, and which useful word of English would the rule also drop?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why does the tokenizer lower-case the text before it splits it?
Answer
So a document and a query produce the same word for the same term
One case for both sides of the search is the property the index depends on.
-
What does this program print?
const text = "Hello, World!"; const words = text.toLowerCase().split(/[^a-z]+/).filter((word) => word !== ""); console.log(words.join(" "));Answer
hello world
The comma and the exclamation mark are separators, and the case was lowered first.
-
What is a stop word?
Answer
A word that every document holds and no query is about
Keeping them would give every document a point for a word that says nothing.
-
What does this program print?
const stop = new Set(["the", "a"]); console.log(stop.has("the"), stop.has("The"), stop.size);Answer
true false 2
A set compares exactly, which is why the text is lower-cased before the lookup.
-
How many words does
tokenize("12 apples, 3 pears.")return?Answer
Two
The digits are separators, so only the two fruit names are words.
-
What does this program print?
const text = "...abc..."; console.log(text.split(/[^a-z]+/).filter((word) => word !== "").length);Answer
1
The punctuation on both sides leaves empty strings, and the filter keeps the one real word.
-
What goes wrong when the tokenizer does not drop stop words?
Answer
Every document scores for the common word, so the ranking says little
A term in every document separates nothing, and the score stops being a difference.
-
What does this program print?
console.log("Dog".toLowerCase(), "Dog".toLowerCase() === "dog");Answer
dog true
toLowerCase returns a new string, and the comparison of the two lower-case forms passes.
-
Why is the stop list a Set rather than an array?
Answer
A Set answers a lookup in one step and cannot hold a duplicate
The list is read once per word, so the lookup is on the hot path of the build.
-
What does this program print?
const words = ["a", "b", "a"]; console.log(words.length, new Set(words).size);Answer
3 2
The array keeps the duplicate and the set does not.
The programs on this page were run with Node.js 24.22.1 and linted with ESLint 10.14.0 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev