NATURAL LANGUAGE PROCESSING / COIT29225
BBC News Classification
Which text representation gives a more accurate and efficient news classifier: TF-IDF or Word2Vec?
A reproducible comparison of multilayer perceptrons trained on term-frequency features and document embeddings, using five BBC News categories.
- Python
- scikit-learn
- Gensim
- NLTK
- MLP
- TF-IDF final test accuracy
- 97.18%
- Word2Vec final test accuracy
- 96.01%
- Held-out news articles
- 426
01 / PROJECT EVIDENCE
Actual results, not mock-ups.
These figures are copied directly from the public GitHub repository and are presented with context.
02 / PROBLEM
The engineering question.
News-topic classification requires extracting useful features from unstructured articles without allowing test-set vocabulary to leak into the training process.
03 / METHOD
What I built and evaluated.
- Removed 99 exact duplicate articles, leaving 2,126 unique documents across Business, Entertainment, Politics, Sport and Technology.
- Preprocessed the articles with NLTK and used the same 1,360/340/426 stratified train/validation/test split for both methods.
- Compared training-only TF-IDF (5,000 features) against Word2Vec mean-aggregated document vectors (100 dimensions), feeding comparable ReLU MLP classifiers.
- Selected checkpoints using validation accuracy and evaluated both models once on the held-out test set; inspected training curves, loss and confusion matrices.
04 / MEASURED FINDINGS
What the evidence shows.
- TF-IDF + MLP correctly labelled 414 of 426 test articles (97.18%). Word2Vec + MLP classified 409 (96.01%).
- The Word2Vec pipeline ran faster in the recorded machine-specific comparison, while TF-IDF obtained higher test accuracy.
- Sport was classified correctly in all 101 test cases; the technology category produced more confusion, particularly with business.
05 / LIMITATIONS
Where the claims stop.
Results describe a specific BBC News coursework dataset and split, not production accuracy on arbitrary news. The article collection is small and both models showed some validation divergence consistent with mild overfitting.
06 / ENGINEERING DECISIONS
Choosing the right representation.
The comparison controls the train/validation/test split so that differences are attributable to the feature representation and fitted classifier, not different test articles. TF-IDF preserves sparse term-level signals; averaged Word2Vec embeddings compress articles into dense semantic representations. The text vectorizer is fitted on training data only to limit leakage.
Recorded held-out comparison
| Pipeline | Correct | Accuracy |
|---|---|---|
| TF-IDF + MLP | 414 / 426 | 97.18% |
| Word2Vec + MLP | 409 / 426 | 96.01% |
Interpretation: TF-IDF won on this particular split, but the outcome alone does not prove it is universally better. The repository's confusion matrix and learning curves provide complementary error evidence. The Word2Vec implementation was faster in the documented machine-specific comparison.
Lesson: For small labelled corpora, a well-controlled baseline can be more useful than selecting a more complex representation by intuition. Inspect errors by category and report the test protocol alongside a headline metric.
PROJECT SOURCE
Inspect the implementation.
See the original code, data-handling choices, documented experiments and repository instructions.