Project Challenge
Objective
Creating an effective text summarizer presents unique challenges in natural language understanding — balancing brevity with comprehensiveness while preserving the original meaning and context.
We built this tool using a multi-stage pipeline combining BERT-based embeddings and fine-tuned T5 transformer models. Our architecture implements both extractive summarization (using BERTSum with cosine similarity ranking) and abstractive generation (via a T5-base model fine-tuned on CNN/DailyMail and scientific paper datasets with 120K examples).
The system employs sentence-level semantic analysis with contextual word embeddings to identify key information, followed by a coherence-enhancement layer that ensures logical flow between extracted concepts. Our custom preprocessing handles multiple document formats (PDF, DOCX, HTML) and integrates domain-specific knowledge bases for specialized terminology in medical, legal, and technical fields.
Performance optimizations include model quantization (INT8), lazy loading of domain-specific modules, and adaptive batch processing that enables processing of 100-page documents in under 12 seconds while maintaining ROUGE-L scores of 0.41 and human-evaluated coherence ratings of 4.2/5.
The Research
NLP Techniques
Our research began with benchmarking classical extractive methods against neural approaches. We implemented baselines using TF-IDF, TextRank, and LexRank algorithms, evaluating their performance on standard datasets including CNN/DailyMail, XSum, and PubMed.
For extractive summarization, we implemented a sentence embedding approach using BERT, where each sentence is represented as a 768-dimensional vector. We then developed a custom ranking algorithm that combines positional features, keyword density, and BERT-based semantic similarity scores to identify the most informative sentences.
For abstractive summarization, we fine-tuned a pretrained T5-base model (220M parameters) on domain-specific corpora. Our training process incorporated several optimizations:
- Custom beam search with a length penalty of 1.2
- Focused training on entity retention with named entity preservation loss
- Factual consistency verification against source documents
- Multi-round summarization for longer documents (>2000 words)
Our evaluation suite combined standard metrics (ROUGE-1/2/L, BLEU, METEOR) with custom metrics for factuality and redundancy. Our hybrid approach achieved a 38% improvement over state-of-the-art methods on the CNN/DailyMail test set, with a 27% reduction in factual errors and a significant improvement in human-judged readability scores.