Text Summarization with BART Model

Introduction

In our data-rich world, making sense of large volumes of text can be overwhelming. Imagine if there was a way to condense long articles or documents into concise summaries automatically. Well, there is! It’s called text summarization, and in this article, we’ll unravel the magic behind it using simple language, formulas, graphs, and practical code.

What is Text Summarization?

Text summarization is the process of distilling the essential information from a piece of text, creating a shorter version while retaining its core meaning. There are two main types: extractive, which selects and stitches together existing sentences, and abstractive, which generates new sentences to convey the main points.

The BART Model:

Our hero for this task is BART (Bidirectional and Auto-Regressive Transformers). BART is a transformer-based model pre-trained on a denoising autoencoder objective, meaning it’s great at reconstructing clean sentences from noisy ones. This ability makes it a fantastic choice for various NLP tasks, including summarization.

Why BART:

The choice of the BART model in the provided code example is just one of many possibilities. BART (Bidirectional and Auto-Regressive Transformers) is a transformer-based model that has shown strong performance in various natural language processing tasks, including abstractive text summarization. It was pre-trained on a denoising autoencoder objective, which makes it proficient at generating coherent and contextually relevant summaries.

The Hugging Face Transformers library provides a variety of pre-trained models, each with its strengths and use cases. BART is just one example; depending on your specific requirements, you might choose a different model. Some other popular models for text summarization include GPT-3 (Generative Pre-trained Transformer 3), T5 (Text-to-Text Transfer Transformer), and Pegasus.

The choice of the model depends on factors such as the size of the model, the amount of training data, the task complexity, and the specific characteristics of the input text and desired summaries. Experimenting with different models and fine-tuning them on your specific task or dataset can help you find the most suitable one for your needs.

BART (Bidirectional and Auto-Regressive Transformers) is a transformer-based model designed for various natural language processing tasks, including abstractive text summarization. Let’s break down the key components of BART in detail.

1. Transformer Architecture:

BART is built upon the Transformer architecture, introduced by Vaswani et al. in the paper “Attention is All You Need.” This architecture uses self-attention mechanisms to capture contextual relationships between words in a sequence.

Self-Attention Mechanism

The self-attention mechanism computes a weighted sum of values for each element in a sequence, where the weights are determined by the relevance of other elements to the current one. The formula for computing the attention score Aij​ between the i-th and j-th elements is as follows:

where eij​ is the attention energy, computed as the dot product of query, key, and a scaling factor:

Here, Qi​ and Kj​ are the query and key vectors, and dk​ is the dimensionality of the key vectors.

2. Tokenization:

Before feeding text into the BART model, the input text is tokenized into smaller units called tokens. Tokenization is crucial for the model to process and understand the input. BART uses a tokenizer to perform this task.

The tokenization process involves breaking down a sequence of characters into tokens. In mathematical terms, if X is the input sequence and T(X) is the tokenized sequence, then:

where xi​ represents the i-th token in the tokenized sequence.

3. BART Architecture:

BART extends the transformer architecture with a specific pre-training objective, making it effective for various tasks, including summarization. It is trained to denoise documents corrupted by various techniques like shuffling and removing sentences.

BART is pre-trained using a denoising autoencoder objective. Given a corrupted input sequence ′X′, the model is trained to reconstruct the original sequence X. The objective is to minimize the reconstruction loss:

4. Summarization:

During the fine-tuning phase for summarization, BART is further trained on summarization-specific datasets to fine-tune its ability to generate abstractive summaries.

The objective during summarization involves generating a summary Y from the input sequence X. BART is fine-tuned to minimize the negative log-likelihood of the target summary given the input:

BART’s strength lies in its transformer architecture, denoising autoencoder pre-training, and its ability to generate coherent and contextually relevant summaries during the fine-tuning phase. Understanding the self-attention mechanism, tokenization, and the specific pre-training and fine-tuning objectives provides insights into how BART operates and excels in text summarization tasks.

Code Implementation:

Now, let’s bring this to life with some code using the Hugging Face Transformers library. We’ll explore a Python script that leverages the PyMuPDF library for PDF processing and the Transformers library for natural language processing to perform extractive summarization. The goal is to summarize the content of a PDF file and save the summary in a new PDF document.

Part 1: Importing Libraries

The script begins by importing the necessary libraries:

import fitz  # PyMuPDF
from transformers import BartForConditionalGeneration, BartTokenizer
import textwrap

• PyMuPDF (fitz): This library provides functionality for working with PDF files. In our case, we use it to extract text from PDF pages
• Transformers (BartForConditionalGeneration and BartTokenizer): These components come from the Hugging Face Transformers library and are essential for using the BART model, a sequence-to-sequence model widely used for natural language processing tasks.
• textwrap: A standard Python library module that helps with formatting text.

Part 2: Extracting Text from PDFs

The first function, extract_text_from_pdf, handles the extraction of text from a PDF file:

def extract_text_from_pdf(pdf_path):
    doc = fitz.open(pdf_path)
    text = ""
    for page_num in range(doc.page_count):
        page = doc[page_num]
        text += page.get_text()
    doc.close()
    return text

This function takes the path to a PDF file as input and uses PyMuPDF to open the file. It then iterates through each page, extracting text and concatenating it. Finally, the function returns the combined text.

Part 3: Generating Summaries with BART

The next function text_summarizer_from_pdf uses BART to generate a summary:

def text_summarizer_from_pdf(pdf_path):
    pdf_text = extract_text_from_pdf(pdf_path)

    model_name = "facebook/bart-large-cnn"
    model = BartForConditionalGeneration.from_pretrained(model_name)
    tokenizer = BartTokenizer.from_pretrained(model_name)

    inputs = tokenizer.encode("summarize: " + pdf_text, return_tensors="pt", max_length=1024, truncation=True)
    summary_ids = model.generate(inputs, max_length=150, min_length=50, length_penalty=2.0, num_beams=4, early_stopping=True)

    summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
    formatted_summary = "\n".join(textwrap.wrap(summary, width=80))
    return formatted_summary

This function utilizes the extracted text from the previous step. It loads the BART model and tokenizer and then generates a summary. The summary is obtained by encoding the input text, decoding the output summary_ids, and formatting it using textwrap for better readability.

Part 4: Creating a Summary PDF

The third function, save_summary_as_pdf, saves the generated summary in a new PDF document:

def save_summary_as_pdf(pdf_path, summary):
    doc = fitz.open()

    page = doc.new_page()
    page.insert_text((10, 100), summary, fontname="helv", fontsize=12)  # Adjust the vertical position as needed

    output_pdf_path = pdf_path.replace(".pdf", "_summary.pdf")
    doc.save(output_pdf_path)
    doc.close()

    return output_pdf_path

This function creates a new PDF document using PyMuPDF, adds a page, and inserts the formatted summary. The resulting PDF is then saved with a modified filename to indicate that it contains a summary.

Part 5: Example Usage

The script concludes with an example demonstrating how to use the functions:

pdf_file_path = "path/to/your/file.pdf"
summary = text_summarizer_from_pdf(pdf_file_path)
output_pdf_path = save_summary_as_pdf(pdf_file_path, summary)
print("Summary saved as PDF:", output_pdf_path)

This part shows how to apply the functions to a specific PDF file. The text is summarized using BART, and the summary is saved as a new PDF. The path to the saved summary PDF is then printed.

Conclusion:

BART’s strength lies in its transformer architecture, denoising autoencoder pre-training, and ability to generate coherent and contextually relevant summaries during the fine-tuning phase. Understanding the self-attention mechanism, tokenization, and the specific pre-training and fine-tuning objectives provides insights into how BART operates and excels in text summarization tasks.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

🤝 Reach out to me Github, Linkedin