Exploring Text and Bounding Box Extraction Anomalies in PDFs with PyMuPDF

[satvik-27199]

Exploring Text and Bounding Box Extraction Anomalies in PDFs with PyMuPDF

Thank you for the excellent work. I'd like to mention an issue related to text and bounding box (bbox) extraction. I've been attempting to extract text values and their corresponding position vectors from tables, and the solution works wonderfully. However, I've encountered a problem with some PDFs (sample attached). When trying to extract information using PyMuPDF, despite there being a significant whitespace between the words "GP" and "Unreserved," it groups them into one block. To understand the root cause, I conducted a word-level bbox extraction and discover ed that the space between "GP" and "Unreserved" is only 2-3 points in the x-coordinate space, which visually does not seem accurate. For example, the space for the "Reserved" vector spans approximately 30 points (274.56 - 244.95). So, why is the gap between "GP" and "Unreserved" only around 3 points (289.20 - 286.53)?

(242.27, 422.79, 339.06, 429.30, ' Reserved GP Unreserved GP\n')

(244.95, 422.79, 274.56, 429.30, 'Reserved', 9, 0, 0)
(277.23, 422.79, 286.53, 429.30, 'GP', 9, 0, 1)
(289.20, 422.79, 327.09, 429.30, 'Unreserved', 9, 0, 2)
(329.76, 422.79, 339.06, 429.30, 'GP', 9, 0, 3)

7.pdf

How to reproduce the bug

import fitz # PyMuPDF

def extract_text_from_pdf(pdf_path):
# Open the provided PDF file
pdf_document = fitz.open(pdf_path)
text = ""
text_block = []

# Iterate through each page of the PDF
for page_num in range(len(pdf_document)):
    # Get the page
    page = pdf_document.load_page(page_num)
    # Extract text from the page
    text += page.get_text()
    print(page.get_text("blocks", sort=False))
    x = page.get_text("blocks", sort=False)
    for word in page.get_text("words", sort=False):
      print(word)
 
# Close the document
pdf_document.close()
return x

Specify the path to your PDF file

pdf_path = '/content/7.pdf'

Extract text

extracted_text = extract_text_from_pdf(pdf_path)
print(extracted_text)

PyMuPDF version

1.23.26

Operating system

MacOS

Python version

3.10

[JorjMcKie]

There is no anomaly. What you see is an image with underlying OCR-ed text.
Looking at it via a viewer will show that the match between text-as-image and OCRed text is sloppy to say it politely: just try to mark text line-wise via the mouse and you will see what I mean.
With a OCR quality like this, you can rely on nothing.

[JorjMcKie]

Anyway - thank you for your appreciation of PyMuPDF.
Here is how to find blocks of text with the "ignore" attribute - which is usually (not always) used for storing OCR results:

bboxes=page.get_bboxlog()

Every item's first sub-item is the block type. For hidden text the type is "ignore-text".

[satvik-27199]

Thank you for your quick response. Is there a method I can use to classify these documents? I have approximately 100,000 documents, and some of them may exhibit the issue I described. I need to identify and remove these problematic documents from the corpus. Is there an efficient way to achieve this?

[JorjMcKie]

No - except you want to exclude all OCRed PDFs.

[satvik-27199]

OCRed PDFs

Yes, we can exclude these documents. Is there any way do it automatically using some function from pymupdf? or we need to do it Manually?

[JorjMcKie]

You can execute above method and check whether "ignore-text" items are present and the page is fully covered by a at least one image.
Then it is probably an OCRed page.

[JorjMcKie]

doc=fitz.open("7.pdf")
page=doc[0]
page.get_images()
[(8, 0, 3800, 5550, 1, 'DeviceGray', '', 'img5', 'JBIG2Decode')]
page.get_image_rects(8)
[Rect(0.0, 0.0, 456.0, 666.0)]
page.rect
Rect(0.0, 0.0, 456.0, 674.0)  # image almost covers the page
bbl=page.get_bboxlog()
set([b[0] for b in bbl if "text" in b[0]])
{'ignore-text', 'fill-text'}  # mixture of normal and OCR text on page

[satvik-27199]

We just tested this on 40-50 documents, and it correctly identifies the OCRed documents. Thank you for the help and the timely response. We really appreciate it and are glad we made the switch to PyMuPDF.

[satvik-27199]

Looks like for OCRed documents, we need to use the combination of detection transformers + pymupdf to make something work.

63 (2).pdf

I have one more question🙏. In the PDF attached (63 (2).pdf), some column values are very close to each other, for example, '2 Lump Sum'. Obviously, the threshold has not been met to separate them into different blocks. I'm curious if there are any strategies or methods available to adjust this threshold based on an analysis of the whitespace according to the PDF's x-coordinate.

[JorjMcKie]

Let's continue under the "Discussions" tab ...

[JorjMcKie]

You are using page.find_tables() for table detection?
Well, anyway, without additional information from outside (= you), the detector cannot be sure here.
You could look at the x0 values of the header words ("Transfer Type", "Outcome", ...) and build a list of vertical lines = [(p1,p2), (p3,p4), ...].
Where p1.x = p2.x and p1.y is the top of words like "Outcome" and p2.y is the page bottom for instance. p1.x ishould equal x0 of "Outcome" and friends in the header row.

Then execute page.find_tables(add_lines=lines). The algorithm will use those lines like vector graphics.

[satvik-27199]

No, we are not using find_tables() for table detection. We are currently using Table Transformer https://github.com/microsoft/table-transformer which works fine for table detection and giving us the table region.

So, the issue is the merging of some headers like Program ID Transfer Type because the whitespace is not high.
(83.61869812011719, 161.50604248046875, 202.13507080078125, 172.53834533691406, 'Program ID Transfer Type\n', 1, 0)

(83.61869812011719, 161.50604248046875, 122.04782104492188, 172.53834533691406, 'Program', 1, 0, 0)
(124.65333557128906, 161.50604248046875, 134.1503143310547, 172.53834533691406, 'ID', 1, 0, 1)
(140.71539306640625, 161.50604248046875, 177.85047912597656, 172.53834533691406, 'Transfer', 1, 0, 2)
(180.38975524902344, 161.50604248046875, 202.13507080078125, 172.53834533691406, 'Type', 1, 0, 3)

If we could look at the x coordinate difference between all words:
we get,
Program - ID --> (124.65 - 122.04) --> 2.5
ID - Transfer --> (140.71 - 134.15) --> 7
Type - Transfer --> (180.3 - 177.8) --> 2.7

Perhaps, it make sense to look at the mean difference? 2.5, 7, 2.7 we get here 250% increase in the difference (between ID and TRANSFER) and tell the algorithm to split the block from original 'Program ID Transfer Type\n', to 'Program ID\n' and 'Transfer Type\n'