Augraphy: A Data Augmentation Library for Document Images

A wide variety of data augmentation tools and pipelines exist for machine learning tasks ranging from natural language processing (e.g., [feng-etal-2021-survey, fadaee-etal-2017-data, wei-zou-2019-eda]), audio and speech processing (e.g., [ko15_interspeech, audiogmenter, audio-framework]), and computer vision and image processing. In the image realm of image processing and computer vision, data augmentation tools and pipelines include Augly [Papakipos2022-gq-augly], Augmentor [augmentor], Albumentations [albumentations], DocCreator [doccreator], and imgaug [imgaug]. Augmentation strategies from these image-centric libraries are typically general purpose, and include image transformations like rotations, warps, and color modifications. Table 1 compares Augraphy with other image-based data augmentation libraries and tools. As can be seen, these other data augmentation libraries do not specifically provide support for imitating the corruptions commonly seen in document analysis corpora.

A notable exception to this is the DocCreator image synthesizing tool [doccreator], which is targeted towards creating synthetic images that mimic common corruptions seen in document collections. DocCreator differs from Augraphy in several ways, however, the first is that DocCreator’s augmentations are meant to imitate those seen in historical (e.g., ancient or medieval) documents, while Augraphy is meant to replicate noise caused by noisy office room procedures. DocCreator is also written in the C++ programming language and is a what-you-see-is-what-you-get (WYSIWYG) tool, and does not have a scripting or API interface to enable use in a broader machine learning pipeline. Augraphy, in contrast, is written in Python and can be easily integrated into machine learning model development and evaluation pipelines, and can easily be used alongside other Python packages.

The introduction of noise-like corruptions and other modifications to image data can be used as a way of estimating and evaluating model robustness. This prior work includes the use of image blurring, contrast and brightness changes, color alterations, partial occlusions, geometric transformations, pixel-level noise (e.g., salt-and-pepper noise, impulse noise, etc.), and compression artefacts (e.g., JPEG) to evaluate image classification and object detection models (e.g., [image-quality-impact, imagenet-c, pathology-recommendations, Hosseini2017-kn-google-api, face-recognition-impact, pathology-schomig, Vasiljevic2016-al-bluring-impact]). Our paper also uses robustness testing as a way to showcase the effectiveness of Augraphy, but rather than general image modifications like those described above, we use document-centric modifications.

There are three unique phases in the Augraphy pipeline: the Ink Phase, Paper Phase, and Post Phase. These three phases are highlighted in Figure 3, which shows how the three phases are composed together. Table 2 lists the various individual augmentations from each phase, and Figure 4 displays example output for several of the augmentations from various phases.

Augraphy’s augmentation pipeline starts with an image of a clean document. The pipeline begins by extracting the text and graphics from the source into an “ink” layer. (Ink is synonymous with printer toner within Augraphy.) The augmentation pipeline then distorts and degrades the ink layer through a series of augmentations. There are 10 different ink augmentations to choose from, and any combination of augmentations can work simultaneously.

In the Paper Phase, a “paper factory” provides either a white page or a randomly-selected paper texture base. Like the ink layer, the paper can also be processed through a series of augmentations, with seven effects to choose from, to further provide random realistic paper textures.

After both the ink and paper phases are completed, processing continues by applying the ink, with its desired effects, to the paper. This merged document image is then augmented further through the Post Phase with distortions, such as adding folds, watermarks, or other physical deformations that rely on simultaneous interactions of paper and ink layers. Figure 5 displays a clean document image next an Augraphy-augmented image that utilized several Ink and Paper phase augmentations. The result is a realistic-looking noisy document image.

Our first application demonstrating the effectiveness of Augraphy is robustness testing optical character recognition (OCR). Here, we use Augraphy to add noise to clean document images to test OCR performance in noisy settings. We use the widely-used open-source Tesseract ²²2https://github.com/tesseract-ocr/tesseract engine and first compiled 15 ground-truth, noise-free document images from a corpus of born-digital documents. We then used Tesseract to generate OCR predictions on these noise-free documents. We considered these OCR predictions as the ground-truth labels for each document. Next, we generated noisy versions of the 15 documents by running them through an Augraphy pipeline, and again used Tesseract to generate OCR predictions on these noisy documents. We compared the word accuracy rate on the noisy OCR results versus the ground-truth noise-free OCR results, and found that the noisy OCR results were on average 52% less accurate, with a range of up to 84%. This example use-case demonstrates the effectiveness of using Augraphy to create challenging test data for evaluating OCR systems.

This section highlights the effectiveness of Augraphy by creating a new evaluation set for the task of document de-noising. Document de-noising is the task of removing noisy artifacts from a document image, and one recent dataset that has emerged for this task is the NoisyOffice dataset [noisyoffice], which itself generated noisy versions of clean documents by applying several basic augmentations. However, both the original documents and the augmentations in NoisyOffice are quite limited, so it is natural to wonder if a model trained on NoisyOffice data can generalize to more diverse data inputs for the de-noising task.

Figure 6 exemplifies test inputs (left) to a convolutional auto-encoder, which was trained on the NoisyOffice dataset. The model’s outputs are shown on the right side of Figure 6. It can be inferred that the model does well on the NoisyOffice input (top row), but underperforms on data that was augmented by Augraphy (bottom two rows), showing that Augraphy’s augmentations are effective at producing challenging testing data for analyzing the robustness of de-noising models.

In this section, we use Augraphy to investigate the robustness of face detection models on images that have been altered by Augraphy. In this way, we move beyond analyzing robustness of text-related tasks, but the images we analyze in this section can nonetheless appear in documents like newspapers, which are often scanned from the physical world by noisy scanners. Hence, it is important for image-focused models like image classifiers and object detection models to be robust to scanner-induced noise.

We begin by sampling 75 images from the FDDB face detection benchmark [fddbTech]. Then, we use Augraphy to generate 10 altered versions of each image and manually remove any augmented image where the face(s) is not reasonably visible. We then test two face detection models on the noisy and noise-free images. First, Microsoft Azure’s proprietary face detection model³³3https://azure.microsoft.com/en-us/services/cognitive-services/face, and second, the freely-available UltraFace⁴⁴4https://github.com/onnx/models/tree/main/vision/body_analysis/ultraface model. Table 3 shows face detection accuracy of both models on the noisy test set, and example images where no faces were detected by the Azure model are shown in Figure 7. We see that both models struggle on the Augraphy-augmented data, with Microsoft Azure’s model dropping to 57.1%, and UltraFace detecting only 4.5% of faces that it found in the noise-free data.