pdftract/docs/research/image-and-figure-extraction.md

Image and Figure Extraction in PDF

1. Image XObjects

PDF images are most commonly embedded as Image XObjects. An XObject is an indirect object whose dictionary contains /Type /XObject and /Subtype /Image. It is invoked from a content stream using the Do operator:

/ImageName Do

where ImageName is a name key in the current resource dictionary's /XObject subdictionary that maps to the image's indirect reference.

The XObject dictionary must contain:

Key	Type	Description
Width	integer	Pixel width of the raster
Height	integer	Pixel height of the raster
ColorSpace	name or array	Color space of the image samples
BitsPerComponent	integer	Bits per color component (1, 2, 4, 8, or 16)
Filter	name or array	Compression filter(s) applied to the stream

The stream body following the dictionary contains the raw (filtered) image data. BitsPerComponent is omitted when the image uses a mask or JBIG2 encoding where bit depth is implied.

Positioning via the CTM

The Do operator renders the image into a 1×1 unit square anchored at the origin. The current transformation matrix (CTM) at the point of Do invocation maps that unit square into page space. A canonical image placement looks like:

q
72 0 0 96 144 432 cm
/Im1 Do
Q

The cm operator concatenates [72 0 0 96 144 432] onto the CTM: this scales the unit square to 72×96 points and translates its origin to (144, 432) in page coordinates. The rendered bounding box in page units is thus derived from the CTM columns — specifically, the x-extent is the length of the first column vector and the y-extent is the length of the second column vector. When the matrix contains rotation or shear, the bounding box must be computed as the convex hull of the four transformed corners: (0,0), (1,0), (0,1), (1,1).

2. Inline Images

Inline images embed pixel data directly into the content stream using a three-operator sequence:

BI
  /W 320 /H 240 /CS /RGB /BPC 8 /F /DCT
ID
<binary image data>
EI

The BI (Begin Image) operator introduces the inline dictionary. Key abbreviations are standardized:

Abbreviation	Full key
/W	Width
/H	Height
/CS	ColorSpace
/BPC	BitsPerComponent
/F	Filter
/DP	DecodeParms

ID (Image Data) marks the transition from dictionary to binary payload. The parser must switch to raw byte mode immediately after the whitespace following ID. The payload ends at the next unescaped EI token. Reliably detecting EI requires either tracking the filter's expected byte count or scanning for EI preceded by a whitespace character.

Inline images are limited to simpler use cases: they cannot be referenced by name, cannot be reused across content streams, and are typically restricted to JPEG, CCITT, or uncompressed data. They carry no indirect object overhead but complicate stream parsing significantly.

3. Filter Decoding

Both XObject streams and inline images may be compressed with one or more filters listed in the /Filter key (a name for a single filter, or an array for a chain). Filters are applied in array order during encoding; decoding reverses the chain.

Common filters:

• DCTDecode — JPEG (ISO 10918). The stream is a complete JFIF/JPEG file. DecodeParms may specify ColorTransform (0 = no transform, 1 = YCbCr→RGB, -1 = automatic). Standard JPEG decoders handle the DCT coefficients, quantization, and Huffman decoding.
• JPXDecode — JPEG 2000 (ISO 15444). The stream is a complete JP2 or J2C codestream. Color space information may be embedded in the JP2 container; when present it overrides the PDF /ColorSpace key.
• JBIG2Decode — Bi-level (1 bpp) compression. DecodeParms may contain a JBIG2Globals key whose value is a stream containing the global segment data that must be prepended before decoding. Requires a full JBIG2 decoder (e.g., the jbig2dec library via FFI).
• CCITTFaxDecode — Group 3 or Group 4 fax encoding. DecodeParms specifies K (0=Group3 1D, -1=Group4, positive=Group3 2D with K rows between EOL), Columns, Rows, BlackIs1, EncodedByteAlign.
• FlateDecode — zlib/deflate (RFC 1950 wrapper). DecodeParms may specify a PNG predictor via Predictor (10–15 for PNG filter types None/Sub/Up/Average/Paeth applied row-by-row). After inflation, the predictor must be undone row by row.
• LZWDecode — LZW compression. DecodeParms supports EarlyChange (1 by default, meaning the code size increases one code early). The LZW variant matches the TIFF LZW convention, not GIF.
• RunLengthDecode — PackBits run-length encoding. Each control byte n signals either (257-n) copies of the next byte (if n > 128) or (n+1) literal bytes (if n < 128). Byte 128 signals end-of-data.

For chained filters — e.g., [/ASCII85Decode /FlateDecode] — the decoder applies ASCII85 first to produce binary, then inflates the result.

4. Color Spaces

The /ColorSpace value determines how to interpret decoded sample bytes.

Device spaces are the simplest: DeviceGray (1 component, 0=black, 1=white), DeviceRGB (3 components), DeviceCMYK (4 components, subtractive).

Calibrated spaces embed a viewing condition: CalGray and CalRGB specify a WhitePoint, optional BlackPoint, and Gamma/Matrix. Lab uses the CIE Lab* model with WhitePoint and Range bounds on a* and b*.

ICCBased references an embedded ICC profile stream. The profile's N value gives the component count. ICC profiles provide a precise device-independent color path; for sRGB output, apply the ICC forward transform to XYZ and then the sRGB matrix.

Indexed defines a palette: [/Indexed base hival lookup]. The base space specifies the color model of palette entries; hival is the maximum index (palette has hival+1 entries); lookup is either a string or stream of (hival+1) * N bytes where N is the component count of the base space. Each 8-bit sample is a palette index.

Separation addresses a single named colorant: [/Separation name alternateSpace tintTransform]. The tint transform (a PDF function) maps a tint value [0,1] to the alternate space. When the target device does not support the colorant, apply the tint transform as a fallback to the alternate space (which is typically DeviceCMYK or DeviceRGB).

DeviceN generalizes Separation to multiple colorants: [/DeviceN names alternateSpace tintTransform attributes]. Each channel maps to a named colorant; the tint transform maps the N-component input to the alternate space.

For pipeline output, convert all color spaces to sRGB: device spaces use standard matrices (CMYK to RGB: R=1-min(1,C+K), G=1-min(1,M+K), B=1-min(1,Y+K)); calibrated/ICC spaces go through XYZ intermediate.

5. Image Geometry

The CTM at Do invocation is a 3×3 affine matrix stored as six values [a b c d e f], representing:

| a  b  0 |
| c  d  0 |
| e  f  1 |

The rendered width in page units is sqrt(a² + b²) and the rendered height is sqrt(c² + d²). Rotation angle is atan2(b, a). Shear is present when the dot product (a·c + b·d) is nonzero.

DPI is computed as:

dpi_x = Width_px / (rendered_width_pts / 72.0)
dpi_y = Height_px / (rendered_height_pts / 72.0)

For rotated images, the bounding box in axis-aligned page coordinates is the AABB of the four corners produced by transforming (0,0), (1,0), (0,1), (1,1) through the CTM.

6. Form XObjects

A /Subtype /Form XObject is a self-contained reusable content stream — not an AcroForm widget. It may embed text, images, paths, and other XObjects (including nested Form XObjects). The parser must recurse into each Form XObject encountered via Do.

The Form XObject dictionary includes:

• /BBox — the bounding box of the form in its own local coordinate system.
• /Matrix (optional) — a transformation applied before the form's content stream executes, in addition to the invoking CTM.
• /Resources — its own resource dictionary, independent of the page's.

When a Do operator names a Form XObject, the renderer pushes the current graphics state, concatenates the form's /Matrix onto the CTM, clips to /BBox, then executes the content stream. The combined CTM (page CTM × form matrix) must be tracked at every Do invocation inside the form to correctly compute image geometry.

7. Soft Masks and Transparency

Images participate in the PDF transparency model in three ways:

• ImageMask (boolean) — when true, the image is a 1-bpp stencil. Samples with value 0 paint the current color; samples with value 1 are transparent. ColorSpace and BitsPerComponent are not used.
• /Mask — either a color key mask (an array of [min max] pairs per component defining transparent ranges) or a reference to a 1-bpp image stream serving as a hard mask.
• /SMask — a reference to a grayscale image stream interpreted as an alpha channel (0=fully transparent, 255=fully opaque). The SMask stream is itself an Image XObject with its own filter, width, height, and ColorSpace of DeviceGray.

For figure detection purposes, an image that is pure stencil (ImageMask=true) or has very low average alpha may be a decorative overlay rather than a content figure.

8. Detecting Figure Regions

Building the figure inventory for a page:

1. Walk the content stream, tracking the CTM stack. At each Do invocation for an Image XObject, compute the AABB bounding box in page coordinates and record the image metadata.
2. Apply size thresholds: images smaller than a minimum area threshold (e.g., 1% of page area) are likely icons or decorative glyphs; images covering more than 90% of the page are likely scanned-page backgrounds.
3. Apply position heuristics: watermarks are typically centered and semi-transparent; logos appear near page margins and are small; content figures appear in the body region with substantial rendered area.
4. Caption association: scan for text runs within a vertical proximity band (e.g., ±2× line-height) below or above the image bounding box. Text beginning with "Figure", "Fig.", or a numeric pattern is a strong caption signal. Associate the nearest qualifying text run as the figure's caption.
5. Detect full-page images: rendered size within 5% of the page's MediaBox and positioned at or near the origin — flag as scanned_page.

9. Output Representation

Each detected figure is emitted as a JSON object in the extraction output:

{
  "kind": "figure",
  "page": 3,
  "bbox": [72.0, 300.0, 540.0, 600.0],
  "width_px": 1200,
  "height_px": 900,
  "dpi": 144.0,
  "color_space": "DeviceRGB",
  "filter": ["DCTDecode"],
  "caption": "Figure 4. Loss curves over 100 training epochs.",
  "image_b64": "<base64-encoded PNG or raw raster, present only if extract_images=true>"
}

bbox is [x_min, y_min, x_max, y_max] in PDF page units (origin at bottom-left per PDF convention, or converted to top-left depending on the caller's coordinate system preference). dpi is the effective horizontal DPI rounded to two decimal places. filter is the original filter chain as decoded from the XObject dictionary. caption is null when no caption is detected.

10. Extraction Use Cases and Caller Options

Decoding image bytes is expensive — allocating a decompressed raster for a 300 DPI full-page image at A4 size requires ~25 MB. The extraction pipeline exposes two caller-controlled options:

• extract_images: bool — when false, only metadata (bbox, width_px, height_px, dpi, color_space, filter) is emitted. The image stream is not decompressed. This is the default for text-extraction workflows where image content is not needed.
• max_image_dpi: u32 — when extract_images is true, images whose effective DPI exceeds this threshold are downsampled before encoding. The downsampled dimensions are round(rendered_width_pts / 72.0 * max_image_dpi) × round(rendered_height_pts / 72.0 * max_image_dpi). A common default is 150 DPI for document previews or 300 DPI for archival quality.

For very large images (e.g., 10,000×10,000 px TIFF-equivalent embedded as FlateDecode), the decoder should process row-by-row rather than inflating the entire stream into a contiguous buffer. FlateDecode with PNG predictors naturally supports row-granularity streaming: inflate one PNG-filtered row, unfilter it (applying Sub/Up/Average/Paeth as indicated), emit to the output buffer, then continue. This keeps peak memory bounded to 2 × stride_bytes regardless of image height.

JBIG2 and JPEG 2000 streams require external codec libraries; callers without FFI dependencies available should fall back to emitting raw stream bytes under a raw_stream_b64 key rather than failing. The filter field in the output indicates which codec is needed for the caller to decode the bytes independently.