jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

chore(bf-a8031): remove tracked debug/scratch artifacts and compiled binaries

Browse source 
Remove 40+ tracked scratch artifacts violating the normative file layout:
- Debug scripts: debug_*.py, debug_*.rs
- Test scratch files: test_*.c, test_*.rs, test_*.py
- Compiled binaries: debug_parse_simple, test_empty, test_page_class, test_parse_simple, test_pdf, test_classifier_corpus, test_trailer_debug*, test_trailer_key*, test_trailer_parse*, libstdin.rlib
- Backup files: serve.rs.bak, v2.pdf.bak
- tests/fixtures/ compiled binaries and generator drafts: gen_fixtures, gen_ocr_fixtures, gen_suspects*, gen_unmapped_*, generate_suspects_fixture*, generate_suspects_fixtures.py
- Scratch project directories: gen_unmapped_bin/, gen_unmapped_configurable_bin/

Extend .gitignore to prevent future tracking:
- Compiled binaries in repo root and tests/fixtures/
- Debug/scratch files and backup files
- Output files (0, --1.ppm, mod, out.pdf, *.rlib)

Note: Some output files (0, --1.ppm, mod, out.pdf, memory-report.json) were not tracked and are now covered by .gitignore.
Useful generator scripts in tests/fixtures/ (generate_*.py, generate_*.rs) are preserved.

Verification: Repository structure now matches normative file layout from docs/plan/plan.md.
This commit is contained in:
jedarden 2026-07-05 13:06:33 -04:00
parent 55da9ddf51
commit bf18b10091
69 changed files with 25 additions and 5278 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
.gitignore vendored
View file

@ -10,3 +10,27 @@ memory-report.json
# Proptest regressions are committed (minimal counterexamples)
# but the .gitkeep keeps the directory in git
# Compiled binaries in repo root
debug_*
test_*
gen_fixtures
gen_ocr_fixtures
# Compiled libraries
*.rlib
# Scratch output files
0
--1.ppm
mod
out.pdf
# Backup files
*.bak
# Compiled binaries in tests/fixtures/
tests/fixtures/gen_fixtures
tests/fixtures/gen_ocr_fixtures
tests/fixtures/gen_suspects*
tests/fixtures/gen_unmapped*

2
.needle-predispatch-sha
View file

@ -1 +1 @@
9bac0f30095fb4ceb5729fe056387a36c15721c1
55da9ddf51a828a22afc7baf9000cb927672401e

923
crates/pdftract-cli/src/serve.rs.bak
View file

@ -1,923 +0,0 @@
//! HTTP serve mode for pdftract.
//!
//! This module implements Phase 6.4's `pdftract serve` subcommand: a long-running
//! HTTP service for multi-tenant extraction with cache integration.
//!
//! # Security Model
//!
//! **NO AUTHENTICATION**: pdftract serve has NO built-in authentication. This is a
//! deliberate design decision - authentication and authorization are the responsibility
//! of the deployment infrastructure (reverse proxy, API gateway, service mesh).
//!
//! Deploy behind a reverse proxy (nginx, Traefik, Caddy, envoy) for production use.
//! The reverse proxy should handle:
//! - TLS termination
//! - Authentication (OAuth2, API keys, mTLS, etc.)
//! - Rate limiting
//! - IP whitelisting/blacklisting
//!
//! # File Path Safety
//!
//! All PDFs arrive via **multipart upload only**. No endpoint accepts a file path
//! parameter from the server filesystem. This design prevents:
//! - Directory traversal attacks (../../etc/passwd)
//! - Unintended file access via request parameters
//! - Path-based injection attacks
//!
//! Routes accept `multipart/form-data` with a `pdf` field containing the file bytes.
//! The server never reads from the server filesystem on behalf of a request.
//!
//! # Endpoints
//!
//! - `POST /extract` — Extract and return JSON with cache status in response body
//! - `POST /extract/text` — Extract and return plain text with X-Pdftract-Cache header
//! - `POST /extract/stream` — Extract and return streaming NDJSON with X-Pdftract-Cache header
//! - `GET /health` — Health check (always returns 200 OK)
//!
//! # Cache headers
//!
//! All endpoints return `X-Pdftract-Cache: hit | miss | skipped` header:
//! - `hit`: Served from cache
//! - `miss`: Ran extraction; populated cache
//! - `skipped`: Cache not configured or --no-cache equivalent
//!
//! # Concurrency model
//!
//! The serve mode uses a two-level concurrency architecture:
//!
//! - **tokio**: Per-request concurrency via the async executor. Each HTTP request
//! is handled asynchronously on tokio's multi-threaded runtime.
//! - **rayon**: Per-document parallelism within each extraction. PDF pages are
//! processed in parallel using rayon's work-stealing thread pool.
//!
//! The bridge between async (tokio) and sync (rayon) is `tokio::task::spawn_blocking`.
//! Each POST handler wraps the synchronous extraction call in `spawn_blocking`, which
//! runs the work on tokio's blocking thread pool (separate from the async reactor).
//!
//! This design ensures:
//! - The async reactor is never blocked by extraction work
//! - Multiple PDFs can be extracted concurrently (one per request)
//! - Within each PDF, pages are processed in parallel (rayon)
//! - Thread pools are sized appropriately (tokio: 512 blocking threads; rayon: num_cpus)
//!
//! # Error codes
//!
//! - `REQUEST_TOO_LARGE`: Request body exceeds --max-upload-mb limit
//! - `BAD_REQUEST`: Invalid request parameters or missing file
//! - `EXTRACTION_ERROR`: PDF parsing or extraction failure
//! - `INTERNAL_PANIC`: spawn_blocking task panicked (indicates a bug)
use crate::middleware::{audit_middleware, AuditState};
use anyhow::{Context, Result};
use axum::{
body::Body,
extract::{DefaultBodyLimit, Multipart, State},
http::{HeaderMap, HeaderValue, StatusCode},
response::{IntoResponse, Json, Response as AxumResponse},
routing::{get, post},
Router,
};
use bytes;
use pdftract_core::audit::AuditLogWriter;
use pdftract_core::cache;
use pdftract_core::diagnostics::DiagCode;
use pdftract_core::extract::{extract_pdf, extract_pdf_ndjson, result_to_json};
use pdftract_core::options::{ExtractionOptions, ReceiptsMode};
use serde::{Deserialize, Serialize};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use tokio::sync::Mutex;
use tower_http::limit::RequestBodyLimitLayer;
use tower_http::classify::SharedClassifier;
use tower_http::response::TraceLayer;
use http::{Request, Response};
use std::task::{Context as TaskContext, Poll};
use std::pin::Pin;
use futures_core::ready;
/// Cache state for the HTTP server.
#[derive(Clone)]
pub struct CacheState {
/// Cache directory path
pub cache_dir: Option<PathBuf>,
/// Cache size limit in bytes
pub cache_size_bytes: u64,
/// Whether cache is disabled
pub cache_disabled: bool,
}
/// Server state for the HTTP serve mode.
#[derive(Clone)]
pub struct ServeState {
/// Cache configuration
pub cache: Arc<Mutex<CacheState>>,
/// Audit log state
pub audit: AuditState,
/// Default maximum decompression size in bytes (from --max-decompress-gb)
pub max_decompress_bytes: u64,
}
impl ServeState {
/// Create a new serve state.
pub fn new(
cache_dir: Option<PathBuf>,
cache_size_bytes: u64,
cache_disabled: bool,
audit_writer: Option<AuditLogWriter>,
max_decompress_bytes: u64,
) -> Self {
let cache = CacheState {
cache_dir,
cache_size_bytes,
cache_disabled,
};
Self {
cache: Arc::new(Mutex::new(cache)),
audit: AuditState::new(audit_writer),
max_decompress_bytes,
}
}
}
/// Cache status for response headers and metadata.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CacheStatus {
Hit,
Miss,
Skipped,
}
impl CacheStatus {
/// Convert to string for header/metadata.
pub fn as_str(self) -> &'static str {
match self {
CacheStatus::Hit => "hit",
CacheStatus::Miss => "miss",
CacheStatus::Skipped => "skipped",
}
}
/// Create header value.
pub fn header_value(self) -> HeaderValue {
HeaderValue::from_static(self.as_str())
}
/// Create from string.
pub fn from_string(s: &str) -> Self {
match s {
"hit" => CacheStatus::Hit,
"miss" => CacheStatus::Miss,
"skipped" => CacheStatus::Skipped,
_ => CacheStatus::Skipped,
}
}
}
/// API error response shape.
///
/// All 4xx and 5xx responses use this JSON shape for consistency.
#[derive(Debug, Serialize)]
pub struct ApiError {
/// Error code (e.g., "BAD_REQUEST", "REQUEST_TOO_LARGE", "ENCRYPTED")
pub error: String,
/// Human-readable error message
pub message: String,
/// Optional hint for actionable errors (e.g., "Supply the correct password via --password")
#[serde(skip_serializing_if = "Option::is_none")]
pub hint: Option<String>,
}
impl ApiError {
/// Create a new API error with code and message.
pub fn new(error: impl Into<String>, message: impl Into<String>) -> Self {
ApiError {
error: error.into(),
message: message.into(),
hint: None,
}
}
/// Add a hint to the error.
pub fn with_hint(mut self, hint: impl Into<String>) -> Self {
self.hint = Some(hint.into());
self
}
}
/// Extraction request parameters.
#[derive(Debug, Deserialize)]
struct ExtractParams {
/// Receipts mode (off, lite, svg)
#[serde(default)]
receipts: String,
/// Disable cache for this request
#[serde(default)]
no_cache: bool,
/// Enable full-render path using PDFium
#[serde(default)]
full_render: bool,
/// Maximum decompression size in GB (overrides server default)
#[serde(default)]
max_decompress_gb: Option<usize>,
}
/// Run the HTTP serve mode.
///
/// # Arguments
///
/// * `bind_addr` — Address to bind (e.g., "127.0.0.1:8080")
/// * `cache_dir` — Optional cache directory
/// * `cache_size_bytes` — Cache size limit in bytes
/// * `cache_disabled` — Whether cache is globally disabled
/// * `max_upload_mb` — Maximum request body size in MB
/// * `audit_log` — Optional audit log file path
pub async fn run(
bind_addr: String,
cache_dir: Option<PathBuf>,
cache_size_bytes: u64,
cache_disabled: bool,
max_upload_mb: usize,
max_decompress_gb: usize,
audit_log: Option<PathBuf>,
) -> Result<()> {
let cache_dir_for_logging = cache_dir.as_deref();
// Create audit log writer if specified
let audit_writer = if let Some(ref path) = audit_log {
Some(
AuditLogWriter::open(path)
.context(format!("Failed to open audit log: {}", path.display()))?,
)
} else {
None
};
// Convert max_decompress_gb to bytes (1 GB = 1 << 30 bytes)
let max_decompress_bytes = (max_decompress_gb as u64) * (1 << 30);
let state = ServeState::new(
cache_dir.clone(),
cache_size_bytes,
cache_disabled,
audit_writer,
max_decompress_bytes,
);
let max_body_bytes = max_upload_mb * 1024 * 1024;
let app = Router::new()
.route("/", get(root_handler))
.route("/extract", post(extract_handler))
.route("/extract/text", post(extract_text_handler))
.route("/extract/stream", post(extract_stream_handler))
.route("/health", get(health_handler))
.layer(axum::middleware::from_fn_with_state(
state.audit.clone(),
audit_middleware,
))
.layer(DefaultBodyLimit::max(max_body_bytes))
.layer(RequestBodyLimitLayer::new(max_body_bytes))
.with_state(state);
let listener = tokio::net::TcpListener::bind(&bind_addr)
.await
.context(format!("Failed to bind to {}", bind_addr))?;
// Print startup banner with security warning
eprintln!("pdftract serve is starting on http://{}", bind_addr);
eprintln!("*** NO BUILT-IN AUTH *** — Deploy behind a reverse proxy for production.");
if let Some(dir) = cache_dir_for_logging {
eprintln!(
"Cache enabled: {} (max {} bytes)",
dir.display(),
cache_size_bytes
);
} else {
eprintln!("Cache disabled");
}
if let Some(ref path) = audit_log {
eprintln!("Audit log: {}", path.display());
}
eprintln!("Max upload size: {} MB", max_upload_mb);
eprintln!("Max decompression size: {} GB", max_decompress_gb);
axum::serve(listener, app)
.await
.context("HTTP server error")?;
Ok(())
}
/// Root handler - returns server info.
async fn root_handler() -> impl IntoResponse {
Json(serde_json::json!({
"service": "pdftract",
"version": env!("CARGO_PKG_VERSION"),
"endpoints": [
"POST /extract - Extract PDF and return JSON",
"POST /extract/text - Extract PDF and return plain text",
"POST /extract/stream - Extract PDF and return streaming NDJSON",
"GET /health - Health check"
]
}))
}
/// Health check handler.
async fn health_handler() -> impl IntoResponse {
Json(serde_json::json!({
"status": "ok",
"version": env!("CARGO_PKG_VERSION")
}))
}
/// Extract handler - returns JSON with cache status in metadata.
async fn extract_handler(
State(state): State<ServeState>,
mut multipart: Multipart,
) -> Result<impl IntoResponse, AxumError> {
let (pdf_file, params) = receive_pdf(&mut multipart).await?;
let options = build_options(&state, &params)?;
// Get cache configuration
let cache_state = state.cache.lock().await;
let cache_dir = cache_state.cache_dir.clone();
let cache_size_bytes = cache_state.cache_size_bytes;
let cache_disabled = params.no_cache || cache_state.cache_disabled || cache_dir.is_none();
drop(cache_state);
// Perform extraction with cache integration
let pdf_file_clone = pdf_file.clone();
let (result, cache_status, cache_age) = tokio::task::spawn_blocking(move || {
let cache_dir_ref = cache_dir.as_deref();
cache::extract_with_cache(
&pdf_file_clone,
&options,
cache_dir_ref,
cache_disabled,
Some(cache_size_bytes),
)
.map_err(|e| AxumError::Extraction(format!("{:?}", e), None))
})
.await
.map_err(|e| {
// Distinguish between cancellation (task dropped) and panic
if e.is_cancelled() {
AxumError::Internal(format!("Task cancelled: {}", e))
} else {
// is_panic() true means the task panicked - indicates a bug
AxumError::InternalPanic(format!("Extraction task panicked: {}", e))
}
})?
.map_err(|e| match e {
AxumError::Extraction(msg, _) => AxumError::Extraction(msg, None),
other => other,
})?;
// Build JSON response with cache status
let mut result = result;
result.metadata.cache_status = Some(cache_status.clone());
result.metadata.cache_age_seconds = cache_age;
let json = result_to_json(&result);
let response = AxumResponse::builder()
.status(StatusCode::OK)
.header("Content-Type", "application/json")
.header(
"X-Pdftract-Cache",
CacheStatus::from_string(&cache_status).header_value(),
)
.body(Body::from(serde_json::to_string(&json).unwrap()))
.map_err(|e| AxumError::Internal(format!("{:?}", e).to_string()))?;
Ok(response)
}
/// Extract text handler - returns plain text with X-Pdftract-Cache header.
async fn extract_text_handler(
State(state): State<ServeState>,
mut multipart: Multipart,
) -> Result<impl IntoResponse, AxumError> {
let (pdf_file, params) = receive_pdf(&mut multipart).await?;
let options = build_options(&state, &params)?;
// Get cache configuration
let cache_state = state.cache.lock().await;
let cache_dir = cache_state.cache_dir.clone();
let cache_size_bytes = cache_state.cache_size_bytes;
let cache_disabled = params.no_cache || cache_state.cache_disabled || cache_dir.is_none();
drop(cache_state);
let (result, cache_status, _cache_age) = tokio::task::spawn_blocking(move || {
let cache_dir_ref = cache_dir.as_deref();
cache::extract_with_cache(
&pdf_file,
&options,
cache_dir_ref,
cache_disabled,
Some(cache_size_bytes),
)
.map_err(|e| AxumError::Extraction(format!("{:?}", e), None))
})
.await
.map_err(|e| {
// Distinguish between cancellation (task dropped) and panic
if e.is_cancelled() {
AxumError::Internal(format!("Task cancelled: {}", e))
} else {
// is_panic() true means the task panicked - indicates a bug
AxumError::InternalPanic(format!("Extraction task panicked: {}", e))
}
})?
.map_err(|e| match e {
AxumError::Extraction(msg, _) => AxumError::Extraction(msg, None),
other => other,
})?;
let mut text = String::new();
for page in &result.pages {
for span in &page.spans {
text.push_str(&span.text);
text.push('\n');
}
}
let response = AxumResponse::builder()
.status(StatusCode::OK)
.header(
"X-Pdftract-Cache",
CacheStatus::from_string(&cache_status).header_value(),
)
.body(Body::from(text))
.map_err(|e| AxumError::Internal(format!("{:?}", e).to_string()))?;
Ok(response)
}
/// Extract stream handler - returns true async streaming NDJSON.
///
/// This handler spawns a background task that extracts pages sequentially
/// and sends them over a channel. The response body is a stream that yields
/// each page as NDJSON immediately after it's extracted.
///
/// Cache status is always "skipped" for streaming since we bypass the cache
/// to provide true incremental output.
async fn extract_stream_handler(
State(state): State<ServeState>,
mut multipart: Multipart,
) -> Result<impl IntoResponse, AxumError> {
use tokio_stream::wrappers::ReceiverStream;
use tokio_stream::StreamExt;
let (pdf_file, params) = receive_pdf(&mut multipart).await?;
let options = build_options(&state, &params)?;
// Get cache configuration (for logging only - streaming bypasses cache)
let cache_state = state.cache.lock().await;
let _cache_dir = cache_state.cache_dir.clone();
drop(cache_state);
// Create a channel for streaming pages
let (tx, rx) = tokio::sync::mpsc::channel::<Vec<u8>>(16);
// Spawn extraction task in background
tokio::task::spawn_blocking(move || {
use pdftract_core::extract::extract_pdf_ndjson;
// Clone sender for error handling
let tx_for_error = tx.clone();
// Write to a custom writer that sends to the channel
struct ChannelWriter {
tx: tokio::sync::mpsc::Sender<Vec<u8>>,
};
impl std::io::Write for ChannelWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
// Clone the buffer since we need to send it
self.tx
.blocking_send(buf.to_vec())
.map_err(|e| std::io::Error::new(std::io::ErrorKind::Other, e))?;
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
let writer = ChannelWriter { tx };
// Extract to NDJSON, streaming each page as it's extracted
if let Err(e) = extract_pdf_ndjson(&pdf_file, &options, writer) {
// Send error as a JSON line
let error_json = serde_json::json!({
"error": format!("{:?}", e)
});
if let Ok(json_bytes) = serde_json::to_vec(&error_json) {
let _ = tx_for_error.blocking_send(json_bytes);
let _ = tx_for_error.blocking_send(b"\n".to_vec());
}
}
Ok::<(), AxumError>(())
});
// Create a stream from the receiver
let stream = ReceiverStream::new(rx).map(|item| Ok::<_, axum::Error>(bytes::Bytes::from(item)));
// Return a streaming body
let body = Body::from_stream(stream);
let response = AxumResponse::builder()
.status(StatusCode::OK)
.header("X-Pdftract-Cache", CacheStatus::Skipped.header_value())
.header("Content-Type", "application/x-ndjson")
.body(body)
.map_err(|e| AxumError::Internal(format!("{:?}", e).to_string()))?;
Ok(response)
}
/// Receive uploaded PDF file and extraction parameters.
async fn receive_pdf(multipart: &mut Multipart) -> Result<(PathBuf, ExtractParams), AxumError> {
let mut pdf_path = None;
let mut params = ExtractParams {
receipts: "off".to_string(),
no_cache: false,
full_render: false,
max_decompress_gb: None,
};
while let Some(field) = multipart
.next_field()
.await
.map_err(|e| AxumError::Internal(format!("{:?}", e)))?
{
let name = field.name().unwrap_or("").to_string();
if name == "file" || name == "pdf" {
let data = field
.bytes()
.await
.map_err(|e| AxumError::Internal(format!("{:?}", e).to_string()))?;
// Create a temp file that will persist for the duration of the request
let temp_dir = std::env::temp_dir();
let temp_file = temp_dir.join(format!("pdftract-upload-{}.pdf", uuid::Uuid::new_v4()));
tokio::fs::write(&temp_file, &data)
.await
.map_err(|e| AxumError::Internal(format!("{:?}", e).to_string()))?;
pdf_path = Some(temp_file);
} else if name == "receipts" {
if let Ok(value) = field.text().await {
params.receipts = value;
}
} else if name == "no_cache" {
params.no_cache = true;
} else if name == "full_render" {
// Check if full_render is requested
if let Ok(value) = field.text().await {
params.full_render = value == "true" || value == "1";
}
// Checkbox without value also means true
if params.full_render == false {
params.full_render = true;
}
}
}
let pdf_path =
pdf_path.ok_or_else(|| AxumError::BadRequest(
"No PDF file uploaded".to_string(),
Some("Upload a PDF file in the 'file' or 'pdf' multipart field".to_string())
))?;
Ok((pdf_path, params))
}
/// Build extraction options from parameters.
///
/// Validates that full_render is only used when the feature is available.
/// If full_render is requested but the feature is not compiled in,
/// the request still succeeds but falls back to direct compositing.
fn build_options(
state: &ServeState,
params: &ExtractParams,
) -> Result<ExtractionOptions, AxumError> {
let receipts_mode = match params.receipts.as_str() {
"lite" => ReceiptsMode::Lite,
"svg" => ReceiptsMode::SvgClip,
_ => ReceiptsMode::Off,
};
// Validate max_decompress_gb if provided (for future use)
// Note: This is currently validated but not applied to ExtractionOptions
// since the extraction pipeline uses a hardcoded DEFAULT_MAX_DECOMPRESS_BYTES.
// This validation is kept for API compatibility and future implementation.
if let Some(gb) = params.max_decompress_gb {
const MAX_DECOMPRESS_GB_HARD_CAP: usize = 4096;
if gb > MAX_DECOMPRESS_GB_HARD_CAP {
return Err(AxumError::BadRequest(
format!(
"max_decompress_gb value {} exceeds hard cap of {} GB",
gb, MAX_DECOMPRESS_GB_HARD_CAP
),
Some(format!("Use a value <= {} GB", MAX_DECOMPRESS_GB_HARD_CAP))
));
}
}
// Check if full_render is requested
if params.full_render {
// Validate that full_render is available at runtime
#[cfg(all(feature = "ocr", feature = "full-render"))]
{
use pdftract_core::render::pdfium_path::has_full_render;
if !has_full_render() {
return Err(AxumError::BadRequest(
"full_render requested but PDFium is not available at runtime. \
Ensure the PDFium native library is installed."
.to_string(),
Some("Install PDFium or build with --features full-render".to_string())
));
}
}
#[cfg(not(all(feature = "ocr", feature = "full-render")))]
{
// Feature not compiled in - fall back to direct compositing
// Log a debug message but don't fail the request
tracing::debug!(
"full_render requested but full-render feature not compiled; using direct compositing path"
);
}
}
Ok(ExtractionOptions {
receipts: receipts_mode,
full_render: params.full_render,
..Default::default()
})
}
/// Error types for the HTTP server.
#[derive(Debug)]
pub enum AxumError {
/// Bad request (400) - invalid parameters or missing file
BadRequest(String, Option<String>),
/// Request too large (413) - body exceeds configured limit
RequestTooLarge,
/// Extraction error (422) - PDF parsing or extraction failure
Extraction(String, Option<DiagCode>),
/// Internal error (500) - server-side failure
Internal(String),
/// Internal panic (500) - spawn_blocking task panicked (indicates a bug)
InternalPanic(String),
}
impl IntoResponse for AxumError {
fn into_response(self) -> AxumResponse {
let api_error = match self {
AxumError::RequestTooLarge => ApiError {
error: "REQUEST_TOO_LARGE".to_string(),
message: "Request body exceeds the configured limit".to_string(),
hint: Some("Reduce the file size or increase --max-upload-mb".to_string()),
},
AxumError::BadRequest(msg, hint) => {
let mut err = ApiError::new("BAD_REQUEST", msg);
if let Some(h) = hint {
err = err.with_hint(h);
}
err
}
AxumError::Extraction(msg, diag_code) => {
let (error_code, hint) = if let Some(dc) = diag_code {
match dc {
DiagCode::EncryptionUnsupported => (
"ENCRYPTED".to_string(),
Some("Supply the correct password via --password, or use an Adobe-side decryption tool first".to_string()),
),
DiagCode::EncryptionWrongPassword => (
"WRONG_PASSWORD".to_string(),
Some("The supplied password is incorrect".to_string()),
),
_ => ("EXTRACTION_ERROR".to_string(), None),
}
} else {
("EXTRACTION_ERROR".to_string(), None)
};
let mut err = ApiError::new(error_code, msg);
if let Some(h) = hint {
err = err.with_hint(h);
}
err
}
AxumError::Internal(msg) => {
// Generate a tracing tag for ops to correlate with logs
let tag = format!("{:x}", rand::random::<u32>());
tracing::error!("Internal error [{}]: {}", tag, msg);
ApiError::new(
"INTERNAL",
"Internal error during extraction".to_string(),
).with_hint(format!("Reference tag {} for debugging", tag))
}
AxumError::InternalPanic(msg) => {
let tag = format!("{:x}", rand::random::<u32>());
tracing::error!("Internal panic [{}]: {}", tag, msg);
ApiError::new(
"INTERNAL_PANIC",
"Extraction task panicked (indicates a bug)".to_string(),
).with_hint(format!("Reference tag {} for debugging", tag))
}
};
let status = match api_error.error.as_str() {
"REQUEST_TOO_LARGE" => StatusCode::PAYLOAD_TOO_LARGE, // 413
"BAD_REQUEST" => StatusCode::BAD_REQUEST, // 400
"ENCRYPTED" | "WRONG_PASSWORD" | "EXTRACTION_ERROR" => StatusCode::UNPROCESSABLE_ENTITY, // 422
"INTERNAL" | "INTERNAL_PANIC" => StatusCode::INTERNAL_SERVER_ERROR, // 500
_ => StatusCode::INTERNAL_SERVER_ERROR,
};
(status, Json(api_error)).into_response()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
/// Test that the AxumError enum converts to correct status codes and error codes.
#[test]
fn test_error_into_response() {
// Test BadRequest
let err = AxumError::BadRequest("test".to_string());
let resp = err.into_response();
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
// Test Extraction
let err = AxumError::Extraction("test".to_string());
let resp = err.into_response();
assert_eq!(resp.status(), StatusCode::UNPROCESSABLE_ENTITY);
// Test Internal
let err = AxumError::Internal("test".to_string());
let resp = err.into_response();
assert_eq!(resp.status(), StatusCode::INTERNAL_SERVER_ERROR);
// Test InternalPanic
let err = AxumError::InternalPanic("test".to_string());
let resp = err.into_response();
assert_eq!(resp.status(), StatusCode::INTERNAL_SERVER_ERROR);
}
/// Test that CacheStatus converts correctly to/from strings.
#[test]
fn test_cache_status_conversions() {
assert_eq!(CacheStatus::Hit.as_str(), "hit");
assert_eq!(CacheStatus::Miss.as_str(), "miss");
assert_eq!(CacheStatus::Skipped.as_str(), "skipped");
assert_eq!(CacheStatus::from_string("hit"), CacheStatus::Hit);
assert_eq!(CacheStatus::from_string("miss"), CacheStatus::Miss);
assert_eq!(CacheStatus::from_string("skipped"), CacheStatus::Skipped);
assert_eq!(CacheStatus::from_string("invalid"), CacheStatus::Skipped);
}
/// Helper to load a valid test PDF.
fn load_test_pdf() -> Vec<u8> {
// Use the existing test fixture from pdftract-libpdftract
let pdf_path = concat!(
env!("CARGO_MANIFEST_DIR"),
"/../pdftract-libpdftract/tests/hello.pdf"
);
std::fs::read(pdf_path).expect("Failed to read test PDF")
}
/// Integration test: 8 concurrent requests complete in parallel.
///
/// This is the critical test from the plan (line 2146). It verifies that:
/// - All 8 requests complete (proves no deadlock or serialization)
/// - Wallclock time is similar to a single request (proves parallelism)
/// - /health responds quickly during concurrent extractions (proves /health doesn't block)
#[tokio::test]
async fn test_concurrent_requests_parallel() {
use axum::{
body::Body,
http::{HeaderMap, HeaderValue, Method, StatusCode},
};
use reqwest::multipart::{Form, Part};
use tokio::time::Instant;
// Start the server in the background
let state = ServeState::new(None, 1024 * 1024 * 1024, true, None, 1 << 30); // No cache, 1 GB decompress limit
let app = Router::new()
.route("/extract", post(extract_handler))
.route("/health", get(health_handler))
.with_state(state);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("Failed to bind");
let addr = listener.local_addr().expect("Failed to get local address");
let port = addr.port();
tokio::spawn(async move {
axum::serve(listener, app).await.expect("Server error");
});
// Give the server a moment to start
tokio::time::sleep(Duration::from_millis(100)).await;
let base_url = format!("http://127.0.0.1:{}", port);
let client = reqwest::Client::new();
let pdf_bytes = load_test_pdf();
// First, test that /health responds quickly
let health_start = Instant::now();
let health_resp = client
.get(format!("{}/health", base_url))
.send()
.await
.expect("Health request failed");
let health_duration = health_start.elapsed();
assert_eq!(health_resp.status(), StatusCode::OK);
assert!(
health_duration < Duration::from_millis(100),
"/health should respond in < 100ms, took {:?}",
health_duration
);
// Now launch 8 concurrent extraction requests
let mut handles = Vec::new();
let start = Instant::now();
for i in 0..8 {
let client = client.clone();
let url = format!("{}/extract", base_url);
let pdf = pdf_bytes.clone();
let handle = tokio::spawn(async move {
let part = Part::bytes(pdf).file_name(format!("test{}.pdf", i));
let form = Form::new().part("file", part);
let resp = client
.post(&url)
.multipart(form)
.send()
.await
.expect("Extraction request failed");
(i, resp.status(), client)
});
handles.push(handle);
}
// Wait for all requests to complete
let mut results = Vec::new();
for handle in handles {
let (i, status, _) = handle.await.expect("Task panicked");
results.push((i, status));
}
let total_duration = start.elapsed();
// The critical test: all 8 requests completed (proves no deadlock or serialization)
// We don't assert OK status because the test PDF might not extract correctly;
// the important thing is that all requests got a response.
assert_eq!(results.len(), 8, "All 8 requests should have completed");
// The critical assertion: if requests were serialized, total time would be
// roughly 8x a single request. With parallelism, it should be much less.
// We use a very loose threshold to account for system load and variability.
let single_request_estimate = Duration::from_millis(100); // Rough estimate
let serialized_estimate = single_request_estimate * 8;
assert!(
total_duration < serialized_estimate,
"Requests appear serialized: completed in {:?}, expected < {:?}",
total_duration,
serialized_estimate
);
// Also verify /health still responds quickly during load
let health_start = Instant::now();
let health_resp = client
.get(format!("{}/health", base_url))
.send()
.await
.expect("Health request failed");
let health_duration = health_start.elapsed();
assert_eq!(health_resp.status(), StatusCode::OK);
assert!(
health_duration < Duration::from_millis(100),
"/health should respond in < 100ms during load, took {:?}",
health_duration
);
}
}

38
debug_content_streams.py
View file

@ -1,38 +0,0 @@
#!/usr/bin/env python3
import zlib
# Read the files
with open('tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf', 'rb') as f:
v1_data = f.read()
with open('tests/fingerprint/fixtures/content_edit_one_glyph/v2.pdf', 'rb') as f:
v2_data = f.read()
# Find the stream data (after 'stream\n' and before 'endstream')
def extract_stream(pdf_data):
stream_start = pdf_data.find(b'stream\n') + 7
endstream_pos = pdf_data.find(b'endstream', stream_start)
return pdf_data[stream_start:endstream_pos]
v1_stream = extract_stream(v1_data)
v2_stream = extract_stream(v2_data)
print('v1 stream hex:', v1_stream.hex())
print('v2 stream hex:', v2_stream.hex())
print()
print('v1 stream length:', len(v1_stream))
print('v2 stream length:', len(v2_stream))
print()
# Decompress
try:
v1_decompressed = zlib.decompress(v1_stream)
print('v1 decompressed:', repr(v1_decompressed))
except Exception as e:
print('v1 decompress error:', e)
try:
v2_decompressed = zlib.decompress(v2_stream)
print('v2 decompressed:', repr(v2_decompressed))
except Exception as e:
print('v2 decompress error:', e)

38
debug_fingerprint.rs
View file

@ -1,38 +0,0 @@
// Debug script to check content stream bytes
use std::path::Path;
fn main() {
let v1_path = Path::new("tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf");
let v2_path = Path::new("tests/fingerprint/fixtures/content_edit_one_glyph/v2.pdf");
// Simple: just read the files and print the raw bytes around content streams
let v1_bytes = std::fs::read(v1_path).expect("Failed to read v1");
let v2_bytes = std::fs::read(v2_path).expect("Failed to read v2");
// Find content stream markers
println!("=== v1.pdf content stream ===");
if let Some(pos) = v1_bytes.windows(3).position(|w| w == b"end") {
// Look for the stream content
let stream_start = v1_bytes.windows(6).position(|w| w == b"stream").unwrap_or(0);
if stream_start > 0 {
let after_newline = stream_start + 6;
while after_newline < v1_bytes.len() && v1_bytes[after_newline] == b'\r' || v1_bytes[after_newline] == b'\n' {
// skip whitespace
}
let endstream = v1_bytes.windows(9).position(|w| w == b"endstream").unwrap_or(v1_bytes.len());
println!("Stream bytes: {:?}", &v1_bytes[stream_start+6..stream_start+200]);
}
}
// Just search for the string literal "(Hello" in both files
println!("\n=== Searching for '(Hello' ===");
let v1_hello = v1_bytes.windows(6).position(|w| w == b"(Hello").unwrap_or(usize::MAX);
let v2_hello = v2_bytes.windows(6).position(|w| w == b"(Hello").unwrap_or(usize::MAX);
if v1_hello < v1_bytes.len() {
println!("v1 found at {}: {:?}", v1_hello, &v1_bytes[v1_hello..v1_hello+20]));
}
if v2_hello < v2_bytes.len() {
println!("v2 found at {}: {:?}", v2_hello, &v2_bytes[v2_hello..v2_hello+20]));
}
}

131
debug_fingerprint_content_diff.rs
View file

@ -1,131 +0,0 @@
//! Debug tool to compare fingerprints of two PDFs
use pdftract_core::document::compute_pdf_fingerprint;
use pdftract_core::fingerprint::{compute_fingerprint, FingerprintInput, PageFingerprintData, ContentStreamData};
use pdftract_core::parser::catalog::parse_catalog;
use pdftract_core::parser::pages::flatten_page_tree;
use pdftract_core::parser::stream::{FileSource, PdfSource};
use pdftract_core::parser::xref::{load_xref_with_prev_chain, XrefResolver};
use std::path::Path;
fn find_startxref(source: &FileSource) -> anyhow::Result<u64> {
let len = source.len()?;
let scan_size = 1024.min(len) as usize;
let scan_start = (len - scan_size as u64) as u64;
let tail_data = source.read_at(scan_start, scan_size)?;
let startxref_pos = tail_data
.windows(9)
.rposition(|w| w == b"startxref")
.ok_or_else(|| anyhow::anyhow!("startxref not found"))?;
let offset_data = &tail_data[startxref_pos + 9..];
let offset_start = offset_data
.iter()
.position(|&b| !matches!(b, b' ' | b'\r' | b'\n' | b'\t'))
.unwrap_or(offset_data.len());
let offset_data_trimmed = &offset_data[offset_start..];
let newline_pos = offset_data_trimmed
.iter()
.position(|&b| b == b'\n' || b == b'\r')
.unwrap_or(offset_data_trimmed.len());
let offset_str = std::str::from_utf8(&offset_data_trimmed[..newline_pos])?;
let offset: u64 = offset_str.trim().parse()?;
Ok(offset)
}
fn main() {
let args: Vec<String> = std::env::args().collect();
if args.len() < 3 {
eprintln!("Usage: {} <v1.pdf> <v2.pdf>", args[0]);
std::process::exit(1);
}
let v1_path = Path::new(&args[1]);
let v2_path = Path::new(&args[2]);
println!("Comparing fingerprints:");
println!(" v1: {}", v1_path.display());
println!(" v2: {}", v2_path.display());
println!();
// Parse v1
let source1 = FileSource::open(v1_path).unwrap();
let startxref1 = find_startxref(&source1).unwrap();
let xref1 = load_xref_with_prev_chain(&source1, startxref1);
let resolver1 = XrefResolver::from_section(xref1.clone());
let root_ref1 = xref1.trailer.as_ref().and_then(|t| t.get("Root")).and_then(|o| o.as_ref()).unwrap();
let catalog1 = parse_catalog(&resolver1, root_ref1, Some(&source1 as &dyn PdfSource)).unwrap();
let pages1 = flatten_page_tree(&resolver1, catalog1.pages_ref).unwrap();
// Parse v2
let source2 = FileSource::open(v2_path).unwrap();
let startxref2 = find_startxref(&source2).unwrap();
let xref2 = load_xref_with_prev_chain(&source2, startxref2);
let resolver2 = XrefResolver::from_section(xref2.clone());
let root_ref2 = xref2.trailer.as_ref().and_then(|t| t.get("Root")).and_then(|o| o.as_ref()).unwrap();
let catalog2 = parse_catalog(&resolver2, root_ref2, Some(&source2 as &dyn PdfSource)).unwrap();
let pages2 = flatten_page_tree(&resolver2, catalog2.pages_ref).unwrap();
println!("v1: {} pages", pages1.len());
println!("v2: {} pages", pages2.len());
// Compare content stream references
println!("\nv1 page 0 contents: {:?}", pages1[0].contents);
println!("v2 page 0 contents: {:?}", pages2[0].contents);
// Resolve and decode content streams
println!("\n=== v1 content streams ===");
for (i, &obj_ref) in pages1[0].contents.iter().enumerate() {
println!("Stream {} (ref {:?}):", i, obj_ref);
match resolver1.resolve(obj_ref) {
Ok(pdftract_core::parser::object::PdfObject::Stream(stream)) => {
println!(" Dict keys: {:?}", stream.dict.keys().collect::<Vec<_>>());
let opts = pdftract_core::parser::stream::ExtractionOptions::default();
let mut counter = 0u64;
let decoded = pdftract_core::parser::stream::decode_stream(&*stream, &source1, &opts, &mut counter);
println!(" Decoded {} bytes: {:?}", decoded.len(), String::from_utf8_lossy(&decoded));
}
Ok(other) => {
println!(" Not a stream: {:?}", std::mem::discriminant(&other));
}
Err(e) => {
println!(" Failed to resolve: {:?}", e);
}
}
}
println!("\n=== v2 content streams ===");
for (i, &obj_ref) in pages2[0].contents.iter().enumerate() {
println!("Stream {} (ref {:?}):", i, obj_ref);
match resolver2.resolve(obj_ref) {
Ok(pdftract_core::parser::object::PdfObject::Stream(stream)) => {
println!(" Dict keys: {:?}", stream.dict.keys().collect::<Vec<_>>());
let opts = pdftract_core::parser::stream::ExtractionOptions::default();
let mut counter = 0u64;
let decoded = pdftract_core::parser::stream::decode_stream(&*stream, &source2, &opts, &mut counter);
println!(" Decoded {} bytes: {:?}", decoded.len(), String::from_utf8_lossy(&decoded));
}
Ok(other) => {
println!(" Not a stream: {:?}", std::mem::discriminant(&other));
}
Err(e) => {
println!(" Failed to resolve: {:?}", e);
}
}
}
// Compute fingerprints
let fp1 = compute_pdf_fingerprint(v1_path).unwrap();
let fp2 = compute_pdf_fingerprint(v2_path).unwrap();
println!("\n=== Fingerprints ===");
println!("v1: {}", fp1);
println!("v2: {}", fp2);
println!("Match: {}", fp1 == fp2);
}

39
debug_fingerprint_detailed.py
View file

@ -1,39 +0,0 @@
#!/usr/bin/env python3
import zlib
# Read the files
with open('tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf', 'rb') as f:
v1_data = f.read()
with open('tests/fingerprint/fixtures/content_edit_one_glyph/v2.pdf', 'rb') as f:
v2_data = f.read()
# Find the stream data
def extract_stream(pdf_data):
stream_start = pdf_data.find(b'stream\n') + 7
endstream_pos = pdf_data.find(b'endstream', stream_start)
return pdf_data[stream_start:endstream_pos]
v1_stream = extract_stream(v1_data)
v2_stream = extract_stream(v2_data)
print('v1 stream hex:', v1_stream.hex())
print('v2 stream hex:', v2_stream.hex())
print()
# Decompress
v1_decompressed = zlib.decompress(v1_stream)
v2_decompressed = zlib.decompress(v2_stream)
print('v1 decompressed:', repr(v1_decompressed))
print('v2 decompressed:', repr(v2_decompressed))
print()
# Hash the decompressed content
import hashlib
v1_hash = hashlib.sha256(v1_decompressed).hexdigest()
v2_hash = hashlib.sha256(v2_decompressed).hexdigest()
print('v1 SHA-256:', v1_hash)
print('v2 SHA-256:', v2_hash)
print('Different?', v1_hash != v2_hash)

36
debug_fingerprint_example.rs
View file

@ -1,36 +0,0 @@
use pdftract_core::document::parse_pdf_file;
fn main() {
let v1_path = std::path::Path::new("tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf");
let v2_path = std::path::Path::new("tests/fingerprint/fixtures/content_edit_one_glyph/v2.pdf");
println!("=== Parsing v1.pdf ===");
let (fp1, _cat1, pages1, _resolver1) = parse_pdf_file(v1_path).unwrap();
println!("v1 fingerprint: {}", fp1);
println!("v1 pages: {}", pages1.len());
if let Some(p) = pages1.first() {
println!("v1 page 0 contents: {:?} ({} streams)", p.contents, p.contents.len());
println!("v1 page 0 media_box: {:?}", p.media_box);
}
println!("\n=== Parsing v2.pdf ===");
let (fp2, _cat2, pages2, _resolver2) = parse_pdf_file(v2_path).unwrap();
println!("v2 fingerprint: {}", fp2);
println!("v2 pages: {}", pages2.len());
if let Some(p) = pages2.first() {
println!("v2 page 0 contents: {:?} ({} streams)", p.contents, p.contents.len());
println!("v2 page 0 media_box: {:?}", p.media_box);
}
println!("\n=== Comparing content refs ===");
println!("v1 content ref: {:?}", pages1[0].contents.get(0));
println!("v2 content ref: {:?}", pages2[0].contents.get(0));
println!("Content refs equal: {}", pages1[0].contents == pages2[0].contents);
println!("\n=== Re-parsing to verify ===");
let (fp1_re, _, _, _) = parse_pdf_file(v1_path).unwrap();
let (fp2_re, _, _, _) = parse_pdf_file(v2_path).unwrap();
println!("v1 fingerprint (re-parsed): {}", fp1_re);
println!("v2 fingerprint (re-parsed): {}", fp2_re);
println!("Fingerprints equal: {}", fp1 == fp2);
}

78
debug_fingerprint_hash.rs
View file

@ -1,78 +0,0 @@
use pdftract_core::document::compute_pdf_fingerprint;
use pdftract_core::parser::lexer::Lexer;
use std::path::PathBuf;
fn main() {
let v1 = PathBuf::from("tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf");
let v2 = PathBuf::from("tests/fingerprint/fixtures/content_edit_one_glyph/v2.pdf");
println!("=== v1.pdf ===");
let fp1 = compute_pdf_fingerprint(&v1).unwrap();
println!("Fingerprint: {}", fp1);
println!("\n=== v2.pdf ===");
let fp2 = compute_pdf_fingerprint(&v2).unwrap();
println!("Fingerprint: {}", fp2);
println!("\n=== Comparison ===");
if fp1 == fp2 {
println!("ERROR: Fingerprints are IDENTICAL but should DIFFER!");
} else {
println!("OK: Fingerprints differ as expected");
}
// Debug: Show raw content stream bytes
println!("\n=== Raw Content Stream Debug ===");
debug_content_stream(&v1, "v1");
debug_content_stream(&v2, "v2");
}
fn debug_content_stream(path: &PathBuf, label: &str) {
use pdftract_core::parser::object::{PdfObject, PdfStream};
use pdftract_core::parser::stream::{FileSource, decode_stream};
use pdftract_core::parser::xref::{XrefResolver, load_xref};
use pdftract_core::parser::catalog::parse_catalog;
use pdftract_core::parser::pages::{flatten_page_tree, PageDict};
use pdftract_core::parser::trailer::parse_trailer_dict;
let source = FileSource::open(path).unwrap();
let file_size = source.len();
// Parse trailer
let tail_data = source.read_range(file_size.saturating_sub(1024), 1024).unwrap();
let trailer_offset = pdftract_core::document::find_startxref_offset(&tail_data).unwrap();
let trailer_data = source.read_range(trailer_offset, file_size - trailer_offset).unwrap();
let (_, trailer_dict) = parse_trailer_dict(&trailer_data).unwrap();
// Load xref
let resolver = load_xref(&trailer_dict, &source).unwrap();
// Get catalog
let root_ref = trailer_dict.get("/Root").and_then(|o| o.as_ref()).unwrap();
let catalog_obj = resolver.resolve_with_source(*root_ref, &source).unwrap();
let catalog = parse_catalog(&catalog_obj).unwrap();
// Flatten pages
let pages: Vec<PageDict> = flatten_page_tree(&catalog.pages_ref, &resolver, &source).collect();
println!("{}: {} pages", label, pages.len());
for (i, page) in pages.iter().enumerate() {
println!("{} page {}: {} content streams", label, i, page.contents.len());
for (j, &stream_ref) in page.contents.iter().enumerate() {
match resolver.resolve_with_source(stream_ref, &source) {
Ok(PdfObject::Stream(stream)) => {
let decoded = decode_stream(&stream, &source, &Default::default(), &mut 0);
println!(" stream {}: {} bytes -> {} decoded", j, stream.raw_len, decoded.len());
println!(" raw bytes (first 100): {:?}", &stream.raw_bytes[..stream.raw_bytes.len().min(100)]);
println!(" decoded: {}", String::from_utf8_lossy(&decoded));
}
Ok(other) => {
println!(" stream {}: NOT a stream: {:?}", j, std::mem::discriminant(&other));
}
Err(e) => {
println!(" stream {}: ERROR: {:?}", j, e);
}
}
}
}
}

43
debug_fingerprint_test.rs
View file

@ -1,43 +0,0 @@
// Debug script to test fingerprint computation with timeouts
use std::path::Path;
use std::time::Instant;
fn main() {
let fixtures = vec![
"tests/fingerprint/fixtures/byte_identical/v1.pdf",
"tests/fingerprint/fixtures/acrobat_resave/v1.pdf",
"tests/fingerprint/fixtures/pdftk_resave/v1.pdf",
"tests/fingerprint/fixtures/qpdf_resave/v1.pdf",
"tests/fingerprint/fixtures/linearization_toggle/v1.pdf",
"tests/fingerprint/fixtures/metadata_only/v1.pdf",
"tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf",
"tests/fingerprint/fixtures/content_edit_one_paragraph/v1.pdf",
];
for path in fixtures {
println!("\n=== Testing {} ===", path);
let path_obj = Path::new(path);
if !path_obj.exists() {
println!(" File not found!");
continue;
}
let start = Instant::now();
match pdftract_core::document::compute_pdf_fingerprint(path_obj) {
Ok(fp) => {
let elapsed = start.elapsed();
println!(" ✓ Fingerprint: {} (took {:?}", fp, elapsed);
}
Err(e) => {
let elapsed = start.elapsed();
println!(" ✗ Error after {:?}: {}", elapsed, e);
}
}
// Safety: if any test takes > 5 seconds, abort
if start.elapsed().as_secs() > 5 {
println!(" WARNING: Test taking too long, aborting");
break;
}
}
}

34
debug_fixtures.py
View file

@ -1,34 +0,0 @@
#!/usr/bin/env python3
import pikepdf
import zlib
# Check v1.pdf
with pikepdf.open("tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf") as pdf:
page = pdf.pages[0]
contents = page.get("/Contents")
if contents:
raw = contents.read_raw_bytes()
print(f"v1 raw hex: {raw.hex()}")
# Try with zlib header (78 9c)
try:
decompressed = zlib.decompress(raw)
print(f"v1 decompressed: {decompressed}")
except Exception as e:
print(f"v1 decompress failed: {e}")
print()
# Check v2.pdf
with pikepdf.open("tests/fingerprint/fixtures/content_edit_one_glyph/v2.pdf") as pdf:
page = pdf.pages[0]
contents = page.get("/Contents")
if contents:
raw = contents.read_raw_bytes()
print(f"v2 raw hex: {raw.hex()}")
try:
decompressed = zlib.decompress(raw)
print(f"v2 decompressed: {decompressed}")
except Exception as e:
print(f"v2 decompress failed: {e}")

138
debug_fixtures.rs
View file

@ -1,138 +0,0 @@
use pdftract_core::parser::stream::{
FlateDecoder, LZWDecoder, ASCII85Decoder, ASCIIHexDecoder,
RunLengthDecoder, DCTDecoder, JpxStreamDecoder, CCITTFaxDecoder,
CryptDecoder, PassthroughDecoder, normalize_filter_name,
StreamDecoder, DEFAULT_MAX_DECOMPRESS_BYTES,
};
use pdftract_core::parser::object::{PdfObject, PdfDict};
use pdftract_core::diagnostics::DiagCode;
use indexmap::IndexMap;
use std::path::PathBuf;
use std::fs;
fn main() {
let fixtures = vec![
("flate_png_pred15_all_six", "FlateDecode", Some(create_png_predictor_params())),
("flate_truncated", "FlateDecode", None),
("lzw_early_change_0", "LZWDecode", Some(create_early_change_params(0))),
("lzw_early_change_1", "LZWDecode", Some(create_early_change_params(1))),
("ascii85_terminator", "ASCII85Decode", None),
];
let fixtures_path = PathBuf::from("tests/stream_decoder/fixtures");
for (name, filter_name, params) in fixtures {
println!("\n=== {} ===", name);
let bin_path = fixtures_path.join(format!("{}.bin", name));
let expected_path = fixtures_path.join(format!("{}.expected", name));
let input = fs::read(&bin_path).unwrap();
let expected = fs::read(&expected_path).unwrap();
println!("Input: {} bytes", input.len());
println!("Expected: {} bytes", expected.len());
println!("Expected hex: {:?}", hex::encode(&expected));
let decoder = get_decoder(filter_name).unwrap();
let mut counter = 0;
let result = decoder.decode(&input, params.as_ref(), &mut counter, DEFAULT_MAX_DECOMPRESS_BYTES);
match result {
Ok(decoded) => {
println!("Decoded: {} bytes", decoded.len());
println!("Decoded hex: {:?}", hex::encode(&decoded));
if decoded != expected.as_slice() {
println!("MISMATCH!");
// Show first difference
for (i, (&exp, &got)) in expected.iter().zip(decoded.iter()).enumerate() {
if exp != got {
println!("First difference at byte {}: expected 0x{:02x}, got 0x{:02x}", i, exp, got);
break;
}
}
} else {
println!("MATCH!");
}
}
Err(e) => {
println!("Error: {:?}", e);
}
}
}
// Test filter array
println!("\n=== filter_array_a85_then_flate ===");
let bin_path = fixtures_path.join("filter_array_a85_then_flate.bin");
let expected_path = fixtures_path.join("filter_array_a85_then_flate.expected");
let input = fs::read(&bin_path).unwrap();
let expected = fs::read(&expected_path).unwrap();
println!("Input: {} bytes", input.len());
println!("Expected: {} bytes", expected.len());
println!("Expected hex: {:?}", hex::encode(&expected));
let mut current = input;
let mut counter = 0;
// First decode ASCII85
let a85_decoder = ASCII85Decoder;
match a85_decoder.decode(&current, None, &mut counter, DEFAULT_MAX_DECOMPRESS_BYTES) {
Ok(decoded) => {
println!("After ASCII85: {} bytes", decoded.len());
println!("After ASCII85 hex: {:?}", hex::encode(&decoded));
current = decoded;
}
Err(e) => {
println!("ASCII85 error: {:?}", e);
return;
}
}
// Then decode Flate
let flate_decoder = FlateDecoder;
match flate_decoder.decode(&current, None, &mut counter, DEFAULT_MAX_DECOMPRESS_BYTES) {
Ok(decoded) => {
println!("After Flate: {} bytes", decoded.len());
println!("After Flate hex: {:?}", hex::encode(&decoded));
if decoded != expected.as_slice() {
println!("MISMATCH!");
} else {
println!("MATCH!");
}
}
Err(e) => {
println!("Flate error: {:?}", e);
}
}
}
fn get_decoder(name: &str) -> Option<Box<dyn StreamDecoder>> {
match normalize_filter_name(name) {
"FlateDecode" => Some(Box::new(FlateDecoder)),
"LZWDecode" => Some(Box::new(LZWDecoder)),
"ASCII85Decode" => Some(Box::new(ASCII85Decoder)),
"ASCIIHexDecode" => Some(Box::new(ASCIIHexDecoder)),
"Crypt" => Some(Box::new(CryptDecoder)),
"DCTDecode" => Some(Box::new(DCTDecoder)),
"JBIG2Decode" => Some(Box::new(PassthroughDecoder::new("JBIG2Decode"))),
"JPXDecode" => Some(Box::new(JpxStreamDecoder)),
"CCITTFaxDecode" => Some(Box::new(CCITTFaxDecoder)),
"RunLengthDecode" => Some(Box::new(RunLengthDecoder)),
_ => None,
}
}
fn create_png_predictor_params() -> PdfObject {
let mut dict = IndexMap::new();
dict.insert("/Predictor".into(), PdfObject::Integer(15));
dict.insert("/Columns".into(), PdfObject::Integer(8));
dict.insert("/Colors".into(), PdfObject::Integer(1));
dict.insert("/BitsPerComponent".into(), PdfObject::Integer(8));
PdfObject::Dict(Box::new(dict))
}
fn create_early_change_params(early_change: i64) -> PdfObject {
let mut dict = IndexMap::new();
dict.insert("/EarlyChange".into(), PdfObject::Integer(early_change));
PdfObject::Dict(Box::new(dict))
}

BIN
debug_parse_simple
View file

Binary file not shown.

22
debug_trailer.rs
View file

@ -1,22 +0,0 @@
use pdftract_core::source::file_source::ParserFileSource;
use pdftract_core::parser::xref::{find_startxref, load_xref_with_prev_chain};
fn main() {
let pdf_path = std::path::Path::new("tests/fingerprint/fixtures/acrobat_resave/v1.pdf");
let source = ParserFileSource::open(pdf_path).unwrap();
let startxref_offset = find_startxref(&source).unwrap();
let xref_section = load_xref_with_prev_chain(&source, startxref_offset);
println!("xref_section loaded");
println!("trailer: {:?}", xref_section.trailer);
if let Some(trailer) = &xref_section.trailer {
println!("\nTrailer contents:");
for (k, v) in trailer.iter() {
println!(" key='{}' value={:?}", k, v);
}
println!("\nLooking for 'Root': {:?}", trailer.get("Root"));
println!("Looking for '/Root': {:?}", trailer.get("/Root"));
}
}

638
gen_unmapped_bin/Cargo.lock generated
View file

@ -1,638 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "adler2"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "android_system_properties"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
dependencies = [
"libc",
]
[[package]]
name = "anyhow"
version = "1.0.103"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a4385e2e34eb35d6b3efe798b9eb88096925d87726c0798709bf56d9ed84af3"
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "block-buffer"
version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array",
]
[[package]]
name = "bumpalo"
version = "3.20.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
[[package]]
name = "cc"
version = "1.2.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96"
dependencies = [
"find-msvc-tools",
"shlex",
]
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "chrono"
version = "0.4.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327"
dependencies = [
"iana-time-zone",
"num-traits",
"windows-link",
]
[[package]]
name = "core-foundation-sys"
version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
[[package]]
name = "crc32fast"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511"
dependencies = [
"cfg-if",
]
[[package]]
name = "crossbeam-deque"
version = "0.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
]
[[package]]
name = "crossbeam-epoch"
version = "0.9.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-utils"
version = "0.8.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28"
[[package]]
name = "crypto-common"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array",
"typenum",
]
[[package]]
name = "deranged"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
[[package]]
name = "digest"
version = "0.10.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer",
"crypto-common",
]
[[package]]
name = "either"
version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"
[[package]]
name = "encoding_rs"
version = "0.8.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3"
dependencies = [
"cfg-if",
]
[[package]]
name = "equivalent"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "flate2"
version = "1.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c"
dependencies = [
"crc32fast",
"miniz_oxide",
]
[[package]]
name = "futures-core"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d"
[[package]]
name = "futures-task"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393"
[[package]]
name = "futures-util"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6"
dependencies = [
"futures-core",
"futures-task",
"pin-project-lite",
"slab",
]
[[package]]
name = "gen_unmapped_comprehensive"
version = "0.1.0"
dependencies = [
"anyhow",
"lopdf",
]
[[package]]
name = "generic-array"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
]
[[package]]
name = "hashbrown"
version = "0.17.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
[[package]]
name = "iana-time-zone"
version = "0.1.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470"
dependencies = [
"android_system_properties",
"core-foundation-sys",
"iana-time-zone-haiku",
"js-sys",
"log",
"wasm-bindgen",
"windows-core",
]
[[package]]
name = "iana-time-zone-haiku"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f"
dependencies = [
"cc",
]
[[package]]
name = "indexmap"
version = "2.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
dependencies = [
"equivalent",
"hashbrown",
]
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "js-sys"
version = "0.3.103"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102"
dependencies = [
"cfg-if",
"futures-util",
"wasm-bindgen",
]
[[package]]
name = "libc"
version = "0.2.186"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
[[package]]
name = "log"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "lopdf"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c5c8ecfc6c72051981c0459f75ccc585e7ff67c70829560cda8e647882a9abff"
dependencies = [
"chrono",
"encoding_rs",
"flate2",
"indexmap",
"itoa",
"log",
"md-5",
"nom",
"rangemap",
"rayon",
"time",
"weezl",
]
[[package]]
name = "md-5"
version = "0.10.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf"
dependencies = [
"cfg-if",
"digest",
]
[[package]]
name = "memchr"
version = "2.8.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
[[package]]
name = "minimal-lexical"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
[[package]]
name = "miniz_oxide"
version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316"
dependencies = [
"adler2",
"simd-adler32",
]
[[package]]
name = "nom"
version = "7.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
dependencies = [
"memchr",
"minimal-lexical",
]
[[package]]
name = "num-conv"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "pin-project-lite"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
[[package]]
name = "powerfmt"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368"
dependencies = [
"proc-macro2",
]
[[package]]
name = "rangemap"
version = "1.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "973443cf09a9c8656b574a866ab68dfa19f0867d0340648c7d2f6a71b8a8ea68"
[[package]]
name = "rayon"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d"
dependencies = [
"either",
"rayon-core",
]
[[package]]
name = "rayon-core"
version = "1.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91"
dependencies = [
"crossbeam-deque",
"crossbeam-utils",
]
[[package]]
name = "rustversion"
version = "1.0.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
[[package]]
name = "serde_core"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "shlex"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
[[package]]
name = "simd-adler32"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214"
[[package]]
name = "slab"
version = "0.4.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
[[package]]
name = "syn"
version = "2.0.118"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "time"
version = "0.3.53"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "18dfaaeddcb932337b5e7866ee7d0ce9b76d2fd092997146f187ec09b4558a50"
dependencies = [
"deranged",
"num-conv",
"powerfmt",
"serde_core",
"time-core",
"time-macros",
]
[[package]]
name = "time-core"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
[[package]]
name = "time-macros"
version = "0.2.31"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c431b87111666e491a90baa837f914fb45cd5dc3c268591b0220ff5057f2085f"
dependencies = [
"num-conv",
"time-core",
]
[[package]]
name = "typenum"
version = "1.20.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "version_check"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
[[package]]
name = "wasm-bindgen"
version = "0.2.126"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4"
dependencies = [
"cfg-if",
"once_cell",
"rustversion",
"wasm-bindgen-macro",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.126"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
]
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.126"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e"
dependencies = [
"bumpalo",
"proc-macro2",
"quote",
"syn",
"wasm-bindgen-shared",
]
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.126"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24"
dependencies = [
"unicode-ident",
]
[[package]]
name = "weezl"
version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a28ac98ddc8b9274cb41bb4d9d4d5c425b6020c50c46f25559911905610b4a88"
[[package]]
name = "windows-core"
version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
dependencies = [
"windows-implement",
"windows-interface",
"windows-link",
"windows-result",
"windows-strings",
]
[[package]]
name = "windows-implement"
version = "0.60.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "windows-interface"
version = "0.59.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-result"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
dependencies = [
"windows-link",
]
[[package]]
name = "windows-strings"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
dependencies = [
"windows-link",
]

13
gen_unmapped_bin/Cargo.toml
View file

@ -1,13 +0,0 @@
[package]
name = "gen_unmapped_comprehensive"
version = "0.1.0"
edition = "2021"
publish = false
[dependencies]
anyhow = "1.0"
lopdf = "0.34"
[[bin]]
name = "gen_unmapped_comprehensive"
path = "../gen_unmapped_comprehensive.rs"

13
gen_unmapped_configurable_bin/Cargo.toml
View file

@ -1,13 +0,0 @@
[package]
name = "gen_unmapped_configurable"
version = "0.1.0"
edition = "2021"
publish = false
[dependencies]
anyhow = "1.0"
lopdf = "0.34"
[[bin]]
name = "gen_unmapped_configurable"
path = "../tests/fixtures/gen_unmapped_configurable.rs"

BIN
libstdin.rlib
View file

Binary file not shown.

126
test_api_null.c
View file

@ -1,126 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include "../../crates/pdftract-libpdftract/include/pdftract.h"
static int json_has_error(const char *json) {
return strstr(json, "\"error\"") != NULL;
}
static int json_has_code(const char *json, const char *code) {
char search[256];
snprintf(search, sizeof(search), "\"error\":\"%s\"", code);
return strstr(json, search) != NULL;
}
int main(void) {
printf("=== pdftract FFI API Surface Test ===\n\n");
// Test 1: pdftract_version (static string, don't free)
printf("Test 1: pdftract_version...\n");
const char *version = pdftract_version();
assert(version != NULL);
printf(" Version: %s\n", version);
printf(" PASS\n\n");
// Test 2: Null source handling - should return error JSON
printf("Test 2: Null source handling...\n");
char *result = pdftract_extract(NULL, "{}");
assert(result != NULL);
assert(json_has_error(result));
assert(json_has_code(result, "NULL_POINTER") || json_has_code(result, "PANIC"));
printf(" Error: %s\n", result);
pdftract_free(result);
printf(" PASS\n\n");
// Test 3: Null options_json handling - should return error JSON
printf("Test 3: Null options_json handling...\n");
result = pdftract_extract("/fake/path.pdf", NULL);
assert(result != NULL);
assert(json_has_error(result));
printf(" Error: %s\n", result);
pdftract_free(result);
printf(" PASS\n\n");
// Test 4: pdftract_free with null - should not crash
printf("Test 4: pdftract_free(null)...\n");
pdftract_free(NULL);
printf(" PASS\n\n");
// Test 5: pdftract_stream_close with null - should not crash
printf("Test 5: pdftract_stream_close(null)...\n");
pdftract_stream_close(NULL);
printf(" PASS\n\n");
// Test 6: pdftract_stream_next with null handle - should return error JSON
printf("Test 6: pdftract_stream_next(null handle)...\n");
result = pdftract_stream_next(NULL);
assert(result != NULL);
assert(json_has_error(result));
printf(" Error: %s\n", result);
pdftract_free(result);
printf(" PASS\n\n");
// Test 7: Memory roundtrip - alloc and free many times
printf("Test 7: Memory roundtrip (100 iterations)...\n");
for (int i = 0; i < 100; i++) {
result = pdftract_extract(NULL, "{}");
assert(result != NULL);
pdftract_free(result);
}
printf(" PASS\n\n");
// Test 8: Invalid JSON in options - should return error
printf("Test 8: Invalid JSON options...\n");
result = pdftract_extract("/fake/path.pdf", "not valid json");
assert(result != NULL);
assert(json_has_error(result));
printf(" Error: %s\n", result);
pdftract_free(result);
printf(" PASS\n\n");
// Test 9: All 12 functions exist and return non-null for valid inputs
printf("Test 9: Function existence check...\n");
// These should all return non-null (even if error JSON) for null inputs
result = pdftract_hash(NULL);
assert(result != NULL);
pdftract_free(result);
result = pdftract_classify(NULL);
assert(result != NULL);
pdftract_free(result);
result = pdftract_search(NULL, "pattern", "{}");
assert(result != NULL);
pdftract_free(result);
result = pdftract_get_metadata(NULL, "{}");
assert(result != NULL);
pdftract_free(result);
result = pdftract_extract_text(NULL, "{}");
assert(result != NULL);
pdftract_free(result);
result = pdftract_extract_markdown(NULL, "{}");
assert(result != NULL);
pdftract_free(result);
void *handle = pdftract_extract_stream_open(NULL, "{}");
// handle might be null on error, which is ok
printf(" PASS\n\n");
printf("=== All API surface tests passed! ===\n");
printf("\nNote: Full PDF parsing tests require Phase 1.2 completion.\n");
printf("The FFI API surface is correctly implemented with:\n");
printf(" - 12 exported symbols\n");
printf(" - Null pointer safety\n");
printf(" - Error JSON format\n");
printf(" - Memory management\n");
printf(" - Panic safety (catch_unwind)\n");
return 0;
}

14
test_audit_debug.rs
View file

@ -1,14 +0,0 @@
use pdftract_core::audit::{AuditLogWriter, AuditRecord};
use tempfile::tempdir;
fn main() {
let temp_dir = tempdir().unwrap();
let temp_file = temp_dir.path().join("audit.ndjson");
let writer = AuditLogWriter::open(&temp_file).unwrap();
let record = AuditRecord::new("extract", Some("pdftract-v1:abcd".to_string()), 1234, 200);
writer.write_record(&record).unwrap();
let contents = std::fs::read_to_string(&temp_file).unwrap();
println!("Output: {:?}", contents);
}

175
test_audit_integration.rs
View file

@ -1,175 +0,0 @@
//! Integration test for audit logging.
//!
//! This test verifies that:
//! 1. The --audit-log flag is accepted by serve, mcp, and inspect subcommands
//! 2. The audit log writer creates valid NDJSON output
//! 3. Log-policy enforcement redacts sensitive values
//! 4. Stdio MCP mode omits client_ip field
use pdftract_core::audit::{AuditLogWriter, AuditRecord};
use std::io::BufRead;
use std::path::PathBuf;
use tempfile::TempDir;
#[test]
fn test_audit_log_creates_valid_ndjson() {
let temp_dir = TempDir::new().unwrap();
let audit_path = temp_dir.path().join("audit.ndjson");
let writer = AuditLogWriter::open(&audit_path).unwrap();
// Write a sample audit record
let record = AuditRecord::new("extract", Some("pdftract-v1:abcd1234".to_string()), 1234, 200)
.with_client_ip("10.0.0.1")
.with_diagnostics(vec!["XREF_REPAIRED".to_string()]);
writer.write_record(&record).unwrap();
// Read back and verify
let file = std::fs::File::open(&audit_path).unwrap();
let reader = std::io::BufReader::new(file);
let lines: Vec<String> = reader.lines().map(|l| l.unwrap()).collect();
assert_eq!(lines.len(), 1, "Should have exactly one line");
let line = &lines[0];
let parsed: serde_json::Value = serde_json::from_str(line).unwrap();
assert_eq!(parsed["tool"], "extract");
assert_eq!(parsed["fingerprint"], "pdftract-v1:abcd1234");
assert_eq!(parsed["duration_ms"], 1234);
assert_eq!(parsed["status"], 200);
assert_eq!(parsed["client_ip"], "10.0.0.1");
assert_eq!(parsed["diagnostics"].as_array().unwrap().len(), 1);
assert_eq!(parsed["diagnostics"][0], "XREF_REPAIRED");
// Verify it has a timestamp field
assert!(parsed["ts"].is_string());
assert!(parsed["ts"].as_str().unwrap().len() > 0);
}
#[test]
fn test_audit_log_omit_client_ip_for_stdio() {
let temp_dir = TempDir::new().unwrap();
let audit_path = temp_dir.path().join("audit.ndjson");
let writer = AuditLogWriter::open(&audit_path).unwrap();
// Write a record without client_ip (stdio mode)
let record = AuditRecord::new("mcp.extract", None, 500, 500);
writer.write_record(&record).unwrap();
// Read back and verify
let file = std::fs::File::open(&audit_path).unwrap();
let reader = std::io::BufReader::new(file);
let lines: Vec<String> = reader.lines().map(|l| l.unwrap()).collect();
let parsed: serde_json::Value = serde_json::from_str(&lines[0]).unwrap();
// client_ip field should be absent for stdio mode
assert!(parsed.get("client_ip").is_none(), "client_ip should be absent for stdio mode");
}
#[test]
fn test_audit_log_appends_multiple_records() {
let temp_dir = TempDir::new().unwrap();
let audit_path = temp_dir.path().join("audit.ndjson");
let writer = AuditLogWriter::open(&audit_path).unwrap();
// Write multiple records
for i in 0..5 {
let record = AuditRecord::new("extract", Some(format!("pdftract-v1:{:x}", i)), i * 100, 200);
writer.write_record(&record).unwrap();
}
// Read back and verify
let file = std::fs::File::open(&audit_path).unwrap();
let reader = std::io::BufReader::new(file);
let lines: Vec<String> = reader.lines().map(|l| l.unwrap()).collect();
assert_eq!(lines.len(), 5, "Should have 5 lines");
}
#[test]
fn test_audit_log_policy_enforcement_redacts_secrets() {
use pdftract_core::log_policy;
// Test that password patterns are redacted
let line_with_password = "user:john password:secret123 action:extract";
let redacted = log_policy::redact_audit_log_line(line_with_password);
assert!(redacted.contains("[REDACTED]"));
assert!(!redacted.contains("secret123"));
// Test that bearer tokens are redacted
let line_with_token = "Authorization: Bearer abc123xyz456";
let redacted = log_policy::redact_audit_log_line(line_with_token);
assert!(redacted.contains("[REDACTED]"));
assert!(!redacted.contains("abc123xyz456"));
// Test that cookies are redacted
let line_with_cookie = "Cookie: session_id=secret_value";
let redacted = log_policy::redact_audit_log_line(line_with_cookie);
assert!(redacted.contains("[REDACTED]"));
assert!(!redacted.contains("secret_value"));
// Test that normal content is preserved
let normal_line = r#"{"tool":"extract","fingerprint":"pdftract-v1:abcd"}"#;
let redacted = log_policy::redact_audit_log_line(normal_line);
assert!(redacted.contains("extract"));
assert!(redacted.contains("pdftract-v1:abcd"));
assert!(!redacted.contains("[REDACTED]"));
}
#[test]
fn test_audit_record_matches_plan_spec() {
// Verify the AuditRecord matches the spec from plan lines 974-978
let record = AuditRecord::new("extract", Some("pdftract-v1:abcd1234".to_string()), 1234, 200)
.with_client_ip("10.0.0.1")
.with_diagnostics(vec!["XREF_REPAIRED".to_string()]);
let json = serde_json::to_string(&record).unwrap();
let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
// Verify all required fields are present
assert!(parsed["ts"].is_string(), "ts field must be present (ISO-8601 timestamp)");
assert!(parsed["client_ip"].is_string(), "client_ip field must be present");
assert!(parsed["tool"].is_string(), "tool field must be present");
assert!(parsed["fingerprint"].is_string(), "fingerprint field must be present");
assert!(parsed["duration_ms"].is_number(), "duration_ms field must be present");
assert!(parsed["status"].is_number(), "status field must be present (u16 HTTP-style)");
assert!(parsed["diagnostics"].is_array(), "diagnostics field must be present (Vec<String>)");
}
#[test]
fn test_audit_log_writer_crash_safety() {
let temp_dir = TempDir::new().unwrap();
let audit_path = temp_dir.path().join("audit.ndjson");
let writer = AuditLogWriter::open(&audit_path).unwrap();
// Write a record and verify it's flushed immediately
let record = AuditRecord::new("extract", Some("pdftract-v1:abcd".to_string()), 100, 200);
writer.write_record(&record).unwrap();
// Read back immediately - the record should be there (flushed)
let contents = std::fs::read_to_string(&audit_path).unwrap();
assert!(contents.contains("extract"), "Record should be flushed immediately");
assert!(contents.ends_with('\n'), "Record should end with newline");
}
#[test]
fn test_audit_record_serialization_is_single_line() {
let record = AuditRecord::new("extract", Some("pdftract-v1:abcd".to_string()), 1234, 200)
.with_diagnostics(vec!["XREF_REPAIRED".to_string(), "STREAM_BOMB".to_string()]);
let json = serde_json::to_string(&record).unwrap();
// Verify it's a single line (no newlines)
assert!(!json.contains('\n'), "Audit record should be single-line JSON");
assert!(!json.contains('\r'), "Audit record should not contain carriage returns");
// Verify it's valid JSON
let _parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
}

50
test_bomb_debug.rs
View file

@ -1,50 +0,0 @@
use std::time::Instant;
// Minimal test to check if FlateDecode bomb limit works
fn main() {
let bomb_data = std::fs::read("tests/stream_decoder/fixtures/flate_bomb_3gb.bin")
.expect("Failed to read bomb fixture");
println!("Bomb fixture size: {} bytes", bomb_data.len());
let start = Instant::now();
let mut counter = 0;
let bomb_limit = 1_000_000_000; // 1 GB
// Try to decode with flate2 directly first
println!("Testing with flate2 ZlibDecoder...");
use flate2::read::ZlibDecoder;
let mut decoder = ZlibDecoder::new(&bomb_data[..]);
let mut output = Vec::new();
let mut chunk = [0u8; 64 * 1024];
let mut total_bytes = 0u64;
loop {
match decoder.read(&mut chunk) {
Ok(0) => break,
Ok(n) => {
total_bytes += n as u64;
if total_bytes > bomb_limit {
println!(" Hit bomb limit after {} bytes", total_bytes);
break;
}
if output.len() < 10_000_000 {
output.extend_from_slice(&chunk[..n]);
}
}
Err(e) => {
println!(" Decode error: {}", e);
break;
}
}
}
let elapsed = start.elapsed();
println!(" Decoded {} bytes in {:?}", total_bytes, elapsed);
println!(" First 100 bytes of output: {:02x?}", &output[..100.min(output.len())]);
}
fn read(_buf: &mut [u8]) -> std::io::Result<usize> {
Ok(0)
}

BIN
test_classifier_corpus
View file

Binary file not shown.

62
test_debug_pdf.rs
View file

@ -1,62 +0,0 @@
use pdftract_core::parser::xref::load_xref_with_prev_chain;
use pdftract_core::parser::stream::{FileSource, PdfSource};
use std::path::Path;
fn main() {
let pdf_path = Path::new("crates/pdftract-core/tests/document_model/fixtures/ocg_default_off.pdf");
// Open the PDF file
let source = FileSource::open(pdf_path).expect("Failed to open PDF file");
// Find the startxref offset
let startxref_offset = find_startxref(&source).expect("Failed to find startxref offset");
println!("startxref offset: {}", startxref_offset);
// Try to load the xref
let xref = load_xref_with_prev_chain(&source, startxref_offset);
println!("Xref trailer: {:?}", xref.trailer);
if let Some(trailer) = &xref.trailer {
println!("Trailer keys: {:?}", trailer.keys().collect::<Vec<_>>());
if let Some(root) = trailer.get("Root") {
println!("Root: {:?}", root);
} else {
println!("No Root key in trailer!");
}
} else {
println!("No trailer found!");
}
}
fn find_startxref(source: &FileSource) -> Result<u64, Box<dyn std::error::Error>> {
// Read the last 1KB of the file to find startxref
let file_size = source.len()?;
let read_size = 1024.min(file_size);
let read_offset = file_size - read_size;
let tail = source.read_at(read_offset, read_size as usize)?;
let tail_str = std::str::from_utf8(&tail)?;
// Find "startxref" keyword
if let Some(pos) = tail_str.find("startxref") {
let offset_start = pos + "startxref".len();
// Find the offset after startxref (whitespace then number)
let offset_str = &tail_str[offset_start..];
let offset_str = offset_str.trim();
if let Some(end) = offset_str.find(|c: char| !c.is_ascii_digit() && c != '-') {
let offset_str = &offset_str[..end];
if let Ok(offset) = offset_str.parse::<u64>() {
return Ok(offset);
}
}
// Try to parse the entire line as the offset
if let Ok(offset) = offset_str.parse::<u64>() {
return Ok(offset);
}
}
Err("startxref not found".into())
}

21
test_debug_serialization.rs
View file

@ -1,21 +0,0 @@
// Quick test to understand serialization format
use pdftract_core::fingerprint::canonicalize::{serialize_dict_canonical, serialize_object_canonical};
use pdftract_core::types::objects::{PdfDict, PdfObject};
use std::sync::Arc;
fn main() {
let mut dict = PdfDict::new();
dict.insert(Arc::from("/Z"), PdfObject::Integer(3));
dict.insert(Arc::from("/A"), PdfObject::Integer(1));
dict.insert(Arc::from("/M"), PdfObject::Integer(2));
let bytes = serialize_dict_canonical(&dict);
println!("serialize_dict_canonical output: {}", String::from_utf8_lossy(&bytes));
println!("bytes: {:?}", bytes);
println!("\n--- serialize_object_canonical ---");
let mut result = Vec::new();
serialize_object_canonical(&mut result, &PdfObject::Dict(Box::new(dict)));
println!("serialize_object_canonical output: {}", String::from_utf8_lossy(&result));
println!("bytes: {:?}", result);
}

BIN
test_empty
View file

Binary file not shown.

17
test_empty.c
View file

@ -1,17 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include "/home/coding/pdftract/crates/pdftract-libpdftract/include/pdftract.h"
int main() {
const char *path = "/home/coding/pdftract/fuzz/corpus/lexer/empty.pdf";
char *result = pdftract_hash(path);
if (result == NULL) {
const char *err = pdftract_last_error();
printf("pdftract_hash returned NULL\n");
printf("last_error: %s\n", err ? err : "(null)");
return 1;
}
printf("Result: %s\n", result);
pdftract_free(result);
return 0;
}

12
test_extract.rs
View file

@ -1,12 +0,0 @@
use pdftract_core::{extract_pdf, ExtractionOptions};
fn main() {
let result = extract_pdf(
"tests/sdk-conformance/fixtures/mixed/mixed.pdf",
&ExtractionOptions::default()
);
match result {
Ok(doc) => println!("Success! Pages: {}", doc.pages.len()),
Err(e) => println!("Error: {}", e),
}
}

13
test_fingerprint_debug.rs
View file

@ -1,13 +0,0 @@
use pdftract_core::fingerprint::canonicalize::normalize_content_bytes;
fn main() {
let v1 = b"\n BT\n /F1 12 Tf\n 50 700 Td\n (Hello World) Tj\n ET\n ";
let v2 = b"\n BT\n /F1 12 Tf\n 50 700 Td\n (Hello Worl) Tj\n ET\n ";
let norm1 = normalize_content_bytes(v1);
let norm2 = normalize_content_bytes(v2);
println!("v1 normalized ({} bytes): {:?}", norm1.len(), String::from_utf8_lossy(&norm1));
println!("v2 normalized ({} bytes): {:?}", norm2.len(), String::from_utf8_lossy(&norm2));
println!("Equal: {}", norm1 == norm2);
}

33
test_fingerprint_debug_content.rs
View file

@ -1,33 +0,0 @@
// Test: Debug fingerprint content stream decoding
use pdftract_core::document::compute_pdf_fingerprint;
fn main() {
let v1_path = std::path::PathBuf::from("tests/fingerprint/fixtures/content_edit_one_glyph/v1.pdf");
let v2_path = std::path::PathBuf::from("tests/fingerprint/fixtures/content_edit_one_glyph/v2.pdf");
let fp1 = compute_pdf_fingerprint(&v1_path).expect("Failed to compute v1");
let fp2 = compute_pdf_fingerprint(&v2_path).expect("Failed to compute v2");
println!("v1 fingerprint: {}", fp1);
println!("v2 fingerprint: {}", fp2);
println!("Equal: {}", fp1 == fp2);
// Let's also check file sizes
let v1_meta = std::fs::metadata(&v1_path).unwrap();
let v2_meta = std::fs::metadata(&v2_path).unwrap();
println!("v1 size: {} bytes", v1_meta.len());
println!("v2 size: {} bytes", v2_meta.len());
// And file hashes
use std::io::Read;
let mut v1_bytes = Vec::new();
let mut v2_bytes = Vec::new();
std::fs::File::open(&v1_path).unwrap().read_to_end(&mut v1_bytes).unwrap();
std::fs::File::open(&v2_path).unwrap().read_to_end(&mut v2_bytes).unwrap();
use sha2::{Digest, Sha256};
let v1_hash = Sha256::digest(&v1_bytes);
let v2_hash = Sha256::digest(&v2_bytes);
println!("v1 SHA256: {}", hex::encode(v1_hash));
println!("v2 SHA256: {}", hex::encode(v2_hash));
}

24
test_fixture_debug.py
View file

@ -1,24 +0,0 @@
#!/usr/bin/env python3
import subprocess
import sys
# Simple debug script to check fixture decoding
fixtures = [
"lzw_early_change_0",
"lzw_early_change_1",
"filter_array_a85_then_flate",
"flate_png_pred15_all_six",
]
for fixture in fixtures:
print(f"\n=== Testing {fixture} ===")
bin_file = f"tests/stream_decoder/fixtures/{fixture}.bin"
exp_file = f"tests/stream_decoder/fixtures/{fixture}.expected"
with open(bin_file, "rb") as f:
bin_data = f.read()
with open(exp_file, "rb") as f:
exp_data = f.read()
print(f" Input ({len(bin_data)} bytes): {bin_data.hex()[:60]}...")
print(f" Expected ({len(exp_data)} bytes): {exp_data[:40]}...")

32
test_flate.rs
View file

@ -1,32 +0,0 @@
use flate2::write::ZlibEncoder;
use flate2::Compression;
use flate2::read::ZlibDecoder;
use std::io::{Write, Read};
fn main() {
let header = b"1 0 2 3";
let obj1 = b"42";
let obj2 = b"true";
let mut stream_data = Vec::new();
stream_data.extend_from_slice(header);
stream_data.extend_from_slice(obj1);
stream_data.extend_from_slice(obj2);
println!("Original data: {:?}", stream_data);
println!("Original data as string: {:?}", String::from_utf8_lossy(&stream_data));
let mut encoder = ZlibEncoder::new(Vec::new(), Compression::default());
encoder.write_all(&stream_data).unwrap();
let compressed = encoder.finish().unwrap();
println!("Compressed: {:?}", compressed);
println!("Compressed len: {}", compressed.len());
// Now try to decompress
let mut decoder = ZlibDecoder::new(&compressed[..]);
let mut decompressed = Vec::new();
decoder.read_to_end(&mut decompressed).unwrap();
println!("Decompressed: {:?}", decompressed);
println!("Decompressed as string: {:?}", String::from_utf8_lossy(&decompressed));
}

BIN
test_page_class
View file

Binary file not shown.

BIN
test_parse_simple
View file

Binary file not shown.

BIN
test_pdf
View file

Binary file not shown.

132
test_stream_decode.rs
View file

@ -1,132 +0,0 @@
use pdftract_core::parser::lexer::Lexer;
use std::env;
use std::fs::File;
use std::io::Read;
use std::path::Path;
fn decode_flate(data: &[u8]) -> Result<Vec<u8>, String> {
use flate2::read::DeflateDecoder;
use std::io::Read;
let mut decoder = DeflateDecoder::new(data);
let mut decompressed = Vec::new();
decoder.read_to_end(&mut decompressed).map_err(|e| format!("Decompression failed: {}", e))?;
Ok(decompressed)
}
fn find_and_decode_stream(pdf_data: &[u8]) -> Option<Vec<u8>> {
let stream_start = pdf_data.windows(7).position(|w| w == b"stream\n")?;
let start = stream_start + 7;
let end = pdf_data[start..].windows(9).position(|w| w == b"endstream")? + start;
let compressed = &pdf_data[start..end];
// Try deflate decompression
match decode_flate(compressed) {
Ok(decompressed) => Some(decompressed),
Err(e) => {
eprintln!("Decompression error: {}", e);
None
}
}
}
fn normalize_content(bytes: &[u8]) -> Vec<u8> {
if bytes.is_empty() {
return Vec::new();
}
let mut lexer = Lexer::new(bytes);
let mut result = Vec::new();
let mut first_token = true;
while let Some(token) = lexer.next_token() {
match token {
pdftract_core::parser::lexer::Token::Eof => break,
_ => {
if !first_token {
result.push(b' ');
}
first_token = false;
serialize_token(&mut result, &token);
}
}
}
result
}
fn serialize_token(output: &mut Vec<u8>, token: &pdftract_core::parser::lexer::Token) {
use pdftract_core::parser::lexer::Token;
match token {
Token::Bool(true) => output.extend_from_slice(b"true"),
Token::Bool(false) => output.extend_from_slice(b"false"),
Token::Integer(i) => {
let s = i.to_string();
output.extend_from_slice(s.as_bytes());
}
Token::Real(r) => {
let s = format!("{:.6}", r);
output.extend_from_slice(s.as_bytes());
}
Token::String(bytes) => {
output.push(b'(');
for &byte in bytes.as_ref() {
match byte {
b'(' | b')' | b'\\' => {
output.push(b'\\');
output.push(byte);
}
_ => output.push(byte),
}
}
output.push(b')');
}
Token::Name(bytes) => {
output.push(b'/');
output.extend_from_slice(bytes);
}
Token::ArrayStart => output.push(b'['),
Token::ArrayEnd => output.push(b']'),
Token::DictStart => output.extend_from_slice(b"<<"),
Token::DictEnd => output.extend_from_slice(b">>"),
Token::Stream => output.extend_from_slice(b"stream"),
Token::EndStream => output.extend_from_slice(b"endstream"),
Token::Obj => output.extend_from_slice(b"obj"),
Token::EndObj => output.extend_from_slice(b"endobj"),
Token::IndirectRef => output.push(b'R'),
Token::Null => output.extend_from_slice(b"null"),
Token::Keyword(bytes) => output.extend_from_slice(bytes),
Token::Eof => {}
}
}
fn main() {
let args: Vec<String> = env::args().collect();
if args.len() < 2 {
eprintln!("Usage: {} <pdf-path>", args[0]);
return;
}
let pdf_path = Path::new(&args[1]);
let mut pdf_data = Vec::new();
if let Err(e) = File::open(pdf_path).and_then(|mut f| f.read_to_end(&mut pdf_data)) {
eprintln!("Failed to read PDF: {}", e);
return;
}
if let Some(decoded) = find_and_decode_stream(&pdf_data) {
println!("Decoded stream bytes:");
println!("{:?}", decoded);
println!();
let normalized = normalize_content(&decoded);
println!("Normalized content:");
println!("{}", String::from_utf8_lossy(&normalized));
println!("Normalized bytes:");
println!("{:?}", normalized);
} else {
eprintln!("Failed to find/decode stream");
}
}

41
test_trailer.rs
View file

@ -1,41 +0,0 @@
use pdftract_core::parser::xref::load_xref_with_prev_chain;
use pdftract_core::parser::stream::FileSource as ParserFileSource;
fn main() {
let source = ParserFileSource::open("tests/document_model/fixtures/tagged_3_level_outline.pdf").unwrap();
// Find startxref
let startxref_offset = find_startxref(&source).unwrap();
println!("startxref offset: {}", startxref_offset);
// Load xref
let xref_section = load_xref_with_prev_chain(&source, startxref_offset);
println!("trailer: {:?}", xref_section.trailer);
if let Some(trailer) = &xref_section.trailer {
println!("trailer keys: {:?}", trailer.keys().collect::<Vec<_>>());
println!("trailer get Root: {:?}", trailer.get("Root"));
}
}
fn find_startxref(source: &ParserFileSource) -> Result<u64, Box<dyn std::error::Error>> {
let file_len = source.len()?;
// Scan last 1024 bytes for startxref
let scan_start = if file_len > 1024 { file_len - 1024 } else { 0 };
let scan_end = file_len;
let scan_size = (scan_end - scan_start) as usize;
let bytes = source.read_at(scan_start, scan_size)?;
let content = std::str::from_utf8(&bytes).ok();
if let Some(content) = content {
if let Some(pos) = content.find("startxref") {
let offset_str = &content[pos + "startxref".len()..];
let offset = offset_str.trim().parse::<u64>()?;
return Ok(offset);
}
}
Err("startxref not found".into())
}

BIN
test_trailer_debug
View file

Binary file not shown.

41
test_trailer_debug.rs
View file

@ -1,41 +0,0 @@
use pdftract_core::parser::xref::parse_traditional_xref;
use pdftract_core::parser::stream::MemorySource;
fn main() {
let pdf_data = std::fs::read("tests/fingerprint/fixtures/byte_identical/v1.pdf").unwrap();
// Find the trailer location manually
if let Some(trailer_pos) = pdf_data.windows(7).position(|w| w == b"trailer") {
println!("Found 'trailer' at offset: {}", trailer_pos);
// Print 100 bytes after "trailer"
let end_pos = (trailer_pos + 100).min(pdf_data.len());
println!("Bytes after 'trailer':");
for i in trailer_pos..end_pos {
if pdf_data[i] >= 32 && pdf_data[i] <= 126 {
print!("{}", pdf_data[i] as char);
} else {
print!("\\x{:02x}", pdf_data[i]);
}
}
println!();
}
let source = MemorySource::new(pdf_data);
let xref = parse_traditional_xref(&source, 439);
println!("\nXref entries: {}", xref.entries.len());
println!("Diagnostics: {}", xref.diagnostics.len());
for diag in &xref.diagnostics {
println!(" - {:?}: {}", diag.code, diag.message);
}
if let Some(trailer) = &xref.trailer {
println!("\nTrailer keys:");
for (key, value) in trailer.iter() {
println!(" '{}': {:?}", key, value);
}
} else {
println!("\nNo trailer found");
}
}

BIN
test_trailer_debug2
View file

Binary file not shown.

85
test_trailer_debug2.rs
View file

@ -1,85 +0,0 @@
use pdftract_core::parser::stream::{MemorySource, PdfSource};
// Manually implement a simple xref parser that logs everything
fn manual_parse(source: &MemorySource, start_offset: u64) {
let mut pos = start_offset;
println!("Starting manual parse at offset {}", pos);
// Read 100 bytes to see what we're starting with
let chunk = source.read_at(pos, 100).unwrap();
println!("Bytes at offset {}:", pos);
for i in 0..chunk.len().min(100) {
if chunk[i] >= 32 && chunk[i] <= 126 {
print!("{}", chunk[i] as char);
} else {
print!("\\x{:02x}", chunk[i]);
}
}
println!();
// Find "xref"
let xref_pos = chunk.windows(4).position(|w| w == b"xref").unwrap();
println!("Found 'xref' at relative offset {}", xref_pos);
pos += xref_pos as u64 + 4;
// Skip line ending
let le_chunk = source.read_at(pos, 2).unwrap();
if le_chunk[0] == b'\n' || le_chunk[0] == b'\r' {
pos += 1;
if le_chunk[0] == b'\r' && le_chunk.len() > 1 && le_chunk[1] == b'\n' {
pos += 1;
}
}
// Read the subsection header
let header_chunk = source.read_at(pos, 100).unwrap();
let header_str = std::str::from_utf8(&header_chunk).unwrap();
let header_line = header_str.lines().next().unwrap();
println!("Subsection header: '{}'", header_line);
// Parse xref entries...
let parts: Vec<&str> = header_line.split_whitespace().collect();
let obj_count: u32 = parts[1].parse().unwrap();
println!("Parsing {} xref entries", obj_count);
// Skip header line and parse entries
let header_end = pos + header_line.len() as u64 + 1;
pos = header_end + (obj_count as u64 * 20); // Assume 20-byte entries
// Now check for trailer
let trailer_chunk = source.read_at(pos, 100).unwrap();
let trailer_str = std::str::from_utf8(&trailer_chunk).unwrap();
println!("Bytes at offset {} (expecting trailer):", pos);
for i in 0..trailer_chunk.len().min(50) {
if trailer_chunk[i] >= 32 && trailer_chunk[i] <= 126 {
print!("{}", trailer_chunk[i] as char);
} else {
print!("\\x{:02x}", trailer_chunk[i]);
}
}
println!();
if trailer_str.trim_start().starts_with("trailer") {
println!("Found trailer keyword!");
let ws_offset = trailer_str.len() - trailer_str.trim_start().len();
let trailer_end = pos + ws_offset as u64 + 7;
let dict_chunk = source.read_at(trailer_end, 100).unwrap();
println!("Bytes at offset {} (expecting dict):", trailer_end);
for i in 0..dict_chunk.len().min(50) {
if dict_chunk[i] >= 32 && dict_chunk[i] <= 126 {
print!("{}", dict_chunk[i] as char);
} else {
print!("\\x{:02x}", dict_chunk[i]);
}
}
println!();
}
}
fn main() {
let pdf_data = std::fs::read("tests/fingerprint/fixtures/byte_identical/v1.pdf").unwrap();
let source = MemorySource::new(pdf_data);
manual_parse(&source, 439);
}

21
test_trailer_key.rs
View file

@ -1,21 +0,0 @@
use pdftract_core::parser::xref::load_xref_with_prev_chain;
use pdftract_core::source::file_source::ParserFileSource;
use pdftract_core::parser::xref::find_startxref;
fn main() {
let source = ParserFileSource::open(std::path::Path::new("tests/fingerprint/fixtures/acrobat_resave/v1.pdf")).unwrap();
let startxref_offset = find_startxref(&source).unwrap();
let xref_section = load_xref_with_prev_chain(&source, startxref_offset);
if let Some(trailer) = &xref_section.trailer {
println!("Trailer keys:");
for key in trailer.keys() {
println!(" '{}'", key);
}
println!("\nLooking for 'Root': {:?}", trailer.get("Root"));
println!("Looking for '/Root': {:?}", trailer.get("/Root"));
} else {
println!("No trailer found!");
}
}

20
test_trailer_parse.rs
View file

@ -1,20 +0,0 @@
use pdftract_core::parser::xref::parse_traditional_xref;
use pdftract_core::parser::stream::MemorySource;
fn main() {
let pdf_data = std::fs::read("tests/fingerprint/fixtures/byte_identical/v1.pdf").unwrap();
let source = MemorySource::new(pdf_data);
let xref = parse_traditional_xref(&source, 0);
println!("Trailer keys:");
if let Some(trailer) = &xref.trailer {
for (key, value) in trailer.iter() {
println!(" '{}': {:?}", key, value);
}
println!("\nTrying to get 'Root': {:?}", trailer.get("Root"));
println!("Trying to get '/Root': {:?}", trailer.get("/Root"));
} else {
println!("No trailer found");
}
}

23
test_trailer_parse2.rs
View file

@ -1,23 +0,0 @@
use pdftract_core::parser::xref::parse_traditional_xref;
use pdftract_core::parser::stream::MemorySource;
fn main() {
let pdf_data = std::fs::read("tests/fingerprint/fixtures/byte_identical/v1.pdf").unwrap();
let source = MemorySource::new(pdf_data);
// The xref starts at offset 439 according to startxref
let xref = parse_traditional_xref(&source, 439);
println!("Trailer keys:");
if let Some(trailer) = &xref.trailer {
for (key, value) in trailer.iter() {
println!(" '{}': {:?}", key, value);
}
println!("\nTrying to get 'Root': {:?}", trailer.get("Root"));
println!("Trying to get '/Root': {:?}", trailer.get("/Root"));
} else {
println!("No trailer found");
}
println!("\nXref entries: {}", xref.entries.len());
}

20
test_trailer_parsing.rs
View file

@ -1,20 +0,0 @@
use pdftract_core::document::parse_pdf_file;
use std::path::Path;
fn main() {
let pdf_path = Path::new("/tmp/valid_test.pdf");
match parse_pdf_file(pdf_path) {
Ok((fingerprint, catalog, pages, resolver)) => {
println!("Success!");
println!("Fingerprint: {}", fingerprint);
println!("Pages: {}", pages.len());
}
Err(e) => {
println!("Error: {}", e);
println!("Error chain:");
for cause in e.chain() {
println!(" - {}", cause);
}
}
}
}

BIN
tests/fingerprint/fixtures/linearization_toggle/v2.pdf.bak
View file

Binary file not shown.

BIN
tests/fixtures/gen_fixtures vendored
View file

Binary file not shown.

BIN
tests/fixtures/gen_ocr_fixtures vendored
View file

Binary file not shown.

BIN
tests/fixtures/gen_suspects vendored
View file

Binary file not shown.

171
tests/fixtures/gen_suspects.rs vendored
View file

@ -1,171 +0,0 @@
//! Generate a minimal valid tagged PDF for testing Phase 7.1.4 coverage check.
//!
//! This creates a PDF with:
//! - /MarkInfo /Suspects true
//! - StructTree with ParentTree
//! - MCID-based content association
//!
//! The PDF is minimal but valid, using manual byte offsets for reliability.
use std::fs::File;
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Generate fixture 1: Suspects true, low coverage -> XY-cut fallback
generate_pdf("tests/fixtures/tagged-suspects-true.pdf", true, 6, 10)?;
// Generate fixture 2: Suspects false, low coverage -> trust StructTree
generate_pdf("tests/fixtures/tagged-suspects-false.pdf", false, 5, 10)?;
// Generate fixture 3: Suspects true, high coverage -> trust StructTree
generate_pdf("tests/fixtures/tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
Ok(())
}
fn generate_pdf(path: &str, suspects: bool, num_claimed: usize, num_total: usize) -> Result<(), Box<dyn std::error::Error>> {
let mut pdf = String::new();
// PDF header
pdf.push_str("%PDF-1.7\n");
// Object 1: Catalog
pdf.push_str("1 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Catalog\n");
pdf.push_str("/Pages 2 0 R\n");
pdf.push_str("/MarkInfo <<\n");
pdf.push_str(" /Marked true\n");
pdf.push_str(format!(" /Suspects {}\n", if suspects { "true" } else { "false" }).as_str());
pdf.push_str(">>\n");
pdf.push_str("/StructTreeRoot 3 0 R\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 2: Pages
pdf.push_str("2 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Pages\n");
pdf.push_str("/Kids [4 0 R]\n");
pdf.push_str("/Count 1\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 3: StructTreeRoot
pdf.push_str("3 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /StructTreeRoot\n");
pdf.push_str("/K [5 0 R]\n");
pdf.push_str("/ParentTree 6 0 R\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 4: Page
pdf.push_str("4 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Page\n");
pdf.push_str("/Parent 2 0 R\n");
pdf.push_str("/MediaBox [0 0 612 792]\n");
pdf.push_str("/Contents 7 0 R\n");
pdf.push_str("/StructParents 0\n");
pdf.push_str("/Resources << /Font << /F1 << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> >> >>\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 5: StructElem (paragraph)
pdf.push_str("5 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /StructElem\n");
pdf.push_str("/S /P\n");
pdf.push_str("/K [");
for i in 0..num_total {
pdf.push_str(&format!("{} ", i));
}
pdf.push_str("]\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 6: ParentTree (number tree with /Nums array)
pdf.push_str("6 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Nums [\n");
pdf.push_str("0 [");
for i in 0..num_total {
if i < num_claimed {
pdf.push_str(" 5 0 R");
} else {
pdf.push_str(" null");
}
if i < num_total - 1 {
pdf.push(' ');
}
}
pdf.push_str(" ]\n");
pdf.push_str("]\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 7: Content stream
pdf.push_str("7 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Length 44\n");
pdf.push_str(">>\n");
pdf.push_str("stream\n");
pdf.push_str("BT\n");
pdf.push_str("/F1 12 Tf\n");
pdf.push_str("100 700 Td\n");
pdf.push_str("(Test) Tj\n");
pdf.push_str("ET\n");
pdf.push_str("endstream\n");
pdf.push_str("endobj\n");
// Calculate xref offset (current position + "xref\n" + start of table)
let xref_offset = pdf.len() + 5; // +5 for "xref\n"
// Build xref table
pdf.push_str("xref\n");
pdf.push_str("0 8\n");
pdf.push_str("0000000000 65535 f \n");
// We need to calculate byte offsets for each object
// Let's do this by building the PDF first, then computing offsets
let pdf_bytes = pdf.as_bytes();
let mut offsets = Vec::new();
let mut current = 0;
// Find each object offset by searching for "N 0 obj"
for n in 1..=7 {
let pattern = format!("{} 0 obj\n", n);
if let Some(pos) = pdf.find(&pattern) {
offsets.push(pos);
}
}
// Add xref entries
for (i, offset) in offsets.iter().enumerate() {
pdf.push_str(&format!("{:010} 00000 n \n", offset));
}
// Trailer
pdf.push_str("trailer\n");
pdf.push_str("<<\n");
pdf.push_str("/Size 8\n");
pdf.push_str("/Root 1 0 R\n");
pdf.push_str(">>\n");
// startxref
pdf.push_str(&format!("startxref\n{}\n", xref_offset));
// EOF
pdf.push_str("%%EOF\n");
// Write to file
let mut file = File::create(path)?;
file.write_all(pdf.as_bytes())?;
eprintln!("Created: {}", path);
eprintln!(" /Suspects: {}", suspects);
eprintln!(" Coverage: {}/{} MCIDs claimed", num_claimed, num_total);
Ok(())
}

BIN
tests/fixtures/gen_suspects_simple vendored
View file

Binary file not shown.

204
tests/fixtures/gen_suspects_simple.rs vendored
View file

@ -1,204 +0,0 @@
//! Simple Rust-based generator for Suspects test fixtures.
//!
//! Generates minimal valid tagged PDFs with:
//! - /MarkInfo /Suspects flag
//! - StructTree with ParentTree
//! - MCID marked content in content streams
use std::fs::File;
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("Generating Suspects test fixtures...");
// Fixture 1: Suspects true, 60% coverage (6/10 claimed) -> fallback to XY-cut
write_fixture("tagged-suspects-true.pdf", true, 6, 10)?;
// Fixture 2: Suspects false, 50% coverage (5/10 claimed) -> trust StructTree
write_fixture("tagged-suspects-false.pdf", false, 5, 10)?;
// Fixture 3: Suspects true, 95% coverage (19/20 claimed) -> trust StructTree
write_fixture("tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
println!("All fixtures generated!");
Ok(())
}
fn write_fixture(
path: &str,
suspects: bool,
num_claimed: usize,
num_total: usize,
) -> Result<(), Box<dyn std::error::Error>> {
// Build the PDF content
let mut pdf = String::new();
// Header
pdf.push_str("%PDF-1.7\n");
// Object 1: Catalog
pdf.push_str("1 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Catalog\n");
pdf.push_str("/Pages 2 0 R\n");
pdf.push_str("/MarkInfo <<\n");
pdf.push_str(" /Marked true\n");
pdf.push_str(&format!(" /Suspects {}\n", if suspects { "true" } else { "false" }));
pdf.push_str(">>\n");
pdf.push_str("/StructTreeRoot 3 0 R\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 2: Pages
pdf.push_str("2 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Pages\n");
pdf.push_str("/Kids [4 0 R]\n");
pdf.push_str("/Count 1\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 3: StructTreeRoot
pdf.push_str("3 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /StructTreeRoot\n");
pdf.push_str("/K [5 0 R]\n");
pdf.push_str("/ParentTree 6 0 R\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 4: Page
pdf.push_str("4 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Page\n");
pdf.push_str("/Parent 2 0 R\n");
pdf.push_str("/MediaBox [0 0 612 792]\n");
pdf.push_str("/Contents 7 0 R\n");
pdf.push_str("/StructParents 0\n");
pdf.push_str("/Resources <<\n");
pdf.push_str("/Font <<\n");
pdf.push_str("/F1 <<\n");
pdf.push_str("/Type /Font\n");
pdf.push_str("/Subtype /Type1\n");
pdf.push_str("/BaseFont /Helvetica\n");
pdf.push_str(">>\n");
pdf.push_str(">>\n");
pdf.push_str(">>\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 5: StructElem (paragraph)
let k_array: String = (0..num_total).map(|i| i.to_string()).collect::<Vec<_>>().join(" ");
pdf.push_str("5 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /StructElem\n");
pdf.push_str("/S /P\n");
pdf.push_str(&format!("/K [{}]\n", k_array));
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 6: ParentTree
pdf.push_str("6 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Nums [\n");
pdf.push_str("0 [");
for i in 0..num_total {
if i < num_claimed {
pdf.push_str("5 0 R");
} else {
pdf.push_str("null");
}
if i < num_total - 1 {
pdf.push(' ');
}
}
pdf.push_str("]\n");
pdf.push_str("]\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 7: Content stream with MCID marked content
let mut content = String::new();
for i in 0..num_total {
let y = 700 - i * 15;
content.push_str(&format!(
"BT\n/F1 12 Tf\n100 {} Td\n/MCID {} BDC\n(Test{}) Tj\nEMC\nET\n",
y, i, i
));
}
let content_bytes = content.as_bytes();
let content_len = content_bytes.len();
pdf.push_str("7 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str(&format!("/Length {}\n", content_len));
pdf.push_str(">>\n");
pdf.push_str("stream\n");
pdf.push_str(&content);
pdf.push_str("endstream\n");
pdf.push_str("endobj\n");
// Now we have all the content, calculate xref
let pdf_bytes = pdf.as_bytes();
let mut offsets = vec![0u64; 8]; // Objects 0-7
// Find each object's offset by scanning the PDF string
let pdf_clone = pdf.clone();
for (obj_num, offset) in find_object_offsets(&pdf_clone) {
if obj_num < 8 {
offsets[obj_num] = offset;
}
}
// Build xref table
let xref_start = pdf_bytes.len() as u64;
pdf.push_str("xref\n");
pdf.push_str("0 8\n");
pdf.push_str("0000000000 65535 f \n");
for i in 1..=7 {
pdf.push_str(&format!("{:010} 00000 n \n", offsets[i]));
}
// Build trailer
pdf.push_str("trailer\n");
pdf.push_str("<<\n");
pdf.push_str("/Size 8\n");
pdf.push_str("/Root 1 0 R\n");
pdf.push_str(">>\n");
pdf.push_str(&format!("startxref\n{}\n", xref_start));
pdf.push_str("%%EOF\n");
// Write to file
let mut file = File::create(format!("tests/fixtures/{}", path))?;
file.write_all(pdf.as_bytes())?;
let coverage = (num_claimed as f64 / num_total as f64) * 100.0;
println!("Created: {}", path);
println!(" Suspects: {}, Coverage: {:.0}% ({}/{})",
suspects, coverage, num_claimed, num_total);
Ok(())
}
fn parse_obj_number(line: &str) -> Option<usize> {
let parts: Vec<&str> = line.split_whitespace().collect();
if parts.len() >= 2 && parts[1] == "0" && parts.get(2) == Some(&"obj") {
parts[0].parse().ok()
} else {
None
}
}
fn find_object_offsets(pdf: &str) -> Vec<(usize, u64)> {
let mut offsets = Vec::new();
let mut pos = 0u64;
for line in pdf.lines() {
if let Some(obj_num) = parse_obj_number(line) {
offsets.push((obj_num, pos));
}
pos += line.len() as u64 + 1; // +1 for newline
}
offsets
}

BIN
tests/fixtures/gen_suspects_simple_local vendored
View file

Binary file not shown.

204
tests/fixtures/gen_suspects_simple_local.rs vendored
View file

@ -1,204 +0,0 @@
//! Simple Rust-based generator for Suspects test fixtures.
//!
//! Generates minimal valid tagged PDFs with:
//! - /MarkInfo /Suspects flag
//! - StructTree with ParentTree
//! - MCID marked content in content streams
use std::fs::File;
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("Generating Suspects test fixtures...");
// Fixture 1: Suspects true, 60% coverage (6/10 claimed) -> fallback to XY-cut
write_fixture("tagged-suspects-true.pdf", true, 6, 10)?;
// Fixture 2: Suspects false, 50% coverage (5/10 claimed) -> trust StructTree
write_fixture("tagged-suspects-false.pdf", false, 5, 10)?;
// Fixture 3: Suspects true, 95% coverage (19/20 claimed) -> trust StructTree
write_fixture("tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
println!("All fixtures generated!");
Ok(())
}
fn write_fixture(
path: &str,
suspects: bool,
num_claimed: usize,
num_total: usize,
) -> Result<(), Box<dyn std::error::Error>> {
// Build the PDF content
let mut pdf = String::new();
// Header
pdf.push_str("%PDF-1.7\n");
// Object 1: Catalog
pdf.push_str("1 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Catalog\n");
pdf.push_str("/Pages 2 0 R\n");
pdf.push_str("/MarkInfo <<\n");
pdf.push_str(" /Marked true\n");
pdf.push_str(&format!(" /Suspects {}\n", if suspects { "true" } else { "false" }));
pdf.push_str(">>\n");
pdf.push_str("/StructTreeRoot 3 0 R\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 2: Pages
pdf.push_str("2 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Pages\n");
pdf.push_str("/Kids [4 0 R]\n");
pdf.push_str("/Count 1\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 3: StructTreeRoot
pdf.push_str("3 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /StructTreeRoot\n");
pdf.push_str("/K [5 0 R]\n");
pdf.push_str("/ParentTree 6 0 R\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 4: Page
pdf.push_str("4 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /Page\n");
pdf.push_str("/Parent 2 0 R\n");
pdf.push_str("/MediaBox [0 0 612 792]\n");
pdf.push_str("/Contents 7 0 R\n");
pdf.push_str("/StructParents 0\n");
pdf.push_str("/Resources <<\n");
pdf.push_str("/Font <<\n");
pdf.push_str("/F1 <<\n");
pdf.push_str("/Type /Font\n");
pdf.push_str("/Subtype /Type1\n");
pdf.push_str("/BaseFont /Helvetica\n");
pdf.push_str(">>\n");
pdf.push_str(">>\n");
pdf.push_str(">>\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 5: StructElem (paragraph)
let k_array: String = (0..num_total).map(|i| i.to_string()).collect::<Vec<_>>().join(" ");
pdf.push_str("5 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Type /StructElem\n");
pdf.push_str("/S /P\n");
pdf.push_str(&format!("/K [{}]\n", k_array));
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 6: ParentTree
pdf.push_str("6 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str("/Nums [\n");
pdf.push_str("0 [");
for i in 0..num_total {
if i < num_claimed {
pdf.push_str("5 0 R");
} else {
pdf.push_str("null");
}
if i < num_total - 1 {
pdf.push(' ');
}
}
pdf.push_str("]\n");
pdf.push_str("]\n");
pdf.push_str(">>\n");
pdf.push_str("endobj\n");
// Object 7: Content stream with MCID marked content
let mut content = String::new();
for i in 0..num_total {
let y = 700 - i * 15;
content.push_str(&format!(
"BT\n/F1 12 Tf\n100 {} Td\n/MCID {} BDC\n(Test{}) Tj\nEMC\nET\n",
y, i, i
));
}
let content_bytes = content.as_bytes();
let content_len = content_bytes.len();
pdf.push_str("7 0 obj\n");
pdf.push_str("<<\n");
pdf.push_str(&format!("/Length {}\n", content_len));
pdf.push_str(">>\n");
pdf.push_str("stream\n");
pdf.push_str(&content);
pdf.push_str("endstream\n");
pdf.push_str("endobj\n");
// Now we have all the content, calculate xref
let pdf_bytes = pdf.as_bytes();
let mut offsets = vec![0u64; 8]; // Objects 0-7
// Find each object's offset by scanning the PDF string
let pdf_clone = pdf.clone();
for (obj_num, offset) in find_object_offsets(&pdf_clone) {
if obj_num < 8 {
offsets[obj_num] = offset;
}
}
// Build xref table
let xref_start = pdf_bytes.len() as u64;
pdf.push_str("xref\n");
pdf.push_str("0 8\n");
pdf.push_str("0000000000 65535 f \n");
for i in 1..=7 {
pdf.push_str(&format!("{:010} 00000 n \n", offsets[i]));
}
// Build trailer
pdf.push_str("trailer\n");
pdf.push_str("<<\n");
pdf.push_str("/Size 8\n");
pdf.push_str("/Root 1 0 R\n");
pdf.push_str(">>\n");
pdf.push_str(&format!("startxref\n{}\n", xref_start));
pdf.push_str("%%EOF\n");
// Write to file (current directory)
let mut file = File::create(path)?;
file.write_all(pdf.as_bytes())?;
let coverage = (num_claimed as f64 / num_total as f64) * 100.0;
println!("Created: {}", path);
println!(" Suspects: {}, Coverage: {:.0}% ({}/{})",
suspects, coverage, num_claimed, num_total);
Ok(())
}
fn parse_obj_number(line: &str) -> Option<usize> {
let parts: Vec<&str> = line.split_whitespace().collect();
if parts.len() >= 2 && parts[1] == "0" && parts.get(2) == Some(&"obj") {
parts[0].parse().ok()
} else {
None
}
}
fn find_object_offsets(pdf: &str) -> Vec<(usize, u64)> {
let mut offsets = Vec::new();
let mut pos = 0u64;
for line in pdf.lines() {
if let Some(obj_num) = parse_obj_number(line) {
offsets.push((obj_num, pos));
}
pos += line.len() as u64 + 1; // +1 for newline
}
offsets
}

190
tests/fixtures/gen_suspects_v2.rs vendored
View file

@ -1,190 +0,0 @@
//! Generate a minimal valid tagged PDF for testing Phase 7.1.4 coverage check.
//!
//! This creates a PDF with:
//! - /MarkInfo /Suspects configurable
//! - StructTree with ParentTree
//! - MCID-based content association
//!
//! The PDF is minimal but valid, with correct xref table offsets.
use std::fs::File;
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Generate fixture 1: Suspects true, low coverage -> XY-cut fallback
generate_pdf("tests/fixtures/tagged-suspects-true.pdf", true, 6, 10)?;
// Generate fixture 2: Suspects false, low coverage -> trust StructTree
generate_pdf("tests/fixtures/tagged-suspects-false.pdf", false, 5, 10)?;
// Generate fixture 3: Suspects true, high coverage -> trust StructTree
generate_pdf("tests/fixtures/tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
Ok(())
}
fn generate_pdf(path: &str, suspects: bool, num_claimed: usize, num_total: usize) -> Result<(), Box<dyn std::error::Error>> {
let mut pdf_parts = Vec::new();
// PDF header
pdf_parts.push(b"%PDF-1.7\n".to_vec());
// Object 1: Catalog
let obj1 = format!(
"1 0 obj\n\
<<\n\
/Type /Catalog\n\
/Pages 2 0 R\n\
/MarkInfo <<\n\
/Marked true\n\
/Suspects {}\n\
>>\n\
/StructTreeRoot 3 0 R\n\
>>\n\
endobj\n",
if suspects { "true" } else { "false" }
);
pdf_parts.push(obj1.into_bytes());
// Object 2: Pages
let obj2 = "2 0 obj\n\
<<\n\
/Type /Pages\n\
/Kids [4 0 R]\n\
/Count 1\n\
>>\n\
endobj\n";
pdf_parts.push(obj2.as_bytes().to_vec());
pdf_parts.push(obj2.into_bytes());
// Object 3: StructTreeRoot
let obj3 = "3 0 obj\n\
<<\n\
/Type /StructTreeRoot\n\
/K [5 0 R]\n\
/ParentTree 6 0 R\n\
>>\n\
endobj\n".to_vec();
pdf_parts.push(obj3);
// Object 4: Page
let obj4 = "4 0 obj\n\
<<\n\
/Type /Page\n\
/Parent 2 0 R\n\
/MediaBox [0 0 612 792]\n\
/Contents 7 0 R\n\
/StructParents 0\n\
/Resources << /Font << /F1 << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> >> >>\n\
>>\n\
endobj\n".to_vec();
pdf_parts.push(obj4);
// Object 5: StructElem (paragraph) with MCID array
let mcid_array: Vec<String> = (0..num_total).map(|i| i.to_string()).collect();
let obj5 = format!(
"5 0 obj\n\
<<\n\
/Type /StructElem\n\
/S /P\n\
/K [{}]\n\
>>\n\
endobj\n",
mcid_array.join(" ")
);
pdf_parts.push(obj5.into_bytes());
// Object 6: ParentTree (number tree with /Nums array)
let mut parent_tree_entries = Vec::new();
for i in 0..num_total {
if i < num_claimed {
parent_tree_entries.push("5 0 R".to_string());
} else {
parent_tree_entries.push("null".to_string());
}
}
let obj6 = format!(
"6 0 obj\n\
<<\n\
/Nums [\n\
0 [{}]\n\
]\n\
>>\n\
endobj\n",
parent_tree_entries.join(" ")
);
pdf_parts.push(obj6.into_bytes());
// Object 7: Content stream
let obj7 = "7 0 obj\n\
<<\n\
/Length 44\n\
>>\n\
stream\n\
BT\n\
/F1 12 Tf\n\
100 700 Td\n\
(Test) Tj\n\
ET\n\
endstream\n\
endobj\n".to_vec();
pdf_parts.push(obj7);
// Build the PDF up to xref and calculate offsets
let mut pdf_before_xref = Vec::new();
for part in &pdf_parts {
pdf_before_xref.extend_from_slice(part);
}
// Calculate object offsets
let mut offsets = Vec::new();
let mut current = 0;
for part in &pdf_parts {
offsets.push(current);
current += part.len();
}
// xref starts after all objects
let xref_offset = current;
// Build xref table
let mut xref = Vec::new();
xref.push(b"xref\n".to_vec());
xref.push(b"0 8\n".to_vec());
xref.push(format!("{:010} 65535 f \n", 0).into_bytes());
for offset in offsets {
xref.push(format!("{:010} 00000 n \n", offset).into_bytes());
}
// Trailer
let trailer = format!(
"trailer\n\
<<\n\
/Size 8\n\
/Root 1 0 R\n\
>>\n\
startxref\n\
{}\n\
%%EOF\n",
xref_offset
);
// Combine everything
let mut final_pdf = Vec::new();
final_pdf.extend_from_slice(&pdf_before_xref);
for part in xref {
final_pdf.extend_from_slice(&part);
}
final_pdf.extend_from_slice(trailer.as_bytes());
// Write to file
let mut file = File::create(path)?;
file.write_all(&final_pdf)?;
eprintln!("Created: {}", path);
eprintln!(" /Suspects: {}", suspects);
eprintln!(" Coverage: {}/{} MCIDs claimed", num_claimed, num_total);
Ok(())
}

BIN
tests/fixtures/gen_suspects_v3 vendored
View file

Binary file not shown.

155
tests/fixtures/gen_suspects_v3.rs vendored
View file

@ -1,155 +0,0 @@
//! Generate a minimal valid tagged PDF for testing Phase 7.1.4 coverage check.
use std::fs::File;
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
generate_pdf("tests/fixtures/tagged-suspects-true.pdf", true, 6, 10)?;
generate_pdf("tests/fixtures/tagged-suspects-false.pdf", false, 5, 10)?;
generate_pdf("tests/fixtures/tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
Ok(())
}
fn generate_pdf(path: &str, suspects: bool, num_claimed: usize, num_total: usize) -> Result<(), Box<dyn std::error::Error>> {
let mut pdf_parts = Vec::new();
pdf_parts.push(b"%PDF-1.7\n".to_vec());
let obj1 = format!(
"1 0 obj\n\
<<\n\
/Type /Catalog\n\
/Pages 2 0 R\n\
/MarkInfo <<\n\
/Marked true\n\
/Suspects {}\n\
>>\n\
/StructTreeRoot 3 0 R\n\
>>\n\
endobj\n",
if suspects { "true" } else { "false" }
);
pdf_parts.push(obj1.into_bytes());
pdf_parts.push(b"2 0 obj\n\
<<\n\
/Type /Pages\n\
/Kids [4 0 R]\n\
/Count 1\n\
>>\n\
endobj\n".to_vec());
pdf_parts.push(b"3 0 obj\n\
<<\n\
/Type /StructTreeRoot\n\
/K [5 0 R]\n\
/ParentTree 6 0 R\n\
>>\n\
endobj\n".to_vec());
pdf_parts.push(b"4 0 obj\n\
<<\n\
/Type /Page\n\
/Parent 2 0 R\n\
/MediaBox [0 0 612 792]\n\
/Contents 7 0 R\n\
/StructParents 0\n\
/Resources << /Font << /F1 << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> >> >>\n\
>>\n\
endobj\n".to_vec());
let mcid_array: Vec<String> = (0..num_total).map(|i| i.to_string()).collect();
let obj5 = format!(
"5 0 obj\n\
<<\n\
/Type /StructElem\n\
/S /P\n\
/K [{}]\n\
>>\n\
endobj\n",
mcid_array.join(" ")
);
pdf_parts.push(obj5.into_bytes());
let mut parent_tree_entries = Vec::new();
for i in 0..num_total {
if i < num_claimed {
parent_tree_entries.push("5 0 R".to_string());
} else {
parent_tree_entries.push("null".to_string());
}
}
let obj6 = format!(
"6 0 obj\n\
<<\n\
/Nums [\n\
0 [{}]\n\
]\n\
>>\n\
endobj\n",
parent_tree_entries.join(" ")
);
pdf_parts.push(obj6.into_bytes());
pdf_parts.push(b"7 0 obj\n\
<<\n\
/Length 44\n\
>>\n\
stream\n\
BT\n\
/F1 12 Tf\n\
100 700 Td\n\
(Test) Tj\n\
ET\n\
endstream\n\
endobj\n".to_vec());
let mut pdf_before_xref = Vec::new();
for part in &pdf_parts {
pdf_before_xref.extend_from_slice(part);
}
let mut offsets = Vec::new();
let mut current = 0;
for part in &pdf_parts {
offsets.push(current);
current += part.len();
}
let xref_offset = current;
let mut xref = Vec::new();
xref.push(b"xref\n".to_vec());
xref.push(b"0 8\n".to_vec());
xref.push(format!("{:010} 65535 f \n", 0).into_bytes());
for offset in offsets {
xref.push(format!("{:010} 00000 n \n", offset).into_bytes());
}
let trailer = format!(
"trailer\n\
<<\n\
/Size 8\n\
/Root 1 0 R\n\
>>\n\
startxref\n\
{}\n\
%%EOF\n",
xref_offset
);
let mut final_pdf = Vec::new();
final_pdf.extend_from_slice(&pdf_before_xref);
for part in xref {
final_pdf.extend_from_slice(&part);
}
final_pdf.extend_from_slice(trailer.as_bytes());
let mut file = File::create(path)?;
file.write_all(&final_pdf)?;
eprintln!("Created: {}", path);
eprintln!(" /Suspects: {}", suspects);
eprintln!(" Coverage: {}/{} MCIDs claimed", num_claimed, num_total);
Ok(())
}

163
tests/fixtures/gen_suspects_v4.rs vendored
View file

@ -1,163 +0,0 @@
//! Generate a minimal valid tagged PDF for testing Phase 7.1.4 coverage check.
use std::fs::File;
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
generate_pdf("tests/fixtures/tagged-suspects-true.pdf", true, 6, 10)?;
generate_pdf("tests/fixtures/tagged-suspects-false.pdf", false, 5, 10)?;
generate_pdf("tests/fixtures/tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
Ok(())
}
fn generate_pdf(path: &str, suspects: bool, num_claimed: usize, num_total: usize) -> Result<(), Box<dyn std::error::Error>> {
let mut pdf = String::from("%PDF-1.7\n");
// Object 1: Catalog
pdf.push_str(&format!(
"1 0 obj\n\
<<\n\
/Type /Catalog\n\
/Pages 2 0 R\n\
/MarkInfo <<\n\
/Marked true\n\
/Suspects {}\n\
>>\n\
/StructTreeRoot 3 0 R\n\
>>\n\
endobj\n",
if suspects { "true" } else { "false" }
));
// Object 2: Pages
pdf.push_str(
"2 0 obj\n\
<<\n\
/Type /Pages\n\
/Kids [4 0 R]\n\
/Count 1\n\
>>\n\
endobj\n"
);
// Object 3: StructTreeRoot
pdf.push_str(
"3 0 obj\n\
<<\n\
/Type /StructTreeRoot\n\
/K [5 0 R]\n\
/ParentTree 6 0 R\n\
>>\n\
endobj\n"
);
// Object 4: Page
pdf.push_str(
"4 0 obj\n\
<<\n\
/Type /Page\n\
/Parent 2 0 R\n\
/MediaBox [0 0 612 792]\n\
/Contents 7 0 R\n\
/StructParents 0\n\
/Resources << /Font << /F1 << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> >> >>\n\
>>\n\
endobj\n"
);
// Object 5: StructElem (paragraph) with MCID array
let mcid_array: Vec<String> = (0..num_total).map(|i| i.to_string()).collect();
pdf.push_str(&format!(
"5 0 obj\n\
<<\n\
/Type /StructElem\n\
/S /P\n\
/K [{}]\n\
>>\n\
endobj\n",
mcid_array.join(" ")
));
// Object 6: ParentTree (number tree with /Nums array)
let mut parent_tree_entries = Vec::new();
for i in 0..num_total {
if i < num_claimed {
parent_tree_entries.push("5 0 R".to_string());
} else {
parent_tree_entries.push("null".to_string());
}
}
pdf.push_str(&format!(
"6 0 obj\n\
<<\n\
/Nums [\n\
0 [{}]\n\
]\n\
>>\n\
endobj\n",
parent_tree_entries.join(" ")
));
// Object 7: Content stream
pdf.push_str(
"7 0 obj\n\
<<\n\
/Length 44\n\
>>\n\
stream\n\
BT\n\
/F1 12 Tf\n\
100 700 Td\n\
(Test) Tj\n\
ET\n\
endstream\n\
endobj\n"
);
// Find the offset of each object by searching for "N 0 obj"
let mut offsets = vec![0usize; 8]; // Index 0 is dummy, 1-7 are actual objects
let mut current_pos = 0;
let pdf_bytes = pdf.as_bytes();
for n in 1..=7 {
let pattern = format!("{} 0 obj\n", n);
if let Some(pos) = pdf.find(&pattern) {
offsets[n] = pos;
}
}
// xref starts after all objects
let xref_offset = pdf.len();
// Build xref table
pdf.push_str("xref\n");
pdf.push_str("0 8\n");
pdf.push_str("0000000000 65535 f \n");
for n in 1..=7 {
pdf.push_str(&format!("{:010} 00000 n \n", offsets[n]));
}
// Trailer
pdf.push_str(&format!(
"trailer\n\
<<\n\
/Size 8\n\
/Root 1 0 R\n\
>>\n\
startxref\n\
{}\n\
%%EOF\n",
xref_offset
));
// Write to file
let mut file = File::create(path)?;
file.write_all(pdf.as_bytes())?;
eprintln!("Created: {}", path);
eprintln!(" /Suspects: {}", suspects);
eprintln!(" Coverage: {}/{} MCIDs claimed", num_claimed, num_total);
Ok(())
}

BIN
tests/fixtures/gen_suspects_v6 vendored
View file

Binary file not shown.

148
tests/fixtures/gen_suspects_v6.rs vendored
View file

@ -1,148 +0,0 @@
//! Generate tagged PDF fixtures for testing Phase 7.1.4 coverage check
//!
//! This creates three fixtures:
//! 1. tagged-suspects-true.pdf - Suspects true, 60% coverage -> fallback to XY-cut
//! 2. tagged-suspects-false.pdf - Suspects false, 50% coverage -> trust StructTree
//! 3. tagged-suspects-true-high-coverage.pdf - Suspects true, 95% coverage -> trust StructTree
use std::fs::File;
use std::io::Write;
fn write_pdf(path: &str, suspects: bool, num_claimed: usize, num_total: usize) -> Result<(), Box<dyn std::error::Error>> {
// Create ParentTree /Nums array with claimed and null entries
// Format: /Nums [0 [ref ref null ref ...]]
let mut nums_content = String::from(" /Nums [\n 0 [");
for i in 0..num_total {
if i < num_claimed {
nums_content.push_str(" 5 0 R");
} else {
nums_content.push_str(" null");
}
if i < num_total - 1 {
nums_content.push(' ');
}
}
nums_content.push_str(" ]\n ]\n");
// Create /K array for StructElem with MCIDs
let k_array = (0..num_total).map(|i| i.to_string()).collect::<Vec<_>>().join(" ");
// Build the PDF content without xref first
let pdf_body = format!(
"%PDF-1.7\n
1 0 obj
<<
/Type /Catalog
/Pages 2 0 R
/MarkInfo <<
/Marked true
/Suspects {}
>>
/StructTreeRoot 3 0 R
>>
endobj
2 0 obj
<<
/Type /Pages
/Kids [4 0 R]
/Count 1
>>
endobj
3 0 obj
<<
/Type /StructTreeRoot
/K [5 0 R]
/ParentTree 6 0 R
>>
endobj
4 0 obj
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Contents 7 0 R
/StructParents 0
>>
endobj
5 0 obj
<<
/Type /StructElem
/S /P
/K [{}]
>>
endobj
6 0 obj
<<
{}
>>
endobj
7 0 obj
<<
/Length 44
>>
stream
BT
/F1 12 Tf
100 700 Td
(Test) Tj
ET
endstream
endobj
",
if suspects { "true" } else { "false" },
k_array,
nums_content
);
// Calculate xref offsets by searching for object markers
let body_bytes = pdf_body.as_bytes();
let mut offsets = vec![0u64; 8]; // 0-7 objects
for i in 1..=7 {
let marker = format!("{} 0 obj", i);
if let Some(pos) = pdf_body.find(&marker) {
offsets[i] = pos as u64;
}
}
let xref_offset = pdf_body.len() as u64;
let xref_table = format!(
"xref\n0 8\n0000000000 65535 f \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \ntrailer\n<<\n/Size 8\n/Root 1 0 R\n>>\nstartxref\n{}\n%%EOF\n",
offsets[1], offsets[2], offsets[3], offsets[4], offsets[5], offsets[6], offsets[7], xref_offset
);
let mut file = File::create(path)?;
file.write_all(pdf_body.as_bytes())?;
file.write_all(xref_table.as_bytes())?;
Ok(())
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("Generating tagged PDF fixtures for Phase 7.1.4 coverage check...");
// Fixture 1: Suspects true, 60% coverage -> fallback to XY-cut
write_pdf("tagged-suspects-true.pdf", true, 6, 10)?;
println!("Created: tagged-suspects-true.pdf");
println!(" - /MarkInfo /Suspects: true");
println!(" - Coverage: 60% (6/10 MCIDs claimed)");
println!(" - Expected: fallback to XY-cut, reading_order_algorithm = 'xy_cut'");
// Fixture 2: Suspects false, 50% coverage -> trust StructTree
write_pdf("tagged-suspects-false.pdf", false, 5, 10)?;
println!("Created: tagged-suspects-false.pdf");
println!(" - /MarkInfo /Suspects: false");
println!(" - Coverage: 50% (5/10 MCIDs claimed)");
println!(" - Expected: trust StructTree, reading_order_algorithm = 'struct_tree'");
// Fixture 3: Suspects true, 95% coverage -> trust StructTree
write_pdf("tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
println!("Created: tagged-suspects-true-high-coverage.pdf");
println!(" - /MarkInfo /Suspects: true");
println!(" - Coverage: 95% (19/20 MCIDs claimed)");
println!(" - Expected: trust StructTree, reading_order_algorithm = 'struct_tree'");
println!("\nAll fixtures generated successfully!");
Ok(())
}

BIN
tests/fixtures/gen_suspects_v7 vendored
View file

Binary file not shown.

171
tests/fixtures/gen_suspects_v7.rs vendored
View file

@ -1,171 +0,0 @@
//! Generate tagged PDF fixtures for testing Phase 7.1.4 coverage check
//!
//! This creates three fixtures:
//! 1. tagged-suspects-true.pdf - Suspects true, 60% coverage -> fallback to XY-cut
//! 2. tagged-suspects-false.pdf - Suspects false, 50% coverage -> trust StructTree
//! 3. tagged-suspects-true-high-coverage.pdf - Suspects true, 95% coverage -> trust StructTree
use std::fs::File;
use std::io::Write;
fn write_pdf(path: &str, suspects: bool, num_claimed: usize, num_total: usize) -> Result<(), Box<dyn std::error::Error>> {
// Create ParentTree /Nums array with claimed and null entries
// Format: /Nums [0 [ref ref null ref ...]]
let mut nums_content = String::from(" /Nums [\n 0 [");
for i in 0..num_total {
if i < num_claimed {
nums_content.push_str(" 5 0 R");
} else {
nums_content.push_str(" null");
}
if i < num_total - 1 {
nums_content.push(' ');
}
}
nums_content.push_str(" ]\n ]\n");
// Create content stream with BDC/EMC marked content sequences for each MCID
// Each MCID gets a marked content sequence
let mut content_ops = String::new();
for i in 0..num_total {
content_ops.push_str(&format!(
"BT\n/F1 12 Tf\n100 {} Td\n/MCID {} BDC\n(Test{}) Tj\nEMC\nET\n",
700 - i * 15, // Move up for each MCID
i,
i
));
}
let content_length = content_ops.len();
// Build the PDF content
let pdf_body = format!(
"%PDF-1.7\n
1 0 obj
<<
/Type /Catalog
/Pages 2 0 R
/MarkInfo <<
/Marked true
/Suspects {}
>>
/StructTreeRoot 3 0 R
>>
endobj
2 0 obj
<<
/Type /Pages
/Kids [4 0 R]
/Count 1
>>
endobj
3 0 obj
<<
/Type /StructTreeRoot
/K [5 0 R]
/ParentTree 6 0 R
>>
endobj
4 0 obj
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Contents 7 0 R
/StructParents 0
/Resources <<
/Font <<
/F1 <<
/Type /Font
/Subtype /Type1
/BaseFont /Helvetica
>>
>>
>>
>>
endobj
5 0 obj
<<
/Type /StructElem
/S /P
/K [{}]
>>
endobj
6 0 obj
<<
{}
>>
endobj
7 0 obj
<<
/Length {}
>>
stream
{}
endstream
endobj
",
if suspects { "true" } else { "false" },
(0..num_total).map(|i| i.to_string()).collect::<Vec<_>>().join(" "),
nums_content,
content_length,
content_ops
);
// Calculate xref offsets by searching for object markers
// The offsets are from the beginning of the file (after %PDF-1.7\n)
let mut offsets = vec![0u64; 8]; // 0-7 objects
let mut current_offset = 10u64; // Start after "%PDF-1.7\n" (10 bytes)
for i in 1..=7 {
offsets[i] = current_offset;
// Find the end of this object by searching for "endobj"
let obj_marker = format!("{} 0 obj", i);
let obj_start = pdf_body[current_offset as usize..].find(&obj_marker)
.expect(&format!("Object {} not found", i));
let obj_end = pdf_body[current_offset as usize + obj_start..].find("endobj")
.expect(&format!("endobj for object {} not found", i));
current_offset += (obj_start + obj_end + 6) as u64; // +6 for "endobj"
}
let xref_offset = current_offset;
let xref_table = format!(
"xref\n0 8\n0000000000 65535 f \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \ntrailer\n<<\n/Size 8\n/Root 1 0 R\n>>\nstartxref\n{}\n%%EOF\n",
offsets[1], offsets[2], offsets[3], offsets[4], offsets[5], offsets[6], offsets[7], xref_offset
);
let mut file = File::create(path)?;
file.write_all(pdf_body.as_bytes())?;
file.write_all(xref_table.as_bytes())?;
Ok(())
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("Generating tagged PDF fixtures for Phase 7.1.4 coverage check...");
// Fixture 1: Suspects true, 60% coverage -> fallback to XY-cut
write_pdf("tagged-suspects-true.pdf", true, 6, 10)?;
println!("Created: tagged-suspects-true.pdf");
println!(" - /MarkInfo /Suspects: true");
println!(" - Coverage: 60% (6/10 MCIDs claimed)");
println!(" - Expected: fallback to XY-cut, reading_order_algorithm = 'xy_cut'");
// Fixture 2: Suspects false, 50% coverage -> trust StructTree
write_pdf("tagged-suspects-false.pdf", false, 5, 10)?;
println!("Created: tagged-suspects-false.pdf");
println!(" - /MarkInfo /Suspects: false");
println!(" - Coverage: 50% (5/10 MCIDs claimed)");
println!(" - Expected: trust StructTree, reading_order_algorithm = 'struct_tree'");
// Fixture 3: Suspects true, 95% coverage -> trust StructTree
write_pdf("tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
println!("Created: tagged-suspects-true-high-coverage.pdf");
println!(" - /MarkInfo /Suspects: true");
println!(" - Coverage: 95% (19/20 MCIDs claimed)");
println!(" - Expected: trust StructTree, reading_order_algorithm = 'struct_tree'");
println!("\nAll fixtures generated successfully!");
Ok(())
}

BIN
tests/fixtures/gen_suspects_v8 vendored
View file

Binary file not shown.

127
tests/fixtures/gen_suspects_v8.rs vendored
View file

@ -1,127 +0,0 @@
//! Generate tagged PDF fixtures for testing Phase 7.1.4 coverage check
//!
//! This creates three fixtures:
//! 1. tagged-suspects-true.pdf - Suspects true, 60% coverage -> fallback to XY-cut
//! 2. tagged-suspects-false.pdf - Suspects false, 50% coverage -> trust StructTree
//! 3. tagged-suspects-true-high-coverage.pdf - Suspects true, 95% coverage -> trust StructTree
use std::fs::File;
use std::io::Write;
fn write_pdf(path: &str, suspects: bool, num_claimed: usize, num_total: usize) -> Result<(), Box<dyn std::error::Error>> {
// Create ParentTree /Nums array with claimed and null entries
// Format: /Nums [0 [ref ref null ref ...]]
let mut nums_content = String::from(" /Nums [\n 0 [");
for i in 0..num_total {
if i < num_claimed {
nums_content.push_str(" 5 0 R");
} else {
nums_content.push_str(" null");
}
if i < num_total - 1 {
nums_content.push(' ');
}
}
nums_content.push_str(" ]\n ]\n");
// Create content stream with BDC/EMC marked content sequences for each MCID
// Each MCID gets a marked content sequence
let mut content_ops = String::new();
for i in 0..num_total {
content_ops.push_str(&format!(
"BT\n/F1 12 Tf\n100 {} Td\n/MCID {} BDC\n(Test{}) Tj\nEMC\nET\n",
700 - i * 15, // Move up for each MCID
i,
i
));
}
let content_length = content_ops.len();
// Build the PDF content objects
let objects = vec![
// Object 1: Catalog
format!(
"1 0 obj\n<<\n/Type /Catalog\n/Pages 2 0 R\n/MarkInfo <<\n /Marked true\n /Suspects {}\n>>\n/StructTreeRoot 3 0 R\n>>\nendobj\n",
if suspects { "true" } else { "false" }
),
// Object 2: Pages
"2 0 obj\n<<\n/Type /Pages\n/Kids [4 0 R]\n/Count 1\n>>\nendobj\n".to_string(),
// Object 3: StructTreeRoot
"3 0 obj\n<<\n/Type /StructTreeRoot\n/K [5 0 R]\n/ParentTree 6 0 R\n>>\nendobj\n".to_string(),
// Object 4: Page
format!(
"4 0 obj\n<<\n/Type /Page\n/Parent 2 0 R\n/MediaBox [0 0 612 792]\n/Contents 7 0 R\n/StructParents 0\n/Resources <<\n/Font <<\n/F1 <<\n/Type /Font\n/Subtype /Type1\n/BaseFont /Helvetica\n>>\n>>\n>>\n>>\nendobj\n"
),
// Object 5: StructElem
format!(
"5 0 obj\n<<\n/Type /StructElem\n/S /P\n/K [{}]\n>>\nendobj\n",
(0..num_total).map(|i| i.to_string()).collect::<Vec<_>>().join(" ")
),
// Object 6: ParentTree
format!(
"6 0 obj\n<<\n{}>>\nendobj\n",
nums_content
),
// Object 7: Content stream
format!(
"7 0 obj\n<<\n/Length {}\n>>\nstream\n{}\nendstream\nendobj\n",
content_length,
content_ops
),
];
// Calculate xref offsets
let mut offsets = vec![0u64; 8]; // 0-7 objects
offsets[0] = 0; // Object 0 is always free
let mut current_offset = 10u64; // Start after "%PDF-1.7\n" (10 bytes)
for (i, obj) in objects.iter().enumerate() {
offsets[i + 1] = current_offset;
current_offset += obj.len() as u64;
}
let xref_offset = current_offset;
let xref_table = format!(
"xref\n0 8\n0000000000 65535 f \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \n{:010} 00000 n \ntrailer\n<<\n/Size 8\n/Root 1 0 R\n>>\nstartxref\n{}\n%%EOF\n",
offsets[1], offsets[2], offsets[3], offsets[4], offsets[5], offsets[6], offsets[7], xref_offset
);
let mut file = File::create(path)?;
file.write_all(b"%PDF-1.7\n")?;
for obj in &objects {
file.write_all(obj.as_bytes())?;
}
file.write_all(xref_table.as_bytes())?;
Ok(())
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
println!("Generating tagged PDF fixtures for Phase 7.1.4 coverage check...");
// Fixture 1: Suspects true, 60% coverage -> fallback to XY-cut
write_pdf("tagged-suspects-true.pdf", true, 6, 10)?;
println!("Created: tagged-suspects-true.pdf");
println!(" - /MarkInfo /Suspects: true");
println!(" - Coverage: 60% (6/10 MCIDs claimed)");
println!(" - Expected: fallback to XY-cut, reading_order_algorithm = 'xy_cut'");
// Fixture 2: Suspects false, 50% coverage -> trust StructTree
write_pdf("tagged-suspects-false.pdf", false, 5, 10)?;
println!("Created: tagged-suspects-false.pdf");
println!(" - /MarkInfo /Suspects: false");
println!(" - Coverage: 50% (5/10 MCIDs claimed)");
println!(" - Expected: trust StructTree, reading_order_algorithm = 'struct_tree'");
// Fixture 3: Suspects true, 95% coverage -> trust StructTree
write_pdf("tagged-suspects-true-high-coverage.pdf", true, 19, 20)?;
println!("Created: tagged-suspects-true-high-coverage.pdf");
println!(" - /MarkInfo /Suspects: true");
println!(" - Coverage: 95% (19/20 MCIDs claimed)");
println!(" - Expected: trust StructTree, reading_order_algorithm = 'struct_tree'");
println!("\nAll fixtures generated successfully!");
Ok(())
}

116
tests/fixtures/gen_unmapped_comprehensive.rs vendored
View file

@ -1,116 +0,0 @@
//! Generate unmapped glyph test fixture for comprehensive Unicode recovery testing.
//!
//! This program creates a PDF with 6 unmapped glyphs and 3 mapped glyphs to exercise
//! the 4-level Unicode fallback chain failure path and verify GLYPH_UNMAPPED diagnostic emission.
//!
//! Based on design: notes/bf-68f9i-design.md
//! Glyph selection: notes/bf-68f9i-glyphs.md
//!
//! Usage: cargo run --bin gen_unmapped_comprehensive
use std::fs::File;
use std::io::Write;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let out_dir = "tests/fixtures/encoding";
std::fs::create_dir_all(out_dir)?;
println!("Generating unmapped-comprehensive fixture...");
generate_unmapped_comprehensive_pdf(out_dir)?;
println!("\nFixture generated successfully!");
println!("PDF: {}/unmapped-comprehensive.pdf", out_dir);
println!("Ground truth: {}/unmapped-comprehensive.txt", out_dir);
Ok(())
}
/// Build a complete PDF with proper xref table and trailer.
fn build_pdf(objects: Vec<Vec<u8>>) -> Vec<u8> {
let mut pdf = b"%PDF-1.4\n".to_vec();
// Track object offsets
let mut offsets = Vec::new();
for obj in &objects {
offsets.push(pdf.len());
pdf.extend(obj);
pdf.extend(b"\n");
}
let xref_start = pdf.len();
// Build xref table
pdf.extend(b"xref\n");
pdf.extend(format!("0 {}\n", objects.len() + 1).as_bytes());
pdf.extend(b"0000000000 65535 f \n"); // Free entry for object 0
for offset in offsets {
pdf.extend(format!("{:010} 00000 n \n", offset).as_bytes());
}
// Build trailer
pdf.extend(b"trailer\n<<\n/Size ");
pdf.extend(format!("{}", objects.len() + 1).as_bytes());
pdf.extend(b"\n/Root 1 0 R\n>>\n");
// Build startxref
pdf.extend(b"startxref\n");
pdf.extend(format!("{}\n", xref_start).as_bytes());
pdf.extend(b"%%EOF\n");
pdf
}
fn generate_unmapped_comprehensive_pdf(out_dir: &str) -> Result<(), Box<dyn std::error::Error>> {
println!(" Generating unmapped-comprehensive.pdf...");
// Content stream with 3 lines:
// Line 1: Three PUA glyphs (codes 0, 1, 2) -> <000102>
// Line 2: Custom and orphaned glyphs (codes 3, 4, 5, 6) -> <03040506>
// Line 3: Mapped AGL glyphs (codes 7, 8, 9) -> <070809>
let content = b"BT\n/F1 12 Tf\n50 700 Td\n<000102> Tj\n50 680 Td\n<03040506> Tj\n50 660 Td\n<070809> Tj\nET\n";
let objects = vec![
// Object 1: Catalog
b"1 0 obj\n<<\n/Type /Catalog\n/Pages 2 0 R\n>>\nendobj".to_vec(),
// Object 2: Pages node
b"2 0 obj\n<<\n/Type /Pages\n/Kids [3 0 R]\n/Count 1\n>>\nendobj".to_vec(),
// Object 3: Page dictionary
b"3 0 obj\n<<\n/Type /Page\n/Parent 2 0 R\n/MediaBox [0 0 612 792]\n/Resources <<\n/Font <<\n/F1 5 0 R\n>>\n>>\n/Contents 4 0 R\n>>\nendobj".to_vec(),
// Object 4: Content stream
format!("4 0 obj\n<<\n/Length {}\n>>\nstream\n{}\nendstream\nendobj",
content.len(),
std::str::from_utf8(content).unwrap()).into_bytes(),
// Object 5: Font dictionary with custom encoding
// Character codes 0-9 map to glyph names:
// 0->/g001, 1->/g002, 2->/g003 (PUA - unmapped)
// 3->/CustomA, 4->/CustomB (custom - unmapped)
// 5->/NotAGlyph (orphaned - unmapped)
// 6->/glyph_0041 (non-AGL algorithmic - unmapped)
// 7->/A, 8->/B, 9->/space (AGL - mapped)
b"5 0 obj\n<<\n/Type /Font\n/Subtype /Type1\n/BaseFont /UnmappedTestFont\n/Encoding <<\n/Type /Encoding\n/Differences [0 /g001 /g002 /g003 /CustomA /CustomB /NotAGlyph /glyph_0041 /A /B /space]\n>>\n>>\nendobj".to_vec(),
];
let pdf = build_pdf(objects);
let pdf_path = format!("{}/unmapped-comprehensive.pdf", out_dir);
let mut file = File::create(&pdf_path)?;
file.write_all(&pdf)?;
// Ground truth: 7 × U+FFFD + "AB "
// Line 1: <20><><EFBFBD> (3 U+FFFD for /g001, /g002, /g003)
// Line 2: <20><><EFBFBD><EFBFBD> (4 U+FFFD for /CustomA, /CustomB, /NotAGlyph, /glyph_0041)
// Line 3: AB (U+0041, U+0042, U+0020 for /A, /B, /space)
let txt_path = format!("{}/unmapped-comprehensive.txt", out_dir);
let mut txt_file = File::create(&txt_path)?;
// Write the expected output: 3 lines
txt_file.write_all(b"\xEF\xBF\xBC\xEF\xBF\xBC\xEF\xBF\xBC\n")?; // Line 1: <20><><EFBFBD>
txt_file.write_all(b"\xEF\xBF\xBC\xEF\xBF\xBC\xEF\xBF\xBC\xEF\xBF\xBC\n")?; // Line 2: <20><><EFBFBD><EFBFBD>
txt_file.write_all(b"AB \n")?; // Line 3: AB + space
println!(" -> {}", pdf_path);
println!(" -> {}", txt_path);
Ok(())
}

297
tests/fixtures/gen_unmapped_configurable.rs vendored
View file

@ -1,297 +0,0 @@
//! Generate unmapped glyph test fixture with configurable glyph names.
//!
//! Run with: cargo run --bin gen_unmapped_configurable
//!
//! This generator reads unmapped glyph names from tests/fixtures/unmapped_config.txt
//! and creates a PDF with:
//! - Configured unmapped glyphs (NO ToUnicode entries)
//! - Standard AGL-mapped glyphs (WITH ToUnicode entries)
//! - All glyphs included in the encoding dictionary
use anyhow::{Context, Result};
use lopdf::dictionary;
use lopdf::{Dictionary, Object, Document};
use std::fs::File;
use std::io::{self, BufRead, Write};
use std::path::Path;
/// Read unmapped glyph names from configuration file
fn read_unmapped_glyph_config(config_path: &Path) -> Result<Vec<String>> {
let file = File::open(config_path)
.with_context(|| format!("Failed to open config file: {}", config_path.display()))?;
let mut glyph_names = Vec::new();
for line in io::BufReader::new(file).lines() {
let line = line?;
let trimmed = line.trim();
// Skip empty lines and comments
if trimmed.is_empty() || trimmed.starts_with('#') {
continue;
}
glyph_names.push(trimmed.to_string());
}
Ok(glyph_names)
}
fn create_simple_page_with_font(
content: &[u8],
font_dict: Dictionary,
doc: &mut Document,
) -> lopdf::ObjectId {
let mut page_dict = Dictionary::new();
page_dict.set("Type", "Page");
page_dict.set("MediaBox", Object::Array(vec![
Object::Real(0.0), Object::Real(0.0),
Object::Real(612.0), Object::Real(792.0)
]));
page_dict.set("Resources", dictionary! {
"Font" => dictionary! {
"F1" => font_dict
}
});
let content_stream_id = doc.new_object_id();
doc.objects.insert(content_stream_id, Object::Stream(lopdf::Stream::new(
dictionary! {},
content.to_vec()
)));
page_dict.set("Contents", Object::Reference(content_stream_id));
doc.add_object(page_dict)
}
/// Create a ToUnicode CMap that excludes configured unmapped glyphs.
///
/// This is the core logic for the unmapped glyph encoding:
/// - We create ToUnicode entries ONLY for glyphs that are NOT in the unmapped config
/// - Configured unmapped glyphs are intentionally excluded from the ToUnicode CMap
/// - All glyphs (mapped and unmapped) are still present in the font's encoding dictionary
fn create_selective_tounicode_cmap(
glyph_mapping: &[(u8, &str, Option<char>)], // (char_code, glyph_name, unicode_value)
) -> Result<Option<Vec<u8>>> {
let mut cmap_lines = Vec::new();
cmap_lines.push(b"/CIDInit /ProcSet findresource begin".to_vec());
cmap_lines.push(b"12 dict begin".to_vec());
cmap_lines.push(b"begincmap".to_vec());
cmap_lines.push(b"/CMapType 2 def".to_vec());
cmap_lines.push(b"/CMapName /SelectiveToUnicode def".to_vec());
cmap_lines.push(b"1 begincodespacerange".to_vec());
cmap_lines.push(b"<00> <FF>".to_vec());
cmap_lines.push(b"endcodespacerange".to_vec());
// Count mappings that have unicode values (i.e., are NOT unmapped)
let mapped_glyphs: Vec<_> = glyph_mapping
.iter()
.filter_map(|&(code, name, unicode)| {
unicode.map(|u| (code, name, u))
})
.collect();
if mapped_glyphs.is_empty() {
// No mapped glyphs - return no ToUnicode CMap at all
return Ok(None);
}
cmap_lines.push(format!("{} beginbfchar", mapped_glyphs.len()).into_bytes());
for (char_code, glyph_name, unicode_char) in mapped_glyphs {
let unicode_hex = format!("{:04X}", unicode_char as u32);
let char_code_hex = format!("{:02X}", char_code);
cmap_lines.push(format!("<{}> <{}>", char_code_hex, unicode_hex).into_bytes());
println!(" ToUnicode: code 0x{} ({}) -> U+{} ({})",
char_code_hex, glyph_name, unicode_hex, unicode_char);
}
cmap_lines.push(b"endbfchar".to_vec());
cmap_lines.push(b"endcmap".to_vec());
cmap_lines.push(b"CMapName currentdict /CMap defineresource pop".to_vec());
cmap_lines.push(b"end".to_vec());
cmap_lines.push(b"end".to_vec());
let cmap_data = cmap_lines.join(b"\n");
println!("Created ToUnicode CMap with {} mappings (excluded {} unmapped glyphs)",
mapped_glyphs.len(),
glyph_mapping.len() - mapped_glyphs.len());
Ok(Some(cmap_data))
}
/// Create the configurable unmapped glyph fixture.
///
/// This demonstrates the key encoding logic:
/// 1. Read configured unmapped glyph names from config file
/// 2. Build encoding dictionary with ALL glyphs (mapped + unmapped)
/// 3. Create ToUnicode CMap ONLY for mapped glyphs (skip unmapped)
/// 4. Unmapped glyphs appear in PDF but have no Unicode mapping
fn create_configurable_unmapped_pdf(
unmapped_glyphs: &[String],
) -> Result<()> {
let mut doc = Document::with_version("1.4");
// Define our glyph mapping: (char_code, glyph_name, unicode_value_if_mapped)
let mut glyph_mapping: Vec<(u8, &str, Option<char>)> = vec![
// Unmapped glyphs (unicode = None)
(0x00, "g001", None),
(0x01, "g002", None),
(0x02, "g003", None),
(0x03, "CustomA", None),
(0x04, "CustomB", None),
(0x05, "NotAGlyph", None),
(0x06, "glyph_0041", None),
// Mapped glyphs (unicode = Some(char))
(0x41, "A", Some('A')),
(0x42, "B", Some('B')),
(0x20, "space", Some(' ')),
];
// Verify that unmapped_glyphs contains the expected unmapped glyph names
println!("Configured unmapped glyphs from config file:");
for glyph in unmapped_glyphs {
println!(" - {}", glyph);
}
// Build encoding differences array
let mut encoding_diffs = vec![Object::Integer(0)]; // Start at code 0
// Add all glyph names to the encoding
for (code, name, _) in &glyph_mapping {
// Add entries in order, ensuring all codes are represented
encoding_diffs.push(Object::Name(name.as_bytes().to_vec()));
}
// Build font dictionary
let mut font_dict = dictionary! {
"Type" => "Font",
"Subtype" => "Type1",
"BaseFont" => "ConfigurableUnmappedFont",
"Encoding" => dictionary! {
"Type" => "Encoding",
"Differences" => Object::Array(encoding_diffs)
}
};
// Create selective ToUnicode CMap (skips configured unmapped glyphs)
let cmap_data = create_selective_tounicode_cmap(&glyph_mapping)?;
// Add ToUnicode to font dictionary only if we have mapped glyphs
if let Some(cmap_bytes) = cmap_data {
let cmap_stream_id = doc.new_object_id();
doc.objects.insert(
cmap_stream_id,
Object::Stream(lopdf::Stream::new(
dictionary! {
"Type" => "/CMap",
"CMapName" => "/SelectiveToUnicode"
},
cmap_bytes
))
);
// CRITICAL: We selectively add ToUnicode ONLY for mapped glyphs
// Unmapped glyphs are intentionally excluded from the CMap
font_dict.set("ToUnicode", Object::Reference(cmap_stream_id));
println!("Added ToUnicode CMap to font (mapped glyphs only)");
} else {
println!("No ToUnicode CMap created (all glyphs are unmapped)");
}
// Content stream showing both unmapped and mapped glyphs
// Line 1: Three unmapped PUA glyphs
// Line 2: Custom and orphaned unmapped glyphs
// Line 3: Mapped AGL glyphs (A, B, space)
let content = b"BT
/F1 12 Tf
50 700 Td
<000102> Tj
50 680 Td
<03040506> Tj
50 660 Td
<414220> Tj
ET";
let page_id = create_simple_page_with_font(content, font_dict, &mut doc);
// Create pages dict
let mut pages_dict = Dictionary::new();
pages_dict.set("Type", "Pages");
pages_dict.set("Count", Object::Integer(1));
pages_dict.set("Kids", Object::Array(vec![Object::Reference(page_id)]));
let pages_id = doc.add_object(pages_dict);
// Update page parent
if let Some(Object::Dictionary(ref mut page_dict)) = doc.objects.get_mut(&page_id) {
page_dict.set("Parent", Object::Reference(pages_id));
}
// Create catalog
let mut catalog_dict = Dictionary::new();
catalog_dict.set("Type", "Catalog");
catalog_dict.set("Pages", Object::Reference(pages_id));
let catalog_id = doc.add_object(catalog_dict);
doc.trailer.set("Root", Object::Reference(catalog_id));
// Save PDF
let pdf_path = "tests/fixtures/encoding/unmapped-configurable.pdf";
doc.save(pdf_path)
.with_context(|| format!("Failed to create PDF: {}", pdf_path))?;
println!("Created: {}", pdf_path);
// Create ground truth .txt file
let txt_path = "tests/fixtures/encoding/unmapped-configurable.txt";
let mut txt_file = File::create(txt_path)
.with_context(|| format!("Failed to create ground truth: {}", txt_path))?;
// Line 1: 3 U+FFFD for g001, g002, g003
writeln!(txt_file, "{}", "\u{FFFD}\u{FFFD}\u{FFFD}")?;
// Line 2: 4 U+FFFD for CustomA, CustomB, NotAGlyph, glyph_0041
writeln!(txt_file, "{}", "\u{FFFD}\u{FFFD}\u{FFFD}\u{FFFD}")?;
// Line 3: "AB " for A, B, space
writeln!(txt_file, "AB ")?;
println!("Created: {} (7 × U+FFFD + \"AB \")", txt_path);
Ok(())
}
fn main() -> Result<()> {
println!("Generating configurable unmapped glyph test fixture...");
println!("{}", "=".repeat(70));
// Ensure output directory exists
std::fs::create_dir_all("tests/fixtures/encoding")
.context("Failed to create fixtures directory")?;
// Read unmapped glyph configuration
let config_path = Path::new("tests/fixtures/unmapped_config.txt");
let unmapped_glyphs = read_unmapped_glyph_config(config_path)
.with_context(|| format!("Failed to read unmapped glyph config: {}", config_path.display()))?;
println!("\nRead {} unmapped glyph names from config", unmapped_glyphs.len());
println!("\n[1/1] Creating unmapped-configurable.pdf...");
println!("- All glyphs included in encoding dictionary");
println!("- ToUnicode CMap created ONLY for mapped glyphs");
println!("- Configured unmapped glyphs excluded from ToUnicode");
create_configurable_unmapped_pdf(&unmapped_glyphs)?;
println!("\n{}", "=".repeat(70));
println!("Configurable unmapped glyph fixture generated successfully!");
println!("\nFixtures created:");
println!(" tests/fixtures/encoding/unmapped-configurable.pdf");
println!(" tests/fixtures/encoding/unmapped-configurable.txt");
Ok(())
}

BIN
tests/fixtures/generate_suspects_fixture vendored
View file

Binary file not shown.

BIN
tests/fixtures/generate_suspects_fixtures vendored
View file

Binary file not shown.

185
tests/fixtures/generate_suspects_fixtures.py vendored
View file

@ -1,185 +0,0 @@
#!/usr/bin/env python3
"""Generate tagged PDF fixtures for testing Phase 7.1.4 coverage check.
Creates three fixtures:
1. tagged-suspects-true.pdf - Suspects true, 60% coverage -> fallback to XY-cut
2. tagged-suspects-false.pdf - Suspects false, 50% coverage -> trust StructTree
3. tagged-suspects-true-high-coverage.pdf - Suspects true, 95% coverage -> trust StructTree
"""
import struct
def write_pdf(path, suspects, num_claimed, num_total):
"""Write a tagged PDF with the given parameters."""
# Create ParentTree /Nums array with claimed and null entries
nums_content = f" /Nums [\n 0 ["
for i in range(num_total):
if i < num_claimed:
nums_content += " 5 0 R"
else:
nums_content += " null"
if i < num_total - 1:
nums_content += ' '
nums_content += " ]\n ]\n"
# Create /K array for StructElem with MCIDs
k_array = ' '.join(str(i) for i in range(num_total))
# Create content stream with BDC/EMC marked content sequences for each MCID
content_ops = []
for i in range(num_total):
y_pos = 700 - i * 15
content_ops.extend([
"BT",
"/F1 12 Tf",
f"100 {y_pos} Td",
f"/MCID {i} BDC",
f"(Test{i}) Tj",
"EMC",
"ET",
])
content_stream = '\n'.join(content_ops)
content_length = len(content_stream)
# Build PDF content
pdf_lines = [
"%PDF-1.7",
"",
"1 0 obj",
"<<",
"/Type /Catalog",
"/Pages 2 0 R",
"/MarkInfo <<",
" /Marked true",
f" /Suspects {'true' if suspects else 'false'}",
">>",
"/StructTreeRoot 3 0 R",
">>",
"endobj",
"",
"2 0 obj",
"<<",
"/Type /Pages",
"/Kids [4 0 R]",
"/Count 1",
">>",
"endobj",
"",
"3 0 obj",
"<<",
"/Type /StructTreeRoot",
"/K [5 0 R]",
"/ParentTree 6 0 R",
">>",
"endobj",
"",
"4 0 obj",
"<<",
"/Type /Page",
"/Parent 2 0 R",
"/MediaBox [0 0 612 792]",
"/Contents 7 0 R",
"/StructParents 0",
">>",
"endobj",
"",
"5 0 obj",
"<<",
"/Type /StructElem",
"/S /P",
f"/K [{k_array}]",
">>",
"endobj",
"",
"6 0 obj",
"<<",
nums_content,
">>",
"endobj",
"",
"7 0 obj",
"<<",
f"/Length {content_length}",
">>",
"stream",
content_stream,
"endstream",
"endobj",
]
# Join content with newlines and calculate offsets
pdf_content = '\n'.join(pdf_lines)
pdf_bytes = pdf_content.encode('latin-1')
# Calculate object offsets
obj_offsets = [0] * 8 # Objects 0-7 (0 is always null)
current_pos = 0
for line in pdf_lines:
# Check if this line starts an object definition
if line.endswith(" 0 obj"):
obj_num = int(line.split()[0])
obj_offsets[obj_num] = current_pos
current_pos += len(line) + 1 # +1 for newline
# Build xref table
xref_lines = [
"xref",
"0 8",
f"0000000000 65535 f ",
]
for i in range(1, 8):
xref_lines.append(f"{obj_offsets[i]:010d} 00000 n ")
xref_table = '\n'.join(xref_lines)
# Calculate startxref (offset to xref table)
startxref = len(pdf_bytes) + 1 # +1 for the newline before xref
# Build trailer
trailer = f"""trailer
<<
/Size 8
/Root 1 0 R
>>
startxref
{startxref}
%%EOF"""
# Write complete PDF
with open(path, 'wb') as f:
f.write(pdf_bytes)
f.write(b'\n')
f.write(xref_table.encode('latin-1'))
f.write(b'\n')
f.write(trailer.encode('latin-1'))
coverage = (num_claimed / num_total) * 100
print(f"Created: {path}")
print(f" - /MarkInfo /Suspects: {suspects}")
print(f" - Coverage: {coverage:.0f}% ({num_claimed}/{num_total} MCIDs claimed)")
if suspects and coverage < 80:
print(f" - Expected: fallback to XY-cut, reading_order_algorithm = 'xy_cut'")
elif not suspects or coverage >= 80:
print(f" - Expected: trust StructTree, reading_order_algorithm = 'struct_tree'")
def main():
print("Generating tagged PDF fixtures for Phase 7.1.4 coverage check...")
print()
# Fixture 1: Suspects true, 60% coverage -> fallback to XY-cut
write_pdf("tests/fixtures/tagged-suspects-true.pdf", True, 6, 10)
print()
# Fixture 2: Suspects false, 50% coverage -> trust StructTree
write_pdf("tests/fixtures/tagged-suspects-false.pdf", False, 5, 10)
print()
# Fixture 3: Suspects true, 95% coverage -> trust StructTree
write_pdf("tests/fixtures/tagged-suspects-true-high-coverage.pdf", True, 19, 20)
print()
print("All fixtures generated successfully!")
if __name__ == "__main__":
main()