Paperful architecture¶

Paperful is a local CLI. Fetch, identifier checks, and proposed metadata patches happen on disk (out/, state/). A library adapter reads the catalogue and, separately, writes PDFs or field patches back. Zotero (local API on localhost:23119) is the first adapter; manager = "mendeley" is reserved.

run never rewrites bibliographic fields. Attach and fix-metadata --apply use the Zotero 10+ write API.

Data flow¶

        flowchart LR
  adapter[LibraryBackend]
  resolve[prepare_identifiers]
  sources[Source lanes]
  disk[out/ and state/]
  writeback[attach / fix-metadata --apply]
  adapter -->|read| resolve --> sources --> disk --> writeback --> adapter

Scope — collection subtree or whole library; run skips items that already have an imported PDF (and by default skip items with only a linked_url PDF).
Prepare identifiers — verify library DOI; optional in-memory swap; PubMed PMID→DOI; title→DOI via Crossref / OpenAlex / Semantic Scholar. Skipped for web/blog/forum types.
Sources — ordered list (Unpaywall, OpenAlex, arXiv, …, CORE, EZProxy, HTML→PDF); per-item routing skips inapplicable sources unless --try-all.
Download — validate PDF size; write under out_dir; extract PDF DOI (pdftotext, then pypdf); append to state/manifest.jsonl.
Attach — optional imported_file upload via local write API; failures recorded as attach_failed with typed reasons.

paperful lint runs step 2 (and PDF-text DOI) for items with and without PDFs. paperful fix-metadata writes state/metadata-patches.jsonl then, with --apply, pushes patches through the adapter.

Disk artifacts¶

Path	Role
`out/<collection>/…pdf`	Collection-mirrored downloads
`state/manifest.jsonl`	Append-only resume ledger. Latest line per item key wins. Fields include `doi` (used this attempt), `library_doi`, `doi_verified`, `pdf_doi`
`state/metadata-patches.jsonl`	Proposed patches (`doi`, `title`, `date`, `publicationTitle`)
`state/pdf-cache/`	Manager PDFs exported so lint reads text on disk
`state/sessions/`	Chromium profile + `meta.json` (login timestamps, no secrets). Netscape dumps for httpx
`state/last-run.json`	Latest `run` report (`paperful.run_report.v1`)
`state/runs/<stamp>-<command>.json`	Historical `run` and `fix-metadata` reports

doi_verified is ok (≥ crossref_min_score), suspect (< doi_suspect_score), swapped (in-memory replacement), unknown (API down, mid-range match, or verify_doi = false), or missing. unknown never swaps.

Library adapter¶

paperful/library.py defines LibraryBackend: list items, export a PDF onto disk, apply a field patch, attach a file. Identifier logic (resolve, lint, pdfid, metadata) must not import Zotero except through this protocol.

Identifiers and lint¶

paperful/resolve.py prepare_identifiers is shared by run and lint. Swap is in memory only. PubMed uses NCBI ID Converter on PMID: / PubMed PMID: in Extra.

paperful/lint.py finding codes (manager-agnostic “library DOI”):

Code	When
`missing_doi`	Scholarly type, no DOI after prepare
`suspect_doi`	Library DOI fails title check, no swap candidate
`swappable_doi`	High-confidence replacement ≠ library DOI
`pmid_no_doi`	PMID present, converter failed
`pdf_doi_mismatch`	PDF-text DOI ≠ library DOI and ≠ prepared DOI
`no_identifier`	No DOI, arXiv id, PMID, or URL

--json prints only findings. Exit 0 unless --strict. Lint prefers a file already on disk (item.pdf_path or manifest path) and calls export_pdf only when has_pdf and nothing is on disk.

paperful/metadata.py whitelist: doi, title, date, publicationTitle. Default fills empty venue/date; --overwrite may replace title/date/venue. Never invents creators.

PDF text¶

paperful/pdfid.py: pdftotext (Poppler) if on PATH, else pypdf (first two pages + /Title). Manager fulltext is last-resort: export the file to state/pdf-cache/ first. paperful doctor reports amber if pdftotext is missing.

Circuit breaker¶

Open-access sources run in parallel (concurrency_oa). Block-like outcomes (CAPTCHA, 429, “sorry”, …) increment a per-source counter; after circuit_breaker_threshold the source is skipped for the rest of the run. Scholar, Sci-Hub, EZProxy, and HTML→PDF stay serial (Scholar/htmlpdf share one Chromium profile lock).

Sci-Hub and presets¶

Sci-Hub is never in the default source list; opt in via config, --scihub, or --sources. The eoi preset (--preset eoi) limits runs to open access plus campus EZProxy (no Scholar, no Sci-Hub). CORE is in the default list but skipped until core_api_key is set.

Disk mirror vs cloud quota¶

When Zotero cloud storage is full, attachments may fail with quota errors; PDFs still land on disk and can be attached later. Linked PDF URLs in Zotero are treated as “already covered” unless --upgrade-linked is set.

Operator tooling¶

paperful doctor — preflight. Colours: green = ready; amber = usable with a degraded path (empty email, missing session, no pdftotext, Zotero 7–9 write API, incomplete grey-lit pack); red on Zotero :23119 / out_dir / state_dir is fatal (doctor and any command that needs Zotero). Reports grey-lit packs (UNGA/undocs · BBNJ/DOALOS · ISA) when builtin is on. See commands.
paperful run --dry-run — no downloads. Per item: Would-hit is the routed source list in order (full sources when --try-all).
paperful lint / paperful fix-metadata — identifier hygiene; apply is explicit.
paperful report / paperful report --last-run — manifest totals plus the latest auditable run report (state/last-run.json, history under state/runs/). Each run prints a Run summary table (downloads, attached, deferred, errors). A one-line banner is a 1.0 tightening.

When Zotero is unreachable, collections, run, and attach exit 2 and print next steps (start Zotero, enable local API, paperful doctor).

Report JSON (`paperful.run_report.v1`)¶

paperful report --json is { counts, by_source, no_identifier, no_doi, attach_failed_by_code, last_run? }. last_run (when present) is the same object as state/last-run.json. 0.x may add keys; 1.0 freezes this schema name.

Field	Meaning
`schema`	Always `paperful.run_report.v1` on run reports
`command`	`run` (or `fix-metadata` on apply reports under `state/runs/`)
`started_at` / `finished_at`	ISO-8601 UTC
`duration_s`	Wall time, or `null` if start unknown
`scope`	Collection path(s) or library
`sources_configured`	Source names for that run
`flags`	CLI flags (`dry_run`, `scihub`, `preset`, …)
`paths.out_dir` / `manifest` / `state_dir`	Absolute paths
`summary.pdfs_downloaded`	Successful downloads (`ok` bumps)
`summary.attached` / `attach_failed`	Write-back counts
`summary.not_found` / `no_identifier` / `captcha` / `error`	Item outcomes
`summary.skipped_manifest` / `linked_url_skipped`	Not attempted this run
`summary.fields_corrected` / `fields_corrected_by_kind`	In-memory DOI enrichments (not library writes)
`summary.identifiers_verified`	`verify:ok` count
`summary.by_source`	Hits per source name
`summary.sources_checked`	Per-source outcome tallies
`summary.errors_by_type` / `attach_failed_by_code`	Typed errors
`items[]`	Per-item: `itemKey`, `title`, `status`, `source`, `reason`, `doi`, `doi_verified`, `attempts`, `fields_corrected`, `path`, `error_type`

Manifest counts keys match ledger statuses (ok, attached, not_found, …).

Grey literature¶

direct runs a playbook engine (paperful/playbooks.py): declarative rewrite / scrape / synthesize rules from config. A builtin ocean/governance example pack (paperful/data/grey_playbooks_ocean.toml) ships named grey-lit packs plus FAO/OECD/IEA/WHO examples — not core product logic; set grey_playbooks_builtin = false or override by name. Skip-host item URLs (YouTube, Scholar, …) still allow Extra/title synthesize. Domain-agnostic OA rewrites (PMC, arXiv, HAL) and DSpace/OAI stay in code. DOI-less report / document items can fall through to htmlpdf. Campus EZProxy is never used for these public hosts. Unpaywall/OpenAlex already skip DOI-less items (no quota burn on institutional reports).

Named packs (BBNJ product)¶

Pack id	Hosts / patterns	Rules
`undocs-unga-vme`	`undocs.org`, `documents.un.org`, `daccess-ods.un.org`	`rewrite` via `parser = "undocs"` → `https://undocs.org/pdf?symbol=…`
`undocs-unga-vme-symbol`	Extra/title symbols `A/RES/…`, `A/N/N`, `A/CONF.…`, `A/AC.…`, `ISBA/…`, `S/…`	`synthesize` → same undocs PDF URL
`bbnj-doalos-prepcom`	`un.org` (`/bbnjagreement/`, `/depts/los/`), `highseasalliance.org`, `iisd.org` (ENB)	`scrape` first same-origin `.pdf` / `sites/default/files` / Download; direct `.pdf` URLs need no rewrite
`isa-deepdata`	`isa.org.jm` (documents / news landings)	`scrape` same-origin PDF/download; OBIS/ODIS links are not treated as PDF sources

Smoke collections (dry-run): HKF7T7EI (UNGA/VME), 7R77ZJFH / XFD86ZFP (BBNJ/PrepCom), J2SEXDC5 (ISA).

Items with no DOI, arXiv id, PMID, URL, or matching synthesize playbook still stop at no_identifier.