Metadata-Version: 2.5
Name: abx-plugins
Version: 1.12.235
Summary: ArchiveBox-compatible plugin suite (hooks, configs, binaries manifests)
Project-URL: Homepage, https://github.com/ArchiveBox/abx-plugins
Project-URL: Source, https://github.com/ArchiveBox/abx-plugins
Project-URL: Documentation, https://github.com/ArchiveBox/ArchiveBox
Project-URL: Bug Tracker, https://github.com/ArchiveBox/abx-plugins/issues
Author: Nick Sweeting, ArchiveBox
License: MIT
License-File: LICENSE
Keywords: archivebox,hooks,plugins,scraping,web-archiving
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: <3.15,>=3.12
Requires-Dist: abxbus==2.5.56
Requires-Dist: abxpkg==1.12.117
Requires-Dist: imagesize>=2.0.0
Requires-Dist: rich-click>=1.9.7
Requires-Dist: uv>=0.8.22
Provides-Extra: opencode
Requires-Dist: httpx>=0.28.1; extra == 'opencode'
Requires-Dist: requests>=2.32.3; extra == 'opencode'
Requires-Dist: websockets>=15; extra == 'opencode'
Description-Content-Type: text/markdown

# [ArchiveBox Plugin Marketplace](https://archivebox.github.io/abx-plugins/)

> [!TIP]
> **[➡️ View The Live Gallery 🌠](https://archivebox.github.io/abx-plugins/)**
> [![](https://github.com/user-attachments/assets/e1c70778-ba8b-4812-8b5a-4d8ebc461eed)](https://archivebox.github.io/abx-plugins/)

ArchiveBox-compatible plugin suite (hooks and config schemas).

This package contains standalone plugin hook scripts and config schemas. A hook
can be run directly as a CLI; runners such as [`abx-dl`](https://github.com/archiveBox/abx-dl)
and [`archivebox`](https://github.com/archiveBox/ArchiveBox) add orchestration,
environment setup, and cache projection around the same scripts.

<img width="1000" height="1082" alt="Screenshot 2026-03-11 at 6 53 03 AM" src="https://github.com/user-attachments/assets/08c5f63b-05e2-4947-adca-f64e8c5ad8b3" />

## Usage

Tools like `abx-dl` and ArchiveBox can discover plugins from this package
without symlinks or environment-variable tricks.

## Plugin Contract

### Directory layout

Each plugin lives under `plugins/<name>/` and may include:

- `config.json` config schema
- optional generic catalog metadata in `config.json`: `category`, `display_order`, `hidden`, and `x-auto-run` (set false for hooks that require explicit host selection)
- `config.json > required_binaries` binary dependency declarations (optional)
- `on_CrawlSetup__...` crawl setup hook scripts (optional) - shared setup/process startup, emit no stdout JSONL records
- `on_Snapshot__...` per-snapshot hooks - emit `ArchiveResult` and may also emit `Snapshot` / `Tag`

Hooks run with:

- **SNAP_DIR** = base snapshot directory (default: `.`)
- **CRAWL_DIR** = base crawl directory (default: `.`)
- **Snapshot hook output** = `SNAP_DIR/<plugin>/...`
- **Crawl hook output** = `CRAWL_DIR/<plugin>/...`
- **Other plugin outputs** can be read via `../<other-plugin>/...` from your own output dir

### Key environment variables

`EXTRA_CONTEXT` is an opaque JSON object used only by shared output emitters to
reflect correlation fields unchanged into JSONL records. Hooks must never read,
branch on, or extract values from it, including IDs. Required inputs belong in
explicit CLI arguments (e.g. `--snapshot-id`, `--depth`, `--url`) or filesystem
data. Unbounded archived content such as titles and tags belongs in files, not
environment variables or shell commands. Sonic reads snapshot metadata from
`SNAP_DIR/index.jsonl`; archived title text is available in `title/title.txt`.

- `SNAP_DIR` - base snapshot directory (default: `.`)
- `CRAWL_DIR` - base crawl directory (default: `.`)
- `ABXPKG_LIB_DIR` - binaries/tools root (default: `~/.config/abx/lib`)
- `PERSONAS_DIR` - persona profiles root (default: `~/.config/abx/personas`)
- `ACTIVE_PERSONA` - persona name (default: `Default`)

### Binary dependency contract (concise)

Lifecycle:

1. `config.json > required_binaries` declares plugin dependencies.
2. Hook config helpers hydrate `*_BINARY` values from env, known local paths, and abxpkg provider state so hooks can run as standalone CLIs.
3. Runners may perform an install preflight from the same declarations. `abx-dl` and ArchiveBox use abxpkg services/cache backends to prepare env/DB state, but hooks must not depend on those services being active.

`config.json` declaration:

```json
[
  {
    "name": "{YTDLP_BINARY}",
    "binproviders": "env,uv,brew,apt",
    "min_version": null,
    "overrides": {
      "uv": {
        "install_args": ["yt-dlp[default]"]
      }
    }
  }
]
```

Runners may project resolved binary metadata internally as `BinaryEvent` records shaped like:

```json
{"type":"Binary","name":"yt-dlp","abspath":"/abs/path","version":"2025.01.01","sha256":"<optional>","binprovider":"pip","machine_id":"<recommended>","binary_id":"<recommended>"}
```

Notes:

- Install resolution is optional runtime preflight work driven directly from `config.json > required_binaries`.
- Binary provider plugins are no longer part of this package; binary provider behavior lives in `abxpkg`.
- `abxpkg` owns the provider cache under `ABXPKG_LIB_DIR`; ArchiveBox may additionally project resolved binary events into DB `machine_binary` rows. Plugins and `abx-dl` stay unaware of both persistence layers.

State/OS:

- working dir: `CRAWL_DIR/<plugin>/`
- durable install root: `ABXPKG_LIB_DIR` (e.g. npm prefix, pip venv, puppeteer cache)
- built-in providers include `apt` (Debian/Ubuntu), `brew` (macOS/Linux), and language/runtime-specific installers; many hooks currently assume POSIX paths

### Hook family contract

Lifecycle:

- optional binary preflight can run before crawl setup, but hook scripts also resolve declared binaries through shared config helpers when run directly
- `on_CrawlSetup__*` runs before snapshot extraction; background setup hooks use their first stdout line as readiness and emit no stdout JSONL records
- `on_Snapshot__*` runs once per snapshot; background hooks use their first stdout line as readiness and then may emit `ArchiveResult`, `Snapshot`, and `Tag` records only

State:

- output cwd is usually `SNAP_DIR/<plugin>/`
- hooks may read sibling outputs via `../<plugin>/...`

Output records:

- `on_Snapshot__*` should finish with an `ArchiveResult` record:

```json
{"type":"ArchiveResult","status":"succeeded|noresults|skipped|failed","output_str":"path-or-message"}
```

- `Snapshot` and `Tag` records may appear before the final `ArchiveResult`

Semantics:

- `stdout`: for background hooks, the first line is the readiness boundary; hook JSONL records may follow
- `stderr`: diagnostics/logging
- exit `0`: succeeded, noresults, or skipped
- exit non-zero: failed

Rules:

- `on_CrawlSetup__*` hooks should communicate only through side effects such as files, sockets, long-lived processes, or the background readiness stdout line, not stdout JSONL records
- `on_Snapshot__*` hooks should not emit `Machine`, `Process`, or `Binary` records

### Base plugin utilities

The `base/` plugin provides shared Python and JS helpers that all other plugins import:

**Python** (`base/utils.py`):
```python
from abx_plugins.plugins.base.utils import (
    load_config,
    emit_archive_result_record,
    emit_snapshot_record,
)
```

- `load_config()` — load plugin `config.json` with env var + alias + fallback resolution, merged with shared base/common runtime vars like `SNAP_DIR`, `CRAWL_DIR`, `ABXPKG_LIB_DIR`, `PERSONAS_DIR`, `EXTRA_CONTEXT`, `TIMEOUT`, and `USER_AGENT`
- `emit_archive_result_record(status, output_str)` — print `{"type":"ArchiveResult",...}` JSONL to stdout
- `emit_snapshot_record(record)` — emit `{"type":"Snapshot",...}` JSONL to stdout
- `write_text_atomic(path, content)` — write file atomically (temp + rename)
- `find_html_source(snap_dir, ...)` — locate HTML from sibling plugins
- `has_staticfile_output(snap_dir, path)` — check if a sibling plugin produced a file
- `enforce_lib_permissions()` — lock down `ABXPKG_LIB_DIR` so snapshot hooks can read/execute but not write

**JS** (`base/utils.js`):
```javascript
const { loadConfig, getEnv, getEnvBool, getEnvInt, getEnvArray, emitArchiveResultRecord, emitSnapshotRecord } = require('../base/utils.js');
```

- `loadConfig()` — load plugin `config.json` merged with shared base/common runtime vars using env var + alias + fallback resolution
- `emitArchiveResultRecord(status, outputStr)` — emit `ArchiveResult` JSONL to stdout
- `emitSnapshotRecord(record)` — emit `Snapshot` JSONL to stdout

**Test helpers** (`base/test_utils.py`):
```python
from abx_plugins.plugins.base.testing import (
    get_hook_script,
    parse_jsonl_output,
    run_hook,
)
```

- `parse_jsonl_output(stdout)` — extract first matching JSONL record from hook stdout
- `run_hook(hook_script, url, snapshot_id=None)` — run a hook subprocess with explicit URL and optional snapshot ID arguments
- `get_hook_script(plugin_dir, pattern)` — find hook script by glob pattern

> **Note:** Use `sys.path.append()` (not `insert(0, ...)`) because the `ssl/` plugin directory would shadow Python's stdlib `ssl` module.

### Rules

- all plugins should:
  - *overwrite* existing files cleanly if re-run in the same dir, do not skip if files are already present (do not delete and then download, because if a process fails we want to leave previous output intact).
  - the exception to always overwriting files is: chrome.pid. target_id.txt, navigation.json, etc. chrome state which gets reused if it's not stale. we should detect if any of it is stale during chrome launch and tab creation, and clear all of it together if it is stale to prevent subtle drift errors / reuse of stale values.
  - status `succeeded` if they ran and produced output
  - status `noresults` if they ran successfully but produced no meaningful output (e.g. git on a non-github url, ytdlp on a site with no media, paperdl on a site with no pdfs, etc.)
  - status `skipped` if only if *config* caused them not to run (e.g. `YTDLP_ENABLED=False`)
  - status `failed` if any hard dependencies are missing/invalid (e.g. chrome) or if the process exited non-0 / raised an exception
  - return a short, meaningful `output_str` e.g. the page title, mimetype, return status code, or the relative path of the primary output file produced like `output.pdf` or `0 modals closed` or `The Page Title Verbatim` or `favicon.io` or `Not a git URL`
  - define execution order solely using lexicographic sort order of hook filenames
  - use bg hooks for either short-lived tasks that can run in parallel, or long-lived tasks that run for the whole duration of the snapshot and get killed for cleanup/final output at the end
  - treat `bg` vs `fg` as the only programmatic hook distinction; words like `daemon` and `finite` in hook filenames are human hints only
  - bg hooks must emit their first stdout line only after they are ready for the next hook to launch; move non-ready startup diagnostics to stderr
  - bg hooks that depend on other bg hook outputs must implement their own waiters internally + check that inputs are truly ready and not just that the files are present, because they may be spawned in parallel/before the earlier one's outputs are actually ready and race. e.g. html/artifact generation should usually be fg so that later bg parsing hooks can safely depend on it being finished and not just part of the file being present
  - use rich_click for cli arg parsing with a uv file header when hooks are written in python. do not depend on archivebox or django, try to only depend on chrome or the output files of other plugins instead of importing code from them. the one exception is to always use chrome_utils.js as the interface for anything involving chrome.


### Hook JSONL interface

Hooks emit plain JSONL records to stdout. The current hook families and records are:

- `on_CrawlSetup__*` → background readiness line only, no stdout JSONL records
- `on_Snapshot__*` → background readiness line when applicable, then `ArchiveResult`, `Snapshot`, `Tag`

`abx-dl` and ArchiveBox map those records into their own internal event systems. Binary request events are produced from plugin config and handled by `abxpkg`, not by plugin hook scripts. Plugins do not need to know or emit any bus envelope format.
