Logging and diagnostics

The observability modules write directly to stdout and expose diagnostic data for a handler to compose. They do not install a handler or automatically answer probe jobs.

Correlate logs with a job

Set and reset job_id_var around each job:

from runpod_doc_worker.obs import logging as worker_logging

token = worker_logging.job_id_var.set(job_id)
try:
    worker_logging.info("processing started", source_kind="b64")
finally:
    worker_logging.job_id_var.reset(token)

The default output is one JSON object per line: ts, level, logger, message, an optional job_id, then the caller's fields. Set LOG_FORMAT=text for compact local output. Set LOG_LEVEL=debug to enable calls to debug(); this variable does not filter the other levels.

The text is under message, matching RunPod's own documented record shape ({"level": ..., "message": ...}). Its log viewer reads those names literally, so a record using anything else renders with an empty MESSAGE column — present, valid, and unreadable where it is read.

The public emitters are info(), warning(), error(), and debug(). Caller fields cannot replace ts, level, logger, message, or job_id. They also cannot replace msg: that is the parameter name the emitters take the text as, so passing it is a TypeError rather than a silently dropped field.

Mirror records to another sink

Set WorkerConfig.log_mirror to a callback with the signature (level, message, fields). Stdout is written first. If the mirror raises, the worker emits a warning and continues the job.

def mirror(level: str, message: str, fields: dict) -> None:
    telemetry_client.write(level, message, fields)

Compact failure text

Use compact() before returning exception text to a caller:

from runpod_doc_worker.obs.redact import compact

safe_message = compact(str(exc))

The helper removes credentials, query strings, and fragments from URLs and limits output length. It does not remove arbitrary API keys, tokens, or every secret that may appear in a message.

Collect diagnostics

from runpod_doc_worker.obs import debug

diagnostics = {
    "gpu": debug.collect_gpu_info(),
    "model_dir": debug.find_model_dir(),
}

if caller_is_operator:
    diagnostics["filesystem"] = debug.probe_filesystem()

collect_gpu_info() inspects device 0 when Torch is available. Missing Torch and inspection errors are returned as data.

find_model_dir() searches the effective Hugging Face Hub cache using model_globs. It reports a recent matching cache directory, not proof of which model weights handled a job.

probe_filesystem() performs bounded, read-only, network-free scans and answers whenever it is called. Results can include partial-scan markers when a limit is reached.

Deciding who may ask is yours

This package reads no probe environment variable and has no probe policy. probe_filesystem() returns the dump to whoever calls it.

That is deliberate. Whether a caller may see worker-local paths and declared env values depends on who a worker's callers are — a single-tenant endpoint and a shared one want opposite answers, and only the worker knows which it is. While this package guessed, the name and the default of an operator-facing knob moved twice in two releases, under endpoints already deployed with the old spelling.

So the worker owns the gate: it names the variable its operators set, picks the default its deployment wants, and calls this function once it has decided. Nothing here can move that out from under it.

Migrating from 0.4.x

probe_enabled() and WorkerConfig.probe_default are gone, and <PREFIX>_ENABLE_PROBE and <PREFIX>_DISABLE_PROBE are no longer read here. A worker that relied on either — including on probe_filesystem() raising PermissionError — implements the gate itself, keeping whatever variable name its operators already have:

import os

_NOT_DISABLED = ("0", "false", "no", "off")


def _probe_allowed() -> bool:
    """Whatever this endpoint's operators already set. The name is yours."""
    value = os.environ.get("ACME_DISABLE_PROBE", "").strip().lower()
    if not value:
        return True                 # unset: this worker's own default
    # Anything that is not an explicit "not disabled" denies, so an operator
    # typo leaves the payload unexposed rather than exposed.
    return value in _NOT_DISABLED


if job_input.get("probe") is True:
    if not _probe_allowed():
        raise ValueError("probe is disabled on this endpoint")
    return {"probe": debug.probe_filesystem()}

Two things that predicate is careful about, both inherited from what this package used to do. An unset variable is not the same as a recognised falsey one — only the first should fall through to the worker's own default. And an unrecognised non-blank value denies: ACME_DISABLE_PROBE=maybe disabled the probe in 0.4.x, and a migration that turns a typo into an exposed filesystem dump is not one that preserved anything.

Keeping the old variable name is the point: an operator's existing setting keeps meaning what it meant, and it stops being a name a dependency can change.

Use worker configuration to select cache patterns and probe fields.