Two systems, 358 calls, one silent level.

Five levels are defined across the two systems. Four are in active use. The distribution is heavily weighted toward INFO and WARN — diagnostic depth at the DEBUG tier is nearly absent, and the highest severity tier (CRITICAL) is dead code.

INFO — phase markers and operation summaries

INFO is used for high-level lifecycle events: job start/end, phase transitions, batch submission and completion, account sync milestones. It always includes entity IDs and summary metrics. It never fires per-item in tight loops — that's left to DEBUG (or not logged at all).

WARN — recoverable degradation

WARN is the second-busiest level and covers three distinct patterns: external system degradation (SnapTrade 429s, network retries), concurrent state machine race conditions in the sync pipeline, and data integrity gaps where processing continues with partial data (missing FX rates, unknown tickers, null brokerage IDs). Every WARN means the operation continued.

ERROR — terminal failures

ERROR marks operations that stopped. It is the signal for incidents and alerting. Always includes machine-readable error codes, entity IDs for correlation, and consequence language where data loss is possible. No ERROR should be informational — if the operation recovered, it was a WARN.

DEBUG — success signals on high-frequency paths

The six DEBUG calls cover narrow cases: market holiday cache refreshes, batch polling status, per-portfolio save confirmations, bulk-save outcomes, and skipped-item notices. These are correct targets for DEBUG. The gap is that almost no high-frequency path produces any diagnostic signal at all.

Level-to-GCP severity mapping

WARN breakdown by pattern

The Severity.CRITICAL level and the logger.critical() method are fully implemented. The schema documents their intended contract precisely: reserved for log entries emitted immediately before exit(1), mapping to GCP's highest alert tier. But the level has zero call sites across the entire codebase.

The actual pattern for fatal exits in every batch job is to log at ERROR and then call process.exit(1) separately. The two signals are disconnected — GCP sees an ERROR entry and then a container exit, but never a CRITICAL entry that would let an alerting policy fire specifically on "job crashed, not just errored."

Current exit pattern (batch-portfolio-insights)

// What actually happens today
catch (err) {
  log(LogLevel.ERROR, "Batch job crashed", { message: err.message });
  Deno.exit(1);
}

// What the schema intended
catch (err) {
  logger.critical("Batch job crashed", {
    event: LogEvent.JOB_FATAL,
    phase: LogPhase.TEARDOWN,
    error: toErrorDetail("JOB_CRASHED", err),
  });
  Deno.exit(1);
}

Scope

The fix is mechanical: replace the terminal log(LogLevel.ERROR) + Deno.exit(1) pair in each job's top-level catch block with logger.critical() using LogEvent.JOB_FATAL. Eight jobs are affected. No schema changes required — the infrastructure is already there.

The six existing DEBUG calls are correctly placed — they cover success signals on high-cardinality operations that don't warrant INFO noise. The problem is scope: whole subsystems that would benefit from per-operation tracing produce no diagnostic output between the WARN/ERROR conditions they do log.

Existing DEBUG call sites

Where DEBUG is absent but needed

• SnapTrade API client (packages/snaptrade/client.ts) Outbound requests are invisible unless they fail. A per-request DEBUG entry (method, URL, status, latency) would make rate limit patterns and latency regressions visible without adding INFO noise.
• API middleware (packages/api/middleware/logging.ts) Only logs procedure entry/exit. Sub-operations — DB queries, cache hits, external calls — within a procedure are silent. No way to trace where time is spent inside a slow oRPC call.
• Database operations No query-level DEBUG anywhere. Failed queries are logged at WARN/ERROR, but slow or unexpectedly large queries are invisible.
• Webhook processing (packages/snaptrade/webhook.ts) Zero log output on success paths. A received-and-processed confirmation at DEBUG would make the webhook pipeline observable without polluting INFO.

Both logging systems emit newline-delimited JSON to standard streams. GCP Cloud Run captures these streams and forwards them to Cloud Logging, where the severity, message, and timestamp fields are promoted to first-class indexed columns. Everything else lands in jsonPayload.

JobLogger entry shape

{
  "severity":    "INFO",
  "message":     "Starting portfolio insights batch job",
  "timestamp":   "2026-05-25T14:32:18.123Z",
  "jobName":     "batch-portfolio-insights",
  "jobType":     "batch",
  "environment": "production",
  "runId":       "550e8400-e29b-41d4-a716-446655440000",
  "event":       "JOB_START",
  "phase":       "INIT",
  "context": {
    "targetDate": "2026-05-23"
  },
  "metrics": {
    "total": 450,
    "durationMs": 5432
  }
}

log(LogLevel) entry shape

{
  "severity":     "INFO",
  "message":      "-> user.deleteAccount",
  "functionName": "oRPC_procedure",
  "timestamp":    "2026-05-25T14:32:18.123Z"
}

GCP filter examples

Top-level fields in the JobLogger output are indexed and filterable directly in GCP Log Explorer. The event and errorCode fields were specifically designed for this — their values were chosen to be useful as alert conditions, not just human-readable labels.

-- Only job-fatal events
jsonPayload.event="JOB_FATAL"

-- All errors from one job
severity>=ERROR AND jsonPayload.jobName="batch-portfolio-insights"

-- Trace all logs for one run
jsonPayload.runId="550e8400-e29b-41d4-a716-446655440000"

-- Follow a specific user across all jobs
jsonPayload.context.userId="<uuid>"

LogEvent taxonomy — 22 defined event types

All ten Cloud Run jobs and their orchestration layers log to GCP Cloud Logging. They arrive under different resource.type labels — so they appear separated in Log Explorer — but they are all in the same logging system. Cloud Scheduler and Cloud Workflows are not separate log destinations; they are separate resource types within Cloud Logging.

Full product × destination matrix

All ten Cloud Run jobs write structured JSON to stdout and stderr. Cloud Run captures these streams automatically and forwards them to GCP Cloud Logging — no SDK or explicit client required. The GCP project is castello-backend, region us-central1, and all images are stored in Artifact Registry under castello-batch.

What the logs look like today

Because the monitoring module's JobLogger is not yet connected, every job currently emits flat JSON via packages/core/logging.ts — not the rich schema defined in monitoring/schema.ts. There are no jobName, runId, event, or phase fields in production logs today. The fields available for filtering are limited to those spread from each log call's data argument, which varies by callsite.

// What GCP receives today (flat, core/logging.ts)
{
  "severity": "INFO",
  "message":  "Fetched X brokerages from SnapTrade",
  "timestamp":"2026-05-25T14:32:18.123Z",
  "count":    42
}

// What GCP will receive after monitoring module is connected
{
  "severity":  "INFO",
  "message":   "Fetched X brokerages from SnapTrade",
  "timestamp": "2026-05-25T14:32:18.123Z",
  "jobName":   "sync-brokerage-info",
  "runId":     "550e8400-e29b-41d4-a716-446655440000",
  "event":     "STEP_SUCCESS",
  "phase":     "PREFETCH",
  "metrics":   { "total": 42, "durationMs": 812 }
}

How to navigate to job logs in GCP Console

Cloud Workflows logs

Cloud Workflows writes to Cloud Logging under resource.type="workflows.googleapis.com/Workflow". By default only errors are logged. Setting call_log_level = "LOG_ALL_CALLS" on the Terraform resource enables step-level entries — which steps executed, what arguments they passed to each Cloud Run job, and what the response was. Custom messages can also be written from within the workflow YAML using sys.log(), which land in the same stream.

Cloud Scheduler logs

Cloud Scheduler writes to Cloud Logging under resource.type="cloud_scheduler_job" on every invocation — the time it fired, the HTTP target it called, and the response status code it received. The "View Logs" button in the Cloud Scheduler console is a shortcut link that opens Cloud Logging with this filter pre-applied. It is not a separate log store.

Beyond the silent code paths identified in §v, three pieces of observability infrastructure exist in the codebase but produce no output in production. Each is fully designed, partially implemented, and then stopped short of being activated.

1. JobLogger — fully designed, never deployed

The monitoring/ module defines a complete structured logging schema: 22 LogEvent types, 5 LogPhase values, typed sub-objects for context, metrics, and error detail, and a createLogger() factory that produces a JobLogger bound to a specific job's identity. This is the infrastructure this branch (feat/logging-monitoring) was created to build.

No batch job or cron job currently imports or calls createLogger(). GCP Cloud Logging receives flat JSON today. The monitoring module exists only on disk.

2. OpenTelemetry — wired but dormant

packages/llm-batch/tracing.ts wraps every Anthropic Batch API call in an OpenTelemetry span via a tracedFetch() helper. It adds gen_ai.* semantic attributes, sanitises URLs, and sets span status on HTTP errors. The implementation is complete and correct.

Activation requires two things that are not currently set in production: the OTEL_DENO=true environment variable (gates all tracing calls behind a no-op check), and an OTEL exporter endpoint (OTEL_EXPORTER_OTLP_ENDPOINT or equivalent). Without both, every tracedFetch() call is a plain fetch() — no spans are emitted anywhere.

3. Cloud Monitoring — no metrics layer configured

There is no Cloud Monitoring (formerly Stackdriver Metrics) setup in the Terraform configuration. No custom dashboards, no uptime checks, no alerting policies, no log-based metrics. Job success and failure are recorded as log events but never elevated to time-series metrics.

This means there is no way to answer operational questions like "how many jobs failed this week?" or "what is the P95 duration of the portfolio insights batch?" without writing ad-hoc Log Explorer queries. There are no alerting policies that fire when a job fails — failure is visible in logs, but only if someone is watching.

• No alerting on job failure A batch-portfolio-insights crash produces an ERROR log in Cloud Logging, but no notification fires. Detection relies on downstream effects (missing insights) or someone manually checking logs.
• No duration tracking Job execution time is not tracked as a metric. Duration regressions — a job that used to take 20 minutes now taking 90 — are invisible unless someone compares log timestamps manually.
• No success rate dashboard There is no view of job success/failure rates over time. Individual runs are auditable but aggregate trends are not surfaced anywhere.