Skip to content

Runtime Configuration

Use Runtime Configuration to control how generation behaves at run time. Adjust logging, paging, snapshot semantics, and integration tuning (Kafka, Redis, Mongo) to fit your workload.

Access

  • Project → Settings → Runtime Configuration

How It Works

The Runtime Configuration panel is capability‑driven. Sections and fields are provided by the server so the UI always matches your backend features.

Note

Changes are auto‑saved after a brief delay. Only modified fields are written; clearing a value removes it from the configuration.

Sections

Logging & Metrics

  • Log Level (INFO, DEBUG, WARNING, ERROR, CRITICAL)
  • Log Source Metrics (CONTROL/PAGE/SUMMARY/LEASE)
  • Log Export Metrics (exporter/target throughput)

Tip

Switch to DEBUG temporarily when investigating issues. Use INFO for normal runs to reduce noise.

Preview & Exporter

  • Preview Record Limit (rows)
  • Disable Preview Exporter (skip UI preview data)

Paging & Snapshot

  • Page Mode (bytes, rows, auto)
  • Target Bytes (MB) / Target Rows
  • Snapshot Mode (auto, rr, scn, snapshot, copy)
  • Statement Timeout (ms)
  • Require Index
  • Table Resume Scope (stmt/table)

Kafka

  • Producer and consumer tuning: batch size, flush intervals, max in‑flight requests, poll timeout, retries

Mongo

  • Allow Aggregate Paging (advanced)
  • Heavy Advance Factor

Memstore & Flow

  • Batch Size
  • Cyclic Max Records
  • Cleanup on Exit
  • Parallel Fetch
  • Cyclic Cache Max (MB)

Redis & Leases

  • Disable Redis
  • Socket/Connect timeouts, health check interval
  • Max Connections
  • Lease Cleanup on Exit, Retention (seconds)
  • Cleanup on Exit (log/preview/test‑result keys)

Planning & Fallback (RDBMS)

  • Allow Unverified User Order
  • Fallback Mode (temp_rownum, cursor_only, pg_ctid)
  • Allow CTID Fallback (Postgres)

Platform (Client Healthcheck)

  • Healthcheck on Registration
  • Timeout (ms)

Tips

  • For memory‑sensitive workloads, use Page Mode: bytes and tune Target Bytes (MB).
  • For large RDBMS exports without a stable order, pick a conservative fallback (cursor_only).

Reference

Supported Runtime Keys (summary)

Key Type Default Description
ds_log_source_metrics bool false Emit source CONTROL/PAGE/SUMMARY/LEASE metrics.
ds_log_export_metrics bool false Emit exporter/target throughput logs. (Legacy alias ds_log_target_metrics also supported.)
preview_record_limit int 20 Number of preview samples at root (nested always 1).
disable_preview_exporter bool false Disable attaching the preview exporter.
disable_redis bool false Disable Redis‑backed handlers/leases (uses local stubs).
memstore_batch_size int 1000 Flush threshold for memstore exporters (records).
memstore_cyclic_max_records int 2000000 Guardrail for cyclic memstore fetches.
memstore_cleanup_on_exit bool true Remove task memstore keys at exit.
kafka_consumer_poll_timeout_ms int — Consumer poll timeout.
kafka_consumer_max_retries int 3 Max empty‑poll retries before giving up.
kafka_exporter_flush_batch int 100 Buffer size for Kafka exporter flush.
kafka_max_messages_per_sec float — Producer throttling (messages/sec).
kafka_throttle_burst int — Throttle burst capacity (tokens).
kafka_producer_flush_every int — Flush after N sends.
kafka_producer_flush_interval_ms int — Periodic flush interval.
kafka_max_inflight_futures int 1000 Max in‑flight producer futures before blocking.
kafka_future_get_timeout_ms int 60000 Timeout for future.get.
ds_page_mode str (bytes\|rows\|auto) bytes Page unit policy.
ds_target_bytes_mb float 128.0 Byte budget per page (bytes mode).
ds_target_rows int 20000 Rows per page (rows mode) and token‑range guidance.
ds_snapshot_mode str (auto\|rr\|scn\|snapshot\|copy) auto Snapshot/transaction semantics.
ds_statement_timeout_ms int 600000 Statement timeout within snapshot transactions.
ds_require_index bool false Fail fast if enforced order key lacks index (RDBMS).
ds_table_resume_scope str (stmt\|table) stmt Continuation scope for table scans.
ds_mongo_allow_aggregate_paging bool false Allow source‑level paging for aggregates.
ds_mongo_heavy_advance_factor int 2 Heuristic for heavy‑advance token‑range decisions.

Advanced Runtime Keys

These keys are functional but typically tuned by engineers rather than operators.

  • Redis client tuning
  • redis_socket_timeout_ms (int, default 5000)
  • redis_connect_timeout_ms (int, default 3000)
  • redis_health_check_interval_s (int, default 0)
  • redis_max_connections (int|null)
  • Lease/cleanup controls
  • ds_lease_cleanup_on_exit (bool, default false)
  • ds_lease_retention_seconds (int, default 604800)
  • redis_cleanup_log_on_exit (bool, default false)
  • redis_cleanup_preview_on_exit (bool, default false)
  • redis_cleanup_testresult_on_exit (bool, default false)
  • Manifest cache (default disabled)
  • ds_manifest_cache_enable (bool, default false)
  • ds_manifest_cache_max_entries (int, default 256)
  • ds_manifest_cache_ttl_minutes (int, default 60)
  • RDBMS ordering/fallback
  • ds_allow_unverified_user_order (bool, default true)
  • ds_fallback_mode (str: temp_rownum|cursor_only|pg_ctid, default temp_rownum)
  • ds_pg_allow_ctid_fallback (bool, default false)
  • Unload thresholds (planning)
  • ds_unload_auto_row_threshold (int, default 50000000)
  • ds_unload_auto_size_gb (float, default 50.0)
  • Mongo/Fastops tuning
  • fastops_mongo_sample_every (int, default 8)
  • fastops_mongo_use_server_stats (bool, default true)
  • Rust/Fast generators
  • rust_fast_generators (bool, default true)
  • rust_fast_buffer_size (int, default 8192)
  • Memstore/data flow
  • memstore_parallel_fetch (bool, default false)
  • ds_cyclic_cache_max_mb (float, default 100.0)

Client Healthcheck (preflight)

Validate connectivity and authentication for registered clients immediately at registration time (setup/include). Defaults are ON and strict fail‑fast.

  • client_healthcheck_on_registration (bool, default true): Runs a short healthcheck for each registered client using the same credentials the run will use.
  • client_healthcheck_timeout_ms (int, default 3000): Target timeout for individual probes where supported.

When to use these settings

  • Logging
  • Increase log_level to DEBUG temporarily during investigations; keep INFO otherwise.
  • Enable ds_log_source_metrics/ds_log_export_metrics to troubleshoot throughput or latency.
  • Preview & Exporter
  • Lower preview_record_limit on very large datasets; disable preview exporter to reduce overhead in batch runs.
  • Paging & Snapshot
  • ds_page_mode: use bytes to bound memory, rows for fixed page sizes, auto for general cases.
  • Tune ds_target_bytes_mb or ds_target_rows according to row/record size and downstream limits.
  • ds_snapshot_mode: pick a transactional mode (e.g., rr, scn) if consistency is required.
  • Kafka
  • Use kafka_max_messages_per_sec/kafka_throttle_burst to throttle producers; adjust flush intervals/batch size based on broker latency.
  • Redis & Leases
  • Set disable_redis only for instances without Redis; otherwise keep enabled for robustness and observability.
  • Planning & Fallback (RDBMS)
  • Keep ds_require_index true in strict environments; use fallback mode cursor_only when indexes are missing.

If you need more detailed guidance for your workload, let us know the source/target, data sizes, and latency constraints and we can recommend concrete values.