Logging Configuration Reference
Bondy logs through Erlang/OTP's Logger application. Every log event carries one of eight severity levels, most to least severe:
| Level | Description |
|---|---|
| emergency | system is unusable |
| alert | action must be taken immediately |
| critical | critical conditions |
| error | error conditions |
| warning | warning conditions |
| notice | normal but significant conditions |
| info | informational messages |
| debug | debug-level messages |
A level filters events less severe than itself: at info (the default), debug events are discarded and everything from info up to emergency is kept.
Setting the level
The primary log level. An event less severe than this is discarded immediately, before it reaches any handler — lowering this is the first step to seeing debug events; raising it is the cheapest way to cut logging volume in production.
Log handlers
A handler takes the events that pass log.level, applies its own level and filters, formats each event as one line, and writes it to its destination. Every handler has its own overload protection, file sync, filter, and formatter settings.
Bondy ships one handler, default, configured under log.handlers.default.*. Additional handlers are declared under log.handlers.$id.*, where $id is a name you choose. Both families accept the same keys with the same defaults, with one exception: log.handlers.default.enabled defaults to on, while log.handlers.$id.enabled defaults to off. Bondy builds a handler only when its enabled key is on, so a new handler needs at least that key set.
Each section below documents the $id keys in full, then lists the matching log.handlers.default.* keys.
The following bondy.conf fragment adds a second handler named errors. It writes events of level error and above to a file, without colour, while the default handler keeps writing to standard output:
log.handlers.errors.enabled = on
log.handlers.errors.level = error
log.handlers.errors.config.type = file
log.handlers.errors.config.file = /var/log/bondy/error.log
log.handlers.errors.formatter.colored = offEvery key the fragment leaves out takes its default. config.type = file is required alongside config.file; see config.file.
Declaring a handler
Whether the handler $id exists. Bondy only builds handlers whose enabled key is on; every other key under log.handlers.$id is ignored while this is off.
The Logger handler module behind this handler: console selects OTP's logger_std_h, which writes to standard output, standard error, or a file, depending on config.type. disk selects OTP's logger_disk_log_h.
disk is rejected in this release
logger_disk_log_h accepts only wrap or halt as its type, but Bondy always passes config.type, whose values are standard_io, standard_error or file. OTP Logger therefore rejects a disk handler with an invalid_config error. To write to a file, keep backend = console and set config.type = file.
The handler's own level, applied after log.level. An event must pass both to be written, so setting this below log.level has no effect. Use log.level to change what Bondy captures at all, and this key to make one handler stricter than the others.
The destination of a console handler: standard output, standard error, or the file named by config.file.
The log file path when config.type is file. A relative path is resolved against the node's working directory. Because config.type always has a value (standard_io unless set), setting config.file alone is not enough: OTP Logger rejects a standard_io or standard_error handler that names a file. Set config.type = file together with this key.
Running under a supervisor
When Bondy runs attached to a terminal (e.g. bondy-connect console) or under a process supervisor that captures standard output (Docker, systemd), leave the default handler on standard_io and let the supervisor own log capture and rotation, rather than pointing Bondy at a file it would then need to rotate itself.
The default handler accepts the same keys under log.handlers.default.
Whether the default handler is active. Turning it off silences Bondy's logging entirely unless another handler is enabled.
The default handler's backend. See log.handlers.$id.backend.
The default handler's level. See log.handlers.$id.level.
The default handler's destination. See log.handlers.$id.config.type.
The default handler's log file. Requires log.handlers.default.config.type = file. See log.handlers.$id.config.file.
Overload protection
Each handler protects itself from overload in three ways. Burst limiting caps the number of events written per time window. Queue-length thresholds switch the handler between asynchronous, synchronous, drop, and flush modes as its message queue grows. Overload kill terminates and restarts a handler whose queue or memory grows past a limit. The defaults suit most deployments; change them only after observing a specific overload problem.
Burst limit
Enables burst limiting. With it on, the handler writes at most burst_limit_max_count events per burst_limit_window_time and drops the rest until the window ends. This keeps a burst of events from filling a log file quickly or slowing down file sync.
The maximum number of events the handler writes within one burst window.
The length of the burst window.
Queue-length thresholds
The handler checks its message queue length as it processes events. The three thresholds below select its mode. Below sync_mode_qlen, processes that log hand events to the handler and continue (asynchronous mode). From sync_mode_qlen, each logging process waits until the handler has processed its event (synchronous mode), which slows producers down. From drop_mode_qlen, new events are dropped before they reach the handler. From flush_qlen, the handler discards every event already in its queue. The values must satisfy sync_mode_qlen ≤ drop_mode_qlen ≤ flush_qlen and drop_mode_qlen > 1, or OTP Logger rejects the configuration. Setting two adjacent thresholds equal disables the lower mode, since the higher one is checked first.
The queue length at which the handler switches from asynchronous to synchronous mode.
The queue length at which new events are dropped.
The queue length at which the handler discards its whole queue.
Overload kill
Enables overload kill. With it on, the handler process is terminated when its queue exceeds overload_kill_qlen or its memory exceeds overload_kill_mem_size, and is restarted after overload_kill_restart_after. This bounds the memory a handler can use when the other mechanisms cannot keep up.
The maximum memory, in bytes, the handler process may use before it is terminated.
The maximum queue length the handler may reach before it is terminated.
The delay before a terminated handler restarts.
Default handler
The default handler accepts the same overload keys under log.handlers.default.config, with the same defaults.
See log.handlers.$id.config.burst_limit_enable.
See log.handlers.$id.config.burst_limit_max_count.
See log.handlers.$id.config.burst_limit_window_time.
See log.handlers.$id.config.sync_mode_qlen.
See log.handlers.$id.config.drop_mode_qlen.
See log.handlers.$id.config.flush_qlen.
See log.handlers.$id.config.overload_kill_enable.
See log.handlers.$id.config.overload_kill_mem_size.
See log.handlers.$id.config.overload_kill_qlen.
See log.handlers.$id.config.overload_kill_restart_after.
File sync
A handler that writes to a file can sync buffered data to disk at a fixed interval. It only syncs when something has been logged since the last sync. Without repeated sync, the operating system decides when buffered data reaches the disk. Handlers that write to standard output or standard error do not sync.
Controls repeated file sync.
The flag is inverted in this release
Bondy's configuration translation turns repeated sync off when this key is on, and keeps filesync_repeat_interval when it is off. With the defaults, a file handler therefore does not sync on an interval. To get repeated sync, set this key to off.
How often the handler syncs to disk while repeated sync is active.
The default handler accepts the same keys under log.handlers.default.config.
See log.handlers.$id.config.filesync_repeat_enable.
See log.handlers.$id.config.filesync_repeat_interval.
Filtering
Every handler runs a fixed chain of filters on each event that passes its level. Bondy builds the chain from the keys below:
- Progress reports are written or dropped according to
allow_progress_reports. - Events from processes whose group leader is on a remote node are dropped.
- Events with no
domainmetadata are written. Bondy's own log events carry no domain, so they pass here. - Events in a domain named in
filter_domainsare written. Events in one of the known domains (otp,ssl,bondy_audit) that is not named are dropped. - An event no filter decided on is handled by
filter_default.
What happens to an event that no filter decided on: log writes it, stop drops it. With stop, an event whose domain does not exactly match one in filter_domains is dropped.
A comma-separated list of the domains this handler writes, drawn from otp, ssl, and bondy_audit. Each name matches an event's domain exactly: otp matches the domain [otp, sasl], which OTP uses for supervisor, crash, and progress reports; ssl matches [ssl]; bondy_audit matches [bondy_audit]. Events whose domain is [otp] alone, or [otp, ssl], match none of these and fall through to filter_default. Leaving a domain out of one handler and naming it in another sends that domain's events to the second handler only.
Whether the handler writes OTP progress reports, the events supervisors and applications emit as they start children. They are noisy and are off by default. Turning this on also requires otp in filter_domains, since progress reports carry the [otp, sasl] domain and are dropped otherwise.
The default handler accepts the same keys under log.handlers.default.
See log.handlers.$id.filter_default.
See log.handlers.$id.filter_domains.
See log.handlers.$id.allow_progress_reports.
Formatting
Every handler formats events with Bondy's bondy_logger_formatter, which writes each event as one line of key=value pairs:
when=2026-01-01T10:00:00.000000+00:00 level=info pid=<0.1234.0> at=bondy_mod:fun/2:42 description="..." node=bondy@127.0.0.1 realm=com.example ...Metadata such as realm, session_id, or trace_id appears only when the event carries it. The keys below control how values are rendered.
How many levels of nested maps the formatter flattens into parent_child=value pairs. Deeper levels are printed as key=.... -1 removes the limit.
The depth limit for printing Erlang terms that are not text, such as tuples and lists of integers. -1 removes the limit.
Whether the formatter wraps the when to description part of each line in a terminal colour chosen by level. Turn it off for handlers that write to a file or to a log collector that does not interpret ANSI escape sequences.
Each colored_<level> key holds the ANSI escape sequence that starts a line of that level. The defaults shown use \e for the escape character (byte 0x1B). bondy.conf does not interpret \e, so a custom value must contain the escape character itself.
The colour of debug lines.
The colour of info lines.
The colour of notice lines.
The colour of warning lines.
The colour of error lines.
The colour of critical lines.
The colour of alert lines.
The colour of emergency lines.
The UTC offset applied to the when timestamp, in microseconds. 0 prints UTC with a +00:00 suffix; 3600000000 prints UTC+1 with +01:00.
The character placed between the date and the time in the when timestamp, as in RFC 3339.
The default handler accepts the same keys under log.handlers.default.formatter.
See log.handlers.$id.formatter.map_depth.
See log.handlers.$id.formatter.term_depth.
See log.handlers.$id.formatter.colored.
The default handler's debug colour.
The default handler's info colour.
The default handler's notice colour.
The default handler's warning colour.
The default handler's error colour.
The default handler's critical colour.
The default handler's alert colour.
The default handler's emergency colour.
See log.handlers.$id.formatter.time_offset.
See log.handlers.$id.formatter.time_designator.
Log messages
Bondy logs structured reports. The description field of each record below is fixed text, so an alert can match on it exactly. Where two records share a description, the alert also matches on reason. Each entry lists the record's other fields, its level, and when Bondy emits it.
Replication and compaction
GC stalled
description => "GC stalled: the trigger returned an error for this instance. For reason unconfirmed or no_frontier, a member is holding it: see member_status (highest last_sync_age_ms, or never_synced). Bring them back online, or retire them with a deliberate cluster membership removal — a stalled member never ages out. For reason no_members, clustering is enabled and no other member is listed: a member that has just restarted holds until it rejoins, and a node meant to run alone needs cluster.enabled = off."| Field | Value |
|---|---|
scheduler | bondy_oplog_gc_scheduler (compaction) or bondy_oplog_reclaim_scheduler (reclamation). |
instance | The oplog instance that did not compact or reclaim. |
reason | The stall reason, for example unconfirmed, no_frontier or no_members. |
missing_members | For unconfirmed, the members with no recorded frontier. Otherwise []. |
member_status | One entry per cluster member. The entry carries the member's last_sync_age_ms, or never_synced when the member has no sync record for this instance. The value is unavailable when the membership cannot be read. |
Level: warning.
Emitted: when a compaction or reclamation run stalls, at most once every 60 seconds for each instance and scheduler. The limit applies to the log only. bondy_oplog_gc_scheduler_events_total counts every trigger outcome, and bondy_oplog_reclamation_stalled_total counts every reclamation stall (see Metrics). Bondy never logs the reason idle. On a node that runs alone, no_members means that cluster.enabled is still on.
Bootstrap refused: peer behind truncation
description => "Refused to install the peer's snapshot: the peer has not seen events this node truncated. If this persists, the two nodes truncated history separately before they were joined (for example, each ran with cluster.enabled = off). Empty one of them and rejoin it."| Field | Value |
|---|---|
instance | The oplog instance. |
peer | The peer whose snapshot this node refused. |
reason | {source_behind_truncation, Uncovered}. Uncovered maps each origin that the peer's frontier does not cover to the sequence number through which this node truncated it. |
Level: warning.
Emitted: each time this node refuses a re-bootstrap from peer. The refused install writes nothing, and the instance keeps compacting and serving. See Enabling clustering for the recovery. A re-bootstrap that fails for any other reason logs bootstrap_catalogue session failed at warning, with the same fields.
Causal hold
description => "Causal hold: this replay batch carries remote-origin events beyond a per-origin contiguity gap, or events whose context observed one; they are excluded from the fold and the applied-frontier merge, and the caller keeps its replay cursor so they re-present until the gap fills or a rebootstrap repairs it."| Field | Value |
|---|---|
instance_id | The oplog instance. |
held | The held events, as origins and sequence numbers. |
Level: info.
Emitted: each time a replay holds events back. bondy_oplog_events_held_total counts the same events. A burst of records is normal while a node rejoins after a long absence. A record that repeats for the same held events on a healthy cluster means a gap that does not fill. See Per-Origin Prefix Closure.
MST root lost pages
description => "The MST root has lost pages. Anti-entropy advertises it as empty, so peers pull nothing from it. Compaction rebuilds the tree once every member covers what this replica applied."| Field | Value |
|---|---|
instance_id | The oplog instance. |
root | The root hash that references pages missing from the store. |
Level: warning.
Emitted: the first time the instance finds that a root references a page the store lacks.
MST rebuilt
description => "Rebuilt an MST whose own root lost pages: every member covers what this replica applied, so the tree was truncated past every lost page and anti-entropy resumes on it."| Field | Value |
|---|---|
instance_id | The oplog instance. |
dropped_events | The number of events that the truncation removed from the tree. |
Level: notice.
Emitted: when compaction rebuilds an MST whose root lost pages. bondy_oplog_mst_rebuilt_total counts each record, with reason unservable_root. Each record means that pages went missing from the store. See Convergence.
Cluster membership
Node removed from the cluster
description => "Node removed from the cluster"| Field | Value |
|---|---|
node | The removed node. |
survey | The pre-removal survey of the remaining members. |
Level: notice.
Emitted: by the node that serves bondy.cluster.leave, after the survey finds every remaining member answering and ready. When the survey refuses a removal, Bondy does not log this record.
Authentication
WAMP authentication failed
description => "WAMP authentication failed"| Field | Value |
|---|---|
reason | Why the authentication was refused, for example bad_signature, invalid_nonce, {no_such_user, Authid}, or temporarily_unavailable when the authentication freshness fence refused. |
authid | The authid the client presented, or undefined. |
authmethod | The method being answered, or undefined when the refusal came before a method was chosen. |
authmethods | The methods the client requested in its HELLO. |
The logger metadata carries realm, when the realm exists, and peername.
Level: info.
Emitted: whenever a session refuses a WAMP authentication. The client receives only the generic wamp.error.authentication_failed, or wamp.error.unavailable for a fence refusal, so this record is the only place the reason is kept.
Listeners and connections
Connection threshold exceeded
description => "Connection threshold exceeded"| Field | Value |
|---|---|
threshold_percent | 75 or 90. |
listener | The listener name. |
alarm_name | num_connections_75 or num_connections_90. |
connections | The number of connections on the listener. |
Level: warning at 75 %, alert at 90 %.
Emitted: when the connections of a listener reach 75 % or 90 % of its max_connections. Each threshold logs at most once every 5 seconds. A listener whose max_connections is infinity never logs this record.
WebSocket closed on idle timeout
description => "Connection closed"
reason => idle_timeout| Field | Value |
|---|---|
reason | idle_timeout. |
idle_timeout | The timeout in force, in milliseconds. |
Level: error.
Emitted: when Bondy closes a WebSocket connection that stayed idle for listeners.$name.websocket.idle_timeout. Other WebSocket closures also use the description Connection closed, with a different reason.
Shutdown
Drain: listeners suspended
description => "Suspending the normal-phase listeners. No new connections will be accepted from now on; the readiness and metrics endpoints stay up for the duration of the drain."Level: notice.
Emitted: when a node starts to shut down. From this record on, the node accepts no new client connections, and existing connections stay open. The next record, Terminating all client connections., marks the point where client connections close. See Node Lifecycle.
See also
- Node Configuration Reference —
vm.W, the separate VM-level warning-message classification, not to be confused with this application-level logging configuration.