Bondy Connect

Appearance

Clustering ​

Bondy is designed from the ground up as a distributed system that forms clusters automatically, providing scalability, high availability, and fault tolerance for your application network.

Overview ​

A Bondy cluster is a group of Bondy nodes that work together as a single logical unit to:

  • Scale horizontally - Handle more concurrent connections and throughput by adding nodes
  • Provide high availability - Continue operating even when nodes fail or are taken offline
  • Distribute load - Balance client connections and message routing across multiple nodes
  • Replicate state - Synchronize Realm configuration, user data, and routing information across all nodes

Cluster

A Bondy cluster is a collection of interconnected Bondy nodes that share state and collaboratively route WAMP messages, appearing to clients as a single, unified WAMP Router.

Architecture ​

Masterless Design ​

Bondy uses a masterless (peer-to-peer) architecture where all nodes are equal. There are no master or slave nodes, which provides several advantages:

  • No single point of failure - Any node can fail without losing cluster functionality
  • Simpler operations - No need to elect or promote masters during failures
  • Symmetric scaling - All nodes contribute equally to cluster capacity
  • Uniform client experience - Clients can connect to any node with identical functionality

Full Mesh Topology ​

Nodes in a Bondy cluster form a full mesh network, meaning:

  • Every node maintains direct connections to every other node
  • Cluster coordination and state replication happen directly between all peers
  • No reliance on external coordination services (like ZooKeeper or etcd)

Cluster Size

While Bondy supports clusters of hundreds of nodes, most production deployments run 3-7 nodes for optimal balance between fault tolerance and coordination overhead. Larger clusters are possible but require careful network and resource planning.

State Replication ​

Control Plane State ​

Bondy replicates control plane data across cluster nodes as per-table CRDTs (Conflict-free Replicated Data Types), converging via a pull-based anti-entropy protocol rather than a gossip broadcast: peers compare Merkle Search Tree root hashes and exchange only the pages that actually differ. See Architecture for how the storage layer (bondy_db/bondy_oplog) fits together. This covers:

  • Realm definitions - URIs, security settings, SSO configuration
  • User accounts - Identities, credentials, group memberships
  • Access control rules - Permissions, role assignments, source definitions
  • API Gateway specifications - HTTP routing rules and transformations

Deleting a value here is itself a replicated operation, and the space it occupied is only physically reclaimed once every node has provably seen the deletion — see Deletion and Reclamation for why that distinction is necessary and how Bondy makes it safe.

Eventual Consistency ​

Bondy's replication follows the AP model from the CAP theorem:

  • Available - Nodes remain operational during network partitions
  • Partition-tolerant - Cluster continues functioning when nodes can't communicate
  • Eventually consistent - State converges when connectivity is restored

This means:

  • Updates are accepted on any node without requiring consensus
  • State may temporarily diverge during partitions
  • The system automatically reconciles conflicts when healed

Active Anti-Entropy

Bondy uses Active Anti-Entropy (AAE) to proactively detect and repair state divergence. AAE periodically compares each shard's Merkle Search Tree against a peer's and pulls only the divergent pages, ensuring convergence even after prolonged partitions. See Convergence for what a sync round does and how a replica that is genuinely behind is detected and repaired, and the Data Storage & Active Anti-entropy Configuration Reference for its tunables, including the authentication freshness fence that refuses to authenticate against provably stale security state.

Message Routing ​

Unlike control plane state, WAMP messages are routed in real-time without replicating every registration or subscription cluster-wide. Instead, each node publishes a compact routing summary of what it can serve — see Registry Routing (RIB) — and cross-node routing is decided from the merged summaries:

  • RPC calls - Routed to Callees on any node in the cluster, node-addressed via the RIB
  • PubSub events - Relayed to the nodes hosting interested Subscribers, then delivered locally on each
  • Registration/Subscription routing - Dynamically maintained across the cluster via the RIB's summary cells, not full-entry replication

Bondy tracks which node hosts each client session and routes messages accordingly. If a node fails, clients on that node disconnect, but other clients continue operating normally.

Clustering on or off ​

Whether a node clusters is decided by one setting, cluster.enabled, on by default. A node with off runs on its own: it opens no cluster port, and it compacts its history and reclaims deleted data without waiting for anyone.

A node with on waits for its members before it forgets anything, and it cannot tell "I am alone" from "I have just restarted and not yet rejoined", because Bondy does not persist the cluster membership. It therefore treats a membership of one as a member that has not rejoined yet.

Run a single node with cluster.enabled = off

A single node left on the default on refuses every authentication under the default fence policy and never compacts its history. See Enabling clustering for this and for the rules on turning clustering on or off for a node that already holds data.

Cluster Formation ​

Automatic Discovery ​

Bondy supports two peer discovery strategies, configured via cluster.peer_discovery.type:

  1. DNS-based — query a DNS record that resolves to every peer's address. This is the recommended approach for production, works in any cloud or on-premise environment, and is also how a Kubernetes deployment clusters — point it at a headless Service's DNS name.
  2. Static list — a manually configured, fixed list of peer addresses. Simpler for a small cluster whose membership doesn't change, at the cost of needing every node's config updated when it does.

See Running a Cluster for the configuration steps, and Cluster Configuration Reference for every cluster.peer_discovery.* key.

Join Process ​

When a new node starts:

  1. Discovery - Node queries configured discovery mechanism for peers
  2. Connection - Node establishes connections to discovered peers and joins their membership
  3. State Sync - Anti-entropy pulls the replicated state the new node does not hold from its peers

The node accepts client connections as soon as its own boot finishes; it does not wait for state sync to complete. See Node Lifecycle. Under the default policy of the authentication freshness fence (db.aae.fence.on_isolation = refuse), the new node refuses authentication until its first sync round with a peer completes.

Version Compatibility

All nodes in a cluster must run the same Bondy release. Rolling upgrades are not supported: upgrade a cluster by stopping every node and starting them all on the new release. See Upgrading to 1.0.0.

Peer plane security

The Partisan peer plane (the connections nodes use to cluster) is plaintext and unauthenticated by default. When automatic peer discovery is configured, Bondy refuses to start unless the peer plane is secured with TLS, or the risk is explicitly acknowledged for a network-isolated cluster — an insecure peer plane would otherwise let an on-path attacker read or modify replicated credentials and realm signing keys. See Cluster Configuration for cluster.tls.*.

Fault Tolerance ​

Node Failures ​

When a node fails or becomes unreachable:

  • Clients disconnect - Sessions on the failed node are terminated
  • Cluster continues - Remaining nodes operate normally
  • State remains available - Control plane data is still on remaining nodes
  • Routing adapts - Calls/events route to active nodes only

Clients should implement automatic reconnection with exponential backoff to connect to surviving nodes.

Network Partitions ​

During a network partition (split-brain):

  • Both sides remain available - Each partition continues accepting updates
  • State diverges - Updates on each side aren't immediately visible to the other
  • Clients stay connected - Sessions remain active on their partition
  • Healing - When connectivity restores, AAE repairs state divergence

Partition Handling

Bondy prioritizes availability during partitions. For scenarios requiring strict consistency, implement application-level coordination (e.g., distributed locks via procedures that check quorum).

Recovery ​

After failures or partitions:

  1. Reconnection - Nodes automatically reconnect when available
  2. State synchronization - AAE detects and repairs divergence
  3. Membership updates - Cluster membership list is reconciled
  4. Normal operation - Cluster returns to fully consistent state

Recovery is not instantaneous, and how long it takes depends on what failed. Ordinary lag closes on the next anti-entropy tick. A replica whose history a peer has already compacted past cannot be repaired by syncing at all, and needs a catalogue re-bootstrap — which Bondy schedules only after the gap has been observed twice, because the remedy transfers a whole projection. So a node that has just returned and is still missing data is normally inside the detection window rather than faulty.

Convergence explains those mechanisms, and Verifying Cluster Convergence is the procedure for telling the two cases apart on a running cluster.

Scaling ​

Horizontal Scaling ​

Add capacity by adding nodes. A new node configured with the same peer discovery settings as the rest of the cluster finds the existing nodes and joins them; there is no join command to run. See Running a Cluster.

Benefits:

  • More client connections - Each node can handle millions of connections
  • Higher message throughput - Message routing distributes across all nodes
  • Increased fault tolerance - More nodes means higher redundancy

Load Distribution ​

Bondy distributes load across nodes through:

  • Client distribution - Use DNS round-robin or load balancer to spread connections
  • RPC load balancing - Router distributes calls across available Callees
  • PubSub fan-out - Events are published to local subscribers and relayed to remote nodes

Session Affinity

For optimal performance, use session affinity (sticky sessions) at your load balancer. This minimizes inter-node routing overhead by keeping a client's messages local to one node when possible.

Operational Considerations ​

Monitoring ​

Key metrics to monitor:

  • Cluster membership - Members and connected peers (bondy_cluster_members, bondy_cluster_connected_peers, bondy_cluster_all_members_connected)
  • Anti-entropy recency - Time since the last completed sync with each peer (bondy_oplog_peer_last_sync_age_seconds)
  • State divergence - The applied frontier on each node (bondy_oplog_instance_frontier_hash) and the gaps that sync rounds detect (bondy_oplog_frontier_gap_verdicts_total)

See the Metrics Reference for the labels of each series.

Best Practices ​

  1. Use odd-numbered clusters with the quorum fence policy - Bondy resolves conflicts by CRDT merge, not by majority vote. A majority matters only when db.aae.fence.on_isolation = quorum, where a node authenticates only while it reaches a majority of the membership
  2. Co-locate with clients - Deploy nodes in same regions as your applications
  3. Size appropriately - Start with 3-5 nodes, scale based on actual load
  4. Monitor network health - Low latency and high bandwidth between nodes is critical
  5. Plan for failures - Test failure scenarios before production
  6. Use DNS discovery - Most flexible and works across all environments

Common Topologies ​

Development (a single node, cluster.enabled = off)

bondy@localhost

Small Production (3-node cluster)

bondy1@host1 ←→ bondy2@host2
       ↖     ↙
      bondy3@host3

Multi-Region (5-node cluster)

Region A          Region B
bondy1 ←→ bondy2  bondy4 ←→ bondy5
   ↖         ↙  ↘    ↙
      bondy3 (core region)

See Also ​