High availability

Arbinel can run as several identical nodes behind a load balancer. Every node serves web, API and Git traffic and processes queue items. One node is elected leader and runs the work that must happen once per cluster, such as scheduled cron tasks.

Forgejo has no guide for running more than one instance, so the Forgejo settings that a cluster depends on are included here.

Enable high availability

Give every node the same app.ini and set:

[high_availability]
ENABLED = true

Key	Default	Meaning
`ENABLED`	`false`	Turn on multi-node mode.
`NODE_ID`	host name, process ID and a random suffix	This node’s name in the cluster, shown on the admin pages and in metrics. Set it to a stable, unique value per node if you want readable names.
`LEASE_TTL`	`30s`	How long the leader lease lasts without renewal.
`RENEW_INTERVAL`	`10s`	How often the leader renews the lease. It must stay below half of `LEASE_TTL`, otherwise Arbinel uses a third of `LEASE_TTL`.

When ENABLED is true, a node checks the rest of the configuration at startup and refuses to start if any of it would break with several nodes. forgejo migrate --check runs the same checks without starting a server.

Shared settings

Arbinel checks each row at startup, except the repository root, the secrets and the SSH host key.

Setting	Requirement
`[database] DB_TYPE`	Not `sqlite3`. Use PostgreSQL or MySQL that every node reaches.
`[cache] ADAPTER`	Not `memory` or `twoqueue`. Use `redis`. `memcache` passes the check, but request budgets do not work with it.
`[session] PROVIDER`	Not `memory` or `file`. Use `redis` or `db`.
`[queue] TYPE`	`redis`. Only the `[queue]` section itself is checked, so do not set another type in a `[queue.*]` section.
`[storage.*] STORAGE_TYPE`	Not `local` for attachments, LFS, avatars, repository avatars, packages, repository archives, Actions logs and Actions artifacts. Use `minio` (any S3-compatible store).
`[indexer] ISSUE_INDEXER_TYPE`	Not `bleve`. Use `db`, `elasticsearch`, `meilisearch` or `typesense`.
`[indexer] REPO_INDEXER_TYPE`	When code search is on, not `bleve`. See Code search on several nodes.
`[repository.upload] TEMP_PATH`	On the same filesystem as the repository root, because the upload form and the commit request often reach different nodes.
`[realtime] CONN_STR`	When `[realtime] ENABLED` is true, a Redis server. It defaults to `[queue] CONN_STR` when the queue type is `redis`, so you rarely set it.
`[repository] ROOT`	One shared directory, for example on NFS or CephFS. Not checked at startup, see below.
Secrets	The same `SECRET_KEY`, `INTERNAL_TOKEN`, `LFS_JWT_SECRET` and `[oauth2] JWT_SECRET` on every node. Not checked.
SSH host key	The same host key on every node, or clients see a changed key when the load balancer sends them to another node. Not checked.

The Forgejo configuration cheat sheet describes each of these settings, for example [cache], [session] and [queue]. Redis becomes a single point of failure for cache, sessions and queues, so run it highly available, for example with Sentinel or a managed service.

Shared repository root

Start Arbinel only after the shared repository root is mounted, for example with RequiresMountsFor= in the systemd unit. As a second safeguard, a node whose repository root is missing, unreadable, or empty while the database lists repositories:

fails the repo-root check of /api/healthz,
does not take or keep leadership,
refuses the delete_missing_repos and reinit_missing_repos cron tasks.

Those two tasks also refuse a run that would touch more than ten repositories and more than a tenth of all repositories. If the repository root takes longer than 5 seconds to respond, the check reports warn instead of fail. Otherwise every node would fail at once when the share stalls.

Leader election

The nodes elect a leader through a row in the codefloe_lease table of the shared database. The leader renews it every RENEW_INTERVAL and gives up leadership before any other node can see the lease expire.

Only the leader runs:

scheduled cron tasks, including tasks set to run at startup,
polling of the [email.incoming] mailbox.

Every node can still start a cron task by hand from the admin pages. Each run takes a lease of its own, so the same task never runs on two nodes at once. A run that finds the task busy is recorded as skipped. Merges, wiki edits, repository transfers, package uploads and schema migrations use leases from the same table, so two nodes never work on the same item at once.

When a node stops cleanly, it releases its lease and another node takes over within one RENEW_INTERVAL. After a crash, the takeover waits up to LEASE_TTL.

In multi-node mode the requeue_undelivered_hook_tasks cron task is on by default and runs every 5 minutes. It adds webhook deliveries that the shared queue lost back to the queue.

Check the current leader

The admin dashboard and the cron monitor page show this node’s name, whether it leads, and which node is the current leader. With [metrics] ENABLED = true, /metrics reports:

gitea_codefloe_ha_is_leader, 1 on the leader, labelled with node,
gitea_codefloe_ha_lease_age_seconds, the time since the lease was last renewed.

Exactly one node should report gitea_codefloe_ha_is_leader 1.

Load balancer health checks

Point the load balancer’s health check at /api/healthz.

Status	HTTP code	When
`pass` or `warn`	200	The node is healthy. `warn` means the repository root answers slowly.
`fail`	424	The database or cache check fails, or the repository root is missing or empty.
`fail`	503	The node is draining.

The JSON body names the failing check under checks.

Drain a node

Take a node out of the load balancer before maintenance:

forgejo manager drain

The node keeps serving requests already in flight, but /api/healthz answers 503 with draining since <time> in checks.drain, so the load balancer stops sending it new ones.

forgejo manager drain --status prints draining since <time> or serving.
forgejo manager drain --off ends the drain.

Arbinel keeps the drain state in the running process only, so a restarted node always comes back serving. A drained node can still be the leader. To hand over cron work, stop the node or wait for the planned restart.

Upgrades

Run forgejo migrate --check with the new binary and the new configuration while the old version keeps serving. It never writes to the database.

Exit code	Meaning	What to do
0	The database is up to date.	Upgrade the nodes one at a time, draining each first.
3	Migrations are pending, or the database is empty.	Stop every node, then start one first. It migrates under the `schema-migration` lease, and the others wait for it.
4	The database is newer than this binary.	Stop. The binary is older than the schema.
1	Any other error, such as a broken configuration or an unreachable database.	Fix the error.

For each migration system with work left, it prints a pending: <system> line.

Code search on several nodes

With REPO_INDEXER_TYPE = zoekt, every node keeps a code search index of its own. REPO_INDEXER_PATH must be on a node-local disk, and startup refuses a path on the same filesystem as the repository root. The node that receives a push indexes it at once, and the other nodes index the new default branch commit within a minute. A node with an empty index rebuilds it at startup.

Arbinel refuses bleve for code search, because each node would index only the pushes that it received. elasticsearch uses a shared index and needs no extra setup.

Two-node example

A minimal cluster has two Arbinel nodes, PostgreSQL, Redis, an S3-compatible store and a Typesense or Meilisearch server. Both nodes use the same file, with only NODE_ID changed.

[high_availability]
ENABLED = true
NODE_ID = node-a

[database]
DB_TYPE = postgres
HOST = postgres.internal:5432
NAME = arbinel
USER = arbinel
PASSWD = ...

[repository]
; NFS or CephFS mount, the same on both nodes
ROOT = /srv/git/repositories

[repository.upload]
; on the same mount as ROOT
TEMP_PATH = /srv/git/uploads

[cache]
ADAPTER = redis
HOST = redis://redis.internal:6379/0

[session]
PROVIDER = redis
PROVIDER_CONFIG = redis://redis.internal:6379/1

[queue]
TYPE = redis
CONN_STR = redis://redis.internal:6379/2

[indexer]
ISSUE_INDEXER_TYPE = typesense
ISSUE_INDEXER_CONN_STR = http://API_KEY@typesense.internal:8108

[storage]
STORAGE_TYPE = minio
MINIO_ENDPOINT = s3.internal:9000
MINIO_ACCESS_KEY_ID = ...
MINIO_SECRET_ACCESS_KEY = ...
MINIO_BUCKET = arbinel
MINIO_USE_SSL = true

[metrics]
ENABLED = true

A matching HAProxy backend:

backend arbinel
    option httpchk GET /api/healthz
    http-check expect status 200
    server node-a 10.0.0.11:3000 check
    server node-b 10.0.0.12:3000 check

The Arbinel repository has a Docker Compose test setup in contrib/ha that runs two nodes behind nginx. Its script stops the leader and checks that the other node takes over. It is intended for testing, not for production.

Limitations

You must set up Redis high availability yourself. Arbinel does not set up Sentinel.
There is no command to make a running node give up leadership while it keeps serving. Stopping the node hands it over within one RENEW_INTERVAL.
If a node crashes while it migrates the schema, the other nodes wait until its schema-migration lease expires, at most 2 minutes later.