Running in production

Everything you need once sites are live: the service and its commands, logs and metrics, backups and updates, and what to do when something does not work.

The service#

Arka runs as the systemd service arka:

sudo systemctl status arka         # running? since when? last messages
sudo systemctl reload arka         # re-read every host file and certificate (no connection dropped)
sudo systemctl restart arka        # needed after changing arka.toml
journalctl -u arka -f              # Arka's own messages, live
journalctl -u arka --since "1 hour ago" -p warning

Before each start, systemd runs arka check, so a broken arka.toml is reported instead of half-starting. If Arka stops unexpectedly, systemd starts it again two seconds later.

Inside Arka, the work that runs for as long as it does — the control panel, the access-log writer, the certificate renewer — is started again on its own if it ever fails, after a few seconds and then longer pauses while it keeps failing; the log says so. Arka's own state — panel accounts, certificates and their keys, bans, settings and host files written from the panel — is written to a new file, flushed to disk and only then put in place, so a crash, a power cut or a full disk leaves the old version, never a half-written one.

The command line#

Run commands as the arka user, pointing at the configuration:

sudo -u arka arka --config /etc/arka/arka.toml check

(Tip: alias arka='sudo -u arka /usr/bin/arka --config /etc/arka/arka.toml' in your shell profile.)

Command	What it does
`arka check`	Validate `arka.toml` and every host, stream and snippet file. Prints the file, line and problem — and warnings worth fixing, such as PROXY protocol ports without trusted senders.
`arka reload`	Re-read every host file and certificate (same as `systemctl reload arka`, `SIGHUP`).
`arka reopen`	Reopen log files after another tool rotated them (`SIGUSR1`) — only needed with `[logs] rotate = false`.
`arka upgrade`	Replace the running server with the binary on disk without refusing a connection (`SIGUSR2`).
`arka stop`	Finish requests in flight and exit (`SIGTERM`). Use `systemctl stop arka` instead under systemd.
`arka token`	Print the control panel's access token. `--rotate` replaces it, and the running server switches at once.
`arka passwd NAME`	Hash a password for `auth` users; prints `NAME:$2b$…`.
`arka sign-url HOST PATH --expires 7d`	Make a signed, expiring link for a route with `signed` links.
`arka new-host ID --domain NAME --root DIR`	Write a new host file for a static site (or `--proxy HOST:PORT` for an application).
`arka init [DIR]`	Create a ready-to-run configuration with a welcome site (for trying Arka outside the package).
`arka backup [--out FILE] [--keep N]`	Archive the configuration and state — see Backups.

How changes are applied#

You…	Arka…
save a host file	rebuilds only that host in about 2 ms; caches and connection pools of the others stay warm.
save a snippet	rebuilds every host that includes it, and only those.
save a broken file	keeps serving the host's last good version — also across restarts — and reports the error with its line in the panel and the log.
give two hosts the same name	lets the host that was already serving keep it, and rejects the newcomer.
add a port	opens it on every worker; removed ports close without cutting requests in flight.
change `arka.toml`	uses it after `systemctl restart arka` (about a second), or `arka upgrade` (no interruption).
renew a certificate	swaps it into the running host — nothing to do.

Every version of every host file is kept; the panel shows the history with diffs and restores any version in one click. A host or stream deleted from the panel waits in the trash (.trash/ beside the files): Hosts → Deleted brings it back, checked first as if it were new.

Logs#

Log	Where
Access log of each host	`/var/log/arka/<host id>.access.log`
The panel's audit log (who changed what, failed sign-ins)	`/var/log/arka/admin-audit.log`
Arka's own messages (start, errors, certificates, warnings)	`journalctl -u arka`, and the latest 1,000 lines under Server → Server log in the panel

Access logs use the familiar combined format by default, so tools such as GoAccess, AWStats and fail2ban read them as they read nginx's. Per host, choose format = "json" (with the upstream, its timings, cache result, request id, country…) or your own line:

[log]
format = "custom"
template = "{time_iso} {client_ip} {method} {uri} {status} {duration_ms}ms {upstream_addr} {cache}"

Logs can also go to syslog (syslog = { address = "udp://10.0.0.5:514" }), or only there (file = false). Logging happens off the request path: if the disk cannot keep up, lines are dropped and counted (arka_log_records_dropped_total), never slowing visitors down. On very busy static sites, turning a host's log off saves about 14% CPU.

Rotation. Arka rotates its logs itself — access logs at 100 MB or once a day, the audit log by size only — keeps 14 rotated files of each, and compresses all but the newest (site.access.log-20261029-060000.gz, times in UTC). A log in a folder Arka cannot rename in (a file you created in /var/log) keeps being written, with one warning an hour. To have logrotate do it instead, set rotate = false and have it run arka reopen.

# /etc/arka/arka.toml
[logs]
max_size = "100MB"       # rotate at this size…
every = "1d"             # …and at least daily (unless empty)
keep = 14
compress = true
disk_reserve = "1GB"     # free space kept for certificates and settings

A full disk. Arka checks every 30 seconds how much room the disks it writes to have. Below disk_reserve (at most a tenth of the disk), access logs pause — certificates, panel accounts and settings can still be saved — until there is twice as much again; the panel's Needs attention shows the disk, and an alert goes out.

Alerts#

Arka can send a message when something needs a person — a JSON POST to a webhook, signed when you give it a secret:

# /etc/arka/arka.toml — or Server configuration → Alerts in the panel
[alerts]
webhook = "https://hooks.example.com/arka"
secret = "a long random string"     # X-Arka-Signature: sha256=<HMAC of the body>
# events = ["certificate", "upstream"]   # all of them when unset

Event	When
`certificate.failed`, `certificate.renewed`	a certificate cannot be obtained or renewed (the first time, then daily), and when it works again
`certificate.expiring`	one expires within 14, 7, 3 or 1 days, or has expired
`upstream.down`, `upstream.up`	an upstream or PHP server fails its health checks, or passes them again
`disk.low`, `disk.ok`	a disk is almost full and access logs pause, or has room again
`firewall.spike`, `firewall.bans`	a site's firewall suddenly refuses many requests, or many addresses are banned at once

Each alert carries event, level, host, message, server and time, and a one-line text that chat webhooks show as is (Slack-compatible ones; Discord's with /slack after the URL). A receiver that cannot be reached gets two more tries; a flood is cut to 20 alerts a minute. Send a test alert in the panel shows it works.

Metrics and monitoring#

The panel's port serves Prometheus metrics at /metrics and a health check at /healthz (no token needed; 200 while Arka runs). /readyz answers 503 while a server is handing over or shutting down, so load balancers stop sending it traffic first.

# prometheus.yml
scrape_configs:
  - job_name: arka
    scheme: https
    tls_config: { insecure_skip_verify: true }     # the panel's self-signed certificate
    authorization: { credentials_file: /etc/prometheus/arka-token }
    static_configs:
      - targets: ["10.0.0.1:9898"]

Or set public_metrics = true under [admin] to allow scraping without the token (only from addresses allow lets in). The most useful series:

Metric	Tells you
`arka_http_requests_total`, `arka_http_request_duration_seconds`	traffic and response times, by host and status class
`arka_http_events_total{event=…}`	`waf_blocked`, `rate_limited`, `banned`, `auth_failed`, `cache_hit`, `cache_miss`, `upstream_error`, `queued`…
`arka_upstream_up`, `arka_upstream_in_flight`, `arka_upstream_failures_total`	health and load of each application server
`arka_connections_active`, `arka_tls_handshakes_total`, `arka_connections_shed_total`	connections, and idle ones closed under pressure
`arka_certificate_expiry_timestamp_seconds`, `arka_certificate_renewal_failing`	when each certificate expires, and whether renewing it keeps failing
`arka_host_serving_previous`	hosts whose current file is broken (serving their last good version)
`arka_cache_bytes`, `arka_cache_entries`	response cache usage
`arka_worker_stall_seconds`	a worker blocked by something — milliseconds are normal, a second is not
`arka_disk_free_bytes`, `arka_disk_low`, `arka_access_log_paused`	room on the disks Arka writes to, and whether logs are paused for want of it

Useful alerts: arka_upstream_up == 0, arka_host_serving_previous > 0, a rising rate of 5xx, arka_certificate_renewal_failing == 1, and arka_certificate_expiry_timestamp_seconds - time() < 14 * 86400 (the panel's Needs attention shows certificates close to expiry too). For distributed tracing, Arka sends OpenTelemetry spans to Jaeger, Tempo or any OTLP collector — see [tracing].

Backups#

Two folders hold everything: /etc/arka (configuration, the token) and /var/lib/arka (certificates and their keys, panel accounts, file history, bans). Logs are not needed to restore a server.

arka backup writes one archive (readable only by its owner, since it holds private keys) to /var/lib/arka/backups/, and the package runs it every day, keeping the newest seven. Server → Backups in the panel lists them, makes one, downloads one and uploads one made elsewhere:

sudo -u arka arka --config /etc/arka/arka.toml backup --keep 7

An archive holds config/ and state/ and a MANIFEST naming where each came from; logs, caches and downloaded updates are left out.

Copy the archives off the server — a backup on the same disk does not survive the disk.

Restoring is Restore next to an archive in the panel. Arka checks it first — only plain files and folders, nothing outside its own tree, an arka.toml that loads — archives what is there now (arka-before-restore-…), puts each part where this server keeps it (a backup from another server works too), and restarts without dropping a connection. Sites, certificates and accounts come back as they were; you may need to sign in again. On a new server: install Arka, sign in with the token, upload the archive under Server → Backups, and restore it.

Updates and rollback#

New versions appear under Server → Updates with their release notes; Install verifies the signature and installs the package. See Install → Updating for the details and for updating by hand.

Rolling back. The update helper keeps the package it installed and the one before it. Roll back under Server → Updates checks the previous package's signature again and reinstalls it (rolling back twice rolls forward again). Without the panel — or to go back further — download the version you want and install it explicitly:

curl -fL -o arka-old.deb https://upgrades.vulntech.com/v1/arka/download/stable/<version>/x86_64-linux
sudo apt install --allow-downgrades ./arka-old.deb

Configuration written for a newer version can contain settings an older one does not know; arka check names them.

The panel under its own name#

Browsers trust the panel, and passkeys work, once it has a domain name and a real certificate. Point a name at the server (panel.example.com), give it a host so Arka gets a certificate for it, and let the panel use that certificate:

# /etc/arka/hosts/panel.example.com.toml
server_names = ["panel.example.com"]

[tls]
acme = true
redirect_http = true

[[routes]]
path = "/"
handler = { type = "redirect", to = "https://panel.example.com:9898{uri}" }

# /etc/arka/arka.toml, under [admin]
tls = { cert = "/var/lib/arka/acme/certs/panel.example.com.pem", key = "/var/lib/arka/acme/certs/panel.example.com.key" }

Restart Arka. Until the first certificate arrives the panel keeps its self-signed one; it switches within a minute of the files appearing and follows every renewal.

Several servers: clustering#

Several Arka servers can work as one: the primary's hosts, snippets, accounts and certificates are copied to the replicas within seconds; bans, cache purges, upstream drains and (optionally) rate-limit counts are shared. Put a load balancer or DNS round robin in front. On replicas the panel is read-only. All it takes is the same secret on every node — see [cluster].

Tuning#

Arka's defaults suit most servers. When a server gets busy:

Workers default to one per CPU core (workers = "auto") — leave it.
Open files: the service may open a million files and connections (LimitNOFILE). max_connections under [server] (16,384 per worker in the package's configuration) makes a flood run out of slots before the machine runs out of memory.
Compression is on by default and compresses static files once; leave it on.
Cache application responses, even for a few seconds, where pages are the same for everyone (recipe).
HTTP/3 (http3 = true under [tls], and UDP port 443 open) helps visitors on mobile networks with packet loss. All of a port's HTTP/3 packets arrive through one UDP socket, so Arka asks the kernel for 8 MB buffers; the package raises the system's limit (net.core.rmem_max) on install, and Arka's log warns when it is lower. HTTP/3 gets half the workers' threads (http3_threads changes that).
Connection floods are handled without refusing ordinary visitors: past a pressure point worked out from memory, idle keep-alive connections of the clients holding the most are closed first (shed_idle_connections).
Access logs cost CPU on very busy static hosts; turn them off where nobody reads them.
Pin workers to cores on a machine dedicated to Arka with cpu_affinity = true.

See Performance for measurements.

File permissions#

Arka runs as the user arka, so:

Site files must be readable by arka — world-readable files (644) and folders (755) are the usual way. Files in home directories are not reachable: the sandbox hides /home; put sites under /var/www or /srv.
Arka writes only to /etc/arka, /var/lib/arka and /var/log/arka. For a WebDAV share or a disk cache elsewhere, allow the folder (see below).
Unix sockets of applications (php-fpm, gunicorn) must be usable by arka: add arka to the socket's group (sudo usermod -aG www-data arka, then restart Arka), or make the application create its socket with group arka.
Symbolic links inside a site's folder work; links that lead out of it answer 404 unless the route says follow_symlinks = true.

To let Arka write to another folder, run sudo systemctl edit arka, add the lines below, and restart Arka:

[Service]
ReadWritePaths=/srv/share /var/cache/arka

Troubleshooting#

Symptom	Likely cause and fix
Arka does not start: address already in use	Another web server holds port 80 or 443: `sudo ss -ltnp '( sport = :80 or sport = :443 )'`, then stop it.
No certificate, browser shows a self-signed one	The name does not point at this server yet, port 80 is blocked, or Let's Encrypt's rate limit was hit. The Certificates page and Arka's log (`journalctl -u arka`) say which.
`502 Bad Gateway`	The application is down, listens on another port, or its socket is not readable by `arka`. The host's page shows each upstream's health and last error.
`504 Gateway Timeout`	The application took longer than `timeout` (60 s) to answer. Raise it on the route, or make the request faster.
`403` on a normal request	Probably the firewall. The Firewall page names the rule — add an exception. Or an `[access]`/`[geo]` rule, or a ban.
`404` for a file that exists	Wrong `root` (the request path is appended to it — use `strip_prefix` for nginx's `alias`), a file `arka` cannot read, a dotfile, or a link out of the root.
`413` on uploads	The body is larger than `max_body_size` (16 MB by default) under `[limits]`. On a large form or JSON post, the firewall may be refusing a body larger than it inspects — the Firewall page then shows `protocol.body-too-large`: raise `max_body` under `[waf]`.
PHP pages hang now and then	Set `max_conns` on the PHP route to php-fpm's `pm.max_children`, with a `queue` (why), and give php-fpm enough workers.
An upstream's name stops resolving	Arka keeps using its last addresses for up to an hour and logs cannot look up upstream; fix the DNS (or use an IP address) before then.
A change does not show	`arka check` — the file may be broken (the site then keeps its last good version). In the panel the host shows amber with the error.
The panel does not open	Port 9898 closed in the firewall (intended), your address missing from `allow`, or you were locked out after ten failed sign-ins (wait fifteen minutes).
Too many open files in the log	Raise `LimitNOFILE` with `systemctl edit arka`, or lower `max_connections`.

Still stuck? journalctl -u arka -n 200 and the request's line in the Logs page usually tell the story. Write to contact@vulntech.com with both.

Questions#

Does Arka read .htaccess files? No. Their rules go into the host file — see Moving from nginx or Apache. This is also why Arka never has to look for them on every request.

Can Arka run next to nginx or Apache? Yes, on different ports — for example Arka on 80/443 in front, passing some sites to Apache on 127.0.0.1:8080 while you migrate.

Which systems does it run on? 64-bit x86 Linux with glibc 2.17 or newer; packages are made for Ubuntu and Debian. Not (yet) ARM, Windows or macOS.

Is it free? Yes, including for commercial use — see the License Agreement.