Kamal deployment with Litestream

ServiceStack VS
1

Kamal deployment is fantastic, however I’m attempting to add Litestream but I’m unsure if I’m doing the right thing.

I’ve added litestream.yml to the config/ folder and I have uncommented all the litestream lines at the bottom of deploy.yml

I have added the necessary secrets to GitHub (R2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEY) for the R2 token.

However when the GitHub actions run, litestream doesn’t seem to get deployed. I’ve tried on a fresh server and existing one and still nothing.

Have I missed something obvious? Thanks for all your support.

2

Here are the litestream + kamal configs we’re using for AI Server, pvq.app and blazordiffusion.com:

3

Strange, they’re the repos I used for a guide.

Files are identical (apart from the project name being different) and nothing is happening. Do I have to install/run any particular commands on the server I’m deploying to?

4

Shouldn’t have to, after the one time setup of SSH keys kamal should be able to remotely manage deployments (e.g. from GitHub Actions or your local repo).

Does anything show up in the logs?

5

I can’t see anything in the logs, I’m assuming it would be in the ‘release’ action? The keys get added, there’s a warning that they’re not github keys.

6

BTW here are kamal docs for managing accessories:
kamal-deploy.org/docs/commands/accessory/

E.g. you can view logs with:

$ kamal accessory logs litestream

and reboot the litestream docker container with:

$ kamal accessory reboot litestream
7

You can also run kamal commands from your local GitHub repo.

E.g. you can view App logs with:

$ kamal app logs -f

and litestream accessory logs with:

$ kamal accessory logs litestream -f

view running docker containers with:

$ kamal details

kamal-deploy.org/docs has lots of different commands to manage and inspect docker commands.

8

Ok, I wiped the deployment server and tried again.

On first deployment I get a lot of errors in the “Check if first run and execute kamal app boot if necessary” step:

ERROR Failed to boot web on ***
(https://github.com/[REDACTED]/actions/runs/14197182373/job/39783107389#step:11:32) ERROR 2025-04-01T15:25:55.267082903Z fail: ServiceStack.ServiceStackHost[0]
(https://github.com/[REDACTED]/actions/runs/14197182373/job/39783107389#step:11:33)2025-04-01T15:25:55.267203603Z RunManagedAction System.Action`1[ServiceStack.ServiceStackHost]:
(https://github.com/[REDACTED]/actions/runs/14197182373/job/39783107389#step:11:34)2025-04-01T15:25:55.267208002Z Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 14: 'unable to open database file'.

An awful lot of these errors on this step. It then runs the “Migration” step which creates the database and the app then successfully runs.

Still no sign of Litestream though in the logs.

9

Are you running fix permissions before migrations?

10

litestream is run as an accessory Docker container, it wont be in the GitHub Action logs, check the Docker App logs, see my example commands above.

11 
$ kamal accessory logs litestream

INFO [8bc2eb2d] Running **docker logs [APPNAME]-litestream --tail 100 --timestamps 2>&1** on [REDACTED]
ERROR (SSHKit::Command::Failed): Exception while executing on host [REDACTED]: docker exit status: 1
docker stdout: Error response from daemon: No such container: [REDACTED]-litestream
docker stderr: Nothing written
12

Yes, that stage runs successfully.

13

The container isn’t running?

See kamal accessory docs for different ways of starting the container.

14 
$ kamal details
  INFO [b080cbc3] Running docker ps --filter name=^kamal-proxy$ on [REDACTED]
  INFO [b080cbc3] Finished in 5.610 seconds with exit status 0 (successful).
Proxy Host: [REDACTED]
CONTAINER ID   IMAGE                         COMMAND             CREATED          STATUS          PORTS                                                                          NAMES
fa5f8fdb05c7   basecamp/kamal-proxy:v0.8.2   "kamal-proxy run"   42 minutes ago   Up 42 minutes   0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp   kamal-proxy

  INFO [e62a78a3] Running docker ps --filter label=service=[REDACTED] --filter label=destination= --filter label=role=web on [REDACTED]
  INFO [e62a78a3] Finished in 0.083 seconds with exit status 0 (successful).
App Host: [REDACTED]
CONTAINER ID   IMAGE                                  COMMAND                  CREATED          STATUS          PORTS      NAMES
ecb5529b26ae   ghcr.io/[REDACTED]:latest   "dotnet App.d…"   41 minutes ago   Up 41 minutes   8080/tcp   [REDACTED]-web-latest

  INFO [57996a4c] Running docker ps --filter label=service=[REDACTED]-litestream on [REDACTED]
  INFO [57996a4c] Finished in 0.085 seconds with exit status 0 (successful).
Accessory litestream Host: [REDACTED] 
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
15

Doesn’t look to be running. Let me check the startup commands.

16 
$ kamal accessory boot all
INFO [e82e77f2] Running **/usr/bin/env mkdir -p .kamal** on [REDACTED]
INFO [e82e77f2] Finished in 5.053 seconds with exit status 0 (**successful**).
Acquiring the deploy lock...
INFO [fa98cefe] Running **docker login ghcr.io -u [REDACTED] -p [REDACTED]** on [REDACTED]
Releasing the deploy lock...
ERROR (SSHKit::Command::Failed): Exception while executing on host [REDACTED]: docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: Cannot perform an interactive login from a non TTY device
17

I don’t know why it’s not running, there must be an issue preventing it from running. Try running litestream locally with your config to see if it works, then try running the litestream/litestream docker image with your config in /etc/litestream.yml with your secrets set as environment variables which is effectively all the litestream docker app is doing.

18

I thought I’d find the issue, the .github/workflows and .kamal/secrets file didn’t have R2_ACCESS_KEY_ID etc in them.

Sadly it hasn’t changed anything. Back to the drawing board. If I find out the issue, I’ll update here.

1 Like
19

Note Litestream expands environment variables in its configuration file, so you can just set your Environment variables in your Docker App or OS if you’re running it locally, which is how we use it:

access-key-id: $R2_ACCESS_KEY_ID
secret-access-key: $R2_SECRET_ACCESS_KEY
20

Thanks for the heads up.

I’m still trying all sorts to try and get it to auto-start with the github actions. I got it to start by adding a kamal accessory boot command into the release workflow, however it didn’t seem to actually do the replication so I assume that’s the issue.