---
name: repo-first-service-staging
description: Use when deploying or updating a homelab self-hosted service through a git-tracked staging repo first, then previewing and applying config into runtime paths like /opt, /etc/systemd/system, /etc/traefik/dynamic, and persistent service directories.
metadata:
  version: 1.2.0
---

# Repo-first staging for homelab self-hosted services

Use this when a service should be managed from a small git repo first, with live runtime files derived from that repo rather than edited ad hoc on the host.

This pattern is for boring homelab deployments: Docker Compose or similar runtime config, native systemd wrappers, persistent service data, and explicit review before apply. Exposing the service over a reverse proxy is a separate concern, handled by the `expose-private-service` skill.

## Files

- `scripts/sync-deploy.sh` — template preview/`--apply` sync script. Copy it
  into the service repo and edit the `SYNC_PATHS` list at the top; used in
  Step 3 below.

## When to use this

Use this skill when:
- a service should live under a repo like `~/dev/personal/<service>-stack`;
- tracked config should be reviewed before touching `/opt/<service>`, `/etc/systemd/system`, or `/etc/traefik/dynamic`;
- the host already has sibling services you can mirror;
- the user wants normal changes to go through staging first, with direct live edits reserved for debugging.

Do not use this when:
- the host already has a stronger deployment system (Ansible, Nix, Helm, etc.) the user wants to keep using;
- the service is intentionally ephemeral and not worth a repo;
- the task is only to hot-patch a live service during debugging.

## Default repo layout

Keep the repo small and literal:

```text
~/dev/personal/<service>-stack/
  docker-compose.yml            # or compose.yaml
  config/                       # tracked app config that is intentionally repo-owned
  scripts/sync-deploy.sh        # preview by default, apply with --apply
  scripts/backup.sh             # optional
  systemd/*.service
  systemd/*.timer
  traefik/<service>.yml
  README.md
  TODO.md
```

Do not add generators, templates, or abstraction layers unless the same repo already truly needs them.

## Ownership model

Treat every file as one of these three classes before writing anything:

1. **Tracked staged config**
   - lives in the repo;
   - safe to preview, review, and sync deterministically;
   - examples: compose files, systemd units, Traefik files, repo-owned `configuration.yaml`.

2. **Live persistent state**
   - never treated as disposable;
   - examples: databases, uploads, generated auth state, app-created registries, secrets, backup archives.

3. **Debug-only live edits**
   - emergency/manual host changes used only to diagnose or temporarily recover;
   - must be pulled back into the repo if they should survive.

If you cannot say which class a file belongs to, stop and decide before editing.

## Set a memory limit on every container, by default

An unbounded container can consume enough host memory to take down every
other service on the box, not just itself — this isn't hypothetical: a
single service doing a large bulk-import (many files processed in a burst)
grew its page cache faster than the kernel could reclaim it, drove the host
into a global out-of-memory condition, and the kernel's OOM killer couldn't
even resolve it — Docker/containerd carry a negative `oom_score_adj` by
default, so the killer skipped every container process and repeatedly killed
small, unrelated system daemons instead. None of the kills freed meaningful
memory. The host livelocked (blank screen, dead keyboard) until a hard power
cycle. A per-container memory limit turns "the whole host freezes" into "one
container gets OOM-killed and restarts" — it's the difference between an
incident and a log line.

Treat `mem_limit` (plain Compose; use `deploy.resources.limits.memory`
instead if the target actually runs under Swarm/a Compose spec that requires
it) as a required field on every service in a staged compose file, sized
like this:

1. **Baseline first, don't guess.** `docker stats --no-stream` against the
   running (or a comparable sibling) service to see actual idle memory
   usage. Sizing from a made-up number produces limits that either strangle
   the service immediately or do nothing.
2. **Size as a generous multiple of idle**, not a tight ceiling — the limit
   exists to cap runaway growth, not to pinch normal operation. A limit that
   gets hit during ordinary use just relocates the incident into the
   container instead of preventing it.
3. **Add extra headroom for bulk-input-processing services** — anything that
   ingests many external items in a burst and does real work per item:
   importers, OCR/document-consumption pipelines, link/page archivers
   (headless-browser rendering), ML/embedding batch jobs, notebook/dev
   environments that can load large datasets. This is the exact risk shape
   that caused the incident above — size these noticeably higher than a
   simple CRUD/API service with the same idle footprint.
4. **Small, stable services get small limits.** A metadata database, a cache,
   a task queue, a lightweight watcher — a few hundred MB to ~1GB is
   typically plenty; don't apply bulk-workload headroom where there's no
   bulk-workload risk.
5. **A memory limit is one layer, not the whole defense.** It bounds a single
   container's blast radius; it doesn't replace host-level OOM tooling
   (`systemd-oomd`/`earlyoom`) that reacts to memory pressure across the
   whole host — that's an OS-level concern outside this skill's scope
   (tracked config in a service repo), but worth having on any host running
   several unbounded-by-default containers.

Completion criterion: every service in the staged compose file has an
explicit memory limit, each one traceable to an idle-baseline measurement
plus a stated reason for its headroom (bulk-workload vs. small-and-stable).

## The workflow

### 1. Inspect the real host pattern first

Do not assume the wrapper shape.

Check sibling services already running on the host:
- systemd unit location (`/etc/systemd/system` vs user units);
- Docker Compose working directory;
- systemd `ExecStart` / `ExecStop` conventions;
- reverse-proxy file placement;
- backup mechanism (timer vs cron);
- runtime data location.

Completion criterion: you can name the exact pattern already used by at least one sibling service and the paths you plan to mirror.

### 2. Create the staging repo

Start with the smallest boring skeleton under `~/dev/personal/<service>-stack`.

Put only tracked deployment artifacts in git. If a live file is not meant to be repo-owned, do not create a tracked twin of it just because the directory exists.

Completion criterion: the repo contains only files you are willing to treat as source of truth.

### 3. Make preview the default, not the live path

`scripts/sync-deploy.sh` must be **non-mutating by default**. Adapt the
template at `scripts/sync-deploy.sh` in this skill: it walks a declared list
of (source, destination) pairs and prints the copy/install commands it would
run, only executing them when passed `--apply`.

Required contract:
- `./scripts/sync-deploy.sh` prints the exact commands it would run and
  **performs no writes at all** — every copy/install/restart command in the
  script must be gated by the same apply check, not just some of them. A
  script that copies unconditionally and only gates the restart is not a
  preview mode, it's a footgun with an announcement.
- `./scripts/sync-deploy.sh --apply` performs the writes **and restarts the
  service** — see "Restart, don't just copy" below. Copy-only is an
  incomplete deploy: the running service keeps serving its old config until
  something else restarts it, and that "something else" tends to be a human
  three days later wondering why a change didn't take.

The preview should show target paths clearly, especially for:
- `/opt/<service>`;
- `/etc/systemd/system/*.service` / `*.timer`;
- `/etc/traefik/dynamic/*.yml`;
- persistent config files that are intentionally repo-owned.

Completion criterion: a plain run changes nothing on disk and still shows the whole apply sequence, including the restart command.

### 3a. Restart, don't just copy

Every sync script needs an apply-time restart/reload step, not just a file
copy — this is a required part of the contract, not an optional nicety. Pick
whichever matches how the service actually runs:

- **Plain Docker Compose** (no systemd wrapper): `docker compose up -d` after
  the copy, in the service's runtime directory. Idempotent — a no-op
  recreate when nothing changed, a real recreate when the compose file
  changed (e.g. a new `mem_limit`, image tag, or env var).
- **systemd-wrapped Compose service**: use `systemctl reload-or-restart
  <unit>`, never `systemctl enable --now <unit>`. A `Type=oneshot` +
  `RemainAfterExit=true` unit (the usual shape for a Compose wrapper, with
  `ExecStart=docker compose up -d`) is already "active" once started, and
  `systemctl start` — which is exactly what `enable --now` does after the
  first deploy — is a **documented no-op on an already-active unit**. It will
  not re-run `ExecStart`. Concretely: the first-ever deploy works fine
  (inactive → active runs `ExecStart`), but every subsequent deploy silently
  stops applying anything to the running container — the new
  `docker-compose.yml`/config lands on disk and nothing picks it up. Give the
  unit an `ExecReload=` line (usually identical to `ExecStart=`) and call
  `reload-or-restart`, which runs `ExecReload` on an active unit and falls
  back to a full start on an inactive one — correct on both first deploy and
  every re-deploy after it.

Completion criterion: re-running `--apply` with a changed tracked file
actually changes the running container/service, not just the file on disk —
verify with `docker inspect`/`systemctl status` after a real config change,
not just "the script exited 0."

### 4. Separate config sync from state clobbering

If you add tracked files under `config/`, make ownership explicit file by file.

Good candidates for repo-owned sync:
- explicit app config the user wants reviewed in git;
- reverse-proxy/trusted-proxy/base-URL settings;
- deterministic include files.

Bad candidates for repo-owned sync:
- generated registries;
- runtime auth stores;
- sqlite/postgres/mysql data;
- uploads/media;
- app-created lock files;
- secrets unless the user explicitly wants a managed secret file path.

If a file is repo-owned but also likely to get manual edits later, document overwrite-vs-merge risk in `TODO.md` instead of pretending it does not exist.

Completion criterion: the sync script only overwrites files that have been deliberately declared repo-owned.

### 5. Validate staged artifacts before apply

Before any apply:
- render/validate Compose (`docker compose config`);
- syntax-check shell scripts (`bash -n`);
- verify systemd unit shape (`systemd-analyze verify`) where possible;
- compare the main wrapper against the real sibling service pattern;
- preview the full sync script once.

When verification errors mention future runtime paths that do not exist yet, distinguish real blockers from expected pre-deploy warnings.

Completion criterion: no unreviewed blocker remains, and preview output matches the intended mutation set.

### 6. Apply explicitly

Only after approval:
- create required runtime directories;
- run `./scripts/sync-deploy.sh --apply`;
- start/enable units or timers only in the apply path, never in preview;
- keep idempotence: a second apply should converge, not drift.

Completion criterion: rerunning apply produces either no meaningful change or the same deterministic end state.

### 7. Hand off exposure, don't reimplement it

If the service needs a private reverse-proxy + DNS route, that's not this
skill's job — use the `expose-private-service` skill for it (idempotent
DNS-record + Traefik-dynamic-config helper). This skill only owns getting
tracked config from repo to runtime; treat exposure as a separate step that
happens after the service is staged and running.

### 8. Verify the staged result end to end

Do not call an apply done from "the container is running" alone.

Verify separately:
- direct backend check on the host;
- reverse-proxy/access-path check, if the service is exposed;
- real client/browser access from the user's device, if applicable.

If the app works locally but fails through its access path, inspect logs for
app-layer issues such as `X-Forwarded-*`/trusted-proxy rejection, bad
host/base-URL validation, or redirect loops — before assuming the wrapper or
route is wrong. Example class of failure: the proxy returns the app
correctly, but the app itself sends `400` until trusted-proxy settings are
added.

Completion criterion: the user confirms the service works through its actual intended access path, not just on the host.

### 9. Document only into allowed locations

Before promising a project note, verify the target repo or vault rules.

Some knowledge stores treat project files as read-only except through a specific workflow. Confirm the destination is writable under current rules before saying you will write there automatically.

Completion criterion: documentation lands in a path that is both correct and allowed.

## Common pitfalls

1. **Assuming the wrapper shape.** Inspect sibling services first; do not silently switch from system-level units to user units.

2. **Making plain sync mutate the host.** Preview must be non-mutating by default; `--apply` is the only mutating path — including the restart step, not just the file copies.

3. **Copying config but never restarting.** A sync script that only copies files leaves the running service on its old config indefinitely — the restart/reload step is part of the required contract (see "Restart, don't just copy"), not an optional extra.

4. **Using `enable --now` to apply a change to an already-running systemd-wrapped service.** It's a no-op on an active `Type=oneshot`/`RemainAfterExit=true` unit — the deploy silently stops taking effect after the first run. Use `reload-or-restart` with an `ExecReload=` line instead.

5. **Leaving a container's memory unbounded.** Every service gets an explicit `mem_limit`, sized from an idle-usage baseline with extra headroom for bulk-input-processing workloads — see "Set a memory limit on every container." Skipping this turns one runaway container into a host-wide incident.

6. **Treating persistent state as tracked config.** Repo ownership must be explicit; do not sync databases, generated registries, or uploads from git.

7. **Ignoring overwrite-vs-merge risk.** If a staged config file may later receive manual edits, track the need for a safer merge helper.

8. **Stopping at container-up verification.** Running containers and enabled units do not prove the app works through its real access path.

9. **Reimplementing exposure here.** Route/DNS/reverse-proxy work belongs to the `expose-private-service` skill, not this one — don't duplicate it.

10. **Calling an apply done without client confirmation.** Host-side `curl` success is necessary, not sufficient when the service is meant to be reached remotely.

11. **Writing docs into governed paths without checking the rules.** Some project/vault trees are intentionally constrained.

## Verification checklist

- [ ] Sibling host pattern inspected and named explicitly
- [ ] Repo layout is minimal and literal
- [ ] Every tracked file has an explicit ownership class
- [ ] Every service has an explicit `mem_limit`, sized from an idle-usage baseline
- [ ] `scripts/sync-deploy.sh` previews by default and mutates only with `--apply`, including the restart step
- [ ] Restart step matches the runtime shape (`docker compose up -d` for plain Compose, `reload-or-restart` — never `enable --now` — for a systemd-wrapped unit)
- [ ] Compose and shell syntax validation passed
- [ ] Systemd unit shape matches the host's existing pattern, and defines `ExecReload=` if wrapping Compose
- [ ] Apply path is idempotent — re-running `--apply` after a real config change actually updates the running container/service, not just the file on disk
- [ ] Direct backend access verified
- [ ] Reverse-proxy/access-path verified, if the service is exposed (via the `expose-private-service` skill)
- [ ] Real client/browser verification completed, if applicable
- [ ] Documentation target validated before writing

## One-shot recipe

For a new service rollout:
1. Inspect sibling service units, runtime directories, and route files.
2. Create `~/dev/personal/<service>-stack` with only the tracked source-of-truth files.
3. Give every service in the compose file an explicit `mem_limit`, sized from a `docker stats` idle baseline plus headroom (more for bulk-input-processing workloads, less for small/stable ones).
4. Write `scripts/sync-deploy.sh` so preview is default, `--apply` performs the copies, and `--apply` also restarts/reloads the service (`docker compose up -d`, or `reload-or-restart` for a systemd-wrapped unit — never `enable --now` on a re-deploy).
5. Validate staged files (`docker compose config`, `bash -n`, unit shape checks, preview run).
6. Apply only after approval.
7. If exposure is needed, hand off to the `expose-private-service` skill rather than reimplementing DNS/reverse-proxy steps here.
8. Confirm backend and, if applicable, proxy/access-path and real client/browser behavior separately.
9. Verify idempotency explicitly: change a tracked file, re-run `--apply`, confirm the running container/service actually picked it up.
10. Record overwrite/merge and documentation-governance follow-ups in `TODO.md` rather than hiding them.