Technical Issues Resolved

Symptom

CI job containers (via act-runner) failed to git clone or push Docker images, throwing what looked like an authentication error against the internal Gitea service.

remote: Invalid username or password.
fatal: Authentication failed for 'http://git.khalilaliouich.com/...'
curl: (7) Failed to connect to gitea-http.gitea.svc.cluster.local port 3000

Discarded Hypotheses

The error read Invalid username or password, so we first regenerated the Gitea token — no change. Then suspected DNS resolution of .svc.cluster.local and patched /etc/hosts in the runner — the timeouts softened but never fully cleared, which was the tell that DNS wasn't the real story.

Root Cause

The standard Docker bridge network created by act-runner suffered from MTU fragmentation and NAT translation issues when communicating with K3s Pod IPs and Services on this specific Oracle Cloud virtualized network — the auth-shaped error was just how the connection drop surfaced.

Fix

We configured the Gitea act-runner to force all CI job containers onto the host's network namespace, letting ephemeral CI containers resolve .svc.cluster.local domains directly, without NAT overhead.

# /data/gitea-runner/config.yaml
container:
  network: "host"
  options: "--add-host=gitea-http.gitea.svc.cluster.local:10.43.0.10"

Symptom

ArgoCD became completely inaccessible, returning 502 Bad Gateway. The argocd-server logs were filled with TLS handshake failures.

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed"

Discarded Hypotheses

A 502 usually means a routing problem, so we first re-checked the Gateway/HTTPRoute config and the certificate — both correct. Then suspected argocd-server itself was crash-looping — but every ArgoCD pod was Running and Ready.

Root Cause

Installing the Linkerd service mesh had globally auto-injected sidecars into the ArgoCD namespace. Linkerd aggressively intercepts gRPC traffic, and ArgoCD relies heavily on internal gRPC between its server, repo-server, and application-controller — the proxy was breaking their TLS handshakes.

Fix

We disabled Linkerd proxy injection specifically for the ArgoCD namespace and recreated the pods to restore internal communication.

kubectl annotate namespace argocd linkerd.io/inject=disabled --overwrite
kubectl delete pods --all -n argocd

Symptom

Right after a successful image build, the pipeline failed at the manifest-update step with Invalid username or password. fatal: Authentication failed.

Discarded Hypotheses

We first assumed the token had expired and regenerated it — same failure. Then suspected a branch-protection rule blocking the push — none was configured on that repository.

Root Cause

The default token injected by Gitea Actions was insufficient for pushing back to the repository from within that specific job's context over HTTPS — a scope problem, not an expiry or policy one.

Fix

We modified .gitea/workflows/deploy.yaml to inject a dedicated access token directly into the remote URL before executing the push.

# Inside the CI/CD Pipeline step:
git remote set-url origin "http://oauth2:${{ secrets.GITEA_TOKEN }}@git.khalilaliouich.com/khalil/tamagotchi-service.git"
git push origin HEAD:main

Symptom

ArgoCD reported the new manifest as Synced, but the running pods kept serving an old image — no error, just stale behavior.

Discarded Hypotheses

We first assumed ArgoCD hadn't actually synced and re-triggered it manually — no change. Then diffed the image tag in the manifest against what was built — they matched exactly.

Root Cause

The registry was configured as the internal service gitea-http.gitea.svc.cluster.local:3000, which K3s's containerd resolved and authenticated inconsistently. With imagePullPolicy: IfNotPresent, once any pull had ever succeeded, later pulls silently kept the cached layer instead of failing loudly.

Fix

We switched the registry target to the external, reliably resolvable domain and set imagePullPolicy: Always to force strict layer validation on every deploy.

# k8s.yaml
spec:
  containers:
    - name: api
      image: git.khalilaliouich.com/khalil/tamagotchi-api:v2
      imagePullPolicy: Always

Symptom

The showcase site advertised guest / a demo password for ArgoCD, but every login attempt was rejected.

Discarded Hypotheses

We first suspected the argocd-rbac-cm policy mapping itself — read it line by line, syntax was correct. Then checked whether the guest account was even enabled — it was.

Root Cause

RBAC and the account definition were both fine; the bcrypt password hash for accounts.guest.password in the separate argocd-secret object was missing.

Fix

We generated a bcrypt hash manually via Python, base64-encoded it, and patched argocd-secret directly.

BCRYPT_HASH=$(python3 -c "import bcrypt; print(bcrypt.hashpw(b'<YOUR_GUEST_PASSWORD>', bcrypt.gensalt()).decode())")
BASE64_HASH=$(echo -n "$BCRYPT_HASH" | base64 -w 0)

kubectl patch secret argocd-secret -n argocd -p '{"data": {"accounts.guest.password": "'$BASE64_HASH'"}}'

Symptom

A routine CSS update brought down the entire site with 502 Bad Gateway, including the live cluster metrics normally embedded on the homepage.

Discarded Hypotheses

Since the change that triggered it was CSS, we first suspected the content itself — but a 502 happens before any HTML is served, which ruled that out as soon as we actually looked at where the error originated.

Root Cause

The build had picked up an older Nginx-based Dockerfile (listening on port 80) instead of the Node.js one, while the Kubernetes Service still targeted port 3000 — a silent mismatch between what got built and what the Service expected.

Fix

We restored the Node.js server.js proxy architecture, rebuilt via nerdctl, and rolled out the corrected image.

sudo nerdctl build -t showcase-website:latest .
kubectl set image deployment/showcase-website website=showcase-website:latest -n showcase
kubectl rollout restart deployment/showcase-website -n showcase

Symptom

Dashboards, users, and settings created by hand in the Grafana UI vanished on every pod restart. Separately, stray Ingress objects from Helm chart defaults were still routing traffic outside the intended Gateway API path.

Discarded Hypotheses

We first assumed a Helm values change had been silently reverted — reviewed release history, values were consistent across upgrades. Then suspected the dashboards-as-code sidecar (the same grafana_dashboard: "1" ConfigMap pattern used for Tamagotchi and Loki) was overwriting manually-created dashboards — but the provisioned ones always survived restarts fine; only the hand-made ones vanished, which pointed at storage, not provisioning.

Root Cause

Grafana's chart defaults to an ephemeral emptyDir volume: every pod restart started from a blank data.db, wiping anything not defined as versioned config.

Fix

We deployed a dedicated hostPath PersistentVolume and PVC for Grafana and pointed the Helm release at it (helm upgrade --reuse-values), then separately deleted the ghost Ingress objects to fully hand routing to HTTPRoute.

Symptom

Switching language on the site sometimes rendered raw keys like issues_title instead of actual text.

Discarded Hypotheses

The name we gave the bug shaped the first hypothesis: a load-order race between setLang() and the dictionaries being defined. We added a DOMContentLoaded guard — the affected strings still broke. Then suspected stale cached JS — hard-refreshed, same result.

Root Cause

There was no timing bug at all. The keys were simply never defined in one or both dictionaries for that section of the page, so setLang() had nothing to find and fell back to printing the raw key.

Fix

We added the missing keys to both the en and fr dictionaries in app.js.

Symptom

After rebuilding the website image locally to ship a frontend change, the deployment failed with ErrImageNeverPull — as if the image had never been built at all.

Discarded Hypotheses

We first assumed the build itself had silently failed — reran it, watched it complete, confirmed the tag existed locally. Then diffed the build tag against the Deployment's image: field — they matched exactly.

Root Cause

nerdctl build had run against the default containerd socket and namespace. K3s runs its own isolated containerd, at /run/k3s/containerd/containerd.sock in the k8s.io namespace — the kubelet simply couldn't see an image built anywhere else.

Fix

We rebuilt the image directly into K3s's containerd, using the isolated socket and namespace explicitly.

sudo nerdctl --address /run/k3s/containerd/containerd.sock \
  --namespace k8s.io build -t showcase-website:v39 .

Symptom

Automated git commit calls from a headless terminal session hung indefinitely — no error, no visible prompt, until the surrounding task eventually timed out.

Discarded Hypotheses

We first suspected a network hang — a pre-commit hook reaching out somewhere. There were no hooks installed. Then checked whether the process was deadlocked or spinning — it was idle, just blocked waiting on input.

Root Cause

commit.gpgsign=true was set globally. Git was waiting on a GPG passphrase prompt that had nowhere to render in that headless context.

Fix

Documented as --no-gpg-sign per commit, to unblock the immediate incident.

git commit --no-gpg-sign -m "fix: ..."