PostgreSQL Container Lifecycle & SRE Automation Overhaul

The Problem: PreStop Hook Lifecycle Paradox & S6 Race Conditions

The root cause of our 15+ minute deployment blocks centered on a misunderstanding of how the Kubernetes container engine manages preStop hooks alongside the s6-overlay process supervisor. Because Kubernetes hooks are strictly blocking, the container engine entirely delays sending the global SIGTERM signal until the script completes execution.

# Orchestration configuration allowing a massive 50-minute deadlock window
spec:
  terminationGracePeriodSeconds: 3000
  lifecycle:
    preStop:
      exec:
        command: ["/bin/sh", "-c", "/scripts/finish.sh > /proc/1/fd/1"]

The Execution Failure Flow:

1. The Blocking Loop: The legacy finish.sh script looped checking database activity via pg_stat_activity. The moment transactions cleared or the internal timeout hit, it fired pg_ctl stop -m fast and instantly exited.

2. The S6 Super-Loop Race: As the script exited, the database began its internal shutdown. However, because the s6 process supervisor layer was still fully active, it detected that the core database PID dropped. Acting as designed to ensure high availability, s6 instantly spawned a brand-new instance of PostgreSQL.

3. The Signal Ghost: Simultaneously, the exited script unblocked Kubernetes, causing it to issue its global SIGTERM. This signal was intended for the dying database, but instead hit the newly spawned zombie instance. Because that new process was wrapped in a legacy login shell wrapper (su - postgres -c ...), it swallowed or misrouted the incoming SIGTERM.

The container would sit completely unresponsive in an infinite loop, entirely deaf to the orchestrator for up to 3,000 seconds (50 minutes) until hitting a hard infrastructure SIGKILL.

Cascading Failures: Un-trapped Sidecars & Standalone Data Risks

Because our platform hosts standalone PostgreSQL instances rather than replicated clusters, any container lifecycle failure carried an immense operational blast radius. Lacking a replica failover safety net, a clean database detachment was paramount to preventing data corruption.

1. Unresponsive Backup Sidecars

The system instability was further amplified by an un-trapped companion backup sidecar container running inside the same StatefulSet pod network.

# The blocking sidecar vulnerability loop
while true; do
    python3 /etc/services.d/backup/backup.py
    sleep 21600 # 6-hour un-interruptible sleep block
done

In Kubernetes, a Pod cannot fully decommission until every active container within its specification exits. Because the companion backup script relied on an un-interruptible 6-hour raw shell sleep, it completely ignored incoming container engine signals. Even if the database container managed to drop, the un-trapped sidecar kept the pod permanently wedged in a Terminating state.

2. The Dangerous Fallback: Force Deletion

Lacking visibility into this complex process loop, the standard engineering response to stalled rollouts was to manually issue a force-evict override command:

Bash

kubectl delete pod <POD_NAME> --grace-period=0 --force

This brute-force override completely bypassed PostgreSQL's graceful shutdown procedure. For standalone databases, this introduced an immediate risk of Write-Ahead Log (WAL) desynchronization and raw data corruption. Furthermore, it left the underlying container processes zombie-running on the host, triggering severe AWS EBS Multi-Attach Errors (already mounted storage). The storage volumes remained exclusively locked to old, decommissioned EC2 hosts, completely blocking new replacement pods from spinning up and entirely breaking customer sandboxes.

The Engineered Fix & Compliance Process Optimization

I resolved this systemic infrastructure bottleneck by introducing strict process synchronization within the container runtime and driving a major operational process shift across our SRE and Database teams.

# Part of the engineered fix in finish.sh to cleanly disarm the supervisor
# Proactively command S6 to down supervision before stopping the engine
s6-svc -o /run/s6/services/postgres
s6-svc -o /run/s6/services/backup

# Directly command immediate fast termination under strict timeout boundaries
exec su - postgres -c "${v_pg_ctl} -t ${shutdown_timeout_seconds} -D ${v_data} stop -m fast"

1. Core Technical Implementation Steps:

• Supervisor Interception: I overhauled the finish.sh hook to proactively issue s6-svc -o commands to both the database and backup service directories. This cleanly disarmed s6's automatic restart triggers before the database shutdown initiated, permanently ending the zombie process loop.

• Signal Trapping: I replaced the backup sidecar's blind 6-hour block with an active POSIX trap loop checking granular 1-second ticks, ensuring it cleanly shuts down within seconds of a rollout notice.

• Flattened Execution Trees: I stripped the shell layer wrapping out of the core image configuration via exec s6-setuidgid, giving the orchestrator clean, unobstructed access to signal the primary database PID.

2. Driving Organizational Process Change

Beyond fixing the underlying code, I recognized that bouncing these standalone databases twice a month (once for the image update and once for the host AMI migration) was introducing unnecessary operational risk and toll.

I designed and guided a new cross-functional deployment workflow that consolidated these two separate maintenance windows into a single, cohesive lifecycle event:

1. Targeted Node Provisioning: Spin up fresh Kubernetes worker nodes pre-baked with the monthly, vulnerability-patched compliance AMI.

2. Node Tainting: Apply strict taints to the legacy worker nodes running the old AMI to prevent new workloads from scheduling.

3. Consolidated Rollout: Execute a targeted PostgreSQL application rollout. As Kubernetes terminated the old database pods, they natively migrated to the pre-provisioned, compliant nodes—upgrading both the database application image and the underlying host infrastructure in a single bounce.

Quantifiable Business Impact

This engineering and process intervention stabilized 200+ production standalone database pods serving 30–40 multi-tenant environments.

By ensuring that standalone engines could exit flawlessly within seconds and optimizing how our teams orchestrated infrastructure lifecycles, this solution completely eliminated data corruption risks, wiped out AWS volume attachment deadlocks, and successfully automated our mandatory FedRAMP High compliance rebuilds.

---