That conversation is usually the first sign that an organisation bought two Oracle products without deciding what each one was for. It is a common outcome, and it is not because the products are bad. It’s because they are sold as a suite, they overlap on purpose, and the sales conversation is about capability rather than authority.

This post is a working comparison of the four Oracle Construction and Engineering products people most often confuse: Primavera P6 vs Unifier vs Aconex, plus Textura. One profile each, what it genuinely owns, where it wins, where it doesn’t, and then the part nobody covers: how to decide which system holds the authoritative number when three of them offer a cost module.

The question that actually sorts them

Feature lists will not separate these products. All four manage documents. Three of them track cost. Two of them hold a schedule. If you compare capability columns you will conclude that you need all four, which is exactly the conclusion the suite is designed to produce.

The question that does separate them is: whose record is this, and who else needs to write to it?

Two of these products are single-organisation systems. You run them, you own the data, and outside parties get in only if you give them a seat and they accept your governance. One of them is deliberately multi-organisation, where every company on the project holds its own workspace and decides what it shares. And one is a transactional network that touches money leaving your bank account, which makes it a finance system wearing a construction badge.

Get that axis right and the rest of the decision mostly falls out.

Primavera P6: the schedule, and only the schedule

P6 is a critical path method scheduling engine with resource and cost loading on top. It exists to answer questions about time: what drives the finish date, what happens if this activity slips, what does the resource histogram look like if we accelerate.

Two things confuse people here. First, P6 comes in two shapes. P6 Professional is the desktop client that planners actually live in. P6 EPPM is the enterprise web layer with the shared database, portfolio views and web services. Most organisations run both against the same database, and most planners will fight you to keep the desktop client.

Second, Oracle Primavera Cloud is a separate product, not P6 in a browser. Oracle says so on its own product page, which is unusually direct for vendor copy. Primavera Cloud shares a lineage with P6 and is positioned as the cloud destination, but the data model is not identical and things planners rely on in P6 do not all have equivalents. If someone tells you a move to Primavera Cloud is a hosting change, they have not tried it.

Where P6 wins

• Contractual programme scheduling, where the schedule is a deliverable and gets audited
• Forensic delay work, because the XER and PMXML formats are what every expert and every third-party analysis tool expects
• Resource-loaded planning across a portfolio with a shared resource pool
• Anywhere the client’s contract names it, which is more places than you would expect

Where P6 doesn’t

• Cost control. P6 has cost fields and they are useful for cash flow shape, but a resource-loaded schedule is not a commitment ledger. It has no concept of a purchase order lifecycle, a variation approval or a payment certificate
• Correspondence and document control. There is nothing here for transmittals, RFIs or drawing registers
• Collaboration across companies. Subcontractors do not get P6 logins, and you would not want them writing to your programme if they did

Primavera Unifier: a business process engine that happens to know about projects

Unifier is the one people describe worst, usually as “the cost one”. That undersells it and also sets bad expectations.

Unifier is a configurable workflow platform for capital programmes. You define business processes: a change order, a funding request, an invoice, a payment application, a site instruction. Each one has a form, a workflow with approval steps and routing rules, and a defined effect on the cost sheet. Oracle ships preconfigured processes and a no-code designer called uDesigner for changing them or building your own. Alongside the business processes sit the managers, including the Cost Manager, Schedule Manager and Document Manager.

It’s sold as base products, principally Project Controls and Facilities and Asset Management, which matters because Unifier’s remit runs past handover into operating the asset.

The honest trade-off with Unifier is this: its configurability is the reason to buy it and the reason implementations run long. A platform that can model any approval workflow will not tell you what your approval workflow should be. If your organisation has not written its cost control process down, Unifier will faithfully automate the argument you are having instead of the process you wanted.

Where Unifier wins

• Owner-side capital programme control, where budget, funding, commitments, forecast and actuals need to roll up across many projects
• Anywhere approvals need an audit trail with defined routing, delegation and thresholds
• Public sector and utility programmes with fund tracking, where money has strings attached and the strings must be reportable
• Organisations that want one governed process applied to every project rather than per-project improvisation

Where Unifier doesn’t

• Supply chain collaboration. It’s your instance, your governance, your seats. External parties can be given access, but they are guests in your system and they behave like it
• Drawing and model control at scale. The Document Manager is real, but it is not a common data environment and it is not built for design coordination
• Fast starts. Configuration is a project in its own right, and treating it as an install is the most common way to burn a first year

Aconex: the project record, owned by nobody in particular

Aconex is a common data environment for correspondence, documents, workflows and models across every organisation on a project. Document control, transmittals, RFIs, submittals, drawing registers, model coordination, field inspections.

Its defining property is not a feature. It is the data ownership model. Every organisation on the project gets its own workspace and controls what it shares and when. That neutrality is the entire commercial argument, and it is the thing you cannot replicate by inviting subcontractors into your instance of something else.

This matters more than it sounds. A designer who believes the main contractor can read their internal drafts will keep their real work somewhere else, and your “single source of truth” quietly becomes a partial one. The reason Aconex accumulates a complete project record is that nobody is afraid of it. Oracle’s own materials lean on that neutrality hard, and in this case the marketing claim and the engineering consequence are the same thing.

On the model side, Aconex Model Coordination is design-neutral and built on open BIM standards including IFC and BCF, so it sits alongside the authoring tools rather than replacing them.

Where Aconex wins

• Multi-party projects where the correspondence record will be read by lawyers later
• Design-build and joint venture delivery, where no single party should hold everyone’s data
• Document and model volume, which Oracle’s datasheets describe as unbounded on data and participants
• Handover, because the audit trail and the linked document set are the deliverable

Where Aconex doesn’t

• Portfolio-level capital planning. Aconex is organised around projects, not around a programme of investment decisions
• Asset lifecycle after handover. The record ends where the project ends
• Deep enterprise cost governance. Connected Cost exists and is genuinely useful for project cost, but if your organisation needs funding, capital planning and multi-year rollup, that is Unifier’s shape

Textura: money leaving the building

Textura came into Oracle by acquisition and still feels different from the rest of the suite, because it is solving a different class of problem.

Textura Payment Management runs the construction payment cycle: initiating a draw, subcontractor pay applications, markups and approvals, owner billing, collecting lien waivers and compliance documents, enforcing compliance holds, payment authorisation and disbursement. It’s a network product. Your subcontractors transact in it, and the value comes from them being there.

The compliance side is the part that decides whether it’s worth buying. If your risk is a subcontractor’s insurance lapsing quietly and nobody noticing until a claim, an automated payment hold tied to a missing document is a control you cannot easily build with a spreadsheet and good intentions.

Where Textura wins

• General contractors with a wide subcontractor base and a monthly draw cycle that eats accounts payable alive
• Jurisdictions with real lien exposure, where waiver collection is a legal control rather than paperwork
• Compliance enforcement at the point of payment, which is the only moment anyone reliably pays attention
• Audit trails on who approved what, signed what and released what

Where Textura doesn’t

• It is not a cost control system. It handles the payment leg, not the budget, forecast or commitment ledger
• It does not replace your ERP. It integrates with it, and that integration is part of the implementation cost
• Adoption is a supply chain problem, not a software problem. If your subcontractors will not onboard, you own a slower process than the one you replaced

The overlap trap, and why the cost module is where it bites

Here’s the failure mode that is invisible at procurement and expensive eighteen months later.

Three of these products can hold cost. P6 has cost-loaded activities. Unifier has a full Cost Manager. Aconex has Connected Cost. Nobody sets out to run three cost systems, but it happens gradually: the schedule gets cost-loaded because the client asked for an S-curve, the finance team runs the commitment ledger in Unifier, and the project team starts logging variations in Aconex because that’s where the correspondence already is.

Now you have three forecasts. They disagree. Not because anyone is wrong, but because they are answering different questions with different cut-off dates and different accrual rules. And the reconciliation lands in a spreadsheet maintained by one person, which is the outcome every one of these products was bought to prevent.

The fix is a decision, not a tool. Before anything is configured, write down one sentence per data class: which system is authoritative, and what the others are allowed to hold.

• Schedule dates and logic: P6 is authoritative. Everything else receives dates, nothing else edits them
• Budget, commitments and forecast: pick one. If you own Unifier, it is Unifier. If you don’t, Connected Cost is a reasonable home, but say so out loud
• Correspondence and controlled documents: Aconex if it’s on the project, and if it is, do not run a parallel drawing register anywhere else
• Subcontractor payment and compliance status: Textura, with the resulting actuals flowing to whichever system owns cost
• Everything the authoritative system does not own: read-only, refreshed by integration, and clearly labelled in reports as a copy

That list takes an afternoon and saves a year. It’s also the single most common thing missing from Oracle Primavera implementations I get asked to look at.

Integration is the real cost, and it has a shape

Once you own two of these, you own an integration. Budget for it as a workstream, not a checkbox.

Oracle offers more than one path. Primavera Gateway is the established broker for P6 and Unifier, with built-in providers per application and a canonical intermediate format so mappings survive version changes. In the P6 to Unifier direction, Gateway maps the P6 Project ID to Unifier’s Project Number, and what flows depends on how the schedule is built: a duration-based schedule sends durations, a resource-loaded one sends durations and units. Oracle also has a lighter integration platform for exchanging data between Unifier, P6, Aconex and Primavera Cloud, so check which one your version and deployment model actually supports before designing anything.

Aconex Connected Cost takes a different route, talking to P6 over its web services API. Two constraints from that integration are worth knowing before you draw an architecture diagram:

• You can configure both a P6 and a Primavera Cloud integration for the organisation, but an individual project can only link to one of them at a time
• Self-hosted or third-party-hosted P6 servers have to be approved by Oracle and reachable over the internet before the integration can be established. Oracle-hosted P6 skips that approval step

That second one has teeth if you are running P6 EPPM on your own infrastructure. “Internet accessible” and “exposed” are not the same thing, and the gap between them is where your security review lives. If you’re standing up middleware or a reverse proxy in front of an on-premise P6 or Unifier deployment, that box needs to be treated as production: a dedicated host from a provider like Contabo or InterServer rather than a corner of an existing server, TLS terminated properly, access restricted to known networks, and remote administrative access over a business VPN such as NordLayer or Surfshark rather than a port forward someone opened for a consultant and forgot about.

Back up the database host properly too, and test a restore. On-premise P6 and Unifier estates are frequently the last thing anyone thinks about until they need it, and imaging tools from vendors like O&O Software earn their keep on exactly this kind of long-lived, rarely-touched server.

How I’d decide

Work through this in order and stop at the first answer that fits. Buying more than you need is far more common than buying too little.

1. Name the pain in one sentence. “We cannot defend our programme in a delay claim” points at P6. “We cannot tell the board what the capital programme will cost” points at Unifier. “We cannot find the approved revision” points at Aconex. “Subcontractor payments take six weeks and compliance is a filing cabinet” points at Textura
2. Count the organisations that must write to the system. One, and you are choosing between P6 and Unifier. Many, and you are choosing between Aconex and Textura
3. Ask whether the problem outlives the project. Assets, maintenance and multi-year capital planning are Unifier’s territory. Anything that ends at handover is not
4. Check what the contract already mandates. Plenty of clients name a scheduling tool or a project CDE in the contract documents. That decision is made, so spend your effort elsewhere
5. Ask the reseller how each product meters. Per named user, per project, per transaction or per value processed changes the architecture, not just the invoice. A per-user product pushes you towards fewer, more senior users, and that changes who does data entry
6. Only then decide on a second product, and write the system-of-record sentence for every data class the two of them share before signing anything

Arguments that don’t survive contact

• “Unifier can do document control, so we don’t need Aconex.” Technically true, practically false on multi-party projects. The obstacle is not features, it is that other companies will not put their real correspondence in your instance
• “Aconex has cost, so we don’t need Unifier.” Fine for project-level cost. It falls down when you need portfolio funding, capital planning and asset lifecycle
• “Primavera Cloud is just P6 in the browser.” Oracle itself states that Primavera Cloud is not P6. Treat any migration as a data model exercise with a gap analysis, not a lift and shift
• “We’ll integrate it later.” Later means after users have created data under two incompatible breakdown structures. Agree the project and cost breakdown codes on day one, even if the integration itself waits
• “The suite integrates out of the box.” Connectors exist and they are good. Mapping your codes, your workflow states and your calendars is still work, and it is work that needs someone who understands both sides

Practices worth adopting early

• Standardise the project ID before anything else. Gateway maps P6’s Project ID to Unifier’s Project Number, and a free-text naming convention will hurt on the day you connect them
• Keep the schedule in one place and let it flow outward. Bidirectional date editing across two systems is a support ticket generator
• Pilot Unifier configuration on one real process end to end before configuring twenty. uDesigner makes it easy to build the wrong thing quickly
• Onboard the supply chain before go-live on Aconex or Textura. Network products fail on adoption, not on uptime
• Export your own data on a schedule. XER and PMXML for schedules, and regular extracts from the cloud products, so you are not negotiating from zero at renewal
• Write the system-of-record statement into the implementation charter, where people will actually see it

FAQ

What is the difference between Primavera P6 and Unifier?

P6 is a critical path scheduling tool that answers questions about time and resources. Unifier is a configurable business process and cost platform that answers questions about money, approvals and governance across a capital programme. They overlap slightly because P6 can hold cost data and Unifier has a Schedule Manager, but neither is a substitute for the other.

Do I need Aconex if I already have Unifier?

If the project involves several independent organisations exchanging drawings and correspondence, usually yes. Unifier’s Document Manager works well for your own records, but external parties are guests in your instance. Aconex gives every organisation its own workspace and control over what it shares, which is what makes the project record complete rather than partial.

Is Oracle Primavera Cloud replacing P6?

Oracle positions Primavera Cloud as the cloud platform and publishes migration guidance from P6, while continuing to sell and support P6 EPPM. Oracle also states plainly that Primavera Cloud is not P6. Plan on the two coexisting for now, and treat any migration as a data model exercise with a proper gap analysis rather than a hosting change.

Can Textura replace our accounting system?

No. Textura handles the construction payment cycle: pay applications, approvals, lien waivers, compliance holds and disbursement. It integrates with ERP systems rather than replacing them, and that integration is a real line item in the implementation.

Which product should a general contractor start with?

It depends on which pain is loudest. If schedules are being disputed, P6. If the project record is scattered across email, Aconex. If the monthly draw cycle and subcontractor compliance are consuming the finance team, Textura. Contractors reach for Unifier less often than owners do, because Unifier’s strengths are programme-level funding and governance.

How hard is it to integrate P6 with Unifier?

The mechanism is well trodden through Primavera Gateway, which ships providers for both. The hard part is not the connector, it’s agreement: matching project identifiers, deciding which system owns dates, and reconciling breakdown structures that were designed independently. Expect the mapping work to take longer than the technical setup.

What data should I be exporting for portability?

For schedules, take regular XER and PMXML exports, since those formats are what independent analysis tools and delay experts read. For the cloud products, set up scheduled extracts of documents, transaction records and audit trails. The point is not distrust of the vendor, it’s that a project record you cannot read without an active subscription is a liability during a dispute.

The one thing worth remembering

The Primavera P6 vs Unifier vs Aconex comparison, with Textura alongside them, is not really a features question. Each of these products is good at the thing it was built for, and Oracle is not trying to trick anyone by giving three of them a cost module.

The damage comes from owning two of them and never deciding which one is authoritative. Sort the products by whose record they hold and how many organisations write to them, pick the smallest set that covers your loudest pain, and write down the system of record for every data class before a single workflow gets configured.

Do that and the monthly report matches the cost system. Skip it and you’ll meet the spreadsheet.

Need help untangling this?

Most of the work in this area is not clicking through screens, it’s the plumbing and the decisions around it. That part I can help with:

• Writing the system-of-record map for your data classes before you commit to a second Oracle product, so cost and dates have one owner each
• Designing and building integrations between P6, Unifier, Aconex, Textura and your ERP, including the field mapping and error handling nobody scopes
• Hardening on-premise P6 EPPM or Unifier deployments that need to be reachable for a cloud integration, with proper TLS, network restrictions and audited access
• Automating schedule and cost data extraction into a warehouse or BI layer, so reporting stops depending on someone manually exporting on a Friday
• Building repeatable backup, restore-testing and disaster recovery for self-hosted Primavera estates
• Second-opinion reviews on an implementation or integration design that already exists and is not behaving

If you’re stuck on something concrete, send it over: an integration mapping spec, a Gateway error, a network diagram for an on-premise P6 that needs to talk to a cloud service. Easier to give you something useful when I can see the actual thing.

The post Primavera P6 vs Unifier vs Aconex vs Textura: Which Oracle Product Do You Actually Need? appeared first on John Nessime.

The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

That gap is the entire problem. Monitoring n8n workflow failures in Grafana is not hard because the tooling is missing. It’s hard because the built-in metrics endpoint answers a question you didn’t ask. It tells you the process is alive and busy. It does not tell you that a workflow stopped producing correct output.

This post covers what n8n’s Prometheus endpoint actually exposes, the three families of failure it handles badly, and the specific PromQL and alert rules that close each gap. There’s also a troubleshooting section for the case where /metrics returns nothing at all, which is the first wall most people hit.

What the n8n metrics endpoint actually gives you

n8n uses the prom-client library and serves metrics on the same port as the editor, at /metrics. The endpoint is off by default, and almost everything interesting inside it is off by default too, behind its own environment variable.

N8N_METRICS=true
N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true
N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL=true
N8N_METRICS_INCLUDE_NODE_TYPE_LABEL=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true

The first line opens the endpoint. The second turns on the workflow and node event counters, which is the part people assume is included and is not. The third attaches a workflow_id label to workflow metrics, without which you get a single instance-wide number and no way to tell which automation broke.

Those labels are opt-in for a reason worth understanding rather than working around. Every distinct label value creates a separate time series. Turn on workflow_id and you get one series per workflow per counter; add node_type and node counters fan out across every integration you use. On a few dozen workflows that’s fine. On several hundred with per-node labels, you’re signing up for a cardinality problem that surfaces later as slow queries and a Prometheus process that keeps getting OOM-killed.

With the event bus metrics enabled, the counters you get include n8n_workflow_started_total, n8n_workflow_success_total, n8n_node_started_total and n8n_node_finished_total. Queue mode adds n8n_scaling_mode_queue_jobs_active, _completed, _failed and _waiting, gathered from Bull and exposed on the main instance only.

Prometheus side, the scrape job is unremarkable:

scrape_configs:
  - job_name: n8n
    static_configs:
      - targets:
          - n8n-main:5678
          - n8n-worker-1:5678
    metrics_path: /metrics

Note both main and workers. Each process exposes its own endpoint and its own counters. Scrape only the main and you lose everything the workers did, which in queue mode is nearly all the real execution work.

One thing the n8n docs are emphatic about, and they’re right: don’t put /metrics on the public internet. It leaks workflow IDs, execution volumes and instance topology to anyone who asks. Keep the scrape path on a private network segment, a WireGuard tunnel, or your provider’s internal networking. If you’re running the whole stack on a single VPS from somewhere like Contabo or InterServer, binding n8n to the Docker network and scraping it by container name is enough. Publishing port 5678 to the host and firewalling it afterwards is not, because you will forget.

Failure family one: the execution that never happened

This is the one that costs the most and shows up the least. A schedule trigger gets deactivated during a deploy. A credential expires and the workflow is switched off rather than fixed. An upstream service stops calling a webhook. Nothing errors. Nothing increments. The workflow simply stops existing as far as your metrics are concerned.

Every error-rate alert you write is blind to this, because an error rate needs a denominator. Zero executions divided by zero executions is not an alert, it’s a NaN.

The instinct is to write this:

sum by (workflow_id) (increase(n8n_workflow_started_total[6h])) == 0

It looks correct and it does not work. When a workflow stops running, its series stops being exported. Prometheus marks it stale after a few missed scrapes and instant queries stop returning it. There is no series left for == 0 to match against. The alert never fires, because the thing you wanted to alert on is exactly the thing that removed the data.

What you want is a function that returns something when a series is absent:

absent_over_time(
  n8n_workflow_started_total{workflow_id="AbC123XyZ"}[6h]
)

absent_over_time returns 1 if the matched series produced no samples in the window and nothing at all if it did. That inversion is the point. The alert fires on silence.

The cost is that absent_over_time needs an exact matcher, so you can’t write it once for everything. That’s more acceptable than it sounds. You don’t need a heartbeat on all two hundred workflows, you need one on the six a human would notice within an hour. Write those six by hand and treat the list as part of the runbook.

One Grafana-specific trap. When a query returns nothing, Grafana’s alert rule enters a No Data state, which is configurable separately from the alert condition and defaults to firing in most versions. Set it deliberately. For an absent_over_time rule, No Data means the whole scrape target is gone, which is a different incident from one workflow going quiet, and you probably want it routed differently.

Failure family two: the failure you can see but can’t attribute

You have the counters. Something is failing. Which workflow?

Without N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL, the answer is: no idea. The counters are instance-wide totals. You know the number moved and that’s it. That’s a fine signal for capacity planning and useless during an incident.

Turn the label on and you get IDs, which are opaque strings like AbC123XyZ. A dashboard full of those is technically attributable and practically unreadable at 3am. Newer n8n versions solve this with a gauge that maps IDs to names, enabled with N8N_METRICS_INCLUDE_WORKFLOW_INFO:

n8n_workflow_info{workflow_id="VaQPuPmx9tPpo6BX",workflow_name="Invoice sync"} 1

It’s a constant-1 gauge that exists only to be joined against. The pattern is a standard Prometheus vector match:

sum by (workflow_name) (
  rate(n8n_workflow_started_total[5m])
  * on(workflow_id) group_left(workflow_name)
  n8n_workflow_info
)

The multiplication is arithmetically a no-op because the gauge is always 1. Its only job is to carry the workflow_name label across with group_left. Do this once in a Grafana variable or a recording rule rather than in every panel, or you’ll be copy-pasting the join for the rest of your life.

On an older build without the info gauge, the fallback is a hand-maintained relabel config or a Grafana value mapping. Both drift, and the relabel version drifts silently.

Failure family three: the failure that counts as a success

This is the invisible one, and it’s why a green n8n dashboard is not evidence of anything.

n8n gives you several deliberate ways to make a node’s failure not be an execution failure. They’re all reasonable features. Together they mean the success counter and “the automation worked” are different statements.

• Continue on error. A node set to continue passes its error downstream instead of stopping the run. The execution finishes. It finishes as a success.
• Error output branches. A node with a separate error output routes failures to a different path. That path might log and stop. The execution still completes normally.
• Never Error on HTTP responses. The HTTP Request node can be configured to treat any status code as a valid response. A run of 500s from an upstream API becomes a workflow that processed zero useful records, successfully.
• Empty result sets. A query that returns nothing is not an error. A sync that syncs nothing, every night, looks exactly like a sync that has nothing to do.

There are also user reports on the n8n community forum of started and success counters matching even when a node failed outright, in queue mode. I have not reproduced that and I’d treat it as unconfirmed rather than a known behavior. But it’s the right instinct: don’t assume the success counter is a strict complement of failure. Verify it on your own instance before you build alerting on the assumption.

For webhook-triggered workflows, newer n8n versions give you a real answer here. The n8n_webhook_request_duration_seconds histogram carries a status_code label, so you can alert on what the caller actually received rather than on what n8n thought about the run:

sum by (workflow_id) (
  rate(n8n_webhook_request_duration_seconds_count{status_code!~"2.."}[5m])
)

Worth flagging: the n8n documentation shows this example with status_code!="2..", using the equality operator. In PromQL, != compares against a literal string, so that matcher excludes only a status code whose literal value is the three characters 2.., which never occurs. It matches everything, including your 200s. Use !~ for the regex form. Test any label matcher you copy from documentation by running it in Prometheus and checking the series count against what you expect.

Getting a second source of truth

For everything that isn’t a webhook, the metrics endpoint alone won’t close this gap. You need a second signal that reads execution outcomes from n8n’s own store.

The public REST API exposes exactly that. Executions can be filtered by status, and the failed ones are one request away:

curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" 
  "https://n8n.example.com/api/v1/executions?status=error&limit=50"

Two ways to get that into Grafana. Write a small exporter that polls the API on a timer and serves its own /metrics, or run a scheduled job that pushes counts to a Prometheus Pushgateway. The exporter is more work and behaves correctly. Pushgateway is faster to build and carries the usual caveat: pushed metrics persist until deleted, so a job that stops running leaves a stale value sitting there looking healthy. That reintroduces failure family one through the back door.

Two caveats on the API. There’s an open report that the status=crashed list filter returns executions whose stored status is something else entirely when re-fetched by ID, which would make crash alerting built on the list endpoint unreliable. Check the behavior on your version before depending on it. And n8n’s log streaming feature, which forwards execution events to syslog or a webhook in real time, is an Enterprise capability. If you’re on the community edition, the API poll is your path.

The third option, and often the best one for a small setup, is an Error Trigger workflow. n8n lets you designate a workflow that runs whenever another workflow fails. Point it at Pushgateway, at Loki, or at a Grafana webhook contact point. It’s native, it needs no API key, and it gives you the error message rather than just a count. The trade-off is that it only fires on real execution failures, so it inherits every blind spot in this section.

The Grafana panels that earn their space

Four panels cover most of what you need. Everything past that is usually decoration.

1. Executions per workflow, by name. A time series of rate(n8n_workflow_started_total[5m]) joined to n8n_workflow_info. Its real job is showing you the lines that flatten to zero.
2. Success ratio. rate(n8n_workflow_success_total[15m]) / rate(n8n_workflow_started_total[15m]). Anything below 1 is worth a look. Anything above 1 means your window is catching a restart and the counters reset.
3. Queue depth and failed jobs. n8n_scaling_mode_queue_jobs_waiting next to increase(n8n_scaling_mode_queue_jobs_failed[1h]). A waiting count that climbs and never drains means workers are dead or starved, which looks identical to “no failures” on every other panel.
4. Webhook non-2xx rate. The query from the previous section. This is the only panel that reports what the outside world experienced.

Skip the node-level breakdown unless you have a specific reason. It is the most expensive thing you can enable in cardinality terms and it answers a question you can answer faster by opening the execution in the n8n UI.

Alert rules worth paging on

1. Named workflow silent. absent_over_time on each business-critical workflow, window set to roughly twice its expected interval. This is the rule that catches the invoice sync.
2. Success ratio below threshold. Sustained for 15 minutes, not instantaneous. A single failed run is not a page.
3. Queue waiting count rising. Compare the current value to fifteen minutes ago rather than to a fixed number, so the threshold survives traffic growth.
4. Webhook error rate. Per workflow, so a single noisy endpoint doesn’t mask the rest.
5. Scrape target down. The up metric. Obvious, and routinely forgotten, and it invalidates every other rule on this list.

Troubleshooting: the endpoint returns nothing useful

• “Cannot GET /metrics”. N8N_METRICS isn’t set on the process you’re hitting, or it’s set as a string that n8n doesn’t read as true. In Docker Compose, quote it as "true" and confirm with docker compose exec n8n env | grep N8N_METRICS.
• Endpoint responds but has no n8n_ counters. You have the default Node.js process metrics and nothing else. Set N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true and restart.
• Counters exist but no workflow_id label. N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL is off. Restart is required; it’s read at startup.
• Queue metrics are zero or missing. They’re gathered from Bull and exposed on the main instance, not the workers. Querying the worker endpoint gives you nothing. Confirm you’re looking at the right target.
• Counters look wrong after a deploy. They reset to zero when the process restarts. Always wrap them in rate() or increase(), which handle counter resets. Raw counter values on a dashboard will lie to you every time you deploy.
• Prometheus can’t reach the target. Usually a Docker network mismatch. Scrape by container name on the shared network rather than by published host port, and check the target page in Prometheus for the actual error rather than guessing.

Common mistakes

• Treating the success counter as proof the automation worked. It proves the run finished.
• Building only error-rate alerts, so the workflows that stop entirely go unmonitored.
• Enabling every N8N_METRICS_INCLUDE_* variable on day one and discovering the cardinality bill a month later.
• Scraping only the main instance in queue mode and losing all worker-side execution data.
• Copying label matchers out of documentation without running them and checking the returned series count.
• Exposing /metrics publicly because it’s “just numbers”.

Best practices

• Decide which workflows are business-critical and write a silence alert for each one by name. Six good heartbeats beat two hundred generic panels.
• Run a second source of truth. Metrics tell you about volume, the executions API tells you about outcomes. You need both.
• Push the ID-to-name join into a recording rule so panels stay readable and cheap.
• Break a staging workflow on purpose and confirm the alert fires. An untested alert is a decoration.
• If you’d rather not run Prometheus and Grafana yourself, Grafana Cloud will scrape a remote endpoint and gives you alerting without a control plane to maintain. Weigh that against your metrics leaving your network.

FAQ

Does n8n expose a failed-executions counter?

Not a reliable per-workflow one. You get started and success counters from the event bus, and a failed-jobs counter for queue mode. Deriving failures by subtracting success from started works only if you’ve verified on your own instance that a failing run genuinely doesn’t increment success, which is exactly the assumption that breaks when nodes are set to continue on error.

Can I monitor n8n Cloud this way?

No. The /metrics endpoint isn’t available on n8n Cloud. On Cloud you’re limited to the public API and the built-in insights views, so the executions-API approach in this post is the only one that transfers.

Why does my workflow error rate look like zero when I know things are failing?

Almost always one of three things: the event bus metrics aren’t enabled so you’re only seeing Node.js process metrics, the failures are being absorbed by continue-on-error settings and recorded as successes, or you’re scraping the main instance while the work happens on workers.

How do I alert when a scheduled workflow stops running entirely?

Use absent_over_time against that workflow’s started counter with an exact workflow_id matcher, and set the window to about twice the expected interval. A threshold comparison won’t work, because a workflow that stops running stops producing the series you’d compare against.

Is enabling the workflow_id label safe on a large instance?

It multiplies your n8n series count by the number of workflows. On a few dozen it’s negligible. Past a few hundred, combined with node-type labels, watch your Prometheus memory and head series count before and after. Enable the workflow label, leave the node label off, and only turn the node one on when you have a specific question it answers.

Should I use OpenTelemetry instead of Prometheus for n8n?

They solve different problems. Metrics answer “how many, how often, how fast” cheaply and forever. Traces answer “where did this run spend its time and what did it call”, which is better for debugging a slow multi-service workflow. Tracing n8n usually means a custom image with the OpenTelemetry SDK loaded before startup, so it’s a project rather than an environment variable. Start with metrics; add tracing when you have a latency question metrics can’t answer.

Where do the n8n counters go when the container restarts?

They reset to zero. They’re in-process counters with no persistence. Prometheus handles this correctly as long as your queries use rate() or increase(). Any panel showing a raw counter value will appear to lose all its history on every deploy.

The one thing to take away

Monitoring n8n workflow failures in Grafana works, but only once you stop asking the metrics endpoint a question it can’t answer. It reports activity. Activity and correctness are different things, and the space between them is where the expensive outages live.

So build for absence first. A silence alert on the handful of workflows a human would miss will catch more real incidents than any error-rate panel, because errors are loud and stopped schedules are not. Then add attribution with the workflow label, then close the false-success gap with the executions API or an Error Trigger workflow. In that order, because each one is cheap and the value drops off fast after the third.

Need help wiring this up?

I build and maintain observability stacks for self-hosted automation platforms, and n8n is a common one. If any of the above sounds like a week you’d rather not spend, here’s what I can take off your plate:

• Enabling and securing the n8n metrics endpoint on a private network path, including queue-mode setups with multiple workers
• Building Prometheus scrape configs, recording rules and the ID-to-name join so your Grafana panels are readable rather than a wall of hashes
• Writing silence and error-rate alert rules that survive counter resets, Grafana No Data states and traffic growth
• Adding a second source of truth from the n8n executions API, either as a small exporter or an Error Trigger workflow feeding Loki or Pushgateway
• Auditing an existing setup for cardinality problems, stale Pushgateway series and alerts that quietly stopped being able to fire
• Running a failure drill against staging so you know which alerts actually work before you need them

Send me your Compose file, your prometheus.yml, or a screenshot of the panel that looks fine while things are on fire, and I’ll tell you what’s missing.

The post Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying appeared first on John Nessime.

Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.

This post compares GitLab CI vs GitHub Actions vs Jenkins on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest “where it wins and where it doesn’t”, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.

The part of the decision that actually bites

Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.

Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.

The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system “for now”, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.

So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the “below the line” half cheapest for your specific situation.

What each platform is really optimising for

Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.

GitHub Actions: shortest path from push to green

Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else’s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.

A minimal hardened job header looks like this, and every line in it is there for a reason:

name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"

The permissions block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The concurrency block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.

Where it wins: your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.

Where it doesn’t: anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.

GitLab CI: one platform owning the whole lifecycle

GitLab’s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.

The pipeline language rewards you for using its graph features rather than treating stages as a queue:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh

The workflow:rules block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. needs turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. interruptible: true makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project’s CI settings; the keyword alone does nothing. And expire_in is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.

Where it wins: you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.

Where it doesn’t: your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.

Jenkins: whatever you need, on hardware you control

Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor’s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.

pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}

Two things in that file are worth pointing at. timeout and buildDiscarder are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And timestamps() and cleanWs() both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.

Where it wins: air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.

Where it doesn’t: the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.

GitLab CI vs GitHub Actions vs Jenkins on cost

Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.

• GitHub Actions. Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.
• GitLab CI. Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why “pay for seats, run your own runners” is such a common GitLab shape.
• Jenkins. No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.

Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.

Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.

Security is where the differences stop being cosmetic

Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.

• Untrusted pull requests. This is the classic GitHub Actions trap. The pull_request trigger runs fork code without access to your secrets. The pull_request_target trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request’s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub’s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.
• Expression injection. Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.
• Self-hosted runners on public repositories. Don’t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.
• Mutable action and image references. Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.
• Credential scope. Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.
• Jenkins specifically. The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project’s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.

A decision procedure you can run in an afternoon

1. Follow platform gravity first. Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.
2. Check whether a runner can reach the deploy target. Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.
3. Name the on-call owner. Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.
4. Model the compute, not the sticker price. Take last month’s build minutes, split them by operating system and machine size, and apply each platform’s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.
5. Inventory what the plugins actually do. If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The “needs writing” pile is your real migration estimate.
6. Trial with your ugliest pipeline. Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.

The hybrid pattern nobody puts on the slide

A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.

Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.

Arguments that don’t survive contact

• “Jenkins is legacy.” It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.
• “We’ll save money with self-hosted runners.” On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.
• “GitLab CI is just GitHub Actions with different YAML.” The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.
• “We’ll stay vendor-neutral by keeping all logic in shell scripts.” Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.
• “The bigger marketplace wins.” Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.

How I’d decide

• Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.
• Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.
• Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.
• Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.
• Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.

Frequently asked questions

Is Jenkins still worth using?

Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.

Which is cheaper, GitLab CI or GitHub Actions?

It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab’s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.

Can I use GitLab CI with a GitHub repository?

There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.

How long does migrating from Jenkins to GitHub Actions or GitLab CI take?

Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.

Are self-hosted runners safe?

They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.

Do I need to pin GitHub Actions to a commit SHA?

For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.

Is it reasonable to run more than one CI system?

Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.

The one thing worth remembering

Comparing GitLab CI vs GitHub Actions vs Jenkins on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.

Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.

Need help choosing or moving a CI pipeline?

This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:

• A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.
• Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.
• Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.
• Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.
• Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.
• CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.
• Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.

If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.

The post GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later appeared first on John Nessime.

Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

The host had removed it. It was on their disallowed list, a routine scan found it, and the platform did exactly what its documentation said it would do. Everyone had agreed to that when they signed up. Nobody had read it.

That incident is the whole managed WordPress hosting vs VPS argument in miniature, and it has almost nothing to do with page speed. Both models can serve a fast WordPress site. They differ on which failures land on you, which land on someone else, and which ones stay invisible until the worst possible moment.

This post compares the two the way I’d compare them for a client: an honest profile of each, the costs that never appear on the invoice, a procedure for modelling the real number, and the decision rules I actually use. No benchmark charts, because your theme and your plugin count will dominate anything the host does.

The real question: who owns the failure at 2am

Most comparisons frame this as control versus convenience. True, but useless, because it doesn’t tell you what to buy. The framing that predicts regret is: when something breaks, whose problem is it, and how fast can they fix it?

Split the failure surface into layers and the answer gets concrete:

• Hardware and hypervisor. Theirs in both models. Your recovery time is still yours.
• OS, web server, PHP, database. Theirs on managed hosting, yours on an unmanaged VPS. This is the big split.
• Backups and restores. Theirs on managed, and tested, because their business depends on it. Yours on a VPS, and almost nobody tests them.
• Core, themes, plugins. Mostly yours either way. Some hosts auto-update, but the compatibility fallout is still yours.
• Your code and your content. Always yours. No host takes this on.

The bottom two rows never move. A managed platform will not make your bloated page builder fast, and it will not stop a vulnerable plugin from being exploited. It moves the middle three rows off your plate and charges you in money and in flexibility.

Managed WordPress hosting, assessed honestly

Where it genuinely wins

The strongest argument isn’t performance. It’s that the boring, high-consequence work gets done by people whose job it is, on a schedule, whether or not you remember.

• Restores that work. One-click recovery from a backup the platform verifies. That’s the difference between a bad hour and a bad week.
• Server-level page caching, already tuned. WP Engine, Kinsta, Pressable and Flywheel run full-page caching in front of PHP. It’s why they ban plugins that try to do the same thing badly.
• Staging that isn’t a science project. Clone, test, push back. On a VPS that’s yours to build, and a half-built staging setup is worse than none.
• Support that knows WordPress. A generic VPS provider’s ticket ends at “the server is up”. A WordPress host will look at your slow query log.
• Edge protection before PHP loads. WAF and bot filtering at the infrastructure layer, the only layer where they meaningfully help. A plugin firewall runs after the request has already cost you a PHP worker.

If nobody on your team wants to own a Linux box, that list is decisive. The rest of this section is about what you’re trading away.

Visit metering is a billing model, not a traffic report

This is the cost that bites hardest, because it doesn’t look like a technical constraint until the invoice arrives.

Most managed plans meter “visits” rather than bandwidth or CPU, and the definition is narrower than it sounds. WP Engine defines a billable visit as one unique IP address logged per day, with static assets like images, CSS and JavaScript excluded, and known and suspected bot user agents filtered out. Kinsta counts the sum of unique IPs seen within each 24-hour period across the plan, and also sells bandwidth-metered plans as an alternative. Flywheel filters IPs belonging to known bots, spammers and attackers before counting.

Three consequences follow, and they’re the ones people miss:

• Your host’s number will not match Google Analytics. Analytics counts sessions from browsers that ran its JavaScript. Your host counts IPs that hit the server. Arguing about the gap is wasted effort.
• Bot filtering is best-effort. Hosts filter what they can identify. A scraper with a browser-shaped user agent and rotating IPs looks, to the meter, like a crowd of new visitors.
• Overages are charged per block of excess visits. Rates change, so check the current figure rather than trusting any article, including this one. What matters is the shape: a viral post or a bad crawl month produces a bill you didn’t budget for.

A VPS has the opposite failure mode. Bad traffic costs you CPU and RAM, not money, right up until it exhausts them and the site falls over. One model converts load into a bill, the other converts it into an outage. Pick the one you’d rather explain to whoever owns the site.

The plugin policy is a real constraint

Managed hosts publish disallowed plugin lists and they enforce them. WP Engine documents that disallowed plugins are found by periodic scans of the site’s filesystem, the owner is notified, and the plugin is removed. That’s not a threat, it’s the operating model, and it’s why the incident I opened with was nobody’s fault.

The banned categories are consistent across vendors, and the reasoning holds up in each case:

• Caching plugins. They fight the platform’s own page cache and produce stale output that support then has to debug.
• Backup plugins. They write large archives onto the same disk the site runs on, which is a useless backup if the disk is what fails.
• Database optimisation plugins. Long-running write-heavy queries on shared infrastructure affect neighbours.
• Some security plugins, or specific features within them. Filesystem-based rule storage and high-volume traffic logging are the usual triggers. Policies vary a lot here.
• Plugins that send mail directly. Platforms push you to a transactional provider instead, for deliverability reasons that are genuinely in your interest.

None of that is unreasonable. The problem is when a plugin your business depends on lands in one of those buckets, or when a host adds something to the list later. Before migrating, read the current disallowed list for that specific host and diff it against your live active plugins. Ten minutes, and it’s the highest-value thing you can do in a hosting evaluation.

You get one shape of application

A managed WordPress platform runs WordPress. If your project also needs a Node service, a Python worker, a queue, or a self-hosted analytics instance, that’s a second bill somewhere else. On a VPS those things are free in cash terms because you already own the machine. They cost you memory and attention instead. Whether that’s a good trade depends on whether you were going to need them anyway.

Running WordPress on your own VPS, assessed honestly

Where it genuinely wins

• Resources per unit of money. Hetzner, InterServer, DigitalOcean and Vultr will sell you more CPU and RAM than an entry managed plan. Whether you can use it is a separate question.
• Many sites, one bill. This is where the economics genuinely flip. Ten small sites on one adequate VPS is one server to patch. Ten managed plans is ten invoices.
• No metering surprises. Traffic spikes hit your load average, not your credit card.
• You choose the stack. Nginx or Apache, PHP-FPM pool sizing, Redis for the object cache, your own MariaDB tuning and firewall rules. If the platform’s opinions were what limited you, this is the fix.
• No lock-in. A tarball and a database dump move anywhere. Platform-specific caching behaviour, redirect rules and deployment workflows do not.

The failure that stays invisible: the backup you never restored

Here’s the one that actually ends projects. Backups on a self-managed VPS fail silently. The cron entry runs, the exit code goes nowhere, the destination bucket quietly rejects writes after a credential rotation, and nothing tells you. You find out the day you need it, which is also the day you have no other option.

The fix isn’t a better tool. Restic and BorgBackup are both excellent and neither will save you. The fix is a restore drill on a schedule, into a scratch directory, where you verify the data is actually there.

# Restore the newest snapshot into a scratch path, never over the live site
restic -r /srv/backups/wp restore latest --target /tmp/restore-drill

# Confirm the dump is real and not a zero-byte file or a truncated
# write from a run that died halfway through
ls -lh /tmp/restore-drill/db/
head -n 20 /tmp/restore-drill/db/site.sql

# Load into a throwaway database and count what came back.
# If wp_posts is empty, your backups have been failing for weeks.
mysql -u drill -p drill_db < /tmp/restore-drill/db/site.sql
mysql -u drill -p -e "SELECT COUNT(*) FROM drill_db.wp_posts;"

Adjust the repository path and database names to your own. The shape is the point: restore somewhere harmless, then check a row count that would be non-zero on a healthy site. A backup you haven’t restored is a hypothesis, not a backup. Uptime checks from something like UptimeRobot tell you the site is down; they tell you nothing about whether you can get it back.

Patching debt compounds quietly

The second invisible cost is the gap between “updates are available” and “updates are applied and the affected services have actually restarted”. Upgrading a package without restarting the process leaves the old, vulnerable code resident in memory. That’s why security scanners and package managers disagree so often.

# RHEL family (AlmaLinux, Rocky, CentOS Stream)
# What security updates are outstanding?
sudo dnf updateinfo list security

# After applying them: does anything still need a restart?
# Exit 0 means no reboot required, exit 1 means reboot required.
sudo dnf needs-restarting -r

# Debian and Ubuntu: what would unattended-upgrades actually do?
sudo unattended-upgrade --dry-run --debug

Run those on any VPS you inherited from someone else. The output is usually educational. Then there’s the WordPress layer, which stays yours regardless of hosting model:

# Do the core files match the official checksums for this version?
# A mismatch means a modified core file or an active compromise.
wp core verify-checksums

# How many plugins are behind?
wp plugin list --update=available --format=count

# Which tables are eating the disk? Usually wp_options, wp_postmeta,
# or a logging plugin nobody remembers installing.
wp db size --tables --format=table

Model the cost before you argue about it

Sticker price comparisons are how people talk themselves into the wrong answer in both directions. This procedure produces a number you can defend.

1. Count your sites. One tilts toward managed. Five or more tilts toward a VPS, because admin time is close to fixed and plan fees are per-site.
2. Estimate your metered visit count from your own logs. If you’re already on a server you control, you can approximate what a metered host would bill you before signing anything.
3. Price the VPS honestly. Instance plus block storage plus offsite backup destination plus any control panel licence. The advertised instance price is rarely the whole line.
4. Put a rate on your own hours. Whatever you’d bill a client, or pay someone else. Zero is not an honest number.
5. Add the one-off build. Standing up a hardened WordPress VPS with TLS, firewall, object cache, tested backups and log rotation is a solid chunk of focused work the first time. Amortise it over twelve months.
6. Add your recovery cost. Hours to rebuild from scratch, times your rate, times how often you honestly think it’ll happen. This term is what makes managed hosting look cheap for a single revenue-generating site.

Step two is worth doing properly, because it’s the number nobody has. On Nginx with the standard combined log format, this gets you close:

# Unique client IPs in the log. A rough upper bound on metered "visits".
awk '{print $1}' /var/log/nginx/access.log | sort -u | wc -l

# Closer to how hosts count: ignore static assets, since most platforms
# exclude images, CSS and JS from billable visits.
awk '$7 !~ /.(css|js|png|jpg|jpeg|gif|svg|webp|woff2?|ico)$/ {print $1}' 
  /var/log/nginx/access.log | sort -u | wc -l

# What is actually hitting you? User agents by request count.
# If the top entries are crawlers, that is your overage risk.
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Field positions assume the combined log format; adjust the column numbers if you’ve customised log_format. The third command is the interesting one. If crawlers dominate your traffic, visit-metered hosting will charge you for whatever the host’s filtering misses, and that’s a risk you can quantify before signing up rather than after.

The middle ground worth looking at first

This gets presented as two options. There are four, and the middle two get skipped far too often.

• Managed VPS. InterServer, Liquid Web and others sell VPS plans where the provider handles OS patching, firewall and backups while you keep root. Read the scope: “managed” often stops at the operating system and leaves the WordPress stack to you.
• Control panel on your own VPS. DirectAdmin, cPanel or CyberPanel gives you WordPress-aware tooling, staging and one-click TLS on infrastructure you still own. You pay a licence and give up some configuration purity.
• Server management platforms. Cloudways, RunCloud, GridPane and SpinupWP provision and manage a stack on a VPS you rent from a provider of your choice. Managed-style workflows without the per-visit meter.
• A CDN in front of anything. Putting Cloudflare or a similar edge ahead of a modest VPS absorbs a large share of the traffic that would otherwise push you onto a bigger plan. Cheapest performance win available to either model, and frequently the actual answer to “we need better hosting”.

If you’re leaning toward a VPS mainly because managed pricing looks steep, look at options two and three before committing to a bare Linux box.

How I’d decide between managed WordPress hosting vs VPS

These are the rules I apply, in order. The first match usually settles it.

• Nobody on the team can or wants to run Linux. Managed, and it isn’t close. An unpatched VPS is worse than the most restrictive managed plan.
• One site, and it generates revenue. Managed. The recovery term dominates everything else in the cost model.
• Five or more sites and someone competent to run them. VPS, or a management platform on top of one.
• You need services WordPress isn’t. VPS. Don’t pay for two platforms to avoid learning one.
• A plugin you can’t remove is on the host’s disallowed list. VPS, or a different managed host. Do not migrate hoping for an exception.
• Traffic is spiky and crawler-heavy. Lean VPS, or pick a bandwidth-metered plan over a visit-metered one. Metering punishes exactly this traffic shape.
• You’re mostly annoyed at the current bill. Neither, yet. Put a CDN in front, cut the plugin count, fix the object cache. Hosting is often not the bottleneck.

Arguments that don’t survive contact

• “A VPS is faster because you get dedicated resources.” Only if it’s configured well. A default LEMP install with no object cache and an untuned PHP-FPM pool loses to a managed platform’s edge cache every time.
• “Managed hosting handles security, so I’m covered.” It handles infrastructure security. The most common route into a WordPress site is a vulnerable plugin, and that’s yours in both models.
• “I’ll just move if I outgrow it.” Migration cost is real and grows with the site. Redirect rules, caching behaviour, deployment workflow and email configuration all need rebuilding.
• “The VPS is only a few dollars a month.” The instance is. Offsite backup storage, monitoring and your own hours are not, and they’re most of the number.
• “Managed hosts back everything up, so I don’t need my own backups.” Retention windows are finite and account access can be lost. Keep an independent copy the host doesn’t control. This applies to both models.

Frequently asked questions

Is a VPS actually cheaper than managed WordPress hosting?

For one site, usually not once you price your own time honestly. For several sites, usually yes, and the gap widens with each site you add. The crossover sits somewhere around three to five sites for most people, depending on what your hours are worth and how much the sites change.

Why does my managed host report more visitors than Google Analytics?

They measure different things. Analytics counts browser sessions that executed its JavaScript, so it misses anything blocking scripts and misses non-browser clients entirely. Your host counts requests reaching the server, deduplicated by IP per day. Bot filtering closes some of the gap but never all of it. The two numbers are not supposed to match.

Can I run other applications alongside WordPress on managed hosting?

Generally no. Managed WordPress platforms run WordPress and nothing else. A background worker, a separate API, a queue or a self-hosted analytics instance needs a VPS or a second platform. Factor that second bill into the comparison.

What happens if I install a disallowed plugin?

It depends on the host, but the documented pattern at WP Engine is that periodic filesystem scans detect it, you get notified, and the plugin is removed. Check the specific host’s current policy and enforcement mechanism before migrating, and check it against your live plugin list rather than what you assume is installed.

Do I still need a caching plugin on a VPS?

You need caching. Whether it’s a plugin is a design choice. The strongest setup is full-page caching at the web server or CDN layer plus a persistent object cache in Redis or Memcached, which handles the repeated database queries page caching can’t touch. A page cache alone leaves logged-in traffic and admin requests hitting the database on every load.

Is managed VPS hosting a real middle ground or just marketing?

It’s real, but the scope varies enormously and the word isn’t standardised. Ask specifically: who applies OS security patches, who restarts services afterwards, who owns backups and where they live, whether restores are tested, and whether the WordPress stack is in scope or the coverage stops at the operating system. Get the answers in writing before comparing prices.

How hard is migrating from managed hosting to a VPS?

Moving files and the database is the easy part. The work is everything the platform did invisibly: redirect rules living in the host’s dashboard, cache purging behaviour, transactional email, TLS renewal, and any deployment workflow tied to the platform. Plan a parallel run with low DNS TTLs rather than a cutover, and keep the old plan alive until the new one has survived a full traffic cycle.

The one thing worth remembering

Managed WordPress hosting vs VPS isn’t a performance question, and it isn’t really a price question either. It’s a question about which failures you’re equipped to own.

Managed hosting converts operational risk into a predictable bill and a set of constraints you have to live inside. A VPS converts that bill into flexibility plus a standing obligation to do the boring work: patch, restart, and prove your restores actually restore. Both are defensible. What isn’t defensible is buying a VPS because it looked cheap and then skipping the work, which is the most common failure in this whole category and the one that ends with a rebuild from a backup nobody ever tested.

Run the restore drill. Whatever you choose.

Need a second opinion on your hosting decision?

Most of the hosting work I get asked for is one of these. If it looks like your situation, I can help:

• Cost modelling before you commit. Working out from your own access logs what a visit-metered plan would actually bill you, against a realistically priced VPS including backup storage and hours.
• Migration in either direction. Managed to VPS or VPS to managed, planned as a parallel run with DNS TTL staging so there’s no blind cutover.
• Building the VPS properly the first time. Nginx or Apache with PHP-FPM sized to the actual RAM, Redis object cache, TLS with automated renewal, firewall and Fail2ban, log rotation that doesn’t fill the disk.
• Backups you’ve actually restored. Offsite destination, retention policy, and a scheduled restore drill that fails loudly instead of silently.
• Plugin policy audits. Checking your live plugin list against a specific host’s disallowed list before migration, and finding replacements for anything that fails.
• Monitoring that answers the right question. Grafana and Prometheus dashboards covering PHP-FPM saturation, database load and cache hit ratio, not just whether the site returns 200.

Send me something concrete and I’ll tell you what I see: an access log sample, your current plan and traffic numbers, your active plugin list, or the output of the commands above. That’s usually enough for a straight answer before either of us talks about scope.

The post Managed WordPress Hosting vs VPS: Who Owns the Failure? appeared first on John Nessime.

Either way you end up looking at Podman, reading a Podman vs Docker comparison table that says “daemonless” and “rootless by default”, and trying to work out whether those words are worth a migration.

Here is the short version, and I will spend the rest of the post defending it. The architectural difference between the two engines is real and it matters more than the feature tables suggest. The migration cost is also real, and it is almost entirely concentrated in four places that comparison posts tend to skip. Whether the switch is worth it depends on which of those four you actually depend on.

This post covers the one architectural difference everything else follows from, an honest profile of each engine with where it wins and where it does not, the four things that genuinely break when you switch, and a decision procedure you can run against your own setup rather than against a generic one.

The difference everything else follows from

Docker is a client and a daemon. The docker CLI is a thin thing that talks to dockerd, which runs as root and owns every container on the host. Podman has no daemon. When you run podman run, the container is started as a child of your own process and supervised by a small monitor, not handed off to a privileged background service.

Two consequences follow, and they are the whole argument.

Membership of the docker group is equivalent to root. Not “close to root”, not “root if misconfigured”. The daemon runs as root and will happily create a container that bind-mounts the host’s root filesystem, so anyone who can talk to the socket can read or write anything on the box. Docker’s own documentation says as much. This is the finding that shows up in access reviews, and it is not a bug you can configure away without switching to rootless mode.

Process supervision belongs to whoever started the container. With Docker, the daemon restarts your containers according to the restart policy, and if the daemon wedges you have a bad afternoon regardless of what the containers are doing. With Podman, containers are ordinary processes, so systemd supervises them the same way it supervises everything else on the machine, with the same logging, dependency ordering and failure handling.

Podman’s rootless mode builds on user namespaces: UID 0 inside the container maps to your unprivileged UID on the host. A break-out lands in a normal user account rather than on a root shell. Docker has a rootless mode too and it works, but it is an opt-in you configure rather than the path of least resistance, which in practice means far fewer installations run it.

Docker: where it wins and where it doesn’t

Where it wins

• The ecosystem assumes it. Testcontainers, VS Code Dev Containers, Traefik’s container discovery, Portainer, most CI runners, half the SDKs on your machine: all of them expect a Docker socket. Some work fine against Podman’s compatible socket. Some do not, and finding out which is your problem.
• Compose is first-party. Docker Compose is maintained by the same people who maintain the engine, tracks the Compose specification closely, and gets features like --wait and profiles as soon as they exist. Nothing in the Podman world matches that.
• Every answer to every question already exists. Fifteen years of Stack Overflow. That is not a technical merit but it is a real operational one at 2am.
• Cross-platform experience is smoother. Docker Desktop on macOS and Windows is a polished product, and the VM it manages is largely invisible.

Where it doesn’t

• The root daemon is a standing liability. One privileged process that everything flows through, plus a group that is root by another name.
• Docker Desktop is licensed, and the terms are specific. Docker’s own license page states Desktop is free for personal use, education, non-commercial open source, and small businesses defined as fewer than 250 employees and less than ten million dollars in annual revenue. Cross either threshold and commercial use needs a paid subscription. Government entities need one regardless. Docker Engine and the Moby project are not affected. This catches people out because the software installs without asking for a key.
• systemd integration is bolted on. You can wrap a Compose stack in a unit file, but systemd is supervising a wrapper, not the containers.
• Restarting the daemon touches everything. An engine upgrade is a whole-host event rather than a per-service one.

Podman: where it wins and where it doesn’t

Where it wins

• Rootless is the default, not a project. You get the safer posture by doing nothing special, which is the only kind of security control that survives contact with a busy team.
• Quadlet is genuinely better than restart policies. You write a declarative unit file, systemd generates a real service from it, and container lifecycle becomes indistinguishable from any other service on the box. This is the feature I would actually miss if I went back.
• No licensing question to answer. Podman is Apache 2.0 with no commercial-use restriction, and on Red Hat Enterprise Linux, Fedora, Rocky and AlmaLinux it is the packaged, supported option already sitting in the repos.
• Pods and Kubernetes YAML are first-class. podman kube play runs a Kubernetes manifest locally, and podman generate kube goes the other way. If your destination is Kubernetes, the mental model transfers.
• Per-user isolation comes free. Two developers on a shared build host cannot see or stop each other’s containers, because they are running in separate user namespaces rather than as siblings under one daemon.

Where it doesn’t

• Compose is a compatibility story, not a native one. More on this below, because it is the single biggest source of migration pain.
• Rootless networking has sharp edges. Traffic goes through a userspace network stack, and the behaviour is not identical to a root bridge. Source IPs, host-loopback access and privileged ports all behave differently, and you will meet at least one of those.
• Swarm and Docker plugins do not exist. Podman does not implement Swarm mode, and Docker volume and network plugins are not supported. If you use either, the conversation is over.
• Desktop platforms mean a VM you can see. Podman on macOS and Windows runs a Podman machine, and the seams show more than Docker Desktop’s do.
• Documentation assumes Docker. Every vendor quickstart, every README, every “run this to try it” snippet is written for docker run. Most translate directly. The ones that do not cost you an hour each.

The four things that actually break

This is the part worth reading twice. Command compatibility is not the problem; podman accepts almost every docker flag you know. The problems are elsewhere.

1. Compose

There are two routes and they are not equivalent.

podman-compose is a separate Python implementation. It covers the common cases and diverges on the uncommon ones. Container naming is the one that bites scripts first: it has historically used underscores where Docker Compose v2 uses hyphens, so anything referencing project-web-1 by name stops finding it.

The Podman socket plus real Docker Compose is the route I would take. Podman exposes a Docker-compatible REST API over a systemd socket, and the actual Compose binary talks to it:

# Start the Docker-compatible API socket for your user.
systemctl --user enable --now podman.socket

# Confirm it exists before pointing anything at it.
ls -l /run/user/$(id -u)/podman/podman.sock

# Point the real Docker Compose at Podman instead of dockerd.
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
docker compose up -d

You keep the tool that tracks the Compose specification and swap only the engine underneath. The catch is that the API is a compatibility layer, not the Docker daemon, so behaviour can differ at the edges. Run your real compose file through it before you commit to anything.

2. Anything that mounts the Docker socket

Traefik discovering containers by label, Portainer, Testcontainers in your test suite, any CI job that does Docker-in-Docker. All of these want /var/run/docker.sock. The Podman socket serves the same API and many of them work against it unchanged, but “many” is doing work in that sentence. Inventory what mounts the socket before you plan the migration, not after.

3. Ports below 1024

A rootless container cannot bind a privileged port, because the process genuinely is not privileged. If your compose file publishes 80 and 443, it will fail, and the error will not obviously say why.

Three ways out, in the order I would try them:

1. Publish a high port and put a host reverse proxy in front. Nginx or Caddy running as a normal system service, listening on 443, proxying to 8443. No kernel tuning, and you probably wanted TLS termination on the host anyway.
2. Lower the privileged port threshold. The kernel exposes net.ipv4.ip_unprivileged_port_start, and setting it to 0 lets any user bind any port. It works, and it is a host-wide change that weakens a defence for every process on the machine, not just yours. Understand that before you write it into a playbook.
3. Run that one container rootful. Podman supports it. Mixing modes is a legitimate choice as long as it is deliberate and documented rather than something that happened.

4. Restart on boot

This is the one that catches people who tested on a laptop and shipped to a server. Rootless containers belong to a user session, and a user’s services are stopped when their last session ends. Reboot the box and nothing comes back, no matter what restart policy you set.

The fix is two parts. Enable lingering so the user’s systemd instance runs without a login, and define the container as a Quadlet unit rather than relying on a restart policy:

# Let this user's services run without an active login session.
sudo loginctl enable-linger appuser

# Verify. Linger=yes is the line that matters.
loginctl show-user appuser --property=Linger

Then the unit itself. Quadlet files look like systemd units with a Podman-specific section, and a generator turns them into real services at boot and on daemon-reload. Rootless units live in ~/.config/containers/systemd/; system ones in /etc/containers/systemd/.

# ~/.config/containers/systemd/api.container

[Unit]
Description=Application API
After=network-online.target

[Container]
# Fully qualified. Podman does not assume Docker Hub.
Image=registry.example.com/api:1.4.2
PublishPort=8080:8080
Volume=api-data.volume:/var/lib/api
Environment=NODE_ENV=production

[Service]
Restart=always

[Install]
# Starts at boot once lingering is enabled for this user.
WantedBy=default.target

# Regenerate services from Quadlet files, then start it.
systemctl --user daemon-reload
systemctl --user start api

# From here it is an ordinary service.
systemctl --user status api
journalctl --user -u api -f

Note that podman generate systemd, which you will find in older tutorials, is deprecated in favour of Quadlet. Podman’s documentation says it receives urgent bug fixes only and no new features. Do not build anything new on it.

One smaller trap while you are here: Podman does not assume Docker Hub for unqualified image names. nginx:alpine may prompt you to choose a registry or fail outright depending on configuration. Write docker.io/library/nginx:alpine, or whatever your registry is, everywhere. This is arguably the correct behaviour and it is still an afternoon of edits.

Podman vs Docker: how I would decide

Run these in order and stop at the first one that answers.

1. Do you use Swarm, or Docker volume or network plugins? Stay on Docker. Podman does not implement them and no workaround changes that.
2. Is a compliance requirement or an access review driving this? Switch, or at minimum move to rootless. “The docker group is root” is a finding you cannot argue with, and rootless Podman is the shortest path to closing it.
3. Are you on RHEL, Fedora, Rocky or AlmaLinux? Lean Podman. It is the packaged, supported, tested-by-the-vendor option on those distributions, and you are swimming with the current instead of against it.
4. Is Docker Desktop licensing a real cost for your organisation? Check the thresholds against your actual headcount and revenue before assuming either way. If you are over, this is a budget conversation with a free technical answer, which is a rare shape of problem.
5. Does your test suite or CI depend on the Docker socket? Prototype that specific dependency against the Podman socket first. It is the highest-risk item and the cheapest to test. If it fails, everything else is academic.
6. None of the above, and Docker is working? Stay. “Podman has nicer architecture” is true and it is not a business case. Migrations cost weeks, introduce risk, and the containers you are already running do not care which engine started them.

The honest summary: Podman is the better-designed engine and Docker is the better-supported one. Design wins when you are building something new or when a policy requirement forces the question. Support wins when you have a working system and a limited number of weeks.

If you do switch, do it in this order

1. Inventory the socket mounts. Grep your compose files and CI config for docker.sock. That list is your actual migration scope.
2. Stand up one non-critical service on Podman. A staging box, an internal tool, something on a spare VPS. Run it for a fortnight through at least one reboot before you draw conclusions.
3. Fully qualify every image reference. Do this in the compose files while they still run under Docker. It changes nothing there and removes a whole class of failure later.
4. Switch the engine, keep Compose. Podman socket plus DOCKER_HOST. Change one variable at a time so you know what broke.
5. Convert to Quadlet last, and only where it earns its place. Long-lived services benefit. A three-container stack you rebuild constantly is fine staying on Compose.

Arguments that don’t survive contact

• “Podman is a drop-in replacement.” The CLI nearly is. The surrounding tooling is not, and that is where the weeks go.
• “Daemonless means faster.” The engine is not your bottleneck. Rootless networking has historically been slower than a root bridge, not faster. Pick Podman for the security model, not for benchmark numbers.
• “Rootless means secure.” It means a break-out lands somewhere less catastrophic. A container running rootless with the host filesystem bind-mounted read-write is still a bad idea.
• “Docker is dying.” It is not. Docker Engine, Compose and Buildx are actively developed and hosted CI runners still ship them by default.
• “Just alias docker to podman and you’re done.” The alias handles the commands. It does not handle the socket, the ports, the boot behaviour or the image names.

Frequently asked questions

Can Podman run docker-compose files?

Yes, by two routes. Install podman-compose, which is a separate implementation with good but incomplete coverage, or enable the Podman socket and point real Docker Compose at it with DOCKER_HOST. The second gives better specification coverage because you are running the actual Compose tool. Neither is a guarantee; test your own file.

Is Podman actually more secure than Docker?

Its default posture is better. Rootless out of the box means no root daemon and no group that is root by another name. Docker can be configured to a similar posture with rootless mode, so the difference is less “Podman is secure and Docker is not” and more “Podman’s safe path is the easy path”. Defaults determine what most installations actually run.

Do I need to pay for Docker?

Docker Engine on Linux is open source and free. Docker Desktop is the licensed product. Per Docker’s terms it is free for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and under ten million dollars in annual revenue; cross either threshold, or be a government entity, and commercial use requires a subscription. If you only run Docker Engine on Linux servers, the licensing question does not apply to you.

Why do my Podman containers disappear after a reboot?

Because rootless containers run under your user’s systemd instance, which is torn down when your last session ends. Run loginctl enable-linger for the service account and define the containers as Quadlet units with WantedBy=default.target. A Podman restart policy alone will not survive a reboot.

What is Quadlet and do I have to use it?

Quadlet is a systemd generator that turns declarative container definitions into real systemd services. You do not have to use it, but the alternative is hand-writing unit files, and the older podman generate systemd command is deprecated in favour of it. For anything long-lived on a server, Quadlet is the current answer.

Can Podman and Docker coexist on the same host?

Yes. They keep separate image and container stores, so nothing collides at the storage layer. Watch for port conflicts if both are running services, and be deliberate about which one a given script is talking to, because DOCKER_HOST and an alias can quietly disagree with each other.

Does Podman work with Kubernetes?

Better than Docker does, in the sense that it speaks the format natively. podman kube play runs a Kubernetes manifest on a single host and podman generate kube produces one from running containers. It is not a substitute for a cluster, but as a local development story for a Kubernetes destination it is the closest thing available.

The one thing worth remembering

The Podman vs Docker decision is not really about container engines. Both run OCI images and both will start your workload. It is about whether a root daemon and a root-equivalent group are acceptable in your environment, and whether you have the weeks to spend on the four things that break.

If a policy requirement is driving the question, switch and budget properly for the compose and socket work. If you are on an enterprise Linux distribution and starting something new, start on Podman, because you will be swimming with the current. If Docker is working and nothing external is forcing the question, the correct engineering answer is to leave it alone and go fix something that is actually broken.

Thinking about a migration and want a second opinion?

I help teams decide this question and then execute it without a surprise outage. Things I am usually brought in for:

• Auditing what on your hosts actually depends on the Docker socket, and scoping the migration honestly before anyone commits to it
• Testing your real compose files against the Podman socket and reporting what breaks, rather than guessing from a feature table
• Converting long-lived services to Quadlet units with proper systemd ordering, health gating and journald logging
• Sorting out rootless networking: privileged ports, reverse proxy placement, and the source-IP behaviour that surprises people
• Closing the “docker group equals root” audit finding, whether that means Podman, rootless Docker, or removing the group entirely
• Writing the decision up so it survives the next person who asks why you chose what you chose

If you are weighing this up, send me a compose file and a list of what mounts the socket. That is usually enough to tell you whether the switch is a fortnight or a quarter.

The post Podman vs Docker: Is the Switch Actually Worth It? appeared first on John Nessime.

That is the shape of most Docker Compose incidents. Not a dramatic architectural failure. A default that was fine on your laptop and wrong on a server, sitting quietly for half a year until it wasn’t.

Running Docker Compose in production is a perfectly reasonable choice for a lot of workloads. The tooling is stable, the file format is readable, and a single host with a handful of services does not need a control plane. What it does need is that you go through the defaults deliberately, because Compose was designed for a developer machine and inherits assumptions from that world.

This post walks the failure families I look for first when reviewing a production Compose setup: the deploy gap, disk exhaustion, state you did not mean to keep, published ports that walk past your firewall, secrets handling, and health checks that report without acting. Then the troubleshooting commands, the mistakes I see repeatedly, and an honest read on when to stop using Compose.

What Docker Compose genuinely gets right

Worth stating the case before pulling it apart, because a lot of writing on this topic is really Kubernetes marketing.

• The whole system fits in one file you can read. Onboarding somebody onto a Compose stack takes minutes. That is not a small thing when you are the only person on call.
• Dev and prod can share a base file. Override files let you keep one source of truth and layer the production differences on top, rather than maintaining two drifting definitions.
• No control plane to operate. A Kubernetes cluster is a system that itself needs upgrading, monitoring and debugging. On a single VPS from a provider like Hetzner, DigitalOcean or InterServer, that overhead buys you very little.
• Recovery is comprehensible. When something breaks at 2am, docker compose ps and docker compose logs tell you nearly everything. There is no scheduler making decisions you have to reverse-engineer.

If your workload is one machine, one team, and downtime measured in seconds rather than zero, Compose is a defensible answer. The rest of this post is about making that answer survive contact with a real server.

Failure family one: the deploy gap

This is the one that surprises people who came from a platform that did rolling updates for them.

docker compose up -d is not a rolling update. When a service’s image or config has changed, Compose stops the old container and then starts the new one. Between those two events the service does not exist. If the container takes twenty seconds to boot a JVM or run migrations, that is twenty seconds of connection refused.

The deploy.update_config block with order: start-first exists in the Compose file specification, but it describes behaviour for orchestrators like Swarm. Do not assume it gives you overlap on a plain Compose host. Test it on your own setup before you rely on it.

What to do instead

Three options, in increasing order of effort:

1. Accept the gap and shrink it. Pull images before you cut over so the restart is not waiting on a network transfer, and make the container boot fast. For an internal tool, a five-second gap is fine and you should not build machinery to avoid it.
2. Put a reverse proxy in front and run two slots. Traefik, Caddy or plain Nginx in a container, with app-blue and app-green services. Start the new one, wait for it to pass its health check, move the proxy, stop the old one. This is real zero-downtime and it is maybe forty lines of config.
3. Drain at the edge. If you already sit behind Cloudflare or a load balancer, take the host out of rotation, deploy, put it back. Simplest when you have more than one host anyway.

Whichever you pick, the deploy itself should pull first and then block on health rather than returning immediately:

# Fetch new images while the old containers are still serving traffic.
docker compose pull

# Recreate changed services, then block until every service with a
# healthcheck reports healthy. Non-zero exit if something never gets there.
docker compose up -d --wait --wait-timeout 120

# --wait only knows about services that define a healthcheck.
# Services without one are treated as ready the moment they start.

That last line matters more than it looks. A service with no health check is invisible to --wait, so a deploy script can report success while your API is still crash-looping.

Failure family two: the disk fills up and nothing tells you

Docker’s default logging driver is json-file, and by default it performs no rotation at all. The docs are explicit that this default exists for backward compatibility, and that the local driver is the recommended alternative because it rotates out of the box and uses a more compact format.

So the failure mode is: a service starts logging every request, or starts emitting a stack trace in a loop, and the log file grows without limit until the filesystem is full. Everything else on that host dies at the same moment, which makes the root cause harder to see, not easier.

Fix it once at the daemon level so it applies to every container on the host, including ones you spin up by hand:

// /etc/docker/daemon.json
{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    { "base": "10.40.0.0/16", "size": 24 }
  ]
}

Two things to know before you restart the daemon. Log options must be strings in this file, quotes included, or Docker will refuse to start. And the change only affects containers created afterwards; existing containers keep whatever config they were created with, so you need to recreate them.

You can also set limits per service, which is worth doing for a known-chatty component. A YAML anchor keeps it from being copy-pasted six times:

x-logging: &default-logging
  logging:
    driver: local
    options:
      max-size: "10m"
      max-file: "3"

services:
  api:
    image: registry.example.com/api:1.4.2
    <<: *default-logging

  worker:
    image: registry.example.com/worker:1.4.2
    <<: *default-logging

Logs are only one of three things eating the disk. The others are old images, which accumulate every time you deploy, and dangling volumes. Put a scheduled cleanup on the host and monitor free space with whatever you already run, whether that is Prometheus and Grafana, a hosted agent, or a plain cron job piping into Healthchecks.io.

# Images not used by any container and older than a week.
# -a includes untagged parents, not just dangling layers.
docker image prune -a --filter "until=168h" --force

# Where the space actually went.
docker system df -v

Failure family three: state you did not mean to keep

Two related traps here.

The first is anonymous volumes. If an image declares a VOLUME in its Dockerfile and your Compose file does not map that path to a named volume, Docker creates an anonymous one. Your data is real and it is on disk, but it has a hash for a name, it is not in your backup script, and the next person who runs a cleanup command has no way to know it matters. Always name your volumes explicitly.

The second is docker compose down. On its own it removes containers and the project’s networks but leaves named volumes alone, which is the behaviour you want. Add -v and it deletes those volumes too. There is no confirmation prompt and no undo. I have seen that flag reach production because somebody had it in a local teardown alias and pasted the alias into a runbook.

The practical rule: on a production host, restarting services is docker compose up -d or docker compose restart. down belongs in a maintenance procedure with a fresh backup, not in a daily habit.

Bind mounts deserve a mention too. They are convenient and they tie the container to a specific host path with specific host permissions. That is fine for config files you want to edit in place. For database data, a named volume gives you something Docker manages and something you can back up as a unit.

Failure family four: ports that bypass your firewall

This is the one that turns into a security incident rather than an outage.

When you write ports: - "5432:5432", Docker publishes that port on all host interfaces and inserts its own rules into the kernel’s NAT table to forward traffic to the container. Those rules are evaluated before the chains that a host firewall such as ufw or firewalld typically manages. The result is a database that is reachable from the internet even though your firewall config says otherwise, and a ufw status output that looks perfectly reassuring.

The fix is to stop publishing what does not need publishing:

services:
  db:
    image: postgres:16
    # No ports: block at all. Other services on the same Compose
    # network reach it as db:5432 by service name.
    networks: [backend]

  api:
    image: registry.example.com/api:1.4.2
    # Bound to loopback only. The reverse proxy on this host can
    # reach it; the internet cannot, regardless of firewall state.
    ports:
      - "127.0.0.1:8080:8080"
    networks: [backend, edge]

Two habits follow from this. Publish a port only when something outside the host genuinely needs it, and when you do, bind it to a specific interface. Everything else talks over the Compose network by service name. Then verify from somewhere else entirely, because checking from the host itself proves nothing.

While you are in daemon config: set default-address-pools as shown earlier. Compose creates a bridge network per project from a default range, and if that range overlaps your office LAN or your VPN subnet, you get routing that fails only for some people, only sometimes. It is a miserable thing to debug and a one-line thing to prevent.

Failure family five: secrets in environment variables

Environment variables are the default way to pass configuration to a container, and they are a mediocre way to pass secrets. They show up in docker inspect, they are readable by anything that can enumerate the process environment, they get inherited by child processes, and they land in crash dumps and error reporters.

The Compose specification supports file-based secrets without any orchestrator. Declare them at the top level and grant them per service; they appear inside the container as files under /run/secrets/:

services:
  db:
    image: postgres:16
    environment:
      # Official Postgres, MySQL and Redis images support the _FILE
      # convention: read the value from this path instead of the env var.
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

The file still lives on the host, so this is not a vault. What it buys you is that the value is not in the container’s environment, not in docker inspect output, and not in your Compose file. Set the file to mode 600 owned by root and keep it out of Git.

One more thing that trips people up: .env and env_file are different mechanisms. The .env file in the project directory feeds variable interpolation inside the Compose file. env_file passes variables into the container. Confusing them produces a service that starts fine with an empty config value and fails somewhere far from the cause.

Failure family six: health checks that report but never act

A container health check marks a container healthy, unhealthy, or starting. On a plain Docker host, that status is a label. Nothing restarts an unhealthy container. The restart policy only reacts to the process exiting.

So the classic silent failure is an app that has deadlocked, or lost its database pool, or wedged a worker thread. The process is alive, so the restart policy sees nothing to do. The health check goes red. Your monitoring, if it only checks that the container is running, sees nothing wrong. The service is down and every automated signal says it is fine.

Health checks are still worth defining, because they gate startup ordering and they gate --wait. Just be clear about what they do not do.

services:
  db:
    image: postgres:16
    restart: unless-stopped
    healthcheck:
      # pg_isready exits non-zero until Postgres accepts connections.
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      # Grace period. Failures during start_period do not count
      # toward retries, so a slow first boot is not marked unhealthy.
      start_period: 30s

  api:
    image: registry.example.com/api:1.4.2
    restart: unless-stopped
    depends_on:
      db:
        # Waits for db to report healthy before creating api.
        # This is a startup-order guarantee only, not a runtime one.
        condition: service_healthy
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 20s

Note restart: unless-stopped rather than always. The difference shows up after a host reboot: always will start a container you had deliberately stopped, unless-stopped respects that you stopped it. If you have ever taken a service down for maintenance and found it running again after a kernel update, this is why.

To close the gap, point an external check at the health endpoint. Whatever you already use for uptime monitoring is fine. The requirement is that something outside the host asks the application whether it is working, rather than asking Docker whether a process exists.

Troubleshooting a Compose stack that is misbehaving

The commands I reach for, roughly in order:

# The fully resolved config: overrides merged, variables substituted,
# anchors expanded. This is what Compose is actually going to run.
# Careful, it prints secret VALUES from interpolation.
docker compose config

# Health state and exit codes, not just up/down.
docker compose ps --all

# Why did it die? Look at the last lines before the restart.
docker compose logs --tail=200 --timestamps api

# The health check's own output, which is where the real error usually is.
docker inspect --format '{{json .State.Health}}' <container>

# Live resource use. Sudden memory growth before a restart means OOM.
docker stats --no-stream

# Did the kernel OOM-killer take it? Exit code 137 is the hint.
dmesg --ctime | grep -i "out of memory"

Two patterns worth recognising. A container that restarts every few minutes with exit code 137 was almost certainly killed for memory, either by a limit you set or by the host running out. And a service that works from inside the network but not from outside is nearly always a published-port or reverse-proxy problem, not an application problem, so check the plumbing before you read application code.

Common mistakes

• Deploying :latest. You cannot tell what is running, and a rebuild elsewhere silently changes what you get on the next pull. Pin a version tag, or pin a digest if you want it to be genuinely immutable.
• Keeping the version: key at the top of the file. Compose v2 ignores it and warns that it is obsolete. Delete it from your base file and from every override.
• No resource limits anywhere. One leaking service takes down every other service on the host. Limits turn a total outage into one restarting container.
• Running docker compose up from an SSH session and walking away. Without -d the stack is tied to your terminal. Use detached mode, and put the stack behind a systemd unit if you want it managed like other services on the box.
• The Compose file lives only on the server. If your production definition is not in Git, you have no history, no review, and no recovery when the disk dies.
• Pulling from Docker Hub anonymously in a deploy script. Anonymous pulls are rate limited per IP, so the deploy that worked all week fails on a busy afternoon. Authenticate, or mirror the images you depend on into your own registry.
• Automatic image updates on production. Tools that watch a registry and redeploy on their own are excellent for a homelab. On a system that matters, you want the update to happen when you are watching.

Best practices for Docker Compose in production

• One base file plus a production override. Keep shared service definitions in the base, and put published ports, resource limits, logging and restart policy in the override. Then docker compose -f compose.yaml -f compose.prod.yaml up -d is your deploy, and the diff between environments is a file you can read.
• Give every service a health check. It gates startup order, it makes --wait meaningful, and it gives you a status worth alerting on.
• Set memory and CPU limits on everything. Watch docker stats under real load first, then set the limit above observed peak with headroom. A limit set from a guess causes the outage it was meant to prevent.
• Segment your networks. An edge network for anything the proxy touches, a backend network for datastores. Only services that need to reach the database should be able to.
• Back up volumes, and restore one. A backup you have never restored is a hypothesis. Restore into a scratch stack on a schedule you actually keep.
• Make the deploy a script, not a memory. Pull, up with --wait, verify, prune. Six lines in the repo beats six commands somebody half-remembers.
• Run docker compose config in CI. It catches a malformed override before it reaches the server, and it costs nothing.

When to stop using Compose

Being straight about the limits is more useful than defending the tool. Compose is a single-host tool. The moment your requirement is “survive the loss of this machine”, you have outgrown it, and no amount of configuration closes that gap.

The signals I treat as a real trigger, rather than as an excuse to rewrite everything:

• You need more than one machine for availability, not just for capacity.
• You need automatic rescheduling when a host dies, without somebody logging in.
• You are scaling components independently and often enough that doing it by hand is a real cost.
• Several teams deploy to the same infrastructure and need isolation from each other.

If none of those are true, migrating to Kubernetes buys you a second system to operate and a longer list of ways to be paged. Plenty of profitable software runs on one well-configured box. The trade-off is real in both directions, and the honest answer depends on what your availability target actually is when written down.

Frequently asked questions

Is Docker Compose production ready?

For a single host, yes, provided you change the defaults that were chosen for development machines: log rotation, resource limits, port binding, named volumes, pinned image tags. Compose is not production ready in the sense of surviving a host failure, because it has no concept of a second host. That is a capability boundary, not a maturity problem.

How do I get zero-downtime deployments with Docker Compose?

Not from Compose alone. Put a reverse proxy in front, run two instances of the service under different names, start the new one, wait for its health check to pass, switch the proxy, then stop the old one. Traefik does the switching automatically based on labels; Nginx or Caddy need you to reload config. If the service is not customer-facing, a few seconds of downtime is usually the cheaper answer.

Does docker compose down delete my database?

Not by itself. Plain down removes containers and the project’s networks and leaves named volumes in place. Adding -v removes those volumes, and there is no prompt and no recovery. Anonymous volumes are the dangerous case, because you may not realise data is living in one until it is gone.

Why does my container keep restarting with exit code 137?

137 means the process received SIGKILL, and in containers that almost always means it was killed for memory. Either it hit a limit you configured, or the host ran out and the kernel OOM-killer chose it. Check docker stats for growth over time and dmesg for OOM entries. Raising the limit is the fix only if the memory use is legitimate; if it grows without bound, you have a leak and a higher limit just delays the restart.

Should I still write version: at the top of my Compose file?

No. Compose v2 validates against the current specification regardless of what that key says, and it emits a warning telling you the key is obsolete. Remove it from the base file and from every override file, otherwise the warning follows you around.

Can I use Docker secrets without Swarm?

Yes, using file-based secrets. Declare a top-level secrets block with a file: source, grant it to a service, and the content appears at /run/secrets/<name> inside the container. It is not a secrets manager, since the plaintext still sits on the host, but it keeps credentials out of the environment and out of docker inspect.

How many services is too many for one Compose file?

There is no hard limit, and the count matters less than the coupling. The signal to split is when a change to one service forces you to think about services that have nothing to do with it, or when one team’s deploy restarts another team’s containers. Compose profiles let you group optional services within a file before you commit to splitting into separate projects.

The one thing worth remembering

Running Docker Compose in production does not fail because Compose is a toy. It fails because the defaults are development defaults, and every one of them is fine right up until the day it isn’t. Unrotated logs, unlimited memory, published ports, anonymous volumes, health checks nobody watches. None of these announce themselves. They wait.

Go through your compose file once, deliberately, and ask of every service: where do its logs go, what is its memory ceiling, what happens to its data on down, who can reach its ports, and what tells you when it is unhealthy. That review takes an afternoon. It is the cheapest reliability work available to you, and it is most of the distance between a Compose stack that quietly burns you and one that just keeps running.

Need a second pair of eyes on your Compose setup?

I work with teams running containerised workloads on their own servers. Things I am usually brought in for:

• Reviewing a production Compose file and daemon config against the failure modes above, with a prioritised list of what to change first
• Building a zero-downtime deploy path with a reverse proxy and two service slots, wired into your existing CI
• Fixing disk exhaustion for good: log driver, rotation, image pruning, and alerting that fires before the filesystem does
• Sorting out network segmentation and published ports so the firewall config on the host is actually the firewall
• Volume backup and restore that has been tested by restoring, not just by running
• An honest assessment of whether you should move to Kubernetes or stay where you are, with the reasoning written down

If something in your stack is behaving oddly, send me the compose file, the output of docker compose ps --all, or the logs from whatever restarted last night. Easier to talk about a real config than a hypothetical one.

The post Docker Compose in Production: What Works and What Quietly Burns You appeared first on John Nessime.

The thing that needed the memory was Redis, installed three weeks earlier to speed up the admin, running with the configuration it shipped with. Redis defaults to no memory limit and a policy of never evicting anything. Left alone it will happily grow until the machine runs out, and then the kernel picks a victim, and it will not necessarily pick Redis.

That is the framing for everything below. Redis object caching for WordPress is genuinely one of the highest-value changes you can make to a busy site, and Redis out of the box is configured as a durable datastore, not a cache. You are storing data you can regenerate at any time. The defaults assume the opposite.

Setup takes ten minutes. The gotchas are the post.

What it does, and what it doesn’t

WordPress has an object cache built in, and by default it lasts exactly one request. Every page load fetches the same options, terms, post meta and user meta from MySQL, uses them, and throws them away. A persistent object cache keeps that data between requests, so the second visitor does not repeat the first visitor’s queries.

Two expectations worth setting before you install anything.

It is not a page cache. Page caching serves a finished HTML file and never runs PHP. Object caching runs the whole application and makes the database part faster. They live in different drop-ins, do different jobs, and coexist happily.

Its biggest effect is where page caching does not reach. Logged-in requests, wp-admin, WooCommerce carts and checkouts, REST API calls. If your front end is already cached and fast while the dashboard crawls, an object cache is aimed exactly at your problem.

Setup

1. Install Redis from your distribution’s packages, and bind it to localhost unless something remote genuinely needs it.
2. Install a PHP Redis client. PhpRedis (the PECL extension) is faster than the pure-PHP Predis and is what you want on a real server. Relay is a newer option worth knowing about if you are pushing volume.
3. Install the Redis Object Cache plugin, then add the constants below to wp-config.php before enabling the drop-in.
4. Enable it, which copies object-cache.php into wp-content/. That file, not the plugin, is what WordPress actually loads.
5. Verify it end to end, which is step five for a reason.

// wp-config.php, above the "That's all, stop editing" line.

define( 'WP_REDIS_HOST',     '127.0.0.1' );
define( 'WP_REDIS_PORT',     6379 );

// Two different protections, and you want both. The database
// number scopes flushes; the prefix prevents key collisions.
define( 'WP_REDIS_DATABASE', 1 );
define( 'WP_REDIS_PREFIX',   'acme-prod:' );

// Ceiling for keys written with no expiry, so nothing lives forever.
define( 'WP_REDIS_MAXTTL',   86400 );

One note if you are following an older tutorial: the prefix constant used to be WP_CACHE_KEY_SALT and was renamed to WP_REDIS_PREFIX in version 2.0 of the plugin, to stop it being confused with a core constant of the same name. The old name is still read for compatibility, but if a guide is still using it, that guide predates a major version and its other advice may too.

Then prove it works rather than trusting a green indicator on a settings page:

# The plugin's own view.
wp redis status

# Write through WordPress, read it back through WordPress.
wp eval 'wp_cache_set("probe","ok","test",60); echo wp_cache_get("probe","test");'

# Then confirm the key landed in the database and prefix you expect.
redis-cli -n 1 --scan --pattern 'acme-prod:*' | head

Site Health under Tools will also tell you whether WordPress considers a persistent object cache to be in use. That is the check to trust, because it reflects what core sees rather than what the plugin believes.

Gotcha 1: the defaults are a datastore’s defaults

The one from the opening paragraph, and the only one on this list that can take a server down.

# /etc/redis/redis.conf

# Without a limit, Redis grows until the machine runs out and the
# kernel kills something. It will not necessarily kill Redis.
maxmemory 512mb

# The default policy is to refuse writes rather than evict. Correct
# for a datastore, wrong for a cache: you want the least recently
# used keys dropped, silently, forever.
maxmemory-policy allkeys-lru

# Cache contents are regenerable by definition. Persisting them buys
# nothing and costs you fork latency on save and slower restarts.
save ""
appendonly no

Size maxmemory against what the machine can spare after MySQL and PHP-FPM have what they need, not against what is currently free. Free memory on a healthy Linux box is mostly page cache, and it is doing useful work.

The honest trade-off with allkeys-lru: under memory pressure Redis will evict things you would rather it kept, and cache misses go up. That is the correct failure mode. The alternative is write errors surfacing inside WordPress, or an outage.

If you are on managed hosting, do not change any of this. Ask them what they have set, because they may have tuned it and may not appreciate you undoing it.

Gotcha 2: one Redis, several sites

Redis is a single key-value store. Two WordPress installs pointed at it with default settings both write post_123 to the same key and read each other’s data. On a server hosting several sites that is a correctness bug. Where it gets genuinely nasty is staging.

Staging is usually a clone of production, which means it inherits production’s wp-config.php, which means it inherits production’s Redis settings. Now both environments share a namespace, and a developer clearing the cache on staging clears production’s, or worse, staging serves production’s cached objects.

The two constants guard against different things and you want both:

• WP_REDIS_DATABASE selects a logical database, of which Redis provides sixteen by default. This scopes flushes, so one site clearing its cache does not clear another’s.
• WP_REDIS_PREFIX namespaces every key. This prevents collisions if two installs ever land in the same database, which is what happens the day somebody copies a config file.

Make the prefix readable and environment-specific: acme-prod: and acme-staging:, not a random string. It is a namespace, not a secret, and you will want to read it in redis-cli at some point.

If you use Relay as the client, a dedicated database and a unique prefix per install is a requirement rather than a suggestion.

Gotcha 3: some things are never cached, and one thing is enormous

Not every cache group is persistent. WordPress marks some as non-persistent, and plugins can mark their own, so those groups keep behaving exactly as they did before you installed anything. If you expected a specific thing to get faster and it did not, this is a likely reason and it is not a misconfiguration.

The more interesting one is alloptions. WordPress stores all autoloaded options as a single cache entry. On a site with a bloated options table, that is one very large key, fetched on every request, and invalidated whenever any autoloaded option is written. Under concurrency that produces a lot of simultaneous re-priming of the same large value.

Which means an object cache does not excuse a bloated options table; it changes where the cost lands. Check what your keys actually look like:

# Reports the largest key per type. If one key dwarfs everything
# else, it is probably alloptions and your options table needs work.
redis-cli -n 1 --bigkeys

Gotcha 4: flushing is not free

Clearing the object cache on a busy site means every subsequent request rebuilds what it needs from MySQL simultaneously. On a quiet site nobody notices. On a busy one you get a load spike at exactly the moment you were trying to fix something.

Know what your flush actually does. Some implementations issue a Redis-wide flush; others delete only keys matching your prefix. Those are very different operations when the instance is shared, and finding out during an incident is a bad time.

Also worth deciding deliberately: whether deploys flush the cache. Stale cached objects after a plugin update cause genuinely confusing bugs, so a flush in the deploy script is often right. Just do it knowing it costs a load spike, and put it at the start of a quiet window rather than at five o’clock on a Friday.

Troubleshooting

“Drop-in: invalid” or “not connected”. Usually the PHP extension is missing, or PHP cannot reach Redis on the host and port you gave it. Check with php -m | grep redis and redis-cli ping from the same machine PHP runs on, which is not always the machine you are logged into.

Everything looks connected but nothing is cached. Something replaced object-cache.php. Optimisation plugins install their own drop-in, and only one can win. Check the file’s contents, not just that it exists.

The site got slower. Latency to Redis. A remote instance across a network adds a round trip to every cache operation, and there are a lot of them per request. Localhost or a Unix socket beats a fast network every time.

Stale data after an update. Flush once, then work out which plugin is caching something it should be invalidating. Do not put a flush on every request to make the symptom go away.

Memory keeps climbing. Either no maxmemory, or keys written with no expiry and no WP_REDIS_MAXTTL ceiling. Check INFO memory and the eviction counters.

Staging and production interfering. They are sharing a database number, a prefix, or both. Fix the config on staging as part of your restore script so it cannot recur.

Common mistakes

• Running Redis with default memory settings on a shared box.
• Leaving persistence enabled for a pure cache workload.
• No prefix and no database number, so sites read each other’s objects.
• Cloning production to staging without changing the Redis settings.
• Trusting the plugin’s status page instead of Site Health and an actual read-back.
• Expecting it to speed up an already page-cached front end.
• Installing it and ignoring an options table full of autoloaded junk.
• Predis on a production site when PhpRedis is available.
• Putting Redis on a different host and adding a network hop to every cache call.
• Flushing the cache during peak traffic to fix a display bug.
• Two plugins competing for the object-cache.php drop-in.

Best practices

• Set maxmemory and allkeys-lru before you point any site at it.
• Disable persistence unless Redis is also doing a job that needs it.
• Unique database number and readable prefix per install and per environment.
• Bind to localhost, or require a password and TLS if it must be remote.
• PhpRedis over Predis, and Redis on the same host as PHP where you can.
• A WP_REDIS_MAXTTL ceiling so nothing lives forever by accident.
• Verify with Site Health plus a write-and-read test, not a status indicator.
• Clean up autoloaded options rather than caching the mess.
• Monitor memory used, evicted keys and hit rate; a collapsing hit rate is an early warning.
• Make staging’s Redis config part of the restore script, not something to remember.

FAQ

Do I need Redis if I already have a caching plugin?

Probably, because they solve different problems. Page caching handles anonymous visitors; object caching handles everything that cannot be page cached, which is the admin, logged-in users and anything transactional. Run both.

Redis or Memcached?

For WordPress, Redis, mostly because the tooling and plugin ecosystem around it is better and you get richer diagnostics. Memcached is perfectly capable and slightly simpler. This is not a decision worth agonising over.

How much memory should I give it?

Start modestly, watch memory used and evicted keys for a week, and raise it if evictions are constant while free memory remains. Sizing from an article is guessing; sizing from your own eviction counter is not.

Is it safe to lose everything in Redis?

Yes, for an object cache. Everything in it is regenerable from MySQL, which is exactly why persistence is unnecessary. Losing it costs a burst of database load, not data.

Why did my dashboard not get faster?

Check the cache is genuinely active first, then look at what else is slow: outbound HTTP calls during admin requests, a large autoloaded options blob, or the heartbeat. An object cache removes repeated database work and nothing else.

Should staging share the production Redis?

It can, with a different database number and a different prefix. It is cleaner not to. Either way, set it as part of the restore process so a clone never inherits production’s cache namespace.

The one thing to remember

Redis ships configured to protect data it assumes you cannot afford to lose. You are storing data you could regenerate in a second. Every default that follows from that assumption, unlimited memory, no eviction, persistence to disk, is wrong for your use case and one of them can take the server down.

So set maxmemory and an eviction policy before you enable the drop-in, give every install its own database and prefix, and verify with Site Health rather than a green dot. Then it is genuinely one of the best changes you can make to a WordPress site.

Want it set up properly?

Object caching is quick to install and easy to get subtly wrong in ways that show up weeks later. Work I take on:

• Installing and tuning Redis object caching on a VPS or dedicated server, including memory limits, eviction and persistence settings.
• Auditing an existing setup for shared namespaces, competing drop-ins and caches that are not actually active.
• Multi-site and multi-environment configuration so staging can never touch production’s cache.
• Cleaning up autoloaded options so the object cache is caching something sensible.
• Monitoring for Redis memory, evictions and hit rate, wired into Prometheus and Grafana.
• Diagnosing sites where caching was installed and nothing got faster.

Send me your Site Health info and redis-cli INFO memory, and I will tell you what is worth changing.

The post The OOM Killer Took MySQL: Redis Object Caching for WordPress appeared first on John Nessime.

The application behind all of this is one Go binary and a Postgres database. It serves a few hundred requests a minute. Nobody has touched its code in a fortnight.

That is the moment worth noticing: when Kubernetes is overkill, the tell is not that anything is broken. Everything works. It is that the cluster has quietly become the thing you operate, and the product has become a tenant.

What it actually buys you

Worth being fair, because the case for Kubernetes is real and I reach for it regularly.

It gives you bin-packing across many workloads, so a fleet of services shares machines efficiently. It gives you a declarative API that a platform team can build on, so twelve product teams deploy without twelve bespoke pipelines. It gives you self-healing, rolling updates, horizontal autoscaling and namespace-level isolation as defaults rather than as things you script. And it gives you a portable vocabulary: an engineer who knows Kubernetes knows most of your infrastructure on day one.

Every one of those benefits scales with the number of services and the number of teams. None of them scale with how modern you want to feel.

The bill nobody itemises

The cost is not the control plane fee. It is the surface area you have agreed to keep current.

Kubernetes ships three minor releases a year and supports each for about fourteen months, and you can skip at most one minor version when upgrading. So there is a permanent, non-negotiable upgrade cadence: roughly every four months something needs planning, testing and a maintenance window, forever. Fall behind and you are doing multi-hop upgrades on an unsupported version, which is the worst combination available.

Underneath that sit the components nobody counts when they say “we’ll just run Kubernetes”: a CNI plugin, a CSI driver, an ingress controller, cert-manager, external-dns, a metrics pipeline, an autoscaler, RBAC, and whatever operators your databases need. Each has its own release cycle, its own breaking changes and its own opinions about the others. That is where the Tuesday goes.

Then there is debugging. A request that used to fail in one place now fails in one of several, and telling apart an application bug from a NetworkPolicy, a readiness probe, a DNS caching issue or a node under memory pressure is a genuine skill. On a good team that skill exists. On a three-person team it exists in one person’s head and goes on holiday with them.

Signals you don’t need it yet

• You have fewer than about five services, and most of them are one binary and a database.
• One team owns everything, so there is nothing to isolate from anyone.
• Traffic is predictable enough that you size for peak and stop thinking about it.
• Nobody on the team has run a cluster in production before.
• Your deploys are already fine. Nobody is complaining about them.
• You cannot name the specific problem Kubernetes is solving, only the general one.
• The cluster would run on two nodes, which means you have a scheduler for a decision with two possible answers.

Signals you probably do

• Multiple teams deploying independently and tripping over each other’s environments.
• Enough services that per-service deployment scripts have become their own maintenance problem.
• Genuinely variable load where autoscaling saves real money, not hypothetical money.
• A compliance or tenancy requirement that maps cleanly onto namespaces and network policy.
• You are already running an operator-based product, like a database platform, that expects Kubernetes.
• Somebody’s whole job is the platform, and it is not also somebody’s whole job to ship features.

That last one is the honest test. Kubernetes is not expensive to install. It is expensive to own, and ownership needs a name attached to it.

What I’d reach for instead

Roughly in order of how much you take on.

One server, systemd, and a reverse proxy. Underrated to the point of being contrarian, and completely adequate for a large number of real businesses. A VPS from Hetzner, InterServer or DigitalOcean, nginx or Caddy in front, systemd units for your services, automated backups off the box. Restart policies and health checks are built in; you are not going without them, you are just getting them from init instead of from a control loop.

Docker Compose on one box. The step up when you want containers without a scheduler. The entire deployment is a file you can read in one screen:

services:
  app:
    image: ghcr.io/acme/api:1.4.2      # pinned, deployed by changing this line
    restart: always
    ports: ["127.0.0.1:8080:8080"]     # proxy handles TLS and the public port
    env_file: .env
    depends_on: [db]

  db:
    image: postgres:16                 # pin the major, upgrade deliberately
    restart: always
    volumes: ["pgdata:/var/lib/postgresql/data"]

volumes:
  pgdata:

The honest limitation: one box is one box. Losing it is an outage, and a redeploy is a brief gap unless you put something in front. For plenty of internal tools and early products, that is an acceptable trade stated out loud rather than an oversight.

A platform that runs containers for you. Cloud Run, App Runner, Fly.io, Render, Railway. You hand over a container image and get scaling, TLS, rolling deploys and health checks without operating any of it. This is the option most teams should look at hardest, because it buys back the majority of what Kubernetes offers at almost none of the operational cost. You pay in per-unit pricing and in reduced control, and both are usually worth it below a certain scale.

ECS on Fargate, or Nomad. Real schedulers with a fraction of the concepts. If you need orchestration but not the ecosystem, these are legitimate destinations rather than waypoints on the road to Kubernetes.

And if you have decided Kubernetes genuinely is the answer, use a managed control plane, and consider a lightweight distribution such as k3s for small or edge deployments. Do not build the cluster yourself to save money. That is the most expensive saving in infrastructure.

Arguments that don’t survive contact

“We’ll need it eventually.” Possibly. Adopting it three years early costs three years of upgrade cycles to buy an option you may not exercise, and the migration is not much harder later than it is now.

“It’s the industry standard.” For orchestrating fleets, yes. Most applications are not fleets. Standard does not mean universally appropriate, and the same argument would put a load balancer in front of a single server.

“It avoids vendor lock-in.” Partly true and frequently oversold. You have swapped a cloud provider’s API for a distribution, a set of controllers, and a pile of YAML that encodes assumptions about all of them. Portability lives in your container images and your data, not in the scheduler.

“It’ll help us hire.” It helps you hire people who want to run Kubernetes. Whether that is the same as the people who will build your product is worth a moment’s thought.

You’re not painting yourself into a corner

This is the part that makes the decision easy to defer, and it is the most useful thing in this post.

The durable investments are containerising your application, externalising configuration into environment variables, keeping state out of the application and in a database or object storage, exposing a health endpoint, and logging to stdout. Do those and your app already satisfies most of what a Kubernetes manifest asks for. The scheduler underneath is genuinely swappable.

So the choice is not Kubernetes now or a painful rewrite later. It is Kubernetes now, or the same containers running somewhere simpler until the complexity is earned.

FAQ

How many services before Kubernetes makes sense?

There is no clean number, and anyone offering one is guessing. The better question is how many teams deploy independently. One team can coordinate without a platform. Four cannot, and that is when the abstraction starts paying for itself.

Is Docker Compose really production-ready?

On a single host, for a workload that tolerates brief restarts, yes. It runs plenty of real systems. What it does not give you is failover across machines, so be clear that losing the host means an outage and decide whether that is acceptable rather than discovering it.

Doesn’t managed Kubernetes remove the operational burden?

It removes the control plane, which is the part that was never your main problem. Ingress, storage, networking, upgrades and the add-on ecosystem all remain yours. Managed offerings genuinely help, and they narrow the gap rather than closing it.

What about autoscaling? I can’t get that elsewhere.

You can. Cloud Run, App Runner, Fly and ECS all autoscale, several of them to zero, with no cluster to maintain. Autoscaling is a common reason to reach for Kubernetes and one of the weakest, because it is the feature most thoroughly commoditised elsewhere.

How hard is it to migrate later?

Much easier than migrating off it. If your app is containerised, stateless, configured by environment and observable, moving to Kubernetes is mostly writing manifests. The hard parts of that migration are the ones you should be fixing anyway.

The one thing to remember

Kubernetes is not complicated by accident. It is complicated because it solves a genuinely complicated problem, and if you do not have that problem you are paying the complexity and collecting none of the benefit.

So make it a decision with a reason attached. Write down the specific thing you cannot do today, and check whether a cluster is the cheapest way to fix it. Sometimes it is. Often the honest answer is one server, a reverse proxy, and an afternoon spent on backups instead.

Trying to decide?

I get called in on both sides of this: teams drowning in a cluster they did not need, and teams who genuinely need one and are putting it off. Work I take on:

• A straight assessment of whether Kubernetes fits your team and workload, with the reasoning written down rather than an opinion delivered.
• Building the simpler thing properly: single-host or Compose deployments with TLS, health checks, deploys and backups that hold up in production.
• Moving off an over-provisioned cluster onto something proportionate, without downtime.
• Getting an application genuinely portable so the decision stays reversible: containerised, stateless, configured by environment, observable.
• Setting up managed Kubernetes properly when it is the right answer, including the upgrade process nobody plans for.
• Reviewing the cost, in money and in engineering hours, of what you currently run.

Tell me how many services you run and how many people deploy them, and I will tell you what I would build.

The post The Cluster Becomes the Product: When Kubernetes Is Overkill appeared first on John Nessime.