That conversation is usually the first sign that an organisation bought two Oracle products without deciding what each one was for. It is a common outcome, and it is not because the products are bad. It’s because they are sold as a suite, they overlap on purpose, and the sales conversation is about capability rather than authority.

This post is a working comparison of the four Oracle Construction and Engineering products people most often confuse: Primavera P6 vs Unifier vs Aconex, plus Textura. One profile each, what it genuinely owns, where it wins, where it doesn’t, and then the part nobody covers: how to decide which system holds the authoritative number when three of them offer a cost module.

The question that actually sorts them

Feature lists will not separate these products. All four manage documents. Three of them track cost. Two of them hold a schedule. If you compare capability columns you will conclude that you need all four, which is exactly the conclusion the suite is designed to produce.

The question that does separate them is: whose record is this, and who else needs to write to it?

Two of these products are single-organisation systems. You run them, you own the data, and outside parties get in only if you give them a seat and they accept your governance. One of them is deliberately multi-organisation, where every company on the project holds its own workspace and decides what it shares. And one is a transactional network that touches money leaving your bank account, which makes it a finance system wearing a construction badge.

Get that axis right and the rest of the decision mostly falls out.

Primavera P6: the schedule, and only the schedule

P6 is a critical path method scheduling engine with resource and cost loading on top. It exists to answer questions about time: what drives the finish date, what happens if this activity slips, what does the resource histogram look like if we accelerate.

Two things confuse people here. First, P6 comes in two shapes. P6 Professional is the desktop client that planners actually live in. P6 EPPM is the enterprise web layer with the shared database, portfolio views and web services. Most organisations run both against the same database, and most planners will fight you to keep the desktop client.

Second, Oracle Primavera Cloud is a separate product, not P6 in a browser. Oracle says so on its own product page, which is unusually direct for vendor copy. Primavera Cloud shares a lineage with P6 and is positioned as the cloud destination, but the data model is not identical and things planners rely on in P6 do not all have equivalents. If someone tells you a move to Primavera Cloud is a hosting change, they have not tried it.

Where P6 wins

• Contractual programme scheduling, where the schedule is a deliverable and gets audited
• Forensic delay work, because the XER and PMXML formats are what every expert and every third-party analysis tool expects
• Resource-loaded planning across a portfolio with a shared resource pool
• Anywhere the client’s contract names it, which is more places than you would expect

Where P6 doesn’t

• Cost control. P6 has cost fields and they are useful for cash flow shape, but a resource-loaded schedule is not a commitment ledger. It has no concept of a purchase order lifecycle, a variation approval or a payment certificate
• Correspondence and document control. There is nothing here for transmittals, RFIs or drawing registers
• Collaboration across companies. Subcontractors do not get P6 logins, and you would not want them writing to your programme if they did

Primavera Unifier: a business process engine that happens to know about projects

Unifier is the one people describe worst, usually as “the cost one”. That undersells it and also sets bad expectations.

Unifier is a configurable workflow platform for capital programmes. You define business processes: a change order, a funding request, an invoice, a payment application, a site instruction. Each one has a form, a workflow with approval steps and routing rules, and a defined effect on the cost sheet. Oracle ships preconfigured processes and a no-code designer called uDesigner for changing them or building your own. Alongside the business processes sit the managers, including the Cost Manager, Schedule Manager and Document Manager.

It’s sold as base products, principally Project Controls and Facilities and Asset Management, which matters because Unifier’s remit runs past handover into operating the asset.

The honest trade-off with Unifier is this: its configurability is the reason to buy it and the reason implementations run long. A platform that can model any approval workflow will not tell you what your approval workflow should be. If your organisation has not written its cost control process down, Unifier will faithfully automate the argument you are having instead of the process you wanted.

Where Unifier wins

• Owner-side capital programme control, where budget, funding, commitments, forecast and actuals need to roll up across many projects
• Anywhere approvals need an audit trail with defined routing, delegation and thresholds
• Public sector and utility programmes with fund tracking, where money has strings attached and the strings must be reportable
• Organisations that want one governed process applied to every project rather than per-project improvisation

Where Unifier doesn’t

• Supply chain collaboration. It’s your instance, your governance, your seats. External parties can be given access, but they are guests in your system and they behave like it
• Drawing and model control at scale. The Document Manager is real, but it is not a common data environment and it is not built for design coordination
• Fast starts. Configuration is a project in its own right, and treating it as an install is the most common way to burn a first year

Aconex: the project record, owned by nobody in particular

Aconex is a common data environment for correspondence, documents, workflows and models across every organisation on a project. Document control, transmittals, RFIs, submittals, drawing registers, model coordination, field inspections.

Its defining property is not a feature. It is the data ownership model. Every organisation on the project gets its own workspace and controls what it shares and when. That neutrality is the entire commercial argument, and it is the thing you cannot replicate by inviting subcontractors into your instance of something else.

This matters more than it sounds. A designer who believes the main contractor can read their internal drafts will keep their real work somewhere else, and your “single source of truth” quietly becomes a partial one. The reason Aconex accumulates a complete project record is that nobody is afraid of it. Oracle’s own materials lean on that neutrality hard, and in this case the marketing claim and the engineering consequence are the same thing.

On the model side, Aconex Model Coordination is design-neutral and built on open BIM standards including IFC and BCF, so it sits alongside the authoring tools rather than replacing them.

Where Aconex wins

• Multi-party projects where the correspondence record will be read by lawyers later
• Design-build and joint venture delivery, where no single party should hold everyone’s data
• Document and model volume, which Oracle’s datasheets describe as unbounded on data and participants
• Handover, because the audit trail and the linked document set are the deliverable

Where Aconex doesn’t

• Portfolio-level capital planning. Aconex is organised around projects, not around a programme of investment decisions
• Asset lifecycle after handover. The record ends where the project ends
• Deep enterprise cost governance. Connected Cost exists and is genuinely useful for project cost, but if your organisation needs funding, capital planning and multi-year rollup, that is Unifier’s shape

Textura: money leaving the building

Textura came into Oracle by acquisition and still feels different from the rest of the suite, because it is solving a different class of problem.

Textura Payment Management runs the construction payment cycle: initiating a draw, subcontractor pay applications, markups and approvals, owner billing, collecting lien waivers and compliance documents, enforcing compliance holds, payment authorisation and disbursement. It’s a network product. Your subcontractors transact in it, and the value comes from them being there.

The compliance side is the part that decides whether it’s worth buying. If your risk is a subcontractor’s insurance lapsing quietly and nobody noticing until a claim, an automated payment hold tied to a missing document is a control you cannot easily build with a spreadsheet and good intentions.

Where Textura wins

• General contractors with a wide subcontractor base and a monthly draw cycle that eats accounts payable alive
• Jurisdictions with real lien exposure, where waiver collection is a legal control rather than paperwork
• Compliance enforcement at the point of payment, which is the only moment anyone reliably pays attention
• Audit trails on who approved what, signed what and released what

Where Textura doesn’t

• It is not a cost control system. It handles the payment leg, not the budget, forecast or commitment ledger
• It does not replace your ERP. It integrates with it, and that integration is part of the implementation cost
• Adoption is a supply chain problem, not a software problem. If your subcontractors will not onboard, you own a slower process than the one you replaced

The overlap trap, and why the cost module is where it bites

Here’s the failure mode that is invisible at procurement and expensive eighteen months later.

Three of these products can hold cost. P6 has cost-loaded activities. Unifier has a full Cost Manager. Aconex has Connected Cost. Nobody sets out to run three cost systems, but it happens gradually: the schedule gets cost-loaded because the client asked for an S-curve, the finance team runs the commitment ledger in Unifier, and the project team starts logging variations in Aconex because that’s where the correspondence already is.

Now you have three forecasts. They disagree. Not because anyone is wrong, but because they are answering different questions with different cut-off dates and different accrual rules. And the reconciliation lands in a spreadsheet maintained by one person, which is the outcome every one of these products was bought to prevent.

The fix is a decision, not a tool. Before anything is configured, write down one sentence per data class: which system is authoritative, and what the others are allowed to hold.

• Schedule dates and logic: P6 is authoritative. Everything else receives dates, nothing else edits them
• Budget, commitments and forecast: pick one. If you own Unifier, it is Unifier. If you don’t, Connected Cost is a reasonable home, but say so out loud
• Correspondence and controlled documents: Aconex if it’s on the project, and if it is, do not run a parallel drawing register anywhere else
• Subcontractor payment and compliance status: Textura, with the resulting actuals flowing to whichever system owns cost
• Everything the authoritative system does not own: read-only, refreshed by integration, and clearly labelled in reports as a copy

That list takes an afternoon and saves a year. It’s also the single most common thing missing from Oracle Primavera implementations I get asked to look at.

Integration is the real cost, and it has a shape

Once you own two of these, you own an integration. Budget for it as a workstream, not a checkbox.

Oracle offers more than one path. Primavera Gateway is the established broker for P6 and Unifier, with built-in providers per application and a canonical intermediate format so mappings survive version changes. In the P6 to Unifier direction, Gateway maps the P6 Project ID to Unifier’s Project Number, and what flows depends on how the schedule is built: a duration-based schedule sends durations, a resource-loaded one sends durations and units. Oracle also has a lighter integration platform for exchanging data between Unifier, P6, Aconex and Primavera Cloud, so check which one your version and deployment model actually supports before designing anything.

Aconex Connected Cost takes a different route, talking to P6 over its web services API. Two constraints from that integration are worth knowing before you draw an architecture diagram:

• You can configure both a P6 and a Primavera Cloud integration for the organisation, but an individual project can only link to one of them at a time
• Self-hosted or third-party-hosted P6 servers have to be approved by Oracle and reachable over the internet before the integration can be established. Oracle-hosted P6 skips that approval step

That second one has teeth if you are running P6 EPPM on your own infrastructure. “Internet accessible” and “exposed” are not the same thing, and the gap between them is where your security review lives. If you’re standing up middleware or a reverse proxy in front of an on-premise P6 or Unifier deployment, that box needs to be treated as production: a dedicated host from a provider like Contabo or InterServer rather than a corner of an existing server, TLS terminated properly, access restricted to known networks, and remote administrative access over a business VPN such as NordLayer or Surfshark rather than a port forward someone opened for a consultant and forgot about.

Back up the database host properly too, and test a restore. On-premise P6 and Unifier estates are frequently the last thing anyone thinks about until they need it, and imaging tools from vendors like O&O Software earn their keep on exactly this kind of long-lived, rarely-touched server.

How I’d decide

Work through this in order and stop at the first answer that fits. Buying more than you need is far more common than buying too little.

1. Name the pain in one sentence. “We cannot defend our programme in a delay claim” points at P6. “We cannot tell the board what the capital programme will cost” points at Unifier. “We cannot find the approved revision” points at Aconex. “Subcontractor payments take six weeks and compliance is a filing cabinet” points at Textura
2. Count the organisations that must write to the system. One, and you are choosing between P6 and Unifier. Many, and you are choosing between Aconex and Textura
3. Ask whether the problem outlives the project. Assets, maintenance and multi-year capital planning are Unifier’s territory. Anything that ends at handover is not
4. Check what the contract already mandates. Plenty of clients name a scheduling tool or a project CDE in the contract documents. That decision is made, so spend your effort elsewhere
5. Ask the reseller how each product meters. Per named user, per project, per transaction or per value processed changes the architecture, not just the invoice. A per-user product pushes you towards fewer, more senior users, and that changes who does data entry
6. Only then decide on a second product, and write the system-of-record sentence for every data class the two of them share before signing anything

Arguments that don’t survive contact

• “Unifier can do document control, so we don’t need Aconex.” Technically true, practically false on multi-party projects. The obstacle is not features, it is that other companies will not put their real correspondence in your instance
• “Aconex has cost, so we don’t need Unifier.” Fine for project-level cost. It falls down when you need portfolio funding, capital planning and asset lifecycle
• “Primavera Cloud is just P6 in the browser.” Oracle itself states that Primavera Cloud is not P6. Treat any migration as a data model exercise with a gap analysis, not a lift and shift
• “We’ll integrate it later.” Later means after users have created data under two incompatible breakdown structures. Agree the project and cost breakdown codes on day one, even if the integration itself waits
• “The suite integrates out of the box.” Connectors exist and they are good. Mapping your codes, your workflow states and your calendars is still work, and it is work that needs someone who understands both sides

Practices worth adopting early

• Standardise the project ID before anything else. Gateway maps P6’s Project ID to Unifier’s Project Number, and a free-text naming convention will hurt on the day you connect them
• Keep the schedule in one place and let it flow outward. Bidirectional date editing across two systems is a support ticket generator
• Pilot Unifier configuration on one real process end to end before configuring twenty. uDesigner makes it easy to build the wrong thing quickly
• Onboard the supply chain before go-live on Aconex or Textura. Network products fail on adoption, not on uptime
• Export your own data on a schedule. XER and PMXML for schedules, and regular extracts from the cloud products, so you are not negotiating from zero at renewal
• Write the system-of-record statement into the implementation charter, where people will actually see it

FAQ

What is the difference between Primavera P6 and Unifier?

P6 is a critical path scheduling tool that answers questions about time and resources. Unifier is a configurable business process and cost platform that answers questions about money, approvals and governance across a capital programme. They overlap slightly because P6 can hold cost data and Unifier has a Schedule Manager, but neither is a substitute for the other.

Do I need Aconex if I already have Unifier?

If the project involves several independent organisations exchanging drawings and correspondence, usually yes. Unifier’s Document Manager works well for your own records, but external parties are guests in your instance. Aconex gives every organisation its own workspace and control over what it shares, which is what makes the project record complete rather than partial.

Is Oracle Primavera Cloud replacing P6?

Oracle positions Primavera Cloud as the cloud platform and publishes migration guidance from P6, while continuing to sell and support P6 EPPM. Oracle also states plainly that Primavera Cloud is not P6. Plan on the two coexisting for now, and treat any migration as a data model exercise with a proper gap analysis rather than a hosting change.

Can Textura replace our accounting system?

No. Textura handles the construction payment cycle: pay applications, approvals, lien waivers, compliance holds and disbursement. It integrates with ERP systems rather than replacing them, and that integration is a real line item in the implementation.

Which product should a general contractor start with?

It depends on which pain is loudest. If schedules are being disputed, P6. If the project record is scattered across email, Aconex. If the monthly draw cycle and subcontractor compliance are consuming the finance team, Textura. Contractors reach for Unifier less often than owners do, because Unifier’s strengths are programme-level funding and governance.

How hard is it to integrate P6 with Unifier?

The mechanism is well trodden through Primavera Gateway, which ships providers for both. The hard part is not the connector, it’s agreement: matching project identifiers, deciding which system owns dates, and reconciling breakdown structures that were designed independently. Expect the mapping work to take longer than the technical setup.

What data should I be exporting for portability?

For schedules, take regular XER and PMXML exports, since those formats are what independent analysis tools and delay experts read. For the cloud products, set up scheduled extracts of documents, transaction records and audit trails. The point is not distrust of the vendor, it’s that a project record you cannot read without an active subscription is a liability during a dispute.

The one thing worth remembering

The Primavera P6 vs Unifier vs Aconex comparison, with Textura alongside them, is not really a features question. Each of these products is good at the thing it was built for, and Oracle is not trying to trick anyone by giving three of them a cost module.

The damage comes from owning two of them and never deciding which one is authoritative. Sort the products by whose record they hold and how many organisations write to them, pick the smallest set that covers your loudest pain, and write down the system of record for every data class before a single workflow gets configured.

Do that and the monthly report matches the cost system. Skip it and you’ll meet the spreadsheet.

Need help untangling this?

Most of the work in this area is not clicking through screens, it’s the plumbing and the decisions around it. That part I can help with:

• Writing the system-of-record map for your data classes before you commit to a second Oracle product, so cost and dates have one owner each
• Designing and building integrations between P6, Unifier, Aconex, Textura and your ERP, including the field mapping and error handling nobody scopes
• Hardening on-premise P6 EPPM or Unifier deployments that need to be reachable for a cloud integration, with proper TLS, network restrictions and audited access
• Automating schedule and cost data extraction into a warehouse or BI layer, so reporting stops depending on someone manually exporting on a Friday
• Building repeatable backup, restore-testing and disaster recovery for self-hosted Primavera estates
• Second-opinion reviews on an implementation or integration design that already exists and is not behaving

If you’re stuck on something concrete, send it over: an integration mapping spec, a Gateway error, a network diagram for an on-premise P6 that needs to talk to a cloud service. Easier to give you something useful when I can see the actual thing.

The post Primavera P6 vs Unifier vs Aconex vs Textura: Which Oracle Product Do You Actually Need? appeared first on John Nessime.

The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

That is where most cost work goes wrong. The pressure is on the total, so people attack the biggest line item first. The biggest line item is almost always compute, and compute is the one place where a wrong guess shows up as latency at peak. Three weeks later the bill is lower and the p99 is worse, and nobody connects the two, because the change that caused it was a Tuesday afternoon instance swap that “went fine.”

This post is about how to cut AWS costs without breaking production by ordering the work differently: not by how much each change saves, but by how far the damage travels if the change is wrong. I will go through the levers in that order, with the traps in each one, the commands I use to find the waste, and the handful of cuts that look harmless and are not.

The cuts that break production are rarely the ones you fear

The line items that look most alarming on a bill are usually networking and storage overhead: NAT Gateway processing, public IPv4 hours, log ingestion, snapshots nobody owns. They feel dangerous to touch because they sit in the middle of the VPC. In practice, almost all of them can be removed or rerouted with zero behavioural change to your application. A gateway endpoint for S3 does not alter a single byte your code sees. Deleting an Elastic IP associated with nothing cannot affect anything, by definition.

The changes that feel safe are the ones with real blast radius. Dropping an instance one size. Switching a volume type. Moving objects to a colder storage class. Each is a single API call, reversible on paper, and each quietly narrows a performance envelope you were relying on without knowing it. The failure does not show up at deploy time. It shows up the next time you hit the load level that used to fit inside the old headroom.

So the rule I work by: sequence cost work by blast radius, not by savings size. Take everything production cannot possibly notice first. Then the things that move a performance envelope, one workload at a time, with a metric to watch. Then, last, the financial commitments, which cannot break your uptime at all but can lock you into paying for architecture you are about to throw away.

First, find out what you are actually paying for

The default Cost Explorer view groups by service, which is close to useless here. “EC2 – Other” is not a thing you can act on. It is a bucket holding EBS volumes, snapshots, NAT Gateway hours, NAT Gateway data processing, data transfer and public IPv4 charges, all mixed together.

Group by usage type instead

Usage type names the actual mechanism being billed. Strings like NatGateway-Bytes, NatGateway-Hours, PublicIPv4:InUseAddress and EBS:VolumeUsage.gp2 tell you what to go fix. This pulls last month’s usage types sorted by cost:

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 month ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --output json 
  | jq -r '.ResultsByTime[0].Groups[] | [.Keys[0], .Metrics.UnblendedCost.Amount] | @tsv' 
  | sort -k2 -gr | head -30

That uses GNU date, so run it on Linux unless you have coreutils on macOS. Cost Explorer API calls are themselves billed per request, which surprises people the first time they wrap this in a cron job.

Billing data cannot tell you where your bytes went

This is the most useful thing to internalise about networking spend. Cost data knows how many gigabytes crossed the NAT Gateway. It has no idea whether they were headed to S3, to ECR, to CloudWatch or to a third-party API. There is no destination breakdown in billing data.

VPC Flow Logs are the missing half. Turn them on for the private subnets, then query them grouped by destination address with Athena or CloudWatch Logs Insights and map the ranges back to AWS services. Now you have the split: this much to S3, this much to ECR, this much genuinely to the internet. That is what tells you which endpoint is worth adding. Flow Logs cost money to ingest, so scope them tightly and turn them off once you have the answer. A week on the right subnets is enough.

Tier zero: savings production cannot notice

Everything in this tier changes billing without changing behaviour. Do all of it before you touch a single instance type.

The orphan sweep

AWS charges for allocated resources, not used ones. A volume attached to nothing bills at full rate. An Elastic IP attached to nothing bills by the hour, and has done ever since AWS began charging for public IPv4 addresses whether or not they are in use.

# Elastic IPs associated with nothing
aws ec2 describe-addresses 
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' 
  --output text

# EBS volumes in the "available" state, meaning attached to no instance
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' 
  --output text

Run the same pass over load balancers with no healthy targets, snapshots whose source volume is gone, and old AMIs still holding snapshots behind them. Snapshot a volume before deleting it. The snapshot costs a fraction of the volume and buys you a rollback if something did care after all.

NAT Gateway is usually the largest risk-free win

A NAT Gateway bills three separate ways: an hourly charge for existing, a per-gigabyte charge for every byte it processes, and then whatever data transfer applies to where that traffic actually went. Teams look at the hourly rate, decide it is a rounding error, and never look at the processing line. The processing line is the one that grows with traffic.

The part that stings: traffic from a private subnet to S3 or DynamoDB in the same region is charged NAT processing even though it never leaves the AWS network. Gateway endpoints for those two services are free, with no hourly and no per-gigabyte charge. If your private subnets talk to S3 through a NAT Gateway, this is the best ratio of saving to risk available anywhere on the platform.

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

A gateway endpoint works by adding a prefix-list route to the route tables you name, so the main failure mode is naming the wrong tables and leaving traffic on the old path. Nothing breaks; you just do not save anything. The real caveat is policy: if you have bucket policies or endpoint policies that restrict access by source, check them first, because requests arriving through an endpoint present differently.

Interface endpoints for other services (ECR, CloudWatch Logs, Secrets Manager, SSM) are a different calculation. They charge per hour per availability zone plus per gigabyte, so they only pay off above a certain volume for that one service. Use the Flow Logs breakdown to decide. Pulling container images from ECR through NAT on every scale-out is the case that usually justifies one.

Log groups that never expire

CloudWatch Logs bills mostly on ingestion and secondarily on stored volume. A log group created without an explicit retention setting keeps data forever, so every debug group someone created during an incident years ago is still accruing storage.

# Log groups with no retention policy set
aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output text

aws logs put-retention-policy 
  --log-group-name /aws/lambda/my-function 
  --retention-in-days 30

Retention is the safe half. Reducing ingestion is a tier-one change, because you are removing evidence you might need during an incident. If you need a long retention window cheaply, exporting to S3 with a lifecycle rule beats paying CloudWatch storage rates indefinitely, and a log platform like Grafana Loki or a hosted equivalent gives you a cheaper long tail while CloudWatch keeps the recent window.

Incomplete multipart uploads

Failed large uploads leave parts behind in S3. Those parts are billed as storage and do not appear in a normal object listing, which is why some buckets bill for far more than they seem to contain. One lifecycle rule ends it:

{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Apply it with aws s3api put-bucket-lifecycle-configuration. That call replaces the entire lifecycle configuration on the bucket, so fetch the existing one and merge rather than overwrite.

Tier one: savings that move a performance envelope

Now a wrong call has consequences. Everything here goes one workload at a time, with a named metric you watch for at least a full traffic cycle afterwards.

gp2 to gp3, and the throughput trap

This is presented everywhere as a free win, and for most volumes it is. gp3 costs less per gibibyte and gives every volume a baseline of 3,000 IOPS regardless of size, where gp2 scales at 3 IOPS per GiB and only reaches 3,000 sustained at 1 TiB. For a small database volume quietly burning through burst credits, gp3 is a straight upgrade in both cost and stability.

The trap is throughput, not IOPS. gp3’s baseline throughput is 125 MiB/s. A large gp2 volume can deliver up to 250 MiB/s. Migrate a big sequential-read volume to gp3 with the defaults and you halve its throughput ceiling while the cost report still shows a saving. Log processing, backup jobs and analytics scratch space are exactly the workloads that hit this, and exactly the ones where the symptom is “the nightly job got slower” rather than an alert.

Check VolumeReadBytes and VolumeWriteBytes in CloudWatch first, and provision throughput explicitly where the volume needs it:

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 
  --volume-type gp3 --iops 3000 --throughput 250

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0 
  --query 'VolumesModifications[].[ModificationState,Progress]' 
  --output text

Elastic Volumes changes the type in place with the instance running, so there is no downtime. Two things to watch anyway. If the root volume is declared inline in a CloudFormation block device mapping, changing its type there can replace the instance rather than modify the volume; modify through the EC2 API instead. And there is a cooldown before a volume can be modified again, so plan on getting it right rather than iterating. On RDS the same change also runs without downtime, but the instance then enters a storage optimization state during which further storage changes are blocked.

Rightsizing without guessing

AWS Compute Optimizer reads CloudWatch metrics and returns recommendations with a confidence rating. Better than eyeballing CPU graphs, with one important gap: without the CloudWatch agent installed it cannot see memory. A JVM service sitting at 12% CPU and 88% heap looks like an obvious downsize candidate and is not.

aws compute-optimizer get-ec2-instance-recommendations 
  --query 'instanceRecommendations[?finding==`OVER_PROVISIONED`].[instanceArn,currentInstanceType]' 
  --output text

My working rules:

• Look at a window that includes your worst week, not a rolling average. Averages hide the peak that justifies the size.
• Change one dimension at a time, family or size, so you know what caused any regression.
• Treat burstable families as a separate decision. A T-family instance that exhausts CPU credits under sustained load either throttles or silently bills you for unlimited mode. Neither is what you wanted.
• Moving to Graviton usually saves more than downsizing x86, but it is an architecture change. Images need to be multi-arch and every compiled dependency needs to build. That belongs in a sprint, not in a cost-cutting afternoon.
• Write down the metric you expect to move and the threshold at which you revert, before you make the change.

S3 storage classes and the small-object trap

Intelligent-Tiering is the low-thought option and it works well for what it was designed for: objects of meaningful size with access patterns you cannot predict. It charges a small per-object monitoring fee, moves objects between tiers automatically, and applies no retrieval fee on the automatic access tiers.

Where it fails is object count. Objects below the eligibility threshold of 128 KB are not auto-tiered at all. They sit in the frequent access tier permanently while still paying monitoring. A bucket holding hundreds of millions of thumbnails, metadata records or small JSON documents can pay for monitoring that produces no saving whatsoever. Check average object size first, from S3 Storage Lens or an inventory report: total bytes divided by object count.

Lifecycle transitions have their own trap, since each transition is a billed request. Moving a billion small objects to a colder class can cost more in transition requests than the storage saving returns in a year. Transitions pay off on size, not on count. Expiry is the exception: deleting costs nothing and removes the storage entirely. While you are in there, check non-current versions in versioned buckets, which accumulate invisibly and need their own rule. And if your workload is genuinely egress-heavy, an object store with no egress charge such as Cloudflare R2 changes the arithmetic enough to model, in exchange for cross-provider latency and an operational split.

Tier two: commitments, which break budgets rather than uptime

Savings Plans and Reserved Instances cannot cause an outage. They are billing constructs. What they can do is hold you to a spend rate for one or three years after the architecture that justified it has changed. What I would want anyone to know before signing:

• Savings Plans cannot be cancelled during the term; AWS documentation is explicit. There is a limited return path with eligibility conditions and quotas per billing family, but plan as though there is no exit.
• Standard Reserved Instances can be listed on the Reserved Instance Marketplace, which is a real exit but not a guaranteed one, since somebody has to buy. Convertible RIs cannot be sold, only exchanged for a different configuration.
• Commit after rightsizing, never before. A plan applied to an oversized fleet locks in the waste and removes the incentive to fix it.
• Savings Plans do not reserve capacity. If you need a guarantee that instances will be available in a specific zone, that is an On-Demand Capacity Reservation, a separate mechanism the discount can then apply to.
• Savings Plans do not apply to Spot usage or to usage already covered by an RI. If much of your fleet is Spot, your committable baseline is smaller than the bill suggests.
• Compute Savings Plans survive architectural drift: family changes, size changes and moves between EC2, Fargate and Lambda stay covered. EC2 Instance Savings Plans and Standard RIs discount more deeply and tolerate drift poorly. Choose based on how stable your architecture actually is, not how stable you would like it to be.

Most teams land on a layered shape: commit at the baseline you are confident will still exist in a year, leave the variable top of the curve on demand or on Spot. Flat usage and a frozen architecture justify deeper, longer commitments. Mid-migration, mid-container-adoption or mid-anything, take the smaller discount for the shorter term.

One honest question worth asking here: does the workload need to be on AWS at all? Steady, non-elastic things like build runners, staging environments and internal tooling are often cheaper on a plain VPS from a provider such as Contabo or InterServer, while genuinely elastic workloads are often cheaper on AWS. The trade is operational, because you take back patching, backups and capacity planning that AWS was doing for you.

Troubleshooting: the bill went down and something broke

The hard part of diagnosing a cost regression is the delay. The change lands on Tuesday, the symptom appears at month-end batch time, and by then three other things have shipped. Work backwards from the symptom:

• Latency got worse but CPU is fine. Look at storage. VolumeQueueLength climbing with throughput flat at a round number means you are sitting on a provisioned ceiling, which is the gp3 default-throughput signature. On gp2, a falling BurstBalance is the tell, and it usually starts hours before anything is user-visible.
• Intermittent timeouts after rightsizing. Check whether the new instance type has lower network or EBS bandwidth, not just fewer vCPUs. Baseline bandwidth scales with size, and smaller sizes burst their network the way T instances burst CPU.
• Something stopped reaching a service. If you added VPC endpoints, check whether an endpoint policy is denying it. Interface endpoints also need the endpoint’s own security group to allow inbound from your workload, which is the most common reason a new endpoint appears to break connectivity.
• Objects are slow or missing from an application path. Check whether a lifecycle rule moved them into a class with retrieval latency. The Glacier tiers inside Intelligent-Tiering are opt-in for exactly this reason, and enabling them turns instant reads into restore requests.

Keep a change log for cost work specifically: a dated line per change, what you expected to save, what metric you were watching. Five minutes each, and it is the difference between diagnosing this in an hour and in a week.

Common mistakes

• Starting with the biggest number. Compute is the biggest line and the riskiest to touch. It is the last tier, not the first.
• Committing before rightsizing. This locks in the exact waste you were trying to remove, for years.
• Accepting gp3 defaults on high-throughput volumes. Cheaper storage, halved throughput ceiling, no alert.
• Enabling Intelligent-Tiering on buckets full of tiny objects. Monitoring fees on objects that will never move a tier.
• Turning off monitoring to save on monitoring. Understandable at 2am on a budget deadline, indefensible during the next incident. Cut cardinality and retention, not coverage.
• Doing it once. Cost drifts back within two quarters unless something structural stops it.

Best practices that make the savings stick

• Make cost visible before it is a crisis. AWS Budgets with alerts at a percentage of forecast, plus Cost Anomaly Detection, catch the runaway job on day two rather than on the invoice. Platforms like Vantage, CloudZero or Kubecost for per-namespace Kubernetes attribution earn their place once several teams share an account structure.
• Tag at creation, in the IaC. Tags applied afterwards do not backfill historical cost data. If it was not tagged when it was created, that spend is unattributable forever.
• Put non-production on a schedule. Dev and staging running nights and weekends is the most common waste in any account, and stopping them has no production blast radius at all.
• Make the cheap thing the default in the module. gp3, sensible log retention, gateway endpoints and lifecycle rules belong in your Terraform or OpenTofu modules, so the next environment is born correct rather than remediated later.
• Review commitment coverage on a schedule. Coverage and utilisation are different numbers and both matter. High coverage with low utilisation means you over-committed.

Frequently asked questions

What is the fastest way to cut AWS costs without breaking production?

Add gateway endpoints for S3 and DynamoDB, delete unassociated Elastic IPs and unattached EBS volumes, and set retention on log groups that have none. All four are billing-only changes with no behavioural effect on your application, and together they often account for a large share of the surprise on a bill.

Why is my “EC2 – Other” line item so large?

It is a mixed bucket: EBS volumes and snapshots, NAT Gateway hours and data processing, data transfer, and public IPv4 address charges. Regroup Cost Explorer by usage type rather than by service to see which of those is driving it.

Is migrating from gp2 to gp3 always cheaper?

No. gp3 has a lower per-gibibyte price, but IOPS above the included baseline and throughput above the included baseline are billed separately. A volume needing high provisioned throughput can cost more on gp3 than it did on gp2. Model the volumes that genuinely need extra performance instead of assuming a flat saving across the fleet.

Can I cancel a Savings Plan if my usage drops?

Assume no. AWS documents Savings Plans as non-cancellable for the term. There is a limited return mechanism with eligibility rules and quotas, and no secondary marketplace of the kind that exists for Standard Reserved Instances. Size the commitment to the baseline you are confident about, not the peak you hope for.

Should I use Spot Instances to cut costs?

For fault-tolerant, interruptible work such as CI runners, batch jobs and stateless workers behind a queue, the discount is hard to ignore. For anything holding state in memory or serving a request that cannot be retried, the interruption notice is short and the failure is abrupt. Spot is an architecture decision, not a billing toggle.

How do I find out where my NAT Gateway traffic is going?

Billing data cannot tell you. Enable VPC Flow Logs on the relevant subnets, query them with Athena or CloudWatch Logs Insights grouped by destination address, and map the results back to AWS service ranges. That gives you the split between S3, ECR, other AWS services and the genuine internet, which is what determines whether an endpoint pays for itself.

The one thing worth remembering

If you take one idea from this, take the ordering. You cut AWS costs without breaking production by sorting the work by blast radius rather than by savings size, and by accepting that the frightening-looking networking and storage line items are usually the safe ones, while the innocuous single-API-call changes are the ones that bite.

Do the tier-zero sweep first and completely. It is free, it is fast, and it buys you the standing to say “not yet” when someone asks you to shrink a production database on a Thursday. Then move one performance envelope at a time, with a metric and a revert threshold. Commit last, at the baseline you actually believe in.

Need a second pair of eyes on your AWS bill?

This is work I do regularly, and it goes faster with someone who has no attachment to how the account got this way. Things I can help with:

• A usage-type level breakdown of your bill, with savings ranked by blast radius, so you know what is safe this week and what needs a change window.
• NAT Gateway and data transfer analysis from VPC Flow Logs, ending in a specific list of which endpoints to add and what each is worth.
• EBS and S3 review: gp3 candidates with throughput checked properly, lifecycle rules, version cleanup, and storage class decisions based on your real object size distribution.
• CloudWatch and logging spend reduction that keeps the signal you need during incidents, including moving long-tail retention off CloudWatch storage rates.
• Savings Plan and Reserved Instance modelling against your actual baseline, including what happens to coverage if the migration you are planning goes ahead.
• Guardrails so it stays fixed: Budgets, anomaly alerting, tagging enforced in Terraform or OpenTofu, and cheap defaults baked into your modules.

If you want a concrete starting point, send me a Cost Explorer export grouped by usage type, or the output of the orphan sweep commands above, and I will tell you what I would touch first and what I would leave alone.

The post Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook appeared first on John Nessime.

The instance snapshot exported fine. The managed database did not. Neither did the load balancer, the CDN distribution, the static IP, or the DNS zone. AWS documents this plainly: the export feature covers instance snapshots and block storage disk snapshots, and nothing else. Every other piece had to be rebuilt by hand, and a maintenance window turned into an afternoon.

That is the part nobody prices in. When you choose compute for a small client workload, you are not just choosing where the process runs. You are choosing how much of the platform you will have to recreate when the client grows, leaves, or hands the account to somebody else.

This post compares Lightsail vs EC2 vs ECS for small client work: single sites, small APIs, internal tools, the kind of thing that runs on one or two boxes with no dedicated ops team behind it. Each option gets a profile with where it wins and where it doesn’t, then a decision procedure, then the arguments I hear most often that fall apart on contact with a real project.

Why the exit path matters more than the launch

All three services will host a small workload. The launch is not the hard part. What separates them is what happens eighteen months later when the requirements change.

Lightsail is the clearest case because its boundaries are documented. Instance and disk snapshots export to EC2 as an AMI and EBS snapshots. Managed databases, container services, distributions, load balancers, static IPs, and DNS records do not export at all. AWS also warns that a Linux instance built from an exported Lightsail snapshot can carry residual SSH keys from the Lightsail service, which you should remove.

So the question is not “which is cheapest for a small site.” It is “how much of this stack is portable, and who operates it after handover.”

Amazon Lightsail: where it wins and where it doesn’t

Lightsail bundles compute, SSD storage, and a monthly data transfer allowance into one flat plan price. There are no separate line items for the instance, the volume, and the egress. For a client who wants to know what the bill will be, that is worth a lot more than a theoretical discount.

Where it wins

• Predictable billing. One number per month, plus egress overage only if you blow through the allowance. Easy to quote, easy to hand to a client.
• The data transfer allowance is generous for content sites. Inbound and outbound both count toward it, but only outbound overage is billed, and transfer between Lightsail resources over private addresses in the same region is free.
• Console surface small enough to hand to a non-engineer. A client can restart an instance or take a snapshot without you, and without an IAM policy discussion.
• IPv6-only bundles exist and cost less than the IPv4 equivalents, which is a real saving if the workload sits behind Cloudflare or another proxy anyway.

Where it doesn’t

• Networking is deliberately shallow. Lightsail resources live in a Lightsail-managed VPC, and connecting to the rest of AWS means enabling VPC peering with the default VPC in the same account and the same region. Not a custom VPC. Not another account.
• Peering is a region-level switch, not a per-instance setting, and the CIDR ranges must not overlap. The route table on the Lightsail side is managed by AWS and you cannot edit it.
• The exit is partial, and the instance type list is short. There is no picking a memory-optimized family because the workload turned out to be memory-bound.
• No Savings Plans, no Reserved Instances, no Spot. The flat rate is the rate.

Worth saying plainly: if all you want is a flat-rate box, Lightsail competes with plain VPS hosts as much as with EC2. Providers like Contabo and InterServer sell the same shape of product, often with more RAM per unit of spend. Lightsail’s edge is that it sits inside an AWS account, so IAM, billing, and eventual migration to EC2 stay in one place. If the client will never touch another AWS service, that edge is worth less than it looks.

Amazon EC2: where it wins and where it doesn’t

EC2 is the general case: a real VPC, real subnets, real security groups, and every instance family AWS sells. You also get to build all of that yourself.

Where it wins

• Instance selection. If profiling shows the workload is memory-bound rather than CPU-bound, you change families instead of buying a bigger everything. For a PHP or Node app with an ARM-clean dependency tree, the Graviton families are usually the easiest price-performance win on the table.
• Commitment discounts. Savings Plans and Reserved Instances make a steady 24/7 workload meaningfully cheaper, and Spot makes non-critical work cheaper still.
• Full networking. Custom VPCs, private subnets, VPC endpoints, cross-account peering, Transit Gateway. If the client already has an AWS footprint, EC2 joins it without a workaround.
• Systems Manager. Session Manager gives you shell access without opening port 22 to the internet, and Patch Manager gives you a defensible patching story at handover. If you would rather have a private network path to admin interfaces than an AWS-native one, something like NordLayer or a self-hosted WireGuard box fills the same slot.

Where it doesn’t

• Every component is billed and configured separately: instance, EBS volume, snapshots, egress, Elastic IP, load balancer. The bill has more surface area, and so does the misconfiguration.
• Public IPv4 addresses are billed per hour, including on running instances. On a fleet of one that is noise. Across a dozen small instances it stops being noise.
• Data transfer out to the internet is metered per GB with no bundled allowance. A media-heavy site that would sit comfortably inside a Lightsail plan can produce a surprising egress line on EC2.
• You own the operating system: patching, kernel updates, log rotation, backup verification. Billable work if you are the consultant, abandoned work if you are not.

The mitigation for the egress point is the usual one: put a CDN in front. CloudFront or Cloudflare both change the shape of that bill substantially, because most bytes stop leaving the origin at all.

Amazon ECS: where it wins and where it doesn’t

ECS is two different products wearing one name, and conflating them is where most of the confusion comes from.

• ECS on Fargate bills per vCPU-hour and GB-hour of the resources you declare in the task definition, not the resources you use. Over-provision a task and you pay for the over-provisioning at full rate.
• ECS on EC2 means you run the cluster instances and ECS schedules containers onto them. Savings Plans and Spot apply again, and you own the host OS again.

Where it wins

• The deployment story is genuinely better. Rolling updates with health checks and automatic rollback are configuration, not a script you wrote and hope still works.
• Immutable artifacts. The image that passed CI is the image that runs. No drift from someone running a package update on the box in March.
• Fargate removes host patching entirely, which for a small team is a real reduction in unowned work.
• Scaling to and from zero is straightforward for scheduled or bursty jobs. A nightly batch task on Fargate costs nothing between runs.
• If the app is already containerized and CI already builds images, you are most of the way there.

Where it doesn’t

• The supporting cast costs more than the compute on small workloads. A load balancer, a NAT gateway, CloudWatch Logs ingestion, and a registry are the floor, and none of them scale down with your one small container.
• Tasks in private subnets pull every image layer through the NAT gateway, which bills per GB processed. Frequent deploys of a large image is a cost pattern that looks like nothing on a diagram.
• Persistent local state is awkward, so stateful workloads generally mean EFS or a managed database.
• Handover is harder. A client’s in-house developer can usually SSH into a box. Far fewer can debug a task that fails its health check before it produces a log line.

The NAT gateway problem has a standard fix, worth knowing before you price the work. Put interface endpoints in the VPC for the ECR API and the ECR Docker registry, plus a gateway endpoint for S3, because ECR stores image layers in S3 and that is where the bulk of the bytes actually move. Add interface endpoints for CloudWatch Logs and Secrets Manager if your tasks use them.

# Interface endpoint for the ECR Docker registry (image layer pulls)
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Interface 
  --service-name "com.amazonaws.$AWS_REGION.ecr.dkr" 
  --subnet-ids "$SUBNET_A" "$SUBNET_B" 
  --security-group-ids "$SG_ID" 
  --private-dns-enabled

# Gateway endpoint for S3: no hourly charge, and it carries most of the data
aws ec2 create-vpc-endpoint 
  --vpc-id "$VPC_ID" 
  --vpc-endpoint-type Gateway 
  --service-name "com.amazonaws.$AWS_REGION.s3" 
  --route-table-ids "$PRIVATE_RTB_ID"

Interface endpoints carry their own hourly charge per endpoint per availability zone, so this is a trade, not a free win. Below some volume of image pulls the NAT gateway is cheaper; above it, endpoints are. Work out which side you are on before you build either.

Lightsail vs EC2 vs ECS on the lines that move the bill

Compute is rarely the biggest number on a small AWS workload. These are the levers that actually decide it, and they are the ones I check first on any account I inherit.

• Egress. Lightsail bundles an allowance. EC2 and ECS meter it. This single difference can invert the comparison for a media-heavy site.
• NAT gateway. Charged per hour and per GB processed. On ECS with private subnets it is usually the largest fixed cost in the account. Lightsail has no equivalent line.
• Load balancing. Lightsail’s load balancer is a flat monthly price. An Application Load Balancer is an hourly charge plus a capacity-unit charge. AWS publishes traffic thresholds above which it recommends moving to an ALB; if you are near them, you have outgrown the simple option already.
• Public IPv4. Billed per address per hour on EC2 and on task ENIs. It is small per unit and easy to forget across an environment.
• Log ingestion. CloudWatch Logs bills on ingested volume. A chatty container at debug level can out-cost the container. Set retention on every log group at creation time, not later.
• Idle resources. Unattached static IPs and orphaned EBS volumes bill quietly forever. Reclaim them with an offboarding checklist, not a quarterly panic.

A decision procedure for Lightsail vs EC2 vs ECS

Run these in order. The first one that gives a clear answer is the answer.

1. Does this workload need to talk privately to other AWS resources? If it needs a custom VPC, a private RDS instance in a non-default VPC, cross-account access, or PrivateLink, Lightsail is out. Its peering only reaches the default VPC in the same account and region.
2. Who operates this in a year? If the answer is “the client’s office manager” or “nobody,” bias hard toward the simplest thing that works. That is usually Lightsail, or a managed platform outside AWS entirely.
3. Is the app already containerized, with CI that builds images? If yes, ECS is cheap to adopt. If no, containerizing it purely to use ECS is a project, and you should price it as one.
4. Is the load steady or spiky? Steady 24/7 favors a committed EC2 instance. Spiky or scheduled favors Fargate, because you stop paying between runs.
5. How much egress? Estimate monthly GB out. If it is large and cache-friendly, put a CDN in front and the question mostly dissolves. If it is large and not cacheable, Lightsail’s bundled allowance is a real argument.
6. What is the audit story? If the client needs detailed IAM boundaries, VPC flow logs, or GuardDuty coverage across the workload, you want a normal VPC. That means EC2 or ECS.

If you land on Lightsail and want to keep the door open, take manual snapshots on a schedule and confirm the export path works before you need it. The commands are short:

# Take a manual snapshot of the instance
aws lightsail create-instance-snapshot 
  --instance-name my-instance 
  --instance-snapshot-name my-instance-export-test

# Export it to EC2 as an AMI plus EBS snapshot, in the same region
aws lightsail export-snapshot 
  --source-snapshot-name my-instance-export-test

# Watch the export until the state reports completed
aws lightsail get-export-snapshot-records

Do this once at project start. Finding out that a Django or Ghost blueprint instance cannot be exported is a fact you want on day one, not cutover day.

If you need Lightsail to reach an RDS database in the default VPC, peering is a region-level toggle. Enable it, then confirm it rather than assuming:

aws lightsail peer-vpc --region eu-west-1
aws lightsail is-vpc-peered --region eu-west-1

You still have to allow the traffic on both sides. The security group on the AWS resource has to accept the Lightsail CIDR range, and the Lightsail instance firewall has to allow the return path. Peering enabled is not the same as traffic flowing, and that gap is where an afternoon disappears.

Whichever you pick, put monitoring on it on day one. CloudWatch covers the AWS-side metrics; Grafana Cloud or Datadog give you a better view across a mixed estate. The failure you are guarding against is not a dramatic outage, it is a slow degradation nobody notices until a client emails.

Arguments that don’t survive contact

“Lightsail is just EC2 with a nicer console”

Same infrastructure underneath, different networking model, and that is the part that bites. A Lightsail instance is not sitting in your VPC. Anything that assumes a normal VPC, from a security group reference to a VPC endpoint, needs a rethink.

“Start on Lightsail, migrate to EC2 when you outgrow it”

Half true. The instance migrates cleanly. The database, load balancer, distribution, static IP, and DNS do not. It is a real migration project, so plan for it as one rather than treating it as an upgrade button.

“Fargate is serverless, so it scales to zero and costs nothing when idle”

The tasks scale to zero. The load balancer, the NAT gateway, and the VPC endpoints do not. On a small workload the always-on infrastructure usually dominates, which is exactly backwards from the intuition the word “serverless” gives you.

“Containers are better practice, so ECS is the professional choice”

Containers are better practice for reproducibility, and that benefit is mostly delivered by the image and the CI pipeline, not by the orchestrator. A container running under systemd on one EC2 instance captures most of the win at a fraction of the operational surface. Reach for ECS when you need scheduling, rolling deploys, and health-check-driven replacement.

“Reserved Instances make EC2 cheaper, so always pick EC2”

Commitment discounts are worth real money on steady workloads. They are also a one or three year commitment on a client relationship that may not last that long. Committing on a client’s behalf is a conversation, not a default.

FAQ

Is Lightsail cheaper than EC2 for a small website?

Usually, for a steady small workload, because the bundle includes storage and a data transfer allowance you would pay for separately on EC2. The comparison flips when you can apply a Savings Plan or Reserved Instance to a long-running EC2 instance, or when you need an instance family Lightsail does not offer. Price both against your actual egress volume rather than comparing headline compute rates.

Can a Lightsail instance connect to an RDS database?

Yes, through VPC peering, with conditions. Peering is between your Lightsail VPC and the default VPC in the same account and region, a default VPC has to exist, and the CIDR blocks must not overlap. Enable it in every region where you have Lightsail resources you want connected.

Does everything migrate when I export Lightsail to EC2?

No. Instance snapshots and block storage disk snapshots export. Managed databases, container services, CDN distributions, load balancers, static IPs, and DNS records do not. Certain blueprint instances, including Django, Ghost, and cPanel & WHM, cannot be exported at all. Check your specific blueprint before you commit to a migration date.

Is ECS overkill for a single small container?

Often, yes. For one container with no scaling requirement, running it under systemd or Docker Compose on a single instance gets the same immutable-artifact benefit without a load balancer, a NAT gateway, or a cluster to reason about. ECS earns its place when you need rolling deploys with automatic rollback, or more than a couple of services.

Why is my Fargate bill higher than the vCPU-hour rate suggests?

Two reasons, usually. Fargate bills the CPU and memory you declared in the task definition, not what the container consumed, so over-provisioning is billed at full rate. And the surrounding infrastructure (load balancer, NAT gateway data processing, public IPv4 addresses, CloudWatch Logs ingestion) is charged separately and often exceeds the compute on small services.

Should I use Lightsail container service instead of ECS?

Reasonable middle option if you want containers with flat-rate billing and no VPC to design. The trade is the same as everywhere else in Lightsail: limited configuration, and container services are among the resources that do not export to EC2. Treat it as a destination, not a stepping stone.

The one thing worth remembering

Lightsail vs EC2 vs ECS is not a question about performance. All three will serve a small client workload perfectly well. It is a question about which constraints you are accepting and how expensive they are to reverse.

One site with no other AWS resources and a client-visible bill: Lightsail. Anything needing a private database, a custom VPC, or an existing AWS footprint: EC2. Already containerized, deploying weekly, with someone technical staying involved: ECS. And if nobody will operate it after handover, push back on all three and price managed hosting instead, because an unpatched instance with no owner is worse than a more expensive managed platform.

Lightsail trades networking depth and portability for a bill you can put in a proposal. EC2 trades operational work for control. ECS trades a higher fixed floor for deployment mechanics that stop being your problem. Pick based on who operates the thing in a year and how much has to be rebuilt when requirements change, not on which service sounds most professional in a kickoff call.

Need a second opinion before you commit?

Most of the cost and portability problems above are cheap to avoid at design time and expensive to unwind later. What I help with on exactly this decision:

• Sizing a Lightsail, EC2, or ECS deployment against real traffic and egress numbers instead of guesses
• Working out whether VPC endpoints or a NAT gateway is cheaper for your specific image size and deploy frequency
• Planning and executing Lightsail to EC2 migrations, including the parts that do not export
• Setting up VPC peering between Lightsail and RDS, and debugging why traffic still is not flowing after you enabled it
• Auditing an inherited AWS account for idle addresses, orphaned volumes, and log groups with no retention policy
• Writing the handover runbook so the next person can operate it without calling you

If you want a straight answer on which of the three fits, send me a month of Cost Explorer output or a task definition and I will tell you what I would change.

The post Lightsail vs EC2 vs ECS: What I Actually Pick for Small Client Workloads appeared first on John Nessime.

So you open the AWS DMS console. The task is green. Full load completed, change data capture running, zero errors. Data validation passed on every table in scope.

Both of those things are true at the same time, and that is the part that catches people out. An Oracle to Aurora migration almost never fails with a stack trace and a rollback. It fails with a green dashboard and a handful of rows that are subtly, permanently wrong.

This post is not a walkthrough of the console wizard. AWS documents that well enough. It covers the four families of failure I would go looking for before signing off on an Oracle to Aurora migration: large objects that arrive truncated, change capture that records less than you assume, the semantic gaps between Oracle and PostgreSQL that compile cleanly and behave differently, and the database state that DMS simply does not move for you. Then how to validate properly, and what to do when a task stalls.

Pick the Aurora engine before you pick the migration tool

Amazon Aurora comes in a PostgreSQL-compatible edition and a MySQL-compatible edition, and the choice shapes everything downstream. Do not let it get decided by whoever on the team happens to like MySQL.

Aurora PostgreSQL is the realistic default coming from Oracle, and the reason is procedural code. PL/SQL and PL/pgSQL are close enough that automated conversion gets you a long way: packages, cursors, exceptions, custom types, and analytic functions all have recognisable counterparts. PostgreSQL also has the orafce extension, which reimplements a chunk of Oracle’s built-in functions and packages inside a compatibility schema, so a lot of converted code can keep calling things it already calls.

The honest case for Aurora MySQL: if the Oracle database is essentially a table store with the business logic living in the application tier, and your team already runs MySQL in production, you will move faster on the engine you can operate at 3am. Migration difficulty is real, but so is the cost of running a database nobody on the rota understands. If your schema has no packages, no triggers worth keeping and no PL/SQL, that argument holds.

Where it falls apart is the moment you find twelve thousand lines of PL/SQL that someone will have to rewrite by hand. At that point the engine choice has stopped being an operational preference and become a rewrite project.

Schema conversion: SCT or the DMS console

There are two ways to convert the schema. The AWS Schema Conversion Tool is a desktop Java application that connects to both databases, reads the source objects, and emits target DDL plus an assessment report of everything it could not convert. DMS Schema Conversion is the same idea run from the DMS console using instance profiles and data providers, with no local install and no JDBC drivers to manage.

Start with the console version. It is less setup and the assessment report is the thing you actually want early: a list of objects with action items, so you can size the manual work before committing to a date. Fall back to the desktop tool when you need something the console flow does not cover, such as converting application SQL files or SQL*Plus scripts.

Either way, treat the assessment report as a project plan, not a progress bar. The red items are the schedule.

Failure family one: LOB columns that arrive shorter than they left

This is the invoice description problem, and it is the single most common way an Oracle to Aurora migration loses data without telling anyone loudly.

DMS gives you three ways to handle CLOB, NCLOB and BLOB columns:

• Full LOB mode moves every LOB regardless of size. DMS has no idea how big they are, so it moves them one at a time, piece by piece. Correct, and slow.
• Limited LOB mode asks you for a maximum size up front. DMS pre-allocates memory and loads LOBs in bulk, which is dramatically faster. Anything larger than the limit is truncated to the limit, and a warning goes into the task log.
• Inline LOB mode sets a threshold. LOBs under it go inline with the row; larger ones fall back to the full LOB path with a lookup against the source. Good when most of your LOBs are small and a few are not.

Read the middle one again. Limited LOB mode is the default answer to “the full load is too slow”, it is the mode most tutorials use, and it silently clips your data. The warning lands in the task log, mixed in with everything else, and nobody is reading task logs line by line on a table with four million rows.

Worse, the task still reports success. Truncation is not an error condition in DMS. It is expected behaviour for the mode you asked for.

Measure before you set the limit

The fix is boring: go and find out how big your LOBs actually are, per column, on the source. Run this against Oracle for every LOB column in scope.

-- Largest LOB values in a single column, biggest first.
-- dbms_lob.getlength returns length in characters for CLOB/NCLOB,
-- bytes for BLOB. Check the top rows, not the average.
SELECT dbms_lob.getlength(description) AS lob_length
FROM   app.invoices
ORDER  BY dbms_lob.getlength(description) DESC
FETCH  FIRST 10 ROWS ONLY;

-- Same thing expressed in KB, which is the unit the DMS
-- "Max LOB size (K)" setting expects.
SELECT MAX(dbms_lob.getlength(description)) / 1024 AS max_kb
FROM   app.invoices;

Two things to know once you have the numbers.

First, with Oracle as a source, DMS treats LOBs as VARCHAR data wherever it can, because bulk-fetching them is much faster than the LOB API. Oracle’s VARCHAR ceiling is 32K, so a limited LOB size under 32K is where the mode performs best. If your real maximum is comfortably under that, limited LOB mode is genuinely the right call.

Second, memory. During full load, DMS pre-allocates roughly the max LOB size multiplied by the commit rate multiplied by the number of LOB columns. Push the limit up and that product grows fast. When the replication instance cannot allocate it, it starts swapping, and your “faster” mode becomes slower than full LOB mode. If limited LOB mode is dragging, drop the commit rate before you drop the size limit.

Split the LOB tables into their own task

The pattern I reach for first: one task with limited LOB mode for the bulk of the schema, a second task for the handful of tables with genuinely large LOBs, using full or inline mode. You can also override task-level LOB settings per table inside the table mapping rules, which keeps it to a single task if you prefer.

{
  "TargetMetadata": {
    "SupportLobs": true,
    "FullLobMode": false,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 16,
    "InlineLobMaxSize": 0,
    "LobChunkSize": 64,
    "BatchApplyEnabled": false
  }
}

That is limited mode with a 16 KB ceiling. To use inline mode instead, set FullLobMode to true and give InlineLobMaxSize a non-zero value, which is the threshold below which LOBs travel inline.

One requirement that bites during ongoing replication: for CDC, tables with LOB columns need a primary key, because DMS looks the LOB value up in the source rather than reading it out of the redo stream. Full load does not care. CDC does. A LOB table with no primary key will load fine and then fail to replicate changes.

Failure family two: CDC that captures less than you think

Ongoing replication is what buys you a short cutover window. It is also where the assumptions hide.

DMS reads Oracle’s redo logs. For that to work at all, the source has to be in ARCHIVELOG mode with enough retention that DMS can still find the logs it needs after a pause, a network blip or an overnight stall. Retention is the one people get wrong: a task that stops for six hours against a two-hour retention window does not resume, it fails, and you restart from a full load.

Supplemental logging is not optional

Oracle’s redo logs, by default, contain enough information for Oracle to recover the database. That is not the same as enough information to reconstruct a row-level change for a different database. Supplemental logging is what closes that gap.

-- Minimal supplemental logging at the database level.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

-- Identification key logging: writes every column of a row's
-- primary key into redo on UPDATE, even when the key didn't change.
-- Without this, DMS cannot reliably match the row on the target.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

-- Per-table alternative, if you only need a subset of tables
-- and want to keep the source overhead down.
ALTER TABLE app.invoices ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

The case that actually bites is a table with no primary key and no unique index. For those, key-level logging gives DMS nothing to work with, and it needs supplemental logging on all columns so the before-and-after image of the row lands in redo. Miss that, and DMS cannot build a safe UPDATE or DELETE statement for the target. You will not notice during full load. You will notice when an update on the source produces no change on the target, or changes the wrong row.

DMS ships premigration assessments that check exactly this. Run them. There are individual checks for supplemental logging on keyed tables, supplemental logging on all columns for unkeyed tables, ARCHIVELOG configuration, the CDC method in use, target permissions, and triggers enabled on the target. It takes minutes and it replaces a category of guesswork.

LogMiner or Binary Reader

DMS can read redo two ways. Oracle LogMiner is an Oracle-side API; DMS calls it and Oracle does the parsing. Binary Reader is DMS’s own parser reading the redo files directly.

If you specify nothing, you get LogMiner. That is usually correct. It needs fewer privileges, it handles ASM without extra plumbing, and it supports Oracle features like encryption and compression that the direct parser does not cover. The grants are small:

GRANT EXECUTE ON DBMS_LOGMNR TO dms_user;
GRANT SELECT ON V_$LOGMNR_LOGS TO dms_user;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO dms_user;
GRANT LOGMINING TO dms_user;

Binary Reader earns its keep in one situation: a source generating redo faster than LogMiner can chew through it, or several concurrent migration tasks against the same database. LogMiner runs inside the source instance, so its CPU cost lands on the database you are trying not to disturb. Binary Reader shifts that work to the replication instance.

The trade-off is setup. Binary Reader needs additional privileges and file-level access to the redo logs, which means either a directory the replication instance can reach or a copy step. If your source is a busy production Oracle instance and CDC latency is climbing steadily rather than spiking, that is the signal to switch. Otherwise stay on LogMiner and spend the complexity budget elsewhere.

Whichever you pick, put CDC latency on a dashboard for the whole replication window, not just cutover day. The DMS task metrics publish to CloudWatch, and pulling them into whatever you already run for alerting, whether that is Grafana Cloud, Datadog or New Relic, is worth the twenty minutes. Source latency climbing while target latency stays flat means you are not reading redo fast enough. Both climbing together usually means the target cannot apply fast enough.

Failure family three: the semantics Oracle and PostgreSQL do not share

Schema conversion produces code that compiles. Compiling is not the same as behaving identically, and this is where a migration goes wrong in ways no tool flags.

Empty string is not NULL

Oracle treats the empty string and NULL as the same thing. PostgreSQL does not. This one difference propagates into comparisons, concatenation, unique constraints, and any procedural code that checks whether a value is “empty”.

-- PostgreSQL
SELECT '' IS NULL;        -- false
SELECT NULL || 'abc';     -- NULL
SELECT '' || 'abc';       -- abc

-- Oracle
SELECT CASE WHEN '' IS NULL THEN 'yes' ELSE 'no' END FROM dual;  -- yes
SELECT NULL || 'abc' FROM dual;                                  -- abc
SELECT '' || 'abc' FROM dual;                                    -- abc

Every one of those lines is a place a report can quietly change its numbers. Unique constraints are worse. Oracle stores an empty string as NULL, and NULLs are not compared for uniqueness, so it will happily accept many rows with an empty value in a uniquely constrained column. PostgreSQL sees zero-length strings, which are equal to each other, so the second one is rejected. Same schema, same data, different outcome, and the failure surfaces as an application error nobody can reproduce locally.

There is no switch for this. Either the application changes, or you wrap the affected functions, or you normalise on write with a trigger or a check constraint. Decide which, per column, before cutover. The orafce extension helps with function behaviour but it is not a global compatibility mode, and enabling the parts that change NULL semantics has its own consequences.

The rest of the list

• ROWID has no PostgreSQL equivalent. DMS Schema Conversion can emulate it with a bigint or a character varying column, but if application code depends on ROWID ordering or reuse semantics, emulation will not save you.
• SYSDATE and time zones. Conversion settings let you choose whether to emulate Oracle’s time zone handling or use native PostgreSQL behaviour. Native is faster. It is only safe if the database and the application genuinely run in the same time zone.
• TO_CHAR, TO_DATE, TO_NUMBER format masks. Oracle accepts parameters PostgreSQL does not. The converter emulates them by default, which is correct and slower. Turning emulation off is a per-codebase decision, not a global one.
• DUAL does not exist. PostgreSQL does not need a FROM clause, so most references just get dropped, but any dynamic SQL that builds a query string around DUAL needs finding.
• Synonyms have no direct equivalent. Search path adjustments and views cover most cases.
• Materialized views can be converted to real materialized views or to plain tables. The refresh semantics differ, so check what the reporting layer expects.
• NUMBER without a scale. An Oracle NUMBER column with no scale specified converts to something with a default, and if the source holds values wider than that default you get rounding or overflow. There is a premigration assessment specifically for this.

One data-level quirk worth knowing: if an Oracle column contains a NULL character (hex U+0000), DMS converts it to a space (U+0020) on a PostgreSQL target, because PostgreSQL cannot store it in a text column. Rare, but if you are migrating anything that stuffed binary into a VARCHAR, that is a real change to your data.

Failure family four: the state DMS never moves

DMS moves rows. It does not move everything that makes a database work, and the gaps are not obvious until you point traffic at the new cluster.

Sequences

Schema conversion creates the sequences. DMS copies the rows. Nothing advances the sequence counters to match the data that just arrived, so your target sequences sit at their starting value while the table already contains eight million rows.

The first insert after cutover collides with an existing primary key. Then the second. Then every write for the next eight million attempts.

Fix this after you stop replication and before you open traffic, not earlier, or you will do it twice.

-- Advance a sequence to match the data already loaded.
-- Run once per sequence, after CDC has stopped.
SELECT setval('app.invoices_id_seq',
              (SELECT max(id) FROM app.invoices));

-- Confirm before you open the doors.
SELECT last_value FROM app.invoices_id_seq;

Triggers on the target

If your converted schema has triggers and they are enabled during the load, they fire on every row DMS inserts. Audit tables get populated with a migration’s worth of fake activity. Derived columns get recalculated against half-loaded data. Anything that writes to another table doubles your write volume.

Disable target triggers for the load, re-enable at cutover. There is a premigration assessment that checks for enabled triggers on target tables in task scope, which is a decent backstop against forgetting.

Secondary indexes and foreign keys

Every secondary index present during full load is an index being maintained row by row while you are trying to move data as fast as possible. Foreign keys are worse, because DMS does not guarantee parent-before-child load order across tables.

Standard sequence: drop or disable secondary indexes and foreign key constraints before full load, recreate them afterwards, then let CDC catch up. Keep primary keys, since CDC needs them. Script the recreation and check it into version control alongside everything else, because rebuilding forty indexes from memory at 2am is not a plan.

Validation you can actually trust

DMS data validation compares source and target row by row and reports mismatches. It supports Oracle and PostgreSQL-compatible endpoints in both directions, so an Oracle to Aurora PostgreSQL path is covered. Turn it on.

Then understand what it will not tell you.

• It needs a primary key or unique index on both sides. Unkeyed tables are not validated, and those are exactly the tables most likely to have gone wrong.
• Collation and sort order differ between Oracle and PostgreSQL. When they disagree, validation reports failures on records that are actually fine, and the noise trains people to ignore the report.
• It does not run when the migration uses custom filtering, or when you are consolidating several source databases into one target.
• It costs real resources. Validation issues its own queries against both databases, on top of the migration traffic. Budget for that on the source.
• For LOB columns it compares checksums rather than values, using DBMS_CRYPTO on the Oracle side.

Because validation runs alongside migration, a task that stops takes validation with it. Validation-only tasks solve that: same endpoints, same table mappings, no data movement, running independently of the migration task. Use one when the validation load is hurting the source and you want to run it off-peak, or when you want validation to survive a migration task restart.

None of that replaces your own reconciliation. Before cutover, run business-level checks on both sides: rows per table, sums of every monetary column, min and max of every date column, distinct counts on anything you join on. Those catch things row comparison does not, because they answer the question the business actually asks.

And specifically for the LOB problem, compare maximum lengths, not just row counts.

-- On Aurora PostgreSQL, after the load.
-- Compare this against dbms_lob.getlength on the Oracle source.
SELECT count(*)                 AS row_count,
       max(length(description)) AS max_len,
       count(*) FILTER (WHERE description IS NULL) AS null_count
FROM   app.invoices;

If the target maximum is suspiciously round, exactly 16384 or exactly 32768, you have found your truncation.

Troubleshooting an Oracle to Aurora migration that stalls

Full load runs, then one table sits at zero

Almost always LOBs. Full LOB mode moves large objects one at a time with a source lookup per value, and on a table with a million large CLOBs that is a very long time with no visible progress. Check the table statistics for that table, then check whether it has LOB columns. Move it to its own task with limited or inline mode and an appropriate size.

CDC latency climbs and never comes down

Look at which latency is climbing. Source latency means redo is arriving faster than DMS can read it, which is the LogMiner to Binary Reader conversation, or a redo volume problem on the source. Target latency means Aurora cannot apply fast enough, which points at indexes still present, triggers still enabled, or an undersized writer instance.

Task fails on restart with missing archived logs

Retention. The logs DMS needed were removed before it came back. Increase archived log retention on the source to comfortably exceed your worst realistic outage, then restart from a fresh full load for the affected tables. There is no recovering the gap.

Updates on the source do not appear on the target

Supplemental logging, nine times out of ten. Check whether the affected table has a primary key or unique index, and whether logging is configured at the right level for that answer. The premigration assessments cover both cases.

Errors about a unique index on the target

DMS does not support replication to a table with a unique index built on a coalesce expression. If schema conversion produced one, or someone added it by hand to work around the empty-string problem, replication to that table will not work. Restructure the constraint.

Common mistakes

• Accepting the default limited LOB size without measuring the source. This is the truncation bug, and it has a specific, preventable cause.
• Treating a green task and a passed validation as proof of correctness. Both are true when data has been truncated.
• Skipping the premigration assessments because the wizard let you continue without them.
• Forgetting to advance sequences, then discovering it under production write load.
• Leaving triggers enabled on the target during full load and polluting audit tables.
• Assuming schema conversion handled empty strings. It converted the syntax, not the semantics.
• Setting archived log retention to the length of a normal outage rather than a bad one.
• Doing the first end-to-end rehearsal on cutover night.

Best practices

1. Run the assessment report first and size the manual work from it. The red items are your timeline, and they are better known in week one than week nine.
2. Inventory LOB columns before configuring anything. Maximum size per column, in KB, written down.
3. Separate the LOB-heavy tables into their own task. Different tables want different modes, and one global setting will always be wrong for someone.
4. Enable supplemental logging deliberately, per table class. Key-level for keyed tables, all-columns for unkeyed ones.
5. Drop secondary indexes and foreign keys for the load. Script the recreation, version it, test it.
6. Rehearse the whole cutover at least twice, including the sequence advance and the trigger re-enable. A scratch environment for the dry run does not have to be expensive: a temporary Aurora cluster you tear down afterwards, or a cheap dedicated box from somewhere like InterServer or Hetzner running a trimmed source copy, both work.
7. Run your own reconciliation alongside DMS validation. Counts, sums, min and max dates, maximum LOB lengths.
8. Keep the Oracle source running and readable for a while after cutover. The bugs that surface at week three are the ones you can only diagnose by comparing against the original.

Frequently asked questions

Should I migrate Oracle to Aurora PostgreSQL or Aurora MySQL?

Aurora PostgreSQL for anything with meaningful PL/SQL, because PL/pgSQL is close enough that automated conversion does most of the work, and the orafce extension covers a chunk of the rest. Aurora MySQL is defensible when the schema is plain tables, the logic lives in the application, and your team already operates MySQL. The deciding question is how many lines of procedural code you would have to rewrite by hand.

Can AWS DMS migrate stored procedures and packages?

No. DMS moves data. Schema and code conversion is a separate job, handled by DMS Schema Conversion in the console or the AWS Schema Conversion Tool on the desktop. Both convert most objects automatically and produce an action-item list for the rest.

Why is my DMS task green when data is missing?

Because truncation in limited LOB mode is expected behaviour, not an error. DMS writes a warning to the task log and carries on. Task status reflects whether the task is running, not whether the data is faithful. Compare maximum LOB lengths between source and target to catch it.

Do I need downtime for an Oracle to Aurora migration?

Some, but it can be short. Full load plus CDC means the target stays current while you test, so the outage is limited to stopping writes, letting CDC drain, advancing sequences, re-enabling triggers, recreating indexes and repointing the application. Rehearse it and that usually fits a maintenance window rather than a weekend.

Does DMS migrate sequences?

It copies row data, not sequence state. Schema conversion creates the sequence objects; you advance them yourself after stopping replication, using setval against the current maximum key value. Skipping this produces primary key collisions on the first write after cutover.

LogMiner or Binary Reader for Oracle CDC?

LogMiner unless you have a reason. It is the default, needs fewer privileges, and handles ASM, encryption and compression without extra work. Move to Binary Reader when redo volume is high enough that LogMiner cannot keep up, when you are running several tasks against the same source, or when the CPU cost of parsing on the source database is unacceptable.

Will DMS data validation catch every problem?

No. It needs a primary key or unique index on both sides, it can report false failures where Oracle and PostgreSQL collations disagree, and it does not run with custom filtering or multi-source consolidation. Treat it as one layer and add business-level reconciliation on top.

The one thing worth remembering

An Oracle to Aurora migration does not usually announce its failures. The task goes green, validation passes, and the damage shows up weeks later as a truncated field, a duplicate key, a report whose totals moved slightly.

So build the migration around proving correctness rather than proving completion. Measure your LOBs before you set a limit. Configure supplemental logging for the tables that have no key, not just the ones that do. Assume the semantic gaps between Oracle and PostgreSQL are still there after conversion, because they are. Advance the sequences. Then reconcile at the level the business cares about, not just row by row.

Green means the pipe is open. It does not mean what came out the other end is what went in.

Need a second pair of eyes on your Oracle to Aurora migration?

Most of the work on these projects is not running the wizard. It is finding the handful of things that will be wrong afterwards. That is the part I help with:

• Reviewing DMS task settings and table mappings for LOB truncation risk, per column, before you run a full load you cannot easily repeat.
• Working through the schema conversion assessment report and turning the red items into an actual estimate rather than a wall of warnings.
• Setting up supplemental logging and CDC correctly for tables with no primary key, including the LogMiner versus Binary Reader call for your redo volume.
• Writing the cutover runbook: index recreation scripts, sequence advance, trigger re-enable, ordered, tested, and timed against a real rehearsal.
• Building the reconciliation layer that sits on top of DMS validation, so you can sign off on the data with something better than a green tick.
• Post-cutover triage when something on Aurora behaves differently from Oracle and nobody can work out which layer changed.

If you have a task settings JSON, an assessment report, or a validation failure log you cannot make sense of, send it over and I will tell you what I see in it.

The post Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover appeared first on John Nessime.

Nothing about the database had changed. Same engine, same instance class, same subnet group. What changed was the resource’s name inside the module. The author had tidied up a resource "aws_db_instance" "this" into something more descriptive, shipped it as a minor release, and moved on. Terraform saw a resource at an address that no longer existed and a resource at an address that did not exist before. Its default reading of that is destroy and create.

That is the real cost of a module, and it is invisible on day one. The first apply is always fine. Modules feel great the week you adopt them. The bill arrives at the third upgrade, when the person who wrote it has moved teams and a dozen repositories consume it.

This post is about judging that cost before you pay it. What separates reusable Terraform modules that are genuinely worth adopting from the ones your team quietly pins at an old version and refuses to touch, how to make your own modules land on the right side of that line, and what to do when a plan comes back looking like the one above.

The bill for a module arrives at upgrade time

Terraform correlates state to configuration by address. Not by tags, not by cloud resource ID, not by anything the provider knows. Just the address: module.database.aws_db_instance.this. Change any part of that path and, unless you say otherwise, Terraform concludes the old object should go and a new one should take its place.

For a security group that is an annoyance. For a database, an object store, or anything holding data, it is an incident with a retro attached.

The fix has existed for a long time and it is genuinely good: the moved block. A module author who renames something declares the rename in code, ships it with the release, and every consumer’s next plan updates state instead of destroying infrastructure.

# Shipped inside the module, in the same release as the rename.
# Terraform rewrites the state address instead of planning a replacement.

moved {
  from = aws_db_instance.this
  to   = aws_db_instance.primary
}

The block is declarative and it chains. If a resource moves in one release and moves again two releases later, someone jumping several versions at once still lands correctly, because Terraform resolves the whole chain. That is why a module that keeps its moved blocks around is doing you a favour, and why one that strips them out after a release or two is quietly making every skipped upgrade riskier.

So here is the whole thesis, and everything below is elaboration: a module is worth reusing if the next version bump is cheap. Everything else is secondary.

Five signals that reusable Terraform modules are worth adopting

These are the things I actually check before letting a module into a codebase. None of them take more than a few minutes. All of them predict the upgrade experience better than star counts do.

1. It keeps state addresses stable, or tells you when it does not

Open the repository. Search for moved. Read the changelog for the last major release and see whether renames are called out.

A module with no moved blocks and no changelog is not necessarily bad code. It just means every internal refactor becomes your problem, discovered in a plan output, usually under time pressure. A module that says “v3.0 renames the launch template resource, here are the moved blocks” is telling you it has thought about consumers.

2. The input surface is narrow and typed

Count the variables. If a module needs forty inputs to cover its use cases, the abstraction was drawn in the wrong place. Usually that means two patterns got merged into one module because they looked similar from a distance.

Width is not the only tell. Look at how the inputs are typed. A module that takes twelve loose strings is very different from one that takes three typed objects with sensible defaults baked in.

variable "logging" {
  description = "Access log settings. Set to null to disable logging."
  type = object({
    bucket         = string
    prefix         = optional(string, "logs/")
    retention_days = optional(number, 30)
  })
  default = null
}

variable "environment" {
  description = "Deployment environment."
  type        = string
  nullable    = false

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of dev, staging or prod."
  }
}

Three things are doing work there. optional() with a default means a caller who wants the simple case writes two lines instead of six. nullable = false means a caller cannot pass null and get a confusing failure three resources deep. The validation block means a typo fails at plan time with a sentence a human wrote, rather than as a provider error about an invalid parameter.

That last one matters more than it looks. Every validation rule a module author writes is an error message you do not have to debug.

3. You can pin it, and the pin means something

This trips people up because the two common source types behave differently.

# Registry source. The version argument applies here.
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"   # allows any 5.x, refuses 6.0
}

# Git source. There is no version argument. Pin with ref.
module "internal_network" {
  source = "git::https://git.example.com/infra/tf-network.git?ref=v2.3.1"
}

The ~> operator only allows the rightmost specified component to increase. Written against a major and minor, as above, it accepts patch and minor releases and blocks the next major. Written as ~> 5.0.1 it accepts patches only. Which one you want depends entirely on whether you trust the author’s reading of semantic versioning, and the honest answer for third-party modules is usually “not enough to auto-accept minors in production”.

For Git sources, pin to a tag, not a branch. A branch reference resolves to whatever the tip happens to be at init time, which turns your infrastructure into a moving target and makes a plan from two weeks ago unreproducible.

4. It has tests you can run without a cloud account

Terraform’s built-in test framework uses .tftest.hcl files and the same HCL you already write. The critical detail for module authors is that a run block set to command = plan creates nothing. No credentials, no resources, no teardown, no cost. It just checks that the configuration produces what you claimed.

# tests/validation.tftest.hcl
# Runs on plan only. Nothing is created and nothing is billed.

run "rejects_unknown_environment" {
  command = plan

  variables {
    environment = "produciton"
  }

  expect_failures = [var.environment]
}

Negative tests like that are the cheapest quality signal in the whole ecosystem. They prove the validation rules actually fire, which is the difference between a rule that catches typos and a rule that was written once and silently broken later.

OpenTofu runs the same format through tofu test, and additionally recognises a .tofutest.hcl extension for tests that should only run there. Useful if you maintain modules that target both tools.

5. Somebody answers issues

Least technical, most predictive. Open the issue tracker, sort by recently updated, and look at whether maintainers reply. A module with a hundred open issues and active triage is in far better shape than one with six open issues and no comments since a provider major release.

Provider major versions are the stress test. When the AWS or Azure provider ships a breaking change, every module has to react. Modules without a maintainer simply stop being usable, and you find out when init refuses to resolve a version constraint.

Three kinds of module, and which ones actually pay off

Not everything that can be a module should be. It helps to name the three shapes you see in the wild.

The thin wrapper

One resource, a handful of variables passed straight through, maybe a tag merge. This is the most common module and usually the least valuable. It adds a layer of indirection, a version to track and a state address prefix, in exchange for saving four lines.

There is one case where it earns its place: when the wrapper is enforcing something, like mandatory encryption, a naming convention, or a tag schema your finance team depends on. Then it is not a wrapper, it is a policy with a friendly interface. If it enforces nothing, inline the resource and move on.

The composition module

Five to fifteen resources that only make sense together. A VPC with its subnets, route tables and gateways. A service with its task definition, target group, listener rule and log group. This is where modules earn back their cost, because the value is in the wiring, and the wiring is exactly what people get wrong when they copy and paste.

Most modules worth reusing live here.

The landing zone

An entire environment behind one module call. Accounts, networking, IAM baseline, logging, the lot. These are genuinely powerful and genuinely expensive. They only work when a team owns them, tests them, and has a release process. A landing zone module maintained by one person as a side project becomes the single scariest thing in the repository, because nobody can upgrade it and nobody can replace it either.

Public modules or your own?

The case for public modules is stronger than the “we have specific needs” crowd usually admits. The community AWS collections in particular have absorbed years of edge cases you have not thought of yet: the ordering constraints, the optional sub-resources, the arguments that only apply in certain regions. Reimplementing that from the provider docs is a real project, not an afternoon.

The case against is equally real. Popular public modules are wide by necessity, because they serve everyone. That width shows up as a large input surface, a lot of conditional logic, and majors that shuffle internals. You inherit an upgrade cadence you do not control.

My rough split: use public modules for undifferentiated infrastructure where the community has clearly solved it, and write your own for anything encoding a decision your organisation made. A VPC is undifferentiated. Your service deployment pattern is not.

There is a licensing dimension worth knowing about. Terraform moved from an open source licence to the source-available Business Source License, which is what prompted the OpenTofu fork now under the Linux Foundation. Module code itself is unaffected by that change, and the two tools read the same HCL, so modules generally work with either. OpenTofu runs its own registry rather than HashiCorp’s. If your organisation has strict open source licensing requirements, that distinction is the one to escalate, and it is a legal question rather than a technical one.

For sharing modules internally, a private registry gives you discovery and a version list in one place. HCP Terraform includes one, and the run platforms built around Terraform and OpenTofu, such as Spacelift, Scalr and env0, all offer module catalogs with policy enforcement attached. You can also get most of the benefit from tagged Git repositories on GitHub or GitLab and a README, which is where I would start before paying for anything. If you self-host your Git server and CI runners, they need somewhere to live, and a small VPS from a provider like InterServer, Hetzner or DigitalOcean handles module CI comfortably, since the plans are cheap and the tests never leave the plan phase.

Making your own modules worth reusing

If you are on the authoring side, this is the order I would do things in.

1. Write it inline first, twice. Do not modularise a pattern you have seen once. The second copy tells you which parts vary and which parts are actually fixed. That is the module’s interface, discovered rather than guessed.
2. Extract with moved blocks in the same commit. Moving a resource into a module changes its address. If the resource holds data, that is the difference between a refactor and an outage.
3. Type the inputs properly. Objects with optional() defaults over flat strings, nullable = false where null makes no sense, validation blocks with error messages that name the fix.
4. Prefer for_each over count for anything collection-shaped. With count, removing an element from the middle of a list shifts every index after it, and Terraform plans a rebuild of resources you never touched. Keyed addresses do not shift.
5. Generate the docs. terraform-docs reads your variables and outputs and writes the README table. Wire it into CI so the docs cannot drift from the code, because hand-written input tables are always out of date.
6. Lint and scan on every PR. terraform fmt -check, terraform validate, tflint for provider-specific mistakes, and a policy scanner such as Checkov or Trivy for the security defaults. All of these run in seconds on a plain GitHub Actions or GitLab CI runner.
7. Tag releases, and mean it. A major for anything that changes an address or removes an input. A minor for new optional inputs. Consumers pin against your tags, so the tag is a promise.

One thing I would skip early on: building a full integration test suite that applies real infrastructure. It is slow, it costs money, and it needs credentials in CI. Plan-mode tests plus a policy scanner catch most of what actually breaks. Add applied tests later, for the two or three modules where the wiring is genuinely subtle.

Troubleshooting the plans that scare you

“Forces replacement” after a version bump

Before you argue with it, find out what it wants to delete. Save the plan and read it as JSON rather than scrolling terminal output.

terraform init -upgrade
terraform plan -out=tfplan

# List every address the plan intends to delete.
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.change.actions[] == "delete") | .address'

If the list is empty, the replacement was an argument change, not an address change, and you should read the diff on that resource. If the list contains things you did not expect, compare the addresses against the new module version’s source. A rename with no moved block is the usual answer, and you can write the moved block yourself in your root configuration as a workaround.

A module with a provider block inside it

Modules should declare which providers they need, not configure them. A module that contains a configured provider block rather than just a required_providers entry limits what callers can do with it, including using for_each or count on the module call. It also makes the module harder to destroy cleanly. If you hit that, the fix is on the module side: move provider configuration up to the root and pass configuration aliases in.

Changes to the module source are not picked up

Remote modules are downloaded into the working directory during init. Changing a ref or a version constraint in your configuration does not re-download anything on its own. Run terraform init -upgrade. If your CI caches the module directory between runs, that cache is the first place to look when a pipeline plans against code you are sure you deleted.

Registry version list looks stale

A registry’s list of versions is a synced view of your Git tags, not an independent database. Reconnecting a VCS integration, moving repositories, or pushing many tags at once can leave the registry out of step with the repository. Treat the tags as the source of truth and check the registry’s view after any change to that integration.

Common mistakes

• Pinning to a branch. ?ref=main is not a pin. It is an unannounced deployment scheduled for whenever someone next runs init.
• Modularising after one use. You end up encoding the first use case’s assumptions as the abstraction, then bolting flags onto it for every case after.
• Copying environments instead of parameterising them. Three near-identical directories drift within a quarter, and the fix that went into dev never reaches prod.
• Passing every provider argument straight through. If a module’s variables are a mirror of the resource schema, it is not an abstraction, it is a rename with extra steps.
• Deleting moved blocks too early. They only cost you a few lines. Removing them is safe only when you are certain every consumer has applied past that version, which you can rarely be sure of for a public module.
• Applying a plan you did not read. Nearly every module horror story ends with someone approving a plan whose destroy count they never looked at.

How I decide whether to adopt a module

Roughly ten minutes, in this order:

1. Count the input variables. Over about thirty, I assume the abstraction is wrong and look for a narrower alternative.
2. Grep for moved. Its presence tells me the author thinks about consumers.
3. Read the last major release notes. Are breaking changes explained, with an upgrade path?
4. Check for a tests directory and whether CI runs on pull requests.
5. Look at the issue tracker’s recent activity, not its total count.
6. Adopt it in one non-production configuration first, then bump it once deliberately and watch the plan. That single bump tells you more than the other five steps combined.

And a standing rule regardless of the module: never let a plan with a non-zero destroy count through review without someone saying out loud what is being destroyed and why.

Frequently asked questions

How many input variables is too many for a Terraform module?

There is no hard limit, but a wide input surface is a symptom rather than a style choice. If covering your use cases needs dozens of inputs, the variation between callers is probably larger than what they share, which means you have found two patterns rather than one. Splitting into two narrower modules usually reads better and upgrades more safely than adding a twelfth boolean flag.

Should I use public modules or write my own?

Both, for different things. Public modules are a good trade for undifferentiated infrastructure where the community has already absorbed the edge cases. Write your own for anything that encodes a decision specific to your organisation, because that is the part nobody else will maintain for you and the part that changes on your schedule rather than someone else’s.

Do Terraform modules work with OpenTofu?

Generally yes. A module is just a directory of HCL, and both tools read the same language and use compatible provider plugins, so registry modules, private modules and Git-sourced modules usually work unchanged. Where you can hit friction is version constraints and features that have diverged between the two projects since the fork. Test one workspace before committing an estate to a migration.

How do I pin a module version from a Git source?

The version argument only applies to registry sources. For Git, append a ref query parameter to the source URL and point it at a tag or a commit SHA. Branch names resolve to whatever the tip is at init time, which defeats the purpose of pinning entirely.

What happens if a module removes a resource in a new version?

Terraform plans to destroy it, because the resource is in state and no longer in configuration. Sometimes that is exactly what the author intended. Sometimes it means functionality moved somewhere you have not adopted yet. Read the release notes before applying, and if the object holds data, take a snapshot first regardless of what the notes say.

Can I test a Terraform module without cloud credentials?

Yes, for a useful subset. Run blocks set to command = plan evaluate the configuration without creating anything, which is enough to assert on computed values, defaults and validation failures. Testing behaviour that only exists after creation still needs an apply and therefore real credentials, which is why most module suites are mostly plan tests with a small number of applied ones.

Do I need a private registry to share modules internally?

No. Tagged Git repositories and a source URL cover the mechanics completely. A registry buys you discovery, a browsable version list and a place to attach policy, which starts mattering once enough teams are publishing that people cannot find what already exists. Below that point it is overhead you do not need yet.

Conclusion

Most advice about reusable Terraform modules focuses on the writing: structure, naming, README quality. That is the easy part, and it is not where things go wrong.

The modules that are actually worth reusing are the ones you can still upgrade in two years. Stable state addresses, a narrow typed interface, a pin that means something, tests that run on plan, and a maintainer who replies. If you only remember one thing, remember what a module is really promising you: not that the first apply will work, but that the next version bump will not wake anyone up.

Need a second pair of eyes on your module library?

I work with teams on the boring, high-consequence parts of Terraform and OpenTofu estates. Things I help with regularly:

• Auditing an existing module library and saying plainly which modules earn their keep and which should be inlined back into root configurations
• Adding moved blocks and upgrade notes to in-house modules so version bumps stop being incidents
• Narrowing wide module interfaces with typed objects, optional() defaults and validation rules that fail with useful messages
• Setting up module CI: fmt, validate, tflint, policy scanning and generated docs on every pull request
• Writing terraform test suites that run on plan, including negative tests that prove your validation rules actually fire
• Planning a Terraform to OpenTofu migration across a module estate, or working out whether it is worth doing at all

Send me a module’s variables.tf and the plan output from your last version bump. That pair is usually enough for me to tell you where the trouble is.

The post Terraform Modules Worth Reusing (And the Ones That Cost You) appeared first on John Nessime.

Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.

This post compares GitLab CI vs GitHub Actions vs Jenkins on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest “where it wins and where it doesn’t”, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.

The part of the decision that actually bites

Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.

Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.

The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system “for now”, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.

So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the “below the line” half cheapest for your specific situation.

What each platform is really optimising for

Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.

GitHub Actions: shortest path from push to green

Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else’s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.

A minimal hardened job header looks like this, and every line in it is there for a reason:

name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"

The permissions block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The concurrency block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.

Where it wins: your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.

Where it doesn’t: anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.

GitLab CI: one platform owning the whole lifecycle

GitLab’s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.

The pipeline language rewards you for using its graph features rather than treating stages as a queue:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh

The workflow:rules block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. needs turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. interruptible: true makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project’s CI settings; the keyword alone does nothing. And expire_in is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.

Where it wins: you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.

Where it doesn’t: your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.

Jenkins: whatever you need, on hardware you control

Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor’s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.

pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}

Two things in that file are worth pointing at. timeout and buildDiscarder are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And timestamps() and cleanWs() both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.

Where it wins: air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.

Where it doesn’t: the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.

GitLab CI vs GitHub Actions vs Jenkins on cost

Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.

• GitHub Actions. Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.
• GitLab CI. Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why “pay for seats, run your own runners” is such a common GitLab shape.
• Jenkins. No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.

Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.

Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.

Security is where the differences stop being cosmetic

Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.

• Untrusted pull requests. This is the classic GitHub Actions trap. The pull_request trigger runs fork code without access to your secrets. The pull_request_target trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request’s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub’s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.
• Expression injection. Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.
• Self-hosted runners on public repositories. Don’t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.
• Mutable action and image references. Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.
• Credential scope. Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.
• Jenkins specifically. The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project’s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.

A decision procedure you can run in an afternoon

1. Follow platform gravity first. Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.
2. Check whether a runner can reach the deploy target. Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.
3. Name the on-call owner. Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.
4. Model the compute, not the sticker price. Take last month’s build minutes, split them by operating system and machine size, and apply each platform’s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.
5. Inventory what the plugins actually do. If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The “needs writing” pile is your real migration estimate.
6. Trial with your ugliest pipeline. Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.

The hybrid pattern nobody puts on the slide

A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.

Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.

Arguments that don’t survive contact

• “Jenkins is legacy.” It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.
• “We’ll save money with self-hosted runners.” On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.
• “GitLab CI is just GitHub Actions with different YAML.” The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.
• “We’ll stay vendor-neutral by keeping all logic in shell scripts.” Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.
• “The bigger marketplace wins.” Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.

How I’d decide

• Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.
• Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.
• Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.
• Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.
• Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.

Frequently asked questions

Is Jenkins still worth using?

Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.

Which is cheaper, GitLab CI or GitHub Actions?

It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab’s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.

Can I use GitLab CI with a GitHub repository?

There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.

How long does migrating from Jenkins to GitHub Actions or GitLab CI take?

Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.

Are self-hosted runners safe?

They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.

Do I need to pin GitHub Actions to a commit SHA?

For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.

Is it reasonable to run more than one CI system?

Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.

The one thing worth remembering

Comparing GitLab CI vs GitHub Actions vs Jenkins on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.

Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.

Need help choosing or moving a CI pipeline?

This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:

• A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.
• Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.
• Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.
• Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.
• Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.
• CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.
• Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.

If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.

The post GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later appeared first on John Nessime.

That framing is where most of these arguments go sideways. On one box serving a normal website, all three of these servers will handle your traffic without breaking a sweat. The web server is almost never the slow part. The difference shows up months later, at 2am, in who gets paged and why.

So this post compares Nginx vs Apache vs Caddy on the three things that genuinely change your on-call life: how each one behaves when it runs out of capacity, who owns the certificate lifecycle, and where configuration lives when someone other than you needs to change it. Then a decision procedure, the mistakes I keep running into, and an FAQ.

Why the benchmark comparison picks the wrong winner

Almost every throughput comparison you’ll find measures static file serving on an idle machine with no application behind it. That is a real measurement of a thing that is not your bottleneck. If you’re proxying to PHP-FPM, Gunicorn, or a Node process, the request spends milliseconds in the web server and everything else in your code and your database.

The honest exception: if you are running a CDN origin, terminating TLS at a high connection rate, or serving genuinely large volumes of static assets, the connection handling model does show up in your bill. That is a real workload and Nginx has earned its reputation there. It just isn’t most projects.

For everyone else, the useful question is not “which is fastest” but “which chores am I choosing to keep.” That’s what the rest of this is about.

Failure family one: the process model decides how you fall over

This is the invisible one. Nobody notices it during a migration and everybody notices it during a traffic spike.

Apache: a hard worker ceiling, and a queue behind it

Apache’s behaviour depends entirely on which Multi-Processing Module is loaded. With prefork, you get one process per connection, capped by MaxRequestWorkers. When you hit that cap, new connections don’t get rejected. They sit in the kernel’s listen backlog waiting for a worker to free up.

The symptom is the worst kind: the site “hangs.” Load average is unremarkable. The application log is quiet because the requests never reached the application. Your uptime check times out but a manual curl from the box works fine. People spend hours looking at the database.

Check which MPM you’re actually running before you theorise about anything:

# RHEL, AlmaLinux, Rocky
httpd -V | grep -i "Server MPM"

# Debian and Ubuntu
apache2ctl -V | grep -i "Server MPM"

The reason prefork survives is mod_php. It isn’t safe with Apache’s threaded MPMs, so PHP stacks historically pinned themselves to prefork and inherited its concurrency profile. The Apache project itself now points people at PHP-FPM over mod_proxy_fcgi instead, which frees you to run mpm_event and gets you HTTP/2 as a side effect, since mod_http2 disables itself under prefork.

If you inherit an Apache box, that migration is usually the single highest-value change available to you, and it does not require leaving Apache.

Nginx: cheap connections, different limits

Nginx runs a small number of worker processes, each with an event loop. An idle keepalive connection costs a file descriptor and a little memory, not a process. Ten thousand of them is not a crisis.

Its ceiling is worker_processes multiplied by worker_connections, plus whatever your file descriptor limit allows. When you hit it, Nginx says so plainly in the error log rather than silently queueing. That’s a genuine operational advantage: the failure announces itself.

The trade is that when Nginx is proxying, saturation moves downstream. You stop seeing worker exhaustion and start seeing 502s and 504s from an overloaded PHP-FPM pool or app server. The web server is fine; you now have to go find the real limit.

Caddy: goroutines, and a softer edge

Caddy is a Go binary and handles connections with goroutines that the runtime multiplexes onto OS threads. Practically, that sits between the other two: connections are cheap like Nginx, but you don’t tune a worker count at all.

The failure mode is correspondingly softer and, honestly, harder to reason about. Instead of a clean “workers exhausted” line, you get memory growth and garbage collection pressure. There’s no knob you turned wrong, which is pleasant right up until you need to explain what happened.

Failure family two: who owns the certificate lifecycle

This is the one that actually causes outages, and it is the sharpest genuine difference between the three.

With Nginx or Apache, certificate management is a separate moving part. Certbot or another ACME client writes new files to disk on a timer, and the running server keeps serving whatever it loaded at startup. If nothing tells it to reload, the certificate on disk is valid and the certificate on the wire is expired. Users see a browser interstitial while your monitoring, if it checks the filesystem, reports everything fine.

The fix is to attach the reload to the renewal itself, not to a separate cron entry that might drift:

# Run the renewal end to end without touching real certificates.
# This exercises the challenge and the hook, which is the point.
certbot renew --dry-run

# Attach the reload to renewal so it only fires when a cert actually changed
certbot renew --deploy-hook "systemctl reload nginx"

The distinction that matters: --deploy-hook runs only when a certificate was actually renewed, so you’re not reloading the server twice a day for nothing. A blanket post-hook reloads regardless.

Caddy removes this category of failure. Certificate issuance and renewal are built into the server itself, it swaps the certificate in memory when it renews, and there is no external hook to forget. Point several instances at the same storage backend and they coordinate rather than each hammering the CA. If you have ever been paged for an expired certificate, this alone is a serious argument.

It is not free of foot-guns, though. On-demand TLS, where Caddy obtains a certificate during the handshake for a hostname it has never seen, is genuinely useful for multi-tenant setups where customers point their own domains at you. Enabled without restriction, it is an open door: anyone who points a DNS record at your IP can make you request a certificate. Caddy’s own documentation is blunt that it must be both enabled and restricted, and the mechanism is an ask endpoint you host:

{
    on_demand_tls {
        ask http://localhost:9123/ask
    }
}

https:// {
    tls {
        on_demand
    }
    reverse_proxy 127.0.0.1:3000
}

Caddy asks that endpoint whether a hostname is allowed before it will issue. It should be a fast indexed lookup against your own tenant table, returning in milliseconds, because a TLS handshake is being held open while it answers. The older interval and burst rate-limiting options are no longer recommended and should be removed if you still have them in a config.

One more Caddy default worth knowing: the admin API listens on localhost port 2019 and accepts a full config replacement without authentication. That is fine bound to loopback and catastrophic if you helpfully expose it while debugging a container.

Failure family three: where the configuration lives

This is the axis people underweight, and it’s usually the one that actually decides the answer.

Apache supports .htaccess: per-directory configuration, read at request time, changeable by anything that can write to the directory. No reload, no root access. That is why the entire shared hosting world is built on Apache, and why control panels like cPanel and DirectAdmin, and hosts like InterServer, assume it. A WordPress plugin can rewrite its own rules and it just works.

The costs are real and they’re the mirror image of the benefit. With overrides enabled, Apache walks the directory tree looking for .htaccess files on every single request. And your security rules can be silently overridden by whatever gets deployed into a directory, by anyone who can deploy.

Nginx has no equivalent and this is deliberate. Everything is central, and a change requires a reload. Rewrites are handled with try_files rather than per-directory rule files. If you own the box, that’s a feature: nothing changes routing behind your back and the running configuration is exactly what’s in version control. If you host other people’s applications, it’s a support burden, because every customer rewrite rule becomes a ticket for you.

Caddy is central too, either a Caddyfile or JSON. The difference is that config changes can go through the admin API as an atomic swap rather than a file edit plus a signal, which suits automated pipelines nicely.

Nginx: where it wins and where it doesn’t

Where it wins: high connection counts on modest hardware, static and cached content, TLS termination in front of an application fleet, and the sheer weight of the ecosystem. Every stack you’ll meet has a known-good Nginx config, and everyone you hire has already read one.

Where it doesn’t: the DNS behaviour catches people constantly. A hostname written directly into proxy_pass, or into an upstream block, is resolved once at startup and cached for the lifetime of the process. When your upstream is a cloud load balancer or a container that moves, Nginx keeps sending traffic to an IP that stopped being yours. Nothing in the log says “stale DNS”; you just get connection timeouts to an address that looks plausible.

Forcing re-resolution means putting the name in a variable and giving Nginx a resolver, because that’s what moves the lookup from config parse time to request time:

location / {
    # resolver moves lookups to runtime; valid= caps how long a result is reused
    resolver 127.0.0.53 valid=30s ipv6=off;

    # the variable is the trick: it defers evaluation to per-request
    set $upstream_host app.internal.example.com;
    proxy_pass http://$upstream_host:8080$request_uri;
}

Note the $request_uri on the end. Once a variable appears in proxy_pass, Nginx stops doing its usual URI rewriting for you, so you have to pass the path explicitly. Miss that and every request lands on the upstream’s root. Also worth knowing: an upstream block defined elsewhere with the same name will take precedence over the runtime lookup, which produces a very confusing afternoon.

The other honest downside is governance. Nginx is owned by F5, and a longtime core developer forked it to freenginx after a dispute over how security issues were being handled, with an earlier fork, Angie, already in existence. Mainline Nginx remains overwhelmingly dominant and I would not choose a fork for a client project today. But “who decides what ships” is a fair question to ask about any dependency, and this one now has a public answer.

Apache: where it wins and where it doesn’t

Where it wins: anywhere applications need to configure the web server without touching the web server. Shared hosting, multi-tenant WordPress, legacy applications that ship their own rewrite rules. The module ecosystem is enormous and modules load at runtime rather than requiring a rebuild. And the documentation is, quietly, the best of the three: nearly every directive has a page explaining its interactions.

Where it doesn’t: memory per connection, if you’re still on prefork. Slow clients holding processes open is the specific scenario that hurts, because a process that’s waiting on a phone on bad mobile data is a process nobody else can use. Putting a CDN or a proxy in front absorbs a lot of that, which is a large part of why Apache origins behind Cloudflare behave so much better than bare ones.

I would not migrate a working Apache installation to Nginx purely for performance. I would move it from prefork and mod_php to mpm_event and PHP-FPM first, then measure, and quite often stop there.

Caddy: where it wins and where it doesn’t

Where it wins: anything where certificate management is the operational risk. Small teams without a dedicated ops person, internal tooling with a long tail of hostnames nobody tracks, multi-tenant products serving customer-owned domains, container stacks where a single static binary with no runtime dependencies is a real simplification. A working HTTPS reverse proxy is genuinely a few lines.

app.example.com {
    encode zstd gzip
    reverse_proxy app:3000
}

That obtains a certificate, redirects HTTP to HTTPS, negotiates compression and proxies, with no ACME client, no cron, no reload hook.

Where it doesn’t: the ecosystem gap is real. When something unusual breaks, there are fewer Stack Overflow answers and fewer engineers who have seen it before. Adding a plugin means rebuilding the binary with xcaddy rather than dropping in a module, which is fine in CI and annoying on a box you SSH into. And there’s a subtler cost: automation you don’t understand is still automation you’re responsible for. Being paged about ACME behaviour you’ve never had to learn is a worse afternoon than being paged about Certbot.

On upstream DNS, Caddy dials static upstreams by name so resolution happens at dial time rather than being frozen at config load. When the set of addresses changes, the dynamic a upstream module with a refresh interval is the built-in answer, which is a cleaner story than Nginx’s variable trick.

A decision procedure for Nginx vs Apache vs Caddy

Work down this list and stop at the first one that matches. It’ll be right more often than a benchmark.

1. Do applications you don’t control need to change server behaviour? If yes, Apache. Nothing else has .htaccess, and reimplementing it in tickets is not a plan.
2. Is TLS renewal your top operational risk? Small team, no dedicated ops, lots of hostnames, or customer-owned domains: Caddy. It deletes a whole class of incident.
3. Are you terminating a very high connection rate, or serving heavy static content? Nginx. This is the workload it was designed around.
4. Does the team already run one of these well? Keep it. Operational familiarity beats a 20% throughput difference you were never going to measure.
5. Is there a working Apache install and no specific complaint? Fix the MPM and PHP handler, then reassess. Most “we need Nginx” tickets are actually “we need to stop using prefork.”
6. None of the above? Nginx for anything that will be handed to another engineer. Caddy for anything you’ll run yourself.

Common mistakes

• Renewing certificates without reloading. The single most common self-inflicted TLS outage on Nginx and Apache. Use a deploy hook and verify it with a dry run.
• Migrating for benchmark numbers. You inherit a new failure mode and a team that has never debugged it, in exchange for latency nobody could perceive.
• Leaving Apache on prefork after moving to PHP-FPM. The migration is only half done, and you’re still paying prefork’s memory cost while also giving up HTTP/2.
• Hostnames baked into proxy_pass with dynamic upstreams. Works perfectly until the upstream IP changes, then fails in a way that looks like a network problem.
• Enabling Caddy’s on-demand TLS without an ask endpoint. You’ve handed strangers the ability to make you request certificates.
• Exposing Caddy’s admin API beyond loopback. Unauthenticated full config replacement is exactly as bad as it sounds.
• Monitoring the certificate file instead of the handshake. The file is almost never the thing that’s wrong. Check what the server actually presents, from outside the box.

How I’d decide, in practice

• Pick based on who has to change the config six months from now, not on requests per second today.
• Whichever you choose, dump the effective configuration rather than reading files: nginx -T, apachectl -S, caddy adapt --config /etc/caddy/Caddyfile. Include files hide surprises.
• Monitor TLS expiry as an external handshake check from a service like Better Stack or a Grafana Cloud synthetic, not as a filesystem check on the box.
• Put something in front. A CDN like Cloudflare or Fastly absorbs slow clients and connection floods, which flattens most of the architectural differences here.
• Run one server type per environment. Mixing Nginx in staging and Apache in production means your rewrite rules are untested by definition.
• Validate before reloading, every time. nginx -t, apachectl configtest and caddy validate all exist so you don’t find out from users.

Frequently asked questions

Is Caddy production ready?

Yes. It’s been running real workloads for years and its certificate automation removes more operational risk than its smaller ecosystem adds, for most teams. The caveat is support surface, not stability: when you hit something unusual, there are fewer people who have hit it before you.

Can I run Nginx and Apache together?

Yes, and it’s a common pattern: Nginx on 80 and 443 handling TLS and static files, proxying dynamic requests to Apache on a local port so applications keep their .htaccess behaviour. You get Nginx’s connection handling and Apache’s per-directory config. The cost is two servers to patch, two logs to correlate, and real client IP to forward correctly.

Which is best for WordPress?

Apache if plugins need to write rewrite rules, which on a typical multi-plugin site they do. Nginx if you control the site and are willing to translate those rules into try_files and static location blocks once. Plenty of high-traffic WordPress runs on Nginx; it just means plugin-generated rules are your job now.

Does Nginx really re-resolve DNS for upstreams?

Not by default. A literal hostname in proxy_pass or an upstream block is resolved at startup and reused indefinitely, regardless of TTL. You get runtime resolution by using a variable together with a resolver directive, and remembering that this changes how the request URI is passed.

Should I switch from mod_php to PHP-FPM?

In almost all cases yes. It’s the change that unlocks Apache’s threaded MPMs and HTTP/2, and it separates PHP’s lifecycle from the web server’s so you can restart one without the other. Check first for anything relying on PHP settings in Apache config or .htaccess, since those move to a .user.ini file with different syntax.

Does the freenginx fork mean I should stop using Nginx?

No. Mainline Nginx under F5 remains the overwhelmingly dominant option with the broadest packaging and support. The fork is worth knowing about as context on project governance rather than as a migration target, and I wouldn’t put a client on a fork without a specific reason.

Which uses the least memory?

Nginx, generally, with Caddy close behind and Apache furthest away when running prefork. But the gap narrows sharply once Apache is on mpm_event, and on a VPS from a provider like Hetzner or DigitalOcean the memory your application uses will dwarf all three. Measure your own stack before treating this as a deciding factor.

The one thing worth remembering

The Nginx vs Apache vs Caddy question is not a performance question. All three are fast enough for what you’re doing. What differs is which operational chores each one hands back to you: Apache gives applications control at the cost of memory and predictability, Nginx gives you central control at the cost of owning every rewrite rule and the certificate reload, and Caddy takes the certificate lifecycle off your plate at the cost of a smaller ecosystem when things get strange.

Pick the set of chores your team can actually carry. Then go fix the thing that’s genuinely slow, which is almost certainly not the web server.

Need a second opinion on your web server setup?

Most of the work I do here is unglamorous and pays for itself quickly. Things I can help with:

• Auditing an existing Nginx, Apache or Caddy config and telling you what will break under load before it does
• Migrating Apache from prefork and mod_php to mpm_event with PHP-FPM, without breaking existing .htaccess behaviour
• Translating a working Apache rewrite set into an Nginx configuration, with a test plan so you know it matches
• Fixing certificate renewal that runs but never reloads, including deploy hooks and external expiry monitoring
• Setting up Caddy for multi-tenant or customer-domain hosting with on-demand TLS properly restricted
• Diagnosing intermittent 502s, 504s and stale-upstream timeouts that look like network faults and aren’t

Send me your config, an error log excerpt, or the output of nginx -T and tell me what you’re seeing. I’d rather look at the real thing than guess from a description.

The post Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects appeared first on John Nessime.

That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between “the server works” and “the server is reproducible” turns into a weekend.

This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client’s credentials from ever touching another client’s box, and how to prove a build is finished instead of assuming it.

The four failure families worth designing against

Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.

1. You cannot rebuild it

The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run apt install, every file edited over SSH at 1am, every “I’ll document that later” is a piece of the machine that exists nowhere but the machine.

This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.

2. Every client is a slightly different server

Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.

Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts “all our clients have…” becomes a guess.

3. One credential, too much blast radius

This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client’s server, and the same API token provisions every client’s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.

Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.

4. The handover you cannot perform

Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.

How to automate server provisioning for new clients

The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.

Layer 1: the machine itself

Terraform or OpenTofu talks to your provider’s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.

Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client’s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.

Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.

Layer 2: first boot with cloud-init

cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.

#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo

Two deliberate omissions. There is no package_upgrade: true, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in runcmd is a script you will never test again.

python3 is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.

Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:

cloud-init schema --config-file user-data.yaml

Layer 3: convergence with Ansible

This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.

The layout that scales across clients separates the shared build from the per-client differences:

infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml

Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.

Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:

cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  > ../inventory/hosts.ini

There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.

Layer 4: the verification gate

This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.

Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:

- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost

Note ssh_unit as a variable rather than a literal: the SSH unit is ssh.service on Debian and Ubuntu but sshd.service on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.

The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.

Keeping clients apart: secrets and access

Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client’s infrastructure is a single, contained action.

For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.

# .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...

Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.

For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.

If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.

Troubleshooting a provisioning run

Ansible connects before cloud-init has finished

The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.

Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:

cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log

One caveat that catches people: never put cloud-init status --wait inside a runcmd or bootcmd block. It will deadlock waiting for the process that is running it.

cloud-init appears to have done nothing

Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in /var/lib/cloud/instance/sem/ are preventing a rerun.

# what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot

Never run cloud-init clean on a production host. It removes cached state and the following boot will behave as a first boot.

Host key verification failed on a rebuilt server

Rebuild a machine on the same IP and every client with the old key in known_hosts refuses to connect. The tempting fix is to disable host key checking in ansible.cfg. Do not. That turns a real integrity check into a permanent no-op across every client you manage.

Remove the stale entry and pin the new one as part of the provisioning flow:

ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 >> ~/.ssh/known_hosts

Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.

The playbook reports changes on every single run

Almost always a shell or command task with no creates, changed_when or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.

Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:

ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff

Common mistakes

• Putting the application build in runcmd. It runs once, never gets tested, and is invisible to every later change.
• One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.
• Copying a role into a client directory to make one small change, instead of extracting a variable.
• Treating “the playbook exited zero” as proof the server is ready. It is proof no task threw.
• Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.
• Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.
• Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.

Best practices

• Rebuild before you need to. Destroy and rebuild a client’s staging host on a schedule. It is the only honest test of reproducibility.
• Version your roles and pin them per client. A shared role that everyone tracks on main means one commit changes six clients at once.
• Keep first boot minimal. If it can wait for Ansible, it should.
• Make monitoring part of provisioning. Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.
• Same for backups. An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.
• Separate credentials per client, always. Keys, tokens, encryption keys, state backends.
• Write the runbook as you build. The provisioning repo’s README is the handover document. If it is accurate, the engagement can end cleanly.

Frequently asked questions

Do I need Terraform if I only manage a few VPS instances?

Not necessarily. If you are creating three servers a year through a provider’s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.

Should I build custom images instead of provisioning at boot?

Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.

Terraform or OpenTofu for client projects?

Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.

How do I automate server provisioning without exposing SSH to the internet?

The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.

How do I stop configuration drift once the server is live?

Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.

What belongs in cloud-init versus Ansible?

cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.

How long should provisioning a new client server take?

Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.

The one thing worth remembering

When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer “can you rebuild this?” with yes, on any day, without opening a shell on the existing machine.

Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.

Start with one client. Get their staging box rebuilding from scratch. Then make the second client’s build reuse the roles from the first. That is the whole path.

Need a provisioning setup you can actually hand over?

Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:

• Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime
• Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets
• Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works
• Writing the verification playbook that decides whether a build is finished, and wiring it into CI
• Fixing idempotency so check-mode runs become a usable drift report instead of noise
• Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves

If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.

The post Automating Server Provisioning for New Clients Without Building a Snowflake Farm appeared first on John Nessime.

That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that Kubernetes zero-downtime rollouts depend on, and it is not the question that usually bites.

This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.

What a zero-downtime rollout actually requires

Three independent things have to be true at the same time, and each is owned by a different part of the system.

• Capacity never dips below what traffic needs. Owned by the Deployment’s rolling update strategy.
• No pod receives traffic before it can serve it. Owned by the readiness probe.
• No pod receives traffic after it stops serving it. Owned by graceful termination, and this is the one nobody configures.

Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.

Failure family one: traffic arrives before the pod can serve it

This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means “the process was forked”, not “the process can answer HTTP”. The pod lands in the Service’s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.

The readiness probe that lies

Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:

A readiness endpoint that returns 200 from a static handler. If /healthz is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.

A readiness endpoint that checks downstream dependencies. This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks every replica unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers “can this pod serve traffic”, not “is the whole system healthy”. Check the dependency in your app’s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.

Let the startup probe own the boot budget

For anything slow to boot, do not stretch initialDelaySeconds on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.

# The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.

Failure family two: traffic arrives after the pod has stopped

This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has “already added readiness probes”.

When a pod is deleted, two things happen in parallel, not in sequence:

1. The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container’s main process.
2. The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.

Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.

The preStop sleep, and what it does not do

The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.

spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15

If you are on an older cluster without the native sleep handler, the equivalent is an exec hook, which does require a shell and a sleep binary in the image:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application’s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.

There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.

The grace period is a shared budget

This trips people up. The terminationGracePeriodSeconds countdown starts when the pod is marked Terminating, which is before the preStop hook runs, not after. The hook and your application’s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.

Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.

One more detail worth knowing before you go looking for it: probes accept their own terminationGracePeriodSeconds override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.

Failure family three: the rollout deletes capacity faster than it adds it

The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1

Setting maxUnavailable: 0 is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise maxSurge to a percentage if you have the node capacity and want the speed back.

minReadySeconds is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.

Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api

Failure family four: the load balancer was never watching EndpointSlices

Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.

Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.

Two things help here:

• Pod readiness gates. These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.
• Match your preStop sleep to the deregistration delay. If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.

The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod’s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.

Verifying a zero-downtime rollout instead of hoping

A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.

1. Generate steady traffic against the real ingress path. Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.
2. Trigger a rollout with no image change, so you are testing the mechanism rather than your new code.
3. Watch EndpointSlices in a second terminal and note how long a terminating pod stays in the list.
4. Count non-200 responses. Anything above zero is a bug, not noise.
5. Repeat under realistic concurrency. A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.

# Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m

Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.

Troubleshooting

Rollout hangs, new pods never become Ready. The readiness probe is failing. kubectl describe pod shows the probe failure and the response it got. Check that the port name in the probe matches a declared containerPort name, that the path exists, and that the app is listening on all interfaces rather than only 127.0.0.1, which is a classic one when moving from a local docker-compose setup.

Probes fail only under load. Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within timeoutSeconds, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.

Errors persist after adding a preStop sleep. Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.

Pods stuck Terminating for the full grace period. Your app is ignoring SIGTERM. This is extremely common when the container’s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of ENTRYPOINT, or an init like tini, so your process actually receives the signal.

Rollout succeeded but the new version is broken. Roll back first, investigate second.

# Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3

Common mistakes

• Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.
• Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.
• Leaving maxUnavailable at the default on a service running near capacity.
• Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.
• Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.
• Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.
• Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.

Best practices

• Three separate endpoints: /healthz/started, /healthz/ready, /healthz/live. They answer different questions and should be allowed to disagree.
• maxUnavailable: 0 plus a maxSurge you have node capacity for, as the default for anything user-facing.
• A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.
• A grace period that covers preStop plus your slowest realistic request, with slack.
• Handle SIGTERM properly in the application, and make sure it reaches PID 1.
• A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.
• minReadySeconds long enough to catch a pod that passes its probe and then dies.
• Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.

FAQ

Do readiness probes alone give me zero-downtime deployments?

No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.

How long should the preStop sleep be?

Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.

Should the readiness probe check the database?

Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.

What is the difference between maxSurge and maxUnavailable?

maxSurge is how many pods you may run above the replica count during a rollout. maxUnavailable is how many you may drop below it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.

Why do I still get 502s on EKS, GKE or another managed cluster?

Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group’s deregistration delay rather than against kube-proxy.

Can I get zero-downtime rollouts with a StatefulSet?

The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.

Does kubectl rollout status prove the deploy was clean?

It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.

The one thing worth remembering

Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.

Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.

Need a second pair of eyes on your rollouts?

Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:

• Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change
• Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number
• Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager
• Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths
• Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks
• Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event

If you have a Deployment manifest, a kubectl describe pod output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.

The post Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn’t Saving You appeared first on John Nessime.