That is the failure mode worth understanding before you touch anything. A Salesforce AWS PrivateLink setup does not fail loudly. It fails by working. The callout succeeds, the payload arrives, the dashboard is green, and the packets went over the public internet anyway because one DNS record was wrong. Nothing in either console tells you.

This post covers what actually breaks when you wire Salesforce to AWS over private connectivity: the DNS trap above, the throughput ceiling nobody budgets for, the TLS and port constraints, why your sandbox cannot rehearse the change, and how to prove after the fact that traffic really is staying on the AWS network.

What Private Connect actually gives you

Salesforce Private Connect is the product name. AWS PrivateLink is the underlying mechanism. Salesforce runs a managed transit VPC in each supported AWS region, and you attach to it: either your VPC creates an interface endpoint pointing at their endpoint service, or their transit VPC creates an endpoint pointing at yours.

Two things people consistently get wrong here.

First, the direction naming is from the org’s point of view, not yours. Inbound means traffic entering Salesforce, so your AWS workload calling the Salesforce REST or Bulk API. Outbound means Salesforce calling out to something you host. If you are an AWS engineer, “inbound” is the one that feels like outbound to you. Half the confused meetings I have seen on this trace back to that single word.

Second, Private Connect only builds the pipe. It does not move data. You still need Apex callouts, External Services, Platform Events, MuleSoft, Amazon AppFlow, or whatever else was already doing the work. If your integration is broken today, a private path will carry the broken requests privately.

Private connectivity is a network property, not an application feature. Nothing above layer 4 changes. Plan the rollout as a routing change, not an integration project.

Failure family one: DNS that quietly resolves to the public internet

This is the one that produced the ticket at the top, and it is by far the most common way a Salesforce AWS PrivateLink deployment ends up decorative.

For the inbound case, your application calls a hostname like yourorg.my.salesforce.com. That hostname resolves publicly. Creating the interface endpoint does not change that. Unless something inside your VPC overrides the answer, the resolver returns Salesforce’s public addresses and your traffic takes the normal internet path. The endpoint sits there costing you money and carrying nothing.

The override is a Route 53 private hosted zone, associated with the VPC that holds the endpoint, containing a record that points the org hostname at the endpoint. Salesforce’s own walkthrough names the zone my.salesforce.com and puts the org-specific record inside it.

Three things have to line up or the override silently does nothing:

• The zone is associated with the right VPC. A private hosted zone answers only for VPCs it is attached to. Query it from anywhere else and the request falls through to public recursion.
• The VPC has DNS support and DNS hostnames enabled. Both attributes must be on, or the VPC resolver will not serve the zone at all.
• Callers actually use the VPC resolver. Workloads pinned to an external resolver, containers with a hardcoded nameserver line, or on-premises clients arriving over Direct Connect will bypass the zone entirely unless you front it with a Route 53 Resolver inbound endpoint.

There is a sharp edge in the other direction too. If a private hosted zone matches the queried name but contains no record of the right type, Route 53 returns NXDOMAIN rather than falling through to public DNS. So a half-finished zone does not degrade gracefully, it takes the hostname down. Creating the zone and the record in separate change windows is how teams turn a routing improvement into an outage.

Verify from inside the VPC, not from your laptop:

# Run this on an instance in the VPC associated with the private zone.
dig +short yourorg.my.salesforce.com

# You want RFC 1918 addresses from your endpoint subnets.
# Public addresses mean the private zone is not being consulted.

# Confirm which resolver answered:
dig yourorg.my.salesforce.com | grep -i "SERVER:"

Failure family two: the throughput ceiling nobody reads

Private Connect is rate limited, and the default entitlement is small enough to catch people out badly.

Salesforce documents an initial license entitlement of 225 MB of data per hour, measured separately for inbound and outbound, with an add-on ceiling far higher. The allowance does not roll over. It resets on the hour.

Now consider what a nightly Bulk API extract looks like. A few gigabytes pushed in a twenty minute window is entirely normal and completely incompatible with a 225 MB hourly bucket. The two directions behave differently when you hit the wall, which matters more than the number itself:

• Outbound is enforced technically. Exceed it and outbound traffic stops until the counter resets at the top of the next hour. Your Apex callouts start failing. This is a production incident.
• Inbound is enforced contractually. Salesforce documents that you are responsible for monitoring and enforcing the inbound limit yourself. Nothing throttles you at the network layer, but you are outside your agreement, and standard enterprise API limits still apply on top.

The outbound allowance is visible through the Limits API as PrivateConnectOutboundCalloutHourlyLimitMB, which is what makes it monitorable. Poll it, put it on a dashboard next to your other API limit metrics in Grafana or CloudWatch, and alert well before exhaustion rather than after the callouts fail.

If your volume genuinely exceeds the entitlement, the honest options are buying more, reshaping the workload to spread across hours, or accepting that bulk movement stays on a different path and Private Connect covers only your latency-sensitive transactional traffic. That last one is a legitimate architecture, not a failure.

Failure family three: TLS, SNI, and the port you cannot use

Private Connect inbound supports port 443 only. Port 8443 is explicitly not supported because Salesforce uses it for mutual TLS. If your integration depends on client certificate authentication on that port, this design does not cover it, and finding out during cutover is expensive.

The subtler problem is SNI. Salesforce documents that Hyperforce requires callers to include the Server Name Indication extension in the TLS ClientHello so the right certificate can be served, and that clients calling a hostname outside the *.my.salesforce.com or *.sandbox.my.salesforce.com patterns will hit HTTPS errors. Browsers always send SNI. Older HTTP libraries, embedded clients, and some middleware stacks do not.

The reason this bites specifically during a PrivateLink migration is that you are changing what the name resolves to while keeping the name identical. A client that was quietly relying on the public path’s behavior now hits a different frontend. Test the handshake explicitly:

# From inside the VPC, forcing SNI to the org hostname.
openssl s_client -connect yourorg.my.salesforce.com:443 
  -servername yourorg.my.salesforce.com </dev/null

# Compare with SNI omitted. If this one fails and the above
# succeeds, your client library is the problem, not the network.
openssl s_client -connect yourorg.my.salesforce.com:443 </dev/null

On the outbound side the mirror image applies. Salesforce is the client calling your endpoint service, and it will validate your certificate against the hostname it was told to use. If your Network Load Balancer terminates TLS with a certificate whose subject alternative names do not include the private DNS name you registered, the callout fails on hostname verification. Issue the certificate for a name you control and that matches what the named credential targets.

Failure family four: you cannot rehearse this properly

This one deserves its own section because it changes how you plan the work.

Private connections are not copied from production into sandboxes. You recreate them. And the ability to recreate them depends on sandbox type:

• Full and Partial Copy sandboxes: you can create connections and provision them. This is where genuine rehearsal happens.
• Developer and Developer Pro sandboxes: you can create connections but cannot provision them.
• Scratch orgs and Developer orgs: create only, no provisioning.

If your organization only has Developer sandboxes, there is no lower environment where the private path can be exercised end to end. That is a procurement conversation, not an engineering one, and it needs to happen at design time. Budget for a Full or Partial Copy sandbox or accept that production is your first real test.

Related trap: connections are packageable while unprovisioned, and unprovisioned connections do not consume a license. That is useful for shipping configuration through a deployment pipeline, but it also means a successful deploy proves nothing about connectivity.

Failure family five: license arithmetic and connection identity

Each Private Connect license covers one provisioned connection in each direction. Each connection is a one-to-one mapping between an org ID and a VPC endpoint ID.

Read that mapping carefully, because it is where multi-account AWS estates get expensive. Separate VPCs mean separate endpoints mean separate connections mean separate licenses. Separate orgs mean the same again. A design with three application VPCs across two orgs is not one connection, it is six.

The architectural answer is usually a shared services or egress VPC that owns the endpoint, with the application VPCs reaching it through a Transit Gateway. You pay for one endpoint and route to it. This is the same centralized-endpoint pattern you would already use for interface endpoints to AWS services, and the reasoning is identical: endpoints are billed per attachment, so consolidate them.

Four inbound connections consume four licenses. If you hold four licenses, you have used all of them and have none left for outbound. Count both directions before the renewal conversation.

Failure family six: the AWS side of an outbound connection

For outbound, you are the service provider. Salesforce’s transit VPC creates the endpoint into your account, which means you have to build a proper endpoint service.

The sequence, in the order that avoids rework:

1. Stand up a Network Load Balancer in front of your service, across at least two Availability Zones.
2. Create the VPC endpoint service backed by that NLB.
3. Copy the IAM role ARN from the Private Connect setup page in Salesforce for your region, and add it to the allowed principals on the endpoint service. Without this, Salesforce cannot connect and the failure looks like a generic timeout.
4. Register a private DNS name on the endpoint service and complete domain ownership verification.
5. Create the outbound connection in Salesforce using the endpoint service name, then sync it to pick up the AWS-side status.
6. Associate the connection with a named credential. Callouts only take the private path when they go through a named credential bound to that connection.

Step four is where people lose an afternoon. Domain verification requires a TXT record in a public zone, because AWS queries the public name servers registered at your domain’s registrar. Put the record in a private hosted zone and verification sits in pending forever with no useful error.

# Register the private DNS name on your endpoint service.
aws ec2 modify-vpc-endpoint-service-configuration 
  --service-id vpce-svc-EXAMPLE 
  --private-dns-name "sf-integration.example.com"

# Read back the TXT record name and value AWS expects.
aws ec2 describe-vpc-endpoint-service-configurations 
  --service-ids vpce-svc-EXAMPLE 
  --query 'ServiceConfigurations[0].PrivateDnsNameConfiguration'

# After publishing the TXT record in your PUBLIC zone:
aws ec2 start-vpc-endpoint-service-private-dns-verification 
  --service-id vpce-svc-EXAMPLE

One detail worth knowing: if verification status later changes away from verified, new connection requests are refused while existing connections keep working. So a lapsed TXT record does not break you immediately. It breaks you the next time something reconnects, which is usually during an unrelated incident.

What this costs, mechanically

Rates change, so model the mechanism rather than memorizing figures from a blog post.

• Interface endpoint hours. Billed per endpoint, per Availability Zone, per hour, whether or not traffic flows. A three-AZ deployment is three billable attachments.
• Data processing. Per GB through the endpoint, on a tiered scale that drops as regional volume rises.
• Cross-AZ transfer through interface endpoints. AWS removed this charge, so multi-AZ endpoint placement is a resilience decision rather than a cost trade-off.
• Cross-region. Still charged, in both directions, plus a per-hour fee on the endpoint service for each remote region with attached endpoints. Match your org region to your VPC region if you can.
• Provider-side infrastructure. For outbound, the NLB and its targets are your bill, not PrivateLink’s.
• The Salesforce license. Separate from all of the above, priced per connection.

Compare honestly against what you are replacing. If the alternative is a NAT gateway carrying the same traffic, NAT has both an hourly charge and a per-GB processing charge of its own, and the comparison is often closer than people assume. If the alternative is a site-to-site VPN or a commercial network overlay, the calculus changes again. PrivateLink wins on the security posture, not automatically on the invoice.

Troubleshooting: proving the traffic is actually private

Do not trust a green status field. Prove it with evidence from the data path.

1. Resolve the hostname from inside the VPC. Private addresses from your endpoint subnets, or nothing else matters.
2. Check the endpoint state. aws ec2 describe-vpc-endpoints should show the endpoint available, with an ENI in each subnet you expect.
3. Read VPC flow logs. Filter for your application’s source addresses and confirm the destination is the endpoint ENI, not a NAT gateway or internet gateway address. This is the actual proof.
4. Check the security group on the endpoint. Interface endpoints carry security groups. If 443 is not permitted from your application subnets, connections hang rather than refuse, which reads as a Salesforce problem in your logs.
5. For outbound, sync the connection in Salesforce before believing its status. The displayed state is the last polled AWS state, not live.
6. Test the callout through the named credential. A direct callout to a hardcoded URL will not use the private path even when everything else is correct.

Two symptom patterns worth recognizing. Connections that hang and time out point at security groups or NLB target health. Connections that fail fast with a TLS error point at certificate naming or SNI. They are diagnosed in completely different places.

Common mistakes

• Treating the setup wizard’s success message as proof that traffic is private. It proves the control plane is configured.
• Creating the private hosted zone in one change and the record in another, producing NXDOMAIN for the org hostname in between.
• Publishing the domain verification TXT record in a private zone instead of the public one.
• Forgetting the Salesforce IAM role in the allowed principals list on the endpoint service, then debugging it as a network problem.
• Sizing the connection for peak batch throughput without checking the hourly data entitlement.
• Assuming sessions locked to their originating IP address will keep working. That behavior is documented as unsupported with Private Connect, along with asynchronous callouts using Apex continuations.
• Building one endpoint per application VPC and discovering the license count at renewal.
• Leaving callouts on hardcoded endpoints instead of named credentials, so outbound traffic never enters the private path.

Best practices for a Salesforce AWS PrivateLink rollout

• Inventory every integration touching the org first, with direction, protocol, hostname, and peak volume per hour. The inventory determines the design.
• Centralize endpoints in a shared services VPC and route from application VPCs through Transit Gateway rather than duplicating endpoints and licenses.
• Put the endpoint in the same region as the org. Cross-region private connectivity works but gives up most of the latency benefit and adds transfer charges.
• Enable VPC flow logs before cutover so you have a clean before-and-after picture. Ship them somewhere you can query, whether that is CloudWatch Logs Insights, Athena over S3, or a platform like Grafana Cloud or Datadog.
• Alert on the outbound hourly limit as a percentage of entitlement, not on callout failures. By the time callouts fail you are already down.
• Terraform the AWS side. The endpoint, the endpoint service, the allowed principals, the private hosted zone, and the records all belong in code. The Salesforce side is click-configured, so document it in the same repository as a runbook.
• Keep a documented rollback: the change that reverts you to public routing is removing or disassociating the private hosted zone, and it should be a one-step action you have practiced.
• Review the configuration on a schedule. Private routing is invisible in normal operation and easy to break with an unrelated DNS or VPC change.

Frequently asked questions

Does Private Connect encrypt my traffic?

Your traffic is already encrypted in transit by TLS. What private connectivity changes is the path, not the cipher. The value is that packets stay on the AWS network rather than traversing the public internet, which removes an entire class of exposure and simplifies a lot of compliance conversations. It is not a substitute for TLS, and it does not authenticate anything.

Can I use Private Connect to access the Salesforce user interface privately?

No. This is API-path connectivity for integrations. Browser access to the org is a separate concern, and if that is what you need, you are looking at conditional access policies, login IP ranges, or a corporate network solution rather than PrivateLink.

What Salesforce features actually route through an outbound connection?

Salesforce documents outbound support for Apex callouts, External Services, Platform Events, Change Data Capture, CRM Analytics, the OData 4.01 adapter and custom adapter for Salesforce Connect, the Salesforce Connect SQL adapter for Amazon Athena, and Agent Actions backed by External Services. Inbound covers the supported public APIs. Anything not on the current list should be verified against the documentation before you design around it, because that list grows.

Do I need Hyperforce for this?

Salesforce states that Private Connect works whether the org sits in first-party data centers or on Hyperforce. That said, Hyperforce brings its own requirements, particularly the SNI behavior described above, so confirm where your org actually runs before you start debugging TLS.

How do I connect on-premises systems, not just AWS workloads?

Bring the on-premises network into AWS first with Direct Connect or a site-to-site VPN, then use the interface endpoint from there. The extra work is DNS: on-premises resolvers need a Route 53 Resolver inbound endpoint to see the private hosted zone, otherwise they resolve the org hostname publicly and you are back to the first failure family.

Can I use one connection for several Salesforce orgs?

No. A connection is a one-to-one mapping between an org ID and a VPC endpoint ID, so each org needs its own provisioned connection and its own license, even if they all point at the same VPC.

Is a VPN a reasonable alternative?

For Salesforce specifically, no. Salesforce does not terminate a customer VPN into the org, so PrivateLink is the mechanism on offer. A VPN or a commercial overlay such as NordLayer solves a different problem, which is getting your own users and sites onto your own network. Where those tools genuinely overlap is the on-premises leg, and even there Direct Connect is usually the better answer for sustained integration traffic.

The one thing to remember

A Salesforce AWS PrivateLink deployment that is misconfigured looks exactly like one that works. Same green statuses, same successful callouts, same clean application logs. The only difference is which path the packets took, and the only place that difference is visible is your flow logs.

So build the verification step into the project rather than the postmortem. Resolve the hostname from inside the VPC, read the flow logs, confirm the destination is the endpoint ENI, and keep the check running afterwards. Everything else in this post is detail. That one habit is what separates private connectivity from a line item that says private.

Need help wiring Salesforce to AWS privately?

This is the kind of work where a second pair of eyes on the DNS and the flow logs saves a security review later. I help teams with:

• Designing the endpoint topology, including centralized endpoints behind Transit Gateway so you are not buying a connection per VPC.
• Building and verifying the Route 53 private hosted zone override, including Resolver inbound endpoints for on-premises callers.
• Standing up the endpoint service side for outbound connections: NLB, allowed principals, private DNS name registration, and domain verification.
• Proving traffic is genuinely private with VPC flow log analysis, before and after cutover.
• Instrumenting the hourly data entitlement so you find out about throttling from a dashboard rather than from failed callouts.
• Codifying the AWS side in Terraform or OpenTofu, with a documented rollback to public routing.

If you are mid-way through one of these and something is not behaving, send me the dig output from inside the VPC, a flow log sample, or your endpoint service configuration and I will tell you what I see.

The post Salesforce AWS PrivateLink: The Failures Nobody Warns You About appeared first on John Nessime.

docker compose up -d

A web application, database, cache, monitoring service, and reverse proxy can all be launched in seconds. Everything is isolated, repeatable, and relatively easy to move between environments.

Then, eventually, something breaks.

A container refuses to start. A port is already occupied. A database loses access to its storage directory. One service cannot resolve the hostname of another. A reverse proxy returns a 502 error even though the application appears to be running.

That is usually the point where Docker stops feeling like a convenient deployment tool and starts becoming a real infrastructure skill.

This guide is based on the way I approach Docker issues on real Linux servers. It is not a list of commands to copy blindly. It is a troubleshooting method for discovering what failed, understanding why it failed, and solving it without making the situation worse.

Docker Is Usually Not the Real Problem

When a container fails, Docker is often blamed first. In many cases, however, Docker is only exposing a problem somewhere else in the system.

• Incorrect Linux file permissions
• A missing environment variable
• An unavailable database
• A port conflict on the host
• An incorrect volume mount
• A DNS or network configuration problem
• A reverse proxy pointing to the wrong destination
• Insufficient memory or disk space
• An application-level configuration error

The container runtime may be working exactly as designed. The application inside the container may simply be unable to operate under the conditions it was given.

This distinction matters because repeatedly deleting and rebuilding containers rarely fixes the underlying issue. It may temporarily hide it, but the same failure usually returns.

My First Rule of Docker Troubleshooting: Do Not Delete Anything Yet

When a container suddenly exits, the first instinct is often to remove it, rebuild the image, or restart the entire stack. That destroys useful evidence.

Before changing anything, I start by collecting information.

docker ps -a

This shows running and stopped containers, their current states, exposed ports, names, and recent exit information.

docker logs container-name

The logs frequently reveal the immediate cause: a missing file, invalid password, failed database connection, permission error, or malformed configuration.

docker inspect container-name

Inspection provides the container’s network settings, environment variables, volume mounts, restart policy, health status, image information, and runtime configuration.

These three commands answer a large percentage of Docker troubleshooting questions before any corrective action is required.

Understanding Docker Container States

A container can exist in several states, and each state tells a different story.

Running

The main process inside the container is active. This does not automatically mean the application is healthy or ready to receive traffic.

Exited

The container’s main process stopped. It may have completed normally, crashed, or failed during startup. The exit code and logs provide the next clues.

Restarting

The container repeatedly starts and fails while Docker applies its restart policy. This commonly indicates an application crash, invalid configuration, inaccessible dependency, or permission problem.

Created

The container was created but its main process has not started successfully.

Dead

Docker could not properly stop or remove the container. This is less common and may indicate a lower-level runtime, storage, or host issue.

Knowing the state prevents random troubleshooting. A networking issue requires a different investigation from a process that exits because of a syntax error.

Start With the Container Logs

Logs are normally the fastest route to the cause of a Docker issue.

docker logs --tail 100 container-name

This displays the latest 100 lines without flooding the terminal with the container’s entire history.

docker logs -f container-name

The follow option streams new log entries in real time. It is useful while restarting a service or reproducing a problem.

docker logs --since 30m container-name

This limits the output to a recent period, which is especially useful for long-running containers.

When reading logs, I do not look only at the final line. The true cause often appears several lines earlier. The last error may simply be a consequence of the first failure.

Docker Networking: Where Many Problems Begin

Docker networking is one of the most common sources of confusion, especially when several containers need to communicate.

Imagine a stack containing:

• Grafana
• Prometheus
• A web application
• MySQL or PostgreSQL
• Redis
• Nginx or Apache

Each service may work independently while communication between them fails.

The Localhost Mistake

Inside a container, localhost refers to that container itself. It does not refer to the Docker host, and it does not refer to another container.

For example, if Grafana needs to connect to Prometheus, the data source address should usually use the Docker Compose service name:

http://prometheus:9090

Using the following from inside the Grafana container would normally point back to Grafana itself:

http://localhost:9090

This small misunderstanding causes a surprising number of connection failures.

Confirm That Containers Share a Network

docker network ls

docker network inspect network-name

The inspection output shows which containers are attached to a network and which IP addresses were assigned.

Containers on unrelated networks cannot automatically communicate by service name. They must share a Docker network or use another explicitly configured route.

Test Connectivity From Inside the Container

docker exec -it container-name sh

Depending on the image, Bash may be available instead:

docker exec -it container-name bash

From inside the container, test name resolution and connectivity using the utilities available in that image:

getent hosts database
curl http://application:8080
wget -qO- http://prometheus:9090/-/ready

A successful result confirms that Docker DNS and network routing are functioning. A failed result narrows the investigation considerably.

Host Ports and Container Ports Are Different

A Docker Compose port mapping such as the following is read from left to right:

ports:
  - "8080:80"

Port 8080 belongs to the Docker host. Port 80 belongs to the application inside the container.

Traffic sent to the host on port 8080 is forwarded to port 80 inside the container.

If another service already occupies host port 8080, Docker cannot bind to it.

ss -tulpn

To check a specific port:

ss -tulpn | grep :8080

A common mistake is changing the container’s internal port when only the host-side mapping needs to change.

For example, this avoids a conflict while leaving the application untouched:

ports:
  - "8081:80"

Published Ports Are Not Required for Container-to-Container Traffic

Containers connected to the same Docker network can usually communicate through their internal ports without publishing those ports to the host.

For example, Prometheus can communicate with an exporter on the shared Docker network without exposing that exporter publicly.

This reduces unnecessary exposure and produces a cleaner architecture. Publish only the ports that must be reached from the host, a reverse proxy, or an external client.

Docker Volumes: Persistent Storage Done Properly

Containers should be treated as disposable. Persistent data should not depend on the writable filesystem inside a container.

Without a correctly configured volume, deleting a database container may also delete the database stored inside it.

Named Volumes

services:
  database:
    image: mariadb:latest
    volumes:
      - database-data:/var/lib/mysql

volumes:
  database-data:

Docker manages the storage location while the application writes to its expected internal directory.

Bind Mounts

volumes:
  - ./config:/etc/application

A bind mount maps a specific host path into the container. It is useful for configuration files, development directories, and data that administrators need to access directly.

Bind mounts also introduce more responsibility. The host directory must exist with the correct permissions, ownership, security context, and contents.

Inspect Existing Volumes

docker volume ls

docker volume inspect volume-name

Before deleting an apparently unused volume, confirm which application created it and whether it contains irreplaceable data.

Permission Problems Are Quiet but Destructive

Permission errors can prevent databases from initializing, web applications from uploading files, monitoring tools from writing data, or services from reading configuration files.

The first checks are simple:

ls -lah
stat path-to-file
stat path-to-directory

The user inside the container may have a different numeric user ID from the owner of the host directory. The names do not matter as much as the numeric UID and GID.

docker exec container-name id

Compare that result with the ownership of the mounted host path.

Changing permissions to 777 may appear to solve the issue, but it is usually an unsafe shortcut rather than a proper solution. Correct ownership and the minimum required permissions are preferable.

SELinux Can Affect Docker Bind Mounts

On distributions such as AlmaLinux, Rocky Linux, CentOS Stream, Fedora, and Red Hat Enterprise Linux, standard Unix permissions may look correct while SELinux still blocks access.

Docker Compose bind mounts may require an SELinux relabeling option:

volumes:
  - ./data:/var/lib/application:Z

The uppercase Z assigns a private SELinux label for the container. A lowercase z is used when the content must be shared between multiple containers.

SELinux should not be disabled simply to make a container work. The better approach is to identify the denied access and apply the correct labeling or policy.

Environment Variables Can Break an Entire Stack

Modern containerized applications depend heavily on environment variables for database credentials, API keys, application URLs, encryption secrets, feature flags, and runtime settings.

One missing or incorrectly escaped value can stop a service from starting.

docker compose config

This command renders the resolved Docker Compose configuration. It helps reveal missing substitutions, merged settings, malformed values, and unexpected defaults.

To inspect the environment available inside a running container:

docker exec container-name env

Be careful when sharing this output because it may include passwords, tokens, and private service credentials.

Docker Compose Configuration Errors

YAML is readable, but it is sensitive to indentation and structure. A service can be valid YAML while still being configured incorrectly for Docker Compose.

Before deploying a change, validate the file:

docker compose config --quiet

If the command returns without an error, the configuration is structurally valid.

For a complete rendered version:

docker compose config

This is especially valuable when Compose files use environment substitutions, overrides, extension fields, or multiple configuration files.

Health Checks Reveal More Than Running Status

A container can be running while the application inside it is unusable.

A database may still be initializing. A web service may be listening but unable to connect to its database. An API may be running but returning errors for every request.

A health check lets Docker test actual application readiness.

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 20s

The application should expose a lightweight endpoint that confirms its critical dependencies are available.

To inspect the current status:

docker inspect --format='{{json .State.Health}}' container-name

A health check does not repair an unhealthy service by itself, but it provides a far more accurate operational signal than process status alone.

Restart Policies Can Hide Repeated Failures

Restart policies are useful in production, but they can create the impression that a service is stable when it is actually crashing continuously.

restart: unless-stopped

If the application exits every few seconds, Docker restarts it every few seconds. The container may appear in the process list, but the service may never become ready.

Check the restart count:

docker inspect --format='{{.RestartCount}}' container-name

A rapidly increasing number indicates that the restart policy is masking an unresolved failure.

Reverse Proxy Errors: 502, 504, SSL Problems, and Redirect Loops

Reverse proxies such as Nginx, Apache, Traefik, and Caddy often sit between users and containerized applications.

When the public website fails, the application container may still be perfectly healthy. The proxy may simply be unable to reach it.

Common Reverse Proxy Problems

• The upstream hostname is incorrect.
• The proxy uses the host port when it should use the container port.
• The proxy and application do not share a Docker network.
• The application listens only on 127.0.0.1 inside the container.
• The proxy forwards HTTP while the backend expects HTTPS.
• The application is unaware that the original request used HTTPS.
• The public hostname does not match the certificate.
• Both the proxy and application force redirects, creating a loop.

Test the application directly before blaming the proxy:

curl -I http://127.0.0.1:published-port

If the application responds directly but fails through the domain, the reverse proxy, firewall, DNS, or TLS configuration becomes the main area of investigation.

An Application Must Listen on the Correct Interface

An application inside a container may start successfully while listening only on 127.0.0.1. In that situation, Docker’s network interface may not be able to reach it.

Containerized web applications should usually listen on:

0.0.0.0

This allows the process to accept connections through the container’s network interfaces.

This setting is commonly controlled by application arguments or environment variables such as HOST, BIND_ADDRESS, or LISTEN_ADDRESS.

Resource Exhaustion Can Look Like a Docker Failure

Containers share the CPU, memory, storage, and network resources of the host. One service can consume enough resources to affect every other workload.

docker stats

This provides a live view of CPU, memory, network, and block I/O usage for running containers.

On the host, I also check:

free -h
df -h
df -i
uptime

Disk space is not the only storage limit. A filesystem can also run out of inodes, preventing new files from being created even when gigabytes remain available.

Kernel logs may reveal out-of-memory termination:

dmesg | grep -i -E "out of memory|killed process|oom"

If the kernel killed the process, the application logs may end suddenly without a useful explanation.

Docker Disk Usage Grows Quietly

Old images, stopped containers, build cache, unused volumes, and unbounded logs can gradually consume the host’s storage.

docker system df

For more detail:

docker system df -v

Cleanup commands should be used carefully:

docker image prune
docker container prune
docker builder prune

A broad command such as docker system prune can remove more than expected. Never add the volume-removal option unless the contents of unused volumes have been verified and backed up.

Container Logs Can Fill the Server

Docker’s default JSON logging driver can create large log files when applications produce continuous output.

Log rotation can be configured in Docker Compose:

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "5"

This limits the number and size of local log files for that container.

For production systems, centralized logging with tools such as Grafana Loki, Elasticsearch, OpenSearch, or a managed logging service makes incident investigation considerably easier.

Image Architecture and Compatibility Problems

An image built for one CPU architecture may not run correctly on another. This becomes relevant when moving between AMD64 servers, ARM-based cloud instances, Apple Silicon machines, and single-board computers.

Check the host architecture:

uname -m

Inspect the image:

docker image inspect image-name

Errors mentioning an invalid executable format frequently indicate an architecture mismatch.

Multi-platform builds can be produced using Docker Buildx:

docker buildx build --platform linux/amd64,linux/arm64 .

A New Image Tag Does Not Always Mean a Safe Upgrade

Using latest may be convenient, but it makes deployments less predictable. A future image update can introduce a breaking configuration change, database migration, removed feature, or incompatible dependency.

Production deployments are usually safer with an explicit version:

image: grafana/grafana:12.0.2

Before upgrading:

• Read the release notes.
• Confirm supported upgrade paths.
• Back up persistent data.
• Record the current working version.
• Test the change outside production where possible.
• Prepare a rollback procedure.

Containers make software easier to replace, but they do not make every application upgrade reversible.

Dependency Startup Order Is Not Readiness

Docker Compose can start services in a defined order, but starting a database container does not mean the database is immediately ready for connections.

An application may launch too early, fail to connect, and exit.

Health checks and dependency conditions can improve this:

services:
  database:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  application:
    image: example/application:1.0
    depends_on:
      database:
        condition: service_healthy

The application itself should still handle temporary dependency failures gracefully. Networks fluctuate, databases restart, and external services occasionally become unavailable.

Backups Matter More Than Recreating Containers

Application containers are normally easy to recreate. Persistent business data is not.

A reliable Docker backup strategy should cover:

• Database-native backups
• Named volumes and bind-mounted data
• Docker Compose files
• Environment files and secrets
• Reverse proxy configurations
• TLS and certificate configuration
• Application-specific settings
• A tested restoration procedure

A backup that has never been restored is only an assumption.

Database backups should normally use the database’s own tools rather than copying active database files blindly. Examples include pg_dump for PostgreSQL and mysqldump or mariadb-dump for MySQL-compatible systems.

Monitoring Changes Troubleshooting From Guesswork to Evidence

Without monitoring, administrators usually investigate after users report a failure. By then, the container may have restarted and the original conditions may no longer be visible.

Metrics help answer better questions:

• When did memory consumption begin increasing?
• Which container experienced the first error?
• Was the disk already nearly full?
• Did network latency rise before the application failed?
• How many times did the service restart?
• Did the host run out of memory?
• Was the reverse proxy still able to reach the backend?

Prometheus, Grafana, node-level exporters, container metrics, uptime probes, and centralized logs provide the historical context that individual Docker commands cannot.

A Practical Docker Troubleshooting Workflow

When a Docker service fails, I use a sequence rather than jumping between unrelated commands.

1. Identify the Exact Symptom

• Does the container fail to start?
• Does it start and then exit?
• Is it running but unreachable?
• Is the application responding with errors?
• Is the public domain failing while the local service works?
• Is the problem intermittent?

2. Check Container Status

docker ps -a

3. Read the Logs

docker logs --tail 200 container-name

4. Inspect Runtime Configuration

docker inspect container-name

5. Validate Docker Compose

docker compose config

6. Check Host Resources

docker stats
free -h
df -h
df -i

7. Test Internal Connectivity

Enter the relevant container and test DNS resolution, ports, and HTTP endpoints from the same network context as the application.

8. Check Volumes and Permissions

Confirm that mounts point to the expected locations and that the container user can read or write them as required.

9. Isolate the Reverse Proxy

Test the backend directly. If the backend works, investigate the proxy, DNS, TLS, firewall, and forwarded headers.

10. Change One Thing at a Time

Multiple simultaneous changes make it difficult to identify the actual solution. Apply one correction, test it, and record the result.

Commands I Frequently Use During Docker Investigations

# Show running containers
docker ps

# Show all containers
docker ps -a

# Read recent logs
docker logs --tail 100 container-name

# Follow logs
docker logs -f container-name

# Inspect a container
docker inspect container-name

# Enter a container
docker exec -it container-name sh

# View live resource use
docker stats

# List networks
docker network ls

# Inspect a network
docker network inspect network-name

# List volumes
docker volume ls

# Inspect a volume
docker volume inspect volume-name

# Validate Compose configuration
docker compose config

# View Compose service status
docker compose ps

# Restart one service
docker compose restart service-name

# Recreate one service
docker compose up -d --force-recreate service-name

# View Docker disk usage
docker system df

# View host listening ports
ss -tulpn

# Check disk usage
df -h

# Check inode usage
df -i

# Check memory
free -h

Docker Troubleshooting Mistakes to Avoid

Deleting Containers Before Reading Their Logs

This removes evidence and often leaves the original problem unresolved.

Using Docker System Prune Without Reviewing the Impact

Cleanup commands can remove useful images, build cache, stopped containers, and potentially important storage.

Using Latest Everywhere

Uncontrolled image updates reduce reproducibility and can introduce breaking changes unexpectedly.

Opening Every Port Publicly

Internal services should remain internal unless external access is genuinely required.

Setting Permissions to 777

This weakens security and avoids fixing the actual ownership or access-control issue.

Restarting the Entire Server Too Early

A restart may temporarily restore service while destroying evidence about the original failure.

Changing Several Variables at Once

Even when the issue disappears, it becomes impossible to know which change solved it.

Docker Compose Is Also Infrastructure Documentation

A well-organized Compose file documents the structure of an application environment.

• Which services exist
• Which image versions are deployed
• Which networks connect them
• Which ports are externally accessible
• Where persistent data is stored
• Which environment variables are required
• Which services depend on others
• Which health checks are available
• How containers restart after failure

Six months after a deployment, this information is often more useful than a separate document that was never updated.

Readable service names, comments for non-obvious decisions, pinned image versions, organized environment files, and clear volume names make future troubleshooting much faster.

What Docker Failures Actually Teach

Some of the most useful infrastructure lessons come from deployments that did not work as expected.

A failed container may teach you how Linux permissions interact with numeric user IDs. A 502 error may reveal the difference between host ports and container ports. A disappearing database may demonstrate why persistent volumes matter. An unreachable service may expose a misunderstanding about Docker DNS or network boundaries.

These incidents are frustrating, but they build practical knowledge that clean tutorial environments rarely provide.

Production systems do not normally fail in textbook ways. Several small issues may combine: a nearly full disk, oversized logs, an automatic image upgrade, and an aggressive restart policy can create one confusing incident.

Final Thoughts

Docker makes application deployment more portable, repeatable, and manageable, but it does not remove the need to understand the systems underneath it.

Effective Docker troubleshooting requires knowledge of Linux, networking, storage, process management, application configuration, security, and monitoring.

The most important habit is simple: follow the evidence.

Check the container state. Read the logs. Inspect the configuration. Test the network from the correct location. Verify volumes and permissions. Check the host’s resources. Separate the application from the proxy. Change one thing at a time.

The next time a Docker container fails, resist the urge to delete everything and begin again. The answer is usually already present in the logs, runtime configuration, network design, storage mapping, or host environment.

Docker troubleshooting becomes much less intimidating once every failure is treated as an investigation rather than an emergency.

Frequently Asked Questions About Docker Issues

Why does my Docker container exit immediately?

A Docker container exits when its main process stops. Common causes include invalid configuration, missing environment variables, permission errors, failed dependency connections, and incorrect startup commands. Use docker logs and docker inspect to identify the cause.

Why is my Docker container running but not accessible?

The application may be listening on the wrong interface, the port may not be published correctly, a firewall may be blocking traffic, or the reverse proxy may point to the wrong hostname or port. Test the application both inside the container and directly through the host port.

Why can one Docker container not connect to another?

The containers may not share a Docker network, the wrong service name may be used, or the destination service may not be listening on the expected port. Inside Docker, use the Compose service name rather than localhost.

How do I find which process is using a Docker port?

Run ss -tulpn on the Docker host and filter for the relevant port. If the host port is already occupied, change the host side of the Docker port mapping or stop the conflicting service.

Can restarting Docker solve container problems?

Restarting Docker may temporarily restore a service, but it can also hide the original cause. Logs, container state, disk space, memory, network connectivity, and volume permissions should be checked before restarting the Docker daemon or server.

How can I prevent Docker logs from filling the disk?

Configure log rotation using the Docker logging options, such as max-size and max-file. Production environments may also benefit from centralized logging with Grafana Loki, OpenSearch, Elasticsearch, or a managed logging platform.

Should I use the latest Docker image tag?

The latest tag is convenient but unpredictable. Pinning a tested image version improves repeatability and makes upgrades and rollbacks easier to control.

Are Docker volumes automatically backed up?

No. Docker volumes provide persistent storage, but Docker does not automatically create external backups. Important volumes and databases need a separate, tested backup and restoration process.

The post Docker Issues & Troubleshooting: Real-World Solutions appeared first on John Nessime.