By the time I got to the hard part, everyone had quietly agreed it wasn't going to happen.

Nobody cancelled anything. It was softer than that. The vendor stopped mentioning it on calls, my management stopped putting it in the summary slides, and the phrase "hybrid multi-cloud" started turning up with "if feasible" attached to it. In corporate, that's how a requirement dies.

I didn't argue with any of it. I just carried on.

Some months later I sat and watched dnf update run on a server with no route to the internet, pulling its packages out of an S3 bucket it couldn't see, over a private interface in a cloud it wasn't running in.

That probably needs some explaining.

The Brief

Stripped of anything that would identify it, the job was this: prove that a set of files could move from one point to another inside a heavily restricted enterprise network, pass through an encryption layer (SDK) on the way, and have every transaction recorded on a private blockchain so that nobody could later alter the data without it being obvious.

Blockchain was the integrity layer and nothing else. It is very good at proving a thing hasn't been tampered with, and useless as a place to actually keep the thing. The real data still had to live somewhere, and the requirement was that it live in three independent storages, so that no single platform going down, going rogue or going out of contract could take the evidence with it.

In this environment, three independent storages meant:

  1. AWS S3, in a corporate account.
  2. A private cloud's Object Store (OSS), on the platform we ran in house.
  3. A third one that didn't exist yet, which I was going to have to build.

A note on vocabulary, because I would rather be accurate than impressive. This got labelled a hybrid multi-cloud and that is how it went into the slides. Strictly, it was two clouds and a pile of bare metal. The third target was MinIO (on the bare metal), which is self hosted object storage and not a cloud by any definition I would defend. What made the three interchangeable wasn't that they were clouds. It was that all three speak the same S3 API, even though the infrastructure underneath each one is completely different.

Then the constraints, which are the actual subject of this post.

No public traffic. Everything traversed the enterprise intranet. Not "prefer internal." Internal by default, with no direct route out. The one flow that genuinely had to leave the estate, the blockchain peering, went out through a proxy and firewall on that same central network, because that was the only way out.

A stateless firewall sat on every subnet boundary. Rules matched on source IP, port, protocol and destination URL. Stateless, so the return path isn't implied. You whitelist the traffic going out, and then you write a second rule for it coming back. If you forget the second one, you get to spend an afternoon working out that you forgot the second one.

Every rule went through a form, with a written justification, read by a human being.

Sitting in the middle of that was a handful of Windows VMs on a single subnet, running on VMware that was nearing end of life. That was what I had been given to work with, and it needed to run a containerised blockchain SDK and peer with an external network.

Docker, on Windows, in a VM, on a hypervisor with an expiry date, with no internet.

The Windows Problem

I never got the containers talking to each other.

That's the honest version, and I sat with it badly for a while before I could say it out loud. Several containers, the SDK among them, all needing to find each other, and they wouldn't. What I got instead was an error about a port. I don't remember the wording and I'm not going to invent it. I remember that it read like a permissions problem and wasn't one, and that something which worked on the Monday failed on the Tuesday with no change from me.

Underneath that, on a loop, Docker Desktop wanted to update the WSL 2 kernel. The update is a download from Microsoft. There was no internet. So it asked, and I couldn't answer, and it asked again. Over and over.

That's the part that actually decided it. Not the networking. The container platform needed the internet in order to keep working, on a network whose entire premise was that there was no internet.

Most of what follows, I worked out afterwards. At the time I knew it was broken and roughly where, and not much more than that.

Linux containers don't run on the Windows kernel. Docker Desktop quietly runs a Linux VM and runs them in there. You type docker run, you get a container, and it isn't on the machine you think it's on. Mine was two machines down.

That inner VM (WSL 2 / Hyper-V Linux VM) needs nested virtualisation. On VMware that's a per VM setting a vSphere administrator has to enable, and my Windows machines were themselves guests on VMware. It wasn't mine to turn on, and I never got as far as asking.

Publishing a port isn't a kernel route there the way it is on Linux. It's a user space proxy on the Windows side forwarding into the Linux VM. Windows reserves blocks of ports for itself at boot, and those blocks move between reboots. That's almost certainly my Monday and Tuesday. The addresses move too, on a NAT'd subnet that can be reallocated across a reboot, which in an environment where every permitted flow is pinned to a source IP is an outage with a delay fuse on it.

Native Windows containers weren't a way out, because they only run Windows images and the SDK shipped as a Linux one.

I didn't root cause the port errors. There is a documented way to see the reserved ranges and pin the ones you need, and with a few more days I could probably have got that stack talking to itself.

I chose not to, and I would make the same call again. Those days would have bought me a fragile arrangement that still sat inside a VM the firewall couldn't address, and still needed a hand carried kernel update at every bump. I wasn't debugging a bug. I was paying interest on a platform choice.

Every one of those problems is solvable on its own. Stacked together, on an air-gapped subnet, behind a stateless firewall, with a form for every rule, they stop being a list of problems and start being a design telling you it's wrong.

Replacing the Foundation

I replaced the whole thing with Linux. Two RHEL 9 VMs, built the way I wanted them, and not on VMware. They went on the private cloud, where the rest of the platform already lived.

So the rebuild moved two things at once. Windows to Linux, because the container runtime belonged on the kernel it was written for. VMware to the private cloud, which I will not dress up as strategy after the fact: the VMware estate was nearing end of life and the private cloud was where everything else already lived. Obvious, not brave.

The notable part of that sentence isn't the technical judgement. It's that I was allowed to make it. I could look at the environment I had been handed, say that it was the wrong foundation and that I was rebuilding it, and nobody made me justify it for three weeks first. That kind of latitude is rarer than good ideas are, and I have since learned to treat it as the thing you actually optimise a job for.

The judgement itself wasn't a gamble. It was the most confident call I made on the project, for a fairly boring reason: I had just spent months of nights configuring Red Hat servers by hand.

I sat the RHCSA in 2025. It's a purely practical exam: no multiple choice, nothing to guess at. They sit you in front of broken systems, you fix them, and the grader is a machine that either finds a working configuration or doesn't. I scored 286 out of 300 against a pass mark of 210. The one thing I dropped was a grep command, in "essential tools", which I think about more often than is healthy.

I didn't take it for a job posting. I took it because I wanted to stop guessing. It paid for itself in about a week here, because at the moment the right answer was to tear out the Windows estate and build on RHEL 9, I didn't have to wonder whether I could operate the thing I was proposing. I knew what the build looked like. I had done it at 1am, repeatedly and badly, until it wasn't bad any more.

Certifications attract a lot of cynicism, most of it earned. The ones that made me better were the ones that made me do something with my hands.

The Build Order

MinIO went first. The third storage had to be genuinely independent, so it couldn't live on either cloud. I built it on the bare metal servers, machines sitting on a separate internal segment that were reclaimed rather than purchased, with RHEL 9 installed from scratch and MinIO on top.

I deliberately didn't do it alone. I pulled in teammates who had never built a server and walked them through it: the OS install, the disks, the service, the checks. It was slower than doing it myself, and it was the only way the thing survives me. By the end they could build it without me in the room. I would rather have a team that can rebuild it than a monument with my name on it.

Then the firewall rules. My VMs were in the private cloud, MinIO was on bare metal on a different internal network, and between them sat two network domains and a stateless firewall that assumed nothing. Every flow had to be requested, specified down to source IP, port, protocol and destination URL, and justified in writing, then written a second time for the return path.

This is the part of architecture that nobody puts in a diagram, so I will put it in this one. It got done quickly because I had friends in the security office. That came from years of not being the engineer who submits a vague form at 5pm on a Friday and then chases it, and years of writing requests that were already correct so the person on the other side didn't have to do my thinking for me. They prioritised my tickets because I had earned it, one boring well written request at a time.

Relationships are infrastructure. They have uptime, and you maintain them, or they aren't there when you need them. I have been turning that thought over for a while now, because the human layer of a system is the one with no documentation, no diagram and no rollback, and it is harder to design than any of the rest of it. That is its own post, and one day I will write it.

Then OSS. The private cloud team, same story and same goodwill, provisioned a bucket for me. I established and tested connectivity from my VPC. That was 2 of the 3 storages built, MinIO and OSS, and both of them sat inside the perimeter.

Routing, Thirteen Years On

I had head knowledge of AWS at that point. I had read a lot, I worked with VPCs and understood them properly, and past the VPC my confidence dropped off into a well organised list of services I could define but had never fought with.

What I was genuinely confident about was routing, and the reason is a year at a Cisco Networking Academy about thirteen years ago, back before exam dumps became an industry. There was no shortcut lying around to tempt me, so I learned it the long way. I drew it. I filled notebooks with my own illustrations of how a packet moves, where the decision points are and what the header looks like at each hop, and I kept redrawing them until I could explain the whole path out loud to another person without looking down. That method has a post of its own, How I Learn Hard Things, because it's behind everything else here too.

Which turned out to be the only test that matters: can you explain it to someone else, unaided. I passed the CCNA at 722 out of 800. The certificate expired a long time ago. The mental model hasn't moved since.

Thirteen years later, sitting in front of an enterprise intranet, a private cloud VPC, three storage platforms and a stateless firewall between all of them, I wasn't lost. I knew where the packet was going, and I knew which boundary was refusing it and in which direction. The cloud part was new to me. The network part wasn't, and the network part was the hard part.

That's the compounding nobody mentions when you are 22 and drawing subnets in a notebook because you can't afford to be confused later.

The Unreachable Bucket

Which left S3. Outside the perimeter, across the boundary that didn't allow public traffic.

This is the part I actually want other people to have.

I didn't start with a list of options to choose from. I started with a question nobody around me could answer: can a machine that isn't allowed to touch the internet reach S3 at all? S3 is a public service with a public DNS name. My estate was internal by default with no route out. On the face of it those two facts can't both hold, and the quiet consensus was that they didn't.

So I asked an AI, and I had a name inside a minute. I'm not going to dress that up as a hunt through the documentation. Finding the term was the cheap part, and it is the cheap part for everyone now.

The expensive part was everything after it. I had a word, very little working knowledge of cloud networking to attach it to, and no appetite for putting something into a regulated environment on the strength of an answer I couldn't explain. So I spent evenings in the AWS networking documentation going backwards from the name until I actually understood the mechanism. Being told the answer and being able to defend it are different things, and I was the one who would be on the hook for it.

The thing is the interface endpoint, PrivateLink. It isn't a route and it isn't a DNS trick. It's an actual elastic network interface with a real private IP address, living in a subnet in your VPC, with its own DNS name. A private IP address in a subnet is something a network can be told how to reach, which means anything with a path into that VPC, whether a peering, a Direct Connect or an intranet attachment, can reach S3 through it.

That was the whole unlock, and it inverts the way the problem is usually framed. The bucket never has to come to my network. The door to it gets an address inside a VPC my network already knows how to reach.

We already had a VPC attached to the intranet, so the path into it existed. I created an interface endpoint for S3 inside it, pointed my clients at the endpoint's own DNS name rather than the public S3 name, and authenticated with an API key and secret against a private bucket. No public traffic, no proxy, and no exception request against the internet policy, because there was no internet involved.

The thing everyone had written off as infeasible took an afternoon, once I knew the primitive existed.

What "Private" Means

I want to be accurate about the security posture, because it's the part I have thought about most since.

What I built was protected by credentials alone, and I only really understood what that meant when I tested it. I sat at my own laptop on a home connection, nowhere near the enterprise network or the VPC, and pointed the API key and secret at the bucket. It let me straight in.

That bothered me for a long time. I had just spent weeks proving that no traffic left the estate, and the storage at the end of it would hand its contents to anyone on the internet holding the right string. So I went looking for what "private" actually means for an S3 bucket, because the word clearly didn't mean what I assumed it meant.

I'm a network person, so private to me means layer 3 unreachable: the packet doesn't arrive, and the thing on the other end never even knows you tried. S3 isn't that and can't be. It's one large shared object store, and every bucket in a region sits behind the same front end fleet on the same addresses. An address serving my bucket is also serving every other customer's, so it can't be told to refuse anyone at the packet level. Follow a denied request through and you can see it: DNS resolves, the TCP handshake completes, TLS negotiates, the request goes across, S3 checks the signature and works out who you are, and only then does an application say no. Every control on a bucket sits at layer 7.

So the fix was policy rather than plumbing. A bucket policy that only accepts requests carrying the right aws:sourceVpce, an endpoint policy narrowing what that endpoint can be used for at all, and an organisation level SCP as the outer guardrail so nobody can quietly undo either. Nothing about the network changes. A leaked key stops working from any laptop on the internet and only works from inside the VPC, which is a much smaller blast radius on a stolen credential.

I didn't apply any of that at the time, and I should be honest about why. I needed the door open. The only machine that could download Red Hat packages was my laptop, on the internet. The machines that needed them couldn't reach anything at all. The single thing those two ends had in common was a bucket that would accept a correctly signed request from either side, and I used exactly that. Lock it to aws:sourceVpce and the air gapped servers can still read from it, and I can no longer put anything in there for them to read.

I passed the AWS Solutions Architect Professional sometime after this. Most of what's in this section is on that syllabus, and I had met all of it on a live system first.

Patching Without a Repository

Now for my favourite part, which is also the part I nearly tripped over.

My two RHEL 9 VMs were up. They were also freshly installed and air gapped, and therefore missing every update and every package released since the image was cut. No repository, no mirror, no internet. And I was about to install a container runtime and a vendor SDK onto them, among other things.

If you know Red Hat, you are already asking why I didn't use Satellite. It is the correct answer to this problem, and exactly what it is built for.

It wasn't there. Nobody in that environment was running one, so using it meant asking for a Satellite to be procured, provisioned and maintained, on a timeline that was already tight, in order to solve a problem I could solve with storage I had been told to build anyway. I was proving a concept, not building a patching platform for the estate.

So I used the door I had just found for something it isn't usually used for, and made S3 the repository.

The shape of it goes like this.

  1. Get a Linux shell. The packages had to be resolved on a machine that speaks RPM, and my laptop is a Mac, so I ran one in a Docker container. Red Hat's own repositories need a subscription, so the container image was AlmaLinux 9, which is binary compatible with RHEL 9 and has public repositories.

    docker run --platform linux/amd64 -it -v ~/rpms:/rpms almalinux:9 bash

    That gives a throwaway el9 shell with a folder from my laptop mounted inside it.
  2. Download the packages and their dependencies. Not just the packages named, but every dependency, and every dependency of those, for the right architecture and release. Inside the container, dnf download --resolve --alldeps walks the whole tree and writes to the mounted folder. It produced 291 RPMs.
  3. Turn the folder into a repository. A directory of RPMs isn't a repository until it has metadata. createrepo_c over the directory writes a repodata folder next to the packages. Without it, dnf reports an empty repository.
  4. Push it to S3. tar czf packages.tar.gz packages/, then aws s3 cp the archive into a new bucket. From my laptop, over the public internet, on a home connection, with an API key and secret. This is the open door from the previous section, used deliberately.The container images went up the same way, since an air gapped machine can't pull from a registry either. docker save writes an image out as a tarball, but on Apple Silicon the pull and save kept losing the architecture I asked for, so I used crane pull --platform linux/amd64.
  5. Pull it down on the other side. dnf can't fetch from a private bucket, because the requests need SigV4 signing and dnf doesn't sign anything. The AWS CLI does the fetching instead.So the AWS CLI goes on the RHEL 9 boxes first, downloaded and carried across the same way as everything else. Then aws s3 cp pulls the archive down through the interface endpoint, with --endpoint-url set to the endpoint's own DNS name so the client never attempts a public lookup. Unpack it, write a .repo file with a baseurl of file:/// pointing at where it landed, then dnf clean all and dnf makecache.

And it works.

dnf update ran on a machine with no route to the internet, from a bucket it couldn't see, over a private interface in a cloud it wasn't running in.

The Rest Was Work

Once the VMs were patched, everything else fell over quietly. The container runtime and the blockchain SDK images went on the same way, since the pattern was already built. Peering to the external blockchain network went out through the one egress firewall on the central network, which was another stack of forms and another round of exactly specified rules in both directions.

Then we ran it. Data moved from one point to another, through the encryption layer, with each transaction written to the chain. Three independent buckets, all of them reachable, none of them touching the public internet.

The thing that had been gently downgraded to "if feasible".

What I Took From It

For a long time I treated the restrictions as obstacles standing between me and the design, and they weren't. No public egress, stateless firewalls, three independent backends. Read those three lines again and the answer is sitting inside them. The design was fully specified by what I wasn't allowed to do, and all I had to do was stop resenting the list long enough to read it properly.

The cloud part of all this was new to me. The part that made it solvable was a year of drawing packet paths in a notebook when I was young and had nothing else to offer. Nobody tells you that the thing you learn properly at 22 is still paying out at 35.

And every hard technical dependency in the project routed through a person. The security office, the firewall and private cloud teams, the teammates who built the MinIO nodes with me. Not one of them owed me anything. The forms got signed because of existing good relationships. They helped because I had spent years being easy to help.

If your environment has told you something is impossible, it's worth checking whether it actually said that, or whether it handed you a very specific list of what it will allow and left you to notice that the list is a design.

What if the rules you are working around aren't in the way of the architecture? What if they are the architecture, waiting for you to read them properly?

A note on what is real here. Some situational details have been altered or invented, and identifying context has been left out, to keep the organisation, the vendor and the people involved unidentifiable. The technical content is not fabricated: the constraints, the failure modes, the primitives and the fixes are all as they actually were.