Our GitOps platform had grown to serve dozens of engineering teams deploying hundreds of applications across multiple Kubernetes clusters in different AWS regions. The core deployment mechanics worked well enough, but the operational challenges had shifted. Teams could not easily see which ApplicationSets belonged to them. Security boundaries between team namespaces were blurry. There was no built-in mechanism to verify that manifests had not been tampered with between commit and deploy. And onboarding new teams meant platform engineers manually configuring RBAC for every new project.
These are not beginner problems. They are the problems that emerge when GitOps succeeds and scales beyond a single team.
Evaluate how Argo CD 3.5 addresses the challenges we were experiencing at scale, specifically around multi-team visibility, security posture, supply chain integrity, and developer self-service. The goal was to determine which new capabilities could be adopted immediately and which required changes to our platform architecture.
Previously, ApplicationSets were invisible in the Argo CD UI. You could see the generated Applications but had no way to trace them back to the template that created them. Version 3.5 introduces first-class ApplicationSet visibility in the dashboard, showing the relationship between sets and their child applications. This alone reduced the number of "who owns this app?" questions from teams significantly.
Before 3.5, all ApplicationSets lived in the argocd namespace. This made it impossible to give teams true ownership over their deployment templates without granting them access to the Argo CD control plane namespace. Now ApplicationSets can exist in any namespace, meaning each team manages their own sets within their own namespace boundary. Platform engineers define the guardrails; teams operate within them.
Supply chain security moved from "nice to have" to "mandatory" after several high-profile attacks in the industry. Argo CD 3.5 introduces source integrity verification, allowing the platform to cryptographically verify that the manifests being deployed match what was committed. This closes the gap between "we trust our Git repository" and "we can prove nothing was altered in transit."
Zero Trust networking within the Argo CD control plane is now achievable through mutual TLS between all internal components. In a shared cluster where multiple teams have workloads running, this prevents lateral movement attacks from compromising the deployment pipeline. We enabled mTLS between the API server, repo server, and application controller without any application-level changes.
One of the most impactful features for our platform. Impersonation allows Argo CD to assume different identities when deploying to target clusters, governed by RBAC policies. This means developers can trigger deployments through self-service workflows while the platform enforces what each team can actually deploy and where. No more platform engineers acting as gatekeepers for every deployment configuration change.
The Source Hydrator addresses a subtle but important problem: ensuring that what you see in Git is exactly what gets deployed, even when Helm values, Kustomize overlays, or parameter overrides are involved. It pre-renders manifests and commits the hydrated output, giving teams a clear audit trail of the exact resources that will be applied. This eliminated an entire class of "it worked in staging but not production" issues caused by environment-specific rendering differences.
With Helm 4 introducing breaking changes to chart packaging and dependency management, having native support in Argo CD prevents teams from being stuck on older chart versions. We migrated several charts to Helm 4 format and confirmed that the rendering pipeline handled them correctly without requiring workarounds.
Argo CD 3.5 represents a maturity shift for GitOps platforms. The features are not about making basic deployments work; they are about making GitOps governable, secure, and scalable across an entire organisation. After adopting these capabilities:
GitOps platforms are maturing from "alternative deployment tool" to "enterprise delivery platform." Argo CD 3.5 is a significant step in that direction, and the features align directly with the challenges platform teams face when operating at scale.
]]>Our engineering organisation operates across multiple AWS accounts following a landing zone pattern: separate accounts for production, staging, development, shared services, security, and logging. Engineers investigating incidents or performing routine operational checks needed to manually switch between accounts, navigate to the relevant service console, and mentally correlate data from CloudWatch, EKS, ECS, Lambda, RDS, and other services. A single investigation could involve five or more accounts and a dozen different dashboards.
The cognitive overhead was significant. Senior engineers could do it from muscle memory, but junior engineers struggled, and everyone found it tedious. Information that should take seconds to retrieve was taking minutes or hours to assemble.
Build an AI-powered agent that integrates with our existing enterprise platform to provide engineers with contextual insights across the entire AWS organisation. The agent needed to respect existing security boundaries, authenticate properly across accounts, and deliver accurate, correlated information in response to natural language questions. Security, governance, and operational efficiency were non-negotiable requirements.
The agent accepts questions in plain English. Engineers ask things like "What caused the spike in errors on the payment service in production yesterday afternoon?" or "Show me the resource utilisation across all EKS clusters in eu-west-1." The agent interprets the intent, identifies which accounts and services to query, and assembles a coherent response.
Authentication uses cross-account IAM role assumption with strict least-privilege policies. The agent assumes read-only roles in target accounts, with each role scoped to the minimum permissions needed for data retrieval. All role assumptions are logged in CloudTrail, providing a complete audit trail of what the agent accessed and when. No long-lived credentials are stored anywhere in the system.
The agent correlates data from across the AWS ecosystem:
Rather than presenting raw data from each service, the agent correlates information to provide explanations. A "why is the service slow?" question might combine RDS connection pool exhaustion, ECS task CPU throttling, and a recent CloudFormation deployment into a single narrative.
Beyond AWS native services, the agent integrates with:
The agent maintains awareness of the organisational structure, knowing which teams own which accounts, which services run where, and what the expected baseline behaviour looks like. This means responses are contextualised: the agent does not just report that CPU is high; it reports that CPU is higher than the baseline for that specific service in that specific account at that time of day.
The impact on operational efficiency was immediate and measurable:
The focus has shifted from collecting data to resolving problems. Engineers ask a single question and receive a consolidated explanation that would have taken dozens of clicks, account switches, and mental correlation to assemble manually. The agent does not replace engineering judgement; it removes the friction that prevented engineers from applying that judgement quickly.
]]>Engineering teams across the organisation needed more than raw Kubernetes clusters. They needed a platform they could trust across multiple AWS regions, one that handled provisioning, deployments, networking, security, and observability without requiring every developer to become a Kubernetes expert. Teams were spending too much time on infrastructure concerns and not enough on their applications. Each team had slightly different cluster configurations, addon versions, and deployment practices, leading to inconsistency, operational drift, and difficulty troubleshooting across environments.
Design and deliver a complete Internal Developer Platform built on Amazon EKS with standardised provisioning, GitOps-driven delivery, production-grade observability, and genuine developer self-service. The platform needed to work identically across multiple AWS regions, support dozens of teams with varying workload profiles, and reduce the time from "new team" to "first production deployment" from weeks to hours.
The foundation is a multi-region EKS deployment with consistent configuration across all regions. Each region runs identical cluster configurations with region-specific networking, ensuring that workloads can be deployed to any region without changes to deployment manifests. Failover patterns are built into the platform layer rather than requiring application teams to implement them.
Every cluster is provisioned through Terraform modules that encode our organisational standards. Node group configurations, VPC layouts, subnet strategies, and security group rules are all defined as code and versioned. Spinning up a new cluster in a new region takes a single pull request rather than days of manual configuration.
Clusters are delivered with a full set of pre-configured addons: ingress controllers, certificate managers, external DNS, metrics servers, log collectors, and policy engines. These are not optional extras that teams install themselves; they are part of the platform contract. Every cluster has them, configured identically, upgraded in coordination.
All application deployments flow through GitOps pipelines. Teams commit their Helm charts or Kubernetes manifests to Git, and the platform handles reconciliation, drift detection, and rollback. There is no kubectl access in production. The deployment pipeline is the only path to production, and it enforces all organisational policies automatically.
Networking is automated end-to-end: VPC peering, transit gateway attachments, security groups, and network policies are all managed by the platform. IAM roles for service accounts (IRSA) are provisioned automatically when teams onboard. RBAC policies follow a least-privilege model, granting teams access only to their own namespaces and resources.
Every service deployed to the platform automatically receives DNS entries, TLS certificates, and properly configured ingress routes. Teams declare their desired hostname in their Helm values, and the platform handles certificate provisioning via Let's Encrypt, DNS record creation in Route 53, and ingress controller configuration. No tickets, no waiting.
Every workload deployed to the platform is automatically instrumented for metrics, logs, and traces. Dashboards are generated automatically based on workload type. Alerting rules follow organisational standards with team-specific notification channels. Teams have full visibility into their applications from the first deployment without configuring any observability tooling themselves.
Applications are packaged as Helm charts following a standardised chart structure. The platform provides base charts that encode best practices for resource limits, health checks, pod disruption budgets, and security contexts. Teams extend these base charts with their application-specific configuration, inheriting platform standards automatically.
We integrated a Radar UI that provides real-time cluster visualisation covering health status, namespace utilisation, workload distribution, traffic flows between services, Helm release status, and resource consumption patterns. This gives both platform engineers and application teams a single pane of glass for understanding what is running, where, and how it is performing.
New engineering teams now onboard to the platform in hours rather than weeks. They deploy securely, scale confidently, and troubleshoot efficiently without needing deep Kubernetes expertise. The platform absorbs the complexity of multi-region infrastructure, networking, security, and observability so that developers deliver value rather than wrestle with infrastructure.
Key outcomes:
The platform is not finished; it never will be. But it has reached the point where developers genuinely trust it, and that trust is the real measure of a successful Internal Developer Platform.
]]>We were operating a hybrid EKS cluster connecting AWS-hosted nodes with on-premises infrastructure. The cluster ran hundreds of services resolving thousands of DNS queries per second. CoreDNS pods were not distributing evenly across all nodes in the cluster. The Kubernetes scheduler, following its default behaviour, concentrated CoreDNS replicas on cloud nodes where resources were more readily available.
This meant on-premises workloads were sending DNS queries across the network to CoreDNS pods running in AWS. When network latency spiked or connectivity became unreliable, those queries failed silently. Applications received SERVFAIL responses or timed out entirely, but the errors were intermittent and did not correlate with any obvious infrastructure event. We spent days tracking the root cause before realising it was a DNS topology problem.
Ensure reliable DNS resolution across the entire mixed topology at scale. The solution needed to handle hundreds of services resolving thousands of queries per second, work correctly regardless of whether a pod was running on a cloud node or an on-premises node, and be resilient to network disruptions between the two environments. Additionally, we wanted the solution to be reusable across other hybrid clusters rather than a one-off fix.
The first step was forcing CoreDNS pods to distribute across both cloud and on-premises nodes. We applied Topology Spread Constraints using both topology.kubernetes.io/region and kubernetes.io/hostname as topology keys with maxSkew: 1 and whenUnsatisfiable: DoNotSchedule. This guarantees that CoreDNS pods spread evenly across regions and individual nodes rather than clustering where the scheduler finds it convenient.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/region
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
k8s-app: kube-dns
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
k8s-app: kube-dns
With pods properly distributed, we needed to ensure there were enough of them under load. We configured an HPA targeting CoreDNS based on CPU utilisation and custom metrics for DNS query rate. During traffic spikes, the HPA scales CoreDNS replicas up, and the topology constraints ensure new replicas distribute correctly across the topology.
Even with properly distributed pods, the default Kubernetes service routing sends traffic to any backend pod regardless of topology. We patched the kube-dns service with trafficDistribution: PreferClose, which instructs kube-proxy to prefer routing to pods that are topologically close to the client. On-premises workloads now prefer on-premises CoreDNS pods, and cloud workloads prefer cloud CoreDNS pods, with cross-topology routing only as a fallback.
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
spec:
trafficDistribution: PreferClose
The final layer deploys NodeLocal DNSCache as a DaemonSet on every node in the cluster. Each node runs a local DNS cache that serves queries from a local cache before forwarding to CoreDNS. This reduces the load on CoreDNS pods dramatically, eliminates network hops for cached queries, and provides resilience during brief network disruptions between cloud and on-premises environments.
NodeLocal DNSCache handles the majority of repeat queries locally, meaning that even if cross-topology connectivity degrades temporarily, applications continue resolving DNS from their local cache without errors.
After deploying all four layers:
We packaged the entire solution as a reusable IDP component: a Helm chart that deploys all four layers with environment-specific configuration. It is now deployed across every hybrid and multi-region cluster we operate. Application teams never think about DNS topology, which is exactly how it should be.
The key insight was that no single fix was sufficient. Topology constraints without traffic distribution still sent queries cross-topology. Traffic distribution without proper pod placement had nothing local to route to. Local caching without proper CoreDNS scaling still had a fragile backend. Each layer addresses a different failure mode, and together they provide robust DNS resolution regardless of topology.
]]>Every EKS version upgrade was a multi-day affair. The process involved checking compatibility matrices for every addon, coordinating maintenance windows with application teams, taking backups, running pre-flight checks, upgrading the control plane, upgrading node groups in sequence, upgrading each addon in the correct order, validating cluster health at every step, and communicating progress to stakeholders. A single upgrade consumed approximately four days of a platform engineer's time, and we had multiple clusters to maintain across environments and regions.
The process was well-documented in runbooks, but the documentation itself was part of the problem. Engineers followed steps manually, occasionally missed validation checks, and spent significant time on communication overhead rather than technical work. The toil was predictable and repeatable, which made it a prime candidate for automation.
Automate the entire EKS upgrade lifecycle while maintaining the same level of confidence in cluster stability that a careful manual upgrade provides. The solution needed to handle the full sequence from pre-upgrade validation through post-upgrade verification, including addon compatibility, proper sequencing, health checks, and stakeholder communication. It also needed to be trustworthy enough that platform engineers could let it run without constant supervision.
We built the upgrade agent using an agentic AI approach with steering files and spec-driven workflows. The agent is not a simple script with conditional logic; it reasons about the upgrade process, adapts to unexpected conditions, and makes decisions about sequencing based on the current cluster state. Steering files define the upgrade policies, required validations, and acceptable risk thresholds. Spec-driven workflows define the expected sequence but allow the agent to adjust when conditions differ from expectations.
The agent begins by taking comprehensive backups: etcd snapshots, Helm release histories, custom resource definitions, and critical ConfigMaps. It then conducts a thorough cluster health assessment, checking node readiness, pod scheduling, persistent volume claims, certificate expiry, and resource utilisation. If any pre-flight check fails, the agent reports the issue and pauses for human review rather than proceeding with a potentially risky upgrade.
The control plane upgrade proceeds first, followed by node groups in a controlled rolling fashion. The agent monitors the upgrade progress, watching for nodes to drain cleanly and new nodes to become ready. It handles the addon upgrade sequence intelligently, understanding dependencies between addons and upgrading them in the correct order. For example, it knows that the VPC CNI plugin must be upgraded before certain other networking addons, and that CoreDNS should be validated before proceeding with application-level addons.
After the upgrade completes, the agent runs the full health check suite again, comparing results against the pre-upgrade baseline. It validates that all workloads are running, all services are reachable, DNS resolution is functioning, and metrics pipelines are reporting correctly. Any regression triggers an alert and detailed diagnostic report.
The agent reports status to the team throughout the process via Slack notifications at key milestones. Upon completion, it publishes a comprehensive summary including the new version capabilities, relevant release notes, any issues encountered and how they were resolved, and the full timeline of the upgrade. This documentation is automatically published to Confluence, creating a permanent record that teams can reference.
The impact was transformative for the platform team:
Customers deploy on upgraded clusters immediately with complete confidence because the agent has already validated everything they would need to check manually. The upgrade process went from being a dreaded quarterly event to a routine operation that barely registers on the team's workload.
The key lesson was that the value of the AI agent is not just speed. It is consistency, thoroughness, and the elimination of human error in a process where a missed step can have significant consequences. The agent never skips a validation check because it is Friday afternoon.
]]>We were setting up a hybrid EKS cluster that connected AWS-hosted nodes with on-premises nodes running on Nutanix infrastructure. The networking requirements were non-trivial: pods on AWS nodes needed to communicate seamlessly with pods on on-premises nodes, NetworkPolicies needed to work consistently across both environments, and we needed deep visibility into traffic flows for troubleshooting and security. We tried multiple CNI plugins, and each one broke in different ways when confronted with the realities of hybrid networking.
Get networking right across the mixed environment with deep visibility and security. The CNI needed to handle cross-environment pod-to-pod communication, provide consistent NetworkPolicy enforcement regardless of where pods were scheduled, offer observability into traffic flows without requiring additional tooling, and perform well enough that the networking layer was never the bottleneck.
We chose Cilium for several reasons that aligned with our hybrid requirements:
On-premises nodes experienced DNS resolution latency of approximately 150ms because every query traversed the network to CoreDNS pods running on AWS nodes. We deployed NodeLocal DNS v2.0.6 with Local Redirect Policy (LRP) in Cilium, which intercepts DNS traffic at the eBPF level and routes it to a local cache without leaving the node. Latency dropped from 150ms to 40ms immediately.
The default IPAM configuration allocated pod CIDRs that overlapped with existing on-premises network ranges. Pods could not communicate across environments because routers did not know which path to take for overlapping addresses. We switched to cluster-pool IPAM with explicitly split CIDRs: one range for AWS nodes and a separate, non-overlapping range for on-premises nodes. This gave us predictable, routable pod addresses across both environments.
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4PodCIDRList:
- "10.100.0.0/16" # AWS nodes
- "10.200.0.0/16" # On-premises nodes
Hubble Relay could not connect to Hubble instances on on-premises nodes. The Relay was configured with an internal FQDN that did not resolve correctly across the hybrid boundary. We corrected the internal FQDN to use the cluster DNS name that was resolvable from all nodes, and added a backoff delay configuration to handle the brief periods during node scaling where some Hubble instances are temporarily unavailable.
We initially attempted to run Cilium in full kube-proxy replacement mode. This worked perfectly on on-premises nodes but conflicted with AWS-specific networking that expects kube-proxy to be present for certain load balancer integrations. Rather than fighting this, we disabled full kube-proxy replacement and let AWS handle kube-proxy on cloud nodes while Cilium managed the data plane. This mixed mode required explicit configuration but proved stable.
kubeProxyReplacement: "false"
nodePort:
enabled: true
After resolving all four issues, we achieved:
The cluster has been running in production for months without networking-related incidents. Cilium's eBPF foundation means we are not accumulating iptables rules as the cluster scales, and Hubble gives us the observability we need to quickly identify and resolve any networking anomalies.
The key takeaway is that hybrid networking is harder than it appears on paper. Every CNI plugin claims to support hybrid environments, but the edge cases around IPAM, DNS, proxy replacement, and service discovery only surface when you actually deploy across mixed infrastructure. Being willing to iterate and fix each issue systematically rather than switching CNIs again was the right call.
]]>The client had been running Ansible Tower for several years to automate provisioning, configuration management, and application deployments across a hybrid cloud estate spanning on-premises data centres and two public cloud providers. Tower was doing its job, but cracks were forming.
Playbooks ran in a shared Python environment with no isolation. One team's dependency upgrade would break another team's workflows. Execution control was limited, meaning there was no way to guarantee reproducible runs across environments. Scaling horizontally required spinning up more Tower nodes, which introduced state synchronisation headaches. Credential management was scattered, with some stored in Tower, others in flat files, and a handful in a partially configured Vault instance.
The writing was on the wall: Tower could not keep pace with the organisation's growth.
Our mandate was clear. Migrate the entire automation estate from Ansible Tower to the Ansible Automation Platform (AAP), making it enterprise-ready, secure, and scalable across three geographic regions. The migration needed to be non-disruptive, meaning production automation could not go dark during the transition.
We started by cataloguing every job template, workflow, credential, inventory, and project in Tower. Using awx-manage export, we extracted the full configuration into a portable format. This gave us a baseline to validate against post-migration.
One of AAP's biggest shifts is the move to containerised Execution Environments (EEs). We used ansible-builder to construct purpose-built EEs for each team, bundling the exact Python dependencies, collections, and system libraries each workflow required. This solved the "it works on my Tower node" problem permanently.
We automated the deployment of Automation Controller, Automation Hub, and Automation Mesh using infrastructure-as-code. Mesh nodes were deployed at each region to provide local execution capacity with centralised orchestration. Vault was integrated as the single source of truth for all secrets and credentials.
Six significant issues surfaced during migration, and each required a targeted fix:
1. Incompatible playbooks with Execution Environments
Many playbooks relied on system packages or Python libraries installed directly on Tower nodes. These broke immediately inside containerised EEs. We used ansible-builder to create layered EE definitions, pulling in the required dependencies. For edge cases, we refactored playbooks to remove host-level assumptions.
2. Credentials migration to Vault
Tower stored credentials in its own encrypted database. Migrating these to Vault required building custom credential plugins that AAP Controller could consume natively. We wrote a migration script that extracted credentials from Tower's export, transformed them into Vault KV paths, and configured matching credential types in Controller.
3. Mesh node sync failures
Automation Mesh communicates over port 27199 using receptor. In the first deployment, hop nodes could not establish connections to execution nodes. Debugging revealed a combination of firewall rules not updated for the new port and DNS resolution failures where short hostnames were not resolving correctly across regions. We fixed the firewall rules and enforced FQDN usage in the mesh topology configuration.
4. Failed job templates post-migration
After importing job templates, roughly 30% failed on first run. Root causes varied: missing EE associations, changed variable names, and inventory source mismatches. We built a dry-run pipeline that executed every imported template in check mode before cutover. This pre-deployment sanity testing caught failures before they reached production.
5. Confused users and broken workflows
Teams accustomed to Tower's interface and workflow struggled with Controller's new layout and RBAC model. We created onboarding guides, pre-built role templates aligned to team functions, and integrated SSO so that users landed in their correct organisational context immediately on login.
6. Content validation in Automation Hub
Teams were pulling collections from Ansible Galaxy without any governance. We deployed an internal Automation Hub instance with curated, signed collections. Only validated and signed content could be consumed by EEs, giving the security team confidence in what was running in production.
The migration completed across all three regions with zero unplanned downtime during cutover. Post-migration metrics told the story:
The platform now serves as the backbone for all infrastructure and application automation across the organisation, handling thousands of job runs daily without the scaling concerns that plagued Tower.
]]>The organisation managed hundreds of AWS accounts across multiple business units. Each team had their own Terraform code, written independently with no shared standards. The problems were predictable:
The platform team needed to impose order without slowing teams down.
Build a composable Terraform module ecosystem following DRY principles with semantic versioning. Each module needed to be single-responsibility, independently testable, and composable into larger platform deployments. Teams should be able to self-serve infrastructure by combining modules without deep Terraform expertise.
We established a clear contract for every module: defined inputs, predictable outputs, sensible defaults, and comprehensive documentation. Each module lived in its own Git repository with its own release lifecycle.
1. VPC Module
Provisions a production-ready VPC with multi-AZ subnets (public, private, database), NAT Gateways with redundancy options, Internet Gateway, and VPC Endpoints for AWS services. Supports customisable CIDR allocation and flow log configuration.
2. EKS Module
Deploys managed Kubernetes clusters for both cloud-native and hybrid workloads. Handles node group configuration, cluster add-on management, OIDC provider setup, and cluster autoscaler prerequisites. Supports both managed node groups and self-managed configurations for on-premises integration.
3. IAM Module
Manages IAM roles, policies, and trust relationships with a focus on IRSA (IAM Roles for Service Accounts) and OIDC federation. Provides pre-built role templates for common patterns like CI/CD runners, application workloads, and cross-account access.
4. Networking Module
Handles Transit Gateway attachments, Route53 zones and records, VPN connections, and VPC peering. Designed for hub-and-spoke topologies where a central networking account manages connectivity for spoke accounts.
5. Security Module
Provisions Security Groups with rule composition, KMS keys with cross-account sharing policies, and WAF web ACLs with managed rule groups. Enforces least-privilege defaults that teams can extend but not weaken.
6. Karpenter Module
Deploys Karpenter for Kubernetes node autoscaling with all supporting infrastructure: IAM roles, SQS queues for interruption handling, EventBridge rules for spot termination notices, and NodePool/EC2NodeClass definitions as Kubernetes manifests.
7. EKS Addons Module
Manages cluster add-ons as a composable unit: CoreDNS, vpc-cni, kube-proxy, EBS CSI Driver, and Pod Identity Agent. Handles version pinning, configuration values, and IAM role associations for each add-on.
8. Tags Module
A utility module that generates standardised tags based on organisational policy. Every other module consumes this to ensure consistent tagging for cost allocation, ownership, environment classification, and compliance.
Individual modules are powerful, but the real value comes from composition. We used Terragrunt to orchestrate module deployments, manage dependencies between them, and keep environment configurations DRY. A single Terragrunt configuration can stand up an entire platform stack by referencing versioned modules.
Each module follows semantic versioning via Git tags. Breaking changes increment the major version. New features increment the minor version. Bug fixes increment the patch version. Consumers pin to a specific version and upgrade deliberately.
To make onboarding frictionless, we built Cookiecutter templates that generate complete project scaffolding. Engineers answer a few questions (account, region, environment, required modules) and receive a ready-to-apply Terraform project with Terragrunt configuration, CI/CD pipeline definitions, and documentation.
The module ecosystem transformed infrastructure delivery:
The ecosystem continues to grow as new patterns emerge, with each module maintained by the platform team and consumed by dozens of product teams across the organisation.
]]>The organisation ran over 40 microservices deployed to Kubernetes. Each team had its own deployment process, ranging from manual kubectl applies to partially automated scripts. Security scanning was inconsistent at best:
The security team was drowning in findings with no way to enforce standards proactively.
Build an end-to-end DevSecOps pipeline integrating security at every stage from commit to production. The pipeline needed to be consistent across all microservices, automated with no manual gates (except for critical findings), and provide real-time visibility into the security posture of every deployment.
Terraform provisions the EKS cluster with all networking, IAM, and security configurations. The cluster runs in a dedicated VPC with private subnets, and node groups are configured with hardened AMIs. This gives us a secure foundation before any application workload lands.
Every push to a repository triggers the pipeline:
Source Code Analysis (SAST)
SonarQube analyses the codebase for security vulnerabilities, code smells, and bugs. Quality gates enforce minimum thresholds. If the gate fails, the pipeline stops immediately and the developer receives actionable feedback.
Dependency Scanning
OWASP Dependency Check scans all project dependencies against the National Vulnerability Database. Known vulnerable libraries are flagged with severity ratings. Critical and high findings break the build.
Container Image Build
Docker builds the application image using a hardened, minimal base image. Multi-stage builds ensure that build tools and source code never make it into the final image.
Container Vulnerability Scanning
Trivy scans the built container image for CVEs in OS packages and application dependencies. The scan runs against continuously updated vulnerability databases. Images with critical vulnerabilities are rejected before they reach any registry.
ArgoCD watches Git repositories for changes to Kubernetes manifests. When a new image passes all security gates and its manifest is updated, ArgoCD synchronises the desired state to the cluster.
The application architecture follows a three-tier pattern:
ArgoCD provides drift detection, automatic rollback on failed health checks, and a clear audit trail of every deployment.
Prometheus scrapes metrics from all application pods, pipeline components, and cluster infrastructure. Grafana dashboards provide real-time visibility into:
Cloudflare sits in front of the infrastructure, managing DNS resolution and providing DDoS protection. Traffic routes through an Application Load Balancer to the appropriate Kubernetes services. This layered approach means malicious traffic is filtered before it ever reaches application pods.
The pipeline transformed the organisation's security posture:
Security shifted from being a bottleneck to being an enabler, integrated so deeply into the workflow that developers barely notice it is there.
]]>The client had a mature GitLab CI/CD pipeline for container image builds and deployments. Their workflow used a combination of Docker, Podman, and Kaniko for building container images depending on the runner environment. For vulnerability scanning, they used Klar as the scanning client, which communicated with a Clair v2 server to check images against known vulnerabilities.
The problem was that Klar only supports Clair v2's API. Clair v4 introduced a completely new architecture and API surface, along with significantly improved vulnerability data, particularly for Red Hat Security Advisories (RHSA). The client's Red Hat-based container images were not being checked against the latest RHSA vulnerabilities because that data only existed in Clair v4's format.
The client's constraint was firm: they did not want to replace Klar, change their scanning workflow, or re-architect their CI/CD pipeline. The existing process was well-understood, battle-tested, and integrated into their compliance reporting. They needed the latest vulnerability data injected into their existing system.
Enable up-to-date RHSA vulnerability scanning without changing the existing build and scan workflow. The solution needed to:
We created a dedicated GitLab pipeline scheduled to run nightly. This pipeline pulls the latest Clair v4 vulnerability database, which contains the most current RHSA advisories, CVE data, and security metadata for Red Hat packages.
The core engineering challenge was transforming the Clair v4 database schema into the Clair v2 format. Clair v4 stores vulnerability data in a fundamentally different structure compared to v2. We built a conversion process that:
This conversion runs as part of the nightly pipeline, producing a fresh database every 24 hours.
The converted database is deployed as a PostgreSQL container accessible to the Clair v2 server. When Klar sends a scan request, Clair v2 queries this database and returns vulnerability matches using the latest RHSA data, all through the existing v2 API that Klar expects.
From the perspective of development teams, nothing changed. Their GitLab CI/CD pipelines still build images with Docker, Podman, or Kaniko. They still scan with Klar. They still see vulnerability reports in the same format. The only difference is that the underlying data is now current and comprehensive.
Builds are automatically broken when vulnerabilities above the configured severity threshold are detected. This enforcement happens at the same pipeline stage as before, maintaining the existing workflow.
Images that pass vulnerability scanning are signed using Simple-Signing before promotion to the production registry. This creates a cryptographic chain of trust: only images that have been built by the pipeline, scanned against current vulnerability data, and found to be clean can be deployed to production.
The solution delivered exactly what the client needed without disrupting their established processes:
The approach demonstrated that constraints do not have to mean compromise. By engineering a bridge between the old and new systems, the client gained modern security capabilities without the cost and risk of a full tooling migration.
]]>The team was managing AWS infrastructure across three environments: development, staging, and production. Deployments followed a familiar but painful pattern:
terraform plan, review the output, then run terraform applyThe consequences were predictable. Development and staging drifted apart because changes were applied inconsistently. Production deployments were nerve-wracking because no one was entirely sure what state the other environments were in. Rollbacks were manual and stressful. New engineers took weeks to learn the deployment process safely.
Automate multi-environment Terraform deployments with zero manual intervention on merge to main. The solution needed to:
We organised the Terraform code with per-environment configurations:
infrastructure/
├── modules/
│ ├── networking/
│ ├── compute/
│ └── database/
├── environments/
│ ├── dev/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ ├── terraform.tfvars
│ │ └── backend.tf
│ ├── staging/
│ │ ├── main.tf
│ │ ├── variables.tf
│ │ ├── terraform.tfvars
│ │ └── backend.tf
│ └── prod/
│ ├── main.tf
│ ├── variables.tf
│ ├── terraform.tfvars
│ └── backend.tf
Each environment has its own backend configuration pointing to a dedicated S3 bucket and DynamoDB table for state locking. This prevents state file conflicts and ensures that concurrent operations against different environments cannot interfere with each other.
Every environment uses an S3 backend with DynamoDB state locking:
terraform {
backend "s3" {
bucket = "company-terraform-state-prod"
key = "infrastructure/terraform.tfstate"
region = "eu-west-1"
dynamodb_table = "terraform-state-lock-prod"
encrypt = true
}
}
This guarantees that only one operation can modify state at a time, preventing corruption from concurrent applies.
The automation workflow triggers on push to main and on pull requests:
On Pull Request:
terraform plan and posts the output as a PR commentOn Merge to Main:
terraform plan followed by terraform apply -auto-approveThe workflow uses matrix strategies to run against multiple environments in parallel where appropriate, with production gated behind a manual approval step in GitHub Environments.
AWS credentials are stored as GitHub Secrets with environment-scoped access. Production credentials are only available to workflows running against the production environment, which requires reviewer approval. This prevents a compromised development workflow from touching production resources.
Branch protection rules require at least one approval before merge. The Terraform plan output is posted as a comment on the PR, giving reviewers visibility into exactly what infrastructure changes will be applied. No one merges blind.
The automation transformed infrastructure delivery:
The pattern has since been adopted across multiple teams in the organisation, with each team running their own infrastructure repositories following the same workflow structure.
]]>Across multiple client engagements, we saw the same container security challenges repeated:
The result was a fragmented security posture where every team solved the same problems differently, introducing gaps and inconsistencies. Audits were painful, remediation was slow, and the risk of deploying a compromised image to production was real.
Evaluate and adopt Docker Hardened Images (HDIs) for production container security across our platform deployments. The goal was to replace the patchwork of manual security practices with a consistent, automated, and compliant foundation that works across AWS, GCP, and Azure environments.
Docker released HDIs as production-grade base images with security built into every layer. We evaluated them against our existing practices and compliance requirements:
Minimalist Base Layers
HDIs ship with only the packages required for the runtime. No compilers, no package managers, no debugging tools in the final image. This dramatically reduces the attack surface compared to standard base images where unused packages introduce potential vulnerabilities.
Non-Root User by Default
Every HDI runs as a non-root user out of the box. This aligns directly with Kubernetes Pod Security Standards, which enforce non-root execution at the cluster level. No more custom USER directives or entrypoint scripts to drop privileges.
SBOM Included
Each image ships with an embedded Software Bill of Materials. This provides complete software transparency without requiring separate SBOM generation tooling. Supply chain audits become a matter of inspecting the included manifest rather than reverse-engineering image contents.
Signed Images (Docker Content Trust)
HDIs are cryptographically signed, providing tamper-evident provenance. We can verify that an image has not been modified since Docker published it, closing the supply chain integrity gap.
Continuous CVE Scanning and Remediation
Docker continuously scans HDIs against vulnerability databases and publishes updated images when fixes are available. This shifts the remediation burden from individual teams to the image provider, with teams only needing to update their base image reference.
Compliance Alignment
HDIs are built to comply with CIS Benchmarks, SLSA (Supply-chain Levels for Software Artifacts), and NIST 800-190 (Application Container Security Guide). This provides auditable evidence of compliance without manual configuration.
We replaced existing base images with HDIs across our platform deployments:
The integration worked consistently across AWS EKS, GCP GKE, and Azure AKS deployments, with no cloud-specific modifications required.
We validated HDIs against our security requirements:
Adopting Docker Hardened Images simplified and strengthened container security across the board:
Docker Hardened Images represent a genuine shift in the container security model. Instead of every team independently solving the same security problems, the foundation is secure by default. Teams can focus on building application value rather than hardening infrastructure.
]]>