Container Security Best Practices

In an era defined by rapid software deployment and the proliferation of microservices, containers have become an indispensable tool for modern developers and operations teams. The flexibility, scalability, and efficiency they offer make them an attractive choice for packaging and deploying applications. However, with great power comes great responsibility, and securing containers is paramount in this dynamic landscape. In this comprehensive guide, we will delve into the best practices for container security, ensuring your applications and data remain protected within this revolutionary paradigm. We are going to touch on the following sections in this blog:

Container Image Hardening: Building a Strong Foundation

Container image hardening is the first line of defense in your container security strategy. Just like fortifying the walls of a fortress, hardening your container images defends them against potential vulnerabilities. We will explore the crucial steps for locking down your containers, including minimizing the attack surface, restricting user privileges, and leveraging namespaces and seccomp profiles to limit system calls.

Container Image Scanning and Signing: Guarding Against Vulnerabilities

In a world where cyber threats continually evolve, staying one step ahead is imperative. Container image scanning is a vital practice that identifies vulnerabilities within your images. Additionally, container image signing employs digital signatures to guarantee the authenticity and integrity of your images, ensuring that only trusted sources can deploy them. We will provide insights into the tools and techniques that make these practices an integral part of your security arsenal.

SBOMs for Container Images: Enhancing Transparency and Trust

The world of container security is all about transparency and trust. We’ll introduce you to the concept of Software Bill of Materials (SBOMs) for container images. By adopting this practice, you’ll gain a clearer view of the components within your containers, facilitating better decision-making, and ultimately bolstering security.

Monitoring Running Containers: Keeping an Ever-watchful Eye

Security does not stop at the deployment phase. Real-time monitoring of running containers is essential to identify any suspicious activities, unusual network traffic, or performance anomalies. We will discuss the tools and strategies to keep a vigilant watch on your containers, allowing you to swiftly detect and respond to security incidents. Our guide will also cover essential security practices for running containers in production, ensuring your containerized applications remain protected throughout their lifecycle.

Hardening Your Containers: Building a Robust Defense

Container hardening is like giving your containers a suit of armor – it’s all about strengthening their defenses to protect your applications and data in the ever-evolving landscape of container security. We’ll walk you through the essential practices to reduce your container’s attack surface and bolster their resilience to cyber threats. By following these steps, you’ll be well-prepared to ensure your containers are as secure as possible.

Harden the Code: Strengthening the Core

The first step in our container hardening process involves examining the very heart of your application – the code itself. Using Static Application Security Testing (SAST), you can scan your codebase for vulnerabilities and misconfigurations. Integrating SAST tools into your CI/CD pipeline helps ensure that the code you’re deploying meets a minimum standard of security and quality. By eliminating code vulnerabilities and misconfigurations early in the process, you significantly reduce your container’s attack surface.

Harden the Image: Slimming It Down for Safety

Once your code is in good shape, it’s time to focus on the container image. Hardening the image means reducing vulnerabilities and removing unused packages and libraries from the base image. Choosing the most minimal base operating system (OS) image is a great start to keeping your container secure. Consider using Distroless images, which contain only the application and its necessary dependencies, leaving out unnecessary components often found in a standard Linux distribution.

Additionally, using multi-stage builds and official base images can help further streamline the image and reduce its attack surface. Always ensure that your base image is sourced from trusted repositories, and if you must use a public image, undergo a rigorous hardening process to remove any unwanted dependencies. By applying these principles, your containers will be leaner, meaner, and more secure.

Remove Sensitive Information: Protecting Your Secrets

Docker images can inadvertently leak sensitive information contained in the Dockerfile, including secrets like passwords, API keys, and tokens. To maintain the integrity of your sensitive information, employ a secrets management tool, such as AWS Secrets Manager or KMS, during the build process. Using Gitleaks, a free and open-source tool, to scan for and eliminate secrets, safeguarding your container images from accidental exposure.

Running Non-root Users: Best Practices for User Privileges

By default, containers run with root privileges, which can pose security risks. It’s generally considered best practice to run containers as non-root users. You can easily achieve this by specifying a non-root user in your Dockerfile. Removing unnecessary users from your container further reduces potential vulnerabilities and helps keep the container configuration to a minimum, enhancing security. You can help enforce this practice with an admission controller such as Kyverno. Kyverno has a policy that is written to protect your cluster by denying any container from running if it is being run as root.

Use COPY Instead of ADD: Mitigating Security Risks

The ADD instruction in a Dockerfile can retrieve files from remote sources, potentially introducing security risks. To mitigate these risks, use the COPY instruction instead, which only copies files from your local machine to the container filesystem. This simple change aligns with best security practices, as it avoids potential security issues associated with ADD.

Avoid SSH Within Containers: Simplify Security Management

Running SSH within a container can complicate security management, making it challenging to control access policies and security compliance, manage keys and passwords, and handle security upgrades. To maintain the principles of immutability and container ephemeral nature, it’s advisable to avoid using SSH within containers. Troubleshooting can be accomplished more securely in lower environments using docker exec.

Container Image Scanning and Signing: Guarding Against Vulnerabilities

When it comes to container security, scanning isn’t a one-and-done affair; it’s a dynamic process that plays a vital role at multiple stages of your container’s journey. Think of it as shining a spotlight on your containers to ensure there are no hidden vulnerabilities lurking in the shadows. Let’s dive into how scanning fits seamlessly into your container security strategy.

Pipeline Scanning: Automation in Action

Your Continuous Integration and Continuous Delivery (CI/CD) pipelines are the unsung heroes in the battle for container security. By automating the scanning process, you ensure that vulnerabilities are caught before they make their way into your container registry. CI/CD pipelines streamline the development and delivery process, allowing developers to make small, iterative changes to their code while maintaining a consistent, reliable delivery pipeline. Meet Grype, an open source tool from Anchore which scans for vulnerabilities before your container gets its moment in the production spotlight.

Registry Scanning: Fortifying the Last Line of Defense

Before your container takes its final leap into the registry, it’s crucial to minimize vulnerabilities. However, it’s a reality that new vulnerabilities and exploits are always on the horizon, regardless of the age of your code. That’s where registry scanning comes into play. It keeps a vigilant eye out for new Common Vulnerabilities and Exposures (CVEs) by checking databases like the National Vulnerability Database (NVD). By scanning images in the registry, you ensure that even the most up-to-date threats won’t go unnoticed.

Runtime Scanning: Guarding Your Containers on the Front Line

Once your container is out in the wild, runtime scanning becomes your frontline defense. It not only unveils misconfigurations within your container but also tracks the assets and the images they’re running. This vigilance ensures that no vulnerable images slip into your production environment unnoticed. Snyk Container is one example of a modern tool able to perform runtime scans on containers used in the production environment. This tool uses an agent installed on the cluster to track the images in use, scan them in the registry, and flag any misconfigurations or known vulnerabilities, all in real-time.

Signing the Image: A Digital Seal of Security

Imagine having a seal of trust on your container image, guaranteeing its authenticity and protecting it from potential Man-in-the-Middle Attacks (MITM). That’s precisely what image signing achieves. It adds a digital fingerprint to your image, making it tamper-evident and verifying its source. When you push an image to the registry, cryptographic testing verifies its trustworthiness. The producer of the container image typically handles this process, and consumers verify signatures when they pull an image. If the signature doesn’t match or fails decryption, it’s a clear warning sign – this image is untrustworthy.

Sigstore, an open-source project, can handle your image signing with the assistance of Cosign. This dynamic duo adds the digital fingerprint to an image before it lands in the registry. This ensures that any future consumers of the image can trust its source and know it hasn’t been tampered with. If the signature doesn’t check out, you know it’s time to exercise caution and avoid deploying that image in your environment. Checkout my previous two blog posts, Signing Software Artifacts With Cosign and How to Sign Container Images Using Cosign, where I go more in depth with the importance of signing container images and how to utilize Cosign to do so.

Software Bill of Materials (SBOM): Illuminating Container Dependencies

In the intricate world of container security, knowing what’s under the hood is crucial, and that’s where the Software Bill of Materials (SBOM) comes into play. SBOMs are like a detailed inventory list of all the dependencies lurking within your containers. Think of them as your X-ray vision, peering deep into your container to expose even the most elusive vulnerabilities.

Why SBOMs Matter: Unearthing Zero-Day Exploits

The true power of SBOMs lies in their ability to uncover those elusive zero-day exploits – vulnerabilities that are so fresh they haven’t even been officially documented yet. Imagine having a map that pinpoints the exact containers housing these newfound vulnerabilities. SBOMs do just that; they dissect your container’s dependencies, leaving no stone unturned to identify what needs patching and eliminate potential vulnerabilities.

The Syft Connection: Unveiling Container Contents

Generating SBOMs is a crucial step in your container security journey, and that’s where Syft comes into the picture. Syft is an open-source tool from Anchore, acting as your trusted detective in the container world. Whether you’re dealing with container images or file systems, Syft is your go-to command-line tool and Go library for generating SBOMs. It meticulously lists out all the components in your containers, helping you understand their inner workings.

A Perfect Pairing: Syft and Grype

But the power of Syft doesn’t stop there. When paired with a scanner like Grype, it becomes an unstoppable force for vulnerability detection. Grype and Syft work hand-in-hand, making sure that you not only know what’s inside your containers but also uncover potential security threats before they become full-blown issues.

So, in the realm of container security, it’s not just about having the containers; it’s about understanding them inside and out. With SBOMs, Syft, and Grype, you’re well-equipped to navigate the labyrinth of dependencies, ensuring that your containerized applications remain secure and your organization stays protected.

Monitoring Running Containers: Keeping an Ever-watchful Eye

In the ever-evolving world of container security, striking a balance between proactive prevention and vigilant oversight is paramount. Below are some best practices for securely running containers, a crucial step in fortifying your digital ecosystem. Running containers in rootless mode, segregating networks, and enforcing resource limits are key steps to safeguard your containers while maintaining system integrity. By diligently following these best practices, you can significantly reduce your attack surface and minimize the impact of potential breaches.

Run in Rootless Mode: To enhance security, run your containers in rootless mode. This involves running the Docker daemon as a non-root user, ensuring that your containers are built entirely in userspace. Tools like kaniko come in handy, as they don’t require root privileges, and every Dockerfile command is executed within userspace. This approach adds an extra layer of security when creating container images.

Avoid Mounting Host Paths: While Docker volumes can be mounted in read-write mode, sensitive data on your host’s file system should remain off-limits. Minimize the use of host path mounting to ensure that your containers don’t have unnecessary access to sensitive host data.

Set Filesystems and Volumes to Read-Only: Running containers with a read-only file system aligns with the principles of immutability and container ephemeral nature. It reduces the risk of malicious activity, such as deploying malware or modifying configurations. You can set this attribute for the docker run command or as part of an ECS task definition when deploying containers in ECS/Fargate. It is important to clarify that this best practice primarily refers to the container’s root filesystem. For specific use cases, such as batch processing or ETL jobs where data needs to be written, containers may require write access to dedicated volumes. In these scenarios, the goal is to limit write access to only the necessary volumes, while keeping the core container filesystem read-only to minimize the potential for unauthorized changes or the introduction of malware to the host system running the containers.

Avoid Running Privileged Containers: Running containers in privileged mode exposes the host’s kernel and hardware resources to potential cyber threats. This practice is generally discouraged in a production environment. Avoid adding the –privileged attribute when starting a container. Privileged containers inherit all of the Linux capabilities assigned to the host’s root user, presenting security risks.

Limit Container Resources: By setting resource limits for containers, you ensure the availability of resources for all containers running on a host. This defensive approach mitigates attacks related to CPU and memory denial-of-service or consumption, reducing the impact of breaches for resource-intensive containers.

Segregate Container Networks: Employ network policies to restrict container network traffic to and from approved destinations. It’s a best practice to deploy containers in a Virtual Private Cloud (VPC) to isolate hosts from external threats. Additionally, using Web Application Firewalls (WAF) can help restrict layer 7 traffic. Custom networks are preferable to relying on the default bridge network, and AWS ECS users should utilize the AWS VPC network mode for enhanced network isolation.

Harnessing the Power of Service Meshes: A service mesh provides a crucial layer of infrastructure to manage and secure communication between containers. By acting as a dedicated, configurable infrastructure layer, a service mesh, such as Istio or Linkerd, can offload security and networking responsibilities from individual applications. It uses “sidecar” proxies, which run alongside each container, to provide features like mutual TLS (mTLS) encryption for all service-to-service communication, fine-grained access control, and network policy enforcement. This not only standardizes security across your containerized environment but also provides valuable observability and network mapping, making it easier to diagnose issues and identify suspicious traffic. The service mesh complements other security best practices by providing a powerful way to enforce zero-trust principles and defend against lateral movement within your container cluster.

Enhance Container Isolation with Namespaces: Linux namespaces provide isolation for running processes, preventing lateral movement if a container is compromised. This is an effective strategy to maintain the security of your containers.

Robust Auditing and Forensics: Collecting ample amounts of data on running containers is essential for troubleshooting and ensuring compliance. Robust log and analytics collection can be a game-changer in detecting malicious threats within your container environment.

Keep Docker Engine and Host Updated: Regularly update the Docker engine and host to mitigate software vulnerabilities. AWS users are recommended to employ Fargate to offload the responsibility of keeping the Docker engine, containers, and the underlying host up to date.

Monitoring the Container: Eyes on the Prize

Security isn’t just about laying down the groundwork; it’s about continuous vigilance. Monitoring is the sentinel that guards your running containers, ready to detect any suspicious activities that might indicate a breach. Whether it’s identifying indicators of compromise (IOCs), zero-day attacks, or ensuring compliance with your organization’s security policies, continuous monitoring is your eyes and ears in the containerized world. The open-source tool Falco can be used to monitor your running workloads, offering real-time detection and immediate response to any unexpected behavior, intrusions, or data theft within your container environment. Check out a great blog about Falco written by two of my coworkers Dustin Whited and Dakota Riley, Threat Detection on EKS – Comparing Falco and GuardDuty For EKS Protection. By marrying best practices with vigilant monitoring, you can confidently navigate the dynamic landscape of container deployments, ensuring the security and stability of your digital infrastructure.

Once your containers are up and running, continuous monitoring is essential. Here’s what to watch for:

Detection of IOCs (Indicator of Compromise): IOCs provide forensic evidence of potential intrusions. They help security professionals and administrators spot intrusion attempts and other malicious activities within your container environment.

Detecting Zero-Day Attacks: User behavior analytics is your best defense against zero-day attacks. By understanding and identifying deviations from normal behavior patterns, you can spot potential zero-day attacks before they escalate.

Compliance Requirements: Use admission controllers to enforce container standards and deployment procedures, ensuring your containers remain compliant with your organization’s security policies.

Detecting Abnormal Behavior in Applications: Analyze events collected from your containers to uncover suspicious behavior patterns that might indicate unauthorized access or other security breaches.

Conclusion: Navigating the Container Security Landscape

In the world of containerization, where agility meets scalability, the journey from development to production is a complex dance of innovation and security. In this comprehensive guide, we’ve delved deep into the realm of container security, from hardening your container images and scanning them at every turn, to securely running containers and maintaining vigilant monitoring practices.

By implementing best practices in container image hardening and continuous scanning, you fortify your containers from the very beginning, minimizing vulnerabilities and ensuring that only trusted and secure containers make their way into your registry. The use of Software Bill of Materials (SBOMs), tools like Syft, and dynamic scanning with Grype guarantees that zero-day exploits remain a distant threat. Further, image signing, through projects like Cosign from Sigstore, adds a digital seal of trust to your container images, safeguarding them from tampering and ensuring their integrity.

The journey doesn’t end with image building; security extends to running containers. Our best practices cover key aspects, from running containers in rootless mode to enforcing resource limits and segregating container networks. Vigilant monitoring is the sentinel that guards your containers in real-time, offering an early warning system for suspicious activities. Through these practices, your containerized applications remain not only innovative and scalable, but also robust and secure, ensuring your organization’s digital ecosystem can flourish without fear of security breaches. In the ever-changing container landscape, striking this balance between security and innovation is key to success. So, as you continue your containerization journey, remember these lessons in container security – your steadfast companions in an ever-evolving digital world.

If you have any questions, or would like to discuss this topic in more detail, feel free to contact us and we would be happy to schedule some time to chat about how Aquia can help you and your organization.