Protecting Your Software Development Pipeline

Software Development

Have you heard of Dependabot? If not, just ask any developer around you, and they’ll probably rave about how it’s changed the tedious task of checking and updating old dependencies in software projects.

Dependabot not only takes care of the checks for you, but also provides suggestions for changes that can be approved with just one click. Although Dependabot is limited to projects hosted on GitHub, it sets a new standard for continuous providers to provide similar capabilities. This automation of “administrative” tasks has become the norm, enabling developers to integrate and deploy their work faster than ever before. Continuous integration and deployment workflows have become the foundation of software engineering, driving the DevOps movement to the forefront of the industry.

But a recent advisory through the security firm Checkmarx shed light on the incident. Malicious actors recently tried to exploit the trust associated with Dependabot by impersonating the tool. By imitating the proposals made by Dependabot (in the form of pull requests), these actors try to trick developers into accepting changes without giving them a second thought.

While Dependabot demonstrates advances in automating software maintenance tasks, this incident also highlights the broader complexities and vulnerabilities inherent in CI/CD pipelines. These pipelines serve as important conduits, connecting the external world of software development tools and platforms to the internal processes of software creation and deployment. Understanding this connection is key to solving the security challenges we face.

CI/CD Pipelines: Connecting the External World with the Internal

Continuous integration (CI) and deployment (CD) workflows are changing the software development process, giving developers the ability to seamlessly integrate their work and deploy it to the production environment. These workflows ensure that code undergoes automated security scans, rigorous testing, and adherence to coding standards, resulting in a more efficient and reliable development process. They become a factor in innovation, enabling teams to focus on building and improving their products with assurance of quality and safety.

To illustrate the concept, consider creating a puzzle. The CI acts as a careful checker, verifying that each new piece of the puzzle fits correctly before moving on. On the other hand, CD takes it a step further by automatically placing each verified piece in the final puzzle, without having to wait for the entire puzzle to be completed. This fast process allows for faster part delivery and ultimately speeds up the overall product development timeline.

However, these CI/CD workflows also connect the external world to the internal development environment, which creates potential risks. For example, consider a scenario where a developer integrates a third-party library into their project through the CI/CD pipeline. If the developer fails to thoroughly check the library or if the pipeline lacks proper security checks, malicious code may unknowingly be included in the project, compromising its integrity. In recent years, there has been an increase in attacks such as typosquatting and confidence confusionwhich aims to exploit the trust in open source software for financial gain.

The rise of automated integrations workflows has changed the economics for attackers by lowering costs and increasing the potential win of an attack. Attackers can target popular central package repositories like PyPi, which hosts thousands of packages and serves millions of downloads every day. The sheer number of operations makes it economically feasible for attackers to try their luck, even with a small chance of success.

Another example is using external APIs within CI/CD pipelines. Developers often need to provide valid credentials for these APIs to enable automated deployment or integration with external services. However, if these credentials are not handled securely or if they are inadvertently exposed in logs or artifacts, they can be exploited by attackers to gain unauthorized access to sensitive resources or manipulate pipeline behavior.

CI/CD breaches often stem from the initial compromise of secrets or developers being the target of specific attacks. However, rather than blaming the developers for these breaches, it is important to recognize that the issue lies in the inherent lack of security in these pipelines. This highlights a bigger problem: CI/CD pipelines are far from secure by default.

The Problem: CI/CD Pipelines Are Far from Secure by Default

Although the idea of ​​implementing secure-by-design workflows is becoming more popular, CI/CD platforms still have a significant way to go. Platforms like GitHub Actions, GitLab CI/CD, and CircleCI, which were originally designed with flexibility in mind, often prioritize ease of use over strong security measures. As a result, there is a lack of default safeguards to prevent potential issues from arising.

A clear example of this is how easy it is for a developer to disclose sensitive information such as secrets. Developers often inject secrets at runtime and rely on the ability to store secrets in the CI provider itself. While this practice is not a problem in itself, it raises at least two security concerns: first, the CI provider hosting the secrets becomes a vault of sensitive information and an attractive target for attackers. As recently as early 2023, CircleCI suffered a breach in its systems which forcing users to turn over “any and all” of their secrets after breaching company systems.

Second, secrets often leak and go undetected through the CI/CD pipelines themselves. For example, if a secret is appended to another string (say, a URL) and then logged, the CI redaction mechanism will not work. Same with encoded secrets. The consequence is that CI logs often expose secrets in plaintext. Similarly, it is not unusual at all to find secrets hard-coded into software artifacts, resulting from a misconfigured continuous integration workflow.

CI/CD providers have already taken steps to improve security, such as GitHub Dependabot security checks. But often, permissive default and complex permission models are still the rule. To protect CI/CD pipelines and prevent code compromise, developers must take additional steps to harden their pipelines against attacks. An important aspect is ensuring the protection of developers’ credentials. Another is to take a proactive security approach with alert factors.

Protecting CI/CD and the Software Supply Chain

To effectively secure CI/CD pipelines, it is important to view them as a high priority, potentially connected to external environments. The key is a mix of best practices:

  • Limit Access and Reduce Privileges: Grant access based on need, not convenience. Broad access to all members of the DevOps team increases the risk of a compromised account giving attackers extensive access to the system. Limit access to critical controls, configuration, or sensitive data.
  • Implement Multi-Factor Authentication (MFA): Important, always use multi-factor authentication (MFA) for logging into the CI/CD platform. MFA adds an important layer of security, making it more difficult for unauthorized users to gain access even if their credentials have been compromised.
  • Use OpenID Connect (OIDC): Employ OIDC for securely connecting workloads to external systems, such as deployment. This protocol provides a robust framework for authentication and cross-domain identity verification, which is critical in a distributed and interconnected environment.
  • Use Pre-Checked Software Dependencies: It is important to provide developers with safe, pre-tested software dependencies. This practice protects the integrity of the supply chain and frees developers from verifying every code in the package. This ensures the integrity of the supply chain, relieving developers from the burden of individually verifying the code of each package.
  • Secure Runtime Secrets: Secure storage of secrets such as API keys and CI/CD platform credentials requires strong security measures, such as enforced MFA and role-based access controls (RBAC). However, it is useless. Secrets are inherentand additional layers of security, such as strict credential hygiene and careful monitoring of internal and external threats, are necessary for comprehensive protection.
  • Implement Advanced Defense Systems: Include an alert system in your security framework. While honeypots are effective but challenging to scale, honeytokens offers a scalable, easy-to-implement alternative. These tokens, which require minimal setup, can greatly improve security for companies of all sizes across a variety of platforms such as SCM systems, CI/CD pipelines, and software artifact registries.
  • Use Enterprise-Scale Solutions: Solutions like GitGuardian Platform offers a single pane of glass to monitor incidents such as leaked secrets, Infrastructure as Code misconfigurations and honeytoken triggers, allowing organizations to identify, remediate and prevent CI/CD incidents on a large scale. This will minimize the impact of potential data breaches.

By integrating these strategies, organizations can comprehensively protect their CI/CD pipelines and software supply chain, adapt to evolving threats and maintain strong security protocols. Start securing your pipelines today with GitGuardian Platform.

Did you find this article interesting? Follow us on Twitter and LinkedIn to read more exclusive content we post.

Leave a comment