Software Build Pipeline = The Finest Hacker Target
Here we would like to share our experience from auditing and securing dozens of software build pipelines in order to provide some understanding where the difficulties in securing those landscapes lie and how to overcome those.
Is this relevant to you? Does your organization do software development in any business area? If the answer to the question is yes, then it applies to your organization as well.
CI/CD Build Pipelines
CI/CD software build pipelines (a.k.a. build systems) are the heart and soul of software production. Compared to a car manufacturer, the build pipelines are the assembly lines where cars get produced. Continuing the analogy, if the assembly line is interrupted (availability is impacted), no cars/software can be produced/delivered. Even worse, if malicious actors manage to manipulate the assembly line (integrity is impacted), the outcome would be malfunctioning unsafe cars or in the case of software delivery, your company may be delivering viruses and malware packaged with your product. Talking about car manufacturers, software build pipelines have already arrived at their enterprises quite some time ago, so they need to take care of both assembly lines.
In the past, successful attacks on such build pipelines have proven to be highly efficient. An example is the CCleaner tool from Avast which was compromised in 2017 and in 2019. The successful attack on the build and delivery process resulted in CCleaner delivering their product packaged with malware. The malicious version of CCleaner was respectively downloaded 2,5 million times.
What makes build pipelines such a great target is that hackers do not have to take care of the "logistics" and distribution of their malware. Vendors with their trusted brands and their delivery channels ensure that the malware is delivered throughout the whole vendor's eco-system, to the vendor's devices, to the vendor's highly protected productive environments.
What is at stake here depends on the importance of the piece of software being built, to whom it is being distributed and where it is being operated. An insecure build environment puts the secure productive run/ops environment at risk as well - while we may have reached highest security standards for our productive environment, a manipulated software delivery to our productive environment may compromise it. Therefore, we need to ensure the same high standards are applied during build.
What are Build Systems
CI/CD Software Delivery Build Pipelines consist of three major components and the utilization, configuration and security of each of those three components is of utmost importance for the security of the delivered service/software. The three components are the Source Code Repository (such as GIT, SVN, Gitlab, Github, Github Enterprise, Perforce, Bitbucket, etc.), the Build Scheduler (Jenkins, TeamCity, Concurse CI, Cruise Control, Bamboo, etc.) and the Binary Repository (JFrog Artifactory, Nexus, etc.). There is a wide range of products that provide the functionality and the complexity may vary widely depending on the scenarios. The following depicts a high level abstraction of a software build pipeline.
The heart of the CI/CD Build Pipeline is the Build Scheduler. Usually, in an automated fashion, the Build Scheduler collects the source code, compiles it on build nodes, packages binary artifacts, performs automated testing and scanning, uploads binary artifacts to binary repositories, triggers automated deployments, reports back the status of a build to the source code management tool, etc. In order to meet these needs, the Build Scheduler requires privileged access to all components of the build landscape. While compromising source code or binary repositories would allow malicious modification of the build outcome, unauthorized access to the Build Scheduler would often yield compromise of the security of all components. Therefore, we need to put the Build Scheduler in the center of our security assessments.
Challenges
Organisational Challenges
Organisationally those CI/CD build pipelines are often operated by the development teams themselves and in some cases are part of the productive environment of the operated service (devops). In any case, these landscapes sit between the development and operations/delivery organizations often with no clearly defined lines of responsibilities. There might by the IT department, which provides infrastructure to development, but operational responsibility above the OS would be usually with the development team.
Vulnerable by Design
Build systems are vulnerable to code execution vulnerabilities by design. In a build system, source code gets compiled to binary often with high privileges during compile time, meaning any developer can potentially hack the build system. But can we trust all developers? Can we design our build system in way that even if a malicious developer pushes code, we have the right controls and build processes in place to mitigate such an attack? This is by far the most difficult technical challenge that a build system needs to overcome and mitigate.
Build System Evolution
Development organizations do not plan for a full-blown, full-fledged all-inclusive build environment from the beginning. The build landscape and its functionality grows and evolves over time with the delivered products. At the beginning the software build may be the part-time responsibility of one developer. Over time this may evolve to a team of ten, with 24/7 support and a release schedule for the build system itself.
All Custom
Software build pipeline teams rely on common off-the-shelf software like Github, Jenkins, Artifactory, etc., but the implemented processes and automation are all custom to the project and the developer's needs. Build teams are the kings of automation/scripting and it is often the case that automation and security are difficult to bring together (keyword: credentials).
Operations and Development
Nowadays we embrace DevOps as a development and operations model, but with this assumption we imply that a developer is automatically a network engineer, a hypersacler (AWS/Azure/GCP) architect, an OS/database administrator, virtualization and storage administrator at the same time... and let's not forget - a security expert. While this may very well work in multi-disciplined and balanced teams, it does not have to be always the case.
Historically priorities of operations and development have been very different and the ask which we have here as security officers is that development (where usually build systems reside) do secure and proper system operations.
Build Systems are Difficult to Audit
Build pipelines implement complex and custom processes. It requires understanding of software development processes which need to be validated on technology level. Little understood and unknown terms like artifacts, branching, branching of feature branches, bugfix branching, merging, merging to main code line, forking, voting, product release management, product maintenance life cycles, etc. scare auditors and pentesters and they often revert back to known common technology ground like network and OS security which leaves a huge blank spot on the security map of build landscapes, namely the development and build processes.
Build Systems Are Fun!
Security of CI/CD pipelines is a challenging and rewarding task. Securing an inherently vulnerable environment is the security expert's challenge. In the build teams of your organization, you will find and work with incredibly skilled technology experts who care about their build landscape and know all nuts and bolts of their complex environment. They do not take criticism lightly, but presenting the right arguments will result in swift actions towards improvement, because they care.
Our Service
We have deep understanding and expertise in the field of Build landscapes. We know the best practices and the operational difficulties of pipeline operations teams, which helps us provide you with viable improvement recommendations.
We perform security assessments for CI/CD build pipelines by organizing workshop sessions with build pipeline operations teams and development process stakeholders in order to get an understanding about the common IT processes mentioned above. We discuss the development and release management processes and map those to the technology layer. Through the workshop sessions we gain valuable knowledge for the next phase where we perform a penetration test on the CI/CD Build pipeline. We document any identified issues and provide mitigation recommendations to improve the security state of the pipeline.
You need support securing your build systems? Contact Us! We help customers achieve secure and reliable builds by ensuring the security of the Continuous Integration/Continuous Delivery Build Pipelines.
Related Articles:
You found the content of this post useful? Register for our Newsletter below to receive email notifications about new posts like this one.