We live in a digitalized world that runs on code. Software now permeates virtually every aspect of our lives – from the things we use day-to-day, to the critical infrastructure of our society. At the same time, hackers are becoming increasingly adept in the art of spreading malware. Infamously, this occurred in the SolarWinds attack, where over 18,000 customers downloaded an infected version of the company’s Orion software. They trusted it, because it was digitally signed and authenticated using a code signing machine identity.
This mode of attack – where hackers implant malware into software as its being built so that it makes it into legitimate software updates – is not new. However, the sophistication and reach of this type of attack has proven to be very successful, and, is likely to encourage others to try. It is paramount that companies prevent this, and while no method will provide a silver bullet, tightening your code signing processes is an important step.
Code signing was a problem; it should have been the solution
So, how can you prevent malware from being inserted into the software you develop? In the SolarWinds attack, its customers trusted that when SolarWinds built their software, it would be free of malicious code. They had implicit confidence in the signed software they received, as there was no evidence it had been tampered with.
While there is no single fix that would have prevented this attack, stronger code signing procedures would have helped. Code signing certificates are machine identities and enable developers to prove that a piece of software is authentic. By digitally signing apps, software or embedded firmware with a private key, proof is provided to end-users that the code is from a trusted and legitimate source. However, if code signing machine identities are poorly protected and fall into the hands of attackers, it can have dire consequences. The machine identity can be used as a weapon, enabling hackers to subvert code signing processes, while appearing to be trustworthy.
A decade or two ago, companies would protect their software build environments by completely sandboxing the process, with no outside network access. Any source code brought in was scanned across all competing AV engines to ensure no known vulnerabilities were present. While this sacrificed some convenience and speed, it enabled the best product possible to be developed. However, this is no longer feasible, due to the speed of developers and digital transformation business timelines. So, if we can’t have a completely offline system building software, how can we better protect our ‘always online’ continuous build and delivery pipelines?
One way to better protect the build pipeline is by locking it down and only allowing software packages specifically approved for installation in the delivery pipeline. There is no reason a very static list of approved binaries with valid signatures couldn’t be checked before being allowed to execute. The second is to tighten code signing procedures to enforce the strict amount of software packages in the workstation, and ensure only trusted code is run. Then only software signed by the vendor would be part of the build system and any malware installed (as in the SolarWinds attack) would not be able to run.
There is no single fix that can prevent future attacks
As with any breach, there is no magic formula that could have prevented the SolarWinds attack and subsequent modifications of source code. The layers-of-security approach will always apply when discussing overall system security. From the network layer to individuals that have access to source code, to vetting the source code to delivering a product that enterprises can prove came from their trusted build system. All of this must be audited, scrutinized, and maintained with the risks in mind.
The pace of DevOps pipelines have already moved the problem beyond human speed and control. Enterprises can no longer rely on manual processes and checklists to ensure that they are running the right software and that no unauthorized changes have been made. As such, there are a few things that security professionals can do to ensure their code signing procedures are tight and will not affect the overall system security.
1. Centralized key protection
By centralizing the storage of an enterprises’ private keys, enterprises won’t have to worry about unauthorized copies or locations of signing keys within their enterprise.
2. Approval of who can sign where
organizations need an approval workflow so they can configure which users can sign code, and on which systems they are approved to request signing operations from on the centrally stored keys.
3. Approval at time of signing (approve what can be signed)
Once you have approved a user and a system to do signing operations, what are they actually signing? Enterprises must also define an approval process that can allow an administrator to inspect the signing request before approving.
4. Ability to control signing, even in a highly automated CI/CD pipeline
All approval and signing operations should be assigned to systems accounts and inspected and scheduled. Automation is required to enable developers to deliver on time while ensuring they have the rules in place to prove they are in control of the signing process.
5. Auditing and reporting
Being able to audit and prove that you are following policy and best practice is critical when using code signing in your software delivery pipeline. For instance, enterprises need to keep a record of code signing operation activity as well as approval operations, so that risk of attacks from opportunistic cyber actors can be mitigated.
Every software organization and their security professionals need to evaluate how they secure machine identities. This goes beyond improving security efforts focused on reducing vulnerabilities in the code. The entire build process needs to be secured with machine identities. Each of these identities needs to be managed; this includes visibility of all signing certificates in use, intelligence about how they are being used and automation to manage their full lifecycle. Without this, malicious actors will continue to successfully target the code-signing process.
Tony Hadfield, Director Solution Architects, Venafi
