Jason GarbisDecember 16, 2020
Software Supply Chain Security and the SolarWinds Compromise
Our approach to securing our product, its supply chain, and learnings to ensure resiliency against similar future attacks.
Like many of you, we’ve been closely following the recent news about a compromise of the SolarWinds network monitoring software by a malicious state actor. This actor embedded malicious software into an official signed SolarWinds release and then used that software to attack enterprise networks onto which it was deployed.
As a security software vendor with our Appgate SDP product deployed across our customers’ enterprise networks, we recognize the important role we play and take this responsibility seriously. As such, we want to proactively communicate on this topic, and also take the time to explore in more depth our approach to ensuring the security of our product and its supply chain.
First and foremost, Appgate and the Appgate SDP product were not at risk or impacted in any way by the SolarWinds compromise.
- Appgate does not and has not used the compromised SolarWinds product on any of our networks
- As a result, neither our company network nor the Appgate SDP product infrastructure was exposed to this threat actor or this malicious code in any way
Taking a step back, we are of course concerned about this type of attack as an enterprise, but also in our special role as a security vendor. Our customers trust our software to control their users’ access to their organization’s most valuable and sensitive assets, deploying it throughout their enterprise. In recognition of this, over the years we have put in place a robust set of security controls, automated tests, and checks and balances to reduce the likelihood of any type of compromise of our software.
Current information from the SolarWinds 8-K  indicates that the malicious actor was able to compromise some aspect of the SolarWinds Orion build system, with the end result being the production and distribution of a valid, signed DLL that contained the malicious code.
As such, let’s examine how we approach these aspects of our software development systems and processes.
First of all, from a holistic perspective, all of our internal source code and build systems are protected by the internal deployment of our own product, Appgate SDP. All developers must be running the Appgate SDP client on their devices and be properly authenticated with multiple factors before they can access any build or source code system. We enforce MFA, conditional access, and device posture checks before developers can access source code
Source Code Control System
Our product is built by compiling source code from our private, internal source code repository. The general risk here is that a user or malware could place malicious code into the repository, where it’d be compiled and included in an official software release.
We mitigate this risk by enforcing several controls. Developers can individually check-in source code changes, but all code changes are made only in a private branch. All checked-in code is automatically scanned for security issues using a static code analysis tool. And, no code is automatically pulled into the master branch – it requires a pull request to be made and approved. In order for this to occur, all code must first pass a manual peer review for validation before it’s merged into the master branch. This ensures that any significant or anomalous changes will be caught and examined. This is an important step, ensuring that there is a human review of code before it’s merged and included in a build.
The product is compiled and packaged by an automated build system, which executes the many steps required to compile, link, and package the software deliverables across the many different platforms we support. The general risk of compromise here is that the build system could pull in or inject malicious code that exists outside the carefully controlled source code systems. And, because it’s created by the build system, it would be signed and trusted, which is apparently what happened with SolarWinds.
In our environment, once code is merged, the build system is automatically run on a nightly basis, with thousands of end-to-end tests. All these tests have to pass successfully in order for a build to complete. Note that these are not just functional tests, they also include integrity checks for changes in expected (normal) memory consumption, network activity, bandwidth consumption, so we have built-in mechanisms for detecting anomalies.
We also protect the build system itself from compromise. Our build agents are virtual and ephemeral – they are only launched as needed, and disappear once a build completes. They’re launched as a clone from a master build snapshot image, and changes to the master build image are carefully controlled.
Release or Distribution system
Once built, the software components are pushed out into various distribution platforms, including systems directly controlled by Appgate, as well as third-party systems such as cloud marketplaces and mobile app stores. The general risks here are that an attacker could have gained access to the release repository and be able to insert malicious code or binaries, replacing the official version of the software with a compromised version that includes malicious code.
Our process is that once a build is selected for release, it’s signed. We publish the hashes and signatures for product releases on our website so that customers can validate them. Not only is each full product image signed, but internal components are individually signed to enable cross-component validation. This ensures that even if a deployed component is modified (definitely a non-zero risk for the client software that’s installed onto user devices) – the other components can recognize this, and respond accordingly. Of course, our signing certificates are themselves carefully secured internally.
In conclusion, we’re very confident in the security and integrity of our Appgate SDP software releases, and our customers should retain a high degree of confidence as well. We are studying the technical information about the SolarWinds supply chain compromise, and learning from it, to make sure that we’re resilient against such an attack, and to see where we could improve even further. We take security seriously – the security of our enterprise as well as yours.
Recommended Actions for Appgate SDP Customers
First and foremost, if you have SolarWinds Orion deployed, follow the steps outlined by that vendor in terms of mitigating the issue , and also look at the Microsoft blog post on this topic .
There is no immediate action required for your Appgate SDP deployment since our product was not impacted by this attack.
But this is a good opportunity to look at how your organization is enforcing access controls around high-value assets:
- How are you controlling admin access to your Authentication server? We recommend using Appgate SDP to enforce MFA for all such access, which could prevent non-human actors from accessing it
- Are you enforcing device onboarding with MFA? This could reduce the likelihood that a forged SAML assertion could be used to access a protected system
- Consider deploying the Appgate SDP client onto your directory servers, which will allow you to carefully control inbound and outbound access to them. This reduces the attack surface and can help limit the ability of malware to perform reconnaissance or attacks
- Likewise, consider deploying the Appgate SDP client onto any servers that use service accounts to access high-value servers
- Check the network entitlements granted to your users, and see if they can be reduced to a more minimal set without impacting user productivity