SaltStack Advisory (Vulnerability, Impact, Remediation)

This blog was authored by Jitendra Chauhan, Head R&D, FireCompass

Salk Stack is

A configuration management system. Salt is capable of maintaining remote nodes in defined states. For example, it can ensure that specific packages are installed and that specific services are running.
A distributed remote execution system used to execute commands and query data on remote nodes. Salt can query and execute commands either on individual nodes or by using arbitrary selection criteria.

It was developed in order to bring the best solutions found in the world of remote execution together and make them better, faster, and more malleable. Salt accomplishes this through its ability to handle large loads of information, and not just dozens but hundreds and even thousands of individual servers quickly through a simple and manageable interface.

It is widely used for Infrastructure orchestration similar to Kubernetes.

Vulnerability Description

Researchers recently discovered several vulnerabilities in the “Salt” management framework by the company SaltStack. The open-source Salt project (https://github.com/saltstack/salt) is at the heart of SaltStack’s (the company) product offerings but is also very popular as a configuration tool to manage servers in datacenters and cloud environments. Salt is used to monitor and update the state of servers. Each server runs an agent called a “minion” which connects to a “master”, a Salt installation that collects state reports from minions and publishes update messages that minions can act on. Typically, such messages are updates to the configuration of a selection of servers, but they can also be used to run the same command in parallel over multiple, even all, managed systems asynchronously. The default communication protocol in salt is ZeroMQ. The master exposes two ZeroMQ instances, one called the “request server” where minions can connect to report their status (or the output of commands) and one called the “publish server” where the master publishes messages that the minions can connect and subscribe to. For more information, see https://docs.saltstack.com/en/getstarted/system/communication.html.

Impact

The vulnerabilities described in this advisory allow an attacker who can connect to the “request server” port to bypass all authentication and authorization controls and publish arbitrary control messages, read and write files anywhere on the “master” server filesystem and steal the secret key used to authenticate to the master as root. The impact is full remote command execution as root on both the master and all minions that connect to it. The vulnerabilities, allocated CVE ids CVE-2020-11651 CVE-2020-11652, are of two different classes. One being authentication bypass where functionality was unintentionally exposed to unauthenticated network clients, the other being directory traversal where untrusted input (i.e. parameters in network requests) was not sanitized correctly allowing unconstrained access to the entire filesystem of the master server.

Remediation

SaltStack engineers patched these vulnerabilities in release 3000.2 and users of Salt are encouraged to make sure that their installs are configured to automatically pull updates from the SaltStacks repository server, see https://repo.saltstack.com/ for more information. A patch release for the previous major release version is also available, with version number 2019.2.4. Adding network security controls that restrict access to the salt-master (ports 4505 and 4506 being the defaults) to known minions, or at least block the wider Internet, would also be prudent as the authentication and authorization controls provided by Salt are not currently robust enough to be exposed to hostile networks.

Detection

A scan revealed over 6,000 instances of this service exposed to the public Internet. Getting all of these installs updated may prove a challenge as we expect that not all have been configured to automatically update the salt software packages. To aid in detecting attacks against vulnerable salt masters, the following information is provided. The exploitation of the authentication vulnerabilities will result in the ASCII strings “_prep_auth_info” or “_send_pub” appearing in data sent to the request server port (default 4506). These strings should not appear in normal, benign, traffic. Published messages to minions are called “jobs” and will be saved on the master (default path /var/cache/salt/master/jobs/). These saved jobs can be audited for malicious content or job ids (“jids”) that look out of the ordinary. Lack of suspicious jobs should not be interpreted as an absence of exploitation however.No reliable log evidence has been identified.

Technical Details

Authentication bypass vulnerabilities (CVE-2020-11651)

The ClearFuncs class processes unauthenticated requests and unintentionally exposes the _send_pub() method, which can be used to queue messages directly on the master publish server. Such messages can be used to trigger minions to run arbitrary commands as root. The ClearFuncs class also exposes the method _prep_auth_info(), which returns the “root key” used to authenticate commands from the local root user on the master server. This “root key” can then be used to remotely call administrative commands on the master server. This unintentional exposure provides a remote unauthenticated attacker with root-equivalent access to the salt-master.

Directory traversal vulnerabilities (CVE-2020-11652)

The wheel module contains commands used to read and write files under specific directory paths. The inputs to these functions are concatenated with the target directory and the resulting path is not canonicalized, leading to an escape of the intended path restriction. The get_token() method of the salt.tokens.locals class (which is exposed to unauthenticated requests by the ClearFuncs class) fails to sanitize the token input parameter which is then used as a filename, allowing insertion of “..” path elements and thus reading of files outside of the intended directory. The only restriction is that the file has to be Deserializable by salt.payload.Serial.loads().ExploitationWe expects that any competent hacker will be able to create 100% reliable exploits for these issues in under 24 hours.