Exploring Syscall Evasion – Linux Shell Built-ins

Originally published by Sysdig.

Written by Jason Andress.

This is the first article in a series focusing on syscall evasion as a means to work around detection by security tools and what we can do to combat such efforts. We’ll be starting out the series discussing how this applies to Linux operating systems, but this is a technique that applies to Windows as well, and we’ll touch on some of this later on in the series.

In this particular installment, we’ll be discussing syscall evasion with bash shell builtins. If you read that and thought “what evasion with bash what now?”, that’s ok. We’ll walk through it from the beginning.

What is a Syscall?

System calls, commonly referred to as syscalls, are the interface between user-space applications and the kernel, which, in turn, talks to the rest of our resources, including files, networks, and hardware. Basically, we can consider syscalls to be the gatekeepers of the kernel when we’re looking at things from a security perspective.

Many security tools that watch for malicious activity taking place are monitoring syscalls going by. This seems like a reasonable approach, right? If syscalls are the gatekeepers of the kernel and we watch the syscalls with our security tool, we should be able to see all of the activity taking place on the system. We’ll just watch for the bad guys doing bad things with bad syscalls and then we’ll catch them, right? Sadly, no.

There is a dizzying array of syscalls, some of which have overlapping sets of functionality. For instance, if we want to open a file, there is a syscall called open() and we can look at the documentation for it here. So if we have a security tool that can watch syscalls going by, we can just watch for the open() syscall and we should be all good for monitoring applications trying to open files, right? Well, sort of.

If we look at the synopsis in the open() documentation:

As it turns out, there are several syscalls that we could be using to open our file: open(), creat(), openat(), and openat2(), each of which have a somewhat different set of behaviors. For example, the main difference between open() and openat() is that the path for the file being opened by openat() is considered to be relative to the current working directory, unless an absolute path is specified. Depending on the operating system being used, the application in question, and what it is doing relative to the file, we may see different variations of the open syscalls taking place. If we’re only watching open(), we may not see the activity that we’re looking for at all.

Generally, security tools watch for the execve() syscall, which is one syscall indicating process execution taking place (there are others of a similar nature such as execveat(), clone(), and fork()). This is a safer thing to watch from a resource perspective, as it doesn’t take place as often as some of the other syscalls. This is also where most of the interesting activity is taking place. Many of the EDR-like tools watch this syscall specifically. As we’ll see here shortly, this is not always the best approach.

There aren’t any bad syscalls we can watch, they’re all just tools. Syscalls don’t hack systems, people with syscalls hack systems. There are many syscalls to watch and a lot of different ways they can be used. On Linux, one of the common methods of interfacing with the OS is through system shells, such as bash and zsh.

NOTE: If you want to see a complete* list of syscalls, take a gander at the documentation on syscall man page here. This list also shows where syscalls are specific to certain architectures or have been deprecated. *for certain values of complete

Examining Syscalls

Now that we have some ideas of what syscalls are, let’s take a quick look at some of them in action. On Linux, one of the primary tools for examining syscalls as they happen is strace. There are a few other tools we can use for this (including the open source version of Sysdig), which we will discuss at greater length in future articles. The strace utility allows us to snoop on syscalls as they’re taking place, which is exactly what we want when we’re trying to get a better view of what exactly is happening when a command executes. Let’s try this out:

1 – We’re going to make a new directory to perform our test in, then use touch to make a file in it. This will help minimize what we get back from strace, but it will still return quite a bit.

5 – Then, we’ll run strace and ask it to execute the ls command. Bear in mind that this is the output of a very small and strictly bounded test where we aren’t doing much. With a more complex set of commands, we would see many, many more syscalls.

7 – Here, we can see the execve() syscall and the ls command being executed. This particular syscall is often the one monitored for by various detection tools as it indicates program execution. Note that there are a lot of other syscalls happening in our example, but only one execve().

8 – From here on down, we can see a variety of syscalls taking place in order to support the ls command being executed. We won’t dig too deeply into the output here, but we can see various libraries being used, address space being mapped, bytes being read and written, etc.

$ mkdir test
$ cd test/
$ touch testfile

$ strace ls

execve("/usr/bin/ls", ["ls"], 0x7ffcb7920d30 /* 54 vars */) = 0
brk(NULL)                               = 0x5650f69b7000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff2e5ae540) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f07f9f63000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=61191, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 61191, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f07f9f54000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=166280, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 177672, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f07f9f28000
mprotect(0x7f07f9f2e000, 139264, PROT_NONE) = 0
mmap(0x7f07f9f2e000, 106496, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f07f9f2e000
mmap(0x7f07f9f48000, 28672, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) = 0x7f07f9f48000
mmap(0x7f07f9f50000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x27000) = 0x7f07f9f50000
mmap(0x7f07f9f52000, 5640, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f07f9f52000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\237\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 48, 848) = 48
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0 =\340\2563\265?\356\25x\261\27\313A#\350"..., 68, 896) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2216304, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2260560, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f07f9c00000
mmap(0x7f07f9c28000, 1658880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f07f9c28000
mmap(0x7f07f9dbd000, 360448, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7f07f9dbd000
mmap(0x7f07f9e15000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x214000) = 0x7f07f9e15000
mmap(0x7f07f9e1b000, 52816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f07f9e1b000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=613064, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 615184, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f07f9e91000
mmap(0x7f07f9e93000, 438272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f07f9e93000
mmap(0x7f07f9efe000, 163840, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6d000) = 0x7f07f9efe000
mmap(0x7f07f9f26000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x94000) = 0x7f07f9f26000
close(3)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f07f9e8e000
arch_prctl(ARCH_SET_FS, 0x7f07f9e8e800) = 0
set_tid_address(0x7f07f9e8ead0)         = 877628
set_robust_list(0x7f07f9e8eae0, 24)     = 0
rseq(0x7f07f9e8f1a0, 0x20, 0, 0x53053053) = 0
mprotect(0x7f07f9e15000, 16384, PROT_READ) = 0
mprotect(0x7f07f9f26000, 4096, PROT_READ) = 0
mprotect(0x7f07f9f50000, 4096, PROT_READ) = 0
mprotect(0x5650f62f3000, 4096, PROT_READ) = 0
mprotect(0x7f07f9f9d000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f07f9f54000, 61191)           = 0
statfs("/sys/fs/selinux", 0x7fff2e5ae580) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7fff2e5ae580)      = -1 ENOENT (No such file or directory)
getrandom("\x9a\x10\x6f\x3b\x21\xc0\xe9\x56", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x5650f69b7000
brk(0x5650f69d8000)                     = 0x5650f69d8000
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 421
read(3, "", 1024)                       = 0
close(3)                                = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=5712208, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 5712208, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f07f9600000
close(3)                                = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=48, ws_col=143, ws_xpixel=0, ws_ypixel=0}) = 0
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(3, "", {st_mode=S_IFDIR|0775, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(3, 0x5650f69bd9f0 /* 3 entries */, 32768) = 80
getdents64(3, 0x5650f69bd9f0 /* 0 entries */, 32768) = 0
close(3)                                = 0
newfstatat(1, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x2), ...}, AT_EMPTY_PATH) = 0
write(1, "testfile\n", 9testfile
)               = 9
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Strace has a considerably larger set of capabilities than what we touched on here. A good starting place for digging into it further can be found in the documentation.

Now that we’ve covered syscalls, let’s talk a bit about system shells.

Linux System Shell 101

System shells are interfaces that allow us to interact with an operating system. While shells can be graphical in nature, most of the time when we hear the word shell, it will be in reference to a command-line shell accessed through a terminal application. The shell interprets commands from the user and passes them onto the kernel via, you guessed it, syscalls. We can use the shell to interact with the resources we discussed earlier as being available via syscalls, such as networks, files, and hardware components.

On any given Linux installation, there will be one or more shells installed. On a typical server or desktop installation, we’ll likely find a small handful of them installed by default. On a purposefully stripped-down distribution, such as those used for containers, there may only be one.

On most distributions, we can easily ask about the shell environment that we are operating in:

1 – Reading /etc/shells should get us a list of which shells are installed on the system. Here we can see sh, bash, rbash, dash, and zsh as available shells.

NOTE: The contents of /etc/shells isn’t, in all cases, the complete list of shells on the system. It’s a list of which ones can be used as login shells. These are generally the same list, but YMMV.

15 – We can easily check which shell we’re currently using by executing echo $0. In this case, we’re running the bash shell.

19 – Switching to another shell is simple enough. We can see that zsh is present in our list of shells and we can change to it by simply issuing zsh from our current shell.

21 – Once in zsh, we’ll ask which shell we are in again, and we can see it is now zsh.

25 – We’ll then exit zsh, which will land us back in our previous shell. If we check which shell we’re in again, we can see it is bash once again.

$ cat /etc/shells

# /etc/shells: valid login shells
/bin/sh
/bin/bash
/usr/bin/bash
/bin/rbash
/usr/bin/rbash
/usr/bin/sh
/bin/dash
/usr/bin/dash
/bin/zsh
/usr/bin/zsh

 $ echo $0

 /bin/bash

 $ zsh

 % echo $0

 zsh

 % exit

 $ echo $0

 /bin/bash

As we walk through the rest of our discussion, we’ll be focusing on the bash shell. The various shells have somewhat differing functionality, but are usually similar, at least in broad strokes. Bash stands for “Bourne Again SHell” as it was designed as a replacement for the original Bourne shell. We’ll often find the Bourne shell on many systems also. It’s in the list we looked at above at /bin/sh.

All this is great, you might say, but we were promised syscall evasion. Hold tight, we have one more background bit to cover, then we’ll talk about those parts.

Shell Builtins vs. External Binaries

When we execute a command in a shell, it can fall into one of several categories:

• It can be a program binary external to our shell (we’ll call it a binary for short).
• It can be an alias, which is a sort of macro pointing to another command or commands.
• It can be a function, which is a user defined script or sequence of commands.
• It can be a keyword, a common example of which would be something like ‘if’ which we might use when writing a script.
• It can be a shell builtin, which is, as we might expect, a command built into the shell itself. We’ll focus primarily on binaries and builtins here.

Identifying External Binaries

Let’s take another look at the ls command:

1 – We can use the which command to see the location of the command being executed when we run la. We’ll use the -a switch so it will return all of the results. We can see there are a couple results, but this doesn’t tell us what ls is, just where it is.

6 – To get a better idea of what is on the other end of ls when we run it, we can use the type command. Again, we’ll add the -a switch to get all the results. Here, we can see that there is one alias and two files in the filesystem behind the ls command.

7 – First, the alias will be evaluated. This particular alias adds the switch to colorize the output of ls when we execute it.

8 – After this, there are two ls binaries in the filesystem. Which of these is executed depends on the order of our path.

11 – If we take a look at the path, we can see that /usr/local/bin appears in the path before /bin, so /usr/local/bin/ls is the command being executed by the ls alias when we type ls into our shell. The final piece of information we need to know here is what type of command this particular ls is.

15 – We can use the file command to dig into ls. File tells us that this particular version of ls is a 64bit ELF binary. Circling all the way back around to our discussion on types of commands, this makes ls an external binary.

21 – Incidentally, if we look at the other ls located in /bin, we will find that it is an identical file with an identical hash. What is this sorcery? If we use file to interrogate /bin, we’ll see that it’s a symlink to bin. We’re seeing the ls binary twice, but there is really only one copy of the file.

$ which -a ls
/usr/bin/ls
/bin/ls


$ type -a ls
ls is aliased to `ls --color=auto'
ls is /usr/bin/ls
ls is /bin/ls

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/sna
p/bin:/snap/bin

$ file /usr/bin/ls
/usr/bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically
 linked, interpreter /lib64/ld-linux-x86-64.so.2,
 BuildID[sha1]=897f49cafa98c11d63e619e7e40352f855249c13, for GNU/Linux 3.2.0,
 stripped

$ file /bin
/bin: symbolic link to usr/bin

Identifying Shell Builtins

We briefly mentioned that a shell builtin is built into the binary of the shell itself. The builtins available for any given shell can vary quite widely. Let’s take a quick look at what we have available in bash:

1 – The compgen command is one of those esoteric command line kung-fu bits. In this case, we’ll use it with the -b switch, which effectively says “show me all the shell builtins.” We’ll also do a little formatting to show the output in columns and then show a count of the results.

2 – We can see some common commands in the output, like cd, echo, and pwd (also note the compgen command we just ran). When we execute these, we don’t reach out to any other binaries inside the filesystem, we do it all inside of the bash shell already running.

17 – We should also note that just because one of these commands is in the builtins list for our shell, it can also be elsewhere. If we use the type command again to inquire about echo, which is in our builtins list, type will tell us it is a shell builtin, but we will also see a binary sitting in the filesystem. If we run echo from bash, we will get the builtin, but if we run it from another shell without a builtin echo, we may get the one from the filesystem instead.

$ compgen -b | pr -5 -t; echo "Count: $(compgen -b | wc -l)"
.          compopt        fc          popd        suspend
:          continue        fg          printf                test
[          declare        getopts      pushd    times
alias          dirs        hash      pwd        trap
bg          disown        help                  read        true
bind          echo        history      readarray    type
break          enable        jobs                  readonly    typeset
builtin          eval        kill                  return                ulimit
caller          exec        let          set        umask
cd          exit         local                  shift        unalias
command export        logout      shopt        unset
compgen  false        mapfile      source    wait
complete
Count: 61

$ type -a echo
echo is a shell builtin
echo is /usr/bin/echo
echo is /bin/echo

It’s also important to note that this set of builtins are specific to the bash shell, and other shells may be very different. Let’s take a quick look at the builtins for zsh.

1 - Zsh doesn’t have compgen, so we’ll need to get the data we want in a different manner. We’ll access the builtins associative array, which contains all the builtin commands of zsh, then do some formatting to make the results a bit more sane and put the output into columns, lastly getting a count of the results.

% print -roC5 -- ${(k)builtins}; echo "Count: ${(k)#builtins}"

-                           compquote     fg                  pushln         umask
.                           compset         float              pwd             unalias
:                           comptags       functions      r                   unfunction
[                           comptry          getln             read             unhash
alias                    compvalues    getopts         readonly      unlimit
autoload              continue          hash             rehash        unset
bg                        declare           history           return          unsetopt
bindkey                dirs                 integer          sched          vared
break                   disable            jobs              set               wait
builtin                   disown            kill                setopt           whence
bye                       echo               let                shift              where
cd                         echotc            limit             source          which
chdir                     echoti             local            suspend       zcompile
command            emulate          log               test               zf_ln
compadd              enable            logout          times           zformat
comparguments  eval                noglob          trap              zle
compcall              exec               popd            true               zmodload
compctl                exit                 print             ttyctl             zparseopts
compdescribe     export             printf            type              zregexparse
compfiles             false               private         typeset         zstat
compgroups         fc                   pushd          ulimit            zstyle
Count: 105

We can see here that there are over 40 more builtins in zsh than there are in bash. Many of them are the same as what we see in bash, but the availability of builtin commands is something to validate when working with different shells. We’ll continue working with bash as it’s one of the more commonly used shells that we might encounter, but this is certainly worth bearing in mind.

Now that we know a bit about the shell and shell builtins, let’s look at how we can use these for syscall evasion.

Syscall Evasion Techniques Using Bash Builtins

As we mentioned earlier, many security tools that monitor syscalls monitor for process execution via the execve() syscall. From a certain tool design perspective, this is a great solution as it limits the number of syscalls we need to watch and should catch most of the interesting things going on. For example, let’s use cat to read out the contents of a file and watch what happens with strace:

1 – First, we’ll echo a bit of data into the test file we used earlier so we have something to play with. Then, we’ll cat the file and we can see the output with the file contents.

5 – Now let’s do this again, but this time we’ll watch what happens with strace. We’ll spin up a new bash shell which we will monitor with strace. This time, we’ll also add the -f switch so strace will monitor subprocesses as well. This will result in a bit of extra noise in the output, but we need this in order to get a better view of what is happening as we’re operating in a new shell. Note that strace is now specifying the pid (process id) at the beginning of each syscall as we’re watching multiple processes.

6 – Here we have the execve() syscall taking place for the bash shell we just started. We can see the different subprocesses taking place as bash starts up.

34 – Now we’re dropped back to a prompt, but still operating inside the shell being monitored with strace. Let’s cat the file again and watch the output.

37 – We can see the syscall for our cat here, along with the results of the command. This is all great, right? We were able to monitor the command with strace and see its execution. We saw the exact command we ran and the output of the command.

$ echo supersecretdata >> testfile
$ cat testfile 
supersecretdata

$ strace -f -e trace=execve bash
execve("/usr/bin/bash", ["bash"], 0x7ffee6b6c710 /* 54 vars */) = 0
strace: Process 884939 attached
[pid 884939] execve("/usr/bin/lesspipe", ["lesspipe"], 0x55aa1d8a3090 /* 54 vars */) = 0
strace: Process 884940 attached
[pid 884940] execve("/usr/bin/basename", ["basename", "/usr/bin/lesspipe"],
0x55983907af68 /* 54 vars */) = 0
[pid 884940] +++ exited with 0 +++
[pid 884939] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
 si_pid=884940, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 884941 attached
strace: Process 884942 attached
[pid 884942] execve("/usr/bin/dirname", ["dirname", "/usr/bin/lesspipe"],
0x559839087108 /* 54 vars */) = 0
[pid 884942] +++ exited with 0 +++
[pid 884941] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=884942, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 884941] +++ exited with 0 +++
[pid 884939] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=884941, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 884939] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=884939,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 884943 attached
[pid 884943] execve("/usr/bin/dircolors", ["dircolors", "-b"], 0x55aa1d8a2d10 /* 54 vars
*/) = 0
[pid 884943] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=884943,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
$ cat testfile

strace: Process 884946 attached
[pid 884946] execve("/usr/bin/cat", ["cat", "testfile"], 0x55aa1d8a9520 /* 54 vars */) = 0
supersecretdata
[pid 884946] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=884946,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---

$ exit
exit
+++ exited with 0 +++

Let’s try being sneakier about things by using a little shell scripting of a bash builtin and see what the results are:

1 – We’ll start a new bash shell and watch it with strace, the same as we did previously.

3 – Here’s the execve() syscall for the bash shell, just as we expected.

31 – And we’re dropped back to the prompt. This time, instead of using cat, we’ll use two of the bash builtins to frankenstein a command together and replicate what cat does:

while IFS= read -r line; do echo "$line"; done < testfile

This uses the bash builtins read and echo to process our file line by line. We use read to fetch each line from testfile into the variable line, with the -r switch to ensure any backslashes are read literally. The IFS= (internal field separator) preserves leading and trailing whitespaces. Then, echo outputs each line exactly as it’s read.

35 – Zounds! We’re dropped back to the prompt with no output from strace at all.

$ strace -f -e trace=execve bash

execve("/usr/bin/bash", ["bash"], 0x7fff866fefc0 /* 54 vars */) = 0
strace: Process 884993 attached
[pid 884993] execve("/usr/bin/lesspipe", ["lesspipe"], 0x5620a56bf090 /* 54 vars */) = 0
strace: Process 884994 attached
[pid 884994] execve("/usr/bin/basename", ["basename", "/usr/bin/lesspipe"],
0x558950f6cf68 /* 54 vars */) = 0
[pid 884994] +++ exited with 0 +++
[pid 884993] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
si_pid=884994, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 884995 attached
strace: Process 884996 attached
[pid 884996] execve("/usr/bin/dirname", ["dirname", "/usr/bin/lesspipe"],
0x558950f79108 /* 54 vars */) = 0
[pid 884996] +++ exited with 0 +++
[pid 884995] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
si_pid=884996, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 884995] +++ exited with 0 +++
[pid 884993] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
si_pid=884995, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 884993] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=884993,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 884997 attached
[pid 884997] execve("/usr/bin/dircolors", ["dircolors", "-b"], 0x5620a56bed10 /* 54 vars
*/) = 0
[pid 884997] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=884997,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
$ while IFS= read -r line; do echo "$line"; done < testfile

Supersecretdata

$

If we can’t see the activity while monitoring for process execution, how do we find it?

Looking for Syscalls in All the Right Places

The problem we were encountering with not seeing the sneaky bash builtin activity was largely due to looking in the wrong place. We couldn’t see anything happening with execve() because there was nothing to see. In this particular case, we know a file is being opened, so let’s try one of the open syscalls. In this particular case, we’re going to cheat and jump directly to looking at openat(), but it could very well be any of the open syscalls we discussed earlier.

1 – We’ll start up the strace-monitored bash shell again. This time, our filter is based on openat() instead of execve().

2 – Note that we see a pretty different view of what is taking place when bash starts up this time since we’re watching for files being opened.

72 – Back at the prompt, we’ll run our sneaky bit of bash script to read the file.

73 – Et voilà, we see the openat() syscall for our file being opened and the resulting output.

$ strace -f -e trace=openat bash
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libtinfo.so.6", O_RDONLY|O_CLOEXEC) =
3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/dev/tty", O_RDWR|O_NONBLOCK) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache",
O_RDONLY) = 3
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/terminfo/x/xterm-256color", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/bash.bashrc", O_RDONLY) = 3
openat(AT_FDCWD, "/home/user/.bashrc", O_RDONLY) = 3
openat(AT_FDCWD, "/home/user/.bash_history", O_RDONLY) = 3
strace: Process 984240 attached
[pid 984240] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 984240] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6",
O_RDONLY|O_CLOEXEC) = 3
[pid 984240] openat(AT_FDCWD, "/usr/bin/lesspipe", O_RDONLY) = 3
strace: Process 984241 attached
[pid 984241] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 984241] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6",
O_RDONLY|O_CLOEXEC) = 3
[pid 984241] openat(AT_FDCWD, "/usr/lib/locale/locale-archive",
O_RDONLY|O_CLOEXEC) = 3
[pid 984241] +++ exited with 0 +++
[pid 984240] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
si_pid=984241, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 984242 attached
strace: Process 984243 attached
[pid 984243] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 984243] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6",
O_RDONLY|O_CLOEXEC) = 3
[pid 984243] openat(AT_FDCWD, "/usr/lib/locale/locale-archive",
O_RDONLY|O_CLOEXEC) = 3
[pid 984243] +++ exited with 0 +++
[pid 984242] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
si_pid=984243, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 984242] +++ exited with 0 +++
[pid 984240] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
si_pid=984242, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 984240] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=984240,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
strace: Process 984244 attached
[pid 984244] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 984244] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6",
O_RDONLY|O_CLOEXEC) = 3
[pid 984244] openat(AT_FDCWD, "/usr/lib/locale/locale-archive",
O_RDONLY|O_CLOEXEC) = 3
[pid 984244] openat(AT_FDCWD,
"/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache", O_RDONLY) = 3
[pid 984244] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=984244,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
openat(AT_FDCWD, "/usr/share/bash-completion/bash_completion", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/init.d/",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
openat(AT_FDCWD, "/etc/bash_completion.d/",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
openat(AT_FDCWD, "/etc/bash_completion.d/apport_completion", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/bash_completion.d/git-prompt", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/lib/git-core/git-sh-prompt", O_RDONLY) = 3
openat(AT_FDCWD, "/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
openat(AT_FDCWD, "/home/user/.bash_history", O_RDONLY) = 3
openat(AT_FDCWD, "/home/user/.bash_history", O_RDONLY) = 3
openat(AT_FDCWD, "/home/user/.inputrc", O_RDONLY) = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD, "/etc/inputrc", O_RDONLY) = 3

$ while IFS= read -r line; do echo "$line"; done < testfile
openat(AT_FDCWD, "testfile", O_RDONLY)  = 3
supersecretdata

We can catch the activity from the shell builtins, in most cases, but it’s a matter of looking in the right places for the activity we want. It might be tempting to think we could just watch all the syscalls all the time, but doing so quickly becomes untenable. Our example above produces somewhere around 50 lines of strace output when we are filtering just for openat(). If we take the filtering off entirely and watch for all syscalls, it balloons out to 1,200 lines of output.

This is being done inside a single shell with not much else going on. If we tried to do this across a running system, we would see exponentially more in the brief period of time before it melted down into a puddle of flaming goo from the load. In other words, there really isn’t any reasonable way to watch all the syscall activity all the time. The best we can do is to be intentional with what we choose to monitor.

Conclusion

This exploration into syscall evasion using bash shell builtins illuminates just a fraction of the creative and subtle ways in which system interactions can be manipulated to bypass security measures. Security tools that solely focus on process execution for monitoring are inherently limited in scope and a more nuanced and comprehensive approach to monitoring system activity is needed to provide a better level of security.

The simple example we put together for replicating the functionality of cat dodged this entirely and allowed us to read the data from our file while flying completely under the radar of tools that were only looking for process execution. Unfortunately, this is the tip of the iceberg.

Using the bash builtins in a similar fashion to what we did above, there are a number of similar ways we can combine them to replicate functionality of other tools and attacks. A very brief amount of Googling will turn up a well-known method for assembling a reverse shell using the bash builtins. Furthermore, we have all the various shells and all their different sets of builtins at our disposal to tinker with (we’ll leave this as an exercise for the reader).