Chaos Malware Quietly Evolves Persistence and Evasion Techniques

Originally published by Sysdig.

Written by Nicholas Lang.

The name Chaos is being used for a ransomware strain, a remote access trojan (RAT), and now a DDoS malware variant too. Talk about chaos! In this case, Sysdig’s Threat Research Team captured attacks using the Chaos variant of the Kaiji botnet malware. There is very little reported information on this malware since September 2022, perhaps because of the unfortunately chaotic naming, or simply because it is relatively new. Kaiji malware was of Chinese origin in 2020 and is written in Golang. Similarly, Chaos is a Chinese Golang malware developed for both Windows and Linux operating systems, and other multiple hardware architectures too.

What makes Chaos interesting is that it puts a lot of effort into persisting on its target, while also implementing defense evasion tactics which are not commonly seen in Linux malware.

Chaos made a lot of noise in our honeypot in mid-January when we directly observed this malware attacking a misconfigured Apache Tomcat environment. We saw it again at the end of February with some evolutions. Previous iterations of this malware were sourced from a publicly available malware repository and analyzed by Lumen’s Black Lotus Labs.

We will go through the analysis of our captured attacks with an emphasis on persistence techniques in this blog, and share our Indicators of Compromise (IOCs) at the end.

Kaiji malware

As far as behavioral attributes, we concur with previous reporting that this Chaos malware is an evolution of the Kaiji botnet, with much of the same previously reported functionality. To summarize, Kaiji was a DDoS botnet that mainly attacked IoT devices via SSH brute-forcing, hence the source language being Go and easy cross-compilation to common IoT architectures like PowerPC and SPARC. The Chaos variant we captured shows all of the same DDoS functionality as the previously reported-on version, and rather than build a Chaos emulator, we identified that the code supporting this functionality was still present in this new version.

Technical analysis

After being installed via the exploitation of a misconfigured Apache Tomcat environment, Chaos malware pivoted to install ALL the persistence mechanisms. We will explain in greater detail below, but the actors behind this attack really wanted to ensure their attack would survive a reboot, which begs the question of whether or not they considered that the world has moved on to containerized workloads – none of these persistence mechanisms would survive a container restart.

We started our analysis using the tool ssdeep for fuzzy hashing, or comparing similar but not identical files. This allowed us to cluster the files we captured based on similarity. The invocation in the screenshot is telling ssdeep to group the files by similarity (-g), and to hash all of the files in the directory (-d). First, we found 10 files that were all the same binary. Turns out, they are copies of the malware itself. The remaining files were scripts used to execute the malware via different persistence mechanisms.

T1053.003 – scheduled task/job: cron

First of all, we saw that persistence was achieved by copying itself to the file path /etc/id.services.conf and creating the file /etc/32678. This action remains unchanged from previous reporting and for this reason, we knew we were likely looking at Chaos. The /etc/32678file contains the following shell script:

<code><span class="hljs-comment">#!/bin/sh</span>
<span class="hljs-keyword">while</span> [ <span class="hljs-number">1</span> ]; 
<span class="hljs-keyword">do</span>
<span class="hljs-keyword">sleep</span> <span class="hljs-number">60</span>
/etc/id.services.conf
Done</code><small><span class="shcb-language__label">Code language:</span> 
<span class="shcb-language__name">Perl</span> <span class="shcb-language__paren">
(</span><span class="shcb-language__slug">perl</span><span class="shcb-language__paren">)
</span></small>

The /etc/32678 file was then executed by adding an entry to crontab and executing cron -f, which makes cron run in the foreground instead of background. The script will attempt to execute the malware while the cron program continues to run. This can be considered the malware’s initial startup. Since running cron like this isn’t persistent, the attacker resorts to a number of additional methods to ensure their malware comes back after a reboot or if it dies.

T1554: compromise client software binary

Chaos attempts to use the user and automated scripts as a persistence mechanism by trojaning common user binaries. When these binaries are executed, the main Chaos payload will be run. The original program is not called, so the expected behavior of the command would not occur. This tactic also has the side effect of making it difficult for a user to see what is happening on the system.

The files replaced are:

The Chaos malware does attempt to hide its presence in a rather uncommon way. Shell functions in the gateway.sh script, when placed in /etc/profile.d/, will run the users’ shell commands and filter out any sign of the malwares’ presence. The example below replaces the find command and uses sed to strip out its own filenames.

<code>function find {
    proc_name=$(/usr/bin/find $@) <span class="hljs-comment"># run the backdoored find with the original arguments specified 
by the user</span>
    proc_name=$(echo <span class="hljs-string">"$proc_name"</span> | sed -e <span class="hljs-string">'/32676/d'</span>)
    proc_name=$(echo <span class="hljs-string">"$proc_name"</span> | sed -e <span class="hljs-string">'/dns-tcp4/d'</span>)
    proc_name=$(echo <span class="hljs-string">"$proc_name"</span> | sed -e <span class="hljs-string">'/quotaoff.service/d'</span>)
    proc_name=$(echo <span class="hljs-string">"$proc_name"</span> | sed -e <span class="hljs-string">'/System.mod/d'</span>)
    proc_name=$(echo <span class="hljs-string">"$proc_name"</span> | sed -e <span class="hljs-string">'/gateway.sh/d'</span>)
    proc_name=$(echo <span class="hljs-string">"$proc_name"</span> | sed -e <span class="hljs-string">'/32676/d'</span>)
    . . . <other IoCs here>
    echo <span class="hljs-string">"$proc_name"</span>
}</code><small><span class="shcb-language__label">Code language:</span> <span class="shcb-language__name">Perl</span> 
<span class="shcb-language__paren">(</span><span class="shcb-language__slug">perl</span><span class="shcb-language__paren">)
</span></small>

T1546.004 – event triggered execution: unix shell configuration modification

Files in /etc/profile/ set the environment variables at startup of the bash shell. The /etc/profile.d/ directory contains other scripts that contain application-specific startup files, which are also executed at startup time by the shell. This is a common place attackers can place their own files in order to gain execution. In this case, it occurs when a shell is launched, such as when a user logs in to the system.

The attacker placed the /etc/profile.d/bash_config.sh file which contains:

<code><span class="hljs-comment">#!/bin/sh</span>
/etc/profile.d/bash_config</code><small><span class="shcb-language__label">Code language:</span> 
<span class="shcb-language__name">Perl</span> <span class="shcb-language__paren">(</span>
<span class="shcb-language__slug">perl</span><span class="shcb-language__paren">)</span></small>

This invokes the specified malware each time a new bash shell is spawned. Note that this is the ELF version of the malware, not another script.

T1053.003 – scheduled task/job: cron

Cron is a system binary that is analogous to scheduled tasks for those coming from the Windows world. Cron allows attackers to ensure that their malware will be restarted after a certain time interval, increasing the persistence of the attack. The crontab file invokes the hidden file .img, which is a shell script that then calls the libdlrpcld.so file, which is yet another copy of the malware. The following cron entry will execute .img every minute.

<code>*<span class="hljs-regexp">/1 * * * * root /</span>.img</code><small>
<span class="shcb-language__label">Code language:</span> 
<span class="shcb-language__name">Perl</span> <span class="shcb-language__paren">
(</span><span class="shcb-language__slug">perl</span>
<span class="shcb-language__paren">)</span></small>

The .img file is a shell script which calls a copy of the malware, named libdlrpcld.so like so:

<code><span class="hljs-comment">#!/bin/sh\n/usr/lib/libdlrpcld.so     
</span></code><small><span class="shcb-language__label">Code language:</span> 
<span class="shcb-language__name">Perl</span> <span class="shcb-language__paren">
(</span><span class="shcb-language__slug">perl</span><span class="shcb-language__paren">)
</span></small>

T1543.002 – create or modify system process: systemd service

Systemd’s primary component is a “system and service manager” – an init system used to bootstrap user space and manage user processes. It also provides replacements for various daemons and utilities, including device management, login management, network connection management, and event logging. In this case, the attackers wrote a systemd service that will run the malware (here named System.img.config) on system init.

The file /usr/lib/systemd/linux.service was created by the attacker as a systemd service that executes the malware on boot:

<code>$ cat linux.service
[Unit]
Description=linux
After=network.target
[Service]
Type=forking
ExecStart=<span class="hljs-regexp">/boot/</span>System.img.config
ExecReload=<span class="hljs-regexp">/boot/</span>System.img.config
ExecStop=<span class="hljs-regexp">/boot/</span>System.img.config
[Install]
WantedBy=multi-user.target </code><small><span class="shcb-language__label">Code language:</span> 
<span class="shcb-language__name">Perl</span> <span class="shcb-language__paren">(</span>
<span class="shcb-language__slug">perl</span><span class="shcb-language__paren">)</span></small>

T1037 – boot or logon initialization scripts

In Unix-based computer operating systems, init (short for initialization) is the first process started during booting of the operating system. Init is a daemon process that continues running until the system is shut down. It is the direct or indirect ancestor of all other processes and automatically adopts all orphaned processes. Init is started by the kernel during the booting process; a kernel panic will occur if the kernel is unable to start it. Init is typically assigned PID 1. Init scripts placed in the <em>/etc/init.d/</em> directory allow for users to write their own startup scripts or programs.

Below, you can see the threat actors behind Chaos leverage init scripts to ensure that their malware (here named <em>System.img.config</em>) will run on system startup. They used both init.d and systemd to better their chances of retaining persistence, presumably because they don’t know in advance which system their target uses. The file used was: /etc/init.d/linux_kill. On boot, it will execute the /boot/System.img.config file, which is the Chaos malware.

<code>cat linux_kill
<span class="hljs-comment">#!/bin/sh</span>
<span class="hljs-comment">### BEGIN INIT INFO</span>
<span class="hljs-comment">#chkconfig: 2345 10 90</span>
<span class="hljs-comment">#description:System.img.config</span>
<span class="hljs-comment"># Default-Start:    2 3 4 5</span>
<span class="hljs-comment"># Default-Stop:</span>
<span class="hljs-comment">### END INIT INFO</span>
/boot/System.img.config
<span class="hljs-keyword">exit</span> <span class="hljs-number">0</span>  
</code><small><span class="shcb-language__label">Code language:</span> 
<span class="shcb-language__name">Perl</span> <span class="shcb-language__paren">
(</span><span class="shcb-language__slug">perl</span>
<span class="shcb-language__paren">)</span></small>

Binary analysis

During our investigation, we observed two versions of the Chaos malware which we will call 32678 and 32676. While the surrounding shell scripts were different and showed an evolution of tactics, the malware itself seems to have remained very similar. First, we used ssdeep to compare them, but as the results show below, they were not the same.

<code>$ ssdeep ./files/System.mod ../chaos/files/System.img.config
ssdeep,<span class="hljs-number">1.1</span>--blocksize:hash:hash,filename
<span class="hljs-number">24576</span>:ae9ufJvk4gQjMNRfktnsIXvZFyD9i+MPCIxyuzNqssZXJjZbdYVVMtIwWz1v:WYMnwRO4ssPJd5Wz1,
<span class="hljs-string">"/Users/nicholaslang/chaos_new/files/System.mod"</span>
<span class="hljs-number">49152</span>:E33d0lGt6UHcFL7Rn2o03wiEhiDmzzd/<span class="hljs-number">9</span>sARlBs/
<span class="hljs-number">00</span>Cpfx9a9uNYp9hW16klbU6V:E33GlbU8FwmzzRDZ9mjqRV,
<span class="hljs-string">"/Users/nicholaslang/chaos/files/System.img.config"</span></code><small>
<span class="shcb-language__label">Code language:</span> <span class="shcb-language__name">Perl</span> 
<span class="shcb-language__paren">(</span><span class="shcb-language__slug">perl</span>
<span class="shcb-language__paren">)</span></small>

Next, we measured the entropy, or randomness, of each binary and plotted them on a graph. Different types of data tend to have different levels of entropy; normal x64 code is less random than an encrypted executable. In order to obfuscate an executable, threat actors will often employ various methods to change the binary so the hashes no longer match, which we saw in this case.

The two versions of the Chaos malware (32676 and 32678, respectively) are extremely different when fuzzy-hashed with ssdeep, but the entropy graphs reveal very similar (near-identical) binary layouts. This suggests that the threat actor did attempt to obfuscate the binary between attacks, but their methods did not significantly alter the binary’s structure. There are likely no major changes to its functionality either. This investigative technique makes it easier to identify additional Chaos variants.

Conclusion

Chaos is either not being deployed very frequently, being miscategorized as its former Kaiji parent version and therefore ignored, or is not being found. Regardless, we were surprised to find so little information on what seems to be a fairly capable piece of malware. The authors of the malware have put more effort than most in trying to persist the malware across reboots and hide its presence.

Our analysis showed that there were several copies of the malware being used in the wild. There are also multiple persistence mechanisms for malware execution, an indication that the actor is thorough but not necessarily competent with containers. While a simple reboot in a containerized environment will rid you of this botnet, you should patch the initial access vector (likely a CVE) to truly rid yourself of this infection.

IOCs

IP Addresses:

98.159.98[.]203

107.189.7[.]51

For additional IoCs associated with this campaign, please visit our GitHub page.