\n\t\t\tJay Beale created two tools used by hundreds of thousands of individuals, companies and governments, Bastille Linux and the Center for Internet Security’s first Linux\/UNIX scoring tool. He has led training classes on Linux security at the Black Hat, CanSecWest, RSA, and IDG conferences, as well as in private corporate training, since 2000. As an author, series editor and speaker, Jay has contributed to nine books and two columns and given roughly one hundred public talks. He is a co-founder, COO and CTO of the information security consulting company InGuardians.\n\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div> If you’ve read my past blog posts or watched my Linux Attack and Defense webinars, you’ll see that one of my favorite hobbies is attacking a single-person capture the flag (CTF) virtual machine, then finding a proactive defense that would break my attack path. The critical point: even if you didn’t know that your machine or application had vulnerabilities, you can break the exploits against them with hardening. In my\u00a0 “Bust-a-Kube” series of blog posts, I’m going to do this with Kubernetes<\/a>, the market-leading open-source container orchestration system.\u00a0 If you enjoy this, watch out for talks that I’m doing with other InGuardians folks at Hushcon 2018, RSA 2019, and online.<\/span><\/p>\n

Kubernetes has become incredibly popular, particularly for DevOps teams who want to automate deployment, horizontal scaling, A\/B testing, and failure tolerance for an application. I’ve had quite a bit of fun compromising Kubernetes clusters in penetration tests, even as I read about illicit Kubernetes compromises, like that of Tesla, Inc.\u00a0 <\/span>In this blog post, I’ll show you how the students in my 2018 training classes compromised and then hardened an intentionally-vulnerable Kubernetes cluster. In the next episode, we’ll attack and defend a multi-tenant Kubernetes cluster.<\/span><\/p>\n

We start by finding a vulnerable application, running in a Kubernetes cluster. Our target will be one of the Kubernetes documentation’s example applications, a Redis-backed Guestbook<\/a>, which has a small vulnerability, to which another has been added to give us remote code execution.<\/span><\/p>\n

<\/p>\n

Compromising the Application<\/span><\/h2>\n

Let\u2019s start by reading the source code (HTML and JavaScript) for this page. We see that this is an AJAX form, not a 1990’s style HTML form \u2013 clicking the submit button doesn\u2019t simply send a GET or POST request directly. Instead, the submit button calls JavaScript from the controller.js file to both pass the form in and update this page with output.<\/span><\/p>\n

<\/p>\n

This means we\u2019ll need to read controller.js to understand more about what\u2019s going on here. We pull down a copy of controller.js\u00a0<\/span>and see two functions.\u00a0 <\/span>Here\u2019s an excerpt from the first, which the submit button triggers:<\/p>\n

<\/p>\n

The JavaScript makes a GET request to guestbook.php, passing it these arguments:<\/span><\/p>\n

CMD:    set<\/span>\r\nKEY:    messages<\/span>\r\nVALUE:  the item we entered in the form\u2019s text box<\/span>\r\n<\/pre>\n
<\/dl>\n
This is where my co-worker Stan shouts out, “my Spidey Sense is tingling!”<\/p>\n
<\/p>\n
\u201cJay,\u201d says our cameo star, \u201cwhy does the GET request need to say what database key the message should go in? What would happen if you specified a different key?\u201d <\/span><\/p>\n
I tell Stan to be patient, but that I want to read the rest of the source before I go off checking that out.\u00a0 <\/span>Here\u2019s the other function in controller.js:<\/span><\/p>\n
<\/p>\n
This is the one that populates the messages on the Guestbook page, by running a request against guestbook.php with two of those three parameters:<\/span><\/p>\n
CMD:    get<\/span>\r\nKEY:    messages<\/span><\/pre>\n
I bet Stan is right, that the server side code is letting the browser choose which Redis key to put data into. We\u2019ll come back to that, once we find something for which that bug is useful. I start to scratch my head, so I think of my enumeration basics. Well, we’ve got a web application, so let’s run OWASP’s DirBuster to find all of the pages and directories that we might not find just by following links.\u00a0 This will expand our view of the application’s attack surface. In my blog posts and webinars, I use stock Kali Linux (link: https:\/\/www.kali.org\/downloads\/<\/span><\/a>) so the reader can practice the same attacks, using the exact same tools as I\u2019m using. Kali has DirBuster set up wonderfully.<\/span><\/p>\n
<\/p>\n
After a couple minutes of trying nearly 500 page names every second, DirBuster finds two pages that could be useful to us: status.php and guestbook.php.<\/span><\/p>\n
<\/p>\n
We know about the guestbook.php page, so let’s investigate the status.php page.<\/span><\/p>\n
<\/p>\n
This is interesting \u2013 we\u2019re getting an error message that implies that this status.php form runs whatever program is in the Redis database\u2019s command key, but that the command key is empty in Redis. The page is setting the command key to “curl http:\/\/10.23.58.50\/status” and asking us to reload the page.<\/span><\/p>\n
When we reload the page, we see this:<\/span><\/p>\n
<\/p>\n
If we try the same curl command on our test system, we\u2019d get the same result: five words of plain text. It appears that we can get remote code execution if we can change the \u201ccommand\u201d key in Redis.<\/span><\/p>\n
Putting that together with Stan\u2019s hunch, we check to see if this gives us remote command execution. We create our own manual GET submission to guestbook.php, setting the cmd<\/em> variable to set<\/em>, the key<\/em> variable to command<\/em> and the value<\/em> variable to whoami<\/em>, a Linux shell command.<\/span><\/p>\n
<\/p>\n
We\u2019re told that the key has been updated. Now let\u2019s reload our status.php page to see if it runs our command:<\/span><\/p>\n
<\/p>\n
Ok \u2013 we\u2019ve got remote command execution! Now, while it is a bit stealthy, this is a pretty unsatisfying web shell to use. We have to use one tab to set a key to a command, then a separate tab to run the command and see its output. It doesn’t feel very interactive or natural. Since I don\u2019t need stealth, I turn to Metasploit.\u00a0 <\/span>I start up a Metasploit console, running the multi\/handler exploit module, with payload set to linux\/x64\/meterpreter\/reverse_tcp and then use msfvenom to create a Linux Meterpreter binary that will connect back to a Metasploit console and stage that binary for download, using Python\u2019s SimpleHTTPServer module:<\/span><\/p>\n
<\/p>\n
Next, I set the Redis command key to pull down that Meterpreter, like so:<\/span><\/p>\n
<\/p>\n
After refreshing status.php to run that curl command, I run a couple more commands through this, renaming the downloaded file to mrsbin and setting it to be executable. Finally, I push my final command through this funky interface, telling the server to run my Meterpreter binary (mrsbin):<\/span><\/p>\n
<\/p>\n
Switching to my Metasploit console, I find that I\u2019ve caught a connection:<\/span><\/p>\n
<\/p>\n
Now it\u2019s time to explore! I use the Meterpreter\u2019s shell command to get a shell, then begin to poke around.\u00a0 <\/span>Eventually, running a mount command, I learn that the web server I\u2019ve compromised is running in a Linux container, on a host running Docker, managed by the Kubernetes container orchestration system.\u00a0 <\/span>At this point, I want to see if there\u2019s a service account token mounted into that container, as is default in Kubernetes. I explore the directory where Kubernetes mounts this, \/run\/secrets\/kubernetes.io\/serviceaccount.<\/span><\/p>\n
<\/p>\n
We\u2019ve got a service account token, a text file that tells us what namespace this service account <\/span>belongs to, and a text file with the root certificate of the cluster\u2019s certificate authority.\u00a0 <\/span>I use the Meterpreter to upload a kubectl (pronounced \u201ckoob-control\u201d) binary. While I could just use curl commands to interact with the Kubernetes API server, kubectl is far quicker and more user friendly.<\/span><\/p>\n
I use my web shell to get the Kubernetes API server IP address and port, then set up a kubectl command that\u2019s configured to use that server IP, the contents of the token file and the location of the ca.crt file, passing it the argument \u201cget pods,\u201d to get a list of pods in the default namespace:<\/span><\/p>\n
<\/p>\n
When I later run this same command with the -o wide (wide output) option, I see that there are three \u201cfrontend\u201d pods, one of which has my Meterpreter running in it.<\/span><\/p>\n
<\/p>\n
Kubernetes automatically tries to distribute identical pods to different nodes, to handle hardware failures, so the three frontend pods are distributed between two worker nodes, k8s-node1 and k8s-node2.\u00a0 <\/span>Each pod gets its own IP address, so the three frontend pods have IP addresses 10.32.0.5, 10.46.0.2, and 10.46.0.3.\u00a0 <\/span>Kubernetes handles the load balancing as well.<\/span><\/p>\n
There are also three other pods running in this namespace: one for a Redis database master and two Redis database slaves, likely tied to the master. We\u2019ll look at those later, but first, let\u2019s figure out if we can deploy a new pod to the cluster.<\/span><\/p>\n
I want to create a YAML file that describes a new pod I\u2019d like to run on the cluster. I start by asking the cluster for a YAML-formatted description of the redis-master pod by running:<\/span><\/p>\n
kubectl get pod redis-master-55dv5f7567-ndjj9 -o yaml<\/span><\/p>\n
I make my own new version of this pod definition that uses the Redis:e2e docker image, but adds a volume mount, mounting the node (Docker host)\u2019s root filesystem (\/) onto \/root in the container in this pod.<\/span><\/p>\n
<\/p>\n
Let\u2019s try deploying it with a kubectl apply -f attack-pod.yaml.\u00a0 <\/span>Kubectl apply is the bulk of how you create things on Kubernetes.<\/span><\/p>\n
<\/p>\n
Darn it! This pod\u2019s service account can get a list of pods, get the YAML definition files for a pod, but can\u2019t create pods to the cluster.\u00a0 <\/span>I wonder if we can move laterally to another pod? Let\u2019s use the Kubernetes API auth module to ask, with the \u201ckubectl can-I exec pods\u201d command.<\/span><\/p>\n
<\/p>\n
Awesome! Let\u2019s exec into one of the containers in the redis-master pod, using the same kind of syntax you\u2019d use in Docker: kubectl exec -it <pod> <command>.<\/span><\/p>\n
<\/p>\n
The hostname in the container is set to the name of the pod, so we\u2019ve just demonstrated that we have a shell in the redis-master pod.\u00a0 <\/span>From here, I pull down a copy of the same Meterpreter binary I put in the frontend pod and run it:<\/span><\/p>\n
<\/p>\n
Backgrounding this session in the Metasploit console, I catch the incoming Meterpreter connection and upload a kubectl binary to the redis-master pod.<\/span><\/p>\n
<\/p>\n
Now I clean up my shell environment to make it more attacker, err, I mean, user-friendly:<\/span><\/p>\n
<\/p>\n
I\u2019ll also use an environment variable and a shell alias so I can run a simple kubectl command without having to copy and paste all those arguments over and over.<\/span><\/p>\n
<\/p>\n
I copy and paste in that same attack-pod.yaml file that I created in the frontend pod.\u00a0 <\/span>Now, let\u2019s try to create that pod on the cluster:<\/span><\/p>\n
<\/p>\n
Success! Let\u2019s see which node that pod ended up on:<\/span><\/p>\n
<\/p>\n
The pod is on the second Kubernetes worker node, k8s-node2. Let\u2019s exec into that pod and see if the volume mount worked, by checking to see if \/root contains the worker node\u2019s root filesystem:<\/span><\/p>\n
<\/p>\n
Bingo! Let\u2019s start a new shell whose root filesystem is the container\u2019s \/root, which is the node\u2019s root (\/) filesystem via the chroot command. We can then take a look at the node\u2019s sudoers file:<\/span><\/p>\n
<\/p>\n
So, anyone in the admin and sudo groups can run any command as root.\u00a0 <\/span>OK, let\u2019s find a member of the sudo group and change their password:<\/span><\/p>\n
<\/p>\n
Now, we can ssh into the node:<\/span><\/p>\n
<\/p>\n
And sudo su – to get a root shell!<\/span><\/p>\n
<\/p>\n
So, we\u2019ve compromised one node in the cluster. We could create a variant of our attack pod that requests that it be placed on a specific node to, one at a time, compromise each node in the cluster, perhaps even the master(s). But, this being Kubernetes, which generally manages not just two worker node machines but instead tens, hundreds or thousands of machines, we really should be thinking about a more scalable method. <\/span><\/p>\n
Kubernetes has a construct that is perfect for this, called a \u201cdaemonset.\u201d The daemonset\u2019s purpose is to run a program on every node in your cluster, like a log collection program or such. We\u2019ll create a daemonset variant of that attack pod, like so:<\/span><\/p>\n
<\/p>\n
Now we\u2019ll apply it and get to see an attack pod running on every node of this three-node cluster:<\/span><\/p>\n
<\/p>\n
Again, going for scalability in our attack, I\u2019ll create a one-line shell script to run a command on a container on every node.\u00a0 <\/span>To demonstrate root privilege with full filesystem access, I grab the password hashes from each of the node machines:<\/span><\/p>\n
<\/p>\n
I leave it to the reader to imagine how one might turn these hashes into interactive root login on the nodes. \u201cBut wait,\u201d cameo Stan says, \u201cyou\u2019ve only demonstrated read access! How about write? Let me show you how we do it in the afterlife, kid!\u201d With that, he seizes my keyboard and modifies my loop:<\/span><\/p>\n
<\/p>\n
Then he logs into the Kubernetes master, using his omniscience, and demonstrates that his 0wn3d.txt file was written.<\/span><\/p>\n
<\/p>\n
I see what he means. If he can write to a file in the root directory of the filesystem, he could change any file on the filesystem (exercise left to the reader or questions via Twitter). <\/span><\/p>\n
So, at roughly the 2,000 word mark, we\u2019ve broken into a container running in a pod on a Kubernetes cluster, moved laterally to another container\/pod, and started pods on the cluster that let us take over every node in the cluster.\u00a0<\/span>Let\u2019s talk about defenses.<\/p>\n
Defending this Kubernetes Cluster<\/h2>\n
You could certainly create a pod security policy to prevent pods from mounting directories from the node filesystems. You could also use a similar policy to prevent pods from running as root. You could also create a network policy that prevented Meterpreter shells from connecting back to you. But since the attack here was basically an exploration of over-privileged service accounts, let\u2019s look at that.<\/span><\/p>\n
Kubernetes’ primary authorization system is its RBAC<\/a> module.\u00a0 RBAC lets us define what subjects can take what actions on what types of objects, in what namespaces. There’s a good deal you can learn about this, but let’s use this practical example to see how easy it is.<\/p>\n
The redis-master pod didn\u2019t need to be able to stage pods on the cluster. Let\u2019s create an RBAC policy that says that the redis-master and redis-slave pods should run with a specific service account, which we\u2019ll create and call \u201credis.\u201d Note how simple this is \u2013 we just hand Kubernetes a YAML file that says we\u2019d like to create an object called \u201credis\u201d of type \u201cServiceAccount.\u201d<\/span><\/p>\n
<\/p>\n
Service accounts and users are \u201csubjects\u201d in RBAC.\u00a0 <\/span>We\u2019re trying to encode the statements:<\/span><\/p>\n
\n
The role \u201cget-pods\u201d can execute the get verb on pods.<\/span><\/li>\n<\/ol>\n
\n
The service account redis has the role \u201cget-pods.\u201d<\/span><\/li>\n<\/ol>\n
We\u2019ll create the \u201cget-pods\u201d role first:<\/span><\/p>\n
<\/p>\n
Now let\u2019s bind the \u201cget-pods\u201d role to the service account \u201credis\u201d by creating a role binding:<\/span><\/p>\n
<\/p>\n
The only remaining step here is to restart the redis-master and redis-slave deployments, with their ServiceAccount set to \u201credis\u201d:<\/span><\/p>\n
<\/p>\n
To test whether our defense was effective, we\u2019ll kubectl exec into the redis-master pod where we had put the kubectl binary and try to deploy a pod onto the cluster:<\/span><\/p>\n
<\/p>\n
Success!<\/span><\/p>\n
I\u2019ll be publishing the intentionally-vulnerable Kubernetes cluster I used in this demo soon, accessible on the site: www.bustakube.com<\/span><\/a> . Go practice this with it! <\/span><\/p>\n
\u00a0\u00a0<\/span><\/span><\/p>\n
[\/et_pb_text][\/et_pb_column][\/et_pb_row][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"
\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div>
\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t<\/div> \n\t\t
\n\t\t\t
\n\t\t\t\t\n\t\t\t<\/div>\n\t\t<\/div> \n\t\t
\n\t\t\t\n\t\t\t
<\/div>\n\t\t<\/div> \n\t\t
\n\t\t\t\n\t\t<\/div>Jay Beale created two tools used by hundreds of thousands of individuals, companies and governments, Bastille Linux and the Center for Internet Security’s first Linux\/UNIX scoring tool. He has led training classes on Linux security at the Black Hat, CanSecWest, RSA, and IDG conferences, as well as in private corporate training, […]<\/p>\n","protected":false},"author":4,"featured_media":1796,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"on","_et_pb_old_content":"
[author] [author_image timthumb='on']https:\/\/www.inguardians.com\/wp-content\/uploads\/2017\/03\/jay-resize.jpg[\/author_image] [author_info]Jay Beale created two tools used by hundreds of thousands of individuals, companies and governments, Bastille Linux and the Center for Internet Security's first Linux\/UNIX scoring tool. He has led training classes on Linux security at the Black Hat, CanSecWest, RSA, and IDG conferences, as well as in private corporate training, since 2000. As an author, series editor and speaker, Jay has contributed to nine books and two columns and given roughly one hundred public talks. He is a co-founder, COO and CTO of the information security consulting company InGuardians.[\/author_info] [\/author] If you've read my past blog posts or watched my Linux Attack and Defense webinars, you'll see that one of my favorite hobbies is attacking a single-person capture the flag (CTF) virtual machine, then finding a proactive defense that would break my attack path. The critical point: even if you didn't know that your machine or application had vulnerabilities, you can break the exploits against them with hardening. In my\u00a0 \"Bust-a-Kube\" series of blog posts, I'm going to do this with Kubernetes<\/a>, the market-leading open-source container orchestration system.\u00a0 If you enjoy this, watch out for talks that I'm doing with other InGuardians folks at Hushcon 2018, RSA 2019, and online.<\/span><\/p>
Kubernetes has become incredibly popular, particularly for DevOps teams who want to automate deployment, horizontal scaling, A\/B testing, and failure tolerance for an application. I've had quite a bit of fun compromising Kubernetes clusters in penetration tests, even as I read about illicit Kubernetes compromises, like that of Tesla, Inc.\u00a0 <\/span>In this blog post, I'll show you how the students in my 2018 training classes compromised and then hardened an intentionally-vulnerable Kubernetes cluster. In the next episode, we'll attack and defend a multi-tenant Kubernetes cluster.<\/span><\/p>
We start by finding a vulnerable application, running in a Kubernetes cluster. Our target will be one of the Kubernetes documentation's example applications, a Redis-backed Guestbook<\/a>, which has a small vulnerability, to which another has been added to give us remote code execution.<\/span><\/p>
<\/p>
Compromising the Application<\/span><\/h2>
Let\u2019s start by reading the source code (HTML and JavaScript) for this page. We see that this is an AJAX form, not a 1990's style HTML form \u2013 clicking the submit button doesn\u2019t simply send a GET or POST request directly. Instead, the submit button calls JavaScript from the controller.js file to both pass the form in and update this page with output.<\/span><\/p>
<\/p>
This means we\u2019ll need to read controller.js to understand more about what\u2019s going on here. We pull down a copy of controller.js\u00a0<\/span>and see two functions.\u00a0 <\/span>Here\u2019s an excerpt from the first, which the submit button triggers:<\/p>
<\/p>
The JavaScript makes a GET request to guestbook.php, passing it these arguments:<\/span><\/p>
CMD:    set<\/span>\r\nKEY:    messages<\/span>\r\nVALUE:  the item we entered in the form\u2019s text box<\/span>\r\n<\/pre>
<\/dl>
This is where my co-worker Stan shouts out, \"my Spidey Sense is tingling!\"<\/p>
<\/p>
\u201cJay,\u201d says our cameo star, \u201cwhy does the GET request need to say what database key the message should go in? What would happen if you specified a different key?\u201d <\/span><\/p>
I tell Stan to be patient, but that I want to read the rest of the source before I go off checking that out.\u00a0 <\/span>Here\u2019s the other function in controller.js:<\/span><\/p>
<\/p>
This is the one that populates the messages on the Guestbook page, by running a request against guestbook.php with two of those three parameters:<\/span><\/p>
CMD:    get<\/span>\r\nKEY:    messages<\/span><\/pre>
I bet Stan is right, that the server side code is letting the browser choose which Redis key to put data into. We\u2019ll come back to that, once we find something for which that bug is useful. I start to scratch my head, so I think of my enumeration basics. Well, we've got a web application, so let's run OWASP's DirBuster to find all of the pages and directories that we might not find just by following links.\u00a0 This will expand our view of the application's attack surface. In my blog posts and webinars, I use stock Kali Linux (link: https:\/\/www.kali.org\/downloads\/<\/span><\/a>) so the reader can practice the same attacks, using the exact same tools as I\u2019m using. Kali has DirBuster set up wonderfully.<\/span><\/p>
<\/p>
After a couple minutes of trying nearly 500 page names every second, DirBuster finds two pages that could be useful to us: status.php and guestbook.php.<\/span><\/p>
<\/p>
We know about the guestbook.php page, so let's investigate the status.php page.<\/span><\/p>
<\/p>
This is interesting \u2013 we\u2019re getting an error message that implies that this status.php form runs whatever program is in the Redis database\u2019s command key, but that the command key is empty in Redis. The page is setting the command key to \"curl http:\/\/10.23.58.50\/status\" and asking us to reload the page.<\/span><\/p>
When we reload the page, we see this:<\/span><\/p>
<\/p>
If we try the same curl command on our test system, we\u2019d get the same result: five words of plain text. It appears that we can get remote code execution if we can change the \u201ccommand\u201d key in Redis.<\/span><\/p>
Putting that together with Stan\u2019s hunch, we check to see if this gives us remote command execution. We create our own manual GET submission to guestbook.php, setting the cmd<\/em> variable to set<\/em>, the key<\/em> variable to command<\/em> and the value<\/em> variable to whoami<\/em>, a Linux shell command.<\/span><\/p>
<\/p>
We\u2019re told that the key has been updated. Now let\u2019s reload our status.php page to see if it runs our command:<\/span><\/p>
<\/p>
Ok \u2013 we\u2019ve got remote command execution! Now, while it is a bit stealthy, this is a pretty unsatisfying web shell to use. We have to use one tab to set a key to a command, then a separate tab to run the command and see its output. It doesn't feel very interactive or natural. Since I don\u2019t need stealth, I turn to Metasploit.\u00a0 <\/span>I start up a Metasploit console, running the multi\/handler exploit module, with payload set to linux\/x64\/meterpreter\/reverse_tcp and then use msfvenom to create a Linux Meterpreter binary that will connect back to a Metasploit console and stage that binary for download, using Python\u2019s SimpleHTTPServer module:<\/span><\/p>
<\/p>
Next, I set the Redis command key to pull down that Meterpreter, like so:<\/span><\/p>
<\/p>
After refreshing status.php to run that curl command, I run a couple more commands through this, renaming the downloaded file to mrsbin and setting it to be executable. Finally, I push my final command through this funky interface, telling the server to run my Meterpreter binary (mrsbin):<\/span><\/p>
<\/p>
Switching to my Metasploit console, I find that I\u2019ve caught a connection:<\/span><\/p>
<\/p>
Now it\u2019s time to explore! I use the Meterpreter\u2019s shell command to get a shell, then begin to poke around.\u00a0 <\/span>Eventually, running a mount command, I learn that the web server I\u2019ve compromised is running in a Linux container, on a host running Docker, managed by the Kubernetes container orchestration system.\u00a0 <\/span>At this point, I want to see if there\u2019s a service account token mounted into that container, as is default in Kubernetes. I explore the directory where Kubernetes mounts this, \/run\/secrets\/kubernetes.io\/serviceaccount.<\/span><\/p>
<\/p>
We\u2019ve got a service account token, a text file that tells us what namespace this service account <\/span>belongs to, and a text file with the root certificate of the cluster\u2019s certificate authority.\u00a0 <\/span>I use the Meterpreter to upload a kubectl (pronounced \u201ckoob-control\u201d) binary. While I could just use curl commands to interact with the Kubernetes API server, kubectl is far quicker and more user friendly.<\/span><\/p>
I use my web shell to get the Kubernetes API server IP address and port, then set up a kubectl command that\u2019s configured to use that server IP, the contents of the token file and the location of the ca.crt file, passing it the argument \u201cget pods,\u201d to get a list of pods in the default namespace:<\/span><\/p>
<\/p>
When I later run this same command with the -o wide (wide output) option, I see that there are three \u201cfrontend\u201d pods, one of which has my Meterpreter running in it.<\/span><\/p>
<\/p>
Kubernetes automatically tries to distribute identical pods to different nodes, to handle hardware failures, so the three frontend pods are distributed between two worker nodes, k8s-node1 and k8s-node2.\u00a0 <\/span>Each pod gets its own IP address, so the three frontend pods have IP addresses 10.32.0.5, 10.46.0.2, and 10.46.0.3.\u00a0 <\/span>Kubernetes handles the load balancing as well.<\/span><\/p>
There are also three other pods running in this namespace: one for a Redis database master and two Redis database slaves, likely tied to the master. We\u2019ll look at those later, but first, let\u2019s figure out if we can deploy a new pod to the cluster.<\/span><\/p>
I want to create a YAML file that describes a new pod I\u2019d like to run on the cluster. I start by asking the cluster for a YAML-formatted description of the redis-master pod by running:<\/span><\/p>
kubectl get pod redis-master-55dv5f7567-ndjj9 -o yaml<\/span><\/p>
I make my own new version of this pod definition that uses the Redis:e2e docker image, but adds a volume mount, mounting the node (Docker host)\u2019s root filesystem (\/) onto \/root in the container in this pod.<\/span><\/p>
<\/p>
Let\u2019s try deploying it with a kubectl apply -f attack-pod.yaml.\u00a0 <\/span>Kubectl apply is the bulk of how you create things on Kubernetes.<\/span><\/p>
<\/p>
Darn it! This pod\u2019s service account can get a list of pods, get the YAML definition files for a pod, but can\u2019t create pods to the cluster.\u00a0 <\/span>I wonder if we can move laterally to another pod? Let\u2019s use the Kubernetes API auth module to ask, with the \u201ckubectl can-I exec pods\u201d command.<\/span><\/p>
<\/p>
Awesome! Let\u2019s exec into one of the containers in the redis-master pod, using the same kind of syntax you\u2019d use in Docker: kubectl exec -it  .<\/span><\/p>
<\/p>
The hostname in the container is set to the name of the pod, so we\u2019ve just demonstrated that we have a shell in the redis-master pod.\u00a0 <\/span>From here, I pull down a copy of the same Meterpreter binary I put in the frontend pod and run it:<\/span><\/p>
<\/p>
Backgrounding this session in the Metasploit console, I catch the incoming Meterpreter connection and upload a kubectl binary to the redis-master pod.<\/span><\/p>
<\/p>
Now I clean up my shell environment to make it more attacker, err, I mean, user-friendly:<\/span><\/p>
<\/p>
I\u2019ll also use an environment variable and a shell alias so I can run a simple kubectl command without having to copy and paste all those arguments over and over.<\/span><\/p>
<\/p>
I copy and paste in that same attack-pod.yaml file that I created in the frontend pod.\u00a0 <\/span>Now, let\u2019s try to create that pod on the cluster:<\/span><\/p>
<\/p>
Success! Let\u2019s see which node that pod ended up on:<\/span><\/p>
<\/p>
The pod is on the second Kubernetes worker node, k8s-node2. Let\u2019s exec into that pod and see if the volume mount worked, by checking to see if \/root contains the worker node\u2019s root filesystem:<\/span><\/p>
<\/p>
Bingo! Let\u2019s start a new shell whose root filesystem is the container\u2019s \/root, which is the node\u2019s root (\/) filesystem via the chroot command. We can then take a look at the node\u2019s sudoers file:<\/span><\/p>
<\/p>
So, anyone in the admin and sudo groups can run any command as root.\u00a0 <\/span>OK, let\u2019s find a member of the sudo group and change their password:<\/span><\/p>
<\/p>
Now, we can ssh into the node:<\/span><\/p>
<\/p>
And sudo su - to get a root shell!<\/span><\/p>
<\/p>
So, we\u2019ve compromised one node in the cluster. We could create a variant of our attack pod that requests that it be placed on a specific node to, one at a time, compromise each node in the cluster, perhaps even the master(s). But, this being Kubernetes, which generally manages not just two worker node machines but instead tens, hundreds or thousands of machines, we really should be thinking about a more scalable method. <\/span><\/p>
Kubernetes has a construct that is perfect for this, called a \u201cdaemonset.\u201d The daemonset\u2019s purpose is to run a program on every node in your cluster, like a log collection program or such. We\u2019ll create a daemonset variant of that attack pod, like so:<\/span><\/p>
<\/p>
Now we\u2019ll apply it and get to see an attack pod running on every node of this three-node cluster:<\/span><\/p>
<\/p>
Again, going for scalability in our attack, I\u2019ll create a one-line shell script to run a command on a container on every node.\u00a0 <\/span>To demonstrate root privilege with full filesystem access, I grab the password hashes from each of the node machines:<\/span><\/p>
<\/p>
I leave it to the reader to imagine how one might turn these hashes into interactive root login on the nodes. \u201cBut wait,\u201d cameo Stan says, \u201cyou\u2019ve only demonstrated read access! How about write? Let me show you how we do it in the afterlife, kid!\u201d With that, he seizes my keyboard and modifies my loop:<\/span><\/p>
<\/p>
Then he logs into the Kubernetes master, using his omniscience, and demonstrates that his 0wn3d.txt file was written.<\/span><\/p>
<\/p>
I see what he means. If he can write to a file in the root directory of the filesystem, he could change any file on the filesystem (exercise left to the reader or questions via Twitter). <\/span><\/p>
So, at roughly the 2,000 word mark, we\u2019ve broken into a container running in a pod on a Kubernetes cluster, moved laterally to another container\/pod, and started pods on the cluster that let us take over every node in the cluster.\u00a0<\/span>Let\u2019s talk about defenses.<\/p>
Defending this Kubernetes Cluster<\/h2>
You could certainly create a pod security policy to prevent pods from mounting directories from the node filesystems. You could also use a similar policy to prevent pods from running as root. You could also create a network policy that prevented Meterpreter shells from connecting back to you. But since the attack here was basically an exploration of over-privileged service accounts, let\u2019s look at that.<\/span><\/p>
Kubernetes' primary authorization system is its RBAC<\/a> module.\u00a0 RBAC lets us define what subjects can take what actions on what types of objects, in what namespaces. There's a good deal you can learn about this, but let's use this practical example to see how easy it is.<\/p>
The redis-master pod didn\u2019t need to be able to stage pods on the cluster. Let\u2019s create an RBAC policy that says that the redis-master and redis-slave pods should run with a specific service account, which we\u2019ll create and call \u201credis.\u201d Note how simple this is \u2013 we just hand Kubernetes a YAML file that says we\u2019d like to create an object called \u201credis\u201d of type \u201cServiceAccount.\u201d<\/span><\/p>
<\/p>
Service accounts and users are \u201csubjects\u201d in RBAC.\u00a0 <\/span>We\u2019re trying to encode the statements:<\/span><\/p>
The role \u201cget-pods\u201d can execute the get verb on pods.<\/span><\/li><\/ol>
The service account redis has the role \u201cget-pods.\u201d<\/span><\/li><\/ol>
We\u2019ll create the \u201cget-pods\u201d role first:<\/span><\/p>
<\/p>
Now let\u2019s bind the \u201cget-pods\u201d role to the service account \u201credis\u201d by creating a role binding:<\/span><\/p>
<\/p>
The only remaining step here is to restart the redis-master and redis-slave deployments, with their ServiceAccount set to \u201credis\u201d:<\/span><\/p>
<\/p>
To test whether our defense was effective, we\u2019ll kubectl exec into the redis-master pod where we had put the kubectl binary and try to deploy a pod onto the cluster:<\/span><\/p>
<\/p>
Success!<\/span><\/p>
I\u2019ll be publishing the intentionally-vulnerable Kubernetes cluster I used in this demo soon, accessible on the site: www.bustakube.com<\/span><\/a> . Go practice this with it! <\/span><\/p>
\u00a0\u00a0<\/span><\/span><\/p>","_et_gb_content_width":"","footnotes":""},"categories":[67,171],"tags":[71,96,95,91,56,66,60],"_links":{"self":[{"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/posts\/1746"}],"collection":[{"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/comments?post=1746"}],"version-history":[{"count":15,"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/posts\/1746\/revisions"}],"predecessor-version":[{"id":2165,"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/posts\/1746\/revisions\/2165"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/media\/1796"}],"wp:attachment":[{"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/media?parent=1746"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/categories?post=1746"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zed.inguardians.com\/wp-json\/wp\/v2\/tags?post=1746"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}