In this chapter we’ll be talking about the Chef community and community cookbooks. The Chef community site provides access to great Chef resources, including cookbooks, knife plug-ins, and the ability to connect to amazing people who create wonderful things built on Chef.
Before we get started, make sure you are running Chef Server and the node you created in Chapter 9. We’ll be making heavy use of the
knife
command line tool in this chapter, and it requires a Chef Server setup to function.Using Community Cookbooks
Although we’ve been writing all the cookbooks and recipes we’ve used so far in this book, so you can learn Chef coding, there is an easier way. There are hundreds of freely available prewritten Chef cookbooks to install and configure a variety of commonly used services and applications in production environments. For instance, there are cookbooks to help you set up Apache, Nginx, and IIS web servers and MySQL, PostgreSQL, Microsoft SQL Server, and Oracle databases. There are cookbooks to support the deployment of apps written in Java, Ruby, Python, PHP, node.js, and much, much more!
You can browse and download these community cookbooks from Chef Software’s Chef Supermarket community hub, and from a variety of other locations including GitHub. Some of these cookbooks were created and are maintained by Chef Software, but the majority are developed by Chef users. In general, as with any third-party software, you should always independently verify the behavior and flexibility of a community cookbook in an isolated environment before using it in production.
WHAT IS A COMMUNITY COOKBOOK?
A community cookbook is a tarball (
.tar.gz
) package of the cookbook structure discussed Chapter 7. It is packaged in this manner for easy cross-platform distribution. At its core, a community cookbook is exactly the same as an internally authored cookbook.
The best place to find free community cookbooks is the Chef Supermarket. This site is a Ruby on Rails application hosted and maintained by Chef Software where you can share, contribute, download, use, rate, and review community cookbooks. Chef Supermarket is akin to rubygems.org, cpan.org, and other focused distribution sites. You can log in to Chef Supermarket using the same credentials as your Hosted Enterprise Chef account. Once you have logged in, you will be able to comment on and follow cookbooks of interest and contribute your own cookbooks to the community.
NOTE
You do not need to log in to the Chef Marketplace to download or use its community cookbooks.
The Chef Supermarket lets you search for cookbooks by name or description as shown in Figure 10-1.
Chef Supermarket will query cookbook names, descriptions, platforms, categories, and other metadata. You can even sort results by most downloaded or most followed. Take time exploring to find the best cookbooks for you.
Chef-Client Cookbook
One community cookbook you should be aware of is the
chef-client
cookbook. Go ahead and search for it now at Chef Supermarket. Figure 10-2 shows the search results you should get for chef-client
.
The arrows in Figure 10-2 point out the most important components on the page when a cookbook is displayed in the Chef Supermarket. Front and center (well, actually, a little to the left) is a rendered version of theREADME.md file. Well-written README files, like the one in the
chef-client
cookbook, tell you what problem the cookbook is trying to solve, plus how to use the cookbook.
On the right are two big buttons, View Source and Download Cookbook. The View Source button will take you to the cookbook source code. (Usually it’s some link on the GitHub source hosting service.) From there you can inspect the cookbook more closely. Finally there is the Download Cookbook button, which lets you download a tarball containing the cookbook source.
The
chef-client
cookbook is a popular cookbook because it makes two things easy:- Configuring
chef-client
to run as a service or a cron job - Deleting the validation.pem file
Because one of the design goals for Chef Server is scalability, the server tries to offload as much processing as it can onto the nodes. So by default, the node is responsible for scheduling and initiating a
chef-client
run and performing all the related processing, not Chef Server. Chef Server itself is really just a dumb artifact repository for cookbooks and other associated metadata about your infrastructure.
When you bootstrap a node with
knife
in order to install chef-client
, as we did in Bootstrap the Node with Knife, the bootstrap process does not configure chef-client
to download any cookbook updates or perform Chef runs at regular intervals. You’ll definitely want to configure all your nodes to do this on a regular basis, say, every 15 to 30 minutes. The chef-client
cookbook makes it easy to configure chef-client
to run as a service or a cron job.
Also, it’s important to delete the validation.pem file after the first Chef run. With Enterprise Chef, this file is called <organization-validator>.pem by default. With Open Source Chef Server, the file is called validation.pemby default. To explain why deleting the validation.pem file is important, we need to provide a quick explanation of how requests by nodes are verified by Chef Server.
Chef Server requires that every request
chef-client
makes to the server be authenticated using a clientpublic/private key pair. Every node has its own special public/private key pair. You have already seen this because users have their own special public/private key pair as well—you needed to download the <username.pem> file to configure knife
to make requests against Chef Server. The <username>.pem file you downloaded contains the private portion of the key pair. The public portion of this key is stored on Chef Server, and the key is used to authenticate you as a valid Chef Server user.
Similarly, there is a .pem for each node that runs
chef-client
containing a private key. We’ll call this client.pem for the sake of discussion. Figure 10-3 presents an overview of how this key is used to verify that requests come from a node. In this example, Node A has a private key, which is a unique client.pem file that lives on the node. When the client.pem file was created, an associated public key was generated and stored on the Chef server. Node A signs all HTTP requests it makes to Chef Server with its private key. When Chef Server receives a request, it verifies that the signature is from Node A by using Node A’s public key to ensure it is a legitimate request from Node A.
When you run
chef-client
for the first time, there is a problem—you don’t have a client.pem file for your node yet, and a corresponding public key for the node does not exist on the Chef server. To solve this bootstrapping issue, a node uses a company-wide, well-known key when it generates the request to register the node as a client. That’s what the validation.pem key is for. The validation.pem is an organization-wide private key used specifically to sign the request to register a new node with Chef Server on the first chef-client
run.
Chef Server performs a validation of a signature using the validator.pem similar to the one it performs with the client.pem. During the bootstrap process, the validator.pem is created with the name /etc/chef/validation.pem on the node.
Although the /etc/chef/validation.pem is secured with root privileges, it’s a good idea to delete it once the node has a proper client key to run
chef-client
. Anyone who obtains the /etc/chef/validation.pem file can create new nodes. Once the node has a client key, it no longer needs the /etc/chef/validation.pem. It’s a good idea to leave the /etc/chef/validation.pem key on the node only during the time it actually needs to create a client public/private key pair for itself and send its client public key to Chef Server.
You can verify that the validation.pem file is still present on the node you bootstrapped in Chapter 9. Make sure the chef-repo/cookbooks/node directory is the current working directory by running one of the following commands. If the parent of your
chef-repo
tree is not $HOME, change the command to reflect the correct parent.
Linux/Mac OS X:
Windows Command Prompt:
Windows PowerShell:
Use
kitchen login
to ssh into the node, as follows. Check the contents of the directory with /etc/chef/validation.pem. Note that it is still there. Then make sure you exit back out to your host prompt:
We’ll talk more about the use of the
chef-client
recipe later, in Chef-Client Recipes. First, we need to talk about the knife cookbook site
plugin.Knife Cookbook Site Plugin
While the Chef Supermarket’s Download Cookbook link is very helpful, you still need to upload the cookbook source to your Chef Server in order to use it in production. Also, although a website is great for discovery, you’ll find yourself wanting to use a command-line tool for day-to-day community cookbook management because it is faster. All recent versions of both the Chef Client and the Chef Development Kit ship with a cookbook site plugin for knife designed to enable command-line interaction with the Chef Supermarket.
Let’s walk you through the most commonly use subcommands now. Feel free to type them in if you like. Make sure your current working directory is the chef-repo from Chapter 9. Assuming that the chef-repo is located in your home directory, run one of the following commands, depending on your platform. Change the root path to suit where you created chef-repo.
Linux/Mac OS X:
Windows Command Prompt:
Windows PowerShell:
Search for Community Cookbooks Using Knife Cookbook Site
You can use the
knife cookbook site search
command to search the Chef Supermarket for cookbooks. You will need to specify a search string as a parameter. knife cookbook site
will search the following fields in the cookbook metadata at Chef Supermarket:- name
- URL
- description
- maintainer
The
knife cookbook site
plugin will perform simple substring matching using your search string. Try it now with a chef-client
query:Manage Chef Supermarket Cookbooks on Your Chef Server Using Knife Cookbook Site
Although the search/show capabilities of
knife cookbook site
are helpful, you’ll use this plugin most often to download cookbooks from Chef Supermarket and upload community cookbooks to your local Chef Server. In order to download a cookbook from Chef Supermarket and upload it to your Chef Server, you must perform three steps:- Download the cookbook using
knife cookbook site download
. - Extract the cookbook package with tar.
- Upload the cookbook using
knife cookbook upload
. - Repeat steps 1-3 for any cookbook dependencies.
Let’s go through this process now for the
chef-client
cookbook. Download the chef-client
with knife cookbook site download
as follows:NOTE
I’ve specified the version in all these commands because cookbooks change frequently, even though you don’t have to specify a version to get the very latest cookbook.
The downloaded cookbook package is a
.tar.gz
file (a gzip-compressed .tar
file). You’ll need to extract it with the tar
command. Use the -C
option to make sure the cookbook is extracted as chef-repo/cookbooks/chef-client. You’ll want to store all downloaded cookbooks in chef-repo/cookbooks. Use the tar
command to extract the archive.
Linux/Mac OS X:
Windows:
NOTE
The
tar
program provided with Chef on Windows does not correctly expand filenames that include wildcards, such as chef-client*.tar.gz
. On Windows, the easiest workaround is to provide the full file name. Thankfully, all the Windows shell programs support tab-completion. So you need only type in the first few letters of chef-client-3.7.0.tar.gz
, then hit the Tab key and the shell should expand to use the full filename.
Unfortunately, if you try to upload the
chef-client
cookbook to your Chef Server right now, you’ll get an error that resembles the following:
If you recall when we introduced
include_recipe
in Include_Recipe cookbooks can contain a chain of references to other cookbooks. These references are called dependencies, and are noted in the metadata.rb of a cookbook using the depends statement. If you take a look at the metadata.rb file for the chef-client
cookbook, you’ll see that it resembles the following:
There are
depends
statements at the bottom of the metadata.rb file that state chef-client
is dependent on thecron
and logrotate
cookbooks. Exactly the two cookbooks mentioned in the error message! This is where you will need to repeat the knife cookbook site
> un-tar > knife cookbook upload
cycle for any cookbook dependencies.
So, download these two additional cookbooks with the
knife cookbook site
command, like so:
Extract them to chef-repo/cookbooks/ using the tar command, like you did for the
chef-client
cookbook.
Linux/Mac OS X:
Windows:
Chef-Client Recipes
Now let’s perform a Chef run adding two recipes to the run-list we touched on in Chef-Client Cookbook:
chef-client::default
recipe—Configureschef-client
to run as a servicechef-client::delete_validation
recipe—Deletes the /etc/chef/validation.pem file
Use
knife node run_list add
to add the chef-client::delete_validation
recipe to the node’s run list. For all knife
command lines, recipes are referenced in a run list in the form "recipe[<cookbook>::<recipe>]"
; for example, "recipe[chef-client::delete_validation]"
.
Run the following
knife node run_list add
command to add "recipe[chef-client::delete_validation]"
to the node-centos65.vagrantup.com
node’s run list that we bootstrapped in Chapter 9:NOTE
If you need a reminder of what the node name is, run the following command:
Also add the
chef-client::default
recipe to the run list. Note that you can use a shorthand notation using just the cookbook name, when you want to use the default
recipe. Run the following command now:
Both the
chef-client::delete_validation
and chef-client::default
recipes have been added to the node’s run list, and will be processed in the order provided.NOTE
You can add more than one recipe at a time to a run list—just separate the recipe names with commas:
Make sure the chef-repo/cookbooks/node directory is the current working directory, by running one of the following commands. If the parent of your chef-repo tree is not $HOME, change the command to reflect the correct parent.
Linux/Mac OS X:
Windows Command Prompt:
Windows PowerShell:
Perform an initial
chef-client
run by using kitchen login
to ssh into the node, and then run sudo chef-client
as follows:
While you are still on the node, verify that the validation.pem was deleted and that the
chef-client
is now running as a daemon:
Return back to the host prompt now:
Configure Knife to Use a Production SSL Setup
In Chapter 6, we mentioned the SSL warning you get running
chef-client
on the node when HTTPS connections are not validated. Before we end this chapter, let’s go over how you would configure SSL in a production environment.
On the node, SSL verification is controlled through settings in the file /etc/chef/client.rb, the file that configures
chef-client
. You can enable this setting by setting attributes in the chef-client
cookbook. Nearly everything in the chef-client
cookbook is controllable through attributes.
SSL setups can be quite complex to set up. So it is recommended that you first validate the setup with
knife
on your development workstation, before trying to configure your nodes to use verified SSL connections. Let’s do that now. We’ll use a simple setup that makes use of the self-signed certificate that was generated when you installed Chef Server in Chapter 9.
You will need to make sure that Chef Server is configured to use a certificate you intend to be used to verify communication over HTTPS. By default, Chef Server automatically generates a self-signed certificate during the installation. If you want to use the self-signed certificate, everything is already set up for you. In this exercise, we’ll use the self-signed certificate.
NOTE
Refer to the Chef documentation for more information on how to configure ChefServer with a certificate authority-verified certificate.
Also, a Chef community member, Mislav Marohnić, wrote an excellent blog post on troubleshooting Chef Server SSL issues and published a set of scripts to help troubleshoot connection issues.
Once the certificate is configured on Chef Server, run
knife ssl check
on your development workstation to find out what you need to do next. You should see output similar to the following:
The
knife ssl check
command says that you need to copy the certificate to your trusted_certs_dir
. Run the command knife ssl fetch
to automatically download the certificate and place it in your trusted_certs_dir
:
Run
knife ssl check
one final time, and it should verify successfully. If not, double-check to make sure the local hosts entry is correct, and that the hostname matched the expected name/IP when you ran knife ssl check
in the previous step:Configure Chef-Client to Use a Production SSL Setup
The
chef-client
cookbook includes a recipe chef-client::config
, which can be used to automatically generate the /etc/chef/client.rb config file with the SSL settings we need. In this way you can automate the configuration of SSL on your nodes.
Log in to the node and check the /etc/chef/client.rb file. The client.rb file was created to configure
chef-client
settings when you ran knife bootstrap
on the node. Notice that it resembles the following. Make sure you log back out to the exist prompt when you are done.
In order to enable verification of the SSL certificate on the server, we need to add the following setting to /etc/chef/client.rb (the default setting is
:verify_none
):
To enable this setting and have the
chef-client::config
recipe generate the appropriate configuration setting, we need to set the following attribute:
We’ll do this by using Chef Server’s management site. Once you log in, click on the Nodes tab, then click on the Edit link for
node-centos65.vagrantup.com
as shown in Figure 10-4.
The node editing page will be displayed. We’ll be using the node attributes portion of the page at the bottom, as shown in Figure 10-5.
Click on the source tab for attributes. Enter in the text as shown in Figure 10-6. This is the attribute setting
node.default['chef_client']['config']['ssl_verify_mode'] = ':verify_peer'
in JSON format. If you are reading this book in electronic format, feel free to copy and paste the following text into the json edit box:
Once you are done typing in the attribute value, click on the Load JSON from Source icon in the editing pane as shown in Figure 10-7. This will update the attribute setting in the
json
tree on the left. Then click on the Save Node button. If there is no syntax error, you should see this message: The node was updated successfully.NOTE
You can also edit node attributes on the command line with the
knife node edit
command.
On the command line, verify that the attribute was set back in your chef-repo directory. Use the
knife node show --attribute
command to display the chef_client.ssl_verify_mode
attribute for the node, as follows:
If the output does not match exactly, make sure that you remembered to click on the
Save Node
button. It is easy to forget because it’s at the bottom of the page outside the editing window. Also make sure that you remembered to click on the Load JSON from Source button, and make sure the json tree rendered on the left matches the figure exactly.
Because we are using a self-signed certificate, we need to set one more attribute setting. We need to tell the SSL library on the node that we trust the self-signed server certificate. In production, you’d do this by writing a recipe that adds the custom certificate to the certificate store. If you are using OpenSSL on your node, you will need to copy the certificate to the SSL_CERT_DIR, the directory where trusted certificates are stored, and run
c_rehash
to register the self-signed certificate.
Figure 10-7. Click on the icon to update the attribute setting in the json tree, then click on Save Node
In our test setup, we simulate this with the synchronized folder we set up for the node that we configured in Chapter 9. The synchronized folder makes the certificate we have in chef-repo/.chef/trusted_certs locally on our host available on the node. This directory was set up when you ran
knife ssl fetch
.
Go back to the Chef Server web interface, and add the
ssl_ca_file
attribute to the node. Copy and paste the following JSON source:
Double-check to make sure the settings for
chef_client.config.ssl_verify_mode
and chef_client.config.ssl_ca_file
are correct by checking them with knife node show --attribute
as follows. Make sure these settings match exactly, and the /chef-repo/.chef/trusted_certs/default-centos65_vagrantup_com.crt
is synced to your node before going further:
Once you have verified all the settings are correct, add the
chef-client::config
recipe to your node’s run list:
Then log in to the node and perform a Chef run. Because you added the
chef-client::config
recipe to your run list, Chef will make sure that the /etc/chef/client.rb matches the settings corresponding to the attributes you set on the node. You’ll get the SSL warning one more time as chef-client
hasn’t yet applied your SSL verification settings:
Now, if you run
chef-client
one more time, finally there is no more SSL warning:
Go ahead and
exit
back out to the host prompt now:
Your tour of the
chef-client
cookbook is now complete. We are now done with both our Chef Server and node. Go ahead and kitchen destroy
both of them.
Linux/Mac OS X:
Windows Command Prompt:
Windows PowerShell:
Summary
In this chapter we showed you the Chef Supermarket, which is the hub of the Chef community. We showed you how you can download cookbooks from the Chef Supermarket using both the web interface and the command line.
We recommend using the
knife
command line tool, because it makes it easy to not only download a community cookbook, but to upload and manage a cookbook on your Chef server so you can use it in your organization.
We presented an overview of the
chef-client
cookbook and how it is useful for configuring a node after it has been bootstrapped with chef-client
.
In the next chapter, we’ll introduce you to Chef Zero, a small, fast-start, in-memory version of Chef Server, great for local testing.
No comments:
Post a Comment