Continuous Deployment with Rundeck and Chef
Here at Shutl HQ we are big fans of Chef, using it to provision our boxes from bare OS up to running application. Chef handles all of our application deployments, which happen many times a day across all of our environments. However, while Chef is excellent at managing the configuration of each machine, it is lacking when it comes to orchestration of deployments across clusters of machines.
This post is about how we addressed our release orchestration headaches with Rundeck, and built a fully automated Continuous Delivery pipeline using Rundeck to integrate our CI platform, releases, and Chef together.
We use this to deploy Ruby applications in the main, with some JavaScript, Erlang and Clojure, but the solution presented here will work equally well for any application stack where the deployment is done by Chef.
Our release flow
Applications are deployed from a Git repo. Chef is configured to checkout the code, run any necessary build steps and database migrations before deploying the new code.
In staging, we always deploy master from Git. When code is pushed to master, a build is triggered in Semaphore, our hosted CI platform. When the build passes, we can log into the staging server and run the Chef client. This will check out and deploy master. This is the simplest workflow case, but there is a problem. Someone needs to watch for the build to turn green and then log into each staging machine in turn and run the deployment.
Enter Rundeck
Rundeck is an open source tool for running jobs on remote machines. It can run predefined or ad-hoc jobs, and it can run many jobs concurrently. It is Java based, and can be easily extended with plugins to perform tasks such as posting a notification in your Slack channel when a job completes. It has a very nice Web UI as well as a fully featured API.
Rundeck executes remote tasks via ssh, as the “rundeck” user by default. So no special agent needs to be running on each machine, you simply need to ensure that the rundeck user has the required permissions to perform the task. Predefined jobs can be configured with options which are passed from the user at run time. This allows jobs to be quite flexible in the tasks they perform. Each job consists of a sequence of steps which can also include other Rundeck jobs.
How we use it
We created a job called “Deploy Application”, which takes three options:
- Name of the application to deploy
- Environment to deploy in
- Version of the app to deploy
The revision is optional, and does not need to be supplied for staging deployments, where we always deploy master. Rundeck uses the application and environment options to find the nodes (machines) to deploy to. In order to find the nodes and tie Rundeck into Chef, we use rundeck-chef, a simple Sinatra app which talks to the Chef API and presents the list of nodes to Rundeck. The Chef environment is visible on each node, and each Chef role becomes a tag on the nodes in Rundeck. We have a role for each application, which allows Rundeck to find the target nodes by searching for the tag which matches the application name.
When it has found the nodes, Rundeck simply executes “sudo chef-client” on them to perform the deployment.
Continuous Delivery (in staging)
In our staging environment, we take this a step further and do fully automated deployments. When the build passes in the CI, a simple script is run which uses the Rundeck API to trigger the deployment:
#!/bin/bash APP=admin_portal ENVIRONMENT=staging JOB_ID=02abce51-a001-46fc-9c4f-080a12e18cb1 TOKEN=XXXX curl --data-urlencode "argString=-application $APP -environment $ENVIRONMENT" "https://RUNDECK_HOST/api/10/job/$JOB_ID/run?authtoken=$TOKEN"
This allows us to go from code push to having the code running in staging. To let us know when deployments happen, we wrote a simple Slack notifier plugin for Rundeck which posts a message in a Slack channel.
Production Deployments
In production, we trigger the Rundeck job manually. However, things are slightly more complicated because we are deploying a tagged release from Git. This means we need a way to pass the tag from Rundeck to Chef.
To solve this, we wrote a simple Sinatra app which we call Deployment Service. This has a simple API which can be used to store and retrieve the tag for each application, in each environment. We then wrote a java plugin for Rundeck to post the tag to the Deployment Service. When chef-client runs on the target node, it calls the Deployment Service API to get the tag to deploy.
Bonus Points – run deployments from the command line
Although Rundeck has a very nice Web UI, sometimes a command line tool is more convenient. So we wrote a knife plugin to trigger the Rundeck job (knife is a command-line client for Chef). This has the advantage of being able to trigger concurrent deployments all our production-like environments with one command:
$ bundle exec knife roller deploy -A admin_portal -E production -T 2.98.14 Environment: production Running job 4553 - https://rundeck.shutl.co.uk/execution/follow/4553 Environment: production-us-west Running job 4554 - https://rundeck.shutl.co.uk/execution/follow/4554 Environment: sandbox-us-west Running job 4555 - https://rundeck.shutl.co.uk/execution/follow/4555
This uses an HTTP call to the Rundeck API, just like the CI deploy script mentioned above but via Ruby.
(CC Image by Rubin)
6 Comments
Hi, thanks for this but I’d like to know how you can pass a required parameter to the job itself using cURL? Here’s what I’ve got. I can authenticate with UID/PW
$ curl -X POST -d j_username=xx -d j_password=xx http://rd.local/j_security_check -c rundeck.out
Then I use “rundeck.out” as the session cookie but I can’t figure out how to pass my required option. I’ve tried all of these:
$ curl http://rd.local/api/1/job/12345/run?OPT=y -b rundeck.out
$ curl http://rd.local/api/1/job/12345/run -OPT=’y’ -b rundeck.out
$ curl -d -OPT y http://rd.local/api/1/job/12345/run -b rundeck.out
$ curl -d -OPT=’y’ http://rd.local/api/1/job/12345/run -b rundeck.out
But they all return the XML below (parsed for simplicity)
result error='true' apiversion='12'
error code='api.error.job.options-invalid'
message = Job options were not valid: Option 'OPT' is required.
Can you please help me?
Hi EV, yes it’s not very clear from the rundeck docs but you should try this:
$ curl –data-urlencode “argString=-OPT y” http://rd.local/api/1/job/12345/run -b rundeck.out
However, I would recommend you use token authentication rather than username/password. You can set a global token in /etc/rundeck/tokens.properties or you can generate a token for your user from the user profile – see http://rundeck.org/docs/api/index.html#token-authentication
Once you have a token you can do something like this:
TOKEN=yourtoken
$ curl –data-urlencode “argString=-OPT y” http://rd.local/api/1/job/12345/run?authtoken=$TOKEN”
Let me know how that works for you.
James
Hi, How can I check whether the job has ran successfully at the node server. Though the rundeck console showed them succeeded I would like to verify that manually at the node. Is there a way to figure that out. I tried checking the logs but it wasn’t present anywhere 🙁
Hi Sujith, that really depends what your job does. If you’re running chef-client then you could always configure it to log to a file in your client.rb. But we just have it print to stdout – you can always see the output in Rundeck by viewing the job and going to the “log output” tab.
If you trigger a build when a current deploy is already running, it fails. Not sure how to get around this, how to make it ‘wait’ for the previous to finish and then build.
Hi ,
I would like to seek your help regarding passing a parameter or option from Jenkins job to its associated RunDeck job. How can I achieve this?
Actually I would like to skip restart of JBoss Application Server in the script at RunDeck side for certain deployments, here in this case I would like to pass some boolean parameter (say isRestartNeeded=yes/no) , by utilizing this parameter in RunDeck job i.e., inside script, I need to skip JBoss restart or opt for choice based upon context(if restart required yes, not required no like that). How can achieve this?
Please help me here.