"Guest" entry by Stéphan, cross-posted from his own blog "Don't make the same mistake twice". We are glad to have Stéphan onboard, and have him share some of the work he is doing here at 8th color - Martin

After a few weeks of devops 'light' practice, I thought I should share my findings. The presentation of Nathen Harvey pushed me a bit further to share them. I will assume you just read deploying rails or a similar book. Started to play with vagrant/puppet/capistrano, and discovered that the real world is not like the sample in the book.

I'd like to live in theory
In theory everything works fine.


Create a single file for your project to describe the type of machine you want, the software that needs to be installed, and the way you want to access the machine (ssh, port forwarding, shared drive, ...). Store this file with your project code.

Run a single command — "vagrant up" — and sit back as Vagrant puts together your complete development environment. Say goodbye to the "works on my machine" excuse as Vagrant creates identical development environments for everyone on your team.

If you are not using it yet ... you should ! Developed by Mitchell Hashimoto, Vagrant is your new IDE. You won't test on beta, pre-production, ... simply on your workstation. With vagrant, you messed up your manifests ? No problem vagrant destroy && vagrant up, you are back on road.

Basebox with veewee : because time is money

Don't waste your time with vagrant sample boxes... often not up to date, not sure about the content, older puppet version, obsolete Virtualbox Guest Additions... Build your own with veewee. 30 minutes download included, just too good to be true. Thanks to Patrick Debois. Don't forget to adapt the mirror in the preseed.cfg file of your vm configuration.

In my case, I had to tweak a bit veewee to fix dns resolution conflicts with my ubuntu laptop :

  • adapt the network settings in lib/veewee/provider/virtualbox/box/helper/create.rb
  • add at the end of the bootorder line : --natdnsproxy1 on --natdnshostresolver1 on

Reuse : be lazy and clever

Reusing existing modules is always a good idea, people smarter than you (or at least me) already worked on the installation process for most open source components

git submodule add https://github.com/puppetlabs/puppet-postgresql.git modules/postgresql
git submodule add https://github.com/puppetlabs/puppetlabs-stdlib.git modules/stdlib
git submodule add https://github.com/blt04/puppet-rvm.git modules/rvm
git submodule add https://github.com/rodjek/puppet-logrotate modules/logrotate
git submodule add https://github.com/blt/puppet-ssh.git modules/ssh

Another (better) option is to use librarian-puppet to avoid the hassle of managing git submodules (specially rm). See Carlos Sanchez post for the benefits.

If you build, build to be maintainable

If you don't find an existing module in the forge don't hesitate to create one or improve an existing one. But don't do it blindly, some patterns are emerging amongst puppeteers. A good layout of your modules, that allows parametrization/separation of concerns/optional concerns like monitoring, firewall, ... is documented in the Example42 foo module. Quality matters, puppeteers push the adoption of rspec for unit testing modules specially if you plan to support multiple distributions.

You can also check your manifests with puppet-lint and see if they conforms to style guide :

sudo gem install puppet-lint
cd /tmp/vagrant-puppet/
puppet-lint --with-filename .

To fix some issues like tabs, trailing whitespaces, All strings that do not contain variables or escape characters like \n or \t should be enclosed in single quotes see some of the bash scripts from bashrc_puppet. Note that the next version of puppet-lint can fix some of them with --fix option. The next step is to enforce puppet-lint reviews with your continuous build : a jenkins-ci sample. During fosdem2013, Bryan Berry demonstrated how to use test-kitchen for multi-node integration tests. The demo is for chef but possible to port to puppet ;)

Performance Vagrant and puppet

If you are using vagrant and provisioning/destroying/reprovisioning/... you don't want to waste your time so we can speed up the installation by sharing a folder for your package manager, here's an example for debian

For puppet, adding --evaltrace will enable performance logs :

Info: /Stage[main]/Ruby/Rvm_system_ruby[ruby-1.9.3-p362]: Evaluated in 261.84 seconds
Info: /Stage[main]/Ruby/Rvm_gem[ruby-1.9.3-p362@rails3.2.11/passenger]: Evaluated in 18.14 seconds
Info: /Stage[main]/Rvm::Passenger::Apache::Ubuntu::Post/Exec[passenger-install-apache2-module]: Evaluated in 43.45 seconds
Info: /Stage[main]/Ruby/Rvm_gem[ruby-1.9.3-p362@rails3.2.11/rails]: Evaluated in 89.32 seconds

You can parse them with a script similar to puppet-profiler. Again a Tim Sharpe (rodjek) project, thanks Tim for puppet-lint and puppet-profiler.

In our case we can still spare a few minutes if we can speed up the rvm/ruby installation. This can be done by skipping the ruby compilation with prebuilt binaries and a corporate repository.

Production environment

You perhaps don't have the need (or the infrastructure) to install puppet-master, puppet-agent, puppet-dashboard, ... so you can start with a master-less setup. Once you have received a new server (from your hosting provider), you can bootstrap the installation of puppet via capistrano and let librarian-puppet take care of the puppet manifest/modules and apply.

Debugging tricks

And finally some debugging tricks that can help you diagnose some issues.

Adding debugging notice

notice("Installing ruby : ${ruby_rails_version} ")

Re-applying puppet manifests without vagrant reload or up/destroy

vagrant ssh
sudo su
cd /tmp/vagrant-puppet/manifests && puppet apply --pluginsync --verbose --modulepath '/tmp/vagrant-puppet/modules-0' /tmp/vagrant-puppet/manifests/default.pp

Debugging cycle or missing dependency

As you probably know, the order isn't guaranteed (if you forgot a dependency) or you accidentally creates cycles in the puppet catalog. It's possible to graph these dependencies to ease the debugging.

Add the graph options --graph and --graphdir '/tmp/vagrant-puppet/modules-0/graphs'

root@host:/tmp/vagrant-puppet/manifests# cd /tmp/vagrant-puppet/manifests && puppet apply --graph --graphdir '/tmp/vagrant-puppet/modules-0/graphs' --pluginsync --verbose --modulepath '/tmp/vagrant-puppet/modules-0' /tmp/vagrant-puppet/manifests/default.pp


  • download Gephi
  • launch ./bin/gephi
  • open the expanded_relationship.dot
  • layout, navigate, filter, ... and finally discover the cycle or the missing dependency
Enhanced by Zemanta