A collection of task oriented solutions in Puppet

 

Vary cronjob start times

Challenge

You want to stop all your cronjobs running at the same time.

Solution

class staggered_cron {

  $minute = fqdn_rand(59)

  notify { "I got ${minute}": }

  cron { 'freshen_cache':
    command => '/usr/local/sbin/freshen_cache',
    user    => 'root',
    minute  => $minute, # this value will vary per host
  }

}

Explanation

Once you start using puppet to ensure your cronjobs are deployed across your hosts you will often want to vary the time the cronjobs start, to avoid placing excessive load on other resources. A good use case for this functionality is when running a yum update or similar expensive command across all your machines. If you hard code the jobs timing, as shown in the first resource below, every machine will contact the remote host at the same time and drive its load up and the performance down. Instead, dynamically set the minute using the fqdn_rand function, shown in the second resource, and the exact start time will be slightly different across the hosts, reducing the pressure.

class good_vs_bad_cron {

  # Hard coded. Every host runs at this exact time
  cron { 'freshen_cache':
    command => '/usr/local/sbin/freshen_cache',
    user    => 'root',
    minute  => 23,
  }

  # Much better - dynamic but consistent value
  $minute = fqdn_rand(59)

  # start time will vary across your hosts
  cron { 'freshen_cache':
    command => '/usr/local/sbin/freshen_cache',
    user    => 'root',
    minute  => $minute,
  }

}

The fqdn_rand function uses the fqdn fact to create a random but repeatable return value. Each node will get a different random number when calling it, but the value will be the same on every run if the host name hasn't changed. This allows puppet to set a value that is dynamic per host but consistent between runs on that host. We can illustrate this by running the same manifest a number of times on the same machine:

class minute_notify {

  $minute = fqdn_rand(59)

  notify { "I got ${minute} with fqdn ${fqdn}": }

}

include minute_notify

for ((i = 0 ; i < 5; i++))
do
  puppet apply -v staggered_cron.pp 2>/dev/null | grep Staggered_cron
  sleep 1
done

Notice: /Notify[I got 41 with fqdn puptest]/snip/
Notice: /Notify[I got 41 with fqdn puptest]/snip/
Notice: /Notify[I got 41 with fqdn puptest]/snip/
Notice: /Notify[I got 41 with fqdn puptest]/snip/
Notice: /Notify[I got 41 with fqdn puptest]/snip/

You can see in the output the number does not change between runs. To verify the number does change between hosts we can override the fqdn fact and re-run our test using the same puppet code as before.

for ((i = 0 ; i < 5; i++))
do
  # FACTER_fqdn overrides the facter value
  FACTER_fqdn=puppet-cookbook puppet apply -v staggered_cron.pp \
    2>/dev/null | grep Staggered_cron
  sleep 1
done

Notice: /Notify[I got 33 with fqdn puppet-cookbook]/snip/
Notice: /Notify[I got 33 with fqdn puppet-cookbook]/snip/
Notice: /Notify[I got 33 with fqdn puppet-cookbook]/snip/
Notice: /Notify[I got 33 with fqdn puppet-cookbook]/snip/
Notice: /Notify[I got 33 with fqdn puppet-cookbook]/snip/

The fqdn fact has changed so the number is different, but still consistent between runs. This is the core of how all our crons continue to run on a consistent, but varied, schedule. If you have cronjobs on a single host that you need to stagger you can customise the fqdn_rand call further and specify a specific name (technically a seed) to use when generating the number. This allows you to vary the numbers on a single host in a consistent way.

class single_host_stagger {

  $batch  = fqdn_rand(59, 'Data batch')
  $repo   = fqdn_rand(59, 'Freshen repos')
  $backup = fqdn_rand(59, 'Backups')

  notify { "Minutes: ${batch}, ${repo} and ${backup}": }

}

# Notice: Minutes: 41, 0 and 20

See also