Merb master/worker monit control setup

Posted by ezmobius Sun, 07 Dec 2008 19:55:00 GMT

Going into 1.0, Merb has changed the way it handles options and processes. This tends to break people’s existing monit configs. So today I’m releasing a custom Engine Yard monit script for controlling merb master/worker processes.

You can get the script from here: monit_merb_mpc. Put this somewhere in your $PATH, or make it live here: /engineyard/bin/monit_merb_mpc and make sure it is executable.

In order to make proper use of this script it assumes certain paths are setup. In this example we will assume you have an app called ‘iggy’ that lives at /data/iggy/current, where /data/iggy is the main app dir that contains the capistrano releases and current directory. The script assumes these paths so if you want top use different paths you will need to hand edit the script as desired.

This script also assumes a logging directory of /var/log/engineyard/$appname, you can edit the script if you want to change this as well.

You need to have one master configured per app, as well as monit configuration for each worker.

Master Config

check process merb_iggy_master
  with pidfile /var/log/engineyard/iggy/iggy-production-merb.main.pid
  start program = "/engineyard/bin/monit_merb_mpc iggy start_master -c3 -n5000" 
  stop program = "/engineyard/bin/monit_merb_mpc iggy stop_master" 
  #if totalmem is greater than 80.0 MB for 2 cycles then restart       # eating up memory?
  group merb_iggy

Note that the start program needs to be adjusted if you need more/less instances or want to run the workers on different ports /engineyard/bin/monit_merb_mpc iggy start_master -c -n . Also, memory monitoring is turned off for this process because the memory includes all processes under its control, including the workers and they are already monitored.

Worker Config (one for each port)

check process merb_iggy_5000
  with pidfile /var/log/engineyard/iggy/iggy-production-merb.5000.pid
  start program = "/engineyard/bin/monit_merb_mpc iggy register_worker 5000" 
  stop program = "/engineyard/bin/monit_merb_mpc iggy restart_worker 5000" 
  if totalmem is greater than 80.0 MB for 2 cycles then restart       # eating up memory?
  group merb_iggy

How it works

Since Merb 1.0 (actually version 0.9.8 and above) uses a master process and a spawner process in addition to the worker processes, the monit_merb_mpc (multi-process-control) script needs to control the master process separately from the worker processes. To accomplish this, there is a fair amount of trickery to make monit treat the merb processes in a way monit was never intended to be able to handle. When the master process is started, it will create all the worker processes, so when monit starts the worker processes, they are infact already started – so the start command for the worker simply “registers” the worker – in other words, it just ensures that the pid files contains the worker pid. Also since the master process will automatically restart the worker processes, the stop action for the workers is actually a restart.

Notes:

Although this looks like it should use the monit dependency directive, the scripts are actually written to not need it, and making the worker processes dependent on the master process, it will cause problems during shutdown. since the master specified the number of worker processes using the -c option, you can’t simply shutdown a single worker process by stopping it with monit – it will simply restart by itself and operate unmonitored, thus making it a possible runaway process.

Thanks to all the EY support guys who worked on this one to make it smooth. Hopefully this clears up how to deploy merb with the master/worker server cluster setup under monit.

7 comments