Django’s FastCGI init.d script for Linux

Update 15 Jan 2007: I have updated the post as I’ve found a better implementation of this script. Indeed, in my previous solution I was running the server as root. Now I’ve simplified the script, eliminated the wrapper and I’m running my servers as a different user (www-data in my case).

In the constant search for the perfect deployment, I’ve decided to have a look at the Django FastCGI option. The idea is to launch several FastCGI servers with Nginx as a proxy redirecting the requests.

The deployment proposed implies that “in most cases you’ll be starting the FastCGI process on your own”. And, of course, we will be stopping it on our own as well. For this they propose this script:

#!/bin/bash

# Replace these three settings.
PROJDIR="/home/user/myproject"
PIDFILE="$PROJDIR/mysite.pid"
SOCKET="$PROJDIR/mysite.sock"

cd $PROJDIR
if [ -f $PIDFILE ]; then
    kill `cat -- $PIDFILE`
    rm -f -- $PIDFILE
fi

exec /usr/bin/env - \
  PYTHONPATH="../python:.." \
  ./manage.py runfcgi socket=$SOCKET pidfile=$PIDFILE

 It’s a short script, and it works just well. But I was looking for a bit more flexibility. So I decided to develop my own init.d script.

The init.d script is called fastcgi,
an admittedly unimaginative name The main requirement is that it
should integrate without problems with the init.d scripts, and that it
should allow to transparently and uniformly start, stop and restart
FastCGI processes for several Django sites with minimum configuration.

#! /bin/sh
### BEGIN INIT INFO
# Provides:          FastCGI servers for Django
# Required-Start:    networking
# Required-Stop:     networking
# Default-Start:     2 3 4 5
# Default-Stop:      S 0 1 6
# Short-Description: Start FastCGI servers with Django.
# Description:       Django, in order to operate with FastCGI, must be started
#                    in a very specific way with manage.py. This must be done
#                    for each DJango web server that has to run.
### END INIT INFO
#
# Author:  Guillermo Fernandez Castellanos
#          .
#
# Version: @(#)fastcgi 0.1 11-Jan-2007 guillermo.fernandez.castellanos AT gmail.com
#

#### SERVER SPECIFIC CONFIGURATION
DJANGO_SITES="site1 site2 site3"
SITES_PATH=/path/to/sites
RUNFILES_PATH=$SITES_PATH/run
HOST=127.0.0.1
PORT_START=3000
RUN_AS=www-data
#### DO NOT CHANGE ANYTHING AFTER THIS LINE!

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="FastCGI servers"
NAME=$0
SCRIPTNAME=/etc/init.d/$NAME

#
#       Function that starts the daemon/service.
#
d_start()
{
    # Starting all Django FastCGI processes
    PORT=$PORT_START
    for SITE in $DJANGO_SITES
    do
        echo -n ", $SITE"
        if [ -f $RUNFILES_PATH/$SITE.pid ]; then
            echo -n " already running"
        else
            start-stop-daemon --start --quiet \
                       --pidfile $RUNFILES_PATH/$SITE.pid \
                       --chuid $RUN_AS --exec /usr/bin/env -- python \
                       $SITES_PATH/$SITE/manage.py runfcgi \
                       host=$HOST port=$PORT pidfile=$RUNFILES_PATH/$SITE.pid
            chmod 400 $RUNFILES_PATH/$SITE.pid
        fi
        let "PORT = $PORT + 1"
    done
}

#
#       Function that stops the daemon/service.
#
d_stop() {
    # Killing all Django FastCGI processes running
    for SITE in $DJANGO_SITES
    do
        echo -n ", $SITE"
        start-stop-daemon --stop --quiet --pidfile $RUNFILES_PATH/$SITE.pid \
                          || echo -n " not running"
        if [ -f $RUNFILES_PATH/$SITE.pid ]; then
           rm $RUNFILES_PATH/$SITE.pid
        fi
    done
}

ACTION="$1"
case "$ACTION" in
    start)
        echo -n "Starting $DESC: $NAME"
        d_start
        echo "."
        ;;

    stop)
        echo -n "Stopping $DESC: $NAME"
        d_stop
        echo "."
        ;;

    restart|force-reload)
        echo -n "Restarting $DESC: $NAME"
        d_stop
        sleep 1
        d_start
        echo "."
        ;;

    *)
        echo "Usage: $NAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac

exit 0

The variables of the script are:

• SITES_PATH: The folder where all the sites are stored, a site per folder. The folder structure i supposed to be a common folder ($SITES_PATH, in
  our case) where each site has a known folder ($SITES_PATH/www_guindilla_eu/,
  $SITES_PATH/www_haruki_eu/, $SITES_PATH/complu_haruki_eu/) where all configuration files
  are stored (manage.py, settings.py,…). More info about this here.
• DJANGO_SITES: The names of the folders where the different sites are.
• RUNFILES_PATH: The path to the folder where .pid and .socket files (if using Unix sockets) are stored.
• HOST: The IP address of the host.
• PORT_START: Each server will be running in a different port. The first server will run on PORT_START, the second on PORT_START+1, etc.
• RUN_AS: We want the FastCGI servers to run under a
  different user than root. RUN_AS is this user. You can put the username
  or the UID of the user.

In order to use, configure the initial variables, put the fastcgi file in the /etc/init.d/ directory and execute update-rc.d /etc/init.d/fastcgi defaults in a terminal. This way the daemon will be automagically be started at each system start-up.

The script is generally pretty easy to understand. Some sensitive parts are:

• #HOST=`/sbin/ifconfig | /bin/sed -n -e ‘s/\(.*\)inet addr:\(.*\)B.*$/\2/p’`
  In my script I used the IP address 127.0.0.1. Sometimes we simply want
  to use the IP of the server. This line finds this address automatically.
• d_start() start-stop-daemon can store the pid file of the daemon it starts in $RUNFILES_PATH. But this requires root access, and by using the option –chuid $RUN_AS we loose the ability to write root-only folders. So we delegate the pid file writting to manage.py. Another problem is, the stored pid would correspond to the FastCGI process while start-stop-daemon will consider the env pid. Thus, what will happen when doing two consecutive fastcgi start is
  that a new FastCGI process will not be spawned, but the pid file will
  be overwritten with a more recent pid, thus making any further fastcgi stop
  command break. The solution is to manually check for already existent
  process (by checking that their pid files exist) and launching the
  FastCGI processes otherwise.
• The d_start() function will increase the port used by one for each different server, starting from the $PORT_START onwards.
  
• d_stop() start-stop-daemon
  is used here, although there is no real need. The pid files are removed
  once the FastCGI process stopped in order to identify which sites are
  running.

This script uses TCP sockets for communication, but it is really simple to modify it in order to use Unix domain sockets:

• The sockets will be stored with the .pid files, and have the same name but with the.socket extension.
• Replate the argument runfcgi socket=$RUNFILES_PATH/$SITE.socket pidfile=$RUNFILES_PATH/$SITE.pid by runfcgi host=$HOST port=$PORT pidfile=$RUNFILES_PATH/$SITE.pid in all the script. Use this last value whenever you need to identify the socket.
• Remove the HOST and PORT variables from the fastcgi file.

And that’s it. Your script will work with Unix domain sockets.

Of course, the script is far from perfect:

• It is the first of the kind I do and my sh mastery is pretty low.
• It does not have a finely grained site management. All are treated alike.
• The port management is done a bit ad-hoc and could cause some configuration synchronization issues if not treated with care.
• The FastCGI server does not work under root, but still has some flaws.
  For example, the .pid files can be read by www-data and can not be
  written in /var/run/. This is due to the fact that Django’s manage.py
  does not seem to have an option to downscale to a lower-permission user
  as www-data.
• The script is known to be working in Debian and Ubuntu based systems, but will not work on Gentoo due to it’s particular init.d system. It should work at is in other Redhat based systems (Mandriva, Suse,…), although this has not been confirmed.

Still, I think it is a script that can be
useful and that will save me an important amount of time. And it’s like
a kid, no matter how ugly it is, it’s mine and I love it

Any feedback would be greatly appreciated!

 

**** OLD POST FOLLOWS FOR REFERENCE **** 

In the constant search for the perfect deployment, I’ve decided to have a look at the Django FastCGI option. The idea is to launch several FastCGI servers with Nginx as a proxy redirecting the requests.

The deployment proposed implies that “in most cases you’ll be starting
the FastCGI process on your own”. And, of course, we will be stopping it on our own as well. For this they propose this script:

#!/bin/bash

# Replace these three settings.
PROJDIR="/home/user/myproject"
PIDFILE="$PROJDIR/mysite.pid"
SOCKET="$PROJDIR/mysite.sock"

cd $PROJDIR
if [ -f $PIDFILE ]; then
    kill `cat -- $PIDFILE`
    rm -f -- $PIDFILE
fi

exec /usr/bin/env - \
  PYTHONPATH="../python:.." \
  ./manage.py runfcgi socket=$SOCKET pidfile=$PIDFILE

 It’s a short script, and it works just well. But I was looking for a bit more flexibility. So I decided to develop my own init.d script.

The first thing I did was to write a wrapper around the python manage.py command. I saved it in /www/run/django_fcgi :

 #! /bin/sh

[ $# = 4 ] || echo -n "Usage: $0 django_site ip port pid_file"

DJANGO_SITE=$1
IP=$2
PORT=$3
PID_FILE=$4

/usr/bin/env python $DJANGO_SITE/manage.py runfcgi \
                    host=$HOST port=$PORT pidfile=$PID_FILE

This script will make the use of the start-stop-daemon easier. Of course, this wrapper is not strictly necessary, as the command line can be directly executed in the init.d script, but this wrapper makes it more convenient.

Then comes the proper init.d script. I called it fastcgi, an admittedly unimaginative name The main requirement is that it should integrate without problems with the init.d scripts, and that it should allow to transparently and uniformly start, stop and restart FastCGI processes for several Django sites with minimum configuration.

#! /bin/sh
### BEGIN INIT INFO
# Provides:          FastCGI servers for Django
# Required-Start:    networking
# Required-Stop:     networking
# Default-Start:     2 3 4 5
# Default-Stop:      S 0 1 6
# Short-Description: Start FastCGI servers with Django.
# Description:       Django, in order to operate with FastCGI, must be started
#                    in a very specific way with manage.py. This must be done
#                    for each DJango web server that has to run.
### END INIT INFO
#
# Author:  Guillermo Fernandez Castellanos
#          .
#
# Version: @(#)fastcgi 0.1 11-Jan-2007 guillermo.fernandez.castellanos AT gmail.com
#

#### SERVER SPECIFIC CONFIGURATION
#HOST=`/sbin/ifconfig | /bin/sed -n -e 's/\(.*\)inet addr:\(.*\)B.*$/\2/p'`
DJANGO_SITES="complu_haruki_eu www_haruki_eu www_guindilla_eu"
SITES_PATH=/www
DAEMON=/www/run/django_fcgi
RUNFILES_PATH=$SITES_PATH/run
HOST=127.0.0.1
PORT_START=3000
#### DO NOT CHANGE ANYTHING AFTER THIS LINE!

set -e

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="FastCGI servers"
NAME=$0
SCRIPTNAME=/etc/init.d/$NAME

# Gracefully exit if the package has been removed.
test -x $DAEMON || exit 0

#
#       Function that starts the daemon/service.
#
d_start()
{
    # Starting all Django FastCGI processes
    PORT=$PORT_START
    for SITE in $DJANGO_SITES
    do
        echo -n ", $SITE"
        # As $DAEMON is a shell script, the process runs with the name "bash",
        # not "$DAEMON".
        # Thus I can not use the start-stop-daemon method here.
        # start-stop-daemon --start --quiet --pidfile $RUNFILES_PATH/$SITE.pid \
        # --exec $DAEMON -- $SITES_PATH/$SITE $HOST $PORT $RUNFILES_PATH/$SITE.pid \
        if [ -f $RUNFILES_PATH/$SITE.pid ]; then
            echo -n " already running"
        else
            $DAEMON $SITES_PATH/$SITE $HOST $PORT $RUNFILES_PATH/$SITE.pid
        fi
        let "PORT = $PORT + 1"
    done
}

#
#       Function that stops the daemon/service.
#
d_stop() {
    # Killing all Django FastCGI processes running
    for SITE in $DJANGO_SITES
    do
        echo -n ", $SITE"
        start-stop-daemon --stop --quiet --pidfile $RUNFILES_PATH/$SITE.pid \
                          || echo -n " not running"
        if [ -f $RUNFILES_PATH/$SITE.pid ]; then
           rm $RUNFILES_PATH/$SITE.pid
        fi
    done
}

ACTION="$1"
case "$ACTION" in
    start)
        echo -n "Starting $DESC: $NAME"
        d_start
        echo "."
        ;;

    stop)
        echo -n "Stopping $DESC: $NAME"
        d_stop
        echo "."
        ;;

    restart|force-reload)
        echo -n "Restarting $DESC: $NAME"
        d_stop
        sleep 1
        d_start
        echo "."
        ;;

    *)
        echo "Usage: $NAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac

exit 0

The variables of the script are:

• SITES_PATH: The folder where all the sites are stored, a site per folder.
• DJANGO_SITES: The names of the folders where the different sites are.
• DAEMON: The path to our wrapper.
• RUNFILES_PATH: The path to the folder where .pid and .socket files (if using Unix sockets) are stored.
• HOST: The IP address of the host.
• PORT_START: Each server will be running in a different port. The first server will run on PORT_START, the second on PORT_START+1, etc.

In order to use, configure the initial variables, put the fastcgi file in the /etc/init.d/ directory and execute update-rc.d /etc/init.d/fastcgi defaults in a terminal. This way the daemon will be automagically be started at each system start-up.

The script is generally pretty easy to understand. Some sensitive parts are:

• #HOST=`/sbin/ifconfig | /bin/sed -n -e ‘s/\(.*\)inet addr:\(.*\)B.*$/\2/p’` In my script I used the IP address 127.0.0.1. Sometimes we simply want to use the IP of the server. This line finds this address automatically.
• d_start() start-stop-daemon stores the pid file of the daemon it starts in $RUNFILES_PATH. The problem is, this stored pid correspond to the FastCGI process while start-stop-daemon will consider the bash wrapper pid. Thus, what will happen when doing two consecutive fastcgi start is that a new FastCGI process will not be spawned, but the pid file will be overwritten with a more
  recent pid, thus making any further fastcgi stop command break. The solution is to manually check for already existent process (by checking that their pid files exist) and launching the FastCGI processes otherwise.
• The d_start() function will increase the port used by one for each different server, starting from the $PORT_START onwards.
  
• d_stop() start-stop-daemon is used here, although there is no real need. The pid files are removed once the FastCGI process stopped in order to identify which sites are running.

This script uses TCP sockets for communication, but it is really simple to modify it in order to use Unix domain sockets:

• The sockets will be stored with the .pid files, and have the same name but with the.socket extension.
• Modify the wrapper so it uses: /usr/bin/env python $DJANGO_SITE/manage.py runfcgi socket=$SOCKET pidfile=$PIDFILE
• Replace the argument $HOST $PORT by $RUNFILES_PATH/$SITE.socket in all the script. Use this last value whenever you need to identify the socket.
• Remove the HOST and PORT variables from the fastcgi file.

And that’s it. Your script will work with Unix domain sockets.

Of course, the script is far from perfect:

• It is the first of the kind I do and my sh mastery is pretty low.
• It does not have a finely grained site management. All are treated alike.
• The port management is done a bit ad-hoc and could cause some configuration synchronization issues if not treated with care.
• The script is known to be working in Debian and Ubuntu based systems, but will not work on Gentoo. I could not test it in other systems (RH, Fedora,…).

Still, I think it is a script that can be useful and that will save me an important amount of time. And it’s like a kid, no matter how ugly it is, it’s mine and I love it

Any feedback would be greatly appreciated!

Tags: design, django, linux