• Author:
  • Published: Dec 31st, 2006
  • Comments: 2

Deployement of Nginx reverse proxy in my network

nginxWhat a better day to improve a network that the last day of the year? Indeed, it’s what I’ve done with mine. Today, I’ve installed a reverse-load-balancer proxy in my network, Nginx.

I was feeling the need for a reverse proxy since the beginning. Once the VServer installed and with the freedom to create as many machines as wished, it seemed logical to have each site’s Apache instance running in a dedicated vserver. This would not only divide each server, thus increasing the security, but would give the owner of each site to play with the server as he sees fit without the risk of breaking someone else’s site.

The implementation on the other hand was a bit problematic. I run behind a router doing NAT, and thus I had to choose a single server as my solely web server. When I decided to create an HTTP driven svn server in a different machine, I had to choose another port (8000) to avoid conflicts. Thus, very fast I saw the interest of installing a reverse proxy.

Due to pure laziness, I’ve spend quite a lot of time documenting and looking for different alternatives. I wanted as much as possible a consistent network with a minimum of software to maintain. As I wanted to substitute in the long term my heavy Django mod-python Apache servers with something lighter, I looked for a load-balancer that could also serve static files and, why not?, FastCGI.

There is several options in the market:

Squid and Apache were discarded because too heavy. I did not want another performance hole in my already strained server.

Pound does not serve any content. Furthermore, it uses important quantities of RAM and has limited documentation.

Perlbal does seem to lack documentation, as does Varnish, another otherwise nice-looking piece of software. Maybe in the future…

Finally, Lighttpd is known for it’s memory leaking, important number of bugs and spotty documentation.

Update: I’ve discovered HAProxy, Pen and Balance so I’ve added them. Pen and Balance seem to be simple TCP proxies, and thus not exactly what I’m looking for. I added them for completeness. HAProxy looks nice though.

I’ve decided to use Nginx. It has all functionalities I was looking for, has a very good reputation of stability, speed and scalability, relatively good documentation and I was feeling adventurous. As I would see later, they also have a very responsive mailing list.

So… Nginx. You can either download the source and compile it or download a package for Debian, Ubuntu or an rpm.

I went the lazy way and used the Ubuntu package. This implies I don’t have the latest version, but that it will work. And indeed it works! I had to solve a couple of dependencies and it was ready to go.

The first thing I did was to configure it to redirect all incoming requests into the correct web server.

All the configuration files are located in /etc/nginx/. Among many files, the main one is (oh, surprise!) nginx.conf. This one holds all the generic (and specific if desired) configuration for web servers and proxies. My file looks like this:

user  www-data;
worker_processes  2; # 5 is the default
error_log  /var/log/nginx/error.log warn;

# pid of nginx master process
pid        /var/run/nginx.pid;

events {
    worker_connections   1024;
       }

http {
    #pull in mime-types. You can break out your config 
    # into as many include's as you want to make it cleaner
    include       /etc/nginx/mime.types;

    # set a default type for the rare situation that
    # nothing matches from the mimie-type include
    default_type  application/octet-stream;

    # configure log format
    log_format main      '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
    log_format combined  '$remote_addr - - [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_host" $request_time "$http_x_forwarded_for" "$http_via" "$gzip_ratio"'; 
    access_log  /var/log/nginx/access.log combined;

    # These are good default values.
    keepalive_timeout              65;
    tcp_nodelay                    on;
    tcp_nopush                     on;
    sendfile                       on;
    server_names_hash_bucket_size  128; # this seems to be required for vhosts

    include         /etc/nginx/mysites/www.guindilla.eu.conf;
    include         /etc/nginx/mysites/svn.guindilla.eu.conf;
    include         /etc/nginx/mysites/www.haruki.eu.conf;
    include         /etc/nginx/mysites/complu.haruki.eu.conf;

     }

Those files are stored in /etc/nginx/mysites/. Each one of the files look very alike:

    server {
        listen           80;
        server_name      www.guindilla.eu;
        # vhost specific logs
        access_log       /var/log/nginx/www.guindilla.eu.access.log combined;

        location / {
            proxy_pass   http://192.168.0.3;
            include      /etc/nginx/proxy.conf;
                   }
           }

I’m simply asking that every request that arrives to my server be redirected to my web server, 192.168.0.3. If I was using a different port, it would have been as easy as adding it at the end: proxy_pass http://192.168.0.3:81;

The file proxy.conf holds my generic pr
ox configuration:

proxy_set_header        Host             $host;
proxy_set_header        X-Real-IP        $remote_addr;
proxy_set_header        X-Forwarded-For  $proxy_add_x_forwarded_for; 

The proxy configuration works, and redirects transparently to the server. But there is still one problem, the logs. Indeed, the web server’s logs will only log 192.168.0.2, the proxy IP address. How to solve that?

The reverse proxies inject a variable in the HTTP headers in order to keep track of the IP of the clients that started the request. This is done in the X-Forwarded-For or X-Real-IP fields. What we want is to use this variable instead of the proxi’s IP.

There is two solutions (thanks Igor and Alexandar):

  • We can use mod_rpaf with Apache. But I find it a bit overkill.
  • We can rewrite the log definition by changing %h with %{X-Real-IP}i or with %{X-Forwarded-For}i in the logs.

I chosed the second solution, by adding in apache2.conf:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" guille

and in each site definition:

CustomLog /var/log/apache2/www.guindilla.eu.log guille

And now… it works. Everything just works. I love easy jobs!

Tags: , ,

2 Responses to “Deployement of Nginx reverse proxy in my network”


  1. iiome
    on Feb 10th, 2007
    @ 02:12

    apache mod_proxy just works for me.

    Lighttpd would have been my second choice; i’m planning to shift some services to it. Where did you read about those memory leaks?

    so did you do any load balancing?


  2. Guille
    on Feb 12th, 2007
    @ 13:09

    I have to admit that I haven’t tried lighty, as I decided to try Nginx directly because of the different reviews. And I must admit that I was more inclined to lighty that some weird Russian tech, as I like the idea of an all-python setup. I know, I know, it’s childish, but you must support the community! :-)

    Still, I kept finding reports about memory leaks everywhere. For instance:
    http://www.ruby-forum.com/topic/96361
    http://hostingfu.com/article/nginx-vs-lighttpd-for-a-small-vps
    http://bob.pythonmac.org/archives/2006/08/04/reverse-proxy-roundup/#comment-3897
    http://forum.lighttpd.net/topic/4802
    but a quick search in Google will show more results.

    Of course, it not be a major problem for some people but for me it made me try Nginx despite the initial problems all young projects face.

    Concerning the load balancing, I did. All my http requests go through a load balancer and are… well… balanced :-)

    Now, I want to try FastCGI with a light process and move each web server to an independent vserver for each. But this will be for later :-)

Leave a Reply

© 2006,2007,2008,2009,2010 Guillermo Fernández Castellanos | Header images by Nick Lobeck