Posted on by Anh

Deploying Django with NGINX and uWSGI

First I want to say that there are many ways of achieving this goal but this is the way I set up. So I do not guarantee that this will work for you!

Concept

A web server faces the outside world. It can serve files (HTML, images, CSS, etc) directly from the file system. However, it can’t talk directly to Django applications; it needs something that will run the application, feed it requests from web clients (such as browsers) and return responses.

A Web Server Gateway Interface (WSGI) does this job. WSGI is a Python standard.

uWSGI is a WSGI implementation. In this tutorial we will set up uWSGI so that it creates a Unix socket, and serves responses to the web server via the WSGI protocol. At the end, our complete stack of components will look like this:

web_client <=> web_server <=> socket <=> uwsgi <=> Django

Install pip

pip will make our job easier later on when we want to add other python packages. pip requires setuptools. Run following commands to install them.

yum update
yum install python-setuptools
curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python get-pip.py

Install Django

Once pip and setuptools are installed, run the following to install Django and create a new project like so:

pip install django
mkdir -p /var/www/django
cd /var/www/django
django-admin.py startproject <new-project>

Where new-project is the name of your project.

To find out where all the Django source files located:

# python -c "import sys; sys.path = sys.path[1:]; import django; print(django.__path__)"
['/usr/lib/python2.6/site-packages/django']

Check which version is installed:

# python -c "import django; print(django.get_version())"
1.5.2

Install South

When deploying code to an application server, it will often be necessary to perform database migrations. In this case, South is the perfect tool for the job.

# easy_install South 2>&1 > install.south.log

The “2>&1 > install.south.log” is to redirect all output stderr and stdout, just for record purposes. You can use the same idea for above installations.

If you’ve already got an old version of South, and want to upgrade, use:

# easy_install -U South

Configuring Django installation

Now you’ve installed South system-wide, you’ll need to configure Django to use it. Doing so is simple; just edit your settings.py and add ‘south‘ to the end of INSTALLED_APPS.

Once South is added, you’ll need to sync the database to make the South migration-tracking tables (South doesn’t use migrations for its own models, for various reasons).

python manage.py syncdb

Example:

# python manage.py syncdb
Syncing...
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table south_migrationhistory
 
You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): 
Email address: [email protected]
Password: 
Password (again): 
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
 
Synced:
 > django.contrib.auth
 > django.contrib.contenttypes
 > django.contrib.sessions
 > django.contrib.sites
 > django.contrib.messages
 > django.contrib.staticfiles
 > south
 
Not synced (use migrations):
 - 
(use ./manage.py migrate to migrate these)

To learn more about South check out the following tutorials:
http://south.readthedocs.org/en/latest/tutorial/part1.html
http://south.aeracode.org/wiki/Tutorial1

Install django-enumfield

django-enumfield provides an enumeration Django model field (using IntegerField) with reusable enums and transition validation. More info: https://github.com/5monkeys/django-enumfield

$ sudo pip install django-enumfield
Password:
Downloading/unpacking django-enumfield
  Downloading django-enumfield-1.0c1.tar.gz
  Running setup.py egg_info for package django-enumfield
 
Installing collected packages: django-enumfield
  Running setup.py install for django-enumfield
 
Successfully installed django-enumfield
Cleaning up...

Install uWSGI

Before we install uWSGI we need to first install some dependencies:

yum install python-devel libxml2-devel zlib-devel openssl-devel pcre-devel gcc autoconf automake
pip install uwsgi

Once uwsgi successfully installed, you will see the following at the end.

    ################# uWSGI configuration #################
 
    pcre = True
    kernel = Linux
    malloc = libc
    execinfo = False
    ifaddrs = True
    ssl = True
    matheval = False
    zlib = True
    locking = pthread_mutex
    plugin_dir = .
    timer = timerfd
    yaml = True
    json = False
    filemonitor = inotify
    routing = True
    debug = False
    zeromq = False
    capabilities = False
    xml = libxml2
    event = epoll
 
    ############## end of uWSGI configuration #############
    *** uWSGI is ready, launch it with /usr/bin/uwsgi ***
Successfully installed uwsgi

Basic test

Create a file called test.py:

def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return "Hello World"

Run uWSGI:

uwsgi --http :8000 --wsgi-file test.py

The options mean:

  • http :8000 – use protocol http, port 8000
  • wsgi-file test.py – load the specified file, test.py

Visit your site at: http://example.com:8000
If you see “Hello World“, it means the following stack of components works:

web_client <=> uWSGI <=> Python

Test your Django project

Now we want uWSGI to do the same thing, but to run a Django site instead of the test.py module.

If you haven’t already done so, make sure that your mysite project actually works. In your Django project’s directory run this:

python manage.py runserver 0.0.0.0:8000

If that works, run it using uWSGI:

uwsgi --http :8000 --module mysite.wsgi
  • module mysite.wsgi – load the specified wsgi module

Point your browser at the server; if the site appears, it means uWSGI is able serve your Django application, and this stack operates correctly:

web_client <=> uWSGI <=> Django

Now normally we won’t have the browser speaking directly to uWSGI. That’s a job for the webserver, which will act as a go-between.

Install NGINX

To add NGINX yum repository, create a file named /etc/yum.repos.d/nginx.repo and paste the configurations below:

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

After that we can install nginx with yum like so:

yum install nginx
chkconfig nginx on
/etc/init.d/nginx start

And now check that the nginx is serving by visiting it in a web browser on port 80 – you should get a message from nginx: “Welcome to nginx!”. That means these components of the full stack are working together:

web_client <=> web_server

If something else is already serving on port 80 and you want to use nginx there, you’ll have to reconfigure nginx to serve on a different port.

Configure NGINX for your site

You will need the uwsgi_params file, which is available in the NGINX directory of the uWSGI distribution.

Copy it into your project directory. In a moment we will tell NGINX to refer to it.

cp /etc/nginx/uwsgi_params /var/www/mysite/

Now create a file called /var/www/mysite/mysite_nginx.conf, and put this in it:

# mysite_nginx.conf
 
# the upstream component nginx needs to connect to
upstream django {
    # server unix:///var/www/mysite/mysite.sock; # for a file socket
    server 127.0.0.1:8001; # for a web port socket (we'll use this first)
    }
 
# configuration of the server
server {
    # the port your site will be served on
    listen      80;
    # the domain name it will serve for
    server_name 172.245.9.46; # substitute your machine's IP address or FQDN
    charset     utf-8;
 
    # max upload size
    client_max_body_size 75M;   # adjust to taste
 
    # Django media
    location /media  {
        alias /usr/share/nginx/html/media;  # your Django project's media files - amend as required
    }
 
    location /static {
        alias /usr/share/nginx/html/static; # your Django project's static files - amend as required
    }
 
    # Finally, send all non-media requests to the Django server.
    location / {
        uwsgi_pass  django;
        include     /var/www/mysite/uwsgi_params; # the uwsgi_params file you installed
        }
    }

This config file tells NGINX to serve up media and static files from the filesystem, as well as handle requests that require Django’s intervention. For a large deployment it is considered good practice to let one server handle static/media files, and another handle Django applications, but for now, this will do just fine.

Symlink to this file from /etc/nginx/conf.d so nginx can see it:

ln -s /var/www/mysite/mysite_nginx.conf /etc/nginx/conf.d

Basic NGINX test

Reload NGINX:

/etc/init.d/nginx reload

To check that media files are being served correctly, add an image called media.png to the /usr/share/nginx/html/media directory, then visit http://example.com/media/media.png – if this works, you’ll know at least that nginx is serving files correctly.

It is worth not just restarting nginx, but actually stopping and then starting it again, which will inform you if there is a problem, and where it is.

NGINX and uWSGI and test.py

Let’s get NGINX to speak to the “Hello World” test.py application.

uwsgi --socket :8001 --wsgi-file test.py

This is nearly the same as before, except this time one of the options is different:

  • socket :8001 – use protocol uwsgi, port 8001

NGINX meanwhile has been configured to communicate with uWSGI on that port, and with the outside world on port 80. Visit: http://example.com to check.

And this is our stack:
web_client <=> web_server <=> socket <=> uWSGI <=> Python

Meanwhile, you can try to have a look at the uswgi output at http://example.com:8001 – but quite probably, it won’t work because your browser speaks http, not uWSGI, though you should see output from uWSGI in your terminal.

Using Unix sockets instead of ports

So far we have used a TCP port socket, because it’s simpler, but in fact it’s better to use Unix sockets than ports – there’s less overhead.

Edit mysite_nginx.conf, changing it to match:

server unix:///path/to/your/mysite/mysite.sock; # for a file socket
# server 127.0.0.1:8001; # for a web port socket (we'll use this first)

Reload NGINX:

/etc/init.d/nginx reload

Run uWSGI again:

uwsgi --uid nginx --gid nginx  --socket mysite.sock --wsgi-file test.py

This time the socket option tells uWSGI which file to use.

Visit: http://example.com

Doesn’t work?

Check your nginx error log (/var/log/nginx/error.log). If you see something like:

connect() to unix:///path/to/your/mysite/mysite.sock failed (13: Permission
denied)

Then probably you need to manage the permissions on the socket so that nginx is allowed to use it.

Try:

uwsgi --uid nginx --gid nginx  --socket mysite.sock --wsgi-file test.py --chmod-socket=666

You may also have to add your user to nginx’s group, or vice-versa, so that nginx can read and write to your socket properly.

It’s worth keeping the output of the nginx log running in a terminal window so you can easily refer to it while troubleshooting.

Run Django application with uWSGI and NGINX

Let’s run our Django application:

uwsgi --uid nginx --gid nginx --socket mysite.sock --module mysite.wsgi

Now uWSGI and NGINX should be serving up not just a “Hello World” module, but your Django project.

Configure uWSGI to run with .ini file

We can put the same options that we used with uWSGI into a file, and then ask uWSGI to run with that file. It makes it easier to manage configurations.

Create a file called mysite_uwsgi.ini:

[uwsgi]
# the base directory (full path)
chdir           = /var/www/mysite
# project's wsgi file
module          = mysite.wsgi
# master
master          = true
# master process id
pidfile         = /tmp/mysite-master.pid
# simple rule is no. of cores on machine
processes       = 1
# user id
uid             = nginx
# group id
gid             = nginx
# using unix socket (use the full path to be safe)
socket          = /var/www/mysite/mysite.sock
# respawn processes after serving 5000 requests (avoid memory leaks)
max-requests    = 5000
# clear environment on exit
vacuum          = true
enable-threads  = true
# background the process
daemonize       = /var/log/uwsgi/mysite.log

And run uswgi using this file:

uwsgi --ini mysite_uwsgi.ini

Once again, test that the Django site works as expected.

Reloading the server

When running with the master process mode, the uWSGI server can be gracefully restarted without closing the main sockets.

There are several ways to make uWSGI gracefully restart.

# using kill to send the signal
kill -HUP `cat /tmp/mysite-master.pid`
# or the convenience option --reload
uwsgi --reload /tmp/mysite-master.pid
# or if uwsgi was started with touch-reload=/tmp/mysite-master.pid
touch /tmp/mysite-master.pid

Or from your application, in Python:

uwsgi.reload()

Stopping the server

If you have the uWSGI process running in the foreground for some reason, you can just hit CTRL+C to kill it off.

When dealing with background processes, you’ll need to use the master pidfile again. The SIGINT signal will kill uWSGI.

kill -INT `cat /tmp/mysite-master.pid`
# or for convenience...
uwsgi --stop /tmp/mysite-master.pid

Emperor mode

uWSGI can run in ‘emperor‘ mode. In this mode it keeps an eye on a directory of uWSGI config files, and will spawn instances (‘vassals’) for each one it finds.

Whenever a config file is amended, the emperor will automatically restart the vassal.

Create a directory for the vassals.

# mkdir /etc/uwsgi
# mkdir /etc/uwsgi/vassals

Symlink from the default config directory to your config file.

ln -s /var/www/mysite/mysite_uwsgi.ini /etc/uwsgi/vassals/

Run the emperor:

uwsgi --emperor /etc/uwsgi/vassals --uid nginx --gid nginx

The options mean:

  • emperor – where to look for vassals (config files)
  • uid – the user id of the process once it’s started
  • gid – the group id of the process once it’s started

Check the site; it should be running.

Start uWSGI when system boots

The last step is to make it all happen automatically at system boot/reboot.

Edit /etc/rc.local and add:

/usr/bin/uwsgi --emperor /etc/uwsgi/vassals --uid nginx --gid nginx

And that should be it!

Further configuration

It is important to understand that this is only a tutorial to get you started. You do need to read the NGINX and uWSGI documentation, and study the options available before deployment in a production environment.

Separating MySQL

When the traffic increases you’ll quickly run into resource contention between the different pieces of software. Database servers and Web servers love to have the entire server to themselves, so when run on the same server they often end up “fighting” over the same resources (RAM, CPU) that they’d prefer to monopolize.

This is solved easily by moving the database server to a second machine.

yum install mysql-server
chkconfig mysqld on
service mysqld start

To set up MySQL run:

/usr/bin/mysql_secure_installation

which will also give you the option of removing the test databases and anonymous user created by default. This is strongly recommended for production servers.

Creating Database

Login to MySQL:

[[email protected] ~]# mysql -u root -p

Creating Database and User:

mysql> CREATE DATABASE django DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_unicode_ci;
mysql> GRANT ALL PRIVILEGES ON django.* TO 'USER'@'192.168.1.100' IDENTIFIED BY 'PASSWORD';

iptables

If you have iptables enabled and want to connect to MySQL from another machine you’ll need to open a port in your server’s firewall (the default port is 3306). You don’t need to do this if the application using MySQL is running on the same machine.

If you do need to open a port (again, only if you’re accessing MySQL from a different machine from the one you’re installing on), you can use the following rules in iptables to open port 3306:

iptables -I INPUT -p tcp --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -I OUTPUT -p tcp --sport 3306 -m state --state ESTABLISHED -j ACCEPT
service iptables save

Testing MySQL remote connection

We need to make sure that our application server is able to connect to the database. On the server where you have Django, uWSGI and NGINX installed, run the following to install MySQL-python:

[[email protected] ~]# yum install MySQL-python
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.gigenet.com
 * extras: yum.singlehop.com
 * updates: mirror.ash.fastserv.com
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package MySQL-python.x86_64 0:1.2.3-0.3.c1.1.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: MySQL-python-1.2.3-0.3.c1.1.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: MySQL-python-1.2.3-0.3.c1.1.el6.x86_64
--> Running transaction check
---> Package mysql-libs.x86_64 0:5.1.69-1.el6_4 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
=================================================================================================================================================
 Package                            Arch                         Version                                     Repository                     Size
=================================================================================================================================================
Installing:
 MySQL-python                       x86_64                       1.2.3-0.3.c1.1.el6                          base                           86 k
Installing for dependencies:
 mysql-libs                         x86_64                       5.1.69-1.el6_4                              updates                       1.2 M
 
Transaction Summary
=================================================================================================================================================
Install       2 Package(s)
 
Total download size: 1.3 M
Installed size: 4.3 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): MySQL-python-1.2.3-0.3.c1.1.el6.x86_64.rpm                                                                         |  86 kB     00:00     
(2/2): mysql-libs-5.1.69-1.el6_4.x86_64.rpm                                                                               | 1.2 MB     00:00     
-------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                            2.4 MB/s | 1.3 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : mysql-libs-5.1.69-1.el6_4.x86_64                                                                                              1/2 
  Installing : MySQL-python-1.2.3-0.3.c1.1.el6.x86_64                                                                                        2/2 
  Verifying  : MySQL-python-1.2.3-0.3.c1.1.el6.x86_64                                                                                        1/2 
  Verifying  : mysql-libs-5.1.69-1.el6_4.x86_64                                                                                              2/2 
 
Installed:
  MySQL-python.x86_64 0:1.2.3-0.3.c1.1.el6                                                                                                       
 
Dependency Installed:
  mysql-libs.x86_64 0:5.1.69-1.el6_4                                                                                                             
 
Complete!

To test the installation, open a command prompt or shell. Enter the Python interactive interpreter and import the MySQLdb module. If this returns without error, then you are all set. If not go back and try to reinstall the MySQLdb package.

[root@web ~]# python
Python 2.6.6 (r266:84292, Feb 22 2013, 00:00:18) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb

Create a file db_connection_test.py and add the following:

#!/usr/bin/python
 
import MySQLdb
 
# Open database connection
db = MySQL.connect("HOST","USER","PASSWPRD","DATABASE")
 
# prepare a cursor object using cursor() method
cursor = db.cursor()
 
# execute SQL query using execute() method.
cursor.execute("SELECT VERSION()")
 
# Fetch a single row using fetchone() method.
data = cursor.fetchone()
 
print "Database version : %s " % data
 
# disconnect from server
db.close()

When you run this script, you will see following output on your machine.

[[email protected] ~]# python db_connection_test.py 
Database version : 5.1.69

So we know that connecting to the remote MySQL is working.