Nginx+uwsgi+django Deploying a Web server

table of Contents

Nginx + uWSGI + Django deployment web server
Environmental description
Foreword
Build the project
Django deployment
Edit luffy / luffy / settings.py
Edit luffy / app01 / views.py
Edit luffy / luffy / urls.py
Run and test
uWSGI deployment
Test running uWSGI
Run django project using uWSGI
uWSGi hot load Djangoa project
Deploy nginx
nginx configure uwsgi and django
Django deploys static files
Reload nginx for testing
Test nginx applications uWSGI and test.py
Replace TCP port with UNIX socket
uwsgi deployment enhancement
Run django project using uwsgi configuration file
Install uWSGI into the real environment
More parameters of uwsgi configuration file
uWSGI boot service
Nginx + uWSGI + Django deployment web server Environment description
You need to build a good environment before proceeding with this article
Linux system, I use openSUSE
Use operation user's home directory as test environment
python3.5.6
virtualenv 16.0
pip3 18.0
nginx 1.13.1
The environment behind the installation
django 1.11
uwsgi-2.0.17.1
Foreword
Many years ago, python-based web projects, common deployment methods are:

fcgi: Use spawn-fcgi or the tools that come with the framework to generate monitoring processes for each project, and then interact with the http service.
wsgi: Use the mod_wsgi module of the http service to run various projects.
However, after uwsgi appeared, most of the deployment will not use the above two protocols.
Because uwsgi uses neither the wsgi nor the fcgi protocol, but created a uwsgi protocol. According to the author, the protocol is about 10 times faster than the fcgi protocol. The main features of uWSGI are as follows:

Super fast performance.
Low memory usage (measured to be about half of mod_wsgi of apache2).
Multi-app management.
Detailed log function (can be used to analyze app performance and bottlenecks).
Highly customizable (memory size limit, restart after a certain number of services, etc.).
Therefore, based on the uwsgi protocol, it is very common to cooperate with nginx to do web project deployment of python (like the framework of django).

uwsgi is actually an http server, but it is only for python web applications.
Although uwsgi is also an http server, it is inconvenient to use it directly to deploy python web applications.

The role played by uwsgi is the back-end http server, the role played by nginx is the front-end http server, and the django project is the data provider that the customer really wants to access. The user sends a request from the web browser. After receiving the request, the nginx server forwards the user's request to the uwsgi server through its uwsgi module. After the uwsgi server processes the request through django, the result is returned to nginx, and the browser will send the final result Show it to users.

Build the project
Create a virtual environment
It is recommended that personal learning and testing should be built directly in the home user directory to avoid various rejection problems caused by permissions
Or formally, the system creates another dev user and gives the dev user permission to run in a special working directory.
The directory name of the virtual environment is recommended to bring an env to point out that it is a virtual environment. If you take the same name as the following project, you will see a triple of the same name, which is a bit difficult to recognize.
virtualenv -p python3 py3env
Start the virtual environment
source py3env / bin / activate

Install django1.11, the reason for installing this version is necessary for learning, and your own projects are better to keep up with the times.
pip install django == 1.11

Create a django project directly using the django-admin command
It is recommended not to put it in the py3env directory, but to create it in a special working directory
django-admin startproject luffy
As follows, a simple project has been initially built:

cd luffy

(py3env) [email protected]: ~ / work / luffy> tree -L 2
.
├── luffy
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── manage.py
Create a new application, for example app01
python3 manage.py startapp app01
The structure is as follows
(py3env) [email protected]: ~ / work / luffy> tree -L 2
.
├── app01
│ ├── admin.py
│ ├── apps.py
│ ├── __init__.py
│ ├── migrations
│ ├── models.py
│ ├── tests.py
│ └── views.py
├── luffy
│ ├── __init__.py
│ ├── __pycache__
│ ├── settings.py
│ ├── urls.py
│ └── wsgi.py
└── manage.py
Django deployment edit luffy / luffy / settings.py
# SECURITY WARNING: do n‘t run with debug turned on in production!
DEBUG = False

ALLOWED_HOSTS = [‘*‘,]

# Application definition

INSTALLED_APPS = [
    ‘Django.contrib.admin’,
    ‘Django.contrib.auth’,
    ‘Django.contrib.contenttypes’,
    ‘Django.contrib.sessions’,
    ‘Django.contrib.messages’,
    ‘Django.contrib.staticfiles’,
    ‘App01’,
]
As above, mainly:

Change DEBUG from True to False (although it is only built for learning and testing, we must also take into account the safety)
Fill in the default empty list of ALLOWED_HOSTS to fill in the domain name you intend to use. When I tested it, I filled in *. It is not recommended to fill in the wildcard * during the actual online deployment, but to fill in the allowed host domain name. Generally django is used as a backend server, and the front end will have a domain name. For details, see the official django documentation.
Add appo1 under the INSTALLED_APPS list, which means that app01 is applied to the installation registration.
In addition, after configuring nginx and creating a new domain name later, I changed ALLOWED_HOSTS = [‘*‘,] to ALLOWED_HOSTS = [‘luffy.tielemao’,], and access via domain name is still normal. Directly knocking on the ip address will report a 400 error.
Edit luffy / app01 / views.py
from django.shortcuts import render, HttpResponse

# Create your views here.

def index (request):
    return HttpResponse (‘Hello Hero’)
Since it is only a simple demonstration, the above is just to let the user access the index page and get a page that displays Hello Hero.

Edit luffy / luffy / urls.py
from django.conf.urls import url
from django.contrib import admin
from app01 import views

urlpatterns = [
    url (r ‘^ admin /‘, admin.site.urls),
    url (r ‘^ index /‘, views.index),
]
Run and test
The first is to run, because it is a separate test for django, so just enter the project root directory and type the following command on the command line:

python manage.py runserver 0.0.0.0:8083
Among them, 0.0.0.0 means to bind all the IP addresses of the network cards on the server, of course, including 127.0.0.1 and the public network IP of the external network.

At first, I tested ALLOWED_HOSTS. The allowed HOSTS is my own domain name. Tielemao.com, and the result is that I directly hit the IP on the browser and add port 8083 to the index. Of course, it is an error:

Bad Request (400) and because I set the DEBUG value to False, it will not expose the debugging information.

Then I changed .tielemao.com to the wildcard * (unsafe, which means that all host headers are allowed).
When you visit http: //39.x.x.x: 8083 / index /, you can access it normally:
Note that I am using port 8083 and that the port has been set to open at the security group on the firewall and cloud control platform.
After django runs correctly, ctrl + c will stop it, and then do the following operations.

uWSGI deployment
Also for the isolation and purity of the environment, this time I also chose to install in the same virtual environment:
pip3 install uwsgi
After the installation is complete, you can view the version by uwsgi --version,
uwsgi --python-version can also indirectly view the python version.

Write a python script for simple testing of uwsgi
vim test-uwsgi.py, the content is as follows:
def application (env, start_response):
    start_response (‘200 OK’, [(‘Content-Type’, ‘text / html’)])
    return [b "Hello Hero, all for one"]
Test running uWSGI
The following command means running the uwsgi service, which also opens web access on port 8083.

Note--After http, there is a space and then connect: port number.
uwsgi --http: 8083 --wsgi-file test-uwsgi.py
Access the 8083 port of the web server and display the text returned in the test-uwsgi.py script normally.
Because what we have to do now is to deploy django based on nginx and uwsgi. From the client's request to the server's response to the request, there will be several links:

the web client <-> the web server (nginx) <-> the socket <-> uwsgi <-> Django
If uWSGI is tested separately, and the access display is normal, the following three links are smooth:

the web client <-> uWSGI <-> Python
Ctrl + c aborts the program, and then performs the following test.

Run django project using uWSGI
In the virtual environment, enter the django root directory and type the following command:

 uwsgi --http: 8000 --module luffy.wsgi
Effects and directly knock the following commands

python manage.py runserver 0.0.0.0:8000
it's the same.

Note: --module luffy.wsgi is to load the specified wsgi module. What is the name of your project, generally it is the project name.wsgi.
This test proves that the web client to Django through uWSGI is smooth:

the web client <-> uWSGI <-> Django
ctrl + c abort the program, and then come to the following test

uWSGi hot load Djangoa project
Add parameters after the start command:

uwsgi --http: 8083 --module luffy.wsgi --py-autoreload = 1
Similarly, this time access to the server port 8083, that is, access to the django project (luffy).
And there is one more parameter:
--py-autoreload monitor python module to trigger reload (only used in development)

# Similar post commands
/ home / operation / work / py3env / bin / uwsgi --http 0.0.0.0: 8000-
-chdir / home / operation / work / luffy --module luffy.wsgi
# At this time modify the django code, uWSGI will automatically load the django program, the page takes effect
# In addition, you can also connect many parameters
ctrl + c aborts the program, then proceed to the following link:

Deploy nginx nginx configuration uwsgi and django
Copy the uwsgi_params file to the project folder.
The uwsgi_params file is generally in the / etc / nginx / directory, and can also be downloaded from the nginx github page.
The uwsgi_params configuration file is as follows:

uwsgi_param QUERY_STRING $ query_string;
uwsgi_param REQUEST_METHOD $ request_method;
uwsgi_param CONTENT_TYPE $ content_type;
uwsgi_param CONTENT_LENGTH $ content_length;

uwsgi_param REQUEST_URI $ request_uri;
uwsgi_param PATH_INFO $ document_uri;
uwsgi_param DOCUMENT_ROOT $ document_root;
uwsgi_param SERVER_PROTOCOL $ server_protocol;
uwsgi_param REQUEST_SCHEME $ scheme;
uwsgi_param HTTPS $ https if_not_empty;

uwsgi_param REMOTE_ADDR $ remote_addr;
uwsgi_param REMOTE_PORT $ remote_port;
uwsgi_param SERVER_PORT $ server_port;
uwsgi_param SERVER_NAME $ server_name;
Copy to the project:
sudo cp / etc / nginx / uwsgi_params / home / operation / work / luffy /

Re-modify the permissions of the owner and the group
sudo chown operation: operation uwsgi_params
sudo chmod 664 / home / operation / work / luffy / uwsgi_params
Create the file luffy_nginx.conf in the project folder, fill in and modify the following:
# luffy_nginx.conf

# the upstream component nginx needs to connect to
upstream django {
    # server unix: ///path/to/your/mysite/mysite.sock;
    # for a file socket
    server 127.0.0.1:8083;
    # for a web port socket (we ‘ll use this first)
}

# configuration of the server
server {
    # the port your site will be served on
    listen 8000;
    # the domain name it will serve for
    server_name luffy.tielemao.com;
    # substitute your machine ‘s IP address or FQDN
    charset utf-8;

    # max upload size
    client_max_body_size 75M; # adjust to taste

    # Django media
    location / media {
        alias / home / operation / work / luffy / media;
        # your Django project ‘s media files-amend as required
    }

    location / static {
        alias / home / operation / work / luffy / static;
        # your Django project ‘s static files-amend as required
    }

    # Finally, send all non-media requests to the Django server.
    location / {
        uwsgi_pass django;
        include / home / operation / work / luffy / uwsgi_params;
        # the uwsgi_params file you installed
    }
}
This configuration file tells nginx to pull media and static files from the file system as services,
At the same time respond to django's request.

To this end, I also established media and static folders under the corresponding project directory.

To facilitate the use of nginx, I created a soft link of luffy_nginx.conf under /etc/nginx/vhosts.d. Of course, some people just use the default sites-enabled directory, just look at it.

sudo ln -s /home/operation/work/luffy/luffy_nginx.conf luffy.conf
Django deploys static files
Under the project folder, create a static file directory:
mkdir static

In django's setting file, add a line:
STATIC_ROOT = os.path.join (BASE_DIR, "static /")

run
python manage.py collectstatic

Under the project folder, create a media media file directory (used to store pictures and music files for testing)
mkdir media

In the media directory, I passed in a picture file princekin_fox.jpg for testing nginx.

Where is the parameter STATIC_ROOT used?
Collect all the static files you use and save them to STATIC_ROOT through python3 manage.py collectstatic!

The STATIC_ROOT folder is used to copy all the files in all the folders in STATICFILES_DIRS and the files in the static in each app. Putting these files together is more convenient when deploying with nginx.

Reload nginx for testing
Check first to see if there is any error in the previous configuration file.
sudo / usr / sbin / nginx -t

Reload nginx to make the soft link's luffy_nginx.conf take effect.
sudo / usr / sbin / nginx -s reload

Visit http://luffy.tielemao.com:8000/media/princekin_fox.jpg to see if you can access the picture resources normally:
Note that before doing this step, I have configured the domain name and sent a picture for testing to the server.

I have successfully accessed the resource, so I will do the next test.

Test nginx applications uWSGI and test.py
Remember the test-uwsgi.py file for a test you wrote earlier?
We use the configured nginx to access the test-uwsgi.py started by uwsgi.
First, start uwsgi, and the port is port 8083 of the load balancing pool in the nginx configuration:
uwsgi --socket: 8083 --wsgi-file test-uwsgi.py

Visit http://luffy.tielemao.com:8000/, in fact, visit the 8083 port of uwsgi.
For example, it can be accessed normally. It shows that the following links are smooth:

the web client <-> the web server <-> the socket <-> uWSGI <-> Python
After the test is successful, the program is terminated in order to proceed to the following link.

Replace TCP port with UNIX socket
Modify luffy_nginx.conf:

# luffy_nginx.conf

# the upstream component nginx needs to connect to
upstream django {
    server unix: ///home/operation/work/luffy/luffy.sock;
    # for a file socket
    # server 127.0.0.1:8083;
    # for a web port socket (we ‘ll use this first)
}
The above is actually annotating the original server 127.0.0.1:8083, and the server unix: ///home/operation/work/luffy/luffy.sock with the original # comment is uncommented, that is, the port is not directly specified , But specified a sock file.

It should be noted that this luffy.sock is not a configuration file written in advance, but is automatically generated by the program.
Reload nginx, then run uWSGI under the project

uwsgi --socket /home/operation/work/luffy/luffy.sock --wsgi-file test-uwsgi.py
Visit http://luffy.tielemao.com:8000/ and report a 502 gateway error.
Check the nginx error log error.log, the general location will be /var/log/nginx/error.log, it is recommended to use tail to view the last 10 lines at the end of the file.
You will find errors similar to the following reported blocked permissions:

2018/08/27 20:17:44 [crit] 25771 # 25771: * 1847865 connect () to unix: ///home/operation/work/luffy/luffy.sock failed (13: Permission denied) while connecting to upstream , client: 223.72.74.11, server: luffy.tielemao.com, request: "GET / HTTP / 1.1", upstream: "uwsgi: // unix: ///home/operation/work/luffy/luffy.sock:" , host: "luffy.tielemao.com:8000"
Then stop the uwsgi process, add socket permissions and run again:

uwsgi --socket /home/operation/work/luffy/luffy.sock --wsgi-file test-uwsgi.py --chmod-socket = 664
If you still get an error, you need to add the operation user to the nginx group. Of course, in some systems, the nginx group may also be www-data.

Add the operation user to the nginx group, remember to add the -a parameter, otherwise the operation will leave the original operation group.

sudo usermod -a -G nginx operation
Then the project directory also belongs to the nginx group

sudo chown operation: nginx -R luffy
This will ensure that nginx can handle luffy.so ck has sufficient permissions.
Sure enough, after setting the permissions, you can access normally.

It is inconvenient to run the above command every time to start the django project. We can write these configurations in the .ini file, which can greatly simplify the work.

After stopping the uwsgi service, see the next link.

uwsgi deployment enhancement Use uwsgi configuration file to run django project
uwsgi supports multiple configuration methods such as ini and xml. Taking ini as an example, create uwsgi_luffy.ini in the / home / operation / work / conf directory. The sample configuration is as follows:

# luffy_uwsgi.ini file
[uwsgi]

# Django-related settings
# the base directory (full path)
# Specify the running directory, which is actually the root directory of the project
chdir = / home / operation / work / luffy

# Django ‘s wsgi file
# Import the wsgi module of the django project
module = luffy.wsgi: application

# Load wsgi-file
wsgi-file = luffy / wsgi.py

# the virtualenv (full path)
# Configure the path of the virtual environment python related dependency packages
home = / home / operation / work / py3env

# Added, users and user groups when uwsgi starts, pay attention to configure according to your own actual permissions
# My side is because the project owner is operation, and operation I set to add to the nginx group
# But in this case uid still cannot set nginx, unless all the file owner of your project directory is changed to nginx
uid = operation
gid = nginx

# process-related settings
# Start the master process
master = true

# maximum number of worker processes
# How many processes are started, workers are also equivalent to processes
# threads item is to set the running thread, the test does not need to set the upper thread
processes = 4

# the socket (use the full path to be safe)
# Set the socket port or socket address used
# socket = 0.0.0.0:8000
# The above socket is recommended to be configured as a luffy.socket file and then use nginx to connect to uWSGI to run, otherwise it is easy to report socket request header errors and permissions errors.
socket = /home/operation/work/luffy/luffy.socket

# ... with appropriate permissions-may be needed
# Configure the permissions of the generated sock file
chmod-socket = 664

# clear environment on exit
# Clearing the environment when exiting is actually killing the automatically generated luffy.sock and related pid files.
vacuum = true
uwsgi specifies the configuration file to start the django project, it is recommended to use nginx users to execute:
uwsgi --ini /home/operation/work/conf/uwsgi_luffy.ini

Note: If you directly specify socket = 0.0.0.0:8000 in the configuration file after the above startup, the following problems may occur: After the browser accesses the server's port 8000 plus the url, the browser will report a connection error, and the server runs in the background You will also see the following error message:
invalid request block size: 21573 (max 4096) ... skip
Before we used the HTTP protocol directly, the block request size would not exceed

 uwsgi --http: 8000 --module luffy.wsgi
The reason is that using the configuration file to start is to provide a communication port in socket mode. The default protocol is tcp, which is different from http. The default size of the socket request header is 4096, so an error will occur when the request header exceeds the default size. Of course, we can solve it later through cooperation with nginx.

And if you just want to test, then just specify the block request size -b 24576 and so on after the above command.

After stopping uwsgi, we re-specify the block size and run the command:
uwsgi -b 24576 --ini /home/operation/work/conf/uwsgi_luffy.ini
Can solve the request header error.

However, the ini configuration file is mainly used to cooperate with nginx, which is why the ini configuration file of uwsgi is discussed after the previous description of nginx deployment.

Put the same options used in uWSGI into a configuration file, and then ask uWSGI to run using this file. This will make it easier to manage the configuration.

It should be noted that because of the need to cooperate with nginx, the generated project name .sock file, nginx needs to be able to read and write.

Such as:

Since I used the operation user when executing the uwsgi command earlier,
In this way, the automatically generated luffy.sock file belongs to a group that is not nginx, so it is best to add uid and gid configuration items to the ini configuration file to use the nginx user to execute uwsgi.

# uwsgi start user name and user group
uid = operation
gid = nginx
I am able to access the django project normally after modifying the group of luffy.sock to nginx, otherwise nginx will report a 502 error.

Install uWSGI into the real environment
So far, we have been working in a virtual environment. Next, we will install uwsgi in the actual environment and add uwsgi to the nginx group (some are the www-data group) to avoid the permissions encountered now And other issues.

Exit the virtual environment
deactivate

Install uWSGI, pay attention to use pip3 in python3 to install. Python2.7 and python3.5.6 coexist in my system, but the default environment is 2.7.
sudo /usr/local/python3.5.6/bin/pip3 install uwsgi

Before I used the default pip to install, the result is the python2 version of uwsgi. When running the python3 django project in the virtual environment, the following error will be reported:
Python version: 2.7.13 (default, Jan 03 2017, 17:41:54) [GCC]
Set PythonHome to / home / operation / work / py3env
ImportError: No module named site
VACUUM: unix socket /home/operation/work/luffy/luffy.sock removed.
The module import error is reported. The reason is that my real environment uses the python2.7 version. The virtual environment home directory configured in ini is a dependent library of python3, and uwsgi installed by pip cannot be imported correctly.

Check again, whether the django project can be run in the real environment as before the virtual environment.
Of course, because python3 in front of me is not imported into the system environment, so here still need to play the full path like pip3 using python3 in front:
sudo /usr/local/python-3.5.6/bin/uwsgi --ini /home/operation/work/conf/uwsgi_luffy.ini
The error that the module could not be imported has been ruled out, but an error of bind binding rejection will also be reported.
This error is still related to file permissions, because the user I set in the uid of the uwsgi ini configuration file is nginx, and the real owner of the project is the user of operation. It is also feasible to set the entire owner of the project as the nginx user.

# I stepped on a lot of pits because of permission issues, so I will continue to emphasize it here.
# Note that you need to configure according to your actual environment
uid = operation
gid = nginx
I have successfully released uwsgi to let you compare it.
Of course, the successful visit to the page will not be put away, the same as the previous successful test.

More parameters of uwsgi configuration file
I have seen some configuration parameters of uwsgi's ini file before. Let's take the luffy project as an example, and then introduce some commonly used parameters:

# set an environment variable
# Setting the variable environment is actually the settings in the django project
env = DJANGO_SETTINGS_MODULE = luffy.settings

# create a pidfile
# Create pid file, pay attention to permissions, so it is generally placed in the tmp directory
pidfile = /tmp/project-master.pid

# background the process & log
# Running in the background, useful, basically will be set, and also output logs
daemonize = /var/log/uwsgi/luffy.log

# With a fixed file size (in kb), cut logs
# The following example is a log file with a size of 50M
log-maxsize = 50000000

# Do not record the request information log. Only log errors and uWSGI internal messages to the log.
disable-logging = true
##
# At the specified address, enable the status service. It should be that you visit the address, you can see the entire uwsgi
The status is similar to nginx. So it is generally only open on the loopback address.
stats = 127.0.0.1:8009

# The following are some performance and resource limitations and deployment
# respawn processes taking more than 20 seconds
harakiri = 20
# limit the project to 128 MB
# Control the total memory of the process
limit-as = 128
# respawn processes after serving 5000 requests
max-requests = 5000
Visit the stats status service page as follows:
uWSGI boot service
Continue to take the django luffy project as an example, let uWSGI start the service after booting, edit the file /etc/rc.local, add the following content before this line of code exit 0:

/usr/local/python-3.5.6/bin/uwsgi --ini /home/operation/work/conf/uwsgi_luffy.ini
It should be noted that the user group, sock file, sock file permissions and background operation settings are set in the ini configuration file.

end
by Tie Le and Cat

reference

uwsgi official documentation
py fish blog related articles
Nginx + uWSGI + Django deployment web server