Using Python's flask framework to implement streaming media transmission of video _python

Flask is a Python-implemented WEB development micro-framework. This article is a detailed tutorial on how to use it to deliver streaming video data.

I'm sure you already know that I posted a book on O ' Reilly media about flask and some video materials. On top of these, the coverage of the flask framework is fairly complete, and for some reason there is not much mention of a small number of features, so I think it's a good idea to write an article about them here.

This article is dedicated to streaming media, and this interesting feature allows flask applications to have the ability to efficiently provide data to large requests by dividing them into small chunks of data, which can take a long time. To illustrate this topic, I will show you how to build a real-time video streaming media server!
What is streaming media?

Streaming media is a technique in which the server responds to requests in the form of blocks of data. I can think of a reason why this technique might be useful:

Very large response. For very large responses, the response collected in memory is returned only to the client, which is inefficient. Another approach is to write the response to disk and then use Flask.send_file () to return the file, but this adds a combination of I/O. Assuming that the data can be generated in chunks, providing a response to the request in small chunks of data is a better solution.
Real-time data. For some applications, you need to request the returned data from a live data source. A good example of this is the provision of a live video or audio. Many security cameras use this technology to spread video data to Web browsers.

Using flask to implement streaming transmission

Flask provides native support by using a generator function convection response. The generator is a special function that can be interrupted and recovered. Consider the following function:

 
Def gen ():  
  yield 1  
  yield 2  
  yield 3

This is a three-step function that returns a value each step. Describing how the builder is implemented is beyond the scope of this article, but if you're a little curious, the following shell session will show you how the generator was used:

 
>>> x = gen ()
>>> x
<generator object Gen at 0x7f06f3059c30>
>>> X.next () C4/>1
>>> x.next ()
2
>>> x.next ()
3
>>> x.next ()
Traceback ( Most recent called last):
 File "<stdin>", line 1, in <module>
stopiteration

In this simple example you can see that a generator function can return multiple results sequentially. Flask uses the feature of the generator function to achieve streaming.

The following example shows how you can use streaming to produce large data tables without having to put the entire table in memory:

 
From flask import Response, render_template to
app.models import stock
 
def generate_stock_table ():
  yield Render_template (' stock_header.html ') for the stock in
  Stock.query.all ():
    yield render_template (' stock_row.html ', Stock=stock
  yield render_template (' stock_footer.html ')
 
@app. Route ('/stock-table ')
def stock_table (): Return
  Response (generate_stock_table ())

In this example, you can see how the Flask and generator functions work together. The route that returns a streaming response (route) needs to return a response object initialized by the generator function. Flask then takes the call builder and sends the result to the client in a chunking way.

For this particular example, if you assume that the database query result returned by Stock.query.all () is an iterator, then you can generate one row of potentially large tables at a time, so no matter how many characters there are in the query, Memory consumption in the Python process does not grow larger because of the larger response string.
Multi-part response

As mentioned above, examples of tables generate a traditional web page in small chunks, and each part is connected to the final result. This is a good example of how to generate a larger response, but the more exciting thing is dealing with real-time data.

An interesting application using streaming is to use each block to replace the place on the original page, which makes the stream animate in the browser window. With this technique, you can make each block of data in a stream an image, which gives you a cool video input signal running in the browser!

The secret to implementing in-place updates is to use a multiple-part response. A multi-part response consists of a header (header) and many parts (parts). The header includes a content type in many parts, and the latter part is delimited by a boundary token, each containing a specific content type in its own section.

For different requirements, there are a number of content types in this section. For streaming transmissions, the Multipart/x-mixed-replace content type must be used for each part to replace the previous part. To help you understand what it really looks like, here's a response structure for a multi-part video stream transmission:

 
http/1.1 OK
content-type:multipart/x-mixed-replace; boundary=frame
 
--frame
content-type:image/ JPEG
 
<jpeg data here>
--frame
content-type:image/jpeg
 
<jpeg data here>
...

As you can see above, the structure is very simple. The main content-type heads are set to multipart/x-mixed-replace, while boundary markers are also defined. Each section then includes a prefix with two dashes, and a boundary string on the line. Each section has its own Content-type header, and each section optionally includes a content-length header that describes the byte length of the part payload, but at least for the image browser, it can handle streams without length.
establish a real-time video streaming media server

There is enough theory in this article that it is time to build a complete application that transmits real-time video streaming to a Web browser.

There are many ways to stream video to a browser, and each method has its advantages and disadvantages. A good way to work with flask flow features is to stream independent JPEG image sequences. This is the dynamic JPEG. This is used for many IP surveillance cameras. This method has a shorter delay time, but the quality of transmission is not the best because JPEG compression is not very effective for dynamic images.

Below you can see a very simple but complete Web application. It can provide a dynamic JPEG streaming transmission:

 
#!/usr/bin/env python from
flask import flask, render_template, Response from
camera import camera
 
app = Flas K (__name__)
 
@app. Route ('/')
def index (): Return  
  render_template (' index.html ')
 
def gen (camera): While
  True:
    frame = camera.get_frame ()
    yield (b '--framern '
        B ' content-type:image/jpegrnrn ' + frame + b ' rn ')
 
@app. Route ('/video_feed ')
def video_feed (): Return
  Response (Gen (Camera ()),
          mimetype= ' Multipart/x-mixed-replace; Boundary=frame ')
 
if __name__ = = ' __main__ ':
  app.run (host= ' 0.0.0.0 ', debug=true)

This application imports a camera class to be responsible for providing frame sequences. In this example, it is a good idea to put the camera control part in a separate module. In this way, Web applications are kept clean, simple, and generic.

The application has two routes (route). /Routing is the homepage service, which is defined in the index.html template. Below you can see the contents of this template file:

This is a simple HTML page that contains only a title and image label. Notice that the SRC attribute of the image label points to the second route of the application, which is where the magic occurs.

The/video_feed route returns a streaming response. Because this stream returns an image to be displayed on a Web page, in the SRC attribute of the image label, the URL points to the route. Because most/all browsers support a partial response (if you find a browser that does not support this, please tell me), the browser will automatically keep the image element updated by displaying the JPEG image stream.

The generator function used in/video_feed routing is called Gen (), which takes an instance of the camera class as its argument. The MimeType parameter is set as shown above and has a multipart/x-mixed-replace content type and a boundary string set to "frame".

The Gen () function goes into a loop where successive return frames from camera are used as response blocks. This function, as shown above, requires camera to provide a frame by calling the Camera.get_frame () method, and then generates a frame, which is formatted as a response block using the Image/jpeg content type.
get frames from the camera

Now all that's left is to implement the camera class, which must connect the camera hardware and download the live video frame from it. The advantage of encapsulating this application hardware-related part in a class is that the class can be implemented differently for different people, while the rest of the application remains unchanged. You can use this class as a device driver that provides a unified implementation regardless of the actual hardware device being used.

Another advantage of separating the camera class from the rest of the application is that it is easy to cheat the application when there is no video camera, so that it thinks there is a camera because the camera class can be implemented as an analog camera without real hardware. In fact, when I run this application, the easiest way to do that is to test the stream to do that without worrying about the hardware until I've made the other parts run correctly. Below, you can see the simple analog camera I use to achieve:

 
From time import Time
 
class Camera (object):
  def __init__ (self):
    self.frames = [Open (f + '. jpg ', ' RB '). Read () For f in [' 1 ', ' 2 ', ' 3 ']]
 
  def get_frame (self): return
    Self.frames[int (Time ())% 3]

This implementation reads three images from the disk 1.jpg, 2.jpg, 3.jpg, and then returns at the rate of one frame per second, sequentially. The Get_frame () function uses the current time, in seconds, to determine which three frames to return at a given moment. It's simple, isn't it?

To run this analog camera, I need to create three frames. I used GIMP to do the following image:

Because the camera is analog, you can run this application in any environment! I put all the files for this application in the GitHub. If you are familiar with Git, you can clone it using the following command:

 
$ git clone https://github.com/miguelgrinberg/flask-video-streaming.git

If you like to download it, you can get a zip file here.

After you install this application, create a virtual environment and install flask inside it. You can then use the following command to run this application:

 
$ python app.py

When you enter http://localhost:5000 in your Web browser to start this application, you will see the analog video streaming over the images 1, 2, 3. Pretty cool, huh?

Once, all of the apps were running, I started the raspberry pie and its camera module, and implemented a new camera class to turn the raspberry pie into a video streaming server, using Picamera packs to control the hardware. I'm not going to discuss the implementation of this camera class here, but you can find it in the camera_pi.py file in the source code.

If you have a raspberry pie and a camera module, you can edit the app.py file to import the camera class from this module, and then you can use the raspberry pie to transmit the streaming video in real time, as I did in the screenshot below:

If you want this streaming application to apply to different cameras, all you have to do is implement different camera classes. I would appreciate it if you could end up writing a project on the GitHub that is available to me.
Limit of Flow

The request cycle is short when the Flask Application Server provides a general request. The worker thread (Web worker) receives the request, invokes the handler function, and finally returns the response. Once the response is sent back to the client, the worker thread is idle and is ready to execute the next request.

When a request is received with a streaming transmission, the worker thread is bound to a client for the duration of the entire streaming transmission. When processing a long and endless stream, such as a video stream from a camcorder, the worker thread locks on a client until the client disconnects. This actually means that the number of clients that the application can serve is the same as the worker thread, unless you take a special approach. When using the debug mode applied by flask, this means that there is only one worker thread, so you will not be able to connect two browser windows simultaneously to view data flows from two different places at the same time.

There is a way to overcome this important limitation. In my opinion, the best solution is to use a Gevent,flask Web server, such as the full support of it. By using the gevent, you can handle multiple clients on a single worker thread because gevent modifies the python I/O function to make the necessary context switches.
Conclusions

If you miss the above content, the code contained in this article is placed in this GitHub library: https://github.com/miguelgrinberg/flask-video-streaming. Here, you can find a universal video stream transfer implementation without needing a camera, and there is also a Raspberry camera module to implement.

I hope this article explains some of the topics about streaming technology. I'm focused on video streaming because it's an area where I've had some experience, but besides streaming video, there are many other uses for transmission technology. For example, this technique can be used to maintain a longer connection between the client and the server, allowing the server to push new information. These days, the network sockets protocol is a more efficient way to achieve this, but the network sockets are quite new, only valid in modern browsers, and streaming technology can be used in any browser you can think of.