Using Python's flask framework for streaming video

Flask is a Python-implemented WEB development micro-framework. This article is a detailed tutorial on how to use it to deliver video data streams.

I'm sure you know now that I posted a book and some video about flask on the O ' Reilly media. On top of these, the flask framework describes the coverage is quite complete, for some reason, there is also a small number of features not much mentioned, so I think it is a good idea to write an article about them here.

This article is dedicated to streaming, and this interesting feature allows flask applications to have the ability to partition small chunks of data efficiently to provide data for large requests, which can take a long time. To illustrate this topic, I will show you how to build a real-time video streaming media server!
What is streaming media?

Streaming media is a technique in which the server responds to requests in the form of chunks of data. I can think of a reason why this technique might be useful:

A very large response. For very large responses, the responses collected in memory are only returned to the client, which is inefficient. Another method is to write the response to disk and then use Flask.send_file () to return the file, but this adds a combination of I/O. It is a better solution to assume that data can be chunked generated, and that responding to requests in a small piece of data.
Real-time data. For some applications, the data returned from the request needs to be from a live data source. A very good example of this is providing a live video or audio. Many security cameras use this technique to stream video data to a Web browser.

Using flask for streaming

Flask provides native support by using the generator function for streaming responses. The generator is a special function that can be interrupted and resumed. Consider the following function:

Def gen ():    yield 1    yield 2    yield 3

This is a three-step function in which each step returns a value. Describing how the generator is implemented is beyond the scope of this article, but if you're curious, the following shell session will show you how the generator is being used:

>>> x = gen () >>> x
 
  
   
  >>> x.next () 1>>> x.next () 2>>> X.next () 3 >>> X.next () Traceback (most recent): File "
  
   
    
   ", line 1, in 
   
    
     
    stopiteration
   
    
  
   
 
  

As you can see in this simple example, a generator function can sequentially return multiple results. Flask uses the feature of the generator function to implement streaming.

The following example shows how streaming can be used to produce large data tables without having to put the entire table into memory:

From flask import Response, render_templatefrom app.models import Stock def generate_stock_table ():  yield Render_ Template (' stock_header.html ')  for the stock in Stock.query.all ():    yield render_template (' stock_row.html ', stock =stock)  yield render_template (' stock_footer.html ') @app. Route ('/stock-table ') def stock_table ():  return Response (Generate_stock_table ())

In this example, you can see how the Flask and generator functions work together. The route that returns the streaming response needs to return a response object initialized by the generator function. Flask then takes the call generator and sends the result to the client in a chunked way.

For this particular example, if you assume that the database query result returned by Stock.query.all () is an iterator, you can generate one row of a potentially large table at a time, so that regardless of the number of characters in the query, The memory consumption in the Python process is not getting larger because of the larger response string.
Multi-part response

The example above mentions the table as a small block of the form of a traditional Web page, each part connected to the final result. This is a good example of how to generate a large response, but the more exciting thing is to deal with real-time data.

One interesting application for streaming is to use each block to replace the place in the original page, which causes the flow to animate in the browser window. With this technique, you can make each chunk of the stream an image, which gives you a cool video input signal that runs in the browser!

The secret to implementing an in-place update is the use of multipart responses. A multipart response consists of a header (header) and many parts (parts). A header consists of a content type in a multipart, followed by a boundary tag, each containing a specific content type in its own part.

There are several multipart content types for different needs. For those with streaming, the Multipart/x-mixed-replace content type must be used for each part to replace the previous part. To help you understand exactly what it looks like, here's a multi-part video streaming response structure:

http/1.1 Okcontent-type:multipart/x-mixed-replace; Boundary=frame--framecontent-type:image/jpeg 
 
   
  
   
    
   --framecontent-type:image/jpeg
  
    ...
 
  

As you can see above, this structure is very simple. The primary Content-type header is set to Multipart/x-mixed-replace, and the boundary tag is also defined. Then each section includes a prefix of two dashes, and a boundary string on this line. Each section has its own Content-type head, and each section can optionally include a content-length header that describes the byte length of the payload in which it is located, but at least for the image browser, it can handle streams with no length.
Establish a real-time video streaming media server

There's enough theory in this article, and it's time to build a complete application that streams live video to a Web browser.

There are many ways to stream video to your browser, and each method has its advantages and disadvantages. A good way to work with flask flow features is to stream independent JPEG image sequences. This is the dynamic JPEG. This is used for many IP surveillance cameras. This method has a short delay time, but the transmission quality is not the best because JPEG compression is not very effective for motion images.

Below you can see a very simple but complete web app. It can provide a dynamic JPEG stream transfer:

#!/usr/bin/env pythonfrom flask Import flask, render_template, responsefrom camera import camera app = Flask (__name__) @ap P.route ('/') def index ():    return render_template (' index.html ') def gen (camera): While  True:    frame = Camera.get_frame ()    yield (b '--framern '        B ' content-type:image/jpegrnrn ' + frame + B ' rn ') @app. Route ('/video_ Feed ') def video_feed ():  return Response (Gen (Camera ()),          mimetype= ' multipart/x-mixed-replace; boundary= Frame ') if __name__ = = ' __main__ ':  app.run (host= ' 0.0.0.0 ', debug=true)

This application imports a camera class that is responsible for providing a sequence of frames. In this example, it is a good idea to put the camera control part into a separate module. This way, web apps remain clean, simple, and generic.

The app has two routes. /routing is a home service, defined in the index.html template. Below you can see the contents of this template file:

    Video Streaming Demonstration    
Video Streaming Demonstration
   

This is a simple HTML page that contains only a title and an image tag. Note that the SRC attribute of the image tag points to the second route of the application, which is where the magic happens.

The/video_feed route returns a streaming response. Because this stream returns the image to be displayed on the Web page, in the SRC attribute of the image label, the URL points to the route. Because most/all browsers support multi-part responses (if you find a browser that doesn't support this, let me know), the browser will automatically keep the image elements updated by displaying the JPEG image stream.

The generator function used in the/video_feed route is called Gen (), which takes an instance of the camera class as its argument. The MimeType parameter is set as shown above and has a content type of multipart/x-mixed-replace and a boundary string set to "frame".

The Gen () function enters a loop in which successive frames are returned from the camera as a response block. As shown above, this function requires the camera to provide a frame by calling the Camera.get_frame () method, and then generates the frame, using the Image/jpeg content type to format the frame as a response block.
Getting frames from the camera

Now all that is left is to implement the camera class, which must connect the camera hardware and download live video frames from it. The benefit of encapsulating this application hardware-related part in a class is that the class can have different implementations for different people, while the rest of the application remains the same. You can use this class as a device driver, providing a unified implementation regardless of the hardware device actually used.

Another advantage of separating the camera class from the rest of the app is that when there is no camera, it's easy to trick the application into thinking it has a camera because the camera class can be implemented as an analog camera without the need for real hardware. In fact, when I run this application, the simplest way is to test the flow to do those without worrying about the hardware until I have made the other parts work correctly. Below, you can see the simple analog camera implementation I used:

From time Import time Class Camera (object):  def __init__ (self):    self.frames = [Open (f + ". jpg ', ' RB '). Read () for F In [' 1 ', ' 2 ', ' 3 ']]   def get_frame (self):    return Self.frames[int (Time ())% 3]

This implementation reads three images from the disk, 1.jpg, 2.jpg, 3.jpg, and then repeats sequentially at the rate of one frame per second. The Get_frame () function uses the current time, in seconds, to determine which three frames to return at a given moment. Very simple, huh?

To run this analog camera, I need to create three frames. I used GIMP to make the following image:

Because the camera is analog, you can run the app in any environment! I've put all the files for this app on GitHub. If you are familiar with Git, you can clone it using the following command:

$ git clone https://github.com/miguelgrinberg/flask-video-streaming.git

If you like to download it, you can get a zip file here.

Once you have installed this app, create a virtual environment and install flask inside it. You can then run the application using the following command:

$ python app.py

When you enter http://localhost:5000 in your Web browser to launch the app, you'll see an analog video stream that plays images 1, 2, and 3 over and over. Pretty cool, huh?

Once, everything in the app was running, I started the Raspberry Pi and its camera module, and implemented a new camera class to turn the Raspberry Pi into a video streaming server, using the Picamera package to control the hardware. I'm not going to discuss the implementation of this camera class here, but you can find it in the camera_pi.py file in the source code.

If you have a Raspberry Pi and a camera module, you can edit the app.py file to import the camera class from this module, and then you can use the Raspberry Pi to stream the video in real-time, as I did in the following:

If you want to make this streaming application suitable for different cameras, all you have to do is implement different camera classes. I would appreciate it if you could end up writing a project on GitHub that was offered to me.
Limit of flow

The request period is short when the Flask application Server provides a regular request. The worker thread (Web worker) receives the request, invokes the handler function, and eventually returns the response. Once the response is sent back to the client, the worker thread is idle and ready to execute the next request.

When a request is received that uses streaming, the worker thread is bound to a client for the duration of the entire streaming transfer. When processing a long, endless stream, such as a video stream from a camera, the worker thread locks on a client until the client disconnects. This actually means that unless you take a special approach, the number of clients the application can serve is the same as the worker thread. When using the debug mode of the flask application, this means that there is only one worker thread, so you will not be able to connect two browser windows simultaneously to view data streams from two different places at the same time.

There are ways to overcome this important limitation. In my opinion, the best solution is to use a co-based Web server, such as Gevent,flask, to fully support it. By using the gevent, you can process multiple clients on a worker thread because gevent modifies the python I/O function to make the necessary context switches.
Conclusion

If you miss the above, the code contained in this article is placed in this GitHub library: https://github.com/miguelgrinberg/flask-video-streaming. Here, you can find a generic video streaming implementation without needing a camcorder, and there is also a Raspberry Pi camera module implemented.

I hope this article explains some of the topics about streaming technology. I focus on video streaming because this is an area where I have some experience, but in addition to streaming video, there are many other uses for transmission technology. For example, this technique can be used to maintain a long-time connection between the client and the server, allowing the server to push new information. These days, the network socket protocol is a more efficient way to achieve this, but the network sockets are quite new, only valid in modern browsers, and streaming technology can be used in any browser you can think of.

