Stream media transmission using the Python Flask framework

This article mainly introduces how to use the Python Flask framework to transmit video streaming media, including digital stream transmission from camera frames to web browsers, if you need it, refer to Flask as a Web development microframework implemented by Python. This article is a detailed tutorial on how to use it to transmit video data streams.

I'm sure you already know that I published a book and some video materials about Flask on O'Reilly Media. In these cases, the coverage of the Flask framework is quite complete. for some reason, there are also a small number of features not mentioned much, therefore, I think it is a good idea to write an article about them here.

This article specifically introduces streaming media. this interesting feature gives Flask applications the ability to efficiently provide data for large requests by dividing them into small data blocks, this may take a long time. To illustrate this topic, I will show you how to build a real-time video streaming media server!
What is streaming media?

Streaming media is a technology in which the server responds to requests in the form of data blocks. I can think of a reason to explain why this technology may be useful:

Very large response. For very large responses, the responses collected in the memory are only returned to the client, which is very inefficient. Another method is to write the response to the disk and then use flask. send_file () to return the file, but this adds a combination of I/O. Assuming that data can be generated in multiple parts, it is a better solution to provide responses to requests in the form of small pieces of data.
Real-time data. For some applications, the data returned by the request comes from the real-time data source. A good example is to provide a real-time video or audio. Many security cameras use this technology to transmit video data streams to Web browsers.

Use Flask for stream transmission

Flask provides local support for stream response by using generator functions. A generator is a special function that can be interrupted and restored. Consider the following functions:

 def gen():    yield 1    yield 2    yield 3

This is a three-step function, and a value is returned for each step. The description of how the generator is implemented is beyond the scope of this article, but if you are a little curious, the following shell session will show you how the generator is used:

 >>> x = gen()>>> x
 
  >>> x.next()1>>> x.next()2>>> x.next()3>>> x.next()Traceback (most recent call last): File "
  
   ", line 1, in 
   
    StopIteration
   
  
 

In this simple example, you can see that a generator function can return multiple results in sequence. Flask uses the generator function to implement stream transmission.

The following example shows how to use stream transmission to generate a large data table without placing the entire table into the memory:

 from flask import Response, render_templatefrom app.models import Stock def generate_stock_table():  yield render_template('stock_header.html')  for stock in Stock.query.all():    yield render_template('stock_row.html', stock=stock)  yield render_template('stock_footer.html') @app.route('/stock-table')def stock_table():  return Response(generate_stock_table())

In this example, you can see how Flask and generator functions work together. The route (route) that returns the stream Response needs to return a Response object initialized by the generator function. Flask then calls the generator and sends the result to the client in multipart mode.

For this special example, if you assume Stock. query. the database query result returned by all () is an iterator, so you can generate a row of a potentially large table at a time. therefore, no matter how many characters are in the query, in Python, memory consumption does not increase because of large response strings.
Multi-part response

The example of a table above generates a traditional web page in the form of small blocks, and each part is connected to the final result. This is a good example of how to generate a large response, but the most exciting thing is to process real-time data.

An interesting application that uses stream transmission is to use each block to replace the original place on the page, which allows the stream to form an animation in the browser window. With this technology, you can make every data block in the stream an image, which provides you with a cool video input signal running in the browser!

The secret to local update is to use multiple responses. A multipart response consists of a header and many parts (parts. The header contains a content type in multiple parts. the subsequent parts are separated by the boundary mark. each part contains a specific content type in its own part.

For different requirements, there are multiple content types. For Stream transmission, each part must replace the previous part with the multipart/x-mixed-replace content type. To help you understand what it looks like, there is a response structure for transmission of multiple video streams:

 HTTP/1.1 200 OKContent-Type: multipart/x-mixed-replace; boundary=frame --frameContent-Type: image/jpeg 
 
  --frameContent-Type: image/jpeg 
  
   ...
  
 

As you can see above, this structure is very simple. The main Content-Type header is set to multipart/x-mixed-replace, and the boundary mark is also defined. Then, each part contains the prefix of two dashes and the boundary string on this line. Each part has its own Content-Type header, and each part can include an optional Content-Length header indicating the Length of the byte of the part where the payload is located, but at least for the image browser, it can process streams with no length.
Create a real-time video streaming media server

There are enough theories in this article. now it is time to build a complete application to stream real-time videos to Web browsers.

There are many ways to stream videos to a browser, and each method has its advantages and disadvantages. A good way to work with Flask stream features is stream transmission of Independent JPEG image sequences. This is dynamic JPEG. This is used by many IP surveillance cameras. This method has a short latency, but the transmission quality is not the best, because JPEG compression is not very effective for dynamic images.

The following shows a simple but complete Web application. It provides a dynamic JPEG stream transmission:

 #!/usr/bin/env pythonfrom flask import Flask, render_template, Responsefrom camera import Camera app = Flask(__name__) @app.route('/')def index():    return render_template('index.html') def gen(camera):  while True:    frame = camera.get_frame()    yield (b'--framern'        b'Content-Type: image/jpegrnrn' + frame + b'rn') @app.route('/video_feed')def video_feed():  return Response(gen(Camera()),          mimetype='multipart/x-mixed-replace; boundary=frame') if __name__ == '__main__':  app.run(host='0.0.0.0', debug=True)

This application imports a Camera class to provide the frame sequence. In this example, it is a good idea to put the camera control part into a separate module. In this way, Web applications are kept clean, simple, and universal.

The application has two routes ). /Serving as the main page, which is defined in the index.html template. You can see the content in this template file below:

    Video Streaming Demonstration    Video Streaming Demonstration   

This is a simple HTML page that contains only one title and image tag. Note that the src attribute of the image label points to the second route of the application. this is where magic occurs.

/Video_feed routes return streaming responses. Because this stream returns the image to be displayed on the web page, in the src attribute of the image tag, the URL points to this route. Because most/all browsers support multi-part responses (if you find a browser that does not support this, please let me know), the browser automatically keeps the image elements updated by displaying JPEG image streams.

The generator function used in/video_feed routing is gen (), and an instance of the Camera class is used as its parameter. The mimetype parameter is set as shown above and has the content type of multipart/x-mixed-replace and the boundary string set as "frame.

The gen () function enters a loop in which consecutive frames returned from camera are used as response blocks. As shown above, this function requires camera to provide frames by calling the camera. get_frame () method, then generate frames, and convert the frames into response blocks using the image/jpeg content type.
Get frame from camera

Now all that is left is the implementation of the Camera class, which must connect to the Camera hardware and download real-time video frames from it. The advantage of encapsulating the hardware-related part of the application in a class is that the class can be implemented differently for different people, while the rest of the application remains unchanged. You can use this class as a device driver to provide a unified implementation regardless of the hardware in use.

Another advantage of separating the Camera class from the rest of the application is that when there is actually no Camera, it is easy to cheat the application and let it think that there is a Camera here, because the camera class can be implemented as a simulated camera without real hardware. In fact, when I run this application, the simplest way is to test what a stream can do without worrying about hardware until I have enabled other parts to run correctly. Below, you can see the simple simulated camera implementation I use:

 from time import time class Camera(object):  def __init__(self):    self.frames = [open(f + '.jpg', 'rb').read() for f in ['1', '2', '3']]   def get_frame(self):    return self.frames[int(time()) % 3]

In this example, we can read three images from images: 1.JPG, 2.jpg, 3.jpg, and then return them at the same rate of one frame per second. The get_frame () function uses the current time and seconds to determine which three frames are returned at a given time point. Is it easy?

To run this simulated camera, I need to create three frames. I used gimp to make the following image:

Because the camera is simulated, you can run this application in any environment! I put all the files of this application on GitHub. If you are familiar with git, you can use the following command to clone it:

 $ git clone https://github.com/miguelgrinberg/flask-video-streaming.git

If you like to download it, you can get a zip file here.

After installing the application, create a virtual environment and install Flask in it. Then you can run the application using the following command:

 $ python app.py

When you enter http: // localhost: 5000 in your Web browser to start this application, you will see the simulated video stream playing images 1, 2, 3 over and over again. Cool, right?

Once, everything in the application was running. I started Raspberry Pi and its Camera module, and implemented a new Camera class to turn Raspberry Pi into a video streaming media server, use the picamera package to control the hardware. I won't discuss the implementation of this camera class here, but you can find it in the camera_pi.py file in the source code.

If you have a Raspberry Pi and a camera module, you can edit the app. the py file imports the Camera class from this module, and then you can use Raspberry Pi to transmit video streams in real time, as I did in the following:

If you want to apply the stream transmission application to different cameras, you must implement different Camera classes. I would be grateful if you could write a Github project and provide it to me.
Stream restrictions

When the Flask application server provides regular requests, the request cycle is short. A working thread (web worker) receives a request, calls a processing function, and finally returns a response. Once the response is sent back to the client, the working thread is idle and is ready to execute the next request.

When a request for stream transmission is received, the worker thread is bound to a client during the duration of the whole stream transmission. When processing a long and endless stream, such as a video stream from the camera, the working thread will be locked in a client until the client is disconnected. This actually means that, unless special measures are taken, the number of clients that the application can serve is the same as the number of working threads. When using the debug mode of the Flask application, this means that there is only one working thread, so you will not be able to connect two browser windows at the same time to view data streams from two different places.

There are ways to overcome this important limitation. In my opinion, the best solution is to use coroutine-based Web servers such as gevent and Flask. You can use coroutine gevent to process multiple clients in one worker thread because gevent modifies Python I/O functions for necessary context switching.
Conclusion

If you miss the above content, the code contained in this article is stored in this GitHub Library: https://github.com/miguelgrinberg/flask-video-streaming. Here, you can find a common video stream transmission implementation without a camera, and there is also a Raspberry Pi camera module implementation.

I hope this article will discuss some stream technology topics. I am focused on video stream transmission because it is an area I have some experience in, but in addition to streaming video, streaming technology has many other uses. For example, this technology can be used to maintain a long connection between the client and the server, allowing the server to push new information. In these days, the network socket protocol is more effective, but the network socket is quite new and only valid in modern browsers, the stream transmission technology can be used in any browser you can think.