Tutorial on completing video processing tasks with the Python Django framework

Stickyworld's web App already supports video dialing for some time, but it's all done via YouTube's embed mode. We started to provide new versions to support video operations, allowing our users to not be subject to YouTube's services.

I've worked on a project in the past, and the customer needs a video transcoding feature, which is not an easy requirement. It takes a lot of reading each video, audio and video container format and then output the video format that conforms to the Web usage and preferences.

With this in mind, we decided to hand over the transcoding work to encoding.com. This website can allow you to encode 1GB size videos for free, and files exceeding 1GB capacity will be taken with tiered pricing fees.

The code developed below, I uploaded a 178KB capacity two-second video to test whether the code worked successfully. I continue to test other larger external files after the test process has not had any exception errors.

Stage One: Users upload video files

Now this new code snippet provides a HTML5-based and quick-to-start upload mechanism. Code written in Coffeescript can upload files from the client to the server side.

$scope. Upload_slide = (upload_slide_form),  file = document.getElementById ("Slide_file"). Files[0]  reader = new FileReader ()  reader.readasdataurl file  reader.onload = (event),   result = Event.target.result   fileName = document.getElementById ("Slide_file"). Files[0].name   $.post "/world/upload_ Slide ",    data:result    name:filename    room_id: $scope. Room.id    (response_data),     if response _data.success? Is isn't yes      Console.error "There was an error uploading the file", Response_data     else      console.log "Upload su Ccessful ", response_data  reader.onloadstart =   console.log" Onloadstart "  reader.onprogress = ( Event)   , Console.log "OnProgress", Event.total, event.loaded, (event.loaded/event.total) *  Reader.onabort =-   console.log "Onabort"  reader.onerror =   console.log "onerror"  Reader.onloadend = (event)   , Console.log "Onloadend", event

It is best to pass ("Slide_file"). Files and upload each file via a separate post instead of uploading all the files by a post requirement. We will explain this later.

Phase two: Verify and upload to Amazon S3

Back end we ran Django and RABBITMQ. The main modules are as follows:

$ pip Install ' django>=1.5.2 ' django-celery>=3.0.21 ' \ ' django-storages>=1.1.8 ' lxml>=3.2.3 ' python-magic>=0.4.3 ' I have built two modules: Slideuploadqueue is used to store each uploaded data, Slidevideomedia is used to store each of the data to upload the movie. Class Slideuploadqueue (models. Model): Created_by = models. ForeignKey (User) Created_time = models. Datetimefield (db_index=true) Original_file = models. Filefield (Upload_to=filename_sanitiser, Blank=true, default= ") Media_type = models. ForeignKey (mediatype) Encoding_com_tracking_code = models. Charfield (default= ", max_length=24, blank=true) status_awaiting_data = 0 status_awaiting_processing = 1 STATUS_PR ocessing = 2 Status_awaiting_3rd_party_processing = 5 Status_finished = 3 Status_failed = 4 Status_list = ((STATU S_awaiting_data, ' awaiting DATA '), (status_awaiting_processing, ' awaiting processing '), (status_processing, ' Process    ing '), (status_awaiting_3rd_party_processing, ' awaiting 3rd-party processing '), (status_finished, ' FINISHED '), (STAtus_failed, ' FAILED '), status = models.    Positivesmallintegerfield (Default=status_awaiting_data, choices=status_list) class meta:verbose_name = ' Slide ' verbose_name_plural = ' Slide upload queue ' def save (self, *args, **kwargs): If not self.created_time:self.cre Ated_time = \ Datetime.utcnow (). Replace (TZINFO=PYTZ.UTC) return super (Slideuploadqueue, self). Save (*args, **kwa  RGS) def __unicode__ (self): If Self.id is None:return ' new
 
  
' Return ' 
  
   
    %d '% self.id class Slidevideomedia (models. Model): converted_file = models. Filefield (Upload_to=filename_sanitiser, Blank=true, default= ") format_mp4 = 0 FORMAT_WEBM = 1 FORMAT_OGG = 2 for  MAT_FL9 = 3 Format_thumb = 4 Supported_formats = ((Format_mp4, ' MPEG 4 '), (FORMAT_WEBM, ' WEBM '), (Format_ogg, ' OGG '), (FORMAT_FL9, ' Flash 9 Video '), (Format_thumb, ' Thumbnail '), Mime_types = ((Format_mp4, ' Video/mp4 ') ), (FORMAT_WEBM, ' VIDEO/WEBM '), (Format_ogg, ' Video/ogg '), (FORMAT_FL9, ' Video/mp4 '), (Format_thumb, ' IMAGE/JP (eg '),) format = models.     Positivesmallintegerfield (Default=format_mp4, choices=supported_formats) class meta:verbose_name = ' Slide video ' verbose_name_plural = ' Slide videos ' def __unicode__ (self): If Self.id is None:return ' new 
    
     ' re Turn ' 
     
      %d '% self.id
      
 
     

   
  
 

All of our modules use Filename_sanitiser. Filefield automatically adjusts the file name to / . format. Organize each file name and ensure its uniqueness. We use a time-sensitive, signed URL column that allows us to control which users are using our services for how long.

def filename_sanitiser (instance, filename):  folder = Instance.__class__.__name__.lower ()  ext = ' jpg '   if ' . ' In filename:    t_ext = Filename.split ('. ') [ -1].strip (). Lower ()     if t_ext! = ":      ext = t_ext   return '%s/%s.%s '% (folder, str (UUID.UUID4 ()), ext)

The file to be tested Testing.mov will be converted to the following URL: https://our-bucket.s3.amazonaws.com/slideuploadqueue/ 3fe27193-e87f-4244-9aa2-66409f70ebd3.mov and uploaded via the Django storages module.

We use Magic to verify the files uploaded from the user-side browser. Magic can detect what type of file is from the file content.

@verify_auth_token @return_jsondef upload_slide (Request):  File_data = Request. Post.get (' data ', ')  file_data = Base64.b64decode (File_data.split ('; base64, ') [1])  description = Magic.from_ Buffer (File_data)

If the file type conforms to the MPEG V4 system or Apple QuickTime movie, we know that the transcoding of the file is not too much of a problem. If the format is not mentioned above, we will sign it to the user.
Next, we will save the video to the queue via the Slideuploadqueue module and send a request to RabbitMQ. Because we used the Django Storages module, the files are automatically uploaded to Amazon S3.

Slide_upload = Slideuploadqueue () ... slide_upload.status = Slideuploadqueue.status_awaiting_processingslide_ Upload.Save () slide_upload.original_file.\  Save (' anything.%s '% File_ext, ContentFile (file_data)) Slide_ Upload.Save () task = Convertrawslidetoslide () task.delay (slide_upload)

Phase 3: Send video to a third party.

RabbitMQ will control the call of the Task.delay (slide_upload).
We now only need to send the video file URL with the output format to encoding.com. The site will reply to one of our work codes and let us check the progress of the video transcoding.

Class Convertrawslidetoslide (Task): queue = ' backend_convert_raw_slides ' ... def _handle_video (self, slide_upload): mp4 = {' Output ': ' mp4 ', ' size ': ' + ', ' bitrate ': ' 256k ', ' audio_bitrate ': ' 64k ', ' Audio_cha Nnels_number ': ' 2 ', ' keep_aspect_ratio ': ' Yes ', ' video_codec ': ' MPEG4 ', ' profile ': ' Main ', ' Vcodecpara Meters ': ' No ', ' audio_codec ': ' Libfaac ', ' two_pass ': ' No ', ' cbr ': ' No ', ' deinterlacing ': ' No ', ' K ' Eyframe ': ' + ', ' audio_volume ': ' + ', ' file_extension ': ' mp4 ', ' hint ': ' No ',} WEBM = {' OUTP UT ': ' webm ', ' size ': ' 256k ', ' bitrate ': ' 64k ', ' audio_sample_rate ': ' 44100 ', ' audio_channels_number ': ' 2 ', ' keep_aspect_ratio ': ' Yes ', ' video_codec ': ' libvpx ', ' profile ': ' Base Line ', ' vcodecparameters ': ' No ', ' audio_codec ': ' Libvorbis ', ' two_pass ': ' No ', ' cbr ': ' No ', ' Dein Terlacing ': ' No ', 'Keyframe ': ' + ', ' audio_volume ': ' + ', ' preset ': ' 6 ', ' file_extension ': ' webm ', ' acbr ': ' No ',} Ogg = {' Output ': ' Ogg ', ' size ': ' 256k ', ' bitrate ': ' 64k ', ' Audio_ ' Sample_rate ': ' 44100 ', ' audio_channels_number ': ' 2 ', ' keep_aspect_ratio ': ' Yes ', ' video_codec ': ' Libtheora '      , ' profile ': ' Baseline ', ' vcodecparameters ': ' No ', ' audio_codec ': ' Libvorbis ', ' two_pass ': ' No ',       ' CBR ': ' No ', ' deinterlacing ': ' No ', ' keyframe ': ' A ', ' audio_volume ': ' The ', ' file_extension ': ' Ogg ', ' ACBR ': ' No ', ' flv = {' Output ': ' fl9 ', ' size ': ' + ', ' bitrate ': ' 256k ', ' audio_bit Rate ': ' 64k ', ' audio_channels_number ': ' 2 ', ' keep_aspect_ratio ': ' Yes ', ' video_codec ': ' libx264 ', ' pro      File ': ' High ', ' vcodecparameters ': ' No ', ' audio_codec ': ' Libfaac ', ' two_pass ': ' No ', ' cbr ': ' No ', ' deinterlacing ': ' No ',     ' Keyframe ': ' + ', ' audio_volume ': ' + ', ' file_extension ': ' mp4 ',} thumbnail = {' output ': ' t    Humbnail ', ' Time ': ' 5 ', ' Video_codec ': ' mjpeg ', ' keep_aspect_ratio ': ' Yes ', ' file_extension ': ' jpg ', } encoder = Encoding (settings. encoding_api_user_id, settings. Encoding_api_user_key) resp = Encoder.add_media (Source=[slide_upload.original_file.url], Formats=[mp4, WEBM, OGG, flv, thumbnail]) media_id = None If Resp is not none and resp.get (' response ') are not none:media_id = resp.ge T (' response '). Get (' MediaID ') if media_id is None:slide_upload.status = slideuploadqueue.status_failed slide _upload.save () Log.error (' Unable to communicate with encoding.com ') return False slide_upload.encoding_com_t Racking_code = media_id Slide_upload.status = \ slideuploadqueue.status_awaiting_3rd_party_processing SLIDE_UPL Oad.save () return True

Encoding.com recommends some useful Python programs that can be used to communicate with their services. I have modified some parts of the module, but I still need to modify some functions to achieve my satisfactory status. Here is the program code that is currently in use after the modification:

Import httplibfrom lxml import etreeimport urllibfrom xml.parsers.expat import expaterrorimport xmltodict encoding_api_ URL = ' manage.encoding.com:80 ' class Encoding (object): Def __init__ (self, userid, UserKey, Url=encoding_api_url): Sel    F.url = URL Self.userid = UserID Self.userkey = UserKey def get_media_info (self, action= ' getmediainfo ', ids=[], headers={' content-type ': ' application/x-www-form-urlencoded '}): query = etree. Element (' query ') nodes = {' UserID ': Self.userid, ' UserKey ': self.userkey, ' action ': Action, ' Medi Aid ': ', '. Join (IDS),} query = Self._build_tree (etree. Element (' query '), nodes) results = self._execute_request (query, headers) return Self._parse_results (results) def Get_status (self, action= ' GetStatus ', ids=[], extended= ' no ', headers={' content-type ': ' application/ X-www-form-urlencoded '}): query = etree. Element (' query ') nodes = {' UserID ': Self.userid, ' UserKey ': self.userkey, ' action ': ACtion, ' extended ': Extended, ' mediaid ': ', '. Join (IDS),} query = Self._build_tree (etree. Element (' query '), nodes) results = self._execute_request (query, headers) return Self._parse_results (results) def Add_media (self, action= ' Addmedia ', source=[], notify= ', formats=[], instant= ' no ', headers={' content-type ': ' Applica Tion/x-www-form-urlencoded '}): query = etree. Element (' query ') nodes = {' UserID ': Self.userid, ' UserKey ': self.userkey, ' action ': action, ' sour Ce ': source, ' Notify ': Notify, ' instant ': instant,} query = Self._build_tree (etree. Element (' query '), nodes) for format in Formats:format_node = Self._build_tree (etree. Element (' format '), format) query.append (format_node) results = self._execute_request (query, headers) return SE        Lf._parse_results (results) def _build_tree (Self, node, data): For K, V in Data.items (): If Isinstance (V, list): for item in v:element = ETRee. Element (k) Element.text = Item Node.append (element) else:element = etree. Element (k) Element.text = v node.append (Element) return node Def _execute_request (self, XML, headers, Path= ', method= ' POST '): params = Urllib.urlencode ({' xml ': etree.tostring (XML)}) conn = Httplib. Httpconnection (Self.url) conn.request (method, path, params, headers) response = Conn.getresponse () data = Respons    E.read () Conn.close () return Data def _parse_results (self, results): Try:return xmltodict.parse (results) Except Expaterror, E:print ' Error parsing encoding.com response ' Print e return None

Other pending items include the use of encoding.com rigorous SSL authentication via Https-only (encrypted online), and some unit tests.
Phase 4: Download all new video file formats

We have a regularly executed program that checks the progress of the video transcoding every 15 seconds through RABBITMQ:

Class Checkuponthirdparties (Periodictask):  run_every = Timedelta (seconds=settings. Third_party_check_up_interval)  ...  def _handle_encoding_com (self, slides):    format_lookup = {      ' mp4 ': slidevideomedia.format_mp4,      ' WEBM ': SLIDEVIDEOMEDIA.FORMAT_WEBM,      ' ogg ': Slidevideomedia.format_ogg,      ' fl9 ': slidevideomedia.format_fl9,      ' Thumbnail ': Slidevideomedia.format_thumb,    }     encoder = Encoding (settings. encoding_api_user_id,      settings. Encoding_api_user_key)     job_ids = [Item.encoding_com_tracking_code for item in slides]    resp = encoder.get_ Status (Ids=job_ids)     If RESP is None:      log.error (' Unable to check up on encoding.com ')      return False

Check the response of the encoding.com to verify that each part is correct to facilitate our continuation.

If Resp.get (' response ') is None:log.error (' Unable to get response node from encoding.com ') return False resp_id = resp.  Get (' response '). Get (' id ') if resp_id is None:log.error (' unable-get media ID from encoding.com ') return False slide = SlideUploadQueue.objects.filter (status=slideuploadqueue.status_awaiting_3rd_party_processing, Encoding_com_ TRACKING_CODE=RESP_ID) If Len (slide)! = 1:log.error (' Unable to find a single record for%s '% resp_id) return False Res   P_status = resp.get (' response '). Get (' status ') if Resp_status is None:log.error (' unable-get status from Encoding.com ') return False if Resp_status! = U ' finished ': Log.debug ("%s isn ' t finished, would check back later"% resp_id) return Tru E formats = resp.get (' response '). Get (' format ') if formats is None:log.error ("No output formats were-found.  Something ' s wrong. ") Return False for format in Formats:try:assert format.get (' status ') = = U ' finished ', \ '%s is not finished. Something ' s wrong. "% Format.get (' id ') output = format.get (' output ') Assert output in (' mp4 ', ' WEBM ', ' ogg ', ' fl9 ', ' thumbnail '), ' Unknown OUTP UT format%s '% output s3_dest = format.get (' s3_destination ') assert ' http://encoding.com.result.s3.amazonaws.com/' \ in S3_dest, ' suspicious S3 url:%s '% s3_dest Https_link = \ ' Https://s3.amazonaws.com/encoding.com.result /%s '%\ s3_dest.split ('/') [-1] file_ext = Https_link.split ('. ') [ -1].strip () Assert Len (file_ext) > 0,\ ' Unable to get file extension from%s '% https_link count = Slidev IdeoMedia.objects.filter (Slide_upload=slide, Format=format_lookup[output]). COUNT () if count! = 0:print ' The Re is already a%s file for this slide '% output continue content = Self.download_content (Https_link) assert Content is not none,\ ' there are no content for%s '% format.get (' id ') except Assertionerror, E:log.error (' A form At didn't pass all assertions:%s '% e) continue

Here we have confirmed that everything is normal, so we can store all the video files:

Media = Slidevideomedia () Media.format = Format_lookup[output]media.converted_file.save (' blah.%s '% file_ext, ContentFile (content)) Media.save ()

Stage 5: Play video files via HTML5

A Web page with a HTML5 image unit has been added to our front-end page. The video is displayed with a video.js that has the best support for each browser.

? Bower Install Video.jsbower caching git://github.com/videojs/video.js-component.gitbower cloning git://github.com/ Videojs/video.js-component.gitbower fetching video.jsbower checking out Video.js#v4.0.3bower copying/home/mark/. Bower/cache/video.js/5ab058cd60c5615aa38e8e706cd0f307bower Installing video.js#4.0.3

On our homepage There are other dependent files included:

!!! 5html (lang= "en", class= "No-js") head  Meta (http-equiv= ' Content-type ', content= ' text/html; Charset=utf-8 ')  ...  Link (rel= ' stylesheet ', type= ' text/css ', href= '/components/video-js-4.1.0/video-js.css ')  script (type= ' text/ JavaScript ', src= '/components/video-js-4.1.0/video.js ')

In the module under the angular.js/jade-based frame, we introduce the volume label and its sub-volume label. Each video file will have a thumbnail displayed by the poster component of the volume label, and the thumbnail image is captured by us from the first few seconds of the video.

#main. Span12  Video#example_video_1.video-js.vjs-default-skin (controls, preload= "Auto", width= "640", height= " ", poster=" {{video_thumbnail}} ", data-setup= ' {" Example_option ": true} ', ng-show=" videos ")    Source (ng-repeat= "Video in Videos", src= "{{video.src}}", Type= "{{video.type}}")

It will also show us the conversion of each video file format and use it in the tag. Video.js will decide which format video to play based on the browser used by the user.

We still have a lot of work to do, a process for building unit tests and communicating with encoding.com services. If you are interested in these jobs, please contact me.

