Example of Avro using php to request the python Service

Avro is one of the Hadoop projects. It is mainly used to support data-intensive applications. It defines a data format and supports this format in multiple programming languages. We primarily use Cassandra and RPC to implement mutual calls between languages. There are many lightweight python scripts in the architecture. For example, if PHP receives a URL, it needs to call

Avro is one of the Hadoop projects. It is mainly used to support data-intensive applications. It defines a data format and supports this format in multiple programming languages. We primarily use Cassandra and RPC to implement mutual calls between languages. There are many lightweight python scripts in the architecture. For example, if PHP receives a URL, it needs to call

Avro is one of the Hadoop projects. It is mainly used to support data-intensive applications. It defines a data format and supports this format in multiple programming languages. We primarily use Cassandra and RPC to implement mutual calls between languages.

There are many lightweight python scripts in the architecture. For example, if PHP receives a URL, it needs to call the Python script to obtain the URL title and then return it. In the past, cli was called directly from the command line. The results were not very good and often got stuck. Sometimes, PHP ports were occupied, which made php-fpm unable to be started. In this scenario, asynchronous processing such as RabbitMQ cannot be used. So we implemented a simple call with Avro and ran it for more than a month. Now everything is normal. Now we can share the relevant code.

Principle: PHP and Python use the same Schema and use the Python HTTPServer as an http service. The PHP end encodes the data in avro/binary mode and transmits it to Python, the Python end decodes the transmitted data in the same way. After processing the data, it returns the result to the PHP end.

The Python module and PHP library officially provided by avro are required. When installing avro with pip, you must first install libsnappy.

Schema

Create a JSON file named avro-protocol.json with the following content:

{"namespace": "com.dmyz.avro",    "protocol": "Post",    "types": [        {"name": "Action", "type": "record",            "fields": [                {"name": "url","type": "string"},                {"name": "charset","type": "string"}            ]        }    ],    "messages": {        "send": {            "request": [{"name": "message", "type": "Action"}],            "response": "string"        }    }}

The preceding content defines an Avro protocol. The Python side accepts parameters and generates services based on their definitions. The PHP side does not provide services, therefore, only schema (type node) in the definition is used ).

Python

#! /Usr/bin/env python #-*-coding: UTF-8-*-from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServertry: from cStringIO import StringIOexcept ImportError: from StringIO import StringIO? Import cgiimport jsonfrom avro import io, ipc, protocol, schema? PROTOCOL_FILE = 'avro-protocol. json 'Protocol _ JSON = json. loads (open (PROTOCOL_FILE ). read () PROTOCOL = protocol. parse (open (PROTOCOL_FILE ). read () SCHEMA = schema. parse (json. dumps (PROTOCOL_JSON ['types'] [0])? SERVER_ADDR = ('localhost', 9090 )? Class PostRequestHandler (BaseHTTPRequestHandler): def do_POST (self ):? Length = int (self. headers. getheader ('content-length') post_data = self. rfile. read (length )? Data_reader = StringIO (post_data) decoder = io. BinaryDecoder (data_reader )? Datum_reader = io. datumReader (SCHEMA) data = datum_reader.read (decoder) url = data ['url']) charset = data ['charset'] # data sent from PHP has been obtained, execute other logic self here. wfile. write ('title') # Return the obtained title? If _ name _ = '_ main _': server = HTTPServer (SERVER_ADDR, PostRequestHandler) server. allow_reuse_address = True server. serve_forever ()

PHP

? $ Avro = json_decode (file_get_contents ('avro-protocol. json'), true );? $ AvroProtocol = new AvroProtocol (); $ protocol = $ avroProtocol-> real_parse ($ avro); $ schema = AvroSchema :: real_parse ($ avro ['types'] [0]);? $ Datum_writer = new AvroIODatumWriter ($ schema); $ write_io = new AvroStringIO (); $ encoder = new AvroIOBinaryEncoder ($ write_io );? $ Message = array ('url' => 'HTTP: // dmyz.org ', 'charset' => 'utf-8'); $ datum_writer-> write ($ message, $ encoder );? $ Content = $ write_io-> string ();? $ Headers = array ("POST/HTTP/1.1", "Host :". self: $ host, "Content-Type: avro/binary", "Content-Length :". strlen ($ content ),);? $ Socket = stream_socket_client ('localhost: 100', $ errno, $ errstr, 5 );? If (! $ Socket) throw new Exception ($ errstr, $ errno );? Fwrite ($ socket, implode ("\ r \ n", $ headers). "\ r \ n"); fwrite ($ socket, $ content );? $ Result = ''; while (! Feof ($ socket) {$ result. = fgets ($ socket) ;}fclose ($ socket );? Return $ result; # obtain the data returned by Python.

Original article address: Avro uses php to request the python service. Thank you for sharing it with me.