PHP obtains hadoop resources through thrift

Introduction: This is a detailed page for PHP to obtain hadoop resources through thrift. It introduces PHP, hadoop, thrift, PHP, Java, hdfs php obtains knowledge, skills, experience, and some PHP Source Code related to hadoop resources through thrift.

Class = 'pingjiaf' frameborder = '0' src = 'HTTP: // biancheng.dnbc?info/pingjia.php? Id = 360990 'rolling = 'no'>
PHP can connect to hbase through thrift, and PHP can also read hadoop resources (HDFS resources) through thrift ).

Preparation:

PHP needs thrift libary

Packages: hadoop-0.20.2 \ SRC \ contrib \ thriftfs \ gen-PHP

Source code:

<? PHP $ globals ['thrift _ root'] = rootpath. '/lib/thrift'; require_once ($ globals ['thrift _ root']. '/thrift. PHP '); require_once ($ globals ['thrift _ root']. '/transport/tsocket. PHP '); require_once ($ globals ['thrift _ root']. '/transport/tbufferedtransport. PHP '); require_once ($ globals ['thrift _ root']. '/protocol/tbinaryprotocol. PHP '); require_once ($ globals ["thrift_root"]. "/packages/hadoopfs/thrifle thadoopfilesystem. PHP "); $ Hadoop_socket = new tsocket ("localhost", 59256); $ hadoop_socket-> setsendtimeout (10000); // ten seconds $ hadoop_socket-> setrecvtimeout (20000 ); // twenty seconds $ hadoop_transport = new tbufferedtransport ($ hadoop_socket); $ transport = new tbinaryprotocol ($ hadoop_transport); $ hadoopclient = new transport ($ hadoop_protocol ); $ hadoop_transport-> open (); try {// create directory $ d Irpathname = new hadoopfs_pathname (Array ("pathname" => "/user/root/hadoop"); If ($ hadoopclient-> exists ($ dirpathname) = true) {echo $ dirpathname-> pathname. "exists. \ n ";}else {$ result = $ hadoopclient-> mkdirs ($ dirpathname );} // put File $ filepathname = new hadoopfs_pathname (Array ("pathname" => $ dirpathname-> pathname. "/hello.txt"); $ localfile = fopen ("hello.txt", "rb"); $ hdfsfile = $ hadoopcl Ient-> Create ($ filepathname); While (true) {$ DATA = fread ($ localfile, 1024); If (strlen ($ data) = 0) break; $ hadoopclient-> write ($ hdfsfile, $ data) ;}$ hadoopclient-> close ($ hdfsfile); fclose ($ localfile); // get fileecho "read file: \ n "; print_r ($ filepathname); $ DATA =" "; $ hdfsfile = $ hadoopclient-> open ($ filepathname); print_r ($ hdfsfile); While (true) {$ DATA = $ hadoopclient-> Read ($ hdfsfile, 0, 1024); If (Str Len ($ data) = 0) break; print $ data ;}$ hadoopclient-> close ($ hdfsfile); echo "liststatus: \ n "; $ result = $ hadoopclient-> liststatus ($ dirpathname); print_r ($ result); foreach ($ result as $ key => $ value) {if ($ value-> isdir = "1") print "dir \ t"; elseprint "file \ t"; print $ value-> block_replication. "\ t ". $ value-> length. "\ t ". $ value-> modification_time. "\ t ". $ value-> permission. "\ t ". $ value-> Owner. "\ t ". $ value-> group. "\ t ". $ value-> path. "\ n" ;}$ hadoop_transport-> close () ;}catch (exception $ e) {print_r ($ e) ;}?>

Start Thrift of hadoop

Hadoop-0.20.2 \ SRC \ contrib \ thriftfs \ scripts \ start_thrift_server.sh 59256

Problem One:

Create a file in the system directory instead of the hadoop directory.

Cause:

Load the default configuration file when thrift starts.

Solution:

Modify the start_thrift_server.sh File

Top =/usr/local/hadoop-0.20.2

Classpath = $ classpath: $ top/Conf

Problem two:

Java. Lang. nullpointerexception

At org. Apache. hadoop. thriftfs. hadoopthriftserver $ hadoopthrifle thandler. Write (hadoopthriftserver. Java: 282)

At org. Apache. hadoop. thriftfs. API. thrifle thadoopfilesystem $ processor $ write. Process (unknown source)

At org. Apache. hadoop. thriftfs. API. thrithadoopfilesystem $ processor. Process (unknown source)

At com. Facebook. Thrift. server. tthreadpoolserver $ workerprocess. Run (unknown source)

At java. util. Concurrent. threadpoolexecutor $ worker. runtask (threadpoolexecutor. Java: 886)

At java. util. Concurrent. threadpoolexecutor $ worker. Run (threadpoolexecutor. Java: 908)

At java. Lang. thread. Run (thread. Java: 662)

Cause:

The map hash ID returned by Java is of the long type, while PhP (32-bit) cannot store the long type of data, resulting in loss of precision after being converted to float data.

Private long nextid = new random (). nextlong ();

Java returned data: 4207488029786584864

PHP Data Retrieval: 4.2074880297866e + 18

Java obtains the data transmitted by PHP: 4207488029786585088

Solution:

Modify hadoop-0.20.2 \ SRC \ contrib \ thriftfs \ If \ hadoopfs. Thrift File

Modify

Struct thrithandle {

I64 ID

}

Is

Struct thrithandle {

String ID

}

Regenerate PHP packages

Thrift -- Gen PHP hadoopfs. Thrift

Modify the org. Apache. hadoop. thriftfs. API. thrithandle class

Modify

Public long ID;

Is:

Public String ID;

Modify the corresponding Program

Org. Apache. hadoop. thriftfs. hadoopthriftserver

Modify

Long id = insert (out );

Thrithandle OBJ = new thrithandle (ID );

Is

Long id = insert (out );

String _ id = string. valueof (ID );

Thrithandle OBJ = new thrithandle (_ id );

Modify the corresponding program

Re-package and start hadoop's thrift:

Hadoop-0.20.2 \ SRC \ contrib \ thriftfs \ scripts \ start_thrift_server.sh 59256

In this way, PHP can connect to and obtain hadoop resources.

Love J2EE follow Java Michael Jackson video station JSON online tools

Http://biancheng.dnbcw.info/php/360990.html pageno: 1.