php通過thrift擷取hadoop資源

簡介：這是php通過thrift擷取hadoop資源的詳細頁面，介紹了和php,hadoop, thrift, php, java, hdfs php通過thrift擷取hadoop資源有關的知識、技巧、經驗，和一些php源碼等。

class='pingjiaF' frameborder='0' src='http://biancheng.dnbcw.info/pingjia.php?id=360990' scrolling='no'>
php可以通過thrift串連hbase，同樣php可以通過thrift讀取hadoop資源(HDFS資源)。

準備：

php需要thrift的libary

packages:hadoop-0.20.2\src\contrib\thriftfs\gen-php

源碼：

<?php$GLOBALS['THRIFT_ROOT'] = ROOTPATH . '/lib/thrift';require_once($GLOBALS['THRIFT_ROOT'].'/Thrift.php');require_once($GLOBALS['THRIFT_ROOT'].'/transport/TSocket.php');require_once($GLOBALS['THRIFT_ROOT'].'/transport/TBufferedTransport.php');require_once($GLOBALS['THRIFT_ROOT'].'/protocol/TBinaryProtocol.php');require_once($GLOBALS["THRIFT_ROOT"] . "/packages/hadoopfs/ThriftHadoopFileSystem.php");$hadoop_socket = new TSocket("localhost", 59256);$hadoop_socket -> setSendTimeout(10000); // Ten seconds$hadoop_socket -> setRecvTimeout(20000); // Twenty seconds$hadoop_transport = new TBufferedTransport($hadoop_socket);$hadoop_protocol = new TBinaryProtocol($hadoop_transport);$hadoopClient = new ThriftHadoopFileSystemClient($hadoop_protocol);$hadoop_transport -> open();try {// create directory$dirpathname = new hadoopfs_Pathname(array("pathname" => "/user/root/hadoop"));if($hadoopClient -> exists($dirpathname) == TRUE) {echo $dirpathname -> pathname . " exists.\n";} else {$result = $hadoopClient -> mkdirs($dirpathname);}// put file$filepathname = new hadoopfs_Pathname(array("pathname" => $dirpathname -> pathname . "/hello.txt"));$localfile = fopen("hello.txt", "rb");$hdfsfile = $hadoopClient -> create($filepathname);while(true) {$data = fread($localfile, 1024);if(strlen($data) == 0)break;$hadoopClient -> write($hdfsfile, $data);}$hadoopClient -> close($hdfsfile);fclose($localfile);// get fileecho "read file:\n";print_r($filepathname);$data = "";$hdfsfile = $hadoopClient -> open($filepathname);print_r($hdfsfile);while(true) {$data = $hadoopClient -> read($hdfsfile, 0, 1024);if(strlen($data) == 0)break;print $data;}$hadoopClient -> close($hdfsfile);echo "listStatus:\n";$result = $hadoopClient -> listStatus($dirpathname);print_r($result);foreach($result as $key => $value) {if($value -> isdir == "1")print "dir\t";elseprint "file\t";print $value -> block_replication . "\t" . $value -> length . "\t" . $value -> modification_time . "\t" . $value -> permission . "\t" . $value -> owner . "\t" . $value -> group . "\t" . $value -> path . "\n";}$hadoop_transport -> close();} catch(Exception $e) {print_r($e);}?>

啟動hadoop的thrift

hadoop-0.20.2\src\contrib\thriftfs\scripts\start_thrift_server.sh 59256

problem one:

在系統目錄建立檔案，而不是在hadoop目錄中建立檔案

原因：

thrift啟動時載入預設的設定檔

解決方案：

修改start_thrift_server.sh檔案

TOP=/usr/local/hadoop-0.20.2

CLASSPATH=$CLASSPATH:$TOP/conf

problem two:

java.lang.NullPointerException

    at     org.apache.hadoop.thriftfs.HadoopThriftServer$HadoopThriftHandler.write(HadoopThriftServer.java:282)

at     org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor$write.process(Unknown Source)

at org.apache.hadoop.thriftfs.api.ThriftHadoopFileSystem$Processor.process(Unknown Source)

at com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown Source)

at         java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

原因：

java返回的map hash id為long類型，而php(32位)無法儲存long類型的資料，導致轉換成float資料後丟失精度。

private long nextId = new Random().nextLong();

java返回資料：4207488029786584864

php擷取資料：4.2074880297866E+18

java獲得php傳遞資料：4207488029786585088

解決方案：

修改hadoop-0.20.2\src\contrib\thriftfs\if\hadoopfs.thrift檔案

修改

struct ThriftHandle {

  i64 id

}

為

struct ThriftHandle {

  string id

}

重建php packages

thrift --gen php hadoopfs.thrift

修改org.apache.hadoop.thriftfs.api.ThriftHandle類

修改

public long id;

為：

public String id;

修改相應的程式

org.apache.hadoop.thriftfs.HadoopThriftServer

修改

long id = insert(out);

ThriftHandle obj = new ThriftHandle(id);

為

long id = insert(out);

String _id = String.valueOf(id);

ThriftHandle obj = new ThriftHandle(_id);

修改相應的程式

重新打包，啟動hadoop的thrift：

hadoop-0.20.2\src\contrib\thriftfs\scripts\start_thrift_server.sh 59256

這樣php就可以串連並且擷取hadoop中的資源了

愛J2EE關注Java邁克爾傑克遜視頻站JSON線上工具

http://biancheng.dnbcw.info/php/360990.html pageNo:1