Hive uses PHP scripts to complete map/reduce

Source: Internet
Author: User
One of the more powerful features of hive SQL is the ability to use external scripting to complete map/reduce. Use the following:
TRANSFORM (...) USING ' ... ' As (...).
Using the inside can use PHP script to handle, the specific use of the following example.

①, build a table:

`member`(    >   `id`int,    >   `user_name`string,    >   `passwd`string    > )    format delimited    "\t"//这句很必要,不然下面的文件导入会导致所有的值都变成null     > stored as textfile;

②, prepare the following data/tmp/member.dat

1     zhangsan     zs10242     lisi     ls19913     wangwu     ww20014     liumang     lm12345     linxing     lx1990

③, importing data

loadlocal'/tmp/member.dat'table member;

can see:

selectfrom member;OK1       zhangsan        zs10242       lisi    ls19913       wangwu  ww20014       liumang lm12345       linxing lx1990

The data is ready, now we do a MD5 encryption in the third column. We use PHP scripts to process the PHP script (/tmp/changepasswd.php) code as follows:

     while(!feof(STDIN)){     $line = rtrim(fgets(STDIN), "\n");      //逐行读取if (empty($line)) continue;     //空,跳出当次循环$data = explode("\t", $line);    //将切分出来的数组保存起来,下面判断使用foreach($dataas &$val){          $val = md5($val);     }     unset($val);     echo implode("\t", $data) . "\n";}

④, add the php script to hive.

add file /tmp/changePasswd.php;

⑤, using PHP scripts to execute Map/reduce

hive> insert overwrite table member    > select TRANSFORM(`id`,`user_name`,`passwd`)"/usr/bin/php changePasswd.php"    > as (`id`,`user_name`,`passwd`) from member;

Finally, we can see that the data for the passwd column has changed:

selectfrom member;OK1       zhangsan        d03eed89429cc3006cc279322c2800c52       lisi    063401506c9d9f0e49a706e3779b74283       wangwu  ac5a8109dbbb46c9f69ffd5fc93c11f84       liumang fda8b97fd723bdbf6a754812b5ecec275       4035378ace8936e93d95aa77e7e224d4

Copyright NOTICE: This article is the original blogger articles, reproduced please indicate the source.

The above describes the use of Hive PHP script to complete the map/reduce, including aspects of the content, I hope that the PHP tutorial interested in a friend helpful.

  • Contact Us

    The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

    If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

    A Free Trial That Lets You Build Big!

    Start building with 50+ products and up to 12 months usage for Elastic Compute Service

    • Sales Support

      1 on 1 presale consultation

    • After-Sales Support

      24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.