About crawling site data, there are 302 redirects!

Source: Internet
Author: User
Target website: http://www.dltm.net/webtmq/free/free_query.php
The first step: Enter the trademark number, submit (post grab)

Step Two: Click on the trademark number

Step three: Data that needs to be crawled


The first two steps have been crawled, but the last step of the crawl is always a 302 redirect, resulting in data fetching is not possible.
Related code:
The first step define (Target_url, ' http://www.dltm.net/webtmq/free/free_query.php ');d efine (reffer_url, ' HTTP// Www.dltm.net '), $url =target_url, $ch =curl_init ($url), curl_setopt ($ch, Curlopt_returntransfer, 1);//The return result is stored in the variable, Instead of the default direct output curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch, Curlopt_referer, reffer_url); $result = Curl_exec ($ch) ///Save the output page to $result curl_close ($ch);p Reg_match_all ('', $result, $rs);//The second step $fields_post = array (' IP ' = = $rs [1][0], ' textarea_explain ' = '%b2%e9%d1%af%c8%ab%b2%bf%c0% E0%b1%f0 ', ' tm_lb ' = ' 0 ', ' tm_key ' = ' 8437927 ', ' tm_key_item ' = ' tm_zch ', ' Query_mode ' and ' 1 '); $fields _ String= '; foreach ($fields _post as $key = = $value) {$fields _string. = $key. '=' . $value. ' & ';} $fields _string = RTrim ($fields _string, ' & ');d efine (TARGET_URL1, ' http://www.dltm.net/webtmq/free/free_res.php ') );d efine (REFFER_URL1, ' http://www.dltm.net/webtmq/free/free_res.php '); $url =target_url1; $ch =curl_init ($url); curl _setopt ($ch, Curlopt_returntransfer, 1);//The return result is stored in the variable instead of the default direct output curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch , Curlopt_referer, REFFER_URL1); curl_setopt ($ch, curlopt_post,1);//POST curl_setopt ($ch, curlopt_postfields,$ fields_string); $result = Curl_exec ($ch);//Save the output of the page to $result curl_close ($ch);//The Third step Preg_match_all (", $result, $res); $ url = ' http://www.dltm.net/webtmq/free/'. $res [1][0]; $ch =curl_init ($url); curl_setopt ($ch, Curlopt_returntransfer, 1 );//return results are stored in the variable instead of the default direct output curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch, Curlopt_referer, reffer_url1); $result = Curl_exec ($ch);//Save the output of the page to $result curl_close ($ch);p rint_r ($result); exit;//This step does not get data

You can copy the code to run a bit, the third step of the data is not, the great God can help me to see, if you can get data, please post your source code, thank you!!!


Reply to discussion (solution)

Increase
curl_setopt ($ch, Curlopt_cookiejar, "cookie.txt");
curl_setopt ($ch, Curlopt_cookiefile, "cookie.txt");

Increase
curl_setopt ($ch, Curlopt_cookiejar, "cookie.txt");
curl_setopt ($ch, Curlopt_cookiefile, "cookie.txt");

No, curl_getinfo. Print ($ch)

Array ([url] = Http://www.dltm.net/webtmq/free/free_detail.php?ip=7f6179b25de2974b0e0b33d01c2fef18&class_ id=25&id=10309774 [Content_Type] = text/html [Http_code] = 302 [header_size] = [Request_size] = 19 3 [FILETIME] +-1 [Ssl_verify_result] = 0 [Redirect_count] + 0 [total_time] = 0.094 [Namelookup_time] => ;  0 [Connect_time] = 0.047 [Pretransfer_time] = 0.047 [Size_upload] = 0 [Size_download] = 0 [Speed_download] + 0 [Speed_upload] = 0 [Download_content_length] = 1 [upload_content_length] + 0 [starttransfer_time] =& Gt 0.094 [Redirect_time] = 0 [Certinfo] = = Array () [Redirect_url] = http://www.dltm.net/webtmq/free/free_query. PHP)

How can it not?
Take registration number 8437927 as an example
Create a new file Cookie.txt, execute the code

$cookie = Realpath (' cookie.txt '); This is increased//first step define (' Target_url ', ' http://www.dltm.net/webtmq/free/free_query.php ');d efine (' Reffer_url ', '/HTTP/ Www.dltm.net '), $url =target_url, $ch =curl_init ($url), curl_setopt ($ch, Curlopt_returntransfer, 1);//The return result is stored in the variable, Instead of the default direct output curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch, Curlopt_referer, Reffer_url); curl_setopt ($ch, Curlopt_cookiefile, $cookie); This is an increase in curl_setopt ($ch, Curlopt_cookiejar, $cookie); This is an increased $result= curl_exec ($ch);//Save the output of the page to $result curl_close ($ch);p Reg_match_all ('', $result, $rs); Step Two $fields_post = array (' IP ' = = $rs [1][0], ' textarea_explain ' = '%b2%e9%d1%af%c8%ab%b2%bf%c0%e0%b1%f0 ', ' TM _lb ' + ' 0 ', ' tm_key ' = ' 8437927 ', ' tm_key_item ' = ' tm_zch ', ' Query_mode ' and ' 1 '); $fields _string= '; foreach ($fields _post as $key = + $value) {$fields _string. = $key. '=' . $value. ' & ';} $fields _string = RTrim ($fields _string, ' & '); Define (' Target_url1 ', ' http://www.dltm.net/webtmq/free/free_res.php ');d efine (' reffer_url1 ', ' http://www.dltm.net /webtmq/free/free_res.php '); $url =target_url1; $ch =curl_init ($url); curl_setopt ($ch, Curlopt_returntransfer, 1);// The return result is stored in the variable, not the default direct output curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch, Curlopt_referer, REFFER_URL1); curl_ Setopt ($ch, curlopt_post,1);//Submit curl_setopt by POST ($ch, Curlopt_postfields, $fields _string); curl_setopt ($ch, Curlopt_cookiefile, $cookie); This is an increase in curl_setopt ($ch, Curlopt_cookiejar, $cookie); This is an increased $result= curl_exec ($ch);//Save the output page to $result curl_close ($ch); Step three Preg_match_All (", $result, $res); $url = ' http://www.dltm.net/webtmq/free/'. $res [1][0]; $ch =curl_init ($url); curl_setopt ($ch, Curlopt_returntransfer, 1);//The return result is stored in the variable instead of the default direct output curl_setopt ($ch, Curlopt_ HEADER, 0); curl_setopt ($ch, Curlopt_referer, REFFER_URL1); curl_setopt ($ch, Curlopt_cookiefile, $cookie); This is an increase in curl_setopt ($ch, Curlopt_cookiejar, $cookie); This is an increased $result= curl_exec ($ch);//Save the output page to $result curl_close ($ch);p rint_r ($result); exit;

How can it not?
Take registration number 8437927 as an example
Create a new file Cookie.txt, execute the code

$cookie = Realpath (' cookie.txt '); This is increased//first step define (' Target_url ', ' http://www.dltm.net/webtmq/free/free_query.php ');d efine (' Reffer_url ', '/HTTP/ Www.dltm.net '), $url =target_url, $ch =curl_init ($url), curl_setopt ($ch, Curlopt_returntransfer, 1);//The return result is stored in the variable, Instead of the default direct output curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch, Curlopt_referer, Reffer_url); curl_setopt ($ch, Curlopt_cookiefile, $cookie); This is an increase in curl_setopt ($ch, Curlopt_cookiejar, $cookie); This is an increased $result= curl_exec ($ch);//Save the output of the page to $result curl_close ($ch);p Reg_match_all ('', $result, $rs); Step Two $fields_post = array (' IP ' = = $rs [1][0], ' textarea_explain ' = '%b2%e9%d1%af%c8%ab%b2%bf%c0%e0%b1%f0 ', ' TM _lb ' + ' 0 ', ' tm_key ' = ' 8437927 ', ' tm_key_item ' = ' tm_zch ', ' Query_mode ' and ' 1 '); $fields _string= '; foreach ($fields _post as $key = + $value) {$fields _string. = $key. '=' . $value. ' & ';} $fields _string = RTrim ($fields _string, ' & '); Define (' Target_url1 ', ' http://www.dltm.net/webtmq/free/free_res.php ');d efine (' reffer_url1 ', ' http://www.dltm.net /webtmq/free/free_res.php '); $url =target_url1; $ch =curl_init ($url); curl_setopt ($ch, Curlopt_returntransfer, 1);// The return result is stored in the variable, not the default direct output curl_setopt ($ch, Curlopt_header, 0); curl_setopt ($ch, Curlopt_referer, REFFER_URL1); curl_ Setopt ($ch, curlopt_post,1);//Submit curl_setopt by POST ($ch, Curlopt_postfields, $fields _string); curl_setopt ($ch, Curlopt_cookiefile, $cookie); This is an increase in curl_setopt ($ch, Curlopt_cookiejar, $cookie); This is an increased $result= curl_exec ($ch);//Save the output page to $result curl_close ($ch); Step three Preg_match_All (", $result, $res); $url = ' http://www.dltm.net/webtmq/free/'. $res [1][0]; $ch =curl_init ($url); curl_setopt ($ch, Curlopt_returntransfer, 1);//The return result is stored in the variable instead of the default direct output curl_setopt ($ch, Curlopt_ HEADER, 0); curl_setopt ($ch, Curlopt_referer, REFFER_URL1); curl_setopt ($ch, Curlopt_cookiefile, $cookie); This is an increase in curl_setopt ($ch, Curlopt_cookiejar, $cookie); This is an increased $result= curl_exec ($ch);//Save the output page to $result curl_close ($ch);p rint_r ($result); exit;


Thank you, just contact Curl,cookie position less release
Problem solved, knot paste!!!!
  • Contact Us

    The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

    If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

    A Free Trial That Lets You Build Big!

    Start building with 50+ products and up to 12 months usage for Elastic Compute Service

    • Sales Support

      1 on 1 presale consultation

    • After-Sales Support

      24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.