CURL, curlhttp

CURL, curlhttp

CURL? Well, it's a long time ~~~~

This is now one of the built-in command line tools on the Apple machine.

1). Let's start from here!

Curl http://www.yahoo.com

After you press enter, the html of www.yahoo.com is displayed on the screen ~~~~~

2) Well, do you want to save the pages you have read?

Curl http://www.yahoo.com> page.html

Yes, of course, but you don't have to worry about it!

Use the built-in option of curl to save the http result. Use this option:-o

Http://www.yahoo.com curl-o page.html

In this way, you can see a download page progress indicator on the screen. When the Progress reaches 100%, it will naturally be OK.

3) What ?! Cannot access? It must be that your proxy has not been set.

When using curl, you can use this option to specify the proxy server used for http access and its port:-x

Curl-x 123.45.67.89: 1080-o page.html http://www.yahoo.com

4) it is annoying to visit some websites. He uses cookies to record session information.

Browsers like IE/NN can easily process cookie information, but what about our curl ?.....

Let's learn this option:-D <-- this is to store the cookie information in the http response to a special file.

Curl-x 123.45.67.89: 1080-o page.html-D cookie0001.txt http://www.yahoo.com

In this case, when the page is stored in the same page as page.html, cookieinformation is also stored in cookie0001.txt.

5) So how can I continue to use the last cookie information during the next visit? You know, many websites rely on monitoring your cookie information,

To determine if you do not access their website according to the rules.

This time, we use this option to append the last cookie information to the http request:-B

Curl-x FIG: 1080-o page1.html-D cookie0002.txt-B cookie0001.txt http://www.yahoo.com

In this way, we can simulate almost all the IE operations to access the webpage!

6). Wait a bit ~~~~~ I seem to have forgotten something ~~~~~

Right! Browser Information ~~~~

Some annoying websites always use some specific browsers to access them. Sometimes, more importantly, some specific versions must be used ~~~~

NND, where can I find these weird browsers for it !?

Fortunately, curl provides us with A useful option, which allows us to randomly specify the browser information we declared during this access:-

Curl-A "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"-x 123.45.67.89: 1080-o page.html-D cookie0001.txt http://www.yahoo.com

In this way, when the server receives an access request, it will think that you are an IE6.0 running on Windows 2000. Hey, maybe you are using an apple!

However, "Mozilla/4.73 [en] (X11; U; Linux 2.2; 15 i686" tells the other party that you are running Linux on a PC and use Netscape 4.73.

7) Another common restriction method on the server side is to check the referer for http access.

For example, if you first access the homepage and then access the specified download page, the referer address for the second visit is the address of the page after the first successful access.

In this way, as long as the server finds that the referer address accessed for a certain download page is not the address of the home page, it can be determined that the referer address is a stolen connection ~~~~~

Hate ~~~ I just want to steal the connection ~~~~~!!

Fortunately, curl provides us with the option to set referer:-e

Curl-A "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"-x 123.45.67.89: 1080-e "mail.yahoo.com"-o page.html-D cookie0001.txt http://www.yahoo.com

In this way, you can lie to the other server. You clicked a link from mail.yahoo.com.

8) I wrote something to discover something important was missed! ----- Use curl to download an object

As mentioned earlier, you can use-o to download a file from the download page.

For example, curl-o 1.jpg http://cgi2.tky.3web.ne.jp /~ Zzh/screen1.JPG

Here we will teach you a new option:-O

Uppercase O, so use: curl-O http://cgi2.tky.3web.ne.jp /~ Zzh/screen1.JPG

In this way, the file name on the server is automatically stored locally!

Another one is better.

If screen1.JPG, screen2.JPG, screen3.JPG,..., and screen10.JPG need to be downloaded, can we write a script to complete these operations?

Don't do it!

In curl, you can write it like this:

Curl-O http://cgi2.tky.3web.ne.jp /~ Zzh/screen [1-10]. JPG

Hehao, amazing ?!~~~

9). Now, let's continue to explain the download!

Curl-O http://cgi2.tky.3web.ne.jp /~ {Zzh, nick}/[001-201]. JPG

This download is

~ Zzh/001.JPG

~ Zzh/002.JPG
...

~ Zzh/201.JPG

~ Nick/001.JPG

~ Nick/002.JPG
...

~ Nick/201.JPG

Is it convenient enough? Hahaha

Why? I'm so happy.

Because the file names under zzh/nick are both 001,002... and 201, the downloaded file has the same name, and the previous file is overwritten by the following ~~~

It doesn't matter. We still have something worse!

Curl-o %2_%1.jpg http://cgi2.tky.3web.ne.jp /~ {Zzh, nick}/[001-201]. JPG

-- This is the download of the custom file name?

-- Right, huh, huh!

#1 refers to the variable {zzh, nick}. The first value is zzh and the second value is nick.

#2 represents the variable, which is the variable part of the second paragraph --- [001-201]. The value is increased from 001 to 201 one by one.

In this way, the downloaded file name is changed to the following:

Original :~ Zzh/001.JPG ---> download: 001-zzh.JPG

Original :~ Nick/001.JPG ---> download: 001-nick.JPG

In this way, you are not afraid of duplicate names of files.

10) continue to download

On windows, we usually use tools like flashget to help us download data in parallel in multiple parts and resume data transmission.

Curl won't lose to anyone in these aspects.

For example, if we download screen1.JPG and suddenly drop the line, we can resume the download.

Curl-c-O http://cgi2.tky.3wb.ne.jp /~ Zzh/screen1.JPG

Of course, you should not use a flash get to download half of the files to confuse me ~~~~ Other Software downloading half-width files may not be used ~~~

For multipart download, we can use this option:-r

Example

For example, we have a http://cgi2.tky.3web.ne.jp /~ Zzh/zhao1.mp3 to download (Instructor Zhao's phone recitation: D)

We can use the following command:

Curl-r 0-10240-o "zhao. part1" http:/cgi2.tky.3web.ne.jp /~ Zzh/zhao1.mp3 &/

Curl-r 10241-20480-o "zhao. part1" http:/cgi2.tky.3web.ne.jp /~ Zzh/zhao1.mp3 &/

Curl-r 20481-40960-o "zhao. part1" http:/cgi2.tky.3web.ne.jp /~ Zzh/zhao1.mp3 &/

Curl-r 40961-o "zhao. part1" http:/cgi2.tky.3web.ne.jp /~ Zzh/zhao1.mp3

In this way, you can download them in multiple parts.

However, you need to merge these broken files by yourself.

If you use UNIX or apple, use cat zhao. part *> zhao.pdf.

If Windows is used, use copy/B.

All of the above are http protocol downloads. In fact, ftp can also be used.

Usage well,

Curl-u name: passwd ftp: // ip: port/path/file

Or you are familiar

Curl ftp: // name: passwd @ ip: port/path/file

11) after the download is completed, we will naturally talk about uploading.

The upload option is-T.

For example, we can upload a file: curl-T localfile-u name: passwd ftp: // upload_site: port/path/to ftp/

Of course, you can also upload files to the http server.

For example, curl-T localfile http://cgi2.tky.3web.ne.jp /~ Zzh/abc. cgi

Note: At this time, the protocol used is the http put method.

Speaking of PUT, hey, naturally, it reminds the old server of the other methods that haven't been mentioned yet!

Neither GET nor POST can be forgotten.

When submitting a form over http, the POST mode and GET mode are commonly used.

No option is required in GET mode. You only need to write the variable in the url.

For example:

Curl http://www.yahoo.com/login.cgi? User = nickwolfe & password = 12345

The option in POST mode is-d.

For example, curl-d "user = nickwolfe & password = 12345" http://www.yahoo.com/login.cgi

This is equivalent to sending a login application to this site ~~~~~

Whether to use the GET mode or POST mode depends on the program settings of the opposite server.

Note that, in POST mode, files are uploaded, such

<Form method = "POST" enctype = "multipar/form-data" action = "http://cgi2.tky.3web.ne.jp /~ Zzh/up_file.cgi ">

<Input type = file name = upload>

<Input type = submit name = nick value = "go">

</Form>

To simulate such an HTTP form with curl, the syntax is as follows:

Curl-F upload = @ localfile-F nick = go http://cgi2.tky.3web.ne.jp /~ Zzh/up_file.cgi

Luo luochao has talked so much about it. In fact, curl still has many skills and usage.

For example, if you use a local certificate for https

Curl-E localcert. pem https: // remote_server

For example, you can use curl to search the dictionary through the dict protocol ~~~~~

Curl dict: // dict.org/d:computer

 

From: http://blog.csdn.net/collin1211/article/details/3075552