C # implementation icon Bulk download

This article is slightly longer, it took several nights to edit the changes, if there is a problem in the wording, please understand. This article is divided into four articles, the following is the main content, but also the basic software development process.

Stage	Describe
requirements Analysis	mainly describes the purpose of implementing the program and analyzing the requirements, that is, why it takes time to write, what functions are required, etc.
scheme design
programming Implementation	through .net programming to achieve the function of batch download, focusing on the problems encountered and the solution.
Results show	Demonstrate the tools and results of sharing and summarize the experience.

First, demand analysis

In peacetime program development, in order to quickly build a more beautiful user interface, often to download some icons as buttons, controls and other appearance, and even need to create some of their own specific icons or pictures. Own motivation, have to say the need for a certain degree of technical and aesthetic Foundation; download, and get online everywhere, find a suit of theme, color, size, beautiful and generous icon is really a difficult thing.

Fortunately, there are a lot of Web sites dedicated to download icons, commonly used are:

http://www.easyicon.net/

https://www.iconfinder.com/

http://www.haotu.net/

http://www.iconpng.com/

http://findicons.com/

http://www.flaticon.com/

http://www.iconspedia.com/

http://icones.pro/

Each of these sites have their own advantages, in common is a large number of icons, I personally prefer to search in the Easyicon, download, but also like its web site. It has some advantages:

(1) Support Chinese and English search. Easyicon support Chinese and English search, of course, its original icon name or English, but before the search, the use of Baidu translation API to Chinese translation into English, and then search.

(2) The user experience is good. Many URLs in the browsing, it is necessary to click on the "next page" and other buttons, and it supports keyboard shortcuts, and the experience is good, its interface, text AH is also more lively, such as by the sort of heat, it gracefully called "publicity priority."

(3) Keep the update. As a code, we are most afraid of the open source of things no longer updated, Easyicon icon Update frequency will be counted.

(4) package download, sometimes, we download more than one icon, you can use its packaging download function. (There are limitations to this feature, such as the limited number of downloads per package, and the inconvenience of downloading dimensions and formatting, which is why you should re-write a bulk download tool.) ）

So, to summarize, we need a program to achieve bulk download of different formats, sizes of icons to the local, in order to search and use.

Second, the programme design

1. Browser Download icon

Design plan is not directly to come out, or to be based on the actual 1.1 points to analyze, determine. We use a browser to download an icon to try.

Target:http://www.easyicon.net/iconsearch/iconset:fatcowhosting-icons/

In this URL, it contains 2000 (40 pages) of different sizes and formats and icons. Fatcowhosting-icons is the category name of these icon sets.

Click the first icon to go to the other Detail page:http://www.easyicon.net/530832-Zoom_Selection_icon.html, here we can see a lot of parameter information.

Click the PNG icon to download and we download this icon. (This time the download is the most inner loop of code in the future.) We see the real:http://download.easyicon.net/png/530832/32/

As long as we have this download URL, regardless of which browser or custom program, can be downloaded.

2. Analysis

Look at the address of each page:

http://www.easyicon.net/iconsearch/iconset:fatcowhosting-icons/1/

Fatcowhosting-icons represents the Icon collection name, 1 indicates the number of pages

Well, let's analyze this. Address: http://download.easyicon.net/png/530832/32/

This can be decomposed into: fixed part + format + icon number + size

Look again, download the required parameters:+ File Save path + file name

Comprehensive analysis can be seen, the format of the icon, size, file save path can be specified by the user, now the key is missing the icon number and file name .

If we already know the icon number and enter the download URL into the browser's address bar to submit, the browser can automatically identify the downloaded file name, why? After the user submits this address to the server, the server returns some messages, including the file name, so you can get the file name by some programming method (mentioned later, without worrying about the query).

Well, now the only missing master is the icon number . By observing the other icons on the site, you can see that the numbers are connected, such as 530832 is the Zoom_selection_icon number, and 530831 is the Zoom_refresh number. And look at the icon fatcowhosting-icons each page of the collection is 50 (except for the last page), and we can't get all the numbers for this icon set based on the number of each icon and the last icon? The answer is yes.

So how do we get the first and last number? If we get to these two numbers by some technical means ... Wait, if you can get these two numbers, why not get all the numbers directly? Yes, one way to crawl through a webpage should be to get all the numbers .

3. Draw a simple flowchart

The following is a diagram using Edraw V7.9 Expert to draw the flowchart:

4. Write a simple interface

After analyzing it for so long, write a simple interface to get a feel for our ideas. (C #)

Private string[] FileType;     File format private int[] FileSize;    File size private string FilePath;     File save path private int totalpages; Total page count//Get icon total pages private int gettotalpages (string iconsurl) {}//Gets the number of the current page private string[] GetIDs (String pageurl) {} private bool Downico (string[] fileType, int[] fileSize, int totalpages) {//Layer: Traverse each page for (int i = 0; i < totalpage S        i++) {//Get current page all numbers string[] Strids = GetIDs ("Pagesurl");  Two layers: Traverse each number for (int j = 0; J < Strids.length; J + +) {//Three layers: traverse each size for (int k = 0; K < Filesize.length;                     k++) {//layer four: Traverse each format for (int m = 0; m < filepath.length; m++) {                    Generate download link string downurl = "http://download.easyicon.net/format/number/size/"; Down (this.                    FilePath, Downurl);            Other operations ...} }//4}//3}//2}//1//Download each icon private bool down (string FilepaTh,string downurl) {} 

　　

5. Key issues

Here's a solution to the key issues used in the code:

(1) If all parameters can be found, which class or method to download? The DownloadFile method of System.Net.WebClient.

(2) How do I get the total number of icons? According to the Observation page, each page has "an icon, turn x page can be read", X is the total number of pages, by crawling the page string can be;

(3) How do I get the number of all the icons on each page? Or, of course, crawling through web pages. For example, by reviewing the elements, you can see the number and name of each icon.

(4) How do I get the name of the download icon? There are two ways to crawl web content, and the other is to extract it by the information returned by the service.

Third, the realization of programming

Programming is relatively simple, the following is a Web page operation of the two core functions (the first crawl of the web, do not know this good or bad)

The first function is to get the page code by the Web address.

<summary>///URL to get page code///</summary>///<param name= "strURL" >url address </param>///< Returns> Web page code string </returns>public static string gethtmlstring (String strurl) {    uri uri = new Uri (strurl);    HttpWebRequest request = (HttpWebRequest) webrequest.create (URI);    HttpWebResponse response = (HttpWebResponse) request. GetResponse ();    Stream stream = Response. GetResponseStream ();    String strhtml = "";    if (stream! = null)    {        StreamReader sr = new StreamReader (stream);        strHTML = Sr. ReadToEnd ();        Sr. Close ();        Stream. Close ();        Response. Close ();    }    return strhtml;}

　　

The second function is to obtain the returned headers information, which contains the name of the icon, based on the download link that submits the icon to the server.

<summary>///Get headers information based on URL///</summary>///<param name= "url" >url address </param>///< Returns>headers Information list </returns>public static dictionary<string, string> getheaders (string URL) {    dictionary<string, string> headerlist = new dictionary<string, string> ();    WebRequest webrequestobject =httpwebrequest.create (URL);    WebResponse responseobject =webrequestobject.getresponse ();    foreach (String Headerkey in Responseobject.headers)    {        headerlist.add (Headerkey, responseobject.headers[ Headerkey]);    }    Responseobject.close ();    return headerlist;}

　　

Issue One: Captcha issues

Programming is not so easy, more or less will encounter some previously not thought of the problem

One of the biggest problems encountered is the validation issue. If a large number of download icons (up to 166 icons for the first time), when submitted to the server, it will pop up the validation window, the following is the results from the WebBrowser control.

This is the result of another page http://www.easyicon.net/api/captcha/captcha.php return

Solution : The first solution is to grab the package, get the submission link and content, just like other programs to let users play code, and then I have to play the code anyway, it is better to let the user directly see this page (of course, this interface is very rough, actually should go to get this icon, And this icon is displayed in front of the user), then use the WebBrowser control; next, you need an input, and then submit: The input takes the VB InputBox, so it is more convenient, do not need to pause the thread, Commit is to use htmlelement getattribute to get the submit button, using the InvokeMember method to execute.

Question two: The problem of suspended animation in procedure

Too many downloads, the program interface will certainly be suspended animation, user experience is very bad. You need to create a new thread, but be aware of the problem of control information interaction between the new thread and the main path.

workaround : The following is a function that implements a delegate to add the message returned to the LISTBOXADV to the download.

delegate void Setvaluecallback (listboxadv lsta,string log);p rivate void setPropertyValue (listboxadv lsta,string log) { C0/>if (lsta.invokerequired)    {        Setvaluecallback d = new Setvaluecallback (setpropertyvalue);        Lsta.invoke (d, new object[] {lsta,log});    else    {        lstA.Items.Add (log);        Lsta.setselected (lsta.items.count-1,true);        lsta.selectedindex=lsta.items.count-1;}    }

　　

Call:

setPropertyValue (LSTADV, "message ...");

　　

Issue three: Download failure issue

Not all icons can be downloaded normally, even if repeatedly downloaded, it is easy to appear, download results only 25-byte size of the icon (repeated download is not valid), probably because of the speed of the network.

fix : Traverse all 25-byte icons, delete and re-download (Which of course takes time).

Iv. Presentation of results

Main interface

Download the icon

I tested the download of the PNG 32 icon, about 8,000, local and cloud disk has, the file is named after the number + name, through the number, I can then download from the official website to other required icons, through the name can search for the desired icon.

SOURCE Download: Http://files.cnblogs.com/files/liweis/EasyDown.rar

Prospect

1. How does the server detect the number of consecutive download icons for this machine? Is it based on IP or other, if the mechanism is clarified, is it possible to bypass its detection by some code operation instead of using a verification code?

2. How to query the name of the icon collection, can be queried by some kind of SQL code? If so, the whole easyicons is not a problem!

C # implementation icon Bulk download