Develop real estate information collectors with WPF + MongoDB (1)

I believe that the House is the greatest burden on the shoulders of many Chinese people. I am also one of them, and I am also paying attention to the house. I am using a Silverlight development project and learning MongoDB.ProgramPersonnel, so naturally I want to develop a "Real Estate Information Collector" for myself and many friends who are interested in the house ".

Real Estate Information Collector:

1. Obtain information about second-hand houses and first-hand houses on the real estate website;
2. You can select the property information of the region you want to download;
3. Filters duplicate and false data;
4. Reminders can be sent when new information is available;
5. You can set whether to pay attention to or not to be interested in a real estate or a real estate in a certain area;
6. ......

Today, we mainly downloaded and parsed data from the website, and did not save the data to the database. First:

To obtain real estate information from a website, you must first know where to download and parse the data.

It is very simple. Every website now has a search function, and all the search results are paginated. These pages are regular. As long as you find this rule, there will be a steady stream of data. Of course, the content of the search result is only an overview. to parse the data, you must go to the corresponding real estate information page and download the resolution. There are no technical difficulties, that is, to analyze the website's htmlCodeIt's a little annoying. All of you have figured it out.

Here we recommend a third-party tool for parsing and downloading HTML: HTML agility pack. As for how to use it, codeplex is a demo and I think it is clear. You can check it yourself.

I want to share with you some questions about using this control today.

1. When you use HTML agility pack to directly download and parse websites, Chinese characters may be garbled. I used httpwebrequest to download the webpage content and return a stream.

   1Httpwebrequest Req = webrequest. Create (NewUri (URL ))AsHttpwebrequest;
   2Req. method ="Get";
   3Webresponse rs = Req. getresponse ();
   4Stream RSS = Rs. getresponsestream ();
   5 ReturnRSS;

   And then use htmlagilitypack. The subsequent parsing uses htmlagilitypack.

2. The path of xpath is incorrect. The path starting with "/" indicates that it starts from the root directory. Therefore, whether you want to obtain its subnodes from a selected htmlnode, note that XPath cannot start.

Today is just the beginning. I will stick to it and continue to complete and improve this tool. If you have any comments or feel that you need to add the functions, I am very grateful.