Webbrowser multithreading brings troubles

When we were doing collection software

It is very troublesome for some websites to analyze HTML text directly.

When winform is used for programming

There is a better way, of course, to analyze htmldocument

However, this htmldoucment cannot be created directly.

It must be generated by the webbroswer control navigate after a page

To obtain WB. htmldocument

Then, you can analyze the elements and labels of htmldocument.

 

In fact

Not only a single page is collected

In this way, it can be completed in the main form

 

For example, to collect some list pages, there are n multiple pages

So, a loop goes down,

Using webbrowser to respond will lead to false positives

At this time, we will certainly think of using multithreading to do this.

 

C # multithreading,

We should all know that there are two modes: STA and MTA.

However, the webbrowser control has a bad feature.

That is: it only supports multi-thread sta Mode

For exampleCode,

Thread tread =   New Thread ( New Parameterizedthreadstart (begincatch ));
Tread. setapartmentstate (apartmentstate. Sta );
Tread. Start (URL );

 

 

Code

Private   Void Begincatch ( Object OBJ)
{
String URL = OBJ. tostring ();
Webbrowser WB =   New Webbrowser ();
WB. scripterrorssuppressed =   True ;
WB. navigate (URL );
WB. documentcompleted + =   New Webbrowserdocumentcompletedeventhandler (wb_documentcompleted );
}

 

To analyze htmldocument generated by webbrowser, you must perform operations in the event documentcompleted.

Webbrowser is loaded only at this time.

 

However, this is just a trap !!!!

Webbrowser has a feature, that is, when the multi-thread sta is used, it simply does not wait for the execution of documentcompleted.

That is, subsequent operations cannot be performed !!!

 

In this case, what should we do?

Someone may think of the WB. Document. Write (string) method as follows:

 

 

Code

Private   Void Begincatch ( Object OBJ)
{
String URL = OBJ. tostring ();
Webbrowser WB =   New Webbrowser ();
WB. scripterrorssuppressed =   True ; String Htmlcode = Gethtmlsource (URL );
WB. Document. Write (htmlcode );
// Perform analysis
}
// WebClient obtains the webpage source code
Private   String Gethtmlsource ( String URL)
{
String Text1 =   "" ;
Try
{
System. net. WebClient WC =   New WebClient ();
Text1 = WC. downloadstring (URL );
}
Catch (Exception exception1)
{}
Return Text1;
}

 

 

However, at this time, we will find that WB. documenttext is always unavailable.

At that time, I was also very depressed.ArticleBoth msdn and documenttext can be used to assign values.

However, I also found a lot on the Internet that there was no result after the operation.

 

Search by effort

I found an example in a useful article in the garden.

After testing, we found that

A document is generated only after webbrowser is navigate.

So suddenly, we can finally implement the operations below multithreading.

The final code is as follows:

 

Code

Private   Void Threadwebbrowser ( String URL)
{
Thread tread =   New Thread ( New Parameterizedthreadstart (begincatch ));
Tread. setapartmentstate (apartmentstate. Sta );
Tread. Start (URL );
}

Private   Void Begincatch ( Object OBJ)
{
String URL = OBJ. tostring ();
Webbrowser WB =   New Webbrowser ();
WB. scripterrorssuppressed =   True ;
// Navigate to a blank page
WB. navigate ( " About: blank " );
String Htmlcode = Gethtmlsource (URL );
WB. Document. Write (htmlcode );
// Perform analysis ...... (Omitted)
}
// WebClient obtains the webpage source code
Private   String Gethtmlsource ( String URL)
{
String Text1 =   "" ;
Try
{
System. net. WebClient WC =   New WebClient ();
Text1 = WC. downloadstring (URL );
}
Catch (Exception exception1)
{}
Return Text1;
}

 

 

Of course, threadpool can be used when processing operations on each node and database in the thread.

The Effect and performance may be better.

 

I hope this article will be helpful to you ~ :)