13.1.2 Asynchronous Download Web page

13.1.2 Asynchronous Download Web page

Before we use asynchronous workflows to crawl Web content, we need to refer to the FSharp.PowerPack.dll library, which contains many asynchronous versions of. NET methods. When developing stand-alone applications, you can use the Add Reference command; In this chapter, we will use the interactive development pattern, so create a new F # script file using the #r directive (listing 13.1).

Listing 13.1 writing code using asynchronous Workflow (F # Interactive)

> #r "FSharp.PowerPack.dll";;

> Open System.IO

Open system.net;;

> Let DownloadURL (url:string) = async{[1]

Let request =httpwebrequest.create (URL)

let! Response =request. AsyncGetResponse () [2]

Use response =response [3]

Let Stream =response. GetResponseStream ()

Use reader = Newstreamreader (stream)

Return!reader.     Asyncreadtoend ()};; [4]

Val downloadurl:string->async<string>

After importing (opening) all the required namespaces, we define a function that uses an async workflow, which uses the async value as the calculation generator [1]. It is easy to prove that it is a normal value; in Visual Studio, if you type a point (.) After the value, IntelliSense displays all the common calculation generator members it contains, such as Bind and Return, as well as several other basic operations that we will be able to do later. The type signature of the output is displayed, and the calculation type is async<string>. In the following, we will discuss this type in detail.

The code in Listing 13.1 uses the let! once structure that executes the asynchronous basic operation AsyncGetResponse provided by the F # Library. This method returns the Async<webresponse> type, so let! The structure combines two asynchronous operations and binds the actual WebResponse value to the symbol response. Thus, once the asynchronous operation is complete, we can use this value.

In the next line [3], the use basic operation is used, and once a particular object is out of scope, it is freed. We've talked about using use in an F # program in general, and it behaves very similarly in an asynchronous workflow. When the workflow finishes, it releases the HTTP response immediately. We use value hiding to hide the original response symbol, declaring that a new value will be released. This is a common pattern, so F # provides an easy way to use the use! Basic operation, simple combination of let! and use. Now that we know, we can replace the above two lines with one line:

use! Response = Request. AsyncGetResponse ()

In the last line of listing 13.1, we've used the basic operations we've never seen before return! [4] It can run other asynchronous operations (like using let! Basic operation), except that when the operation is complete, the result is returned instead of being assigned to the symbol. Like do! Basically, this is also a simple syntactic sugar (syntactic sugar). The calculation generator does not need to implement any other members, and the compiler can think of the code as written (the actual conversion is simpler):

let! Text = reader. Asyncreadtoend ()

return text

Now that we have the DownloadURL function for creating asynchronous computations, we should also determine how to use it to download the contents of a Web page. As you can see in Listing 13.2, we use the functions in the Async module to run the workflow.

Listing 13.2 for asynchronous computations (F # Interactive)

> Let Downloadtask = DownloadURL ("http://www.manning.com");; [1] <--generating asynchronous workflow

Val downloadtask:async<string>

>async.runsynchronously (Downloadtask); [2] <--running workflow, waiting for results

Val it:string = "<! DOCTYPE htmlpublic "-//W3C//DTD XHTML 1.0

transitional//en "" Http://www.w3.org/TR/xhtml1/DTD/xhtml1-tr

Ansitional.dtd ">

> Let tasks =

[DownloadURL ("http://www.tomasp.net");

DownloadURL ("http://www.manning.com")];;

Val tasks:list<async<string>>

> Let all =async.parallel (tasks); [3] <--combines several workflows into one

Val all:async<string[]>

> async.runsynchronously (All);;

Val it:string[] = ["..."; " ..." ]

Code written using an asynchronous workflow is automatically deferred, so when we execute the first line of the DownloadURL function, it will not start downloading the page [1]. The returned value (async<string> type) represents the calculation that you want to run, just as the function value represents code that can be executed later. The Async module provides a way to run the workflow, and table 13.1 describes a subset of them.

Table 13.1 Several basic operations for handling asynchronous workflows in the Async module of the standard F # Library

Basic operation	Types and descriptions of basic operations
runsynchronously	async< ' t>–> ' T Launches the given workflow in the current thread. When an asynchronous operation is used in a workflow, the workflow restarts the thread and is used to invoke the asynchronous callback. This action blocks the caller thread and waits for the result of the workflow.
Start	async<unit>–> Unit In the background (make Starts a given workflow with thread pool threads, and returns immediately. The workflow executes in parallel with the subsequent caller code. As the signature indicates, the workflow does not return a value.
Createastask	async< ' t>-task< ' t& gt;  This method is used only in. NET 4.0. It packages an asynchronous workflow into a task< ' t> object that can be used to execute it. The task can be started with the start or runsynchronously method, and behaves like an Async basic operation. To get the results of a workflow, you can use the result property, which is blocked if the workflow is not completed.
Parallel	seq<async< ' a>>-Asyn c<array< ' A>> Gets the collection of asynchronous workflows and returns a workflow that executes all parameter values in parallel. The returned workflow waits for all operations to complete, and then returns the result in an array.

In Listing 13.2, we initially used async.runsynchronously[2], blocking the calling thread, which is useful for testing the workflow interactively. In the next step, we create a list of workflow values. In addition, here, do not start any run. Once we have this set, we can use the Async.Parallel method [3] to generate a workflow that executes all the workflows in the list in parallel. At this point, no original workflow is executed, and to execute, you need to use async.runsynchronously again to start the combined workflow and wait for the result. The combined workflow starts all workflows and waits for all workflows to complete.

While waiting for the overall result, the code is still blocked, but it is more efficient to run, and it uses the. NET line pool to balance the maximum value of the running thread. If we were to create hundreds of tasks, it would not be possible to create hundreds of threads, because it would be inefficient to use a small number of threads instead. When a workflow uses let! Structure, when a call is made to an asynchronous basic operation, a callback is registered in the system and the thread is freed. Because. NET uses thread pooling to manage threads, threads that do work can be reused to initiate additional asynchronous workflows. When we use asynchronous workflows, the number of tasks that run in parallel can be far greater than the number of threads that are used directly.

In this chapter, we need to get the data interactively, so we are interested in running the workflow in parallel rather than developing responsive graphics applications. The latter type of application (also known as the Responsive Application (reactive applications)) is also important and we will discuss this topic in chapter 16th. Now that we've seen the code for working with asynchronous workflows, let's see how they're implemented.

13.1.2 Asynchronous Download Web page