Grid Computing Based on azure cloud computing platform, Part 1: Running grid applications

ArticleDirectory

• Grid Application Solution Structure
• Set local database
• Configuration Solution
• Test and step-by-step advancement of grid applications across Environments
• Local test run
• View Grid
• View task
• View results
• Run grid applications in the cloud
• Performance and Tuning

In part 1 of this series, we introduced the design model for grid computing development on Azure, and in part 2, we wrote a fraud scoring Grid Computing Application.ProgramAnd name it fraud check. Here, in part 1, we will run this application in the future, first running locally, and then moving to the cloud.

 

The framework we use is azure grid, which is from the neudesic grid computing framework.Community.

Before running our grid application, there are several things that need to be processed-such as configuration tracking database and configuration information-but let's first see what the application runs in the cloud:

Let's complete the settings in the following steps so that we can run it.

Grid Application Solution Structure

Azure GRID provides you with a basic solution template where you can add application-specificCode. Last time, we added code for the fraud check grid application, but did not explain the solution structure itself. Let's talk about it now.

Our grid computing solution consists of the cloud and the Enterprise. The Azure grid framework includes all of these in a separate solution, as shown below. So what are there in the solution?

• The gridapplication project contains the application-specific code we have written in the second part of this series of articles. Recall that we have compiled three pieces of code: task code, loader, and aggregator. Other projects in the solution contain application code: some code will be executed in the cloud, and some will be executed in the enterprise system.
• The grid actuator code in azure is located in the azuregrid and azuregrid_workerrole projects. This is a standard azure hosting application with executor roles that can be deployed to Azure hosting projects through the azure portal. Your task application code will be executed here.
• The enterprise code is located in the gridmanager project. This is a desktop application used to start and monitor grid work. It also runs your loader code and aggregator code in the background.
• The storageclient Project is a function library that accesses cloud storage and inherits from the azure SDK storageclient example. This function library is used by grid actuators in the cloud and grid management software in enterprise systems.

To run this solution, the cloud and basic systems are simple:

• To run the grid manager, right-click the gridmanager project and choose debug> Start.
• To run the grid Actuator on a local machine in a local development environment, right-click the azuregrid project and choose debug> Start.
• To run the mesh Actuator on Azure, use the azure portal to publish the azuregrid project to Azure in the usual steps to form a hosted application.

To test all functions locally, setting azuregrid and gridmanager as startup projects is useful, so that the two applications can be started by pressing F5 only once.

Set local database

Azure grid tracks running, tasks, parameters, and results in a local SQL Server or SQL Server express database (2005 or 2008. The project source code downloaded from codeplex also contains the SQL script used to create the database.

Configuration Solution

A few configurations are required for the cloud and basic systems in the solution. The most important thing is related to cloud storage.

The mesh actuator configuration in the cloud is set in the serviceconfiguration. cscfg file of the azuregrid project, as follows:

• Projectname: name of your application
• Taskqueuename: name of the queue used to receive results from the mesh Actuator in cloud storage
• Queuetimeout: How long a task can be executed before it is overwritten by another Executor (in seconds)
• Sleepinterval: The interval (in seconds) between sleep before the grid executor checks the new task to be executed)
• Queuestorageendpoint: the queue storage endpoint used for cloud storage projects or local development and storage.
• Accountname: name of the cloud storage project
• Accountsharedkey: the Key of the cloud storage project.

The configuration used for grid manager is set in the application. config file of the gridmanager project. Most settings have the same name and meaning as the configuration file in the cloud, and must be set to the same value. Set the connection string of an extra local SQL Server database for tracking records of grid applications.

Test and step-by-step advancement of grid applications across Environments

Because grid applications can be greatly scaled, you should test them carefully. In the azure-based grid computing scenario, we recommend that you test the new grid application in the following order:

1. developer Testing

Use a small number of tasks to test grid applications on local development machines. Here, you need to check whether the application runs correctly and whether the correct data is moved between the grid application and the basic storage.

• For cloud computing, the grid application is executed in the local development environment.
• The cloud storage part is also replaced by a local development environment.

Of course, grid Manager (always) runs on a local machine.

2. QA testing

In the local environment, you can use a large number of tasks to test applications on multiple executor machines. The purpose of this test is to verify that the same work can be performed in a single-host environment and in a multi-host environment using cloud storage.

• For cloud computing, the grid application is executed in the local development environment, but on multiple local machines.
• For cloud storage, use the azurestorage project.

Note that this method is still mainly a local test, but it only uses azure cloud storage. This method is required when we cannot coordinate the work across multiple machines.

3. Staging Environment

Deploy the grid application to the azure stage environment and run it with a small number of tasks. At this stage, you can verify whether the local work can run normally in the cloud.

• For cloud computing, the grid application is deployed in the azure hosting project-staging.
• For cloud storage, the azure storage project is used.

4. Production Environment

Now, you are ready to push your application to the level of azure managed production environment and run it with saturated load.

• For cloud computing, grid applications are hosted in the azure hosting project-production.
• For cloud storage, the azure storage project is used.

You can use azure manager to monitor whether grid applications are running properly.

Local test run

Now we are going to perform a local test on our development machine. According to our test plan, we initially tested it on a local development machine with a small amount of load.

The input data is displayed in the following CSV worksheet, which is created and saved as a CSV file in Excel. The opening account number of the input file is the same as the account used to start gridmanager.exe. In this test, we will use 25 records.

To ensure correct database settings, configurations in the cloud and enterprise must be reflected in local development and storage.

Compile the application and start azuregrid and gridmanager projects.

The azuregrid project does not display anything visible, but you can check its running status through the development environment UI (developer fabric UI) if you want.

The gridmanager application is a WPF-based console, as shown in the following figure after it is started:

To start a job in the application, select job> new job from the menu. In the displayed dialog box, enter the name and description.

Click OK, and you will see that the job has been created but not yet running. The work panel is displayed in red, indicating that the work has not started.

Now, we are ready to start the grid application. Click the start button. Soon you will see the work panel yellow and the grid of the task is displayed.

The result is updated every 10 seconds. Soon you will see some tasks in red, yellow, and green. The task is displayed in red while waiting, yellow when executed, and green when executed.

After all tasks are completed, the Work panel is displayed in green.

After the grid application is executed, let's check the results. Fraud check is written to read input parameters from the input data table, and then write the results to the output data table. Open the newly created output workbook and we can see that the grid application creates a scoring, plain text, and approval decision for each input record.

View Grid

In the test run we just completed, we see a grid view, so that we can understand the overall implementation of the grid application. Each small box represents a grid task, and the task id is displayed in the box. In the figure below, you can see a grid view that shows a large number of tasks.

• Red tasks are waiting, meaning they are not yet executed. These tasks are stored in the cloud storage task queue (the loader puts the tasks in this queue ). That is to say, these tasks have not been extracted by grid actuators.
• The yellow task is in progress. You will usually see a small yellow box with the same number of executor role instances.
• The green task is finished. These tasks have been executed and their results have been put on the aggregation queue to wait for the aggregation to be extracted.

In addition to grid view, you can also view tasks and results. The grid view, task view, and results view buttons allow you to see different views of grid applications.

View task

You can click the View button to view information about a task during or after running the task. Task view displays a row of data for each task, including the task id, task type, and worker machine name. When running locally, the name of your local machine is always listed. When running in the cloud, you will see the name of the specific machine responsible for processing each task.

View results

Result view is similar to task view-each task has a row of data-but the input parameters and results are displayed for each task in XML format. You can also extend the form to view the complete information.

Run grid applications in the cloud

After reading the grid application running on the local machine, let's run it on the cloud. This requires us to modify the configuration information to use the cloud storage project rather than local development and storage, and deploy the azuregrid project to the cloud hosting project.

The steps for deploying to the cloud are similar to any managed azure application:

1. Right-click the azuregrid project and select publish.

2. Access the managed azure project on the azure portal, and click deploy under staging to upload your application package and configuration file.

3. Click Run to run your instance.

The next step is to prepare your input data. In our local test environment, the data entered into the application is an electronic data table containing 25 rows of application information to be processed. This time, we will use a big data with 1000 rows.

Since some of the applications in the cloud have been running on the azure platform, we only need to start the grid Manager application locally.

We use the new job menu item in the azure manager menu again to select a job to start.

As before, click OK to complete the input of the dialog box, and then click Start to start the operation. In the grid view, you can see that the loader has generated 1000 tasks.

Switch to task view. You can see the machine name of the cloud executor that executes each task.

You can check the results through results view even before the grid application completes computing (as shown previously ).

After the work is completed, the cloud storage has been cleared. As there is no work to do, we can pause the applications deployed in the cloud to avoid extra computing costs.

Once the application completes running, the expected results (in the example of fraud check, the output result is a CSV file) will be ready, the number of rows is as expected (consistent with the number of input data), and each task has a computing result.

We can see that 1000 rows of data have been processed, but the records are not in the initial order. This is normal: we cannot control the sequence of task execution, and all we care about is that each task guarantees its atomicity. One reason for copying input parameters to the output result is that the input information and result information can be associated.

Performance and Tuning

Currently, it is difficult to estimate the performance of grid computing applications in azure, because we can only use the azure preview version, but it is not yet available in mainland China. For details, see here. ], It provides developers with only two instances for testing.

However, there is one way to make your grid computing application more than two instances, that is, hosting the grid Actuator in multiple places. You can use multiple managed accounts to work with other azure account owners. You can also run some executors locally. This is feasible because the queue in cloud storage is the communication mechanism between the grid actuator and the enterprise loader/aggregator. As long as you use a general storage project, you can place grid execution in various places.

We also hope that grid computing applications can also come true in azure: for example, computing-intensive applications can achieve huge returns through grid computing.

As azure's official launch approaches, we should be able to collect a lot of meaningful data about grid computing performance and how to optimize it.

View Original English text: Grid Computing On the azure cloud computing platform, Part 3: running a grid application.