In this three-part series, we will take a look at grid computing using the Azure cloud computing platform. In part 2, we will see the design patterns involved and some useful ideas. In sections 2nd and 3rd, we will see a code example to demonstrate a grid computing framework specifically developed for Azure.
Not everyone knows the difference between grid computing and cloud computing, so let's give a brief explanation of the two. Although grid computing and cloud computing are not the same thing, there are many areas of collaboration between them and it is very meaningful to use them together.
Grid computing
Grid computing uses a large number of computers working in parallel mode rather than standalone mode to handle computing problems. This method has many advantages:
- Save time: if a single machine needs to handle the task for one month, it may take one day if there are 30 computers dedicated to this problem. The largest Grid Computing Project in history is the SETI @ home project. by leveraging the capabilities of hundreds of thousands of volunteer computers, it took two million years of total computer processing time.
- Cost saving: You can use lower-cost resources to complete your work, rather than purchasing a large number of servers with the highest level of processors and memory. Even if you have to purchase so many computing resources-but you can buy smaller and cheaper machines, they are more likely to be used for other purposes.
- Reliability: grid computing systems must take into account faults in advance, or changes in the availability of individual computers will not impede the successful completion of work.
Not all types of work are suitable for grid computing. Only jobs that can be divided into multiple small tasks are suitable, and computers that process these tasks can run in parallel in a loosely coupled network. To distribute tasks, collect results, and manage systems, it is necessary to establish an intelligent infrastructure. Don't be surprised. early adopters of grid computing need to solve some huge computing problems. Therefore, you will see that grid computing is now used for genetics, Insurance Statistics computing, astronomical analysis, and animated film rendering. However, it has changed: Grid computing is paying more and more attention to common business problems, and the competition brought by cloud computing is accelerating this development. Computing tasks do not have to be huge enough to benefit from grid computing, nor do they have to be computing intensive tasks for grid computing. Any task with repeated executions is suitable for grid computing. Whether you are a fortune 500 company that needs to process 4 million orders per month, or a medium-sized enterprise that only needs to approve thousands of credit applications, grid computing can be used for you. Grid computing emerged more than 10 years ago, so today's grid computing naturally does not use the cloud method. The most common method is:
- Dedicated machines: purchase a large number of computers for grid processing.
- Cycle Stealing: when other machines in your organization are idle (such as at night), they are temporarily used for grid processing. A business workstation used during the day can become a grid processing computer at night.
- Global Cycle misappropriation: The concept of application cycle misappropriation on the Internet worldwide. This is how the SETI @ home Project works. It uses Over 300000 active computers.
Cloud computing can be another Implementation Method of grid computing. It has many attractive features and can be flexibly scaled according to your business model, in addition, many traditional methods must be used to customize the development support infrastructure.
Business applications for Grid Computing
To bring grid computing into the mainstream, some compelling business applications need to appear. Let's take a look at three types of business applications that are suitable for grid computing and the huge commercial value they have gained. The first example is data mining. Data mining and other forms of data analysis can identify interesting relationships and patterns from business data.
- Grocery stores can use data mining to study purchase models and make strategic decisions on discounts for certain commodities.
- Video Rental chain stores can use data mining to determine which new videos to recommend to customers based on their previous rental records.
- E-commerce websites can use data mining to study users' navigation modes in real time and adjust the display of commodities and advertisements to be displayed.
The second example is decision analysis, which is to execute a group of forward-moving business rules to make business decisions. In some scenarios, decisions involving complex computing need to be made very quickly. The parallelism of the grid can be used to obtain a fast response time, and there will be no delay when the workload increases.
- Credit institutions can use decision-making engines to calculate credit ratings.
- Financial institutions can use the Decision-Making engine to quickly approve loans.
- Insurance companies can use the Decision-Making engine to calculate the risks of applicants and verify their billing rate.
The third type of case is batch processing. When you occasionally need to handle a large amount of sudden workloads, and there are not so many internal capabilities to handle them, you can use grid computing.
- The fax service needs to generate and deliver electronic fax documents. When sending faxes, the workload is huge, and at other times it is relatively low. Using a grid to process document generation and delivery can avoid purchasing a large amount of internal computing power that is useless for most of the time.
- Marketing activities may need to automatically send emails, faxes, letters, or voice messages to a large number of addresses. The concurrency of the grid allows such sending activities to be completed at the same time in a short period of time, regardless of the workload.
- E-commerce websites need to start a business process workflow for each order, and may need to process massive workflow instances during busy seasons. Using the grid in order processing workflows can be avoided. Using limited local resources to process these orders may result in bottlenecks and delay in processing.
These examples describe how grid computing moves beyond the limits of application types to a common form that can be considered for all business types.
Cloud computing and Microsoft's Azure Platform
Cloud computing uses a large-scale data center with intelligent infrastructure to provide you with the required computing resources. Cloud computing spans application hosting and storage and provides services for communication, workflow, security, and synchronization. Cloud computing has the following advantages:
- On-demand Scaling: there is no limit on the capacity you need.
- Billing by usage: the pay-as-you-go business model means you only need to pay based on your usage. This is not a long-term agreement. When your usage level changes, it will not limit you.
- No upfront investment required: No need to purchase hardware, maintain them, or patch the operating system. Infrastructure Costs are converted into operational costs.
- Smaller IT teams and fewer IT incidents: features such as high availability, scalability, redundant storage, and Failover have all been built into this platform.
Microsoft's cloud computing platform, also known as Azure, is currently composed of four major service fields:
- Windows Azure provides application hosting and storage services. Application hosting means that software similar to Web applications, Web services, or background work processes can be run on the cloud (that is, in the cloud computing data center. The application can be load balanced to run any number of instances you need, or you can change the number of instances at any time. Cloud storage provides Blob storage, queues, and data tables similar to file systems.
- SQL Data Services provides a complete relational database in the cloud, with the same features as SQL Server Enterprise Products.
- . NET Services provides the functions required for enterprise business processing. The Service Bus connects multiple locations or organizations on the Internet by publishing and subscribing to messages. The Access Control Service provides enterprise and joint security for applications. The Workflow Service can execute workflows in the cloud.
- Live Services provides a virtual desktop to synchronize data across computers and devices with Web applications, as well as various communication and collaboration functions on social networks.
Azure is a brand new platform. At the time of writing this article, it is still in the preview stage and is expected to be commercially available by the end of this year.
Put grid computing and Azure cloud computing together
Azure is designed to support multiple application types, but does not provide specific features for grid computing. However, Azure provides a lot of functionality required by the grid computing system. To make Azure an excellent grid computing platform, you only need to use the correct design patterns and frameworks to provide grid-specific functions. Now let's take a look at the design pattern. In the second part, we will study the frameworks that support this pattern.
The first thing you need to note about this model is that there are some software/data and some basic functions on the Azure cloud and the enterprise end. What are they doing and why are they arranged like this?
- The cloud is used to complete grid computing. The use of cloud resources is mainly for temporary work and can minimize costs. When you do not run the grid computing solution, you will not be charged.
- An enterprise is a fixed storage location for data. It is the source of input data required for grid operation and the final storage location of work results.
In this mode, the software role is:
- Grid Worker: a Grid executor is a cloud software that can execute the tasks required by Grid applications. This software runs on the cloud as a Worker Role with multiple instances. The framework uses the exchange instruction arrangement method so that any mesh executor can execute any task requested by it. The mesh actuator runs in a loop, reads the next task from the task queue, executes the task, and writes the result to the result queue. When the grid executor does not have a queue task to run, a close request is sent.
- Grid Manager is an enterprise-side software that manages the running of Grid computing. The grid manager has three components:
- Loader: The Loader generates tasks to be executed for the mesh actuator to start grid applications. The loader runs on the enterprise side to access some basic resources, such as the database that provides the required input data for each task. When the loader is running, the generated Task is written into the Task Queue in the cloud.
- Aggregator: the Aggregator reads results from the result queue and stores the results to basic functions at a fixed position. The aggregators also know whether the execution of grid applications is complete.
- Console: the Console is a series of management functions used to configure a project, start a job, and view its status during grid execution. It also provides a view similar to displaying the flight status at the airport to display tasks to be processed and completed tasks.
In this mode, the data role is:
- Task Queue: A cloud storage Queue that stores tasks. The Enterprise-side loader writes the tasks it generates to this queue. Grid executors in the cloud read tasks from this queue and execute them.
- Result Queue: This is a cloud storage Queue that stores Results. The grid executor outputs the results of each task to this queue. The aggregators running on the Enterprise end read the results from the queue and store them permanently on the Enterprise end.
- Tracking Table: This is an enterprise database Table that tracks tasks and their statuses. The loader writes related records to this tracking table, and the aggregator updates the results when receiving the results. The tracking table allows the console to display the Grid status and let the system know when the grid application is finished.
- Enterprise Data: An Enterprise provides Data storage or services to provide input Data for tasks or receive Task results. This should be set according to their respective organizations and projects; the Code for writing loaders and aggregators should be integrated with these data storage.
Walkthrough: Create and execute a Grid Computing Application on Azure
Let's mix the things mentioned above, and practice how to use this mode and the appropriate framework to develop and run grid computing applications from the beginning to the end:
1. Establishment of grid computing application requirements. Is to confirm the task to be completed, input data and the purpose of the result.
2. Using the framework, developers can add custom content to their projects:
- Azure Worker Role is created by a template and added implementation code for each task.
- The loader is created by a template and some code is added to read input data from local resources, generate tasks, and route them to the task queue.
- The aggregators are created by a template, and some code is added to receive results from the result queue and store them in the underlying database.
3. the Azure project of this application is configured for hosting and storage through the Azure portal. The mesh actuator is packaged and deployed to the cloud host. After testing, it is upgraded to the product version (Production ).
4. Use the grid Manager Console to define and start grid jobs. The loader is also started here.
5. The loader reads local enterprise data, generates tasks, and writes each task to the task queue.
6. When the grid Actuator project is started on the Azure portal, it will generate multiple instances of the grid actuator.
7. Each grid executor continuously receives new tasks from the task queue, determines the Task Type, executes appropriate code, and sends the task results to the result queue. The Azure queue method is useful here: if an executor error occurs and the task fails to be completed, the task will reappear in the queue after the timeout, it will be obtained by other mesh actuators.
8. The aggregator reads the results from the result queue and writes them to the local enterprise repository.
9. When the grid is executed, the administrator can use the grid Manager Console to observe the status of the grid executor in almost real time.
10. When the aggregator learns that all tasks in the plan have been completed, it sends a notification about this status through the console. By now, the mesh completes all its work, and its results are stored securely on the enterprise side.
11. The Azure portal allows you to pause grid actuators to avoid additional computing costs. After all the queues are fully read, cloud storage is empty, so there is no additional storage fee.
Added value of using Azure for Grid Computing
From the technical and economic aspects, the Azure platform is best for Grid Computing:
- Cost-effectiveness: The use of cloud-hosted applications avoids the need to purchase computers for grid computing. Instead, you pay monthly for the computing time and queue storage used by the grid executor. Once the grid application is processed, no additional cost will be incurred in computing time and storage time.
- Scalability and flexibility: Whether you want to get a lot or a small amount of capacity, it can be satisfied. Your grid computing application can do this by only using a single grid executor instance.
- Reliability: the reliability mechanism built in the Azure queue ensures that all tasks are executed, even if the mesh actuator is accidentally disabled. If the mesh actuator is accidentally disabled, Azure Fabric starts an alternative instance.
- Coordination: The Worker Role + queue mechanism is simple, load balancing, and works well. Using it avoids the need to write complex coordination software.
- Simple: the Core of the grid computing mode on Azure is very simple. Roles are also clearly defined. There are no very complex software elements, and the number of migration parts can be minimized as much as possible.
Next time, let's look at how to implement this mode in code using the Azure grid computing framework.
