Cloud design mode (21)--sharding Shard mode

Cloud design mode (21)--sharding Shard mode

Stores one data to a set of horizontal partitions or fragments. This mode can improve scalability when storing and accessing large amounts of data .

Background and issues

Data stores hosted by a single server may be subject to the following limitations:
• Storage space. A data storage for a large cloud application can be expected to contain a huge amount of data that can be significantly increased over time. A server typically provides only a limited amount of disk storage, but it can be as large as replacing an existing disk, or adding additional disks to the machine as the volume of data increases. However, thus, systems that cannot easily increase the storage capacity on a given server will eventually reach a hard limit.
• Compute resources. Cloud applications may need to support a large number of concurrent users, each of which runs a query that stores information for the retrieved data. A single server-hosted data store may not be able to provide the required computing power to support that load, thereby prolonging the response time for users and failures as applications attempt to store and retrieve data timeouts. It may increase memory or upgrade the processor, but when it does not increase computing resources any further system will reach the limit.
• Network bandwidth. Finally, the performance of a data store running on a single server is constrained by the ability of the server to receive requests and send response rates. It is possible that the amount of network traffic may exceed the capacity of the network that is used to connect to that server, resulting in a failed request.
Geographical It may be necessary to store those users that are generated by a particular user in the same zone as legitimate, compliant, or performance reasons, data, or to reduce data access delays. If users are dispersed in different countries or regions, they may not be able to store the entire data for application in a single data storage area.

Vertical scaling by adding more disk capacity, processing power, memory and network connectivity may delay some of these limitations, but it is likely to be just a temporary solution. Business cloud applications that can support a large number of users and large amounts of data must be able to scale almost unlimited, so vertical scaling is not necessarily the best solution.

Solution Solutions

Partition data to be stored horizontally or fragmented. Each fragment has the same pattern, but saves its unique subset of data. A fragment is a data memory that is in its own right (it can contain data for many different types of entities) to run on a server that serves as a storage node.

This mode has the following advantages:
• You can extend the system by adding additional shards running on the extra storage nodes.
• The system can use off-the-shelf commodity hardware instead of dedicated (and expensive) computers for each storage node.
• You can reduce contention and improved performance by balancing workload across fragments.
• In the cloud, fragments can be physically close to the users who will be accessing the data.

When a piece of data is stored as fragmented, decide what data should be placed on each fragment. A fragment typically contains items that fall within a specified range determined by one or more properties in the data. These attributes form The Shard key (sometimes referred to as the partition key). The key to the Shard should be static. It should not be based on data that may vary.

The data of the fragmented body tissue. when an application stores and retrieves data, the logical indication of that Shard is applied to the corresponding fragment. This split logic can be implemented as part of the data access code in the application, or it can be implemented by the data storage system if it transparently supports sharding.

The physical location of the abstracted data of the split logic provides a high level of control over the fragmentation of the contained data, and the migration of data between shards, while the business logic of the reprocessing application should need to be introduced after the fragmented data (for example, if fragmentation becomes unbalanced). The tradeoff is in determining the location of each data item, because it is retrieved with the overhead of additional data access required.

In order to ensure optimal performance and scalability, it is important to split the data in a way that fits the query type of application execution. In many cases, it is not possible that the split plan will fully meet the requirements of each query. For example, an application in a multiuser system may need to retrieve tenant data by using the tenant ID, but it may also need to be looked up based on other attributes, such as the tenant's name or location. To handle these situations, implement a sharding strategy, which is the key to supporting a sub-library of the most commonly used query execution.

If the query periodically retrieves data by using a combination of property values, this may be a composite shard key that can be defined by combining the attributes. In addition, use patterns, such as index tables, to provide fast lookups to non-covered fragments of critical base attribute data.

Sharding Policy

Three strategies are commonly used when selecting the Shard key and deciding how to distribute data across fragments. Note that it is not necessary to be fragmented and to host them-one-to-many correspondence between servers in a single server that can host multiple tiles. These strategies include:
• Find policies. In the above strategy, the logic of the Shard implements a graph that routes requests to the data to contain fragments by using the Shard key data. In a multi-user app, all the data for a tenant may be stored together in a shard by using the tenant ID as a sharding key. Multiple tenants may share the same fragmentation, but data for a single tenant will not be scattered across multiple fragments. Figure 1 shows an example of this strategy.

Figure 1-Shard tenant data based on tenant ID


The mapping of sharding keys and physical storage can be based on physical chunking, with each shard key mapped to a physical partition. Replaceable, this technique provides greater flexibility when rebalancing fragments by using the virtual partitioning method, where the Shard key is mapped to the same number of virtual fragments, which in turn maps to fewer physical partitions. In this approach, an application uses a fragment key that refers to a virtual fragment to locate the data and transparently map the virtual shards of the system to the physical partitions. The virtual fragments and physical partitions that are modified to use a different set of fragments of the keys can be changed without the need for mapping between application code.

• Range of policies. This is continuous with the same shard-related items in the policy group, and placing their orders through the Shard key Sharding key. It is useful for applications to retrieve itemsets frequently by using a range query (that is, to return a set of data items for a fragment key query that falls within a given range). For example, if an application often needs to find all orders placed in a given month, that data can be retrieved faster if all commands for one months are stored in the same shard in the order of the date and time. If each order is stored in a different fragment, they will have to be taken out separately by making a large number of point queries (queries that return a single item of data). Figure 2 shows an example of this strategy.

Figure 2-Fragment storage order collection in the data (range)


In this example, the Sharding key is a key combination, including the order month as the most significant component, followed by the day and time. When you create and attach a new order to a fragment, the data in the order is naturally sorted. Some data store support includes a two-part shard key that identifies the fragment and the line key that uniquely identifies the item partition key element in the sub-Library. Data is usually held in the order in which the fragments are sorted. The scope of the query and the combination of required items can use a Shard key that has the same value as the unique value for the partition key but the row key.

• Hash policy. The goal of this strategy is to reduce the chance of a data hotspot. Its purpose is to allocate the data in the entire fragment in a way that implements the size and average load of each fragment, each of which encounters a balance between each fragment. The logical calculation of a shard, which is based on the hashing of one or more properties of the data to store the sub-Library of the item in. The selected hash function should be evenly distributed across the entire data fragment, possibly by introducing some random elements into the calculation. Figure 2 shows an example of this strategy.

Figure 3-Hash shard tenant data based on tenant ID


• Learn more about the benefits of hashing policies over other Shard policies and consider how to enroll new tenants in a multi-tenant application may allocate data stores in tenant fragmentation. When using the scope of the policy, the tenant 1 to n data will be stored in Shard A, the data for the household of N +1 to M will be stored in Shard B, and so on. If the most recently registered tenant is also the most active, the most data activity will occur in a few fragments, which may lead to hotspots. In contrast, the hash policy allocates a tenant based on the hash fragment of its tenant ID. This means that sequential tenants are most likely to be assigned to different fragments, 3 as shown by tenants 55 and 56, which will distribute the load on those fragments.

The following table lists the major advantages and considerations for these three shard policies.

Strategy strategy	Advantages Advantages	Considerations consider
Lookup lookup	More control over the shards is configured and used. Using virtual shards reduces the impact when rebalancing data because new physical partitions can is added to even out The workload. The mapping between a virtual Shard and the physical partitions that implement the Shard can be modified without affecting Application code, uses a shard key to store and retrieve data.  Better control how fragmentation is configured and used. When rebalancing data, because the new physical partition can be added to the effect of flattening the workload using virtual fragmentation reduction. The mapping between the virtual fragments that are modified by the application code that uses a sharding key to store and retrieve data, and the physical partitions that implement the Shard can be used without affecting.	Looking up shard locations can impose an additional overhead.
range  Scope	Implement and works well with range queries because they can often fetch multi ple data items from a single shard in a single operation. Easier data management. For example, if users in the same region is in the same shard, updates can is scheduled in each time zone based on the LO Cal load and demand pattern. Easy to implement and use range queries work well because they can typically take multiple data items from a single shard in a single operation. for easier data management. For example, if the user in the same region is in the same sub-library, the update can be scheduled in each time zone based on the local load and demand pattern.	may not provide optimal balancing between shards. Rebalancing shards is difficult and may not resolve the problem of uneven load if the majority of activity are for Adjac Ent shard keys. The may not provide the best balance between fragmentation. Rebalancing fragmentation is difficult and may not solve the problem of uneven loads if most of the activity is on adjacent shard keys.
Hash hashes	Better chance of a more even data and load distribution. Request routing can accomplished directly by using the hash function. There is no need to maintain a map. A better chance of even more data and load distribution. Request routing can be implemented directly by using a hash function. There is no need to maintain a map.	Computing The hash may impose an additional overhead. Rebalancing shards is difficult. The calculation of the hash may incur additional overhead. It is difficult to rebalance debris.

The most common split scenarios implement one of the above methods, but you should also consider the business requirements of your application and their data usage patterns. For example, in a multi-user application:
• Can be fragmented based on workload data. You can separate the data of volatile tenants in different fragments. The speed of data access for other tenants can be increased as a result.
• Can be fragmented based on the tenant's location data. It may take data on tenants in a particular geographic area offline during off-peak backup and maintenance, while for households in other areas the data remains online and accessible during their working hours.
• High-value households can share their own private high performance, light-load debris, while lower-value households may be expected to be sharing more densely stacked, busy fragments.
• For heights that require data isolation and privacy, you can store data for tenants on a completely separate server.

Scaling and data movement operations

Each shard policy means the scale of different functions and complexity management, scaling out, data movement, and maintaining the horizontal state.

The find policy allows zooming and data movement operations to take place at the user level, whether online or offline. The technology pauses some or all of the user activity (perhaps during off-peak hours), moves the data to a new virtual partition or physical fragment, alters the mapping, invalidates or refreshes the cache holding the data, and then lets the user activity recover. Typically this type of operation can be centrally managed. The status of the Find strategy requirement is highly cacheable and replica friendly.

The scope of the policy specifies scaling and data movement operations, which usually must be done when part or all of the data is stored offline, because the data must be segmented and some limitations of the entire fragment merge. Moving the data to rebalance the fragments may not solve the problem of uneven load, if most of the activity is within the same range as the adjacent Shard key or data identifier. The scope of the policy may also need to be maintained in some States of the physical partition within the graph range.

The hashing policy makes the extension and data movement operations more complex because the partition key is a hash of the fragment key or data identifier. The new location of each fragment must be determined from the hash function, or the function is modified to provide the correct mapping. However, the hash policy does not require a maintenance state.

Issues and considerations

When deciding how to implement this pattern, consider the following points:
• Shards are complementary to other forms of partitioning, such as vertical partitioning and functional partitioning. For example, a single sub-library can contain vertically divided entities and functional partitioning can be implemented as multiple fragments. For more information about partitioning, see data partitioning guidance.
• Keep the balance fragments so that they all deal with similar volumes of I/O. As data is inserted and removed, it may be necessary to periodically rebalance the fragments to ensure uniform distribution and reduce the chance of hotspots. Rebalancing can be an expensive operation. In order to reduce frequency and rebalance becomes necessary, you should ensure that each fragment contains enough free space to handle changes in expected volume plan growth. You should also develop strategies and scripts that you can use to quickly rebalance the shards that should become necessary.
• The key to using stable data shards. If the Shard key changes, the corresponding data items may need to be moved between fragments, increasing the amount of work performed by the update operation. For this reason, avoid debris critical based on potential volatility information. Instead, look for key attributes that are immutable or naturally formed.
• Make sure that the Shard keys are unique. For example, you should avoid using auto-incrementing fields as the key to Sharding. In some systems, auto-increment fields can not be reconciled across fragments, which can result in items that have different fragments of the same shard key.

Note:

Not including sharding keys can also cause problem fields to automatically increment. For example, if you use the self-increment field to generate a unique identity and distribute two different items in different fragments, the same ID may be assigned.


• It may not be possible to design a sharding key that meets every possible query for data. Shards of data to support queries that are most frequently made and, if necessary, to create a Level two index table to support queries that retrieve data by using attribute criteria that are not part of a shard key. For more information, see indexed table mode.
• Query access only a single fragment is more efficient than those that are retrieved from multiple tiles, thus avoiding the execution of a shard scheme that causes a large number of applications to perform queries on the data that the connection maintains in different fragments. Keep in mind that broken tiles can contain data for multiple types of entities. Considering non-normalized data to keep the same fragmentation is often considered a query (such as customer details and orders that they have put) related entities to reduce the number of individual reads performed by the application.

Note:

If an entity in one sub-library references an entity that is stored in another shard, including the Sharding key for the second entity, as part of the schema of the first entity. This can help improve the performance of referencing data queries across fragments.


• If an application must perform a query that retrieves data from multiple tiles, it is possible to obtain the data by using a parallel task. Examples include fan-out queries, where data from multiple shards is retrieved in parallel and then aggregated into one result. However, this approach inevitably adds some complexity to the data access logic of the solution.
• For many applications, the larger number of small fragments produced can be more effective than having a small amount of large fragments because they can provide increased opportunity for load balancing. It can also be useful if it is expected that fragmentation needs to be moved from one physical location to another. Moving small fragments is faster than moving a large one.
• Ensure that the resources provided to each shard storage node are sufficient to handle the scalability requirements of the scale and throughput of the data. For more information, see Data Partitioning boot section, "Design Partition extensibility”。
• Consider copying all fragments of the reference data. If the operation retrieving data from a sub-library also references static or slow-moving data as part of the same query, the data is added to the fragment. The application can then read all the data used for easy querying without having to make additional round trips to a separate data store.

Attention:

If the base data is kept in a plurality of chunked changes, the system must synchronize all the fragments for these changes. When this synchronization occurs, there may be some degree of confusion in the system. If you follow this approach, you should design your own application to handle this contradiction.


• It can be difficult to maintain the referential integrity and consistency between fragments, so you should minimize the impact on multiple fragmented data operations. If the application must modify the data through fragmentation, it is actually a requirement to evaluate whether the full data consistency is true. Conversely, a common approach in the cloud is to achieve eventual consistency. The data in each partition is updated separately, and the application logic must assume that the responsibility of the update is successfully completed, and that the processing can result in inconsistencies from querying the final consistent operation of the data. For more information on achieving eventual consistency, see data consistency Primer.
• Configuring and managing large amounts of fragmentation can be a challenge. Tasks such as monitoring, backup, checking consistency, and logging or auditing must be completed on multiple fragments and servers, which are likely to remain in multiple locations. These tasks may be done by using scripts or other automation solutions, but scripting and automation may not completely eliminate the additional administrative requirements.
• Fragments can be geo-positioned so that they contain data that is close to an instance of the application that uses it. This approach can significantly improve performance, but requires extra consideration for tasks that must access different locations in multiple tiles.

when to use this mode

Use this mode:
• When data storage may need to extend beyond the limits of a single storage node's resources.
• Improve performance by reducing contention for data storage.

Attention:

The main focus of sharding is to improve the performance and scalability of the system, and as a by-product, it is also possible to improve availability by dividing the data into separate partitions. Failure in one partition does not necessarily prevent the application from accessing the data stored in other partitions, and the operator does not need to make the entire data accessible to the application for maintenance or recovery of one or more partitions that can be performed. For more information, see Data partitioning guidelines.

Example

The following example uses a set of SQL Server databases that act as fragments . Each database contains a subset of the data used by an application. The application retrieves the data that is distributed throughout the fragment by using its own shard logic (which is an example of a fan-out query). The details of the data that will be located in each of the sub-Libraries are returned by this method called Getshards. This method returns the Shardinformation object, where the Shardinformation type contains an identifier for each fragment and the SQL Server connection string. The application should use an enumerated list that is attached to the fragment (no code example in the connection string is shown).

[CSharp]View Plaincopy

2. Private ienumerable<shardinformation> Getshards ()
4. {
6. //This retrieves the connection information from a shard store
8. //(commonly a root database).
10. return new[]
12. {
14. New Shardinformation
16. {
18. Id = 1,
20. ConnectionString = ...
22. },
24. New Shardinformation
26. {
28. Id = 2,
30. ConnectionString = ...
32. }
34. };
36. }

The following code shows how the application uses the Shardinformation object list to perform a query that fetches data from each fragment in parallel. The details of the query are not shown, but the data retrieved in this embodiment includes strings that can hold information, such as the customer's name, if the fragment contains the details of the customer. The results are processed by aggregation of applications into Concurrentbag collections.

[CSharp]View Plaincopy

2. Retrieve the shards as a shardinformation[] instance.
4. var shards = getshards ();
8. var results = New concurrentbag<string> ();
12. Execute the query against each shard in the Shard list.
14. This list would typically is retrieved from configuration
16. Or from a Root/master shard store.
18. Parallel.ForEach (shards, shard =
20. {
22. //note:transient fault handling is not included,
24. //But should is incorporated when used in a real world application.
26. using (var con = new SqlConnection (Shard). ConnectionString))
28. {
30. Con. Open ();
32. var cmd = new SqlCommand ("SELECT ...  From ... ", con);
36. Trace.traceinformation ("Executing command against Shard: {0}", Shard.  ID);
40. var reader = cmd. ExecuteReader ();
42. //Read The results in to a THREAD-SAFE data structure.
44. While (reader. Read ())
46. {
48. Results. ADD (reader. GetString (0));
50. }
52. }
54. });
58. Trace.traceinformation ("fanout query Complete-record Count: {0}",
60. Results. Count);

This article is translated from msdn:http://msdn.microsoft.com/en-us/library/dn589797.aspx

Cloud design mode (21)--sharding Shard mode