How to get started with Dynamodb and some considerations

What is Dynamodb

Amazon DYNAMODB Distributed NoSQL Database service supports dynamic scaling and reliable performance

Dynamodb basic concept of dynamodb composition

• Table: Represents the table used to store dynamodb data, similar to a relational database
• Items: The table can have 0 to n (n > 0) items that resemble the "rows" of a relational database, where there is no limit to the number of items in Dynamodb.
• Properties: Multiple properties make up an item that resembles a field in a relational database. The tables in Dynamodb are non-schema in addition to the primary key, so they can have different properties, sizes, data types in the project

DYNAMODB PRIMARY Key

A primary key is used in Dynamodb to identify a unique item, and a primary key must be specified when the table is created. DYNAMODB primary key supports two types

• A primary key consisting of a single partition key: consists of a partition key attribute that identifies the unique item and which partition the item is stored in. Dynamodb calculates the value of the partition key based on the internal hash function, and the result is used to determine which partition the item is stored in.
• Partition key + sort key: Adds a sort key based on a single partition key, which consists of two properties that compose a composite primary key. Similarly, the partition key determines which partition the item is stored in, and the sort key determines the order in which the items of the same partition key are arranged within the same partition.

Dynamodb Stream

DYNAMODB streams can capture data modification events in tables that are written to the stream in the order in which they occur. The flow has a life cycle, which is 24 hours from the count to the end and is automatically removed from the stream when it expires.

The following events will trigger the generation of the stream

• When you add a new project to a table, the stream captures the entire item, all properties
• When you update an item, the stream now moves the wave or modified property in the project
• When you delete an item, the stream is captured before the item is deleted

Dynamodb Level Two Index

DYNAMODB provides a local two-level index and a global two-level index, which is somewhat different.

	Local Level Two index	Global Level Two Index
Concept	The meaning of a local level two index is "local", meaning that the index of a level two index is scoped to a table partition with the same partition key. This means that the index can only be indexed in the same partition	Globally, a query executed against an index can span all the data of all partitions in the base table
Consistency	Support Strong consistency	Only final consistency is supported
Limit	Only 5 local two-level indexes can be created in a table	Only 5 Global two-level indexes can be created in a table

Dynamodb Read-Write restrictions

When you create a table, you specify the read, write throughput of the table. In a production environment, if the actual throughput exceeds the currently provisioned throughput for DYNAMODB, an exception will eventually be thrown after retrying (if the DYNAMODB client sets a retry mechanism).

• Read throughput units: Read throughput depends on the size of the project and whether final or strong consistency is required
  Final consistent read capacity Unit = 2 times 4KB Project read/sec
  Strong consistent Read capacity Unit = 1 times 4KB Reads/sec
  If you read an item larger than 4KB at one time, Dynamodb consumes an additional unit of read capacity.
• Write throughput units: Write capacity units depend on the size of the write item.
  One write capacity unit = 1 times maximum of 1KB Project Writes/sec
  If you need to write a project larger than 1KB, Dynamodb consumes additional write capacity units.
• Other restrictions: DYNAMODB Specifies a project size limit of 400KB, which will consume more capacity units if the size of the project exceeds this limit.

Once the dynamodb limit is exceeded, the request is throttled, and the error is not automatically recovered in the program, so the throughput and size of the project need to be set appropriately.

Dynamodb Throughput Preset value calculation

Throughput can be specified when the table is created, and the in-line business can be dynamically adjusted via the DYNAMODB background visualization interface or the provided API. The API approach is relatively flexible. Note: Dynamodb allows each table to be adjusted only 4 times per day, with no limit to the upward adjustment .

This preset value has a set of calculations, if you can estimate the amount of business read and write, please refer to the following formula

• Strong consistent read capacity calculation: Rounding up (project size/4KB) * Estimated number of reads per second
  For example: Strong consistency read requirements, a project size of 3KB, expect to read 80 items per second.
  3KB/4KB = 0.75, rounding up = 1
  1 * 80 = 80 Read capacity units
• Final consistent read capacity calculation: As with strong consistent read capacity calculation, on the final result * 2
• Write capacity calculation: Rounding up (project size/1KB) * Estimated number of writes per second
  For example, the project size is 512 bytes and is expected to hit 100 items per second.
  512 Bytes/1KB = 0.5 rounding up = 1
  1 * 100 = 100 Write capacity units

DYNAMODB partition initial partition count calculation

When the data is stored, DYNAMODB divides the table's items into multiple partitions and is supported by the SSD. The distribution of data is mainly determined by the partition value.

The DYNAMODB service is solely responsible for the management of the partition, including the number of start table partitions and partition splitting.

First we need to know that a partition in Dynamodb can hold approximately 10GB of data, supporting up to 3,000 read requests per second and 1000 write requests per second.

When you create a table, the number of table partitions is initialized according to the read and write throughput of the preset, and the formula is as follows:

Number of initialized partitions = rounding up (read preset throughput/3000) + (write preset throughput/1000))

For example: Read preset 5000, write preset 2000, then apply the formula (5000/3000) + (2000/1000) = 3.6667 rounding up = 4 partition number

Then each partition can support 5000/4 = 1250 reads and (2000/4) = 500 writes

Partition splitting

Dynamodb can be dynamically scaled, and throughput is guaranteed. But these are based on its partitioning design.

Partition Split mode

Since DYNAMODB is solely responsible for partition management, the timing and manner of partition splitting is determined by Dynamodb. When necessary, Dynadmodb automatically splits existing partitions to provide more partitions to support throughput. Let's start by understanding how to split

1. In the 1th step, Dynamodb assigns two new partitions
2. In the 2nd step, Dynamodb distributes the original partition data evenly to the new partition
3. In the 3rd step, Dynamodb no longer assigns data to the original partition.

Partition Split Trigger condition

As mentioned above, "a partition in Dynamodb can hold approximately 10GB of data, supporting up to 3,000 read requests per second and 1000 write requests per second"

Then the partition trigger condition is also triggered around the actual storage and reading and writing of the partition.

• Provisioned throughput Increase: If the current partition table does not meet the new provisioned throughput, Dynamodb expands the current number of partitions by one
  
  

  The diagram shows that the initial table is assigned 4 partitions (read preset 5000, write preset 2000, then apply formula (5000/3000) + (2000/1000) = 3.6667 rounding up = 4 number of partitions)
  Each partition has 1250 read units and 500 write units, at this time the read capacity from 5000 to 8000, at this time, 4 partitions can not be satisfied, Dynamodb doubled the number of partitions 4 * 2 = 8 partitions, so each partition soy sauce 1000 read units and 250 write units

• Storage requirements Increase: If a partition's data exceeds the limit, 10G,DYNAMODB will split the partition into two, and the data is evenly distributed to the new two partitions. There are many reasons why a partition exceeds the limit of 10G, which is more common is that the partition key is not hashed enough, causing the data to be shifted to a partition.
  
  

  , when the red box partition fills up the data, DYNAMODB will split the partition out of two new partitions, with a total capacity of 8 partitions * 10GB capacity = 80GB, and a total capacity of 9 partitions after the split * 10GB capacity = 90GB
  Note: This split of two partitions from a partition will only share the read and write throughput of the original partition. As an example:
  5000 reads and 2000 writes, DYNAMODB creates 4 partitions at this time, and each partition reads and writes capacity of
  5000/4 = 1250 Read capacity
  2000/4 = 500 Write capacity
  Assuming that one of the partitions is about to full 10g,dynamodb will split the partition out of two partitions, the table has 5 partitions, and their write capacity is
  of which 3 partitions read and write capacity is still 1250 read capacity and 500 write capacity, two partitions are removed read and write for 1250/2 = 625 reads and 500/2 = 250 writes

Reference: https://docs.aws.amazon.com/zh_cn/amazondynamodb/latest/developerguide/Introduction.html

How to get started with Dynamodb and some considerations