DynamoDB is Amazon’s managed nonrelational database service. It’s designed for highly transactional applications that need to read from or write to a database tens of thousands of times a second.
The basic unit of organization in DynamoDB is an item, which is analogous to a row or record in a relational database. DynamoDB stores items in tables. Each DynamoDB table is stored across one or more partitions. Each partition is backed by solid-state drives, and partitions are replicated across multiple Availability Zones in a region, giving you a monthly availability of 99.99 percent.
Each item must have a unique value for the primary key. An item can also consist of other key-value pairs called attributes. Each item can store up to 400 KB of data, more than enough to fill a book! To understand this better, consider the sample shown in Table 9.2.
Table 9.2 A Sample DynamoDB Table
| Username (Primary Key) | LastName | FirstName | FavoriteColor |
|---|---|---|---|
| hburger | Burger | Hamilton | |
| dstreet | Street | Della | Fuchsia |
| pdrake | Drake | Paul | Silver |
| perry | Perry |
Username, LastName, FirstName, and FavoriteColor are all attributes. In this table, the Username attribute is the primary key. Each item must have a value for the primary key, and it must be unique within the table. Good candidates for primary keys are things that tend to be unique, such as randomly generated identifiers, usernames, and email addresses.
Other than the primary key, an item doesn’t have to have any particular attributes. Hence, some items may contain several attributes, while others may contain only one or two. This flexibility makes DynamoDB the database of choice for applications that need to store a wide variety of data without having to know the nature of that data in advance. However, every attribute must have a defined data type, which can be one of the following:
Scalar : A scalar data type has only one value and can be a string, a number, binary data,or a Boolean value.
Set : A set data type can have multiple scalar values, but each value must be unique within the set.
Document : The document data type is subdivided into two subtypes: list and map. Document data types can store values of any type. List documents are ordered, whereas map documents are not. Document data types are useful for storing structured data, such as an IAM policy document stored in JavaScript Object Notation (JSON) format. DynamoDB can recognize and extract specific values nested within a document, allowing you to retrieve only the data you’re interested in without having to retrieve the entire document.
DynamoDB uses the primary key to distribute items across multiple partitions. Distributing the data horizontally in this fashion makes it possible for DynamoDB to consistently achieve low-latency reads and writes regardless of how many items are in a table. The number of partitions DynamoDB allocates to your table depends on the number of write capacity units (WCU) and read capacity units (RCU) you allocate to your table. The higher the transaction volume and the more data you’re reading or writing, the higher your RCU or WCU values should be. Higher values cause DynamoDB to distribute your data across more partitions, increasing performance and decreasing latency. As demand on your DynamoDB tables changes, you can change the number of RCU and WCU accordingly. Alternatively, you can configure DynamoDB Auto Scaling to dynamically adjust the number of WCU and RCU based on demand. This automatic horizontal scaling ensures consistent performance, even during times of peak load.
Recall that nonrelational databases let you quickly retrieve items from a table based on the value of the primary key. For example, if the primary key of a table is Username, you can perform a query for the user named pdrake. If an item exists with that primary key value, DynamoDB will return the item instantly. Searching for a value in an attribute other than the primary key is possible, but slower. To locate all items with a Username that starts with the letter p, you’d have to perform a scan operation to list all items in the table. This is a read-intensive task that requires scanning every item in every partition your table is stored in. Even if you know all the attributes of an item except for the primary key, you’d still have to perform a scan operation to retrieve the item.
In general, scan operations are less efficient than other operations in DynamoDB. A scan operation always scans the entire table or secondary index. It then filters out values to provide the result you want, adding the extra step of removing data from the result set. If possible, you should avoid using a scan operation on a large table or index with a filter that removes many results. Also, as a table or index grows, the scan operation slows. The scan operation examines every item for the requested values and can use up the provisioned throughput for a large table or index in a single operation. For faster response times, design your tables and indexes so that your applications can use query instead of scan. (For tables, you can also consider using the GetItem and BatchGetItem APIs.)
Alternatively, design your application to use scan operations in a way that minimizes the impact on your request rate. When you create a table, you set its read and write capacity unit requirements. For reads, the capacity units are expressed as the number of strongly consistent 4 KB data read requests per second. For eventually consistent reads, a read capacity unit is two 4 KB read requests per second. A scan operation performs eventually consistent reads by default, and it can return up to 1 MB (one page) of data. Therefore, a single scan request can consume (1 MB page size / 4 KB item size) / 2 (eventually consistent reads) = 128 read operations. If you request strongly consistent reads instead, the scan operation would consume twice as much provisioned throughput—256 read operations. This represents a sudden spike in usage, compared to the configured read capacity for the table. This usage of capacity units by a scan prevents other potentially more important requests for the same table from using the available capacity units.
As a result, you likely get a ProvisionedThroughputExceeded exception for those requests. The problem is not just the sudden increase in capacity units that the scan uses. The scan is also likely to consume all of its capacity units from the same partition because the scan requests read items that are next to each other on the partition. This means that the request is hitting the same partition, causing all of its capacity units to be consumed, and throttling other requests to that partition. If the request to read data is spread across multiple partitions, the operation would not throttle a specific partition.
Because a scan operation reads an entire page (by default, 1 MB), you can reduce the impact of the scan operation by setting a smaller page size. The scan operation provides a Limit parameter that you can use to set the page size for your request. Each query or scan request that has a smaller page size uses fewer read operations and creates a “pause” between each request. For example, suppose that each item is 4 KB and you set the page size to 40 items. A query request would then consume only 20 eventually consistent read operations or 40 strongly consistent read operations. A larger number of smaller query or scan operations would allow your other critical requests to succeed without throttling.
Isolate scan operations; DynamoDB is designed for easy scalability. As a result, an application can create tables for distinct purposes, possibly even duplicating content across several tables. You want to perform scans on a table that is not taking mission-critical traffic. Some applications handle this load by rotating traffic hourly between two tables: one for critical traffic and one for bookkeeping. Other applications can do this by performing every write on two tables: a missioncritical table and a shadow table.
Configure your application to retry any request that receives a response code that indicates you have exceeded your provisioned throughput. Or, increase the provisioned throughput for your table using the UpdateTable operation. If you have temporary spikes in your workload that cause your throughput to exceed, occasionally, beyond the provisioned level, retry the request with exponential backoff.
Many applications can benefit from using parallel scan operations rather than sequential scans. For example, an application that processes a large table of historical data can perform a parallel scan much faster than a sequential one. Multiple worker threads in a background “sweeper” process could scan a table at a low priority without affecting production traffic. In each of these examples, a parallel scan is used in such a way that it does not starve other applications of provisioned throughput resources.
Although parallel scans can be beneficial, they can place a heavy demand on provisioned throughput. With a parallel scan, your application has multiple workers that are all running scan operations concurrently. This can quickly consume all of your table’s provisioned read capacity. In that case, other applications that need to access the table might be throttled.
A parallel scan can be the right choice if the following conditions are met:
The best setting for TotalSegments depends on your specific data, the table’s provisioned throughput settings, and your performance requirements. You might need to experiment to get it right.
In RDBMS, you design for flexibility without worrying about implementation details or performance. Query optimization generally does not affect schema design, but normalization is important. In DynamoDB, you design your schema specifically to make the most common and important queries as fast and as inexpensive as possible. Your data structures are tailored to the specific requirements of your business use cases.
For DynamoDB, you should not start designing your schema until you know the questions it will need to answer. Understanding the business problems and the application use cases up front is essential. You should maintain as few tables as possible in a DynamoDB application. Most well-designed applications require only one table.
The first step in designing your DynamoDB application is to identify the specific query patterns that the system must satisfy.
In particular, you need to understand three fundamental properties of your application’s access patterns before you begin:
Data size: Knowing how much data will be stored and requested at one time will help determine the most effective way to partition the data.
Data shape: Instead of reshaping data when a query is processed (as an RDBMS system does), a NoSQL database organizes data so that its shape in the database corresponds with what will be queried. This is a key factor in increasing speed and scalability.
Data velocity: DynamoDB scales by increasing the number of physical partitions that are available to process queries and by efficiently distributing data across those partitions. Knowing in advance what the peak query loads might be helps determine how to partition data to best use I/O capacity.
After you identify specific query requirements, you can organize data according to general principles that govern performance:
Keep related data together: Research on routing-table optimization in the late 1990s found that “locality of reference” was the single most important factor in speeding up response time: keeping related data together in one place. This is equally true in NoSQL systems today, where keeping related data in close proximity has a major impact on cost and performance. Instead of distributing related data items across multiple tables, you should keep related items in your NoSQL system as close together as possible.
As a general rule, maintain as few tables as possible in a DynamoDB application: Most well-designed applications require only one table, unless you have a specific reason for using multiple tables. Exceptions are cases where high-volume time series data are involved, or data sets that have very different access patterns, but these are exceptions. A single table with inverted indexes can usually enable simple queries to create and retrieve the complex hierarchical data structures that your application requires.
Use sort order: Related items can be grouped together and queried efficiently if their key design causes them to sort together. This is an important NoSQL design strategy.
Distribute queries: It is also important that a high volume of queries not be focused on one part of the database, where they can exceed I/O capacity. Instead, you should design data keys to distribute traffic evenly across partitions as much as possible, avoiding hotspots.
Use global secondary indexes: By creating specific global secondary indexes, you can enable different queries from those your main table can support, and that are still fast and relatively inexpensive.
These general principles translate into some common design patterns that you can use to model data efficiently in DynamoDB:
The primary key that uniquely identifies each item in a DynamoDB table can be simple (a partition key only) or composite (a partition key combined with a sort key).
Generally speaking, you should design your application for uniform activity across all logical partition keys in the table and its secondary indexes. You can determine the access patterns that your application requires and estimate the total Read Capacity Units (RCUs) and Write Capacity Units (WCUs) that each table and secondary index requires.
As traffic starts to flow, DynamoDB automatically supports your access patterns using the throughput you have provisioned, as long as the traffic against a given partition key does not exceed 3000 RCUs or 1000 WCUs.
DynamoDB provides some flexibility in your per-partition throughput provisioning by providing burst capacity. Whenever you are not fully using a partition’s throughput, DynamoDB reserves a portion of that unused capacity for later bursts of throughput to handle usage spikes.
DynamoDB currently retains up to 5 minutes (300 seconds) of unused read and write capacity. During an occasional burst of read or write activity, these extra capacity units can be consumed quickly—even faster than the per-second provisioned throughput capacity that you’ve defined for your table.
DynamoDB can also consume burst capacity for background maintenance and other tasks without prior notice.
It is not always possible to distribute read and write activity evenly all the time. When data access is imbalanced, a “hot” partition can receive a higher volume of read and write traffic compared to other partitions. In extreme cases, throttling can occur if a single partition receives more than 3000 RCUs or 1000 WCUs.
To better accommodate uneven access patterns, DynamoDB adaptive capacity enables your application to continue reading and writing to hot partitions without being throttled, provided that traffic does not exceed your table’s total provisioned capacity or the partition maximum capacity. Adaptive capacity works by automatically increasing throughput capacity for partitions that receive more traffic.
Adaptive capacity is enabled automatically for every DynamoDB table, so you do not need to explicitly enable or disable it.
Typically, a 5- to 30-minute interval occurs between the time that throttling of a hot partition begins and the time that adaptive capacity activates.
In a DynamoDB table, the primary key that uniquely identifies each item in the table can be composed not only of a partition key but also of a sort key.
Well-designed sort keys have two key benefits:
They gather related information together in one place where it can be queried efficiently. Careful design of the sort key lets you retrieve commonly needed groups of related items using range queries with operators such as starts-with, between, >, and <.
Composite sort keys let you define hierarchical (one-to-many) relationships in your data that you can query at any level of the hierarchy.
Amazon DynamoDB supports two types of secondary indexes:
Global secondary index: An index with a partition key and a sort key that can be different from those on the base table. A global secondary index is considered “global” because queries on the index can span all of the data in the base table, across all partitions. A global secondary index has no size limitations and has its own provisioned throughput settings for read and write activity that are separate from those of the table.
Local secondary index: An index that has the same partition key as the base table but a different sort key. A local secondary index is “local” in the sense that every partition of a local secondary index is scoped to a base table partition that has the same partition key value. As a result, the total size of indexed items for any one partition key value can’t exceed 10 GB. Also, a local secondary index shares provisioned throughput settings for read and write activity with the table it is indexing.
Each table in DynamoDB is limited to a maximum of five global secondary indexes and five local secondary indexes. For global secondary indexes, this is less restrictive than it might appear because you can satisfy multiple application access patterns with one global secondary index by overloading it.
In general, you should use global secondary indexes rather than local secondary indexes. The exception is when you need strong consistency in your query results, which a local secondary index can provide but a global secondary index cannot (global secondary index queries only support eventual consistency).
Keep the number of indexes to a minimum. Do not create secondary indexes on attributes that you do not query often. Indexes that are seldom used contribute to increased storage and I/O costs without improving application performance.
Avoid indexing tables that experience heavy write activity. In a data capture application, for example, the cost of I/O operations required to maintain an index on a table with a very high write load can be significant. If you need to index data in such a table, a more effective approach may be to copy the data to another table that has the necessary indexes and query it there.
Because secondary indexes consume storage and provisioned throughput, you should keep the size of the index as small as possible. Also, the smaller the index, the greater the performance advantage compared to querying the full table. If your queries usually return only a small subset of attributes, and the total size of those attributes is much smaller than the whole item, project only the attributes that you regularly request.
If you expect a lot of write activity on a table compared to reads, follow these best practices:
Consider projecting fewer attributes to minimize the size of items written to the index. However, this advice applies only if the size of projected attributes would otherwise be larger than a single write capacity unit (1 KB). For example, if the size of an index entry is only 200 bytes, DynamoDB rounds this up to 1 KB. In other words, as long as the index items are small, you can project more attributes at no extra cost.
Avoid projecting attributes that you know will rarely be needed in queries. Every time you update an attribute that is projected in an index, you incur the extra cost of updating the index as well. You can still retrieve nonprojected attributes in a query at a higher provisioned throughput cost, but the query cost may be significantly lower than the cost of updating the index frequently.
Specify ALL only if you want your queries to return the entire table item sorted by a different sort key. Projecting all attributes eliminates the need for table fetches, but in most cases, it doubles your costs for storage and write activity.
To get the fastest queries with the lowest possible latency, project all the attributes that you expect those queries to return. In particular, if you query a local secondary index for attributes that are not projected, DynamoDB automatically fetches those attributes from the table, which requires reading the entire item from the table. This introduces latency and additional I/O operations that you can avoid.
When you create a local secondary index, think about how much data will be written to it and how many of those data items will have the same partition key value. If you expect that the sum of table and index items for a particular partition key value might exceed 10 GB, consider whether you should avoid creating the index.
If you cannot avoid creating the local secondary index, anticipate the item collection size limit and take action before you exceed it.
For any item in a table, DynamoDB writes a corresponding index entry only if the index sort key value is present in the item. If the sort key does not appear in every table item, the index is said to be sparse. Sparse indexes are useful for queries over a small subsection of a table.
Global secondary indexes are sparse by default. When you create a global secondary index, you specify a partition key and optionally a sort key. Only items in the parent table that contain those attributes appear in the index. By designing a global secondary index to be sparse, you can provision it with lower write throughput than that of the parent table, while still achieving excellent performance.