DynamoDB Enhanced Client: Support Querying and Marshalling Heterogeneous Item Collections

[helloworldless]

Item collections are a core concept in DynamoDB, especially for "well-designed" applications using a single table design. So ideally, this concept should have first class support in the SDK. However, there doesn't seem to be a way to query a heterogeneous item collection with the Enhanced Client. I did see a feature request (#1870) for supporting polymorphic types, but this doesn't seem to have item collections in mind.

Here's an example of a heterogeneous item collection, a customer and their associated orders:

PK	SK	Type
CUSTOMER#Customer1	A	Customer
CUSTOMER#Customer1	ORDER#Order1	Order
CUSTOMER#Customer1	ORDER#Order2	Order

To retrieve the item collection we use a Query with KeyConditionExpression: "PK = CUSTOMER#Customer1". Now suppose our application has a value class (ideally immutable) corresponding to each of these item types: Customer.java and Order.java. But with the Enhanced Client, there doesn't seem to be a way to marshall heterogeneous query results into their respective value classes.

With the AWS SDK for Java v1 and DynamoDBMapper, I've been doing the following:

1. Use AmazonDynamoDB#query to query an item collection, e.g. KeyConditionExpression: "PK = CUSTOMER#Customer1"
2. QueryResult#getItems returns a List<Map<String, AttributeValue>>
3. For each item, check the Type attribute and use DynamoDBMapper#marshallIntoObject to marshall the item's Map<String, AttributeValue> into the appropriate value class, e.g. dynamoDBMapper.marshallIntoObject(Customer.class, currentItemKeyAttributeValueMap)

...but I don't think we can do something similar when using the Enhanced Client since the DynamoDBMapper wouldn't work with the AWS SDK v2 annotations and the Enhanced Client doesn't expose such a marshallIntoObject method. I think working with immutable value classes would make it more challenging to come up with a similar workaround for the Enhanced Client.

I did experiment with the Flat map attributes from another class features in the Enhanced Client, but this doesn't help with the issue of marshalling heterogeneous results into their respective value classes.

[helloworldless]

I've been thinking about this a bit more. Below are some workarounds I came up with. I think it's useful to document them because they all have significant drawbacks.

1. Use Lower Level API and Manually Construct the Entities
Fall back on the lower level DynamoDbClient#query and manually construct the entities. This gets quite messy when there are converted attributes (AttributeConverter). This manual mapping is exactly what DynamoDBMapper#marshallIntoObject helps us avoid. Another downside to this approach is that we must maintain each entity's attributes in two place, for example, in the Order.java and in this item collection mapping code.

QueryResponse queryResponse = dynamoDbClient.query(queryRequest);
List<Map<String, AttributeValue>> items = queryResponse.items();
// For each item, manually construct the appropriate immutable value class based on the Type attribute
// Omitting some code here... We know this item is an Order because it has an attribute Type with the value "Order"
Order.builder()
  .orderId(item.get("OrderId").s())
  .orderDate(LocalDateAttributeConverter.create().transformTo(item.get("OrderDate")))
  .build();

2. Make Multiple Round Trips
Make N Get/Query requests to build up the item collection with N items. This is perhaps the cleanest workaround, but it requires multiple round trips which I think goes against the principles of a well-designed application. This would also consume extra RCUs, and all the requests to fetch a single item collection would be going to the same partition which I think could be problematic.

3. Use a Super Entity and Mappings
Have a super entity like CustomerOrderSuperEntity.java which is a superset of all the attributes in the item collection, in this case all the attributes in a Customer and in an Order. We can use the Enhanced Client to query the item collection and get back a List<CustomerOrderSuperEntity> and then use write mappers to map the super entity to the Order and Customer entities. Again, we have to maintain each entity in two places, for example, in Order.java and in CustomerOrderSuperEntity.java. One other downside is that there may be conflicts between attributes with the same name on different entities if the attributes are of different types.

[debora-ito]

Hi @helloworldless thank you for reaching out, marking as a feature request. It is similar to the polymorphic types request, except within a collection of items instead of an item.

[helloworldless]

I think by far the simplest solution here would be to expose the Enhanced Client's marshalling mechanism just like DynamoDBMapper provides marshallIntoObject. In this way, the same three steps I outlined in my original post can be used except with the SDK v2's DynamoDbClient and the Enhanced Client annotations.

[helloworldless]

After digging into the Enhanced Client code, I discovered this, software.amazon.awssdk.enhanced.dynamodb.TableSchema#mapToItem, which is exactly what I was after!

Used together with software.amazon.awssdk.services.dynamodb.DynamoDbClient#query, I'm able to query an item collection and then marshall each item into its respective entity.

I've described how to do this in detail here.

I'm fine with closing this!

[debora-ito]

Glad you are not blocked anymore. Closing this, feel free to reach out if you have any more questions.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.