Add Wish List

also known as: Introduce Payload Feature Toggle, Support Response Shaping and Expansion

Context and Motivation

An API offers one or more endpoints exposing operations to retrieve structured data. Not all API clients are interested in the same Data Elements that can be retrieved, and both clients and providers want to reduce unnecessarily transmitted and processed data.

As an API client, I want exact control over response message content so that I receive just the data I require to realize an application feature.

Ideally, a single call should be enough to retrieve all required data. So one solution would be to offer custom operations for different clients, but this requires much knowledge about the clients on the provider side, which might not be easy to obtain and change dynamically. It would also increase maintenance effort and coordination needs.

Stakeholder Concerns

#client-information-needs

APIs are often used by multiple clients with different information needs, which makes it difficult for their providers to offer one-size-fits-all solutions.

#evolvability, #flexibility

The information needs of clients and users vary over time, and an API and its provider-side implementation should not have to be adjusted every time something changes in its clients. Maintaining several variations of the same operation for different clients is possible but increases the maintenance effort and coordination needs, which in turn might constrain the flexibility of the API provider.

#green-software, #performance, #data-parsimony

Overall response time, throughput, client-side and server-side processing time are qualities that concern API clients and their providers. Unused data that is prepared, transported, and processed wastes resources and ought to be avoided.

Initial Position Sketch

The initial response shown in Figure 1 comprises several, possibly nested, Data Elements, also known as attributes.

Figure 1: Add Wish List: Initial Position Sketch. The API provider responds to a request from a client (1) with a message (2) that contains Data Elements. The client does not require all the received data.

The targets of this refactoring are API operations with their request and response messages. The refactoring applies in different situations: the provider might return data the client is not interested in, as shown in Figure 1, where the client wants the ability to filter elements to avoid overfetching. The refactoring is also applicable when the provider omits data the client must retrieve through a second request from a different API operation. Clients can use the Wish List to expand the message content to avoid underfetching in that case.

Design Smells

Underfetching

Clients have to call multiple API operations to get all data they require because these operations do not offer any way to define the targeted Data Elements.

Overfetching

Clients throw away large parts of the received data because the API design follows a one-size-fits-all approach, and the provider includes all data in responses that any present or future client might be interested in. For example, in an e-commerce API, product procurement information might only interest a few clients, while most want to learn about current prices and items in stock. Another phenomenon is “sell what is on the truck”: implementation data is exposed just because it is there, without any client-side use case.

API does not get to the POINT

According to the POINT principles for API design, any operation should have a purpose. It should also be T-shaped (both broad and deep, that is). Underfetching and overfetching indicate that these two principles are violated or only partially met.

Instructions

Adding a Wish List to request messages lets clients select what Data Elements they want the response to contain. Applying this pattern reduces the mismatch between client expectations and provider capabilities:

1. Analyze the current response message structure with respect to nesting, optionality, and data usage profiles. All its elements to become selectable must be optional; otherwise, not wishing for them cannot have any effect.
2. Apply Introduce Data Transfer Object (DTO) or adjust existing DTOs as needed.
3. Add a set-valued request parameter that allows clients to enumerate the desired Data Elements. The name of the parameter could, for example, be called wishList, select, or expand. Decide on a list separator that is easy to transport over the given networking protocol (for instance, slashes “/” might not be a good choice in HTTP APIs, but commas “,” work well).
4. Populate this parameter when preparing requests on the client side.
5. Process this parameter when responding to requests on the provider side. Evaluate which Data Elements to return and prepare the response message accordingly. Avoid retrieving data from the data store only to discard it afterward. For example, customize SQL queries to only fetch the required attributes and avoid unnecessary joins.

Complete the refactoring with the following steps that apply to most/all IRC entries:

• Test the changes to ensure that the new design works and that the end-to-end capabilities of the API remain unchanged. Use this opportunity to learn about its effectiveness; establish success criteria derived from the stakeholder concerns. For example, performance might be an important metric that can be tracked by a benchmark.
• Enhance the external API Description and document the rationale for the new design.
• Upgrade the version number (indicating a backward-compatible feature enhancement in this case) and inform the clients when the new version has been released.
• Analyze whether the refactoring application has been successful according to the success criteria.

Several approaches to evolution exist. To preserve the original behavior for clients that omit the Wish List and ensure backward compatibility, the API should make the new parameter optional and keep the current response message structure and content. This approach does not break the clients, but they might miss the opportunity to learn about the Wish List and will continue to request the complete response structure. A different, possibly incompatible approach is only to return a minimal response message by default so that clients are forced to state their wishes.

Target Solution Sketch (Evolution Outline)

The Wish List in the operation signature lets clients specify certain Data Elements they are interested in. The provider then tailors the response to the client’s wishes. This solution is sketched in Figure 2. Chapter 7 of [Zimmermann et al. 2022] shows a solution sketch with a dedicated List Evaluator component to handle the client’s wish.

Figure 2: Add Wish List: Target Solution Sketch. In its request (1), the client also sends a Wish List. The provider uses this wish to decide whether some Data Element should be included in the response message (2). The provider implementation might also delegate this filtering to the repository component to avoid retrieving unneeded data from the data store.

Example(s)

The following example from our Lakeside Mutual sample application shows a response of the Policy Management backend microservice before refactoring. Note that no customer master data is included, only a customerId:

curl http://localhost/policies/fvo5pkqerr
{
  "policyId" : "fvo5pkqerr",
  "customerId" : "rgpp0wkpec",
  "creationDate" : "2021-07-07T13:40:52.201+00:00",
  "policyPeriod" : {
    "startDate" : "2018-02-04T23:00:00.000+00:00",
    "endDate" : "2018-02-09T23:00:00.000+00:00"
  },
  ...
}

A second request to the respective endpoint could then be performed to retrieve the customer data. However, the API description specifies that the GET operation may take an optional expand parameter:

'/policies/{customerIdDto}':
  get:
    tags:
      - customer-information-holder
    summary: Get a customer's policies.
    operationId: getPoliciesUsingGET
    produces:
      - '*/*'
    parameters:
      - name: customerIdDto
        in: path
        description: the customer's unique id
        required: true
        type: string
      - name: expand
        in: query
        description: a comma-separated list of the fields  
          that should be expanded in the response
        required: false
        type: string
    ...

Adding expand=customer to the query string results in the following response, which now includes customer master data:

curl http://localhost/policies/fvo5pkqerr?expand=customer
{
  "policyId" : "fvo5pkqerr",
  "customer" : {
    "customerId" : "rgpp0wkpec",
    "firstname" : "Max",
    "lastname" : "Mustermann",
    "birthday" : "1989-12-31T23:00:00.000+00:00",
    "streetAddress" : "Oberseestrasse 10",
    "postalCode" : "8640",
    "city" : "Rapperswil",
    "email" : "admin@example.com",
    "phoneNumber" : "055 222 4111",
    "moveHistory" : [ ],
    ...
  },
  "creationDate" : "2021-07-07T13:40:52.201+00:00",
  "policyPeriod" : {
    "startDate" : "2018-02-04T23:00:00.000+00:00",
    "endDate" : "2018-02-09T23:00:00.000+00:00"
  },
  ...
}

The client does not have to issue a second request for the data when using the Wish List.

Hints and Pitfalls to Avoid

If some of the data that can be referred to in the wishes resides in another API endpoint, the API implementation is now depending on that other endpoint, which could not be permitted or desired according to the overall architecture that has been decided for.

Remember that security measures might have to be adjusted because customers might retrieve more data in a single call than before the refactoring. Measures could include checking for authorization and other quality management concerns, like accounting for Rate Limits.

If providing a single, general-purpose operation for different clients does not feel right, there is nothing wrong with having specialized operations, endpoints, or even entire APIs for clients. See the Backends for Frontend pattern for a deeper discussion of the pros and cons of this approach.

Too many optional parameters can lead to difficult-to-use APIs, impeding learnability and resulting in a complex implementation logic.

Related Content

Step 2 of this refactoring applies Introduce Data Transfer Object. The MDSL Tools contain an implementation of this refactoring.

Split Operation could be used to undo this refactoring: if the Wish List is small (for example, there might only be a single data element that can be selected), it might be better to offer two separate operations instead of one with an optional Wish List parameter.

In the example above, we saw that the entire customer entity was included in the response. If only parts of that data are used, the Wish Template pattern provides a mock object-based approach to further tailor the response to the client’s wishes. A Wish List can be evolved and graduated into such a Wish Template with the Add Wish Template refactoring.

The Known Uses¹ section of the Wish List pattern explains variants and implementation options. For example, Atlassian JIRA has a concept of resource expansion.

A Wish List that is introduced with this refactoring can also provide flexibility regarding the content of request messages. For example, when partially updating server-side data, instead of offering many distinct operations that each update a specific field, a single operation can be provided. The request parameters of a Wish List are not used to tailor the response but instruct the endpoint on what data to update. “Practical API Design at Netflix, Part 2: Protobuf FieldMask for Mutation Operations” in the Netflix technology blog shows this in the context of gRPC using Field Masks, the gRPC and Protocol Buffers pendant to our Wish List pattern.

References

Zimmermann, Olaf, Mirko Stocker, Daniel Lübke, Uwe Zdun, and Cesare Pautasso. 2022. Patterns for API Design: Simplifying Integration with Loosely Coupled Message Exchanges. Addison-Wesley Signature Series (Vernon). Addison-Wesley Professional.

1. https://www.api-patterns.org/patterns/quality/dataTransferParsimony/WishList#sec:WishList:KnownUses ↩