Add Wish List

Updated:

 Note: This is a preview release and subject to change. Feedback welcome! Contact Information and Background (PDF)

also known as: Introduce Payload Feature Toggle, Support Response Expansion

Context and Motivation

An API offers various endpoints with operations to retrieve structured data. Not all API clients are interested in the same data elements, and both clients and providers want to reduce unnecessarily transmitted and processed data.

As an API client, I want exact control over response message content so that I only receive the data I require to realize an application feature.

One solution would be to offer custom operations for different clients, but this requires a lot of knowledge about the clients on the provider side, which might not be easy to obtain.

Stakeholder Concerns (including Quality Attributes and Design Forces)

#client-information-needs
APIs are often used by multiple clients with differing information needs, which makes it difficult for their providers to offer one-size-fits-all solutions.
#evolvability, #flexibility
The information needs of clients change over time, and an API and its provider-side implementation should not have to be changed every time something changes in its clients. Maintaining several variations of the same operation for different clients is possible but increases the maintenance effort and coordination needs, which in turn might constrain the flexibility of the API provider.
#performance
Response time, throughput, and processing time are qualities that concern both API clients and their providers. Unused data that is prepared, transported, and processed wastes resources and ought to be avoided.

Initial Position Sketch

The initial response comprises several, possibly nested, Data Elements (represented by icons) or attributes.

The target of this refactoring is a single operation with its request and response messages.

Smells / Drivers

Underfetching
Clients have to call multiple API operations to get all data they require because these operations do not offer any way to define the targeted representation elements (publishing parts or all of a domain model’s entities and their attributes).
Overfetching
Clients throw away large parts of the received data because the API design follows a “one size fits all” approach and the provider includes all data in responses that any present or future client possibly might be interested in. Another phenomenon is “sell what is on the truck”: implementation data is exposed just because it is there, without any client-side use case.
API does not get to the POINT
According to the POINT principles for API design, any operation should have a purpose. It should also be T-shaped (both broad and deep, that is). Underfetching and overfechting indicate that these principles are violated or only met partially.

Instructions (Steps)

Adding a Wish List to request messages enables clients to decide what data elements they would like the response to contain. This reduces the mismatch between client expectations and provider capabilities:

  1. Analyze the current response message structure with respect to nesting, optionality, and data usage profiles. All its elements to be included in the Wish List must be optional.
  2. Apply Introduce Data Transfer Object or adjust existing ones as needed.
  3. Add a string set-valued request parameter. Decide for a list separator that is easy to transport over the given networking protocol (for instance, slashes "/" might not be a good choice in HTTP APIs, but commas "," work well).
  4. As for all API refactorings, test and document the changes and learn about its effectiveness.
  5. Upgrade the version number (indicating a backwards-compatible feature enhancement) and inform the clients when the new version is released.

Several approaches to evolution exist. To preserve the original behavior for clients that do not specify a Wish List and ensure backwards compatibility, the API should make the new parameter optional and keep the current response message structure and content. This does not break the clients, but they miss the opportunity to learn about the Wish List and will continue to request the full response structure. A different, incompatible approach is to only return a bare minimum by default so that clients are forced to state their wishes.

Target Solution Sketch (Evolution Outline)

The Wish List in the operation signature lets clients specify certain data elements they are interested in. The provider then tailors the response to the client wishes.

Example(s)

The following example from our Lakeside Mutual sample application shows a response of the Policy-Management-Backend microservice (note that no customer master data is included):

To retrieve the customer data as well, a second request to the respective endpoint could be performed. However, the API description specifies that the GET operation may take an optional expand parameter:

'/customers/{customerIdDto}/policies':
  get:
    tags:
      - customer-information-holder
    summary: Get a customer's policies.
    operationId: getPoliciesUsingGET
    produces:
      - '*/*'
    parameters:
      - name: customerIdDto
        in: path
        description: the customer's unique id
        required: true
        type: string
      - name: expand
        in: query
        description: a comma-separated list of the fields that should be expanded in the response
        required: false
        type: string
    ...

Adding expand=customer to the query string results in the following response:

Hints and Pitfalls to Avoid

The advice given in the Inline Information Holder refactoring applies here as well: if some of the data that can be referred to in the wishes resides in another (micro-)service, the API implementation is now depending on that service, which could not be permitted or desired according to the overall architecture that has been decided for.

Keep in mind that security measures might have to be adjusted because customers may now be able to query more data than before.

If providing a single, general-purpose operation for different clients does not feel right, there is nothing wrong with having specialized operations, endpoints, or even entire APIs for different clients. See the Backend for Frontend pattern for a deeper discussion of the pros and cons of this approach.

The Known Uses section of the Wish Lists pattern explains its different variants. For example, Atlassian JIRA calls the concept resource expansion.

GraphQL as an implementation of the Wish Template pattern is a more sophisticated alternative, if several endpoints of the API require this flexibility. Also see the Add Wish Template refactoring.

Too many optional parameters can lead to difficult to use APIs, impeding #learnability and resulting in a complex implementation logic.

The Inline Information Holder offers a less flexible but simpler solution. In the example above, we saw that the full customer entity was included in the response. If only parts of that data are used, the Wish Template pattern provides a mock object-based approach to further tailor the response to the client wishes. A Wish List can be evolved and graduated into a Wish Template, also a MAP quality pattern, with a Add Wish Template refactoring.

Step 2 of this refactoring applies Introduce Data Transfer Object. Undoing it is possible and straightforward; our refactoring catalog does not contain a dedicated inverse refactoring.

Wish Lists can also be used in request messages: “Practical API Design at Netflix, Part 2: Protobuf FieldMask for Mutation Operations” in the Netflix technology blog mentions this refactoring in the context of gRPC but then moves on to present a more flexible solution based on Field Masks, the gRPC and Protocol Buffers pendant to our Wish List pattern.

The (still emerging) MDSL tool prototypes contain an implementation of this refactoring.