Add Wish Template

Context and Motivation

An API offers one or more endpoints exposing operations to retrieve structured data. Not all API clients are interested in the same Data Elements the API provides.

As an API client developer, I want to express the desired response structure and content with a mock object or a descriptive specification so that I can be selective and precise when expressing my wishes about the response data that I am interested in.

Stakeholder Concerns

#client-information-needs

As motivated in Add Wish List, APIs are often used by multiple clients with differing information needs, which makes it difficult for providers to offer one-size-fits-all solutions. Some clients may require overviews of information elements, for instance lists of identifiers; others may require a subset or all of the data fields (attributes) of these elements.

#maintainability, #flexibility

Client information needs change over time, but an API and its implementation should not have to be updated every time a client does. Maintaining multiple variants of the same operation for different clients is possible, but increases maintenance cost and coordination effort, which in turn can limit the flexibility of the API provider.

#performance

Response time and throughput are qualities that concern both API clients serving end users and API providers. Preparing, transporting, and processing data but not using it on the client side wastes resources and ought to be avoided when sustainability and “Datensparsamkeit” are required.

Initial Position Sketch

A client might request and receive nested and or flat response data via one or more operation calls, as shown in Figure 1.

Figure 1: Add Wish Template: Initial Position Sketch. The API provider responds to a request from a client (1) with a message (2) that contains Data Elements. The client then issues follow-up requests (3 and 5) to retrieve more Data Elements (4 and 6).

Any operation with response data structures that clients want to customize is a potential target. A Wish List might have been introduced to let clients explicitly mark the response data they are interested in (to reduce response message size); however, this list is considered insufficient and/or cumbersome to use. The Wish Template pattern extends Wish List by allowing clients to define a structured, possibly nested template for the data they want. This Add Wish Template refactoring can also be applied to improve performance if a Wish List has not been used yet, as shown in Figure 1. Both patterns aim to give clients fine-grained control over API responses.

Design Smells

Overfetching

Clients may throw away large parts of the received data because the API design follows a one-size-fits-all approach and includes all data that any present or future client might be interested in. Another phenomenon is “sell what is on the truck”: implementation data is exposed just because it is there, without any client-side use case.

Structured artifact serialized and therefore mangled

Some nested data representation is serialized into a custom string format because the API response message contract is not expressive enough to accommodate a complex data structure. This results in a string containing a lot of information, but is hard to parse and causes surprises in testing and operations. This smell often materializes during development. For instance, a remote API that is implemented as an object-oriented program might work with objects that contain other objects. These nested object clusters (e.g., aggregates with entities and value object in domain-driven design) often have to be mapped to and from text notations such as JSON. This process is also called serialization and deserialization. When the entire nested object cluster is squeezed into a plain string, the reserved characters of the notations (for instance, curly braces {} and double quotes " in JSON) have to be escaped during serialization, which can be tedious.

Underfetching

Clients may have to call multiple API operations to obtain all required data because the interfaces of these operations do not offer any way to define the targeted representation elements (thus publishing parts or all of the entities of a domain model and their attributes).

Instructions

Introducing a Wish Template requires careful consideration, implementation, and enforcement:

1. Review the structure of the response message of the operation to define the scope of the wishes to be expressed; for instance, all elements must be optional. If necessary, wrap the structure in a Data Transfer Object (DTO) by applying the Introduce Data Transfer Object refactoring. If the request message already contains a Wish List that is supposed to be upgraded and reified, review its syntax and semantics.
2. Copy the schema definition of the eligible part of the response message to the response message schema to create the mock DTO template. Define which values mark inclusion and exclusion, respectively (for example, "in" and "out" for strings, 1 and 0 for integers, and lists with one or zero entries).
3. Optionally, change the data type of all atomic elements in the structure (such as strings and integers) to boolean. In this case, true marks inclusion and false marks exclusion. Apply these optional changes to the entire nested/hierarchical data structure recursively: in lists, turn all list elements into booleans; in Parameter Trees, process all child elements recursively.
4. Replace an existing flat, list-oriented Wish List (if any) with the mock DTO serving as Wish Template. If no such list already exists, add the Wish Template to the request message.
5. As for most, if not all, refactorings, apply the TELL steps that we introduced in [Stocker and Zimmermann 2023] according to the smell description in Add Wish List. Testing edge cases and combinations of input data is particularly important here. For instance, test empty wishes and a representative set of tree populations such as entire nested structure requested, only top-level information requested, and only leaves requested.

The revised API design is not necessarily backward-compatible. Making the template parameter optional helps preserve backward compatibility.

Target Solution Sketch (Evolution Outline)

The hierarchical structure of the Wish Template matches the structure of the response data. It is shown in Figure 2. A variant that boolean to indicate the desired response values is sketched in a UML class diagram in Figure 3.

Figure 2: Add Wish Template: Target Solution Sketch. In its request (1), the client sends a Wish Template. The provider uses a template processor to evaluate the wish Metadata Elements to decide whether some Data Element should be included in the response message (2).

Figure 3: Variant of Wish Template using booleans to indicate desired values. The template mirrors the output DTO on two nesting levels.

Example(s)

To demonstrate the Wish Template in an API contract, we will start with an API operation that uses a simple Wish List. A Wish List for a customer response payload might look as the "desiredElements":MD<string>* in the request messages of getCustomerMasterData in the following example. Note that all response elements are optional, which is indicated by the question marks ? (notation: MDSL):

operation getCustomerMasterData 
 with responsibility RETRIEVAL_OPERATION 
  expecting payload {
    "queryParameters", 
    <<Wish_List>> "desiredElements":MD<string>*
  }  
  delivering payload {
    "name":D<string>?, 
    "address": {
        "street":D<string>?, 
        "zip":D<int>?, 
        "city":D<string>}?
  }

Listing fields or attributes of provider-side resources (in the broadest sense of the word) works fine and is used in many APIs, both public and community- or solution-internal ones. This works ok as long as the field names are unique and the resource structure does not cause complex navigation in nested, hierarchical name spaces. A downside of this approach is that it does not catch typos and other mistakes in field/attribute names (at least not on the client side).

Type safety and fine-tuning of the wishes can be achieved in the example by upgrading the flat string list to a mockCustomer Wish Template that mirrors the customer structure appearing in the response:

operation getCustomerMasterData 
 with responsibility RETRIEVAL_OPERATION 
  expecting payload {
    "queryParameters",  
      <<Wish_Template>> "mockCustomer": {
        "mockName":D<string>?, 
        "mockAddress":{"mockStreet":D<string>}?, 
        "mockZip":D<int>?, 
        "mockCity":D<string>}?}  
  delivering payload {
    "name":D<string>?, 
    "address": {
        "street":D<string>?, 
        "zip":D<int>?, 
        "city":D<string>}?
  }

If such copy of the response structure is used, marker values for the basic types have to be defined in the API contract (for instance, -1 for int values and "" empty strings and lists). Alternatively, all basic types can be made optional or changed to boolean:

<<Wish_Template>> "inclusionMarkers": {
    "includeName":D<bool>?, 
    "includeAddress":{
        "includeStreet":D<bool>?, 
        "includeCity":D<bool>
    }?
} 

The API Patterns website provides another example of a Wish Template.

A large integration interface to core banking systems that applied the pattern is described in [Zimmermann et al. 2004].

Hints and Pitfalls to Avoid

The design of a Wish Template requires more effort than that of a plain Wish List:

• Explicitly define a schema for the template and the corresponding response message elements. Use the same level of detail as for the elements in the response messages themselves, e.g. NULL values, optionality, etc.
• Have the template design be reviewed by different client developers before implementing it. Following a variation of the 80-20 rule, also known as the Pareto Principle, 80% of the required features (here: Wish Template design elements) often can be elicited from, and tested by, 20% of the client developers.
• Focus on cardinalities and n to m relations in nested/linked data when designing, implementing, and testing the template. Complex requests with deep nesting can put a strain on your infrastructure. For instance, is it possible to include entire subtrees when specifying inclusion of nodes that are neither roots nor leaves? In a study on the “Semantics and Complexity of GraphQL” conducted by Hartig and Pérez [2018], the performance of the GitHub GraphQL API was examined, and an “exponential increase in result sizes” was observed as the query level depth was increased.
• Avoid the god endpoint API design smell: having one endpoint that does everything. See the Move Operation refactoring for an explanation of this smell.
• Be pragmatic and driven by client/user needs in the detailed design. Resist the temptation to optimize the wish declaration syntax just because it is technically challenging and fun to experiment with; focus on the business and domain logic using it instead. In other words, do not become a middleware provider/vendor but use existing languages and engines such as GraphQL for advanced scenarios.
• Do not forget to update security policies so that clients cannot suddenly access more data than they should.

Keep an eye on server side performance. The Wish Template can lead to many combinations of wishes, which can result in a combinatorial explosion of the data that has to be processed and sent to the client. This can lead to performance problems on the server side, especially if the template is not designed carefully.

Developer experience (DX) can suffer if the Wish Template is too complex or not well documented. Clients may struggle to understand how to use it effectively, leading to frustration and potential misuse. Further discussion of the benefits and liabilities of the Wish Template can be found in [Zimmermann et al. 2022].

Wish Lists and Wish Templates may be applied to several operations called in sequence. When improving API performance and rightsizing one or more messages, also consider applying API design patterns such as Embedded Entity and Linked Information Holder.

Related Content

An application of Add Wish List may provide the starting point for this refactoring. A Wish Template can also be introduced if no Wish List existed before; the steps dealing with the existing list are simply skipped in this case.

An alternative to introducing a Wish Template is to simply offer API clients two or more distinct operations. This simpler solution can be sufficient if the number of clients is small and their information needs are well understood.

GraphQL engines can be seen as a tool- and middleware-supported application of this refactoring, providing a single endpoint for flexible queries and mutations. The MDSL Tools contain an implementation of this refactoring.

References

Hartig, Olaf, and Jorge Pérez. 2018. “Semantics and Complexity of GraphQL.” In Proc. World Wide Web Conference (WWW), 1155–64. https://doi.org/10.1145/3178876.3186014.

Stocker, Mirko, and Olaf Zimmermann. 2023. “API Refactorings to Patterns: Catalog, Template and Tools for Remote Interface Evolution.” In Proceedings of the 28th European Conference on Pattern Languages of Programs. EuroPLop ’23. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3628034.3628073.

Zimmermann, Olaf, Sven Milinski, Michael Craes, and Frank Oellermann. 2004. “Second Generation Web Services-Oriented Architecture in Production in the Finance Industry.” In Companion to the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications, 283–89. OOPSLA ’04. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/1028664.1028772.

Zimmermann, Olaf, Mirko Stocker, Daniel Lübke, Uwe Zdun, and Cesare Pautasso. 2022. Patterns for API Design: Simplifying Integration with Loosely Coupled Message Exchanges. Addison-Wesley Signature Series (Vernon). Addison-Wesley Professional.