Add Wish Template

also known as: Add Expansion Mock Object, Reify Wish List, Turn Wish List into Wish Template

Context and Motivation

A Wish List might have been introduced to let clients explicitly mark the response data they are interested in (to reduce response message size); it is considered insufficient and/or cumbersome to use.

As an API client developer, I want to express the desired response structure as a mock object so that I can be precise and selective when expressing my wishes about structured response data that I am interested in.

This refactoring can also be applied to improve performance if a Wish List has not been used yet.

Stakeholder Concerns

#client-information-needs

As motivated in Add Wish List, APIs are often used by multiple clients with differing information needs, which makes it difficult for providers to offer one-size-fits-all solutions. Some clients may require overviews of information elements, for instance list of identifiers, others may require a few or all of the data fields/attributes of these elements that the API is able to provide.

#performance

Response time and throughput are qualities that concern both API clients serving end users and API providers (also explained in Add Wish List). Preparing, transporting, and processing data but not using it on the client side wastes resources and ought to be avoided.

#maintainability, #flexibility

Client information needs change over time, but an API and its implementation should not have to be updated every time a client does. Maintaining multiple variants of the same operation for different clients is possible, but increases maintenance cost and coordination effort, which in turn can limit the flexibility of the API provider.

Initial Position Sketch

A client might request and receive nested and or flat response data via one or more operation calls:

Add Wish Template: Initial Position Sketch

Any operation with response data structures that clients want to customize is a potential target.

Design Smells

Underfetching

Clients may have to call multiple API operations to obtain all required data because the request messages of these operations do not offer any way to define the targeted representation elements (thus publishing parts or all of the entities of a domain model and their attributes).

Overfetching

Clients may throw away large parts of the received data because the API design follows a one-size-fits-all approach and includes all data that any present or future client might be interested in (according to the smell description in Add Wish List). Another phenomenon is “sell what is on the truck”: implementation data is exposed just because it is there, without a client-side use case.

Structured artifact serialized and therefore strangled

Some data representation that is nested is serialized into a custom string format that is hard to parse and keeps on causing surprises in testing and production. In object-oriented programming, for instance, objects that contain other objects often have to be mapped to and from text notations such as JSON. The reserved characters of the notations (for instance, curly braces {}and double quotes " in JSON) have to be escaped during serialization, which can be tedious.

Instructions

Introducing a Wish Template requires careful consideration and enforcement:

1. Review the structure of the response message of the operation to define the scope of the wishes to be expressed; for instance, all elements must be optional. If necessary, wrap the structure in a Data Transfer Object (DTO) by applying the Introduce Data Transfer Object refactoring. If the request message already contains a Wish List that is supposed to be upgraded and reified, review its syntax and semantics.
2. Copy the schema definition of the eligible part of the response message to the response message schema to create the mock DTO template. Define which values mark inclusion and exclusion, respectively (for example, "in" and "out" for strings and 1 and 0 for integers and lists with one or zero entries).
3. Optionally, change the data type of all atomic elements in the structure (such as strings and integers) to boolean. In this case, true marks inclusion and false marks exclusion. Apply these optional changes to the entire nested/hierarchical data structure recursively: in lists, turn all list elements into booleans; in Parameter Trees, process all child elements recursively.
4. Replace an existing flat, list-oriented Wish List (if any) with the mock DTO serving as Wish Template. If no such list already exists, add/append the template to the request message.
5. As for most, if not all, refactorings, apply the TELL steps – test, extend API Description, let clients know, leverage logging – that are generally explained at the bottom of this page). Here, testing edge cases and combinations of input data is particularly important. For instance, test empty wishes and a representative set of tree populations such as entire nested structure requested, only top-level information requested, and only leaves requested.

The revised API design is not necessarily backward-compatible. Making the template parameter optional helps preserve backward compatibility.

Target Solution Sketch (Evolution Outline)

The hierarchical structure of the wish matches the structure of the response data. It is shown in the following figure:

Add Wish Template: Target Solution Sketch

In the online description of the Wish Template pattern, the variant that changes the response data types to boolean is sketched in a UML class diagram:

The template mirrors the output DTO on two nesting levels.

Example(s)

A Wish List for a customer response payload might look as the "desiredElements":MD<string>* in the request messages of getCustomerMasterData in the following example. Note that all response elements are optional, which is indicated by the question marks ? (notation: MDSL):

operation getCustomerMasterData with responsibility RETRIEVAL_OPERATION 
  expecting payload {"queryParameters", <<Wish_List>> "desiredElements":MD<string>*}  
  delivering payload {"name":D<string>?, 
      "address": {"street":D<string>?, "zip":D<int>?, "city":D<string>}?}

Listing fields or attributes of provider-side resources (in the broadest sense of the word) works fine (and is used in many APIs, both public and community- or solution-internal ones) — as long as their names are unique, and the resource structure does not cause complex navigation in nested, hierarchical name spaces. A downside of this approach is that it does not catch typos and other mistakes in field/attribute names (at least not on the client side).

Type safety and fine-tuning of the wishes can be achieved by upgrading the flat string list to a "mockCustomer" Wish Template that mirrors the customer structure appearing in the response:

operation getCustomerMasterData with responsibility RETRIEVAL_OPERATION 
  expecting payload {
    "queryParameters",  
      <<Wish_Template>> "mockCustomer": {
        "mockName":D<string>?, 
        "mockAddress":{"mockStreet":D<string>}?, 
        "mockZip":D<int>?, 
        "mockCity":D<string>}?}  
  delivering payload {"name":D<string>?, 
    "address": {"street":D<string>?, "zip":D<int>?, "city":D<string>?}?}

If such copy of the response structure is used, marker values for the basic types have to be defined in the API contract (for instance, -1 for int values and "" empty strings and lists). Alternatively, all basic types can be made optional or changed to boolean type:

      <<Wish_Template>> "inclusionMarkers": {
        "includeName":D<bool>?, 
        "includeAddress":{"includeStreet":D<bool>?}, "includeCity":D<bool>}?} 

See MAP website for another example of a Wish Template.

Hints and Pitfalls to Avoid

The design of a Wish Template requires more effort than that of a plain list:

• Have the template design be reviewed by different client developers before implementing it; apply the 80-20 rule, also known as the Pareto Principle, along the way.
• Focus on cardinalities, optionality, and n:m relations in nested/linked data when designing, implementing, and testing the template. For instance, is it possible to include entire subtrees when specifying inclusion of nodes that are neither roots nor leaves?
• Resist the temptation to focus on the wish declaration syntax more than on the business and domain logic using it. In other words, do not become a middleware provider/vendor but use existing languages and engines such as GraphQL for advanced scenarios.

Wish Lists and Wish Templates ay be applied to several operations called in sequence. When improving API performance and rightsizing one or more messages, also consider to apply Microservice API Patterns such as Embedded Entity and Linked Information Holder.

Also remember to update security policies so that clients cannot suddenly access more data than they should.

Related Content

An application of Add Wish List may provide the starting point for this refactoring. A Wish Template can also be introduced if no Wish List existed before; the steps dealing with the existing list are simply skipped in this case. A large integration interface to core banking systems that applied the pattern is described in Zimmermann et al. [2004].

GraphQL engines can be seen as a tool- and middleware-supported application of this refactoring, prpviding a single endpoint for flexible queries and mutations. The MDSL Tools contain an implementation of this refactoring.

References

Zimmermann, Olaf, Sven Milinski, Michael Craes, and Frank Oellermann. 2004. “Second Generation Web Services-Oriented Architecture in Production in the Finance Industry.” In Companion to the 19th Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages, and Applications, 283–89. OOPSLA ’04. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/1028664.1028772.