Compoxure Composition Middleware

Composition proxy replacement for ESI or SSI uses htmlparser2 to parse HTML from backend services and compose fragments from microservices into the response. This is exposed as Express middleware to allow quick creation of a proxy server.

If you are looking for a almost 'ready to use' server that has the middleware configured with some examples, please look here: https://github.com/tes/compoxure-example

For rationale (e.g. why the heck would anyone build this), please see the rationale section at the bottom.

Running Example App

git clone git@github.com:tes/compoxure.git
cd compoxure
npm run start

Visit https://localhost:5000/

What is it

Compoxure is a composition proxy - you put it in front of a back end service that acts as a template into which content from microservices is composed. It is designed to be simple, fast and failure tolerant - e.g. you won't need to build complex patterns across and around all of your micro services to deal with failure scenarios, they just serve out HTML and should try to fail fast.

Examples of usage

• Branding: Imagine that you need to share common branding (masthead, navigation, footer) across a range of applications. You can put this in front of the application and it can then include these elements in the page from a common set of microservices responsible for each.

• Expose an app within a CMS: Instead of building your apps in your CMS (arrgh) instead include them in CMS managed pages with a simple declaration. The apps won't need to know about the site branding, the CMS takes care of all of that. Also makes it easy to have those apps expose small amounts of functionality across your site (e.g. small Top 10 lists etc.).

• Complex application decomposition: Image you have to deliver a page that has some product data, as well as a range of content that depends on the user. Compoxure will let you serve these fragments of the page at different TTLs, and if you add middleware that parses your user cookie it can actually pass back user information to these services so they dont have to - making those easier to test and reason about.

How it works

You have a back end service (e.g. a CMS) that returns HTML containing the declarative markup explained below. Compoxure then responds to requests, calling the backend service that matches the URL, parses the HTML it returns and makes any requests to other micro services whose responses are then inserted into the HTML on the way through.

The various responses (e.g. the backend HTML page, each service fragment response) can all be cached at differing TTLs (or not at all) and using a simple key construction method that covers the scenario that some fragments may differ for not logged in users vs logged in users.

Typically the backend service will be a CMS or a static HTML page containing specific markup, with the thing that is included being the output of more complex applications.

<div>
     This content will be replaced on the way through
</div>

Configuration

The full configuration options can be found in /examples/example.json, but it is expected that you will store this configuration within your own application and pass it through into the middleware.

The configuration object looks as follows:

{
    "backend": [{
        "pattern": ".*",
        "target":"https://www.tes.co.uk",
        "host":"www.tes.co.uk",
        "ttl":"10s",
        "quietFailure":false,
        "leaveContentOnFail":false,
        "dontPassUrl":false,
        "passThrough":false,
        "cacheKey":"{{url:path}}",
        "contentTypes":["html"]
    }],
    "parameters": {
        "urls": [
            {"pattern": "/teaching-resource/.*-(\\d+)", "names": ["resourceId"]}
        ],
        "urlGroups": [
            {"pattern": "/teaching-resource*", "group": "resources"}
            {"pattern": ".*", "group": "home"}
        ],
        "servers": {
            "local": "https://localhost:5001"
        }
    },
    "cdn": {
        "url": "https://server.cloudfront.net",
    },
    "environment":{
        "name":"development"
    },
    "hogan":{
        "delimiters":"{{ }}"
    },
    "cache": {
        "defaultNoCacheHeaders": {
          "cache-control": "private, no-cache, max-age=0, must-revalidate, no-store"
        },
        "engine": "redis"
    },
    "followRedirect": false,
    "fragmentDepth": 5
}

Backend

These properties configure the backend server that the initial request goes to grab the HTML that is then processed.

Property	Description
pattern	The regexp used to select this backend based on the incoming request to compoxure. First match wins.
fn	Name of a selector function of type: function(req, variables) { return true; } to allow more dynamic selection of backend (instead of pattern). Function must be passed in on the config.functions object, and is referenced by name.
target	The base URL to the backend service that will serve HTML. URLs are passed directly through, so /blah will be passed through to the backend as https://backend/blah
host	The name to be passed through in the request (given the hostname of compoxure is likely to be different to that of the backend server. The host is used as the key for all the statds stats.
ttl	The amount of time to cache the backend response (TODO : make it honor cache headers)
timeout	Time to wait for backend to respond - should set low
quietFailure	Used to determine if compoxure will serve some error text in response to a microservice failure or fail silently (e.g. serve nothing into the space).
dontPassUrl	Used to decide if the URL in the request is passed through to the backend. Set to true if the backend should ignore the front URL and just serve the same page for all requests (e.g. a fixed template)
contentTypes	An array of content types which are accepted by this backend. Defaults to ['html']. See the accepts documentation regarding how headers are parsed. *Note: The order is important! We recommend that you always put html as the first item in the array; an exception to this is if you need to allow requests for Javascript files through unparsed - for example if you need to make a request for a service worker script to the same domain as your page and can't request it from a CDN host; in which case use ['application/javascript', 'html'] in combination with passThrough: true.
passThrough	Defaults to false, but if set to true it will proxy all non HTML assets through completely un-parsed. It's best to serve static assets like Javascript and CSS from a CDN so the requests don't go through Compoxure, but there could be exceptions to this, eg service workers that have to be requested from the same domain as the app (for this case, also see contentTypes config option above).
headers	An array of header names to specify headers forwarded to the backend server.
cacheKey	Cache key to use for requests to this backend
noCache	Do not cache this backend. Deprecated - caching is now off by default
tags	a space separated list of tags. They are used to mark a cached item, so they should be easy to bulk delete
passThroughHeaders	Pass this array of headers to the client if the backend sets them. E.g., "passThroughHeaders": ["x-robot-tag","cache-control"]
addResponseHeaders	Set the response headers in this map always. E.g., "addResponseHeaders": { "x-robots-tag": "noindex" }. You can use template variables in these headers.

You can define multiple backends, by adding as many declarations for backends as you like, with differing patterns. The first match wins e.g.

 "backend": [{
        "fn": "selectResource",
        "target":"https://www.tes.co.uk",
        "host":"www.tes.co.uk",
        "ttl":"10s",
        "cacheKey":"{{param:resourceId}}",
        "replaceOuter":false,
        "quietFailure":true
    },
    {
        "pattern": ".*",
        "target":"https://localhost:5001",
        "host":"localhost",
        "ttl":"10s",
        "replaceOuter": false,
        "quietFailure": true
    }]

In this example, the selectResource function is passed in with the configuration when configuring the middleware:

config.functions = {
  'selectResource': function(req, variables) {
    if(variables['param:resourceId']) return true;
  }
}

Parameters

The parameters section provides configuration that allows compoxure to use data from the initial request (and config) to pass through to microservices (e.g. url parameters, values from the url path via regex, static server names to avoid duplication).

Property	Description
urls	This is an array of objects containing pattern / names values. Pattern is a regex used to parse the request hitting page composer, names are the names to assign to the matches from the regex. All of the patterns are executed against the incoming URL, and any matches added to the parameters that can then be used in the microservice requests. The full list of these are found below.
query	This is an array of objects containing key / mapTo values. Key is the query parameter name, mapTo is the name to assign the value. The values are added to the parameters that can then be used in the microservice requests. The full list of these are found below.
servers	A list of server name to server URL configurations that can be used to avoid repetition of server names in your fragments

Allowing backends to render their own error pages

You can now add an array parameter to config:

{
  renderContentOnErrorCode: [403]
}

This configuration means that compoxure will take the content rendered (along with the error), and render that through to the end user, this is in place of a bland compoxure error string indicating that the backend served a 403.

This response is parsed.

Cache Engine

Compoxure allows caching of both the back end response and page fragments. This is currently done using Redis, but other cache engines could be put in place.

Note that the cache implementation in Redis is not done using TTLs, as this means that once Redis expires a key if the backend service is down that compoxure will serve a broken page. It is instead done slightly differently to allow for the situation that serving stale content is better than serving no content (this is one of the safe failure modes).

To disable caching simply delete the entire config section.

Property	Description
defaultNoCacheHeaders	Headers to automatically append to any response that is not cached (optional)
engine	The engine to use: 'nocache', 'memorycache', 'redis'.
url	If Redis, set a url for the redis server - e.g. localhost:6379?db=0
host	If Redis, set the host explicitly (this and params below are an alternative to using url)
port	If Redis, set the port explicitly
db	If Redis, set the db explicitly
options	If redis, any option described at https://github.com/NodeRedis/node_redis#options-object-properties (e.g. to allow authentication)
compress	Use snappy compression for caching
namespace	Optional prefix for redis keys

Conditional GET request & Etag

HTTP conditional requests are requests that are executed differently, depending on the value of specific headers. In the htmlparser.js, we implement the entity tag, or the etag, as the validator to determine if the resource/content stored on the server matches a specific version. We calculate the etag hash if we can cache the page (meaning, there are no fragments with no-cache specified), and return statusCode 304 (Not modified) when the ETags match to that of the request header "If-None-Match", 200 otherwise. When we return statusCode 304 it indicates that there is no need for the server to send a resource/content because the request indicates that the client, which made the request conditional, already has a valid content; the server is therefore redirecting the client to make use of that stored content as if it were the payload of a 200 OK response.

CDN Configuration

Compoxure can pass through CDN configuration through to each of the backend services, this is to allow them to render any links required to images or other static assets.

The configuration here is passed through to the services via two headers:

• x-cdn-host : maps to cdn host property (deprecated)
• x-cdn-url : full url to use as based for CDN

Property	Description
host	The hostname of the CDN - e.g. cdn.tes.co.uk
url	Full url - e.g. https://cdn.tes.co.uk/assets/ - required
resolver	Resolver function (see below)

If you provider a resolver function, it needs to be of the form:

config.cdn.resolver = function(service) {
  if (service = 'one') {
    return 'https://base.cdn.path/';
  }
}

This allows you to lookup the CDN to use for a specific bundle based on the name of the service (however you choose to). You must still pass the default url, as this is the fallback. If you use the option "useRelativeURL: true" the CDNURL will be translated (if possible) in an url relative to the domain.

Status Code Handlers

You can specify a function to be executed if any of your backend services return with a specific HTTP Status Code. This allows you to do things like redirect a user request to a login page if a request returns 401 or 403.

Property	Description
statusCode:fn	The name of the function to be executed when this statusCode is encountered. This function has to be available as a property on the config.functions and will receive the arguments req, res, variables, data
statusCode:data	An object that will be passed to the function when called. An example of using this function would be to add the redirect link when a 401 of 403 code is encountered.

 "statusCodeHandlers": {
   "403": {
     "fn":"handleUnauthorised",
       "data": {
         "redirect":"https://account.tes.co.uk/LogOn?rtn=<%= url%>"
       }
   },{
    "401": {
      "fn":"handleUnauthorised",
       "data": {
         "redirect":"https://account.tes.co.uk/LogOn?rtn=<%= url%>"
       }
    }
 }

The function has a signature of:

(req, res, variables, data, options, error) => {}

Follow Redirect

There is one final parameter, which is 'followRedirect'. This applies to the request library used to retrieve fragments. The default for this value is true, which means that if this configuration is not supplied as false, that if a fragment issues a 301 or 302 then Compoxure will follow it and retrieve the content from that new location and cache it.

If you set this property to false, this means that you can write a status code handler (see above) to capture a 301 or 302 from a fragment, and have the entire page follow to the location specified.

An example handler below (from the Compoxure tests):

 'handle302': function(req, res, variables, data, options, err, responseCallback) {
      res.writeHead(302, {location: err.headers.location});
      res.end('');
  }

You can use the responseCallback function to respond with HTML that would be rendered within any errored fragment:

 'handle600': function(req, res, variables, data, options, err, responseCallback) {
      responseCallback('html to render on any status code of this type');
  }

Environment

Environment can be set in config, so it can be used as a parameter.

Property	Description
name	Environment name, defaults to NODE_ENV or 'development' if NODE_ENV not supplied.

Hogan

Compoxure uses Hogan to parse string templates. If you are also using a mustache language as your templating language, and are embedding Compoxure directives in templates, this will conflict.

The best option here is to use a different delimiter for Compoxure, and this can be configured via the hogan configuration option:

Property	Description
delimiters	New delimiter string - e.g. '<% %>'

Layout engine and Slots

You can use this feature to push some sections of a template in a layout.

To get this feature to work, you need a few things setup:

1. An endpoint (service) that serves a HTML page that forms the layout, that contains 'define slots'.
2. An application that serves the correct headers, and a fragment of HTML that with 'use slots'

The headers your application needs to send are:

{
  'cx-layout': 'https://www.example.com/layout.html',
  'cx-parse-me': 'true'
}

The layout should contain some slot definition:

...
<h1>I am the layout</h1>
<div>
  <p>Default content of the slot 1</p>
</div>
...
<div>
  <p>Default content of the slot 2</p>
</div>
...

The application then should respond with some HTML that describes what to put in each slot:

...
<div>
  <p>This is the content of slot 1</p>
</div>
...

If a slot is not provided the default content will be used.

So this will be output:

...
<h1>I am the layout</h1>
<div>
  <p>This is the content of slot 1</p>
</div>
...
<div>
  <p>Default content of the slot 2</p>
</div>
...

If there is more than one cx-use-slot with the same name, the content of all of them will be concatenated.

This can be useful if fragments from different services want to add the content in the same place.

Note: By default only content assigned to a slot from the initial content request will be considered for loading into a slot.

If you want to have nested fragments load content into the slots defined as part of the layout then you will need to have an additional header returned by your first content response. In addition to the cx-layout and cx-parse-me headers you will need to assign the cx-allow-slot-use header. For example:

{
  'cx-layout': 'https://www.example.com/layout.html',
  'cx-parse-me': 'true',
  'cx-allow-slot-use': 'true',
}

This configuration causes an additional parse of the initial content response to take place so has been added as an opt in option to keep the performance stable for exising applications.

Nested Fragments

Compoxure will parse fragments within fragments up to fragmentDepth deep. If not specified with default to 5. For this to remain performant, a service responding with a fragment containing compoxure directives will need also send a response header of cx-parse-me: true.

i.e.

Component1

<div>
  <p>Widget 1</p>
</div>

Component2

<div>
  <p>Widget Two</p>
  <div></div>
</div>

Root

  <div></div>

Result of rendering Root:

  <div>
    <div>
      <p>Widget Two</p>
      <div>
        <div>
          <p>Widget 1</p>
        </div>
      </div>
    </div>
  </div>

Declarative Parameters

To use compoxure in a declarative fashion, simply add the following tags to any HTML element. The replacement is within the element, so the element containing the declarations will remain. The element can be anything (div, span), the only requirement is that the cx-url attribute exists.

Warning: As composure uses a mustache syntax for variable substition, when using compoxure params within a mustache/handlebars template you must escape the page composer params e.g. \{{server:resource-list}} or change the delimiter in the hogan configuration.

Property	Description
cx-url	The url to call to get the content to put into the section matching the selector (specific to replacement).
cx-cache-key	The key to use to cache the response, if none provided compoxure will not cache.
cx-cache-ttl	The time to cache the response.
cx-statsd-key	The key to use to report stats to statsd, defaults to cache-key if not set.
cx-timeout	The timeout to wait for the service to respond.
cx-no-cache	Explicit instruction to not cache when value value eval's to true, overrides all other cache instruction (deprecated - compoxure no longer caches by default)
cx-replace-outer	Whether to completely replace the outer HTML element. Overrides configuration in backend.
cx-test	Use to test a string, it will parse the string and output that (e.g. change cx-url to cx-test to test)
cx-ignore-404	If this call returns a 404 then dont pass it up and 404 the entire page, defaults to TRUE.
cx-ignore-error	Set to true to ignore all errors (non 200), or provide a comma delimited list to ignore specific errors on this fragment

<div>
     This content will be replaced on the way through
</div>

Example of dynamic cx-no-cache:

<div>
     This content won't be cached if the user is the author of the widget
</div>

Using parameters and substitution in strings

Page composer allows you to use mustache like templates for a number of strings, specifically URL and Cache Key fields tend to allow the use of variables. The possible variables are:

Prefix	Description	Example
param	Parameters matched from the parameters configuration (regex + name) pairs in the configuration	/resource/{{param:resourceId}}
query	Parameters matched from any query string key values in the incoming URL	/user/{{query:userId}}
url	Any elements of the incoming url (search, query, pathname, path, href, group)	/search{{url:search}}
cookie	Any cookie value	/user/{{cookie:TSL_UserID}}
header	Any incoming header value	/user/feature/{{header:x-feature-enabled}}
server	A server short name from the configuration in the parameters section of config	{{server:feature}}/feature
env	Environment, property available is name e.g. {{env:name}}
cdn	CDN configuration, properties available are host and url e.g. {{cdn:url}}
user	User properties, if set on the request as property user - e.g. req.user = {name: bob} >> {{user:name}}. Also a 'loggedIn' boolean property is set on user eg {{user:loggedIn}}
device	Device type {{device:type}} = desktop	phone
layout	This is the layout url (available only after getting the response, useful for cacheKey)	{{layout}}

Note: It also passes the device:type down to downstreams in a header - as 'x-device'.

Note that you can add an additional :encoded key to any parameter to get the value url encoded (e.g. {{url:search:encoded}}). An example set is shown below. Note that they will vary depending on your request, parameter configuration, cookies etc.

{ 'param:resourceId': '123456',
  'url:protocol': 'https:',
  'url:slashes': true,
  'url:auth': null,
  'url:host': 'localhost:5000',
  'url:port': '5000',
  'url:hostname': 'localhost',
  'url:hash': null,
  'url:search': '?param=true',
  'url:query': 'param=true',
  'url:pathname': '/resource/123456',
  'url:path': '/resource/123456?param=true',
  'url:href': 'https://localhost:5000/resource/123456?param=true',
  'url:group': 'resources',
  'cookie:example': '12345',
  'header:host': 'localhost:5000',
  'header:connection': 'keep-alive',
  'header:accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
  'header:user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36',
  'header:referer': 'https://localhost:5000/',
  'header:accept-encoding': 'gzip,deflate,sdch',
  'header:accept-language': 'en-GB,en-US;q=0.8,en;q=0.6',
  'header:cookie': 'example=12345',
  'server:local': 'https://localhost:5001',
  'env:name': 'development',
  'user:userId': '_',
  'device:type': 'phone'
  }

Multiple urls in cx-url and strategy

You can optionally have multiple cx-url attributes with different suffixes. All the contents will be retrieved and will be added to the html. You can modify the default behaviour using the cx-strategy attribute. These are the currently supported strategies:

• default: all the contents
• first-non-empty: this is returning the first non empty content
• random: this is returning a random content When you have multiple urls you can also have multiple versions of these attributes having the same suffix: cx-cache-key, cx-timeout, cx-ttl, cx-no-cache. For example:

<div></div>

From Cache Only

Note that the reason that we designed compoxure to allow full control over cache keys was to support the use case of pre-warming a cache with content at given keys - e.g. so you could have a backend job that populated the cache with content so you never actually got a cache miss and called the backend service itself.

e.g. the HTML below would fail silently and quickly in the instance the cache didn't have content, but expects the cache to be loaded:

<div>
     This will contain the latest news that is expected to be pre-populated by a job in the cache.
</div>

To do: build an API for the cache that enables jobs to do this without directly talking to Redis.

Static Bundles

Using Compoxure in conjunction with Bosco gives you a way to manage the static assets (JS, CSS, Images) that go with the services.

This will resolve the src or href properties to:

{{cdn:url}}/{{environment:name}}/vendor/library/{{library}}

e.g.

https://duqxiy1o2cbw6.cloudfront.net/tes/vendor/library/bootstrap-3.2.0.min.css

A library is imported only once, any duplicated will be ignored.

Client hints

You can apply the cx-client-hint=true attribute to a cx-library, cx-bundle or cx-src tag. This transforms the tag in a client hint:

By default rel="preload" is used, unless you specify a rel attribute with a different value.

Images

In addition to bundles, you can use a directive to get access to images from a service that also respect the service version and cdn url.

<img>

This would resolve to: