Tapjaw Importer - v2.0.0

Create a stream of payload messages to STDOUT from any API.

• Tapjaw Importer - v2.0.0
• Description
• Purpose
• Documentation
• Why use Tapjaw Importer?
• Dependancies
• Setup a new project
• Components
  • Commands TapjawCommand
  • Adapters TapjawContract.TapjawAdapter
  • Authenticators TapjawAuthenticator
  • Configs TapjawConfig
  • Connectors TapjawConnector
  • Contracts TapjawContract
  • TapjawMessage TapjawMessage
  • Iterators TapjawIterator
  • Dates TapjawDate
• Examples & Tutorial
• Changelog
  • v2.0.0
  • v1.2.0 (no longer supported)
  • Previous versions

Description

Tapjaw Importer is a Typescript JSON-based utility framework purposefully designed to simplify the development of commands that get chained together to consolidate data for persistence or further processing in the cloud or on existing system infrastructure.

The chained commands communicate with JSON over an STDIO stream using a pre-defined class called a TapjawMessage.TapjawMessage, containing some default metadata and payload properties, allowing for easy extending from the TapjawMessage.TapjawMessage Class based on the business rules of implementation.

The TapjawMessage.TapjawMessage schema enables the creation of a standardised streaming paradigm with JSON delimited by newlines, using the Unix pipe syntax, e.g.:-

$ bin/run apis my-network | bin/run filters remove-duplicates | bin/run stores s3 s3://my-bucket

In practice, one API command gets used in conjunction with multiple filter commands, finishing with a store command to persist the result in an external location.

To stream a single output into several store commands, create a Tools command to boot up multiple child processes and fan out to the multiple store commands. Please refer to the example below for a pseudo-example:

$ bin/run apis my-network | bin/run filters remove-duplicates | bin/run filters clean | bin/run tools fan "stores s3 s3://my-bucket" "stores sqs sqs://sqs-server/my-queue"

The project provides several additional useful features to decrease the complexity of:

• Access HTTP/API endpoints with Connectors (TapjawConnector.TapjawDefaultConnector).
  • Security implementations for Bearer, Basic, JWT and OAuth authentication.
• Iterate seamlessly over multiple pages of response data using Adapters (TapjawContract.TapjawAdapter).
• Parser tools to convert XML or CSV to JSON (TapjawParser.xmlToJson & TapjawParser.csvToJson).
• Strong focus on Typescript data typing, employing contracted interfaces throughout the framework.
• Rate limiting of messages getting emitted through a stream from any API command.
• Each command type is has a base class designed for its purpose, requiring a minimal amount of code to achieve a workable command.
• Date helper functions, providing a common toolkit that we have found useful in previous projects (using Luxon).
• A quick and simple .env configuration loading strategy with TapjawConfig.DotEnvConfig, featuring the ability to namespace groups of configurations.
• Base TapjawContract.TapjawLogger interface to allow easy implementation of your logging library. The project includes two basic implementations, ConsoleLogger and NullLogger, which NullLogger is generally used in testing.

Purpose

Tapjaw Importer started life in 2019 due to a requirement of needing to import data from across dozens of APIs into a consistent internal schema, each API has unique data schemas, various security infrastructures, request/response strategies and general differences in approach.

Tapjaw Importer's inspiration comes from the singer.io approach, which provides many API taps and targets. However, as an Affiliate Marketing company, we could not rely on singer.io to have all the APIs we needed. Instead, we created this framework that provides a more diverse approach to extracting data and doing something useful with it.

Tapjaw Importer provides a more abstract approach than singer.io by employing customisable API, filter, store and tool commands.

As a developer, system administrator or operations manager, it is possible to chain together as many commands as needed to fulfil a business models requirements.

As the developer, little concern is necessary with how data traverses through the commands since Tapjaw Importer supplies abstract command types to complete the task quickly and easily.

digidip GmbH has successfully written and deployed four separate internal projects into production using this framework, allowing us to experience such a process's complexities. The project attempted to provide the necessary components to set up a command and data stream quickly.

Documentation

The overview documentation in the docs/markdown/ directory, the latest API documentation is also available.

Why use Tapjaw Importer?

1. Require node.js for setting up ingestion pipelines.
2. Require a framework to quickly create API commands to stream each record to a STDOUT buffer.
3. Be able to mix and match commands for business purposes demands quickly.
4. Slots in easily into existing system infrastructure.
5. The ability to create a custom toolkit of bespoke scalable commands.
6. Easy to implement data consolidation for cloud-based serverless projects.
7. An simple interface for creating streams of data between commands.

Dependancies

• Node JS v16
• Commander.js
• reflect-metadata
• Typescript 4
• Unix based operating system (it might work in Windows, but compatibility can't be assured)

Setup a new project

Firstly, execute cd to your workspace or development directory.

~/workspace $> git clone "https://github.com/digidip/tapjaw-example.git#semver^2" my-project

> Caution: Please make sure the host system is using Node version 16. A package such as nvm may help manage your node versions during development.

Install all the frameworks dependencies. If the host system is not configured with at least node version 16, yarn will throw an error.

~/workspace/my-project $> yarn install

Build the project with:

~/workspace/my-project $> yarn tsc

To validate the project, issue the following command:

~/workspace/my-project $> bin/run

After the command completes, the following out should be displayed:

Usage: run [options] [command]

Options:
  -V, --version     output the version number
  -h, --help        display help for command

Commands:
  apis         Execute an API command
  filters