Kamau Washington — Posts

Off By Cents

Kamau Washington — Sun, 06 Sep 2026 00:00:00 GMT

I want to introduce BOE, my new rule engine platform at boerules.com. But the announcement is the easy part. The more honest post is the one that explains why someone who walked away from rule engines over a decade ago spent last year building thirty five of them.

So here is the long version. It starts fifteen years ago, goes quiet for most of the time in between, and then turns on a single bug I found last year.

MeshAlpha

Almost fifteen years ago I wrote MeshAlpha, a Forward Chaining RETE-OO engine in C#, tested and debugged by hand, before any of us had an AI assistant looking over our shoulders. Mesh for the discrimination network and a nod to the Von Neumann machine architecture I was studying at the time, Alpha for the first node type every RETE tutorial hands you. I read Forgy’s original papers and Robert Doorenbos’s 1995 CMU thesis on RETE/UL, kept a running study of Drools and the lesser known engines beside it. All of it traced back to FICO, where I started my career working with rule engines in my twenties. The part that ends most rule engine side projects, mine included, is the business rule management system wrapped around it: an authoring UI, version control for rules business users can actually touch, testing, simulation and deployment, all built for people who are not engineers. ILOG, Pega, Blaze Advisor, Sparkling Logic and above all Drools already had that, and I could not compete with any of it as one person on a side project. So I put MeshAlpha down and stayed on my architecture track. That is where things sat for more than a decade.

Last year, and then Off By Cents

That changed last year, in 2025, when AI got genuinely good at writing code and the gap between having an idea and having a working implementation of it stopped being the obstacle. I was building an integrations scaffold on an AWS native platform, and part of it was a cost skill: tell me what a given integration costs to run at different volumes. Ahead of a demo it ran clean five times in a row, memory cleared between each run, numbers exact every time. On the sixth run the calculation came back Off By Cents, and took longer than it should have to get there. Nobody else in the room noticed. An answer that drifts by cents while deciding what a system costs to run is a real problem, and I had spent enough of my career on rule engines to know exactly where that kind of drift comes from. It sent me straight back to FICO, back to Forgy, back to everything I had read and built and set aside years earlier. So I built the Pricing Engine.

Why I built the Pricing Engine

I did not build the Pricing Engine for the company I was working for. I built it for myself, to prove something I had believed for a long time and never had a clean chance to prove: less is more, simple is fast, and fast can still be reliable and deterministic.

Once the Pricing Engine worked the way I wanted, I kept going. Forward Chaining next, then Backward Chaining, Bayesian, Sequential, Constraint Satisfaction, Utility and Multi Criteria Decision Making. Twenty engines by January 1st, 2026. Today there are more than thirty five, and the list keeps growing.

The gap I actually found

Somewhere in that stretch of building I noticed the actual shape of the market, not the shape I had assumed it had back when I put MeshAlpha down. On one end sit the large commercial players. On the other sits a long tail of open source rule engines, each with its own conventions, little to no support, and rarely anything you could call a real BRMS. Most of them hand you a config file, a YAML document, or a JSON schema and call that authoring.

This time the BRMS came first. The part that ended MeshAlpha is the part BOE opens with, and it is built for the agentic era rather than retrofitted to it.

Authoring in the BOE Console. The change is described in plain language, staged against a draft, and nothing goes live until the tests pass and someone publishes.

I also noticed something about scale that the industry rarely says out loud. Most production rulesets sit well under 2,500 rules. Most of them are closer to 250 or 500. Past that, a ruleset stops being something a team can reason about, and honestly even 500 rules is pushing it for most teams. Fact volume tells a similar story. In my entire career I have never seen an engine handed millions of facts in a single pass. What I have seen, over and over, are millions of small fact sets, 10 or 20 objects each, triggering an engine individually or in async batches.

That gave me a number to build toward. If I could run 2,500 rules against 2,500 facts and get an answer back in 250 milliseconds, I would have something worth shipping.

Micro Engines

The last realization shaped BOE more than any of the others. Most rule engines on the market, commercial or open source, ship one engine type, or maybe as many as five, and then bend every problem to fit whichever one you licensed. A Scoring problem gets modeled as a Forward Chaining problem because Forward Chaining is what the vendor sells. An Eligibility check runs through the same general purpose evaluator as a pricing calculation, because that evaluator is the only tool in the box.

I built the opposite of that. Every engine in BOE is its own thing: its own shape, its own runtime, its own input and output schema, its own algorithm tuned for exactly the decision it makes. A Scoring engine does scoring. An Eligibility engine does eligibility. Nothing gets reused just because it happens to already be sitting there.

You see it the moment you open the console. The overview is the way into the BRMS, and every ruleset on it names the engine it runs on.

The overview, where the console opens. The ENGINE column is the whole argument: every ruleset runs on the engine its decision calls for, not on one shared evaluator.

I never needed to compete with ILOG or Pega or Drools on their own terms. Their architecture solves a much bigger problem than most teams actually have, and that scope is exactly what makes them heavy to adopt and slow to change. My problem was smaller and clearer than that from the start: thirty five specific Micro Engines, each one built the way it should have been built if nobody had to compromise for the sake of a shared platform underneath it.

That is BOE! You can look through all of it at boerules.com.

SOUNDEX Algorithm in Python3+

Kamau Washington — Thu, 25 May 2023 00:00:00 GMT

This post is purely illustrative and educational. It showcases a working version of the Soundex algorithm in Python3+. This code is not to be used in production as is and should be modified or rewritten to meet the requirements and standards of a production-level implementation.

If you would rather skip to the code, or this article is TLDR, no worries! See the code here.

The Soundex Algorithm is a phonetic algorithm that converts an alphanumeric string into a 4 character code by how it is pronounced in English. The 4 character code is comprised of one letter followed by three digits. The first time I came across this algorithm was with a task I was given to perform address and name matching for a skiptracing worflow using Stored Procedures in SQL Server 2000 (eeeeek! 😬 ). It made absolutely no sense to me at the time how it was able to determine how “Geoff” and “Jeff” SOUND alike to a computer. As I made a list of algorithms to familiarize myself with and implement this year, SOUNDEX made it on the list, and here I am. This algorithm can be implemented in any language, I decided to use Python3+ to become more fluent in it.

So how do we get V532 from “VanDeusen”?

The algorithm seems to be straight-forward. However, it has a few gotchas that can be observed in many available open-source implementations which will be addressed below. Let’s list the rules of the algorithm (they can be implemented in varying orders) :

From SOUNDEX Wikipedia

Remove all non-alpha characters
If two of the same characters are adjacent in the original alphanumeric string, only retain the first character.
If two of the same characters are separated by h, w, y only retain the first character.
If two of the same characters are separated by a, e, i, o, u retain both characters
Retain the first character of the alphanumeric string and drop all other occurrences of a, e, i, o, u, y, h, w, y
Replace consonants with digits as follows (after the first letter):
- b, f, p, v = 1
- c, g, j, k, q, s, x, z = 2
- d, t = 3
- l = 4
- m, n = 5
- r = 6
If there are too few characters in the alphanumeric string to assign three numbers, append zeros until there are three numbers. If there are four or more numbers, retain only the first three

Vowels and the Highway (hwy)

One of the first steps is to Keep it Short and Simple (KISS Principle). We don’t need a clever algorithm to handle vowels and h, w, y we simply need a code. Uppercase characters will be used to make a lexicon given the rules above.

A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z
0	1	2	3	0	1	2	9	0	2	2	4	5	5	0	1	2	6	2	3	0	1	9	2	9	2

0’s are vowels, 9’s are h,w,y

Implementation Step #1 – Create the lexicon

Moving from A to Z, use the mappings provided by the algorithm to create a string that can later be iterated character by character. 0’s represent vowels, and 9’s represent h, w, y.

python

# codings is a representation of the alphabet according to the SOUNDEX algorithm
# 0 represents vowels : 'A','E','I','O','U', where 9 represents 'H','W','Y'.
# Both 0 and 9 are skipped charcters.
codings : str = "01230129022455012623019292"

Implementation Step #2 – Lexicon translation

This step is the “coding” step, where the input is transformed into its coded output as per the lexicon. This implementation breaks apart the transformation step from the code generation step for readability and separation of concerns.

python

# use an uppercase representation of the input for easier validation and iterate over each character
for char in list(input.upper()) :

    # get the ascii character code for the current character
    charCode : int = ord(char)

    # VALIDATON : if the char is not [A-Z] skip it
    if charCode < 65 and charCode > 90 :
        continue

    # SOUNDEX : Step 1a. Retain the first letter of the input
    if len(finalCode) < 1 :
        finalCode.append(char)

    # SOUNDEX : Step 2. Replace consonants with digits (after the first letter):
    code : int = codings[charCode - 65]
    inputCoded.append(code)

Get the ASCII representation of the current uppercase character of the iteration
Given A=65 and Z=90 as ASCII character codes, determine if the character is inclusively between 65 and 90, if not the character skips any further processing (the equivalent of deleting the character in the output of the routine)
Extract the first character of the input to use as the prefix of the final code
Use the lexicon to obtain the character value per the specification. When using ASCII it is easy to subtract A from 65 to get “0” which can be used as an index for the lexicon. Append this character to the representation of the input as coded numbers

The above steps will translate a name into a string of numbers to be parsed in the next step so VanDeusen will be translated into 105300205

V	A	N	D	E	U	S	E	N
1	0	5	3	0	0	2	0	5

Lexicon translation of VanDeusen

Implementation Step #3 – SOUNDEX Code Creation

This step creates the 4 character code from the lexicon translation. While this implementation can be merged with the previous step, this iteration is kept apart from the prior iteration for readability and separation of concerns.

python

# iterate over the coded input, the index is needed to compare like characters with a 0 or 9 between them
for index, char in enumerate(inputCoded) :

    # prevent further iteration once the finalCode has reached the max length according to the SOUNDEX algorithm
    if len(finalCode) == 4 :
        break

    # determine if the current character is a skippable (or to be removed) character as per the algorithm
    skipchar : bool = char in ['0','9']

    # SOUNDEX : Step 3b. Remove all the zero (0 or 9) digits (performed by skipping)
    if skipchar :
        continue

    # store the previous character if available
    if index > 0 :
        previousChar = inputCoded[index-1]

    # the first character should be skipped for processing
    else :
        continue

    # SOUNDEX : Step 3a. Replace all adjacent same digits with one digit (performed by skipping)
    if char == previousChar :
        continue

    # SOUNDEX : Step 3.
    #
    # If two or more letters with the same number are adjacent in the original word (before step 1),
    # only retain the first letter; also two letters with the same number separated by 'h' , 'w' or 'y'
    # are coded as a single number, whereas such letters separated by a vowel are coded twice.
    # This rule also applies to the first letter.
    if index > 2 and inputCoded[index-2] == char :
        if previousChar == '0' :
            finalCode.append(char)
            continue

        # characters equaling 9 are skipped if the previous and next value are the same
        elif previousChar == '9' :
            continue

    finalCode.append(char)

If the length of the finalCode is 4 we can stop iterating as that is the maximum number of characters needed for code completion
If the current character is 0 or 9 it can be skipped or not added to the final code
If there is a previous character available it should be stored for future iterations
If this character is the first character it can be skipped as it was already added in Implementation Step #2
Deduplicate adjacent identical characters
If two identical characters are separated by a vowel ( 0 ) append the character to the final code
If two identical characters are separated by h, w, y then skip this character (it is not added to the final code)
Finally, if this iteration hasn’t skipped, append the character to the final output

V	N	D	S
V	5	3	2

Lexicon translation of VanDeusen

The above steps will convert 105300205 into the final output of V532. Remember that the first character “V” was extracted in Implementation Step #2

Thats it!

This was an extremely interesting algorithm that proved to be challenging at times. This implementation may differ from SQL implementations as eluded to by the Wiki article, but this is the traditional approach that meets the conditions specified :

Using this algorithm, both “Robert” and “Rupert” return the same string “R163” while “Rubin” yields “R150”. “Ashcraft” and “Ashcroft” both yield “A261”. “Tymczak” yields “T522” not “T520” (the chars ‘z’ and ‘k’ in the name are coded as 2 twice since a vowel lies in between them). “Pfister” yields “P236” not “P123” (the first two letters have the same number and are coded once as ‘P’), and “Honeyman” yields “H555”.

Take me to the Code!!!

NATS.io Queue Group REST API w/ Sanic and Python3+

Kamau Washington — Mon, 22 May 2023 00:00:00 GMT

This post is purely illustrative and educational. It showcases a working version of a Queue Group + Request-Reply pattern implementation in Python3+ with Nats.io and Sanic. This code is not to be used in production as is, and should be modified and or rewritten to meet the requirements and standards of a production-level implementation.

NATS is a connective technology much like Kafka or Kinesis though far much simpler to use and adopt. The Publish-Subscribe pattern is easy to understand, if one has worked with any message queue, authored/used a file watcher, and has not been exposed to “streaming” technologies; NATS is Open Source, easy to adopt, especially with its built-in asynchronous patterns. However, the goal is not to explain NATS ad infinitum, and add to the pool of NATS explanations across the web or to be preferential.

The goal is to showcase one of the many built-in patterns in NATS and accrue hands-on time with the technology to better understand it.

What is a Queue Group in 3 sentences or less?

A Queue Group is essentially a pool of subscribers load-balanced on a NATS subject. Adding two of the same subscriptions to a subject will result in both executing. If these same subscriptions are part of a Queue Group, each message is received by one and only one random subscriber in the subscriber pool.

What is a valid use case for Queue Groups?

One of the designs I have favored is having message validation IO completely decoupled from computationally intensive APIs. This is to prevent invalid messages from entering and/or leaving a system, and to offload validation (both light and heavy) to a separate tier that responds immediately vs bundling validation with valuable compute.

This design can be seen in API Gateway, where a JSON Schema can be defined and executed prior to invoking a Lambda or transmitting to another AWS Service.

For this implementation, the goal is much simpler, to gauge the degree of difficulty to implement Queue Groups, and observe them in action.

Let’s get started!

There are three parts needed to prove this pattern and its usefulness :

A running NATS Server. This is extremely easy to install and configure NATS with docker.
An API Framework. Python3+ is used for this implementation along with the Sanic Framework for the API as it is quick to get up and running.
A Subscriber that will listen to a NATS subject and respond using NATS Request-Reply

With these 3 components, implementing an API with Request-Reply to a NATS Subject with Subscribers is extremely straightforward.

Create the Subscriber

Subscriber implementation can be seen HERE if TLDR

The subscriber should be able to identify itself to the requestor, distinguishing itself from other subscribers in the Queue Group on reply. If created correctly, multiple requests to a single NATS subject with multiple queued subscribers should result in a randomly selected subscriber reply on each request.

Step #1 Identify the Subscriber

A UUID could be used, but what would be the fun in that? Boring! Random human-readable “slugs” or names seem much easier to align which subscriber responded without having to manually scan a few letters of a UUID. For this, a Python library called ‘coolname‘ was used.

python

from coolname import generate_slug
# generate a coolname for this subscriber
subscriptionName : str = generate_slug(2)

Using 2 as a parameter to generate_slug(..), generates names like ‘macho-baboon’, or ‘coy-turtle’

Step #2 Create a callback w/ an adjustable delay

While there is no shortage of algorithms that would slow responses through computationally intensive algorithms, the goal is to prove the load balancing aspect vs cpu performance as this varies from one environment to another.

To keep it simple, all that is needed is a delay, an async delay to be exact, and use the message sent to the callback to issue a reply back to the NATS reply-to token. An async delay is used as this type of event-based programming lends itself to an async environment like Python’s asyncio.

Asynchronous programming was favored as I have become used to it over the years of writing NodeJS APIs. When applicable use the best tool for the job..

python

# this method executes an async delay for the duration defined in the environment variable : DURATION
async def delayedResponse(msg : NATS.msg_class) -> None :
    global subscriptionName

    # important to use await here, this is a non blocking sleep
    await asyncio.sleep(settings.DURATION)

    messageCount : str = msg.data.decode()
    replyMessage : str = "Message %1s recieved by '%2s.'" % (messageCount,subscriptionName)
    print(replyMessage)

    # using the Request-Reply pattern msg.respond publishes a message to the reply-to (message must be in bytes)
    await msg.respond(replyMessage.encode())

Note : ‘messageCount’ can be thought of as ‘message-id’, this may be changed after this article is published.

Step #3 Create the subscriber

This was the most interesting implementation hurdle. There are plenty of extremely lengthy Python examples for creating a NATS long-running subscriber. They were ignored in lieu of a simple while loop with an async.sleep (exactly how I would have done it in NodeJS 😁 )

With any long-running process like this, it is important to unsubscribe and close the NATS connection to prevent orphaned processes. It is common to close all processes without doing the aforementioned, only to find that messages sent to a subject continue returning responses.

IMPORTANT – the most essential part of this function is setting ‘queue‘. This is how the load balancing is achieved within the Queue Group.

python

# this is the main subscriber, note that it is asynchronous as there are multiple awaits within needed by NATS
async def main () -> None :
    # identify subscriber
    global subscriptionName
    print("Starting subscriber '%1s'." % (subscriptionName))

    # provide access to the close connectionGlobal var for use in the keep alive loop
    global closeConnection

    # NOTE - this is anycio compatible NATS, not the synchronous NATS functionality
    nc = NATS()

    # connect to the server defined in the environment variable : SERVER
    await nc.connect(servers=settings.SERVER)
    print("Connection to NATS Server '%1s' established by '%2s'." % (settings.SERVER,subscriptionName))

    # create the subscription to the subject defined in the environment variable : NATS_SUBJECT
    subscription = await nc.subscribe(
        settings.NATS_SUBJECT,

        # callback to the delayed response method defined above
        cb=delayedResponse,

        # IMPORTANT -> specifying queue allows multiple instances of this process to load balance
        # the queue specified in the environment variable : NATS_QUEUE_NAME.
        queue=settings.NATS_QUEUE_NAME
    )

    # while the global variable closeConnection is False, loop every one second to keep alive
    # this can be a larger amount if needed, it is set to 1 for quick exiting only
    while not closeConnection :
        await asyncio.sleep(1)

    ###################### CLEANUP and EXIT ######################

    # unsubscribe from the above NATS subscription
    await subscription.unsubscribe()
    print("\nUnsubscribed '%2s' from '%1s.'" % (settings.NATS_SUBJECT,subscriptionName))

    # close the NATS connection, there have been instances where a process was not properly closed w/ orphan connections
    await nc.close()
    print("Closed connection to Nats Server '%1s'." % (settings.SERVER))

    # exit this process
    sys.exit()

Create the API

Sanic API implementation can be seen HERE if TLDR

The API is the most simplistic implementation as it simply is a wrapper around NATS Request-Reply mechanics. It is essentially a NATS HTTP proxy. There are before_server_start and before_server_stop for initialization and cleanup respectively, the code can be seen in the link above, as they feel immaterial for the purposes of this article.

That is all there is to it! Line 10, NATS Request-Reply in action! This makes me 🤦🏽‍♂️ (facepalm) at how simple this is. It is almost too simple.

Keep it Short and Simple (Kiss Principle) – I hope that developers take note, implementation should be this simple in 2023+ (should have been this simple in most libraries a decade or more ago).

python

# basic root endpoint http://<SERVER>:<PORT> (defaults to http://localhost:8000)
@app.get("/")
async def root_request(request : Request):
    try:

        # increment the message count (to identify messages when multiple subscribers are attached)
        app.ctx.messageCount += 1

        # use NATS Request/Reply pattern to publish to NATS_SUBJECT (it should be load balanced and sticky to the receiving subscriber)
        msg : NATS.msg_class = await app.ctx.nc.request(settings.NATS_SUBJECT, str(app.ctx.messageCount).encode(),timeout=60)

        # return the subscriber response to the requestor (message returned is in bytes must decode to utf-8)
        return text(msg.data.decode())

    except asyncio.TimeoutError:

        # there should not be timeouts unless a large duration is set
        print("Timed out waiting for response")
        return text("Request timed out sending message %1s." % (app.ctx.messageCount))

Lets run this thing!

Running this application is straightforward. Follow the steps in the README.md of the repository for prerequisites. Once the environment is ready for a run…

Step #1 Launch a Subscriber

Ensure a new terminal/console is used for each creation when creating a Subscriber. In each terminal run python3 subscriber.py Creating multiple separate Subscriber processes allows the Queue Group to be formed providing the load-balancing capabilities we set out to prove.

shell

my-username@a-computer-name nats-queue-api-python % python3 subscriber.py

python3 subscriber.py

Starting subscriber 'analytic-honeybee'.
Connection to NATS Server 'localhost' established by 'analytic-honeybee'.

Note that the make command is merely a python3 command underneath and that the subscriber name was randomly generated as described in the above Subscriber creation section.

NOTE – open multiple terminals/consoles to the root of this project and issue the ‘make subscriber’ command to scale up

Step #2 Launch the Sanic API

This is a standard Sanic API launch. Ensure that a terminal/console is open to the root of this project and run the command python3 api.py

shell

my-username@a-computer-name nats-queue-api-python % python3 api.py

sanic api:app
[2023-05-21 21:07:06 -0400] [52986] [INFO]
  ┌──────────────────────────────────────────────────────────────────┐
  │                          Sanic v23.3.0                           │
  │                Goin' Fast @ http://127.0.0.1:8000                │
  ├───────────────────────┬──────────────────────────────────────────┤
  │                       │     mode: production, single worker      │
  │     ▄███ █████ ██     │   server: sanic, HTTP/1.1                │
  │    ██                 │   python: 3.11.3                         │
  │     ▀███████ ███▄     │ platform: macOS-13.3.1-x86_64-i386-64bit │
  │                 ██    │ packages: sanic-routing==22.8.0          │
  │    ████ ████████▀     │                                          │
  │                       │                                          │
  │ Build Fast. Run Fast. │                                          │
  └───────────────────────┴──────────────────────────────────────────┘

[2023-05-21 21:07:06 -0400] [52986] [WARNING] Sanic is running in PRODUCTION mode. Consider using '--debug' or '--dev' while actively developing your application.
Connection to NATS Server 'localhost' established.
[2023-05-21 21:07:06 -0400] [52991] [INFO] Starting worker [52991]

It can be seen that ‘make api’ is running the standard Sanic app start command underneath.

Step #3 Load Test!

Using Artillery it is extremely easy to run load tests and get a report showing what effect general load or sustained load over time has on the response time.

Open a new terminal/console to the root of this project and run artillery quick —count 20 -n 20 http://<HOST>:<HOST_PORT>/ —output report.json && artillery report report.json

shell

my-username@a-computer-name nats-queue-api-python % artillery quick --count 20 -n 20 http://127.0.0.1:8000/ --output report.json && artillery report report.json

artillery quick --count 40 -n 20 http://127.0.0.1:8000/ --output report.json
Log file: report.json
artillery report report.json
Report generated: report.json.html

1 Subscriber in Queue Group ~2 seconds median

Median of ~2000ms (2 seconds) with 40 users sending 20 requests each

3 Subscribers in Queue Group ~500 milliseconds median

Median of ~500ms (1/2 second) with 40 users sending 20 requests each

As can be seen, the Queue Group pattern opens many possibilities for handling targeted scalability. This was a feature I didn’t expect would perform this well and be implemented with such ease. All in all, this will definitely be a pattern I will use with NATS, and possibly with other connective technologies if supported. Woot!

Take me to the Code!!!

Creation of ION-Query for MeshAlpha.io (Pt. 1)

Kamau Washington — Wed, 18 Jan 2023 00:00:00 GMT

This post chronicles the journey of creating an Open Source TypeScript Rule Engine from conception to release. While the code will be open source (MIT License), it is asked that this or other articles are referenced, and attributed to Kamau Washington if a strategy or code block is used outside of meshalpha.io repositories.

NOTE : MeshAlpha.io is a work in progress, updates will be found on this site pre-release

Earlier in my career I was introduced to the world of Rule Engines, and immediately found a subject that was far more intriguing and challenging than standard Full-Stack SDLC. Blaze Advisor, BizTalk BRE, ILOG JRules were areas of study, as well as the works of Charles L. Forgy the designer of the RETE algorithm. It took the better part of a year to understand RETE-OO (object oriented RETE), Conflict Resolution, Agendas, Working Memory, Alpha and Beta nodes, discrimination networks, forward chaining, backward chaining, and more. The following year, I completed my first RETE based Rule Engine in C# which I named “MeshAlpha”.

While the Engine was fast, extendable, well documented, and passed nearly every Rule Engine test I could get my hands on, there were a few main issues that stood out :

An extremely simple DSL (Domain Specific Language) to represent and author rules outside of code, treating Rules as external configuration vs compiled syntax.
An extremely simple UI that allows Rules to be authored without knowledge of the underlying engine, or with limited to exposure to Rule Engines.
An extremely simple code base, following patterns and practices that would allow authors besides myself to contribute to the codebase with minimal ramp-up.

Starting over, DSL first

There are many Rule Engines complete with their own DSL, often requiring a significant amount of onboarding for the language itself let alone the Engine ecosystem. This should be avoided at all costs this time around.

PROBLEM STATEMENT
Build a text-based DSL for Querying Facts (Objects, Classes, Models, Interfaces, Types…) using only JSON and or YAML to empower business users, and developers alike. The DSL should be easily readable, pattern based, and allow for hinting in IDEs as well as by schema.

Start small, with a reusable pattern

This DSL should be comprised of repeatable patterns that when parsed and executed alter their behavior based on their location in a query. It should also be familiar to read and write. A pattern that stuck out was S-Expressions or Symbolic Expressions : a tree structured notation of data.

text

// this is an example of an S-Expression (pseudo code)
(multiply 3 (add 2 5))

This may look very simple, and appear as if functionality would be lost, however this is an extremely simple, and readable notation to express the following :

“multiply three by the sum of two and five”

This can easily by translated into something familiar like JSON

text

// this is an example of an S-Expression in JSON
[
    "multiply"
        3,
        [
            "add",
                2,
                5
        ]
]

This then can be easily translated into executable JavaScript

typescript

function multiply(...args) {
    return args.reduce((total, value) => total * value, 1);
}
function add(...args) {
    return args.reduce((total, value) => total + value, 0);
}
console.log(multiply(3,add(2,5))) // = 21

Immediately, the possibilities can be realized!

After running permutations of functions, selectors, constants, etc. the S-Expression seems to be the most robust and simple method of implementing a DSL for MeshAlpha.io’s ION (the Rule Engine). A simple repeatable method signature provides > 80% of needed capabilities to query fact sets.

typescript

// this is a simple signature (pseudo code) that provides powerful functionality
type pseudoOperation = [
    string, // operation
    BaseTypes | PropertyQuery,
    ...(BaseTypes | PropertyQuery)[]
]

Given the above, I was armed with enough to create the first pass of the DSL, with VS-Code hinting by means of well defined Types and Type Guards in TypeScript. Below is a working POC of the DSL in ION-Query…

typescript

// rule-fact-query.json
{
    person : {
        type : "Person",
        where : [
            // NOTE* "and" is optional at root
            "and",
            // firstName is "John", "Jane", or "Jim"
            [ "eq", { $self : "firstName"},...["John","Jane","Jim"]],
            /*
            * lastName matches (Regex) strings ending with variations of "smith" ,"Smith",
            * "smithson", "Smithson"
            */
            [ "mt", { $self : "lastName"},"[sS]mith(?:son)$"],
            // age is greater or equal to 21
            [ "gte", { $self : "age"},21]
        ]
    },
    address : {
        type : "Address",
        where : [
            // join on personId between address and person
            [ "eq", { $person : "personId"},{ $self : "personId"}],
            // state is California
            [ "eq", { $self : "state"},"CA"],
            [
                "or",
                // city is in "Los Angeles", "Alameda", or "Culver City"
                [ "in", { $self : "city"},["Los Angeles","Alameda","Culver City"]],
                // addressLine1 starts with 10 or 40
                [ "sw", { $self : "addressLine1"},10,40]
            ]
        ]
    }
}

To be continued…

This was a very challenging exercise that will play a huge part in the creation of the ION-Query library. While this initial step is to prove the theory that a simple readable DSL could be written in JSON, TypeScript, and JavaScript (YAML will be added later), it opens the door to the next steps of supporting provided and custom functions, as well as prototyping the design for the actions supported when a query produces a result from a Fact Set.

ISO 6346 – Shipping Container Codes

Kamau Washington — Thu, 15 Sep 2022 00:00:00 GMT

This post is purely illustrative and showcases a few implementations that achieve the same goal. Feel free to use any of them in your implementation.

I often spend time researching and implementing algorithms in multiple languages as a way to keep my programming skillset sharp, learn a new language and its syntactical sugar, and to have the algorithms at my disposal when and if I need them.

This article references
Container Container ISO 6346
Wikipedia ISO 6346

ISO 6346 is an international standard that describes the identification of a shipping container. The standard is maintained by the BIC (International Container Bureau) and covers the serial number, owner, country code, and size of any given shipping container.

An example of a compliant ISO 6346 Code is CSQU3054383

Owner Code	Category Identifier	Serial Number	Check Digit
CSQ	U	305438	3
[A-Z]{3}	[UJRZ]	\d{6}	\d

A container owned by Hapag-Lloyd AG.

It is relatively straightforward to validate via Regex as shown above. However, the fun algorithm portion that validates a shipping container code is as follows.

Calculation 1

An equivalent numerical value is assigned to each letter of the alphabet, beginning with 10 for the letter A (11 and multiples thereof are omitted) for Owner Code and Category Identifier.

A	B	C	D	E	F	G	H	I	J	K	L	M
10	12	13	14	15	16	17	18	19	20	21	23	24

N	O	P	Q	R	S	T	U	V	W	X	Y	Z
25	26	27	28	29	30	31	32	34	35	36	37	38

Calculation 2

Each of the numbers calculated in step 1 is multiplied by 2^position, where position is the exponent to base 2. Position starts at 0, from left to right.

C	S	Q	U	3	0	5	4	3	8
13	30	28	32	3	0	5	4	3	8
1	2	4	8	16	32	64	128	256	512
13	60	112	256	48	0	320	512	768	4096

Calculation 3

Sum up all results of Step above = 6185
Divide them by 11 = 562.272..
Round the result down towards zero i.e. make the result a whole number (integer) = 562
Multiply the integer value by 11 = 6182
Subtract the result of (4) from the result of (1): This is the check digit = 3

Calculation 1 Implementations

While there are multiple calculations as outlined above, the Calculation 1 implementation is the most interesting. The goal for me was to retrieve the number mapped to an A-Z character without the use of if statements.

Implementation 1 (Dictionary)

This seems simple enough, create a dictionary/object that represents these values. This will be the most common implementation seen on the web.

typescript

const alphaReductionDictionary: { [character: string]: number } = {
        "A": 10, "B": 12, "C": 13, "D": 14, "E": 15, "F": 16, "G": 17, "H": 18, "I": 19, "J": 20, "K": 21, "L": 23, "M": 24,
        "N": 25, "O": 26, "P": 27, "Q": 28, "R": 29, "S": 30, "T": 31, "U": 32, "V": 34, "W": 35, "X": 36, "Y": 37, "Z": 38
 }

Implementation 2 (ASCII Character Code w/ Math)

However, I wanted to see if this can be solved via ASCII char code and math alone, as this seems to be a common theme in alphanumeric codes: omit multiples of n

typescript

const charCodeReduced = shippingContainerChar.charCodeAt(0) - 55; // this reduces ASCII A to 10, as per algorithm
const charCodeRemainder = charCodeReduced % 11; // identifies occurrences of multiples of 11, and the position of items that arent
const weight = Math.trunc((charCodeReduced - charCodeRemainder) / 10); // generates a number that indicates any addition to charCode needed to skip multiples of 11
return Math.trunc((charCodeReduced + weight) / 11) + charCodeReduced; // generates an integer as per the ISO 6346 specification

Implementation 3 (Most Elegant)

While I have two solid implementations, I realized that position played a part in the calculation, and with A starting at 10, 0-9 fit nicely in front of it. I thought I was original in this concept however, it seems that this is a valid way of solving this problem as well. It took some digging, but it was found on bitcoinwiki.

typescript

const characterIndexMap: string = "0123456789A?BCDEFGHIJK?LMNOPQRSTU?VWXYZ";
return Number(characterIndexMap.indexOf(shippingContainerChar, 1));

With these implementations in place, creating a function that completes the algorithm is easy. Note, the full implementation allows the implementations above to be injected into the calculation process.

typescript

export const isISO6346Compliant = function (containerNumber: string, getCharValue: (c: string) => number): boolean {
	// fail fast if any parameter is null or undefined
    if (!containerNumber || /^\s+$/.test(containerNumber) || !getCharValue) return false;

	// validate for ISO 6346 compliant container number
	const isoPattern: RegExp = /^[A-Z]{3}[UJRZ]\d{6}\d$/;

	// fail fast if the containerNumber does not meet ISO 6346 standards
    if (!isoPattern.test(containerNumber)) return false;

    // get check digit (last character of containerNumber)
    const checkDigit: number = Number(containerNumber[containerNumber.length - 1]);

    /*
     * Calculation Step 3a
     *
     * calculate sum of container number chars multiplied by exponent to base 2,
     * the last character (check digit) is omitted from calculation as per spec
     */
    let calcSumExpB2: number = 0;
    for (let i = 0; i < containerNumber.length - 1; i++) {
        let extractedValue: number = 0;
        const contNumChar: string = containerNumber[i];
        /*
         * the first 4 characters are alphabetical, they need to be converted with
         * multiples of 11 omitted
         */
        extractedValue = (i < 4)? getCharValue(contNumChar) : Number(contNumChar);
        // multiply the extracted value by exponent to base 2 using the position of the char
        calcSumExpB2 += extractedValue * Math.pow(2, i);
    }

    // Calculation Step 3b: Division by 11
    let stepCalc = calcSumExpB2 / 11;

    // Calculation Step 3c: Erase decimal digits
    stepCalc = Math.trunc(stepCalc);

    // Calculation Step 3d: Multiply by 11
    stepCalc *= 11;

    // Calculation Step 3e: a) - d) === CheckDigit
    return calcSumExpB2 - stepCalc === checkDigit;
}

There it is!

This exercise was a lot of fun and showcased various ways of implementing an algorithm. The full source code for this exercise can be found here.

AWS CDK Static Web + API Multiple Branches in One Account

Kamau Washington — Tue, 06 Sep 2022 00:00:00 GMT

This post is purely illustrative and is to be considered a deployment option that may suit an organization’s CICD workflow and feedback loop for Website and API resource deployment.

With one development team working on a single project in multiple branches, it may be advantageous to deploy each branch in an isolated manner that grants other developers and product owners access to a work in progress remotely.

A use case for this is a client-rendered SPA or Static Website with an API being deployed in AWS, three developers working on three separate new features in one Micro-frontend. The organization feedback loop standard is to publish to the development environment for visual review after completing code review.

The development environment would be written over multiple times with each commit, leaving reviewers to gamble as to which feature they are experiencing.

50,000ft view of namespace deployment

Technically namespace deployment is already performed in the manner of subdomains, resource tagging and prefixing, and other methods. If building a Stack on each commit, and tearing it down after a few hours is a desired or current workflow, this can be achieved via namespaced deployment as shown in the “dev” site below (for which I used a random phrase generator).

dev.example.com
- infinity-mouse.example.com
- highfalutin-cord.example.com
- makeshift-foot.example.com
qa.example.com
stage.example.com
http://www.example.com

So how do we do that in AWS?

We are going to need a few things first, but it can be done using the AWS CDK. These items should be complete or available before attempting to deploy.

AWS CDK installed
AWS credentials (best defined via profile)
A domain name that points to AWS nameservers
A CNAME for the wildcard domain (*.example.com – created by default)
A wildcard SSL certificate issued for the domain through ACM in the us-east-1 region as this is required by CloudFront for HTTPS (we want to test on HTTPS)

Create constants from Environment Variables

These environment variables will help with the naming of the resources while authoring the stack, these should be required.

DOMAIN_NAME
APPLICATION_NAME
NAMESPACE
CERTIFICATE_ARN

Add tags to all resources in the Stack

Stack-created resources may have unique names that do not include the Namespace as a helper construct may have generated them. Applying tags to all resources makes them easier to find when searching, and can be found all at once using the AWS Tag Editor.

To add tags to all resources in one go, it can be done when initializing the stack (in the bin dir).

typescript

const app = new cdk.App();
new AwsCdkNamespaceDeploymentWebStack(app, `${NAMESPACE}`, {});
cdk.Tags.of(app).add('NAMESPACE', NAMESPACE);
cdk.Tags.of(app).add('APPLICATION_NAME', APPLICATION_NAME);

Gotcha #1 – Plan to use “HostedZones.fromLookup()“? env needs to be set at the stack level!!!

This seems like an anti-pattern and doesn’t follow the convention of most constructs however, it was easy enough to remedy using process.env vs hard coding values. These environment variables get set regardless of how the AWS account and Region are loaded

typescript

export class AwsCdkNamespaceDeploymentWebStack extends CDK.Stack {
  constructor(scope: Construct, id: string, props?: CDK.StackProps) {
		super(scope, id, {
			env: {
				account: process.env.CDK_DEFAULT_ACCOUNT,
				region: process.env.CDK_DEFAULT_REGION,
			},
		});

Gotcha #2 S3 Bucket naming conventions

Since S3 bucket names are unique, they have a ton of restrictions on the bucket name. This should be ok, as the bucket name needs to match the domain name to be resolved by Route53 (not so much by CloudFront). If the final domain is infinity-mouse.example.com, then the bucket name should be infinity-mouse.example.com.

The bucket name must also be lowercased.

typescript

const websiteBucket: Bucket = new Bucket(this, `${NAMESPACE} S3 Bucket`, {
			websiteIndexDocument: 'index.html',
			bucketName: domainWithNamespace.toLowerCase(),
			removalPolicy: CDK.RemovalPolicy.DESTROY,
			autoDeleteObjects: true,
		});

Deploying Static Assets

For client-side rendering SPAs and Static Websites, the assets will either be compiled into or worked on in a directory in the project. CDK makes this deployment easy through the BucketDeployment construct.

typescript

const bucketDeployment: BucketDeployment = new BucketDeployment(
			this,
			`${NAMESPACE} S3 Bucket Deployment`,
			{
				sources: [Source.asset('./dist')],
				destinationBucket: websiteBucket,
			}
		);

An Origin Access Identity (OAI) must be created to grant S3 bucket read access to CloudFront.

The OAI allows the S3 bucket to remain private while allowing exposure through the controls of CloudFront.

typescript

const originAccessIdentity: OriginAccessIdentity = new OriginAccessIdentity(
			this,
			`${NAMESPACE} CloudFront OAI`
		);
		// grant read rights to the S3 Bucket for the OAI
		websiteBucket.grantRead(originAccessIdentity);

ApiGateway stageName needs to be set

In the ApiGateway documentation, it is stated that ApiGateway creates a “prod” stageName by default when creating a RestAPI. While this may be the case, errors were being thrown until this was set manually. This will need further review, however, this stage name could be anything as the environments are isolated from one another.

typescript

const api = new RestApi(this, `${NAMESPACE} Rest Api`, {
			description: `API for ${NAMESPACE} environment`,
			deployOptions: {
				stageName: 'prod',
			},
      restApiName : NAMESPACE
});

CloudFront Target and Aaaa record

For the solution to work properly the CloudFront distribution needs to be the next in command once a request reaches the hosted zone. The logic is saying, on the arrival of a request to this hosted zone (the root domain), send the request to an AWS service, in this case, CloudFront to handle routing, serving content, etc.

The use of Aaaa record is to use the IPv6 address of CloudFormation, there are benefits to IPv6 all of which can easily be found in IPv4 vs IPv6 comparison articles.

There it is!

This exercise was fun, and a bit challenging to say the least. Hopefully, it proves to be a good starting point for someone looking for this style of deployment.

The repository associated with this article will more than likely be refactored to clean it up a bit, possibly a neater file structure, the use of constructs, and or child stacks.

SHOW ME THE CODE!!!

https://github.com/kamauwashington/aws-cdk-namespaced-web-api

Using AWS CDK with dotenv -r and TypeScript

Kamau Washington — Tue, 06 Sep 2022 00:00:00 GMT

Note, I prefer the node —require option of loading .env variables over importing or requiring in application code.

TLDR;

install dotenv as a development dependency
in the cdk.json in the root of the project directory add the following in bold :
- { “app”: “npx ts-node -r dotenv/config —prefer-ts-exts bin/<stack-name>.ts” }

When developing via AWS CDK it is easy to set environment variables on resources as they are being defined to be stored in AWS. However, in some repositories there is a need to set environment variables for the CDK Stack to use.

This can be done from the command line, or by setting the environment variables for the user, terminal, or machine. However these options do not provide the convenience of dotenv especially when many environment variables are needed during local development (See my notes on dotenv for local only here)

Steps

confirm that ts-node can be found in the package.json of the AWS CDK project

it should be present as it is a part of cdk init
if ts-node is not present run npm install -D ts-node
npx can also be used

run npm install -D dotenv for the current AWS CDK TypeScript project
create a .env file in the project root and add the necessary environment variables
ensure the .env is added to the .gitignore file as this should not be kept in the repository
open the cdk.json file

find the app keyword
add -r dotenv/config right after npx ts-node
the end result should look like this :
- “app”: “npx ts-node -r dotenv/config —prefer-ts-exts bin/<stack-name>.ts”

Once this is done, the cdk commands will utilize the environment variables from the .env file. This is a very convenient way to load environment variables using a nodejs standard that most developers know, without having to install additional libraries, clis, or custom scripts.

Before

json

{
  "app": "npx ts-node --prefer-ts-exts bin/my-stack.ts",
  "watch": {
    "include": [
      "**"
    ],
    "exclude": [
      "README.md",
      "cdk*.json",
      "**/*.d.ts",
      "**/*.js",
      "tsconfig.json",
      "package*.json",
      "yarn.lock",
      "node_modules",
      "test"
    ]
  },
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
    "@aws-cdk/core:stackRelativeExports": true,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
    "@aws-cdk/aws-lambda:recognizeLayerVersion": true,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
    "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
    "@aws-cdk/core:checkSecretUsage": true,
    "@aws-cdk/aws-iam:minimizePolicies": true,
    "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
    "@aws-cdk/core:validateSnapshotRemovalPolicy": true,
    "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
    "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
    "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
    "@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
    "@aws-cdk/core:enablePartitionLiterals": true,
    "@aws-cdk/core:target-partitions": [
      "aws",
      "aws-cn"
    ]
  }
}

After

json

{
  "app": "npx ts-node -r dotenv/config --prefer-ts-exts bin/my-stack.ts",
  "watch": {
    "include": [
      "**"
    ],
    "exclude": [
      "README.md",
      "cdk*.json",
      "**/*.d.ts",
      "**/*.js",
      "tsconfig.json",
      "package*.json",
      "yarn.lock",
      "node_modules",
      "test"
    ]
  },
  "context": {
    "@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
    "@aws-cdk/core:stackRelativeExports": true,
    "@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
    "@aws-cdk/aws-lambda:recognizeVersionProps": true,
    "@aws-cdk/aws-lambda:recognizeLayerVersion": true,
    "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true,
    "@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
    "@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
    "@aws-cdk/core:checkSecretUsage": true,
    "@aws-cdk/aws-iam:minimizePolicies": true,
    "@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
    "@aws-cdk/core:validateSnapshotRemovalPolicy": true,
    "@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
    "@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
    "@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
    "@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
    "@aws-cdk/core:enablePartitionLiterals": true,
    "@aws-cdk/core:target-partitions": [
      "aws",
      "aws-cn"
    ]
  }
}

Avoid using ts-node when performance testing

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

Often times when performance testing, or comparing resource utilization (ex NodeJS vs Java), I have seen developers load testing, performance testing, and even one-off testing, with ts-node. ts-node is an awesome piece of kit, but it is for rapid prototyping, watching, and development only, it is not a production runtime.

When performance testing, even one offs, use tsc my-nodejs-app.ts && node my-node-app

Simple test sample

typescript

const numberArray : Array<number> = [2,6,1,9,0,5,8,7,3,4];
numberArray.sort();
const heapUsed = Math.round((process.memoryUsage().heapUsed / (1024*1024)));
const cpuTime = Math.round((process.cpuUsage().user / 1000));
console.log(`Approximately ${heapUsed}MB and ${cpuTime}ms of CPU time was used by this process.`);

Using ts-node

text

Approximately 61MB and 1466ms of CPU time was used by this process.

Using tsc index.ts && node index

text

Approximately 2MB and 25ms of CPU time was used by this process.

Notes

heapUsed is the actual memory used during the execution of a Node process
memoryUsage provides rss, heapTotal, heapUsed, external, arrayBuffers metrics as well
the command NODE_ENV=production tsc index.ts && node index will ensure that your one off test mimics your production environment build and runtime
this is an illustration, not an accurate representation of resource usage
- an APM + a performance test with sustained load over time will provide the most accurate information

A great article that discusses NodeJS profiling here…

AWS OpenSearch Search Request Logging

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

There may be rules and regulations for logging in your production environment, please consult with your web engineering, or infrastructure equivalent prior to logging in this fashion to prevent unwanted charges and or side-effects.

This one took some time to figure out, and the AWS documentation Monitoring audit logs in Amazon OpenSearch Service did not point directly to how to log inbound REST search requests to OpenSearch to CloudWatch. There wasn’t much on the web either in terms of discussions, or questions on the matter, so I hope this helps someone.

Steps

Enable Audit logs on [your-domain] in the AWS Console, or via REST, or via CLI
Create a CloudWatch log group, or choose an existing one.
Choose an access policy that contains the appropriate permissions, or create a policy
Add “logs:CreateLogStream” to the $.Statement.Action of the policy if not present

If using an existing policy, consult with the owner of the policy to determine if this action is already present and or if a new policy should be created for you

In OpenSearch Dashboard go to the Security Plugin
Configure General Settings
Enable REST Layer
Disable FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES unless of-course these are needed
Disable Transport layer unless of-course this is needed
Enable Request Body

Notes

It may be a good idea to ignore requests sent to _dashboard or any other paths that should be excluded from the logs to prevent overflow
*{$.audit_rest_request_path = _search} is an easy way to find the incoming the requests, though adding a custom header may be a better option to ensure that only valid searches are being pulled in the resultset

Sample request log record

json

{
    "audit_cluster_name": "111111111111:test-cluster",
    "audit_rest_request_params": {
        "index": "test"
    },
    "audit_node_name": "33b23f3f2e8e9607bc355b9c17cf21af",
    "audit_request_initiating_user": "prototype-user",
    "audit_rest_request_method": "GET",
    "audit_category": "AUTHENTICATED",
    "audit_request_origin": "REST",
    "audit_request_body": "YOUR SEARCH FILTER WOULD BE FOUND HERE AS TEXT vs JSON OBJECT",
    "audit_node_id": "4AIFR8dTTFuEkk9ce5Yd2w",
    "audit_request_layer": "REST",
    "audit_rest_request_path": "/test/_search",
    "@timestamp": "2022-04-14T02:24:05.539+00:00",
    "audit_request_effective_user_is_admin": false,
    "audit_format_version": 4,
    "audit_request_remote_address": "1.1.1.1",
    "audit_rest_request_headers": {
        "Accept": [
            "*/*"
        ],
        "Connection": [
            "close"
        ],
        "User-Agent": [
            "PostmanRuntime/7.28.4"
        ],
        "Host": [
            "search-chdxDWWhtdv3d2jxenfkdjq.us-east-1.es.amazonaws.com"
        ],
        "Accept-Encoding": [
            "gzip, deflate, br"
        ],
        "Cache-Control": [
            "no-cache"
        ],
        "Postman-Token": [
            "4d2342ca-0d38-44e0-8da6-2d8fd85f65d9"
        ],
        "Content-Length": [
            "1721"
        ],
        "Content-Type": [
            "application/json"
        ]
    },
    "audit_request_effective_user": "random-user"
}

Developer Ethos

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

During code review, triage, and early stages of sprint development we are so focused on the deliverables that we often forget that our commits, communication, teamwork, and ownership are more than simple roles and responsibilities… they are all part of our development culture.

I will ABC (Always Be Collaborative). Do not build in isolation. Collaborate with other developers working on an application, the end user, and the testers.
I will seek business requirement clarity. Don’t imagine and build. If there is a lack of clarity – discuss, record then build towards the exact business requirement for acceptance.
I will document my code, for myself and my peers. Self-documenting code is a myth. The best developers clearly and thoroughly explain each step of functionality, inputs, outputs, side effects, parameters, and often the reason why the functionality was introduced.
I will remain focused. Working on various applications in parallel is a surefire way to become overwhelmed, and potentially introduce one size fits all solutions that don’t scale. Tackle development serially in small, achievable blocks of time and code. Break larger tasks into smaller more manageable tasks whenever possible.
I will avoid shortcuts. Shortcuts have proven over time to create defect pile-ups. While tasks may take longer and require more code, files, thought, and collaboration – doing it immediately with tenacity will spare you and your peers from unneeded and unwanted cycle waste.
I will support the code that I author. As a project grows by feature, fix, or refactor, a developer should take pride in the ownership of the code introduced, and support changes if a peer has difficulty achieving a desired result or if the code introduced requires enhancement.
I will author code with performance in mind. All of the lazy loading, dynamic invocation, and tree-shaking in the world will not convert a slow application into a fast one on their own. Understand the common performance concerns of the architecture, framework, and language and code with these in mind.
I will code defensively. Remember Murphy’s Law : “Anything that can go wrong will go wrong”. Programming languages attempt to protect developers from mistakes, and ease complex implementations – but this is happy path. The opposite and or edge cases will happen.
I will unit test my code. Unit testing builds confidence for refactoring, maintenance, bug fixing, feature introduction, and can do all this locally before deployment. When issues are found, write unit tests for those cases as well. Remember, the tests are your safety net.
I will refactor my code. Refactoring is one of the best practices that all developers must have. Code in most cases can be improved after the first iteration. Upon completion of a deliverable, take the time to analyze what improvements can be made in the next iteration.
I will be an expert on the fundamentals. Knowing the latest and greatest is fashionable, and adds more tooling to your problem-solving arsenal. However, knowledge of the fundamentals often negates the use of a new approach or framework – for something much simpler.
I will be honest. I don’t know everything. My code is not perfect. I need to learn more. I just don’t get “this”. I think we could collaborate better as a team. There were some issues with your last merge request. I feel that I am not performing at the same level as my peers. I’m awesome! It is healthy to be honest with yourself, your peers, and your leadership. It is just as healthy to accept honesty.
I will raise awareness to problematic areas in a codebase. Defects and bugs found during sanity testing and or in higher environments are often created by anti-patterns, lack of documentation, happy path programming, indecipherable lengthy implementation, etc. Notify the original/last commit author, and team, mark the offending areas with a //TODO:, and create a backlog item.
I will write clean, readable code. Code is written for humans to read and understand. It is compiled for computers to understand. Taking shortcuts in the code to save a few characters or new lines is an anti-pattern that makes it more difficult for the next developer to read and maintain. “the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code. …Therefore, making it easy to read makes it easier to write.” – Robert C. Martin

Double vs Triple Equals

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

A very common point of confusion in JS/TS programming is what double equals and triple equals actually do. While this is base JS functionality, the method by how it works is quite simple (maybe) : Type Coercion

Because JavaScript is weakly typed, values can be easily converted from one type to another. Type Coercion is the implicit conversion from one type to another. There are only three types of conversion in Javascript: to string, to boolean, to number, allowing for “sameness” or “equality checking”

Double Equals checks for sameness and performs Type Coercion before the equality operation.
Triple Equals checks for strict type and value equality without performing Type Coercion.

JS Sameness (Double Equals) in action

it is easy to see the trouble one can get into

javascript

console.log("\n\t\r\n" == 0); // true
console.log("00000005" == 5); // true
console.log(0 == false); // true

JS Strict Equality (Triple Equals) in action

desired equality outcomes

javascript

console.log("\n\t\r\n" === 0); // false
console.log("00000005" === 5); // false
console.log(0 === false); // false

For fun, a brain teaser!

WTNaN???

javascript

console.log(!NaN == true); // true
console.log(!NaN === true); // true!?!?!?

Notes

IMPORTANT use ==, != when comparing a value or property with null or undefined (if allowed in linter), to ensure expected truthy/falsey with possibly null/undefined values
Purists will suggest that double equals should never be used. However, this should be a configurable lint rule, deciding to use or not to use double equals by a team or enterprise.
Type Coercion is invoked by many of the other operators (beware) <,<=,>,>=,~,!,+,-,*
- use typeof(x) === typeof(y) if you need to be certain beforehand
If using defined types, in 99% of cases TS transpilation should fail and force correction (thus, use more TS and use it appropriately)

A great article that discusses Type Coercion [here](https://www.freecodecamp.org/news/js-type-coercion-explained-27ba3d9a2839/#:~:text=Type%20coercion%20is%20the%20process,Symbol%20(added%20in%20ES6)…

Enum Flags … are … awesome!

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

An often overlooked feature of most if not all languages is the concept of Enum Flags. In a nutshell, Enum Flags are an efficient way of storing and representing a collection of boolean values with one value 😎. This applies to all SQL languages, C#, Java, Python, Rust, GoLang, the list goes on, and illustrated here in TypeScript.

Is it possible to store if a field is required, read-only, and visible in the number 13?

Before

typescript

interface IFieldRule {
    isReadOnly : boolean;
    isMasked : boolean;
    isRequired : boolean;
    isVisible : boolean;
    isHeaderVisible : boolean;
}

// fieldA should be readOnly, required, and visible
let fieldARules : IFieldRule = {
    isReadOnly : true,
    isMasked : false,
    isRequired : true,
    isVisible : true,
    isHeaderVisible : false
}

console.log(`the representation in JSON would be : ${JSON.stringify(fieldARules,null,2)}`)
/*
// the JSON representation will be ->
{
  "isReadOnly": true,
  "isMasked": false,
  "isRequired": true,
  "isVisible": true,
  "isHeaderVisible": false
}
*/
console.log(`Is fieldA readOnly? ${fieldARules.isReadOnly}`); //true
console.log(`Is fieldA required? ${fieldARules.isRequired}`); //true
console.log(`Is fieldA visible? ${fieldARules.isVisible}`); //true

After

typescript

enum FieldRule {
    None = 0,
    isReadOnly = 1, // 1 << 0
    isMasked = 2, // 1 << 1
    isRequired = 4, // 1 << 2
    isVisible = 8, // 1 << 3
    isHeaderVisible = 16, // 1 << 4 try it! console.log(1<<4) prints 16
}

// fieldA should be readOnly, required, and visible
let fieldARulesEnum : FieldRule = FieldRule.isReadOnly | FieldRule.isRequired | FieldRule.isVisible;

console.log(`the representation in JSON would be : ${fieldARulesEnum}`);
/*
// the JSON representation will be ->
   13
*/
console.log(`Is fieldA readOnly? ${(fieldARulesEnum & FieldRule.isReadOnly) == FieldRule.isReadOnly}`); //true
console.log(`Is fieldA required? ${(fieldARulesEnum & FieldRule.isRequired) == FieldRule.isRequired}`); //true
console.log(`Is fieldA visible? ${(fieldARulesEnum & FieldRule.isVisible) == FieldRule.isVisible}`); //true

Bonus

sql

SELECT
    COUNT(1)
FROM FieldRules
WHERE
    fieldName = 'fieldA' AND
    fieldRule & 4 <> 0 -- testing for isRequired

Notes

IMPORTANT only use when necessary, and where applicable. While this functionality is cool, determine if a use case warrants it.
Bitwise operations can be used to determine exclusivity, inclusivity, remove items, add items, if all flags are set, etc.
INSIGHT : Using the example above imagine a form with 40 fields and the JSON that would be needed if using booleans (200 fields), vs 40 with Enum Flags.
- This is a very useful exercise when thinking about data transmission, slow connections, compression/decompression
It is not uncommon to see a set of enum flags defined across an enterprise or vertical and used between different languages and datastores supporting bitwise operations (which is most if not all)

Visit the TypeScript Deep Dive for Enums for more in-depth information!

JS Array Functions Instead of Lodash

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

Lodash is an amazing library and is often used to minimize the hassle of working with arrays, numbers, objects, strings, etc. However, it is often seen that one of its primary uses is array manipulation, and accessing elements within.

TypeScript (and ES6) have the majority this functionality built in!

The example below illustrates the find method in Lodash vs TS/JS :

Before

typescript

import * as _ from 'lodash';
const workingArray : Array<number> = [0,1,2,3,4,5,6,7,8,9];
const foundResult : number = _.find(workingArray,(item : number) => item > 8);
console.log(foundResult); // prints 9

After

typescript

const workingArray : Array<number> = [0,1,2,3,4,5,6,7,8,9];
const foundResult : number | undefined = workingArray.find((item : number) => item > 8);
console.log(foundResult); // prints 9

But wait! I may need an index as well as the individual item!

typescript

const workingArray : Array<number> = [0,1,2,3,4,5,6,7,8,9];
const foundResult : number | undefined = workingArray.find((item : number, index : number) => index > 8 || item > 8);
console.log(foundResult); // prints 9

Notes

Lodash comes at a cost in file size and performance (in many cases compared to native), consider existing functionality before importing Lodash
IMPORTANT While inline anonymous functions work in most cases for simple logic, opt for a separate named function (imported from a separate file if warranted) to house complex manipulation/accessor logic promoting reuse and independent testability.
The above is a simple example using find, there are more methods available :
- every, some, includes, filter, map, reduce, and more!

Visit the Array Prototype Documentation for more in-depth information!

Literal Types / Type Guards

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

When using TypeScript, we often (and hopefully as a manner of practice, always) declare the type of a variable or property. TypeScript allows the possibility of a variable or property to be of one or more types. This means that a variable or property can be either a number, string, or custom type by declaring it like so :

typescript

let myVariable : number | string | ICustomType;

The interesting part is that this can be done with primitives, mixed primitives, and types for variables, properties, and functions!

Example 1

typescript

type TennisScore = "love" | 15 | 30 | 40 | "all" | "deuce" | "ad-in" | "ad-out";

let score : TennisScore = 30; // allowed
score = "love"; // allowed
score = 1;// Type '1' is not assignable to type 'TennisScore'.(2322)

Example 2

typescript

type CardinalDirection = "North" | "South" | "East" | "West";

function whichWay() : CardinalDirection {
  return "N"; // Type '"N"' is not assignable to type 'CardinalDirection'.(2322)
}

Example 3

typescript

type CardinalDirection = "North" | "South" | "East" | "West";
type CardinalDirectionShorthand = "N" | "S" | "E" | "W";

function whichWay() : CardinalDirection | CardinalDirectionShorthand {
  return "N"; // allowed
}

Notes

IMPORTANT this feature is NOT the same as runtime validation. This is often called “Narrowing” in scripting languages, which takes a loosely defined value, and narrows the possibilities of types and or primitives to defined constraints.
Literal types can be used to return one or more types, or pass one or more types of values to and from a function.
There is awesome intellisense support, notifying individuals working in a codebase what values or types are possible.
if..then logic will only allow comparison with values or types defined (with intellisense guidance as well). The exception is the else statement.

Visit the TypeScript Documentation on Literal Types for more in-depth information!

Preload dotenv, don’t require it

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

Oh my dotenv! If you work with NodeJS at some point you will use and implement dotenv. It loads environment variables from a .env file (configurable name btw), into process.env, following the 12 Factor App methodology of separating environment from code. Pure awesomeness, and makes development configuration loading a breeze mimicking your higher environments.

An important note. Do not use .env in production! Your environments be them containers, RHEL, Cloud, CentOS, Windows Server are responsible for and capable of providing environment variables without dotenv.

Given the above, one should also avoid bundling dotenv as a dependency, exclude configuration in the codebase, and consider dotenv a developer workstation tool only.

Avoid

dotenv documentation states “As early as possible in your application, require and configure dotenv.“. To avoid loading in higher environments, most perform the following check :

javascript

if (process.env.node_env == 'development') require('dotenv').config() //this should not be in your code

Better

Preload dotenv in your start script, by installing dotenv as a development dependency npm install —save-dev dotenv and using Node’s built in -r flag to preload the configuration module from dotenv, you no longer have to _require(‘dotenv’).config() in your code!

shell

$ node -r dotenv/config your_script.js

But I use ts-node!!!

Node’s -r flag applies to ts-node as well!

shell

$ts-node -r dotenv/config your_script.ts

Notes

It is still possible to use the dotenv_config_path argument using the preload method, for custom file name and path
this should be placed in your npm scripts section in start or even better start-dev
- important note, avoid using npm start in production (though Heroku and similar services do for simplicity and workflow), best practice is to use the NodeJS binary instead of the package manager like so node my-app.js

Template Literals

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

In many TS code-reviews one may find string concatenation (with the addition operator) used to assemble a message, query, multi-line comments, or other manners of literals. While this gets the job done, it is often prone to error, difficult to read, and in many cases tedious to refactor and or implement.

TypeScript (and ES6) added Template Literals, and the TS/JS world couldn’t be happier!

Before

typescript

const firstName : string = "Jane";
const lastName :  string = "Smith";
const message : string = "A person with the first name of " + firstName + " and the last name of " + lastName + " says hello!";
console.log(message) // A person with the first name of Jane and the last name of Smith says hello!"

After

typescript

const firstName : string = "Jane";
const lastName :  string = "Smith";
const message : string = `A person with the first name of ${firstName} and the last name of ${lastName} says hello!`;
console.log(message) // A person with the first name of Jane and the last name of Smith says hello!"

Notes

this “magic” happens when backticks are used to encapsulate a string vs ticks or quotes
you can use full expressions inside of the ${expresssion|variable-name} directive
use the backtick to create multi-line strings as well without using newline and the addition operator
TS Template Literals are POWERFUL much more than is shown above!

Visit the TypeScript Documentation for Template Literals for more in-depth information!

The awesome Readonly Utility Type

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

Lets all be honest, side effects are terrible. They are especially terrible when working with a runtime that doesn’t implicitly discern between mutable and immutable pre transpilation or compilation (like JavaScript). I have often found functions during code review where an Array or Object is manipulated in the subroutine returning the input Array or Object. Ideally, functions should remain pure.

A pure function is a function where the return value is only determined by its input values, without observable side effects…

So how do we enforce immutability pre-transpilation? Is that even possible in TypeScript?

Before

typescript

let myNumArray : Array<number> = [0,1,2,3,4,5];

function removeFirstItem (inArray : Array<number>) : Array<number> {
    inArray.shift();
    return inArray;
}
removeFirstItem(myNumArray);
console.log(myNumArray); // prints [1, 2, 3, 4, 5]

After (will not compile enforcing pure function practice)

typescript

let myNumArray : Array<number> = [0,1,2,3,4,5];

function removeFirstItem (inArray : Readonly<Array<number>>) : Readonly<Array<number>> {
    inArray.shift(); // Property 'shift' does not exist on type 'readonly number[]'.(2339)
    return inArray;
}
removeFirstItem(myNumArray);
console.log(myNumArray);

After pt. 2

typescript

let myNumArray : Array<number> = [0,1,2,3,4,5];

function removeFirstItem (inArray : Readonly<Array<number>>) : Readonly<Array<number>> {
    let numArrayOut : Array<number> = inArray.concat(); // returns new mutable array;
    numArrayOut.shift();
    return numArrayOut; //implicit cast to Readonly<T>
}
removeFirstItem(myNumArray);
console.log(myNumArray); prints [0, 1, 2, 3, 4, 5] untouched

Notes

IMPORTANT The above code WILL RUN (in the “After” block), but if using TypeScript it will not compile
The use of Readonly as a return value of a pure function is a design choice not a mandate. It is however good practice as it requires any chained operations or next use to explicitly create a new mutable instance to work with, further enforcing pure functions principles.
This is very useful in function type definitions as it explicitly requires that the implementation be pure.

Visit the TypeScript Deep Dive for Readonly for more in-depth information.

Use for..of in lieu of traditional incremental for loops when appropriate

Kamau Washington — Mon, 05 Sep 2022 00:00:00 GMT

For loops are for loops, sure. There is absolutely nothing wrong with the traditional incremental iterator approach. However, this recommendation is about readability, maintenance, decreasing code smells, as well as best practice when using loops in TypeScript (and in this case ES6+ as well). Often during code review, one will find references to an item via its parent array and index numerous times (think myArray[i].firstName and guess how lastName is accessed? 😞). It is a best practice in most if not all languages to create a variable out of this value accessed by index once.

The example below illustrates for..of in TS and ES6+

Before

typescript

const workingArray : Array<number> = [0,1,2,3,4,5,6,7,8,9];
for (let i=0;i<workingArray.length;i++) {
  console.log(workingArray[i]);
}

After

typescript

const workingArray : Array<number> = [0,1,2,3,4,5,6,7,8,9];
for (const item of workingArray) {
  console.log(item);
}

But wait! I may need an index as well as the individual item!

it is so awesome that someone thought about this beforehand!

typescript

const workingArray : Array<number> = [0,1,2,3,4,5,6,7,8,9];
for (const [index,item] of workingArray.entries()) {
  console.log(`${item} has an index of ${index}`);
}

Notes

for..of and for..in are not the same. for..of is for items in an array, for..in is for properties of an object. It is syntactically confusing when said aloud admittedly 😕
utilize const and let keywords where appropriate, to explicitly define mutability pre-transpilation, and prevent simple coding mistakes.
for..of gets transpiled to the incremental iterator style when targeting < ES6 for backward compatibility
Why use for..of over Array.prototype.forEach? Each has its own use case. However, at the time of authoring, it is not possible to stop or break a forEach without throwing an exception which is not always desirable.
- It is important to note that using await in a loop, only works in sequence with for..of
INSIGHT: A reverse iterator is missing IMHO, as it is not always desirable to reverse an array then loop. A reverse iterator would be the equivalent of traditional for loops with i–;

Visit the TypeScript Deep Dive on for..of for more in-depth information!