Table of Contents

1 Background

This consists of an overview of some of the core technologies available in 2019.

Most of the examples here will be a combination of JavaScript, Python, Scheme, and Clojure.

Some of the core elements of https://www.thoughtworks.com/radar/techniques will be included in the review specifically as it relates to ML.

  • CD4ML
  • productionizing Jupyter Notebooks

2 Compiler

2.2 AST: Esprima

http://esprima.org/

const esprima = require('esprima');
const program = 'const answer = 42';
esprima.tokenize(program);

2.3 AST: CSS

2.4 REPL

Task: Incrementally build a lisp interpreter capable of running the following:

(define a 1)

(define b (lambda (x) (+ 1 x)))

(b 1)

(+ 1 1)

(quote 1)

(car (quote (1 2)))

(cdr (quote (1 2)))

(cond (true) 1 2)

(define fib
  (lambda (n)
        (if (or (= n 1)
                        (= n 2))
                  n
                  (+ (fib (- n 1))
                         (fib (- n 2))))))

(fib 5)

3 Interpreters

4 Evaluation

5 Data Structures

5.1 Array

5.2 Node

5.3 Linked Lists

5.4 Dictionary

5.6 BST

5.7 Heapsort

5.8 Graphs

6 Types

7 Design

7.1 Swagger

Provide some background on OpenAPI v3 particularly as it relates to paths, requests, and responses.

7.2 SOLID

  • Single responsibility principle
  • Open/closed principle
  • Liskov substitution principle
  • Interface segregation principle
  • Dependency inversion principle

https://en.wikipedia.org/wiki/SOLID(object-orienteddesign)

7.3 SRS

  • The Single Responsibility Principle

There should never be more than one reason for a class to change.

http://www.objectmentor.com/resources/articles/srp.pdf

7.5 Chaining

7.6 Immutability

8 Security

8.2 OWASP

There should be some familiarity with mechanism for designing secure systems that take the OWASP top ten into account.

As part of design there should be concepts of standard request and response objects, schema parsing, field validation, default values, and error handling.

8.2.1 Injection

8.3 CORS

Indicate how to enable CORS on a server.

https://enable-cors.org/server_nginx.html

server = http.createServer(function(req,res){
        res.setHeader('Access-Control-Allow-Origin', '*');
        res.setHeader('Access-Control-Request-Method', '*');
        res.setHeader('Access-Control-Allow-Methods', 'OPTIONS, GET');
        res.setHeader('Access-Control-Allow-Headers', 'authorization, content-type');
        if ( req.method === 'OPTIONS' ) {
                res.writeHead(200);
                res.end();
                return;
        }
});

8.4 JWT

See the encoding example noted in https://github.com/jwalsh/jwt-example/blob/master/index.js or generate a token from https://jwt.io/ .

const encoded = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c';

const base64UrlEncode = (string) => {
        return btoa(encodeURIComponent(string));
};

const base64UrlDencode = (string) => {
        return decodeURIComponent(atob(string));
};

console.log(base64UrlDencode(base64UrlEncode(JSON.stringify([1, 2, 3]))))

const decode = (encoded) => {
        const [h, p, s] = encoded.split('.');
        return base64UrlDencode(p);
}

console.log(JSON.parse(decode(encoded)))

8.5 Static analysis

8.6 SOX

https://www.sarbanes-oxley-101.com/sarbanes-oxley-checklist.htm

  • Establish safeguards to prevent data tampering
  • Establish safeguards to establish timelines
  • Establish verifiable controls to track data access
  • Ensure that safeguards are operational
  • Periodically report the effectiveness of safeguards
  • Detect Security Breaches
  • Disclose security safeguards to SOX auditors
  • Disclose security breaches to SOX auditors
  • Disclose failures of security safeguards to SOX auditors

8.7 Risk assessment

8.7.1 DREAD

https://en.wikipedia.org/wiki/DREAD_(risk_assessment_model)

  • Damage – how bad would an attack be?
  • Reproducibility – how easy is it to reproduce the attack?
  • Exploitability – how much work is it to launch the attack?
  • Affected users – how many people will be impacted?
  • Discoverability – how easy is it to discover the threat?

8.7.2 STRIDE

  • Spoofing of user identity
  • Tampering
  • Repudiation
  • Information disclosure (privacy breach or data leak)
  • Denial of service (D.o.S)
  • Elevation of privilege

8.8 SSL

server=dev.wal.sh
openssl req -out $server.csr -new -newkey rsa:4096 -sha256 -nodes -keyout $server.key -subj "/C=US/ST=Massachusetts/L=Boston/O=Walsh/OU=IT Department/CN=$server"

9 Data

9.1 GraphQL

10 UI

10.2 CSS Preprocessors

  • SASS
  • LESS
  • Garden

10.3 CSS Styles

11 Coding

11.1 Python: core

11.2 Python: numpy

11.3 Python: sklearn

11.4 JavaScript: core

11.5 JavaScript: web

11.6 JavaScript: React

11.7 JavaScript: lodash

11.8 TypeScript: core

11.9 Java: core

  • Enumeration
  • BitSet
  • Vector
  • Stack
  • Dictionary
  • Hashtable
  • Properties

11.10 Clojure: core

11.11 Clojure: core/async

11.12 ClojureScript: core

11.13 Rust

11.14 Go

11.15 Ruby: Rails

12 DataViz

13 Statistics

13.1 stochastic

https://github.com/jwalsh/stochastic

var stochastic = require('@jwalsh/stochastic');
const exp = stochastic.exp(20);

14 Machine Learning

14.1 Linear regression

x,y
-26,-52
-279,-558
-163,-326
-254,-508
72,144
-120,-240
401,802
-297,-594
238,476

import matplotlib.pyplot as plot
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

14.2 Features and models

{
  "version" : "1.0",
  "rowId" : null,
  "rowWeight" : null,
  "targetAttributeName" : "y",
  "dataFormat" : "CSV",
  "dataFileContainsHeader" : true,
  "attributes" : [ {
    "attributeName" : "x",
    "attributeType" : "NUMERIC"
  }, {
    "attributeName" : "y",
    "attributeType" : "NUMERIC"
  } ],
  "excludedAttributeNames" : [ ]
}


14.3 Schemas

14.4 Resource

14.5 Estimation

14.6 Interpretibility

  • What-If Tool - model comparison

15 Tooling

15.1 Eslint

15.2 Prettier

15.3 Emacs

15.4 Postman

16 Testing

16.1 pytest

16.2 Jest

17 AWS

17.1 Overview

  • Region > Availability zone
  • Compute, sstorate, db, tools, security, crypto, ml, scaling, migration, mobile, networking, analytics, integrations, IoT, VR, SDKs

17.2 Ansible

17.3 CLI

touch .tmp
aws s3 cp .tmp s3://$USER-tmp/

18 Environments

18.1 Ansible

18.2 Kubernetes

19 Services

19.1 Maven

19.2 Artifactory

19.3 GitHub

19.4 GitLab

19.6 Docker

docker run \
  --rm \
  -u root \
  -p 8080:8080 \
  -v jenkins-data:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$HOME":/home \
  jenkinsci/blueocean

19.7 Ansible

19.8 Slack

19.9 Jira

19.10 DataDog

19.11 Kafka

  • Broker
  • Consumer
  • Group
  • Message
  • Offset
  • Partition
  • Pipeline
  • Producer
  • Stream
  • Topic

19.12 Vault

Secrets as a service.

20 Deployment

20.1 CI/CD

21 Operations

  • unix services

22 Logging

https://docs.google.com/presentation/d/14PiMsvqs7f5A3_leDE07HoQkbHf_sUfuNHDShUMyT3A/edit#slide=id.g731d2a787_0_13

  • Crash
  • Enter debugger
  • Throw exception
  • Return special value
  • Set error flag
  • Recover, keep going

23 Debugging

23.1 Defects

  • Defect: An incorrect program code (a bug in the code).
  • Infection: An incorrect program state (a bug in the state).
  • Failure: An observable incorrect program behavior (a bug in the behavior).

Why Programs Fail: A Guide to Systematic Debugging, 2ed.

23.2 TRAFFIC

  • Track the problem in the database.
  • Reproduce the failure.
  • Automate and simplify the test case.
  • Find possible infection origins.
  • Focus on the most likely origins:
    • Code smells
    • Known infections
    • Causes in state, code, and input
  • Isolate the cause
  • Correct the defect.

http://www.whyprogramsfail.com/slides.php

24 Types

24.1 Pairs

const pair = (_first, _rest) => (_) => _(_first, _rest);
const first = (_pair) => _pair((_first, _rest) => _first);
const rest = (_pair) => _pair((_first, _rest) => _rest);

25 Refactoring

  • Duplicate Observed Data
  • Encapsulate Downcast
  • Encapsulate Field
  • Extract Interface
  • Introduce Foreign Method
  • Introduce Local Extension
  • Pull Up Constructor Body
  • Pull Up Field
  • Push Down Field
  • Replace Delegation with Inheritance
  • Replace Type Code with Class
  • Replace Type Code with Subclasses
  • Add Parameter
  • Change Bidirectional Association to Unidirectional
  • Change Reference to Value
  • Change Unidirectional Association to Bidirectional
  • Change Value to Reference
  • Collapse Hierarchy
  • Consolidate Conditional Expression
  • Consolidate Duplicate Conditional Fragments
  • Decompose Conditional
  • Dynamic Method Definition
  • Eagerly Initialized Attribute
  • Encapsulate Collection
  • Extract Class
  • Extract Method
  • Extract Module
  • Extract Subclass
  • Extract Superclass
  • Extract Surrounding Method
  • Extract Variable
  • Form Template Method
  • Hide Delegate
  • Hide Method
  • Inline Class
  • Inline Method
  • Inline Module
  • Inline Temp
  • Introduce Assertion
  • Introduce Class Annotation
  • Introduce Expression Builder
  • Introduce Gateway
  • Introduce Named Parameter
  • Introduce Null Object
  • Introduce Parameter Object
  • Isolate Dynamic Receptor
  • Lazily Initialized Attribute
  • Move Eval from Runtime to Parse Time
  • Move Field
  • Move Method
  • Parameterize Method
  • Preserve Whole Object
  • Pull Up Method
  • Push Down Method
  • Recompose Conditional
  • Remove Assignments to Parameters
  • Remove Control Flag
  • Remove Middle Man
  • Remove Named Parameter
  • Remove Parameter
  • Remove Setting Method
  • Remove Unused Default Parameter
  • Rename Method
  • Replace Abstract Superclass with Module
  • Replace Array with Object
  • Replace Conditional with Polymorphism
  • Replace Constructor with Factory Method
  • Replace Data Value with Object
  • Replace Delegation With Hierarchy
  • Replace Dynamic Receptor with Dynamic Method Definition
  • Replace Error Code with Exception
  • Replace Exception with Test
  • Replace Hash with Object
  • Replace Inheritance with Delegation
  • Replace Loop with Collection Closure Method
  • Replace Magic Number with Symbolic Constant
  • Replace Method with Method Object
  • Replace Nested Conditional with Guard Clauses
  • Replace Parameter with Explicit Methods
  • Replace Parameter with Method
  • Replace Record with Data Class
  • Replace Subclass with Fields
  • Replace Temp with Chain
  • Replace Temp with Query
  • Replace Type Code with Module Extension
  • Replace Type Code With Polymorphism
  • Replace Type Code with State/Strategy
  • Self Encapsulate Field
  • Separate Query from Modifier
  • Split Temporary Variable
  • Substitute Algorithm

26 SDLC

26.2 Agile

27 Estimation

28 Metrics

28.1 Four key metrics

  • lead time,
  • deployment frequency,
  • mean time to restore (MTTR)
  • change fail percentage

29 Projects

See some of the external repositories for the project work.

29.1 Make a Lisp

29.4 Data Structures

Re-implement the core APIs for stacks, enumerations, lists, etc.

29.5 React+Apollo

https://www.typescriptlang.org/docs/handbook/react-&-webpack.html

cd ~/sandbox
if [ ! -d apollo-example ]
then
        nvm use 10
        npx create-react-app apollo-example
fi
cd apollo-example
mkdir -p src/components
npm i parcel-bundler -D
npm i typescript -D
npm i -D @types/react @types/react-dom

30 IT

30.1 LDAP

30.2 VPN

30.3 On-boarding/Off-boarding

31 Conferences

32 Training

Author: Jason Walsh

Created: 2019-07-13 Sat 11:26

Validate