Leveraging jq for JSON Data Manipulation in Various AWS Services

jq
- GraphQL
- package.json
  - Contains
- AWS
  - RDS
    - Proof of Testing
  - ECR
    - Null and Select
  - SES
    - Sort and Raw Encoding
  - Lambda
    - CSV
  - Secrets Manager
Pipeline diagram
Related notes
Postscript (2026)

jq

https://jqplay.org/

GraphQL

{
  viewer {
    commitComments(first: 10) {
      edges {
        node {
          author {
            login
          }
          commit {
            comments(first: 1) {
              edges {
                node {
                  commit {
                    message
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

{
  "data": {
    "viewer": {
      "commitComments": {
        "edges": [
          {
            "node": {
              "author": {
                "login": "jwalsh"
              },
              "commit": {
                "comments": {
                  "edges": [
                    {
                      "node": {
                        "commit": {
                          "message": "Data files for Alexa"
                        }
                      }
                    }
                  ]
                }
              }
            }
          },
          {
            "node": {
              "author": {
                "login": "jwalsh"
              },
              "commit": {
                "comments": {
                  "edges": [
                    {
                      "node": {
                        "commit": {
                          "message": "Specify a minimum version for Buffer.allocUnsafe\n\nhttps://nodejs.org/api/buffer.html#buffer_class_method_buffer_allocunsafe_size"
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        ]
      }
    }
  }
}

Extracting

cat response.json | jq '.data.foo.edges[]|.node'

Conversions

cat response.json | jq '.data.foo.edges[]|.node.bar|{id:.id,number:.number|tonumber}'

Filtering

cat response.json | jq '.data.foo.edges[]|.node.bar|{id:.id,number:.number|tonumber}|select(.number > 200)'

Recombining

cat response.json | jq '.data.foo.edges[]|.node.bar' | jq -s .

package.json

Contains

cat package-lock.json  | jq '.dependencies[]|select(.resolved|contains("core-js"))'

AWS

RDS

Proof of Testing

aws rds describe-db-instances | jq -r '.DBInstances[]|[.DBInstanceIdentifier,.EngineVersion,.DBInstanceClass,.CACertificateIdentifier]|@csv'

ECR

Null and Select

aws ecr list-images --repository-name foo | jq '.imageIds[]|select(.imageTag != null)|select(.imageTag|contains("1.0.0"))'

SES

Sort and Raw Encoding

aws ses list-identities  | jq -r '.Identities|sort|.[]'

Lambda

aws lambda list-functions | jq '[.Functions[].FunctionName]|sort'

CSV

aws lambda list-functions | jq -r '.Functions[]|[.FunctionName,.LastModified]|@csv'

Secrets Manager

aws secretsmanager list-secrets | jq -r '.SecretList[]|select(.Name=="dev" or .Name=="qa" or .Name=="stg")|.ARN'| cut -d : -f 7 | cut -d '-' -f 1

#!/bin/sh
for E in $(aws secretsmanager  list-secrets | jq -r '.SecretList[]|.Name'); do
    echo $E
    aws secretsmanager get-secret-value --secret-id $E | jq '.SecretString|fromjson'
done

Pipeline diagram

Listing 1: A composable jq filter pipeline — input flows left-to-right through identity, field access, iteration, predicate filter, and transformation.

// jq pipeline — composable filters from input to output, with parallel branches
digraph jq_pipeline {
    rankdir=LR;
    graph [bgcolor="white", fontname="Helvetica", fontsize=11,
           pad="0.3", nodesep="0.3", ranksep="0.35"];
    node  [shape=box, style="rounded,filled", fontname="Helvetica",
           fontsize=10, fillcolor="#f5f5f5", color="#888"];
    edge  [color="#aaa"];

    input  [label="Input JSON\n{...}", color="#d36", fontcolor="#d36"];
    ident  [label="."           , color="#d63", fontcolor="#d63"];
    field  [label=".foo"        , color="#d63", fontcolor="#d63"];
    iter   [label=".[]"         , color="#693", fontcolor="#693"];
    sel    [label="select(.n>0)", color="#369", fontcolor="#369"];
    mapb   [label="map(.x*2)"   , color="#639", fontcolor="#639"];
    sortb  [label="sort_by(.n)" , color="#639", fontcolor="#639"];
    join   [label="@csv / -s"   , color="#963", fontcolor="#963"];
    output [label="Output\nstream", color="#d36", fontcolor="#d36"];

    input -> ident -> field -> iter;
    iter  -> sel   -> mapb  -> join;
    iter  -> sortb -> join  [label="parallel branch", fontsize=9, fontcolor="#888"];
    join  -> output;
}

Related notes

GraphQL — jq is the canonical post-processor for GraphQL responses; this note covers query shape and what shows up in data.*.
Clojure + GraphQL — server-side schema resolution that produces the JSON jq consumes; useful for understanding why fields nest the way they do.
dataLayer schema — analytics events arrive as deeply nested JSON; the same .[] | select(...) | map(...) pattern applies when slicing GTM payloads.
Terraform — terraform show -json and terraform output -json make jq the natural tool for state inspection and CI assertions.

Postscript (2026)

The jq ecosystem diversified considerably after 2023. Two reimplementations now ship in package managers everywhere: gojq (Go, embeddable, slightly different number semantics) and jaq (Rust, faster on most pipelines, uses applicative semantics). For YAML the de-facto tool is mikefarah/yq, which speaks jq syntax over YAML/TOML/XML and is what most CI scripts now reach for. Upstream jq 1.7 (2023) finally landed after a six-year gap, bringing SIMD-accelerated parsing via simdjson, much better error messages with source locations, and SQL-style operators (INDEX, GROUP_BY). For new pipelines on FreeBSD I default to jaq for speed and yq for anything touching Kubernetes manifests; classic jq remains the lingua franca for shell snippets and AWS CLI examples like the ones above.