wal.sh
home > research > jq · May 20, 2026

Leveraging jq for JSON Data Manipulation in Various AWS Services

Table of Contents

1. jq

1.1. GraphQL

{
  viewer {
    commitComments(first: 10) {
      edges {
        node {
          author {
            login
          }
          commit {
            comments(first: 1) {
              edges {
                node {
                  commit {
                    message
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}


{
  "data": {
    "viewer": {
      "commitComments": {
        "edges": [
          {
            "node": {
              "author": {
                "login": "jwalsh"
              },
              "commit": {
                "comments": {
                  "edges": [
                    {
                      "node": {
                        "commit": {
                          "message": "Data files for Alexa"
                        }
                      }
                    }
                  ]
                }
              }
            }
          },
          {
            "node": {
              "author": {
                "login": "jwalsh"
              },
              "commit": {
                "comments": {
                  "edges": [
                    {
                      "node": {
                        "commit": {
                          "message": "Specify a minimum version for Buffer.allocUnsafe\n\nhttps://nodejs.org/api/buffer.html#buffer_class_method_buffer_allocunsafe_size"
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        ]
      }
    }
  }
}

1.1.1. Extracting

cat response.json | jq '.data.foo.edges[]|.node'

1.1.2. Conversions

cat response.json | jq '.data.foo.edges[]|.node.bar|{id:.id,number:.number|tonumber}'

1.1.3. Filtering

cat response.json | jq '.data.foo.edges[]|.node.bar|{id:.id,number:.number|tonumber}|select(.number > 200)'

1.1.4. Recombining

cat response.json | jq '.data.foo.edges[]|.node.bar' | jq -s .

1.2. package.json

1.2.1. Contains

cat package-lock.json  | jq '.dependencies[]|select(.resolved|contains("core-js"))'

1.3. AWS

1.3.1. RDS

1.3.1.1. Proof of Testing
aws rds describe-db-instances | jq -r '.DBInstances[]|[.DBInstanceIdentifier,.EngineVersion,.DBInstanceClass,.CACertificateIdentifier]|@csv'

1.3.2. ECR

1.3.2.1. Null and Select
aws ecr list-images --repository-name foo | jq '.imageIds[]|select(.imageTag != null)|select(.imageTag|contains("1.0.0"))'

1.3.3. SES

1.3.3.1. Sort and Raw Encoding
aws ses list-identities  | jq -r '.Identities|sort|.[]'

1.3.4. Lambda

aws lambda list-functions | jq '[.Functions[].FunctionName]|sort'
1.3.4.1. CSV
aws lambda list-functions | jq -r '.Functions[]|[.FunctionName,.LastModified]|@csv'

1.3.5. Secrets Manager

aws secretsmanager list-secrets | jq -r '.SecretList[]|select(.Name=="dev" or .Name=="qa" or .Name=="stg")|.ARN'| cut -d : -f 7 | cut -d '-' -f 1

#!/bin/sh
for E in $(aws secretsmanager  list-secrets | jq -r '.SecretList[]|.Name'); do
    echo $E
    aws secretsmanager get-secret-value --secret-id $E | jq '.SecretString|fromjson'
done

2. Pipeline diagram

// jq pipeline — composable filters from input to output, with parallel branches
digraph jq_pipeline {
    rankdir=LR;
    graph [bgcolor="white", fontname="Helvetica", fontsize=11,
           pad="0.3", nodesep="0.3", ranksep="0.35"];
    node  [shape=box, style="rounded,filled", fontname="Helvetica",
           fontsize=10, fillcolor="#dbeafe", color="#888", fontcolor="#555"];
    edge  [color="#888", fontcolor="#555"];

    input  [label="Input JSON\n{...}", fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];
    ident  [label="."           , fillcolor="#fef3c7", color="#b45309", fontcolor="#b45309"];
    field  [label=".foo"        , fillcolor="#fef3c7", color="#b45309", fontcolor="#b45309"];
    iter   [label=".[]"         , fillcolor="#dcfce7", color="#15803d", fontcolor="#15803d"];
    sel    [label="select(.n>0)", fillcolor="#dbeafe", color="#1d4ed8", fontcolor="#1d4ed8"];
    mapb   [label="map(.x*2)"   , fillcolor="#ede9fe", color="#6b21a8", fontcolor="#6b21a8"];
    sortb  [label="sort_by(.n)" , fillcolor="#ede9fe", color="#6b21a8", fontcolor="#6b21a8"];
    join   [label="@csv / -s"   , fillcolor="#fef9c3", color="#a16207", fontcolor="#a16207"];
    output [label="Output\nstream", fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];

    input -> ident -> field -> iter;
    iter  -> sel   -> mapb  -> join;
    iter  -> sortb -> join  [label="parallel branch", fontsize=9, fontcolor="#555"];
    join  -> output;
}

diagram-pipeline.png

3. Related notes

  • GraphQL — jq is the canonical post-processor for GraphQL responses; this note covers query shape and what shows up in data.*.
  • Clojure + GraphQL — server-side schema resolution that produces the JSON jq consumes; useful for understanding why fields nest the way they do.
  • dataLayer schema — analytics events arrive as deeply nested JSON; the same .[] | select(...) | map(...) pattern applies when slicing GTM payloads.
  • Terraform — terraform show -json and terraform output -json make jq the natural tool for state inspection and CI assertions.

4. Postscript (2026)

The jq ecosystem diversified considerably after 2023. Two reimplementations now ship in package managers everywhere: gojq (Go, embeddable, slightly different number semantics) and jaq (Rust, faster on most pipelines, uses applicative semantics). For YAML the de-facto tool is mikefarah/yq, which speaks jq syntax over YAML/TOML/XML and is what most CI scripts now reach for. Upstream jq 1.7 (2023) finally landed after a six-year gap, bringing SIMD-accelerated parsing via simdjson, much better error messages with source locations, and SQL-style operators (INDEX, GROUP_BY). For new pipelines on FreeBSD I default to jaq for speed and yq for anything touching Kubernetes manifests; classic jq remains the lingua franca for shell snippets and AWS CLI examples like the ones above.