ADS-B Pipeline Implementation Guide

1. Overview
2. Phase Overview
3. Phase 1: Core Parsing
4. Phase 2: Validation Layer
5. Phase 3: Multi-Receiver Deduplication
6. Phase 4: Clean Output
- 6.1. Tasks
- 6.2. Output Schema
7. Phase 5: Identity Resolution (Future)
8. Phase 6: Enrichment (Future)
9. Success Metrics
10. Test Data Generation
11. Key Insight: Device Serial as ID

1. Overview

Implementation guide for building a clean ADS-B data pipeline. Each phase is self-contained and testable.

2. Phase Overview

Phase 1: Core Parsing (Foundation)
    │
    ▼
Phase 2: Validation Layer
    │
    ▼
Phase 3: Multi-Receiver Deduplication
    │
    ▼
Phase 4: Clean Output
    │
    ▼
Phase 5: Identity Resolution (Future)
    │
    ▼
Phase 6: Enrichment (Future)

3. Phase 1: Core Parsing

Goal: Parse SBS messages from dump1090 reliably.

3.1. Tasks

Task	Input	Output	Handle
SBS Parser	Raw SBS line	RawSBSMessage	Malformed lines, encoding
Multi-Receiver Ingestor	(host, port) list	Tagged messages	Connection drops
File Reader	CSV path	Message iterator	Extract receiver from filename

3.2. Sequence Diagram

┌────────┐     ┌────────┐     ┌────────┐
│dump1090│     │Ingestor│     │Consumer│
└───┬────┘     └───┬────┘     └───┬────┘
    │              │              │
    │ "MSG,3,..."  │              │
    │─────────────>│              │
    │              │              │
    │              │ parse()      │
    │              │─────┐        │
    │              │<────┘        │
    │              │              │
    │              │ RawSBSMessage│
    │              │─────────────>│
    │              │ (receiver_id │
    │              │  = serial)   │

3.3. Validation Criteria

[ ] Parse MSG types 1-8 correctly
[ ] Handle empty fields (callsign may be blank)
[ ] Tag each message with device serial
[ ] No exceptions on malformed input (return None)

4. Phase 2: Validation Layer

Goal: Detect bad data without discarding it.

4.1. Tasks

Task	Input	Output	Handle
Callsign Validator	callsign string	(valid, classification, confidence)	"NULL", empty, formats
Position Validator	lat, lon, alt, speed	PositionValidation	Out of range, teleportation
Spoof Detector	position history	(is_spoof, indicators, confidence)	Position jumps, impossible speeds
Mode S Validator	hex_ident	(valid, country, confidence)	Format, country ranges

4.2. Sequence Diagram

┌────────────┐  ┌────────────┐  ┌────────────┐
│RawSBSMessage│ │CallsignVal │ │PositionVal │
└─────┬──────┘  └─────┬──────┘  └─────┬──────┘
      │               │               │
      │ callsign      │               │
      │──────────────>│               │
      │               │               │
      │  (valid,      │               │
      │   "airline",  │               │
      │   0.9)        │               │
      │<──────────────│               │
      │               │               │
      │ lat,lon,alt,speed             │
      │──────────────────────────────>│
      │               │               │
      │  Validation(valid, 0.8, [])   │
      │<──────────────────────────────│

4.3. Validation Criteria

[ ] "NULL" callsign marked as weird_but_valid
[ ] Position > 600 knots implied speed flagged
[ ] Altitude > 60,000 ft flagged (not rejected)
[ ] Teleportation (>50nm in <60sec) detected

5. Phase 3: Multi-Receiver Deduplication

Goal: Merge sightings from multiple receivers for same aircraft.

5.1. Tasks

Task	Input	Output	Handle
Time Window Grouper	message stream	grouped messages	Out of order, clock drift
Aircraft Grouper	message groups	groups by (hex, time)	Same aircraft from N receivers
Field Merger	message list	single MergedSighting	Conflicting values
Conflict Recorder	merged with conflicts	conflict record	Different callsigns

5.2. Merge Priority Rules

┌─────────────────────────────────────────────────────────────┐
│                    FIELD MERGE PRIORITY                     │
├─────────────────────────────────────────────────────────────┤
│  Field          Priority                                    │
│  ─────────────  ─────────────────────────────────────────   │
│  hex_ident      Must match (grouping key)                   │
│  timestamp      Use most recent                             │
│  callsign       Prefer non-empty, most recent               │
│  position       Prefer highest confidence source            │
│  altitude       Prefer geometric over barometric            │
│  receiver_ids   Append all (for coverage tracking)          │
│  source_count   Count of merged messages                    │
└─────────────────────────────────────────────────────────────┘

5.3. Validation Criteria

[ ] Messages within 5 seconds merged
[ ] Messages > 10 seconds apart kept separate
[ ] receiver_ids contains all receivers that saw it
[ ] Conflicting callsigns recorded, not lost

6. Phase 4: Clean Output

Goal: Produce reliable, queryable output.

6.1. Tasks

Task	Input	Output	Handle
CSV Writer	MergedSighting stream	CSV file	Escaping, atomic writes
SQLite Writer	MergedSighting stream	SQLite DB	WAL mode, concurrent reads
Quarantine Manager	failed validations	quarantine table	Preserve original, reason
Stats Collector	pipeline events	metrics	Real-time, periodic summary

6.2. Output Schema

CREATE TABLE sightings (
    id INTEGER PRIMARY KEY,
    timestamp TEXT NOT NULL,
    mode_s_hex TEXT NOT NULL,
    callsign TEXT,
    latitude REAL,
    longitude REAL,
    altitude_feet INTEGER,
    confidence REAL,
    is_potential_spoof INTEGER,
    receiver_ids TEXT,
    source_count INTEGER
);

7. Phase 5: Identity Resolution (Future)

Goal: Link observed data to known aircraft and flights.

Task	Input	Output	Sources
Aircraft Resolver	mode_s_hex	Aircraft record	OpenSky, local cache
Flight Resolver	callsign, timestamp	Flight record	Schedules, patterns
Operator Resolver	callsign prefix	Operator record	ICAO database

8. Phase 6: Enrichment (Future)

Goal: Add metadata from external sources.

Task	Input	Output	Handle
OpenSky Enricher	mode_s_hex	Registration, type	Rate limiting, caching
Route Enricher	origin, destination	Common routes	Unknown routes
Historical Context	sighting	Previous sightings	First-time sightings

9. Success Metrics

Phase	Metric	Target
1	Parse rate	100% of valid SBS lines
2	Validation accuracy	<0.1% false positives
3	Dedup effectiveness	>90% multi-receiver merge
4	Output quality	0 corrupted records
5	Resolution rate	>80% aircraft identified
6	Enrichment coverage	>70% with metadata

10. Test Data Generation

# Capture 60 seconds from each receiver
for receiver in receiver_a receiver_b; do
    timeout 60 nc ${receiver} 30003 > test_${receiver}.csv
done

# Verify SBS format
head -5 test_receiver_a.csv

11. Key Insight: Device Serial as ID

Use USB device serial numbers, not hostnames:

# This allows multiple receivers per host
RECEIVERS = {
    "serial_001": ReceiverConfig(
        device_serial="serial_001",
        host="host_a",
        port=30003,
        name="receiver-1090",
        antenna="1090MHz dipole",
    ),
    "serial_002": ReceiverConfig(
        device_serial="serial_002",
        host="host_a",  # Same host, different device
        port=30004,
        name="receiver-rabbit",
        antenna="rabbit ears",
    ),
}

Get device serial with:

# Linux
rtl_test -t 2>&1 | grep "SN:"

# FreeBSD
usbconfig -d ugenX.Y dump_device_desc | grep iSerialNumber