sources[] list pre-vets subreddit relevance, so a separate keyword pass was redundant. Officially retired 2026-05-25.
data/raw/data/raw/ → data/filtered/data/filtered/ → data/extracted/<dir>/<id>.jsondata/authors/<shard>/<author>.jsonl + _features.jsondata/pipeline/<condition>_*data_lineage.mmd. Companion to pipeline.mmd (processing-stage view). Guarded by scripts/check_no_dup_diagrams.py.
View fullscreen ↗
item.id — Reddit's globally-unique post/comment ID. Every persisted layer carries it. The mentions table (stage 6) is the inner join of data/extracted/<dir>/<id>.json with data/filtered/<sub>/*.jsonl on this key.
python ingest/ingest_sub.py --sub <sub> --kind both --after YYYY-MM-DD --before YYYY-MM-DD
Output:data/raw/<sub>/{posts,comments}_<after>_<before>.jsonl
Granularity:One JSONL line per Reddit submission or comment
Idempotency:Skips if output file already exists
Submission record — Arctic Shift returns the full Reddit JSON (~100 fields). We keep them all; downstream uses only ~10. Currently retained per ingest: 99 fields/submission
| Field | Type | Used downstream by |
|---|---|---|
id | string (base36) | universal join key |
author | string | stage 6 credibility, stage 8 distinct_authors, drill-down |
subreddit | string | stage 8 distinct_subs, drill-down |
title | string | stage 4+5 LLM input, drill-down |
selftext | string | stage 4+5 LLM input, body_snippet, length filter |
score | int | stage 2 hard filter |
created_utc | int (epoch) | provenance |
permalink | string | (unused, available for drill-down link) |
num_comments, upvote_ratio, over_18, ... | various | (unused) |
_source, _retrieved_at, _meta | provenance | future multi-source disambiguation |
Comment record — same shape minus title/selftext, with body and parent_id/link_id instead. ~80 fields/comment
data/raw/<sub>/*.jsonl
Command:python ingest/filter_posts.py --sub <sub> + filter_comments.py --sub <sub>
Output:data/filtered/<sub>/{posts,comments}_<after>_<before>.jsonl
Schema:identical to raw — same fields per record, fewer rows
| Filter | Posts | Comments |
|---|---|---|
| Length | 200 ≤ len(title + selftext) ≤ 8000 | len(body) ≥ 75 |
| Score | ≥ 1 | ≥ 1 |
| Author | not deleted, not [removed], not bot | not deleted, not [removed], not bot |
| Keyword gate | (none — assumed in-condition) | at least one treatment-verb keyword ("tried", "helped", "worked for me", "switched to", "dose", etc.) |
| Pass rate (T1D run) | 49–61% (3 subs) | 16–19% (3 subs) |
data/filtered/<sub>/*.jsonl (one item at a time)
Command:python extract/run_extraction.py --sub <sub> --date-range ... --output-name <extracted_dir> --max-cost 5.00 --yes
Output:data/extracted/<extracted_dir>/<item_id>.json — one file per item
Model:claude-haiku-4-5-20251001 (pinned), prompt-cached, async concurrency 5, idempotent skip-if-exists
Cost:~$0.0025/item, ~$3-5 per condition for 1-week multi-source
Extracted JSON schema — top-level fields (one per item):
| Field | Type | Meaning |
|---|---|---|
contains_treatment_info | bool | True if poster (or someone they know) tried a treatment, even if framed as rant/question |
stance | enum | embodied | secondhand | hearsay | question | rant | meta | none |
about_condition | bool | Whether the post is actually about the seed condition |
commercial_intent | enum | none | weak | strong (POSTER's promotion of own product/service, NOT mentions) |
skip_reason | string? | Set when contains_treatment_info=false |
symptoms[] | array | Symptoms the poster reports — independent of treatments |
treatments[] | array | One object per atomic treatment claimed |
symptoms[] sub-schema: { name: string, duration: string }
treatments[] sub-schema — 12 fields per treatment:
| Field | Type | Notes |
|---|---|---|
intervention | string (free text) | Canonicalization is downstream — never the LLM's job. The cluster key. |
category | enum | supplement | medication | diet | exercise | posture | behavioral | device | procedure | other |
form, dose, frequency, timing, duration_tried | strings (free text) | Granularity for drill-down only |
outcome | enum | helped_a_lot | helped_some | no_effect | worsened | mixed | unclear |
outcome_detail | string | Side effects live here, not in symptoms[] |
symptoms_targeted[] | string[] | Subset of post-level symptoms[].name. Drives the symptom-first frontend. |
combined_with[] | string[] | Other interventions in the same protocol — preserves protocol signal across atomic units |
caveats | string | Free text qualifications |
Current state on disk: 102,458 extractions across 26 conditions / 133,190 treatment mentions (one extraction can yield N mentions via the treatments[] array). Per-condition breakdown lives in dev_status.html and corpus.html.
scripts/pull_author_history.py — per-author Arctic Shift pulls, atomic .tmp → .jsonl per author, manifest at data/authors/_manifest.jsonl; circuit breaker (5fail→30min) handles the ~1.2k authors/IP rate cliff
Features:scripts/compute_author_features.py — sqlite3 over data/glean.db (the author_history_items warehouse); full overwrite of data/authors/_features.json
Persisted:data/glean.db author_history_items (raw history; legacy per-author JSONL shards archived to data/authors/legacy_jsonl_archive.zip) + data/authors/_features.json (the composite, keyed by author)
Read by:extract/build_pipeline.py (stage 8 cluster weighting); manifest tracked via shared_inputs._features_json
Six feature families, deterministic via SQL over data/glean.db:
| Feature family | Source | Status |
|---|---|---|
Longevity (account_age_days, log-transformed with diminishing returns past ~2 yrs) | Per-author Arctic Shift API pull | built |
Activity & dispersion (total_posts, distinct_subs, sub_entropy) | Per-author Arctic Shift API pull | built |
Karma signal (median_score, pct_engaged ≥10, pct_at_floor ≤1) — downweight unengaged; do NOT upweight popular | Per-author Arctic Shift API pull | built |
Commercial ratio (commercial_link_ratio + promo_code_pattern_rate regex + aggregated commercial_intent from LLM stage) | Author history domains + extraction stage | built |
| Condition-fit U-curve (1 cluster → 0.50 baseline; 3 tight d=1 clusters → 0.70 max boost; 8 scattered d=2 clusters → 0.20 max penalty) | conditions/<name>_proximity_map.csv + 8×8 cluster distance matrix + author cross-sub posting | built |
Data sufficiency (user_data_sufficiency ∈ [0,1] from total_posts) — sparse authors are downweighted, never filtered | derived from total_posts | built |
Composite score:
Scores land in [0.3, 1.3]. Initial weights guessed; calibration via 40-60 hand-labeled high/medium/low authors is a remaining open question (PLAN.md Gap 3 follow-up). Wired into scripts/run_condition.py steps 4c (pull) + 4d (features) as of 2026-06-01 — every new condition reaches Phase B parity from the orchestrator alone.
Not a numbered pipeline stage — this is the in-memory fact-table assembly inside build_pipeline.py that feeds stages 7-10. Shown here because it's the load-bearing data structure the rest of the pipeline operates on.
extractions{item_id → ext} ⨝ source_items{item_id → src} ON item_id, then flatmap over ext.treatments[] (one mentions row per treatment-with-non-empty-intervention).
build_pipeline.py:208-242
Cardinality:≈ extractions × avg_treatments_per_extraction filtered to contains_treatment_info=true and non-empty intervention
Persisted?No. Lives only in memory during build_pipeline.py.
Mention row schema (18 fields) — this is effectively the wide fact table:
| Field | From | Notes |
|---|---|---|
intervention | extracted.treatments[].intervention | The cluster key (after normalize) |
item_id | both | Join key |
author | source_items[id].author | Only path to author — extracted JSONs do not carry it |
subreddit | source_items[id].subreddit | For distinct_subs, drill-down |
title | source_items[id].title | "" for comments |
body_snippet | source_items[id] selftext or body, truncated 200ch | Drill-down preview |
stance | extracted.stance | |
category | extracted.treatments[].category | |
outcome, outcome_detail | extracted.treatments[] | |
dose, frequency, duration_tried | extracted.treatments[] | |
combined_with[] | extracted.treatments[] | Protocol siblings |
symptoms_targeted[] | extracted.treatments[] | |
author_credibility | credibility{author} | Lookup, defaults 0.5 if author missing |
posts[] arrays in the three <condition>_pipeline_output.json files on author.
build_pipeline.py:cluster_interventions()
Output:in-memory list of {aliases: set, mentions: list, canonical_name: str}
| Step | What happens |
|---|---|
| Normalize | lowercase, strip punctuation |
| Dedupe | build unique-name list (preserves first-seen order for stable canonical naming) |
| Embed | fastembed all unique names in one batch → 384-dim vectors |
| Pairwise sim | full N×N cosine similarity matrix |
| Cluster | greedy agglomerative complete-link: at each step, merge the pair with highest min pairwise sim across members; stop when no pair ≥ 0.75 |
| Canonicalize | each cluster's canonical_name = most-frequent normalized alias |
scipy.cluster.hierarchy.linkage(method='complete') + fcluster. NN-chain gives O(N²) memory and time. Runtime: IBS ~12 sec, Endo ~14 sec, T1D ~21 sec — bit-identical partitions verified against the pre-fix outputs across all three conditions.
build_pipeline.py:248-297
Output:output_clusters[] — the table that lands in pipeline_output.json
Cluster row schema (16 fields) — this is what surfaces in the dashboard:
| Field | Computation |
|---|---|
id | 1-indexed sequential |
canonical_name | Most-frequent normalized alias in cluster |
aliases[] | Sorted list of all normalized intervention strings in cluster |
category | Counter(mention.category).most_common(1) |
mention_count | len(mentions) |
distinct_authors | len(set(mention.author)) |
distinct_subs[] | sorted(set(mention.subreddit)) |
outcomes{} | Counter(mention.outcome) — helped_a_lot, helped_some, no_effect, worsened, mixed, unclear |
positive_ratio | (helped_a_lot + helped_some) / (positive + negative + mixed) |
cluster_weight | sum(mention.author_credibility) — primary ranking signal in the prototype |
cluster_breadth | distinct_authors × len(distinct_subs) |
pubmed_count, evidence_level, novelty_score | Filled in stage 9 (initially null) |
posts[] | List of 13-field drill-down rows: id, author, subreddit, title, body_snippet, stance, outcome, outcome_detail, dose, frequency, duration_tried, combined_with[], symptoms_targeted[] |
pubmed_articles[] | Filled in stage 9 |
posts[] embeds drill-down per cluster: the dashboard is a single static HTML file with all data inline. Embedding the post details in each cluster keeps the surface self-contained — no DB, no API. Trade-off: <condition>_pipeline_output.json is ~2 MB for 1 week of T1D data.
Before clusters land in the persisted output, a post-aggregation filter drops two classes of low-signal clusters that were polluting the top-N novelty surface:
| Filter | Rule | Drops |
|---|---|---|
| Generic catch-alls | Canonical name in GENERIC_BLOCKLIST (14 phrases: "pain meds", "diet change", "various supplements", etc.) or matches ^u\d+ insulin$ (concentration metadata) | ~10–15 clusters/condition |
| Single-mention single-author | distinct_authors == 1 AND mention_count == 1 — one person said it once | ~190–250 clusters/condition |
data/pipeline/<condition>_pipeline_output.json (clusters with null pubmed_count)
Command:python extract/pubmed_enrich.py --condition <name>
Brand→generic step:For category=="medication" clusters, lookup canonical_name via extract/rxnorm.py → NIH RxNorm REST API (cached in data/rxnorm_cache.json) + a hand-curated non-US fallback dict (motilium, novorapid, trurapi, visanne, slynd, slinda)
For each cluster:NCBI eUtils esearch with (brand OR generic_1 OR generic_2 ...) AND (pubmed_terms OR-joined) → esummary for top results
Output:SAME file, in place — adds pubmed_count, evidence_level, novelty_score, pubmed_articles[], and generic_names[] (when expanded) per cluster
Evidence buckets (Gap 1 fix, 2026-04-30):
| pubmed_count | evidence_level | evidence_score |
|---|---|---|
| 0 | none | 0.00 |
| 1–5 | weak | 0.35 |
| 6–20 | moderate | 0.70 |
| 21+ | strong | 0.95 |
Novelty score (the final ship axis):
High novelty = high community signal × low literature coverage. The "what's the crowd seeing that the literature isn't?" surface.
pubmed_articles[] per cluster: { pmid, title, year, journal } — top results stored for drill-down.
data/pipeline/<condition>_pipeline_output.json — the canonical persisted artifact| Top-level field | Type | Source |
|---|---|---|
condition, display_name, description | strings | condition config |
sources[] | array | condition config (subreddit, extracted_dir, filtered_dir, date_range) |
total_extractions | int | count of valid extracted JSONs |
total_with_treatment | int | count where contains_treatment_info=true |
total_mentions | int | len(mentions) |
total_clusters | int | len(clusters) |
clusters[] | array | The cluster rowset (16 fields/row, posts[] embedded), sorted by cluster_weight desc |
evidence_summary | object | Counter of evidence_level (added by stage 9) |
data/extracted/<condition>_summary.md — markdown report (summarize.py)Aggregate stats: stance distribution, commercial intent, top treatments, top symptoms, category/outcome breakdowns. No new joins — pure rollup of extracted JSONs.
data/pipeline/<condition>_results.html — the dashboard (build_ui.py)Self-contained tabbed HTML. Reads pipeline_output.json + summary.md and inlines the data. 7 tabs: Project, Pipeline, Data Flow, Results Table, Results Charts, Browse, Project Files. (The in-page password gate was removed; the Cloudflare deploy bundle injects a soft passcode via gate.js at build time instead.)
To answer "is the same person posting across conditions?" you join the three <condition>_pipeline_output.json files on cluster.posts[].author:
Asymmetry to be aware of: most conditions ingest a single canonical subreddit (r/IBS, r/endometriosis, r/Asthma, …); a few have multi-source sources[] lists (Type 1 Diabetes pulls r/diabetes_t1 + r/Type1Diabetes + r/diabetes; Long COVID pulls multiple). A poster active in a sub the condition doesn't ingest is invisible to this condition's pipeline.
sources[] across conditions. The Stage 6 author-authority track (now built) pulls each author's full cross-sub history via Arctic Shift, so the U-curve and condition-fit signals are not limited by which subs the conditions ingest. This in-pipeline view remains a useful narrow check for "did the same author surface in multiple condition extractions?" — a first cut visible without the per-author history.