Glean Health collates Reddit treatment mentions with PubMed literature coverage to help patients with chronic conditions arrive at their appointments better-prepared. We mine condition-specific subreddits, score each poster's credibility from their full cross-Reddit history, cluster the treatments they mention, and surface how much published research exists for each. Not a recommender — a doctor-visit prep tool. The patient picks what to ask about; we save them the hours of Reddit and PubMed searching.
Patients with chronic conditions share what's worked for them on Reddit — both standard-of-care and long-tail options. Finding it requires hours of clicking through threads and cross-checking PubMed. The hypothesis: collating those mentions across credibility-scored patients, alongside PubMed presence per treatment, gives a 60-second version of that research so a patient arrives at their appointment with real questions instead of vague impressions.
Extract blind. The LLM mines Reddit for treatment mentions without seeing PubMed first — patient signal isn't contaminated by literature priming. PubMed is then added as a parallel column (paper count + search link), so the patient sees community endorsement and literature coverage side-by-side. We collate, we don't combine.
Not all posters carry the same weight. We pull each author's full cross-Reddit history — every subreddit they post in, not just the ones in our corpus — and score credibility from their behavior: posting longevity, subreddit diversity, commercial-link ratio, engagement quality. A U-curve catches single-purpose accounts; a commerce signal catches affiliate-driven promoters. Real patients with broad Reddit footprints get higher weight than focused shills.
Arctic Shift — full Reddit archive, 2005-2026, every public subreddit, 2.5B items. We pull per-subreddit dumps from condition-specific communities across 25 chronic conditions, plus per-author cross-Reddit histories for credibility scoring. Reddit API only fills gaps in the live tail.
r/eczema · 2026-02-27 → 2026-03-27. 729 submissions + 6,905 comments.
Hard filter: posts need len 200–8000 and score ≥ 1; comments need len ≥ 75, score ≥ 1, and a treatment-verb keyword. No deleted or bot content.
1924/2490 items (77%) had extractable treatment information, including those framed as rants or questions.
4,570 total mentions clustered into 702 distinct treatments.
Every box in the diagram is clickable. Numbered boxes are pipeline stages; "Arctic Shift" and "Reddit API" describe the data sources; "Condition Config" describes the per-condition manifest that parameterizes stages 8 through 9.5.
Pull raw posts and comments from Arctic Shift API for target subreddits.
Drop noise: deleted content, bots, too short/long, low score. Comments also require treatment-verb keyword match.
Haiku 4.5 reads each item and produces structured JSON: treatment info presence, stance, commercial intent, and extracted treatment details.
Score each author's credibility from their posting behavior. Runs in parallel with the LLM extraction track; only joins back at stage 8.
Current rollout (this condition): 1,392 of 1,397 authors scored on Phase B (99%). The remaining 5 are on Phase A — in most cases because their cached cross-Reddit history is too thin to compute Phase B features (very new accounts, very few items). For the other 26 conditions, the Arctic Shift author-history puller has cached ~86% of authors and is paused (see data/authors/_TRUNCATION_NOTE.md); each condition's Phase B coverage will lift once we re-run build_pipeline.py for it.
Group treatment mentions by semantic similarity, so "L-glutamine", "l glutamine powder", and "Lglutamine" all collapse into one cluster — and so do harder cases like "low-FODMAP" and "FODMAP elimination" that pure string matching would miss.
Combine post-level quality and user-level credibility into a single cluster weight.
unclear outcomes are excluded from the denominator; if no rated outcomes, defaults to 0.5Query PubMed for each treatment cluster to tag evidence coverage level.
For each cluster's top-N PubMed articles (from stage 9), fetch the abstract via NCBI EFETCH and ask Haiku 4.5 to classify the paper as pro (supports the treatment for this condition), con (refutes or warns), or neutral (mention/background/review without stance, or empty abstract). Conservative prompt — defaults to neutral when unsure. Honest sample-of-top-N framing: this classifies the top 5 most-relevance-sorted papers per cluster, not all pubmed_count papers.
pubmed_articles[].pmid from stage 9pubmed_pro_count / pubmed_con_count / pubmed_neutral_count / pubmed_classified_count. Per article: classification + fetched abstract{pro} supportive · {con} against · {neutral} background with tooltip "of the top N most-relevant PubMed papers"Compute novelty_score as an internal QC metric — credibility-weighted Reddit signal scaled down by literature coverage. The patient-facing Browse tab does not use this; it sorts by net-positive credibility-weighted endorsement instead. Novelty is kept for the Results Table sort and the novelty histogram, useful when scanning for high-signal treatments the literature hasn't touched.
Bulk Reddit archive that powers every ingest. Per-subreddit dumps + per-author cross-Reddit history search. Public mirror of Reddit going back to its first commit.
Not yet integrated. Planned for the live tail: Arctic Shift releases monthly and lags ~1-2 months behind real-time, so a production deployment needs Reddit API to close that gap.
conditions/<name>.jsonPer-condition manifest that parameterizes the pipeline. The --condition flag on stages 8 through 9.5 reads this file to drive what gets ingested, how PubMed is queried, and how the U-curve is fit. Onboarding a new condition is a config file + sample ingest — no code change.
sources[] — {subreddit, extracted_dir, filtered_dir, date_range} per source. pubmed_terms[] — condition terms ORed and wrapped in [Title/Abstract]. proximity_map_csv — pointer to the subreddit → cluster table for U-curve scoring.eczemaconditions/eczema_proximity_map.csvStage 3 was specified in early-prototype design as a condition-context keyword filter sitting between hard-filter (2) and LLM extract (4+5). It was never implemented and was formally removed from pipeline.mmd on 2026-05-25.
sources[] list pre-vets subreddit relevance (e.g., r/migraine items are already on-topic by construction). A separate keyword-matching pass would just re-filter what subreddit selection already filtered for free.Eczema — r/eczema — 2026-02-27 → 2026-03-27 — every number below is from this run
/api/posts/search and /api/comments/search, sorted ascending by timestamp. Sliding after parameter for pagination. Rate limit: 0.5s between requests, honors X-RateLimit-Remaining header. Retry with exponential backoff on 429/5xx._source: "arctic_shift" and _retrieved_at: <unix_ts> appended — so we can distinguish data sources when Reddit API is added later.arctic_client.author_submissions() + author_comments() — no subreddit filter. We want the author's entire Reddit footprint to gauge whether they're a real human (broad activity across many subs) or a single-purpose account (concentrated in one or two health subs — the shill / health-anxiety signature).data/authors/_TRUNCATION_NOTE.md).
cluster_weight = Σ(user_credibility) per mention — this is where the parallel track rejoins (Step 8 input)claude-haiku-4-5-20251001) via the Anthropic Python SDK (AsyncAnthropic) — async with a concurrency semaphore and ephemeral prompt caching on the system prompt. Hard per-run cost cap via --max-cost. Realized cost ~$0.0023/item.true.false.contains_treatment_info is false, brief explanation why. Used for calibration.{name, duration} extracted regardless of whether treatments are present.combined_with. Clustering needs atomic units; protocol links preserved.if contains_treatment_info: extract treatments[] with schema fields | else: skip_reason = "why not"combined_with pointing to the other.fastembed BAAI/bge-small-en-v1.5 (a sentence-embedding model), L2-normalize, then compute pairwise cosine distance and run scipy agglomerative complete-linkage clustering (NN-chain implementation, O(N²)). Cut the dendrogram at cosine similarity ≥ 0.75.extract/rxnorm.py, applied at stage 9 (PubMed query expansion).data/authors/_TRUNCATION_NOTE.md). Per-author breakdown lives in pipeline_output.author_credibility_index.unclear outcomes are excluded; defaults to 0.5 if no rated outcomes.
"<treatment_name>" AND (<condition.pubmed_terms> joined with OR) — terms come from each condition's config file.eczema): "<treatment_name>" AND ("eczema" OR "atopic dermatitis" OR "atopic eczema")novelty_score = cluster_weight × (1 - evidence_score)novelty_score as an internal QC metric. The patient-facing Browse tab does not use this — it sorts by net-positive credibility-weighted endorsement so well-studied and under-studied treatments are surfaced together. Novelty is retained for the dev-facing Results Table sort and the novelty histogram, where it's useful for scanning the long tail.novelty_score = cluster_weight × (1 - evidence_score)| # | Treatment | Category | Mentions | Authors | Outcomes | PubMed | Evidence | Novelty |
|---|
How the codebase is organized. Data files (raw, filtered, extracted) are generated by the pipeline and gitignored.
--max-cost). Production prompt in prompt.mdBelow are the things real patients with Eczema have talked about, surfaced from 2,490 Reddit posts. For each one you can see how many people endorsed it, how it worked for them, and how much published research exists. Bring this list to your next appointment as ideas to discuss with your doctor — both well-studied options and ones the literature hasn't caught up to yet are mixed in by how strongly the community endorses them.
Crowd-sourced from Reddit. Not medical advice. Talk to your doctor before trying anything. ← Back to all conditions
Ranked by how strongly the community endorses each — well-studied and under-studied treatments mixed together. Click any card for posts, doses, and PubMed links.