Graphify product signal report
Overview
A quick orientation: what this report covers, how to read it, and where to go next.
- Evidence unit: one distinct problem an LLM extracted from a thread (a thread can yield several). Cluster: evidence units grouped by semantic similarity; the name and description are LLM-written. Only clusters with 2+ unique people are shown.
- Unique people = thread/comment authors plus users who said they are also affected (maintainers and bots excluded). Users (u) never authored a PR in this repo; contributors (c) have. Shown as "N (Uu/Cc)".
- Fix coverage: "k of n threads fixed" counts a cluster's threads that have a deterministic link (PR, commit, release note or maintainer comment) to a fix. Clusters often mix related-but-distinct bugs, so partial coverage is common.
- Status: SHIPPED fixes linked for all threads, no confirmed recurrence · IN_PROGRESS open PR touching the remaining threads · PARTIALLY_ADDRESSED some fix, other threads still open · UNRESOLVED no fix or PR found · STILL_REPORTED_AFTER_FIX similar reports dated after a fix release · UNKNOWN too little evidence. Each cluster has a "Why this status" line.
- Confidence: L1 links are deterministic GitHub references (high; medium when the release is inferred from a merge date). L3 links are an LLM judgement (0 to 1); those below 0.7 are shown but not used for status. LLM judgments are not ground truth.
Where to go
Key findings
Which patterns in user reports are clear enough to state plainly, with the evidence behind each?
These observations are hand-checked against the underlying threads. Each links to its evidence. They describe what users report and what shipped. They make no claim about what Graphify should prioritise.
1. Same-named files merging into one node: fixed in 6+ releases, new variants still arriving
11 threads, 14 people, April → September. Files that share a basename (config.py, index.tsx, Program.cs, layout.py vs layout.ts) collapse into one graph node, which cross-wires edges and creates phantom god-nodes.
Fixes shipped for specific variants in v0.4.2, v0.5.1, v0.8.27, v0.8.51, v0.9.0 and v0.9.11. Four reports are still open:
- #438 (Apr 18)
- #952 (May 20)
- #1829 (Jul 12)
- #3407 (Sep 8, symbol IDs qualified only by the parent directory)
The recurrence check judged a May report "same problem" after the April v0.4.2 fix (confidence 0.75). The pattern is consistent with one underlying ID scheme being patched case by case. The evidence does not show whether that is intended.
2. Incremental and hook rebuilds losing graph data: the most re-fixed problem family
17 threads, 18 people, April → September. graphify update, the post-commit/post-checkout hooks, watch mode and --force runs silently drop or overwrite existing nodes: semantic/doc nodes, unchanged files, hyperedges.
Distinct fixes shipped in at least 12 releases, from v0.5.1 (Apr) to v0.9.53 (Aug). In September, four more variants were reported:
Three of these have open PRs. Two neighbouring clusters, deleted-file ghost nodes and hooks rebuilding wrongly, add up to 35 more people (overlap not removed). Taken together, it is likely the largest bug family in the dataset by people affected.
3. The Codex PreToolUse hook: fixes in 9 releases over 5 months for one integration
11 threads, 12 people. The Codex hook failed in successive ways: unsupported additionalContext, nested systemMessage, permissionDecision, a missing binary, and a no-op hook-check.
Fixes shipped in v0.3.14, v0.3.18, v0.3.21, v0.3.28, v0.4.4, v0.4.10, v0.6.4, v0.9.27 and v0.9.62. After the v0.3.18 fix, a user on 0.4.32 still reported the original failure (confidence 0.80), and #533 (Apr 24) is still open. The report can't say from this evidence whether the churn comes from Graphify or from Codex's changing hook contract.
4. A discoverability gap: how .gitignore, .graphifyignore and graphify-out/ interact
10 threads, 16 people, only 3 of the 10 linked to a fix. Users can't tell whether both ignore files apply together, whether .gitignore still applies when .graphifyignore exists, or what in graphify-out/ should be committed. The neighbouring bug cluster ("ignore rules mismatch gitignore semantics", 29 people) is mostly fixed: 18 of 23 threads. The questions about the rules keep coming. This is a documentation/discoverability signal, not a code defect.
5. Agent-integration demand is the widest signal; engineering activity tracks user signal closely overall
- Most people: "Cannot use Graphify with other coding agents" is the cluster with the most people, 32 (29 users), asking for Kilo Code, Qwen CLI, Windsurf, Hermes, Copilot CLI and more. 7 of 20 requests are linked to a shipped integration.
- Signal vs activity: across all 18 product areas, the share of user signal and the share of engineering activity differ by at most about 1.5×. No area shows high user pain with little visible engineering.
- Where signal runs ahead: CLI/CI workflow (1.5×), docs & discoverability (1.4×), agent integrations (1.3×) and multi-repo workspaces (1.3×).
- Caveat: much of the activity is attributed to areas through the issues it fixed, which pulls these ratios toward 1. Treat them as rough.
6. Docs & discoverability has the largest unresolved share of any area
9 of its 18 multi-person problem groups have no linked fix. No other area has a higher unresolved share. The groups are about users who can't work out:
- how ignore files interact
- what to commit from
graphify-out/ - what a command does
- that a feature already exists (see finding 4)
Code defects in the same areas are mostly fixed. This is a different kind of gap: explanation rather than behaviour. It is also the area with the highest share of threads that got no maintainer comment at all (63%, the highest of any area; the lowest is 36%). Many fixes land as commits without a comment, so read this as "less discussed", not "ignored".
7. When something is fixed, it ships fast
Median of 22 hours from an issue being opened to the release containing its fix (979 fixed issues with a release link):
- 74% ship within 2 days, and 92% within a week.
- The median is similar across areas. Slow fixes are long tails, up to 146 days, concentrated in graph build and dedup.
With near-daily releases, "fixed" in this dataset usually means "fixed and released within days". That is why reports arriving after a fix (findings 1–3) stand out.
8. 👍 reactions point at different demand than head-counts do
Reactions and the number of people reporting a problem are only weakly related (rank correlation 0.25). Only 3 clusters appear in both top-15 lists.
The clearest overlap is agent-integration requests: most people (32) and most 👍 (38).
Several small groups draw outsized silent support:
| Group | 👍 | People who wrote in | |---|---|---| | Custom OpenAI/Anthropic-compatible backends | 23 | 6 | | Terraform/Ansible support (#187) | 19 | 5 | | Unclear token savings | 18 | 5 |
Reactions are noisy, and only 152 of 602 groups have any. They are shown as a second demand signal, not a ranking.
Method notes: statuses combine deterministic GitHub links (88% of closed issues carry a maintainer "Fixed in vX" comment, closing PR or release reference) with Sonnet 5.5 checks of candidate matches and recurrences. "Still reported after a fix" requires a later report, from a different thread dated after the fix release, that an LLM judged the same problem with confidence ≥ 0.7. Lower-confidence cases appear separately as "possible recurrence".
Top problems
What are users currently struggling with most, ranked by how many different people reported it?
Rising problems
Which problems are being reported by noticeably more people lately than before?
Method: unique people (authors + also-affected) in the last 30 days vs the previous 60 days (as a per-30-day rate), measured back from the newest evidence date. Flagged rising when the last 30 days have at least 3 people and at least twice the earlier rate. No forecasting.
Clusters
Themes (groups of related clusters, cut at a looser similarity)
Demand
Do 👍 reactions point to the same problems as the people counts do?
Reactions are an extra, noisy demand signal: one 👍 (issues) or upvote (discussions) on a thread's opening post, summed over a cluster's threads. They favour visible, older or well-shared threads and only some users react, so read them next to the people count, not instead of it. The 👍 column is also in the Top problems table and the All problems explorer.
Top clusters: by people vs by 👍
Left: the 15 clusters with the most unique people. Right: the 15 with the most 👍. A line joins clusters that appear in both.
Themes
What are the bigger topics that many individual problems fall under?
Themes group related clusters (the same clustering cut at a looser similarity). 440 themes; 315 are named (2+ people), the rest are single-person themes labelled with a representative problem. Click a row for its clusters; a cluster name opens it in Top problems. Status bar = share of the theme's clusters by status. Sorted by unique people by default.
All problems
Can I find a specific problem, including ones only a single person reported?
Every extracted problem (2775), including 631 that sit in one-off clusters (a single problem, not repeated anywhere) and so are not in Top problems. Text search covers problem, context and requested solution. "Grouped" = the problem is in a cluster of 2+ problems. Status = status of the problem's cluster. Filters combine; 50 rows per page.
Shipped fixes
Which reported problems have a fix, and which release carried it?
Chain: problem (cluster) → report → PR / commit → release and date. Clusters with status SHIPPED, newest release first. "L1" = deterministic reference; "release inferred" = the first release published after the merge/commit rather than one named in a release note.
Unresolved problems
Which recurring problems have no fix, or only a partial one, so far?
Clusters with status UNRESOLVED, or PARTIALLY_ADDRESSED with fix coverage of 50% or less. Default order: unique people, then number of threads, recency, average severity. This lists evidence only; it says nothing about what should be built.
Reported after fix
Where did people keep reporting a problem after a fix release shipped?
A later report counts only if it is dated after the fix release, is from a different thread, is close in embedding space to the fixed report (cosine ≥ 0.82), and an LLM judged it the same problem or a regression. Possible interpretations, not chosen between: the fix was incomplete; a regression; a documentation / discoverability gap; or a different underlying issue that reads similarly.
Confirmed (LLM confidence ≥ 0.7)
Possible recurrence (0.5–0.69 confidence)
Lower-confidence verdicts, computed in this report only. Not counted in any status. Read as leads, not conclusions.
Areas
Which parts of the product draw the most user signal, and does engineering activity match it?
Areas are approximate LLM-assigned labels and activity counts are proxies (release-note bullets, shipped PRs, issues closed as completed; routed to areas by deterministic links where possible, otherwise by embedding similarity). Ratio = share of user signal divided by share of visible engineering activity; a descriptive comparison only.
Cluster status by area
Question: which areas are mostly fixed and which are not? One bar per area, clusters with 2+ people, sorted by unique people.
User signal vs engineering activity
Question: where does user signal outrun engineering? Each point is an area; the dashed line is parity.
Unique people per month, by area
Question: what is growing? Same y-scale in every panel (0 to people per month), busiest area first. The dashed last segment is the current, partial month.
Area table
Speed
How quickly do maintainers reply and ship fixes, and how does that track the reporting rate?
Time to fix
Question: how long does a fix take? Days from the issue's creation to the release that carries its fix, over threads that have a fix release (threads whose fix predates the report are excluded); the area is the thread's most common evidence area. Dot = median, bar = middle half (25th to 75th percentile). Areas with fewer than 5 fixed threads are hollow. Sorted by median.
Time to first maintainer response
Question: how quickly does a maintainer (safishamsi) first reply? Hours from a thread's creation to the first maintainer comment (issues and discussions; threads opened by a maintainer or bot excluded). Dot = median, bar = middle half, over threads that got a reply; areas with fewer than 5 replied threads are hollow. "No reply" = share of the area's threads with no maintainer comment at all (a maintainer may have closed or fixed it silently). Sorted by median.
Release cadence vs signal over time
Weekly releases and weekly unique people reporting problems, same weeks on both. Two aligned panels rather than a dual-axis chart, because the two counts have very different scales. Weeks start on Monday;
Contributors
Who actually writes the fixes: the reporters themselves, maintainers, or other outside contributors?
Descriptive only. Over threads opened by users or contributors (not maintainers or bots) whose fix is linked (status "fixed"), who authored the change that fixed it: self-fixed = the reporter authored the fixing PR or commit; maintainer = a maintainer authored the fix (and the reporter did not); contributor = another outside person did; unknown = no PR or commit author could be identified (e.g. a release-note-only link). Fixing PRs are those linked as primary or linked and merged or closed; the maintainer usually lands fixes as commits, and where a contributor PR and a maintainer commit are both linked the thread counts as maintainer.
Who fixed the fixed threads, by area
Top 10 fixers among outside contributors
Public GitHub logins, by number of fixed user-reported threads whose linked fix they authored (including their own issues).
PR review bot: is it working?
Graphify reviews its own pull requests with graphify-labs[bot]: a verdict, LLM "advisory findings", inline coupling notes, test selection and a gate. This tab checks how much of that output lands. Bot reviews from 2026-07-28 to 2026-10-08.
Verdict: it shows up reliably, but most of its output doesn't affect what happens to a PR
- Coverage and speed are solved.
graphify-labs[bot]reviewed 1,071 of 1,115 PRs (96%) opened since 2026-07-28, usually within 5 minutes. About a third of the 44 PRs it missed are drafts. - The verdict doesn't predict what ships. Outside PRs whose first review said "Looks safe to merge" shipped 83% of the time. PRs flagged "Worth a look" shipped 84%. PRs with a high-severity finding shipped 80%. Comparing PRs of the same size gives the same result: flagged PRs ship as often as safe ones, or more. Most contributor PRs land as maintainer cherry-picks, so the bot's verdict is not what gates them.
- But the flag does trigger rework, and sometimes a real fix. 42% of flagged outside PRs got another push after the review, against 18% of "safe" ones. The gap holds within every size band (50–299 lines: 32% vs 16%; 300+: 59% vs 43%). In 72 flagged PRs (16%) the author said they fixed a bot finding, 134 fixes in all. Those PRs then ship at the usual rate (22 shipped, 6 closed, 44 still open). So the flag changes the code that lands, not whether it lands. That is where it helps.
- The gate never fails. All 1,862 reviews say
PASS. Every advisory finding is tagged "Escalate … NOT verified … needs human review". Formal verification ran on 37 reviews and proved 0 functions. - LLM findings are useful about half the time. When PR authors replied to the advisory findings (202 responses on 111 PRs), they judged 278 findings valid and 200 wrong (58%). Authors often wrote things like "the bot was right, fixed in …". Many of the wrong ones misread the code: a renamed flag file, a sentinel the code uses on purpose, a function the changed path never calls.
- Inline coupling notes are mostly noise. 99% of the 2,022 inline comments are deterministic "Health regression: fans out to N callees" notes, and 44% repeat an earlier note word for word after each push. Only 11% were resolved, and they got 4 👍 in total. When authors did answer one, it led to a code change 17 times out of 168. Most replies said the shape was deliberate, or that the note pointed at an untouched function or at test code.
- Re-reviews don't converge. 42% of flagged authors pushed again, but only 54 of 454 flagged PRs ever reached "Looks safe". Only 9 of 542 re-review pairs repeated a finding title, so each run mostly reports a new set of findings. Some authors went 4 to 8 rounds before stopping, e.g. #2905 and #3207.
- The maintainer doesn't engage with it. The maintainer's formal reviews fell from 34 a month (June) to 1 in August and none since. The maintainer never replies to or resolves a bot thread. The bot has effectively become the only formal reviewer, while the real merge decision happens through cherry-picks.
Where to improve: drop or collapse the repeated coupling notes, and skip test code. Keep findings stable across pushes so authors can clear them. Make the gate able to fail, or stop showing it. Spend the effort on the LLM findings, the one output authors treat as real review.
Coverage and verdicts, week by week
PRs opened each week (non-bot authors), split by the bot's verdict on its first review of the PR. Weeks start on Monday; the last week is partial.
Does the verdict predict what happens to the PR?
Outside contributors' PRs only. "Shipped" = merged, or closed by the maintainer with a "shipped in / landed in / cherry-pick" note (most contributor PRs land as cherry-picks, so the GitHub merge flag alone undercounts). Ship rate = shipped / (shipped + closed unshipped); still-open PRs are left out. Flagged PRs are bigger, so the same comparison is shown within size bands.
Does the flag cause rework?
Outside contributors' PRs. "Pushed again" = at least one commit dated after the bot's first review (a rebase also counts, for both groups). "Fixed a bot finding" = the author said so in a reply that was labelled valid-and-fixed (a floor: silent fixes are not counted). Compared within size bands, because flagged PRs are bigger and bigger PRs get more pushes anyway.
What happens after a "Worth a look"
From the first flagged review: did the author push again, did the bot re-review, and did the verdict clear? Commits are matched by commit date, so a rebase can count as a push.
Speed and anatomy of a review
Before and after the bot
Context, not cause: other things changed at the same time (PR volume roughly doubled). Maintainer formal reviews per month, and the share of outside PRs shipped within 14 days of opening (PRs younger than 14 days are left out).
What PR authors say about the bot's findings
The best available check on precision: PR authors often answer the bot point by point ("the bot was right, fixed in …", "neither holds up on inspection"). Each response was labelled by an LLM (Claude Sonnet) from the text, then spot-checked.
Verdicts on the findings, by kind of bot output
Inline coupling notes
What the LLM panel flags
Every labelled response
Reviewed PRs
Every PR the bot reviewed, with its first and latest verdict and what happened to the PR. Click a row for the first review's findings.
Method & data
Where does the data come from, how was it processed, and what are the limits?
Data
The underlying data (derived fields only, no raw GitHub bodies) is in share/ (SQLite, CSV and JSON bundle) with the column-by-column description in docs/DATA_DICTIONARY.md. Both paths are relative to this file's location in the repo (output/), so they only work while the report sits alongside the repo; they are not embedded in this page.
Methodology
Data window. The sync fetches everything updated since 2026-04-01. The repository was created after that date, so this covers its full public history. User reports in the data run from 2026-04-06 to 2026-09-29. GitHub data was pulled on 2026-10-08.
Data flow
- Deterministic: GitHub API sync of issues, discussions, PRs, commits, releases and comments into SQLite; reference parsing (closes/fixes, closing-issue references, "fixed in" maintainer comments, commit references, release-note bullets); noise filtering; L1 fix resolution per thread; unique-user counting, coverage, trend and status rules.
- Embeddings (local model Qwen3-Embedding-0.6B-Q8_0): clustering (run t0.20, average-linkage cosine distance 0.20), themes (looser cut 0.27), L2 candidate matches, area activity routing.
- LLM (Claude Sonnet 5.5): evidence extraction (extract-v3), area taxonomy, cluster/theme naming, L3 verification of candidate fixes, recurrence checks. The only LLM prose shown is the cluster name / description / distinction and the short reasons attached to L3 and recurrence verdicts.
Thresholds
- L3 verdicts count toward status only at confidence ≥ 0.7; recurrence candidates need cosine ≥ 0.82; confirmed recurrence needs LLM confidence ≥ 0.7; the "possible" tier uses 0.5–0.69.
- An open PR counts as IN_PROGRESS only if updated in the last 30 days. Rising = last-30-day people ≥ 3 and ≥ 2x the previous 60-day per-30-day rate.
- Status counts (clusters with 2+ people): PARTIALLY_ADDRESSED 229, STILL_REPORTED_AFTER_FIX 17, IN_PROGRESS 112, UNRESOLVED 91, SHIPPED 151, UNKNOWN 2.
LLM calls and cost (from the local call log)
| Stage | Model | Calls | Input tok | Output tok | USD |
|---|---|---|---|---|---|
| extract-v2 | claude-haiku-4-5 | 39 | 150109 | 20747 | 0.30 |
| extract-v2 | claude-sonnet-5-5 | 39 | 170714 | 25107 | 0.67 |
| extract-v3 | claude-sonnet-5-5 | 1846 | 8571067 | 884378 | 28.05 |
| name-v1 | claude-sonnet-5-5 | 917 | 2667528 | 327181 | 10.75 |
| recur-v1 | claude-sonnet-5-5 | 519 | 1311947 | 96171 | 4.33 |
| verify-v1 | claude-sonnet-5-5 | 1358 | 3691912 | 255630 | 13.01 |
| Total | 4718 | 16563277 | 1609214 | 57.10 |
extract-v2 rows are earlier calibration runs; the report uses extract-v3 evidence from claude-sonnet-5-5 only.
Limitations
- Discord and other community chat are excluded by design; this is GitHub only.
- Discussion comments are not parsed for fix links, so discussion threads rarely show a fix and read as "no fix signal".
- Area labels are noisy; clusters can mix related-but-distinct problems (a warning icon marks clusters the naming step flagged as incoherent), which is why coverage is shown next to status.
- LLM judgments (extraction, naming, L3 verification, recurrence) are not ground truth. Unique-people counts depend on extraction of "also affected" mentions.
- Releases marked "inferred" are the first release after the merge or commit, not confirmed.
- Many verified matches are "partially addresses": clusters usually contain several related bugs.