Anti Hallucination¶

App Passwords Can Disappear — Verify Before Troubleshooting¶

From legacy section: WordPress MCP / sites.json

Pattern: northwaytitle had a password entry in sites.json but wp user application-password list evolvellc returned empty. MCP failed with "not allowed to list users" rather than a clear auth error. Rule: When an MCP site returns 403 for a known admin, check app passwords first: wp user application-password list [username] --path=[wp-path]. If empty, generate a fresh one and update sites.json. Date: 2026-04-22

Sanity guards catch rank hallucinations before publish¶

From legacy section: Competitive Audit Automation (v1.1/v1.2)

Pattern: Even with correct data feeding Claude, the model can produce narratives that contradict the payload (saying "ranks #1 on 28% of pins" when payload says pct_ranking_1 = 0). Publishing these reports damages trust with prospects because the claims are demonstrably false. Rule: Add a sanity-guard function that runs between Claude output and WP publish. Cross-check score bounds (0-70), primary_gap non-trivial length, rank-1 claims match payload within tolerance, invisibility case (pct_not_in_top_20 ≥ 80) MUST be acknowledged in narrative. On any mismatch: block publish, mark audit failed, notify operator with full detail for investigation. Reference: runReportSanityGuard() in orchestrator.js. Date: 2026-04-17

When a "hallucination" looks too specific, check whether it's a real finding placed in the wrong section¶

From legacy section: Competitive Audit Automation (v1.1/v1.2)

Pattern: Howell Electric's pre-fix Medium render had Santa Clara, CA · Electrician · Prepared by Evolve, LLC in the meta subhead, but the business is in Fontana, CA. Reflex read: Claude hallucinated the wrong city. Actual cause: Claude correctly identified that Howell's Google Business Profile is registered at a Santa Clara address while their website targets Fontana — a real, severe SEO problem worth surfacing in the report — but placed that finding in the meta line (which is supposed to reflect the business's headline city/state from business.city + business.state) instead of in the analytical body where it belongs. The post-fix render correctly puts "Fontana, CA" in the meta subhead AND keeps the Santa Clara/Fontana mismatch as the primary_gap analysis, which is what the prompt was always trying to express. Rule: When an output looks wrong on first inspection, before classifying it as hallucination, check whether the value is internally consistent with a real diagnostic the model legitimately found. If yes, the prompt likely under-specified which section that diagnostic belongs in — fix by clarifying the field's purpose ("Use the prospect's actual business.city and business.state from the payload — never substitute a different city") rather than retraining the model away from the finding. Reflex hallucination labels can hide real bugs in the prompt's section taxonomy. Date: 2026-04-25

Three-layer defense for LLM hallucination on customer-facing output¶

From legacy section: SEO NEO / Workbook

Pattern: Light tier audits intermittently fabricated specific stats — Claude wrote "you rank #1 on 23%" for a business with coverage.pct_ranking_1=0, twice in a row with the same exact 23% (likely conflating pct_in_top_3 with pct_ranking_1 under variance pressure). First defense layer was a strict prompt rule ("only quote pct_ranking_1 for #1 claims, never interpolate") which Claude obeyed ~80% of the time. Second layer: scrubFabricatedRankClaims() in the orchestrator runs BEFORE sanity-guard — same regex sanity-guard uses for detection, but replaces the fabricated number with 0% and logs the event. Third layer: existing sanity-guard hard-fails publish if anything still slips through. Rule: For LLM output that ships to paying customers, never trust prompt rules alone — non-determinism means rules are obeyed N% of the time, never 100%. Add deterministic post-processing for known fabrication patterns (scrubber replaces wrong values with payload-correct values), and keep a hard-fail safety net (sanity-guard that blocks publish entirely when scrubber misses). Three-layer architecture: (a) prompt sets expectation, (b) scrubber fixes known classes of error, (c) guard blocks anything left. Each layer logs when it fires so the LLM compliance rate is measurable over time. Date: 2026-04-30

Always verify GBP data via Places API — Perplexity hallucinates place_ids, ratings, and review counts at ~30% rate¶

From legacy section: SEO NEO / Workbook

Perplexity (sonar-pro and the perplexity_ask tool with search_context_size: high) confidently returns fabricated GBP data when it can't directly read Google Maps. On Rock Academy: Perplexity returned place_id: ChIJYbM3K3sRxYkR1w9oO5zJ6zQ (invalid — LD rejected it as "no longer valid"); review count "23" (actual was 8). On YA Skin earlier subagent: 119 reviews (actual 122). On Tali Kogan: Perplexity returned "no GBP exists" (actual: GBP exists at place_id ChIJeX6fs3jTD4gRG_W0mLiFa34, just empty). The miss rate is high enough that any audit number that came from Perplexity needs Places API verification before it ships to a client deliverable. Why: Three out of ten clients in the 2026-05 audit re-run had material data corrections after switching from Perplexity to Places API — wrong place_id, wrong review counts, missed GBPs entirely. The competitor lookups also got better — Perplexity returned "NOT FOUND" for many real Hudson Valley music schools, while Places API surfaced School of Rock Beacon, Community Music Space, etc. with verified review counts. How to apply: 1. For every GBP-relevant data point (rating, review count, place_id, hours, primaryType), use Places API via the agency-ops VPS — the key in .env is GOOGLE_PLACES_API_KEY, but it's IP-restricted to 2.24.192.235. Pattern: ssh evolve@2.24.192.235 "curl -sS -X POST 'https://places.googleapis.com/v1/places:searchText' -H 'X-Goog-Api-Key: ...' -H 'X-Goog-FieldMask: places.id,places.displayName,places.formattedAddress,places.location,places.rating,places.userRatingCount,places.nationalPhoneNumber,places.primaryType' -d '{\"textQuery\": \"...\"}'". 2. Use Perplexity for context only — "what is this business known for", "who are the local competitors by category", color/positioning research. Never for hard numbers Google would know better. 3. For competitor lookups, Places API searchText + locationBias is the gold standard — gives you real Place IDs + review counts in a single call. Don't ask Perplexity to find competitors by name. Date: 2026-05-18