Methodology
sfpulse fetches public data from SF government sources on a fixed schedule, caches the results locally, and renders them as-is. Nothing is summarized, reworded, or inferred. The text you read came directly from the source.
How data gets to you
A set of Python scripts runs in the background, each targeting a single data source. Each script accepts a district number and a lookback window (in days), queries the source, filters the results to the district, and returns structured JSON. The Go server stores that JSON in SQLite and serves it to the page. When cached data is older than its refresh threshold, the script reruns in the background while the old data stays visible.
No data transformation happens between script output and what you see. If a permit shows "HIGH" tier, that label came directly from the source data. If a 311 category is flagged as a spike, that flag was computed by the script comparing the current week to the prior week using the same Socrata dataset the city publishes.
Sources and refresh schedule
| Section | Source | How | Refreshed every |
|---|---|---|---|
| Building permits | SF Open Data i98e-djp9 | Socrata API, filtered by supervisor district | 2 hours |
| Housing pipeline | SF Planning qvu5-m3a2 & acdm-wktn | Socrata API, filtered by district | 4 hours |
| 311 activity | SF Open Data vw6y-z8j6 | Socrata API; compares current 7-day window to prior 7-day window per category to detect spikes | 1 hour |
| Eviction notices | SF Open Data 5cei-gny5 | Socrata API, filtered by district; Ellis Act and owner move-in flagged separately | 6 hours |
| Board of Supervisors | Legistar RSS | RSS feed parsed for upcoming and recent meetings; items filtered to district supervisor | 2 hours |
| Planning Commission | sfplanning.org | HTML scrape of published hearing calendars; PDFs parsed for item addresses cross-referenced to district streets | 3 hours |
| SFMTA engineering hearings | sfmta.com | HTML scrape; items filtered by district street list | 1 hour |
| Board of Appeals | SF.gov Wagtail API | REST/JSON endpoint | 2 hours |
| Ethics & lobbying | SF Ethics Commission | Lobbyist contact filings filtered to supervisor; campaign contribution data via Socrata & RSS | 4 hours |
| Local news | Mission Local, SF Standard, Streetsblog SF, SF YIMBY, 48 Hills, Indybay (RSS) | RSS feeds deduplicated by title similarity; articles tagged to district by street/neighborhood name matching | 30 minutes |
| SFMTA Board | sfmta.com | HTML scrape of meeting calendar | 6 hours |
| Rent Board | SF Rent Board (SF.gov) | Wagtail REST/JSON endpoint | 12 hours |
| Rec & Park | SF Rec & Park (Granicus) | REST/JSON endpoint | 4 hours |
| SFUSD Board | SFUSD Board (BoardDocs) | BoardDocs API + public Google Sheets calendar | 12 hours |
| BART Board | BART Legistar API | REST/JSON; BART advisory bodies included | 12 hours |
| SFCTA | sfcta.org | HTML scrape of meeting calendar | 12 hours |
| Civic actions | mobilize.us + Indybay RSS | mobilize.us event search for SF; Indybay RSS filtered for civic events | 30 minutes |
| Volunteer cleanups | Refuse Refuse SF + SF DPW | HTML scrape + DPW volunteer calendar | 4 hours |
District assignment
Each supervisorial district is pre-configured with a list of neighborhoods and streets. Scripts that query address-level data (permits, evictions, planning) filter by supervisor district number directly in the API query where the dataset supports it. Scripts that scrape text (planning PDFs, news articles) match against the district's street and neighborhood lists.
The 11 district boundaries follow the official supervisorial district map. Neighborhood-to-district assignments are manually curated and updated when the Board redraws district lines.
Tier classification
Permits are classified HIGH, MEDIUM, or LOW by the building permit script based on permit type and estimated cost. HIGH covers major new construction, large additions, and demolitions. MEDIUM covers significant alterations. LOW covers minor work, mechanical permits, and routine maintenance. These thresholds are defined in the script and applied to the raw permit data. They are not adjusted per district or per time period.
The "This week" digest on each district page counts items in each section and flags anything above these thresholds. The counts reflect exactly what is in the cached data at render time.
Limitations
- Planning Commission agendas are sometimes posted late; items can appear within 72 hours of the hearing.
- Legistar RSS covers approximately 3 weeks of Board activity; older votes are not shown.
- SFMTA engineering hearings and Rec & Park Commission sections may be empty between hearing cycles. This is expected, not a data error.
- News district tagging is based on street and neighborhood name matching; articles that discuss a district without naming a specific street may not appear.
- All data is public and unauthenticated. Socrata rate limits apply; requests use a shared app token when available.