--- name: alt-data-monitoring description: > Alternative data sources and integration frameworks for investment monitoring. Reference when evaluating non-traditional data signals, building monitoring dashboards with satellite imagery, transaction data, web scraping, NLP sentiment, or geolocation data for investment decisions. metadata: author: nirav version: "1.0" compatibility: Designed for Claude Code --- # Alternative Data Monitoring — The Signal Layer Alternative data is any dataset used for investment insight that is not traditional financial data (prices, earnings, filings). The industry has shifted from "alternative" to "essential" — an estimated 90% of institutional investors now use at least one alternative data source, up from 62% in 2023. Systematic integration of alt data has been associated with an estimated 3% annual return advantage for firms that use it well. The advantage accrues not from the data itself but from extracting ACTIONABLE SIGNALS before the market prices them in. ## The Alt Data Taxonomy ### What Makes Data "Alternative" Traditional data: prices, volumes, financial statements, analyst estimates, economic indicators. Alternative data: everything else that can inform investment decisions. The line blurs as "alternative" sources become mainstream — what was alternative in 2015 (satellite imagery) is table stakes for many hedge funds today. ### The Alt Data Value Chain ``` Raw Data → Collection → Cleaning → Signal Extraction → Integration → Action ``` Each step has failure modes: - **Collection**: Legal compliance, data quality, vendor reliability - **Cleaning**: Missing data, outliers, changing formats, survivorship bias - **Signal Extraction**: Overfitting, spurious correlation, look-ahead bias - **Integration**: How to combine with fundamental analysis without double-counting - **Action**: Position sizing, conviction weighting, signal decay management ## Satellite Imagery The original "alternative data" — seeing the physical world from orbit to infer economic activity. ### Parking Lot Analysis **What it measures**: Retail store traffic by counting cars in parking lots from satellite imagery. **Signal**: More cars = more foot traffic = higher revenue. Compare period-over-period and year-over-year. **Application**: - Track Walmart, Target, Costco store traffic before earnings - Monitor restaurant chains during new product launches - Detect regional demand shifts (new store cannibalizing old store) **Providers**: RS Metrics, Orbital Insight, Planet Labs **Limitations**: - Weather affects parking (snow covers lots, rain changes behavior) - E-commerce shift reduces predictive power over time - Cloud cover prevents satellite capture on some days - Resolution: typically daily to weekly depending on satellite pass frequency ### Oil Storage Monitoring **What it measures**: Crude oil inventory levels by measuring the height of floating-roof storage tanks from satellite shadows. **Signal**: Rising inventories = oversupply = bearish for oil prices. Falling inventories = tightening supply = bullish. **Application**: - Track Strategic Petroleum Reserve levels - Monitor Chinese oil imports via coastal storage - Verify (or contradict) official inventory reports - Detect supply disruptions in real-time **Why it works**: Official inventory data is reported weekly (EIA) or monthly. Satellite data is available within 24-72 hours of capture, giving a multi-day to multi-week information advantage. ### Construction and Development Activity **What it measures**: Physical construction progress — foundation laying, structural framing, project completion timelines. **Application**: - Real estate development tracking: Is the housing project on schedule? - Factory construction: When will a new semiconductor fab come online? - Infrastructure projects: Actual progress vs reported progress - Disaster recovery: How quickly are damaged areas being rebuilt? ### Agricultural Monitoring **What it measures**: Crop health, acreage planted, growing conditions, water availability, deforestation. **Signal types**: - NDVI (Normalized Difference Vegetation Index): Plant health from spectral analysis - Soil moisture levels from radar satellites - Planted acreage vs USDA estimates - Frost damage assessment within days of weather events **Application**: - Commodity trading: corn, soybean, wheat price forecasting - Agribusiness equity research: input demand, yield estimates - Climate risk assessment for agricultural regions **Delivery latency**: Typically 24-72 hours from satellite capture to processed, actionable analysis. Some near-real-time services offer same-day delivery for premium clients. ## Credit Card and Transaction Data Aggregated, anonymized consumer spending data — one of the highest-signal alt data categories. ### How It Works - Payment processors and fintech companies aggregate transaction data - Data is anonymized and aggregated (no individual cardholder data) - Panels of millions of consumers provide statistically representative spending estimates - Revenue estimates available days to weeks before official earnings reports ### Signal Strength Transaction data is one of the strongest predictors of company revenue because: - It measures ACTUAL spending, not surveys or estimates - It's available in near-real-time (1-3 day lag) - It covers both online and in-store purchases - Panel sizes of 5-10 million cardholders provide statistical reliability ### Application Examples - **Retail**: Track same-store sales growth for Target, Walmart, etc. before quarterly earnings - **Restaurants**: Monitor average ticket size and visit frequency for Chipotle, McDonald's - **Subscriptions**: Detect churn rates for Netflix, Spotify, gym memberships - **Travel**: Airline and hotel spending as a real-time economic indicator - **Sector rotation**: Aggregate spending categories to detect consumer behavior shifts ### Providers - **Second Measure** (acquired by Bloomberg): Wide merchant coverage, daily updates - **Bloomberg Transaction Data**: Integrated into Bloomberg Terminal - **Earnest Research**: Strong methodology, institutional focus - **Mastercard SpendingPulse**: Broad coverage, macro-level insights ### Privacy and Compliance Considerations - **GDPR** (EU): Requires explicit consent for personal data processing. Aggregated, anonymized data may be exempt but legal grey areas exist. - **CCPA** (California): Similar consent requirements. "Do Not Sell" provisions affect data collection. - **Panel bias**: Transaction panels skew toward certain demographics, income levels, and geographies. Correct for this or your signal is biased. - **Vendor risk**: If the data provider's collection method is later deemed non-compliant, your signal disappears. ## Web Scraping and Digital Exhaust Harvesting publicly available digital information for investment signals. ### Product Pricing Data **What it tracks**: Real-time pricing changes across e-commerce platforms and competitor websites. **Application**: - Detect pricing power: Is a company raising prices without losing volume? - Monitor competitive dynamics: Who's discounting? Who's holding price? - Inflation tracking: Build a real-time inflation index from millions of online prices (Billion Prices Project methodology) - Input cost monitoring: Track raw material prices from wholesale platforms **Implementation**: Automated scrapers that capture prices daily or hourly. Requires handling dynamic websites, anti-scraping measures, and data normalization across different formats. ### Job Posting Analysis **What it tracks**: Companies' hiring and firing patterns as a leading indicator of growth or contraction. **Signal logic**: - Increasing job postings in engineering/product = company is investing in growth - Decreasing postings across all departments = potential cost-cutting or slowdown - Shifts in job description language = strategic pivots (e.g., suddenly hiring AI/ML engineers) - Geographic patterns = expansion plans, office openings/closings **Lag**: Job postings lead actual hiring by 2-4 months, and hiring leads revenue impact by 3-6 months. This gives a 5-10 month forward view. **Data sources**: Indeed, LinkedIn, Glassdoor, company career pages (scraped directly). ### App Download and Usage Data **What it tracks**: Mobile application download counts, daily active users (DAU), session length, retention. **Application**: - Track growth trajectory for mobile-first companies (Uber, DoorDash, dating apps) - Detect user churn before it shows up in earnings - Compare competitor apps' traction in real-time - Monitor new market entry and geographic expansion **Providers**: Sensor Tower, data.ai (formerly App Annie), Apptopia **Limitations**: Download counts do not equal revenue. Usage data is more valuable than download data. iOS and Android panels may have different coverage quality. ### Website Traffic Analysis **What it tracks**: Unique visitors, page views, time on site, traffic sources for web-based businesses. **Application**: - E-commerce revenue estimation: Traffic x conversion rate x average order value - Lead generation tracking for SaaS companies - Content engagement for media and advertising businesses - Competitive benchmarking: Who's gaining/losing share of web traffic? **Providers**: SimilarWeb, Semrush, Ahrefs (primarily SEO-focused but useful for traffic estimates) ### Google Trends **What it tracks**: Relative search interest for any keyword over time and across geographies. **Application**: - Brand interest tracking: Is consumer awareness growing or declining? - Product demand signals: Search spikes for "buy [product]" or "[product] review" - Macro sentiment: Search volume for "recession," "unemployment," "inflation" - Disease surveillance: Search patterns for symptoms predict outbreaks (Google Flu Trends concept) **Strengths**: Free, real-time, global coverage, long history. **Limitations**: Relative scale (not absolute), seasonal adjustment needed, search behavior changes over time (voice search, AI assistants alter patterns). ## Social Media and NLP Extracting investment signals from unstructured text at scale. ### Twitter/X Financial Community **Signal types**: - Sentiment aggregation: Bullish vs bearish tone across financial Twitter - Breaking news detection: Twitter often surfaces material events before news wires - Retail positioning signals: What are individual investors excited about or scared of? - Influencer tracking: Key accounts that move markets (Elon Musk effect) **Challenges**: Bot contamination, pump-and-dump schemes, signal noise, data access restrictions (API pricing changes). ### Reddit and Retail Investor Signals **Signal types**: - WallStreetBets mention frequency: What stocks are retail investors focused on? - Sentiment scoring: Bullish/bearish/neutral classification of posts and comments - "DD" (due diligence) post analysis: What theses are driving retail conviction? - Short squeeze detection: Identifying heavily shorted stocks with rising retail interest **Application**: As a CONTRARIAN indicator or early warning of retail-driven volatility. Not as a directional signal — retail sentiment is a poor predictor of returns beyond very short time frames. ### Earnings Call NLP **What it analyzes**: Tone, language patterns, and content of quarterly earnings conference calls. **Signal types**: - **Tone analysis**: Positive vs negative word frequency compared to prior calls - **Hedging language**: Increased use of "may," "could," "uncertain" correlates with future disappointments - **Confidence metrics**: Vocal analysis (pitch, pace, pauses) in live calls — experimental but promising - **Topic shifts**: What topics are management emphasizing or avoiding compared to prior quarters? - **Q&A dynamics**: How defensive or forthcoming is management when challenged by analysts? **Research finding**: CEO linguistic tone changes predict future stock returns and earnings surprises. Deception detection models have shown modest but statistically significant predictive power. ### News Sentiment Scoring **What it does**: Automated scoring of news articles by topic, entity, and sentiment. **Approaches**: - Dictionary-based: Count positive/negative words (simple, fast, crude) - Machine learning: Train classifiers on labeled financial news (more accurate, requires training data) - Transformer-based: Use large language models fine-tuned on financial text (most accurate, most expensive) **FinBERT**: A BERT model fine-tuned specifically on financial text. Outperforms general-purpose sentiment models on financial news, earnings calls, and analyst reports. Available open-source. Key capabilities: - Financial sentiment classification (positive/negative/neutral) - Aspect-based sentiment: What ASPECT of the company is the sentiment about? - Domain-aware entity recognition: Distinguishes between Apple (company) and apple (fruit) ### Aggregation and Alpha Decay Social media signals face rapid alpha decay: - When a signal is first discovered: maximum alpha - As more investors use the same signal: alpha decays - When the signal becomes consensus: alpha approaches zero or reverses - Social media signals have among the FASTEST alpha decay of any alt data category — often weeks to months ## Geolocation Data Mobile device location data as a proxy for economic activity. ### Foot Traffic Analysis **What it measures**: Store visits, restaurant traffic, airport throughput, mall activity. **Application**: - Retail revenue estimation from foot traffic counts - Restaurant same-store traffic trends - Mall occupancy and retail health - Airport passenger throughput as travel industry indicator **Providers**: SafeGraph (now part of Dewey), Placer.ai, Foursquare **Key metric**: Visits per location, normalized for seasonal patterns and weather effects. ### Supply Chain Tracking **AIS (Automatic Identification System) data**: - Every commercial vessel broadcasts its position, speed, destination - Track shipping volumes, port congestion, trade flow disruptions - Detect supply chain bottlenecks before official data shows them - Monitor oil tanker movements to estimate crude trade flows **Truck movement data**: - GPS tracking of commercial trucking fleets - Freight volume as a leading indicator of economic activity - Just-in-time supply chain disruption detection - Cross-reference with rail and shipping for multimodal picture ### Real-Time Economic Activity **Mobile phone movement** as a macro indicator: - Aggregate mobility data shows economic reopening/closing in real-time - Commuting patterns indicate return-to-office trends and commercial real estate demand - Consumer mobility to stores, restaurants, entertainment venues - Cross-border movement for tourism and trade estimation **Note**: Post-COVID, this data category has permanently expanded. Investors who tracked mobility data in March 2020 had a significant information advantage in predicting the economic impact timeline. ## Government and Regulatory Data in Real-Time Publicly available data from government sources, processed faster than most investors manage. ### FRED (Federal Reserve Economic Data) **What it is**: Over 800,000 economic data series from the Federal Reserve Bank of St. Louis. **Key series for investors**: - Yield curve data: 10Y-2Y spread (recession indicator) - Inflation expectations: 5Y5Y forward breakeven inflation - Labor market: Initial claims, continuing claims, JOLTS - Financial conditions: Chicago Fed National Financial Conditions Index - Money supply: M2, bank reserves, monetary base - Housing: Case-Shiller, housing starts, mortgage rates **Advantage**: Free API access, updated in real-time as releases occur. Most investors check FRED manually — automated monitoring gives you the data within seconds of release. ### SEC EDGAR **What it is**: All public filings with the Securities and Exchange Commission. **High-value signals**: - **13F filings**: Quarterly holdings reports from institutional investors with >$100M AUM. Shows what smart money is buying and selling (with a 45-day lag). - **Form 4**: Insider transactions (officer/director buys and sells). Insider buying clusters are among the strongest traditional signals. - **8-K filings**: Material event disclosures — earnings, M&A, executive changes. Real-time alert monitoring. - **10-K/10-Q full text**: Textual analysis of risk factor changes, MD&A language shifts, accounting policy modifications. - **SC 13D/13G**: Activist investor position disclosures. Often move stock prices significantly. **Implementation**: EDGAR full-text search API is free. Set up alerts for specific filers, filing types, or keywords. ### Patent Filings **What it tracks**: Innovation at the company level by monitoring patent applications and grants. **Signal logic**: - Patent velocity: Companies increasing patent filings are investing in R&D - Patent quality: Citation-weighted patent scores correlate with future innovation output - Technology overlap: Detect competitive encroachment before product announcements - Talent mapping: Inventor names on patents reveal R&D team movements between companies **Sources**: USPTO PAIR, Google Patents, Lens.org (all free). ## Alt Data Integration Challenges ### Signal Decay **The problem**: As more investors use the same alternative dataset, the informational advantage (alpha) decays. **Lifecycle of an alt data signal**: 1. **Discovery** (few users): Maximum alpha, typically 3-5 years 2. **Early adoption** (dozens of funds): Alpha begins to decay, 2-3 years 3. **Mainstream adoption** (hundreds of funds): Alpha mostly gone, <1 year 4. **Commoditization** (available to all): Alpha is zero or negative (crowded trade) **Implications**: - The most valuable alt data is the data nobody else is using - By the time a dataset is discussed at conferences, its alpha is decaying - Proprietary data collection (building your own scrapers, sensors) retains value longer - The PROCESSING and interpretation of data matters more than access to the raw data ### Overfitting Risk **The problem**: Testing many datasets guarantees some will show spurious correlations with returns. **The numbers**: If you test 100 uncorrelated signals at 95% confidence, you expect 5 to appear significant by pure chance. **Protection**: - Out-of-sample testing: Always reserve data for validation - Multiple hypothesis correction: Apply Bonferroni or Benjamini-Hochberg adjustments - Economic rationale: Only act on signals that have a LOGICAL reason to predict returns - Walk-forward testing: Test on sequential time periods, not random splits - Minimum sample size: Require at least 36 months of data before acting on a signal ### Legal and Compliance Risk **The line between alternative data and insider trading**: Legal (generally): - Satellite imagery of public spaces - Web scraping of public websites (though ToS may restrict) - Aggregated, anonymized transaction data - Government filings and public records - Social media posts Grey area: - Expert network calls (legal if compliant, but several insider trading cases have involved expert networks) - Aggregated employee data (may reveal material non-public information about a company) - Geolocation data with small sample sizes (may identify individuals) Illegal: - Data obtained by bribing insiders - Hacked or stolen data (even if bought from a third party) - Data that reveals material non-public information from a source with a duty to protect it **Rule of thumb**: If a reasonable person would consider the data source "unfair," it probably crosses a line. When in doubt, consult compliance. ### Cost Considerations **Premium alternative data pricing**: | Data Type | Annual Cost Range | Typical Users | |-----------|------------------|---------------| | Satellite imagery (full coverage) | $100K - $500K+ | Hedge funds, commodity traders | | Transaction/credit card data | $200K - $1M+ | Fundamental equity funds | | App usage/download data | $50K - $200K | Tech-focused funds | | Web traffic data | $30K - $100K | E-commerce analysts | | NLP/sentiment platforms | $50K - $300K | Quantitative funds | | Geolocation/foot traffic | $50K - $200K | Real estate, retail funds | | Custom data builds | $100K - $500K+ | Anyone with unique needs | **ROI calculation**: If a dataset costs $200K/year and manages $100M AUM, it needs to generate 0.2% alpha to break even. This is achievable but NOT guaranteed. ## Free and Accessible Alt Data Sources Not all valuable alternative data costs six figures. Many of the highest-signal sources are free. ### FRED (Free) - 800,000+ economic time series - API access with Python, R, Excel integration - Updated in real-time as economic data releases occur - Build automated alerts for yield curve changes, employment surprises, inflation data ### OpenInsider (Free) - Tracks SEC Form 4 insider transactions in a user-friendly format - Filter by: company, insider role, transaction type, date range - Cluster buys (multiple insiders buying within 2 weeks) are the strongest signal - Insider selling is less informative (many legitimate reasons to sell) ### Quiver Quantitative (Freemium) - Government contracts: Which companies are winning federal spending - Congressional trading: Stock trades by members of Congress (disclosed with a lag) - Corporate lobbying expenditure - Insider trading aggregation - Some features free, advanced analytics require subscription ### Google Trends (Free) - Relative search interest for any keyword, any geography, any time period - Compare up to 5 search terms simultaneously - Download data in CSV format for quantitative analysis - Category filtering to reduce noise (e.g., "Finance" category only) - Useful for brand awareness, product demand, macro sentiment ### CFTC Commitments of Traders Reports (Free) - Weekly futures positioning data for commercial hedgers, large speculators, and small speculators - Covers all major commodity, currency, and financial futures - Extreme positioning often precedes reversals - Available every Friday at 3:30 PM ET for positions as of the prior Tuesday - Long history enables regime analysis and percentile rankings ### SEC EDGAR Full-Text Search (Free) - Search across all SEC filings by keyword, company, filing type, date range - XBRL data for structured financial statement extraction - RSS feeds for real-time filing alerts - Build custom monitoring for specific companies, filing types, or keywords - Python libraries (sec-edgar-downloader, edgartools) simplify access ## Building a Monitoring Dashboard A practical framework for integrating free alt data into a monitoring system. ### Minimum Viable Dashboard **Layer 1 — Macro Regime (updated daily)**: - FRED yield curve (10Y-2Y spread): recession probability - FRED financial conditions index: stress indicator - Google Trends "recession" search volume: retail fear gauge - CFTC positioning extremes: crowded trade alerts **Layer 2 — Sector/Industry (updated weekly)**: - Google Trends for key product/brand searches in your portfolio sectors - Job posting trends for portfolio companies and competitors - EDGAR 8-K alerts for material events in portfolio holdings - Insider transaction monitoring via OpenInsider **Layer 3 — Position-Level (updated as available)**: - EDGAR 13F tracking: What are top holders doing with your positions? - Google Trends for company-specific brand interest - App download/usage data for mobile-centric holdings - News sentiment scoring for earnings and event-driven alerts ### Dashboard Architecture ``` Data Sources → Collectors → Database → Signal Engine → Alert System → Review ``` **Collectors**: Python scripts (cron jobs or cloud functions) that pull data from APIs and websites on schedule. **Database**: SQLite for small portfolios, PostgreSQL for larger setups. Store raw data AND processed signals. **Signal Engine**: Rules-based alerts for threshold breaches. Start simple — complex ML models come later. **Alert System**: Email, Slack, or SMS notifications when signals fire. Prioritize by signal strength and portfolio relevance. **Review**: Weekly synthesis of all signals. Monthly deeper analysis of signal performance and false positive rates. ### Signal Quality Assessment Not all alt data signals are created equal. Score each on: | Criterion | Low (1) | Medium (3) | High (5) | |-----------|---------|------------|----------| | Economic rationale | No clear logic | Plausible story | Strong causal mechanism | | Historical correlation | <0.1 | 0.1-0.3 | >0.3 | | Lead time | Same-day or lagging | 1-4 weeks | 1+ months | | Data quality | Frequent gaps/errors | Occasional issues | Consistently clean | | Exclusivity | Everyone uses it | Moderate adoption | Few users | | Cost | >$100K/year | $10K-$100K/year | Free or <$10K/year | **Decision rule**: Prioritize signals scoring >20 out of 30. Signals scoring <12 are likely not worth the integration effort. ### Common Pitfalls 1. **Data snooping**: Testing dozens of alt data sources and reporting only the ones that "work" historically 2. **Ignoring base rates**: Most stocks go up most of the time — your signal needs to beat this base rate 3. **Confusing correlation with causation**: Satellite data of parking lots correlates with revenue because BOTH correlate with economic activity 4. **Overweighting novelty**: New data feels more valuable than it is because of the novelty premium in attention 5. **Neglecting maintenance**: Scrapers break, APIs change, data formats shift — alt data pipelines require ongoing engineering 6. **Ignoring signal decay**: A signal that worked for 3 years may have already been discovered and arbitraged away 7. **Insufficient sample size**: Drawing conclusions from 6 months of data when you need 36+ months for statistical reliability ## Cross-Domain Connections - **Data-science/data-wrangling/feature-engineering**: The alt data value chain (raw → cleaned → signal → integration) IS the feature engineering pipeline. Satellite imagery requires spatial features, NLP requires text embeddings (FinBERT), transaction data requires temporal aggregation — all feature engineering applied to financial signals. - **Data-science/statistical-analysis/statistical-testing**: Signal validation requires hypothesis testing with multiple comparison correction. Testing dozens of alt data signals and reporting only the ones that "work" is textbook p-hacking — the statistical-testing skill provides the correction methods (Bonferroni, BH). - **Data-science/ml-engineering/drift-detection**: Alt data signal decay is drift — a signal that worked for 3 years degrades as it's discovered and arbitraged. Monitoring signal quality over time uses drift detection methodology. ## Related Skills - **sentiment-signals** — Alternative data feeds (credit card, satellite, scrape) often serve as raw inputs for sentiment signal construction. Alt-data-monitoring is one input pipeline; sentiment-signals is one downstream framework that consumes it.