---
name: large-data-storage-planning
title: Large-scale data storage strategy with cost, free-at-scale levers, and preprocessing
description: Class-level workflow for minimizing burn on PB-scale storage while preserving reliability and processing throughput.
version: 1.0.0
author: Hermes
license: MIT
---

# Large-scale data storage strategy with cost, free-at-scale levers, and preprocessing

## Scope
Use this skill whenever the user asks for multi-PB/EB-class storage strategy, cheapest-at-scale architecture, or ways to reduce storage burn through preprocessing/credits.

## Core rule
At 12 PB order, **free tiers are not a strategy** — they are a bridge.
If a requirement implies “for free at scale,” only realistic paths are:
1. Make eligible data public and apply for open-data sponsorship/public hosting routes,
2. Stack startup credits/partner grants for storage and processing,
3. Reduce bytes before persistence (dedupe/compression/format transforms).

## Decision framework (always use)
1. **Classify first** (mandatory before pricing)
   - Public vs private data.
   - Hot/active vs cold/archival.
2. **Compute TCO, not headline rate**
   - Storage rate ($/TB-mo)
   - retrieval + egress penalties
   - API/transaction costs
   - minimums and hidden floor charges
   - durability / replication assumptions
3. **Prefer cheapest private archival for the immutable bulk**, with a small hot/cache slice for active work.
4. **Only keep “free” as a primary lever if it changes runway** (credits/sponsorship consuming spend where possible).

## Baseline architecture pattern
- **Layer 1: Cold private archive**
  - Archive class or equivalent with strict lifecycle policies.
  - If ops capacity is high and risk tolerance allows, tape-class/hybrid patterns may beat cloud archive.
- **Layer 2: Hot/cook layer**
  - Small, high-velocity object storage for preprocessing and short-term compute windows.
- **Layer 3: Public route (if eligible)**
  - Move open datasets into sponsorship/public channels to reduce private paid footprint.

## “Cheapest realistic” sequence
1. Run **classify + preprocess** pass first.
2. Submit sponsorship/public-data options for eligible slices.
3. Apply all feasible startup credit/grant programs in parallel.
4. Finalize private cold tier only for restricted or sensitive data.

## Proven execution template
1. Slice dataset by sensitivity and access pattern.
2. Estimate current monthly footprint for hot + cold.
3. Run credits/sponsorship timeline (who, what, deadlines, docs).
4. Build cost model: `monthly_cost = TB * rate + egress + retrieval + ops overhead`.
5. Pilot one shard through chosen architecture for one-week burn validation.
6. Expand only after retrieval/reduce loop confirms expected cost and latency.

## Processing lever (often bigger than provider choice)
- Parse/clean before cold write.
- Deduplicate aggressively via content hash.
- Transcode to compressed archival-friendly formats.
- Drop temporary artifacts and keep only canonical artifacts.
- Enforce immutable manifests and checksums.

## Frequent mistakes (pitfalls)
- **Using free tiers as scale strategy** (10–25 GB quotes are noise at PB scale).
- **Ignoring retrieval economics** (egress or rehydrate can erase apparent savings).
- **Overbuilding hot layer** and carrying unnecessary active storage for months.
- **No credits-first execution plan**, especially in startups/AI startup programs.
- **Assuming managed object prices are all that matter** and skipping region, minimum duration, and API charges.

## Verification workflow
For each candidate stack, capture and archive:
- provider pricing excerpts (storage + retrieval + egress + minimums),
- credit program terms and cap amounts,
- 12 PB total monthly burn and credits runway in a one-page comparison.
- Decision rationale: why this option is “lowest-risk + lowest-burn” for this dataset profile.

## Maintenance
- Keep all numeric evidence in `references/` files.
- If new provider classes or credit programs appear, append them to:
  - `references/provider-pricing-notes.md`
  - `references/credit-programs.md`

## References
- Start with `references/provider-pricing-notes.md` for session-specific evidence and rate sanity checks.