--- name: explore-datasources description: Discover what datasources, metrics, labels, and log streams are available in a Grafana instance. Use when the user asks what data exists, what metrics are available, what services are being monitored, or needs to find a datasource UID. --- # Datasource Discovery > If gcx is not configured, see the setup-gcx skill first. ## Instructions ### Step 1: List Available Datasources Start by identifying all datasources in the Grafana instance. ```bash # List all datasources gcx datasources list # Filter by type if you know what you need gcx datasources list --type prometheus gcx datasources list --type loki ``` **Expected output:** Table showing UID, NAME, TYPE, URL, and DEFAULT columns. **Important:** Always use the UID (not the name) in subsequent commands. ### Step 2: Explore Datasource Contents Choose the appropriate exploration path based on datasource type. #### For Prometheus Datasources ```bash # List all available labels gcx metrics labels -d # Get values for a specific label to understand what's being monitored gcx metrics labels -d --label job # List all available metrics with descriptions gcx metrics metadata -d # Check what systems are being scraped gcx metrics query -d 'up' ``` **Expected output:** Tables showing labels, metrics, or query results depending on command. #### For Loki Datasources ```bash # List all available labels gcx logs labels -d # Get values for a specific label gcx logs labels -d --label job # List log streams matching a selector (required) gcx logs series -d -M '{job="varlogs"}' ``` **Expected output:** Tables showing labels or log stream series. **Note:** The `series` command requires at least one `-M` (match) selector using LogQL syntax. ### Step 3: Test Queries (Optional) Once you've identified available data, verify with a test query. ```bash # For Prometheus - instant query (current value) gcx metrics query -d 'up' # For Prometheus - instant query at a specific point in time gcx metrics query -d 'rate(http_requests_total[5m])' --time 2026-05-14T12:00:00Z # For Prometheus - range query (time series over a window) gcx metrics query -d 'rate(http_requests_total[5m])' --from now-1h --to now ``` **Choosing instant vs range:** Use `--time` for a value *at* a moment ("what is the current rate?", "what was the rate an hour ago?"). Use `--from`/`--to` for anything *across* a window — peak detection, totals, trends, or ranking over time. For point-in-time comparisons, run two `--time` queries rather than a range. If the PromQL uses `increase()`, `max_over_time()`, or similar over a long window (hours/days), always use `--from`/`--to` — the range lets you verify the full series and aggregate with `sum by (label)`, not `count ... > 0` which loses the magnitude. **Expected output:** Table showing metric values with labels and timestamps. ### Step 4: Set Default Datasource (Optional) To avoid passing `-d ` repeatedly, configure defaults: ```bash # Set default Prometheus datasource gcx config set contexts..default-prometheus-datasource # Set default Loki datasource gcx config set contexts..default-loki-datasource ``` After setting defaults, you can omit the `-d` flag in datasource commands. ## Examples ### Example 1: Finding HTTP metrics **User says:** "What HTTP metrics are available?" **Actions:** 1. List Prometheus datasources: `gcx datasources list --type prometheus` 2. Get datasource UID from output 3. Search for HTTP metrics: `gcx metrics metadata -d -o json 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin)['data']; [print(k,d[k][0]['type'],d[k][0]['help']) for k in sorted(d) if 'http' in k]"` 4. Get details on specific metric: `gcx metrics metadata -d --metric http_requests_total` **Result:** Metric name, type (counter/gauge), and help text showing what the metric measures. ### Example 2: Discovering which services are logging to Loki **User says:** "What applications are sending logs to Loki?" **Actions:** 1. List Loki datasources: `gcx datasources list --type loki` 2. Get datasource UID from output 3. List label names: `gcx logs labels -d ` 4. Get job values: `gcx logs labels -d --label job` 5. List streams for a specific job: `gcx logs series -d -M '{job="varlogs"}'` **Result:** List of job names and their associated log streams. ### Example 3: Troubleshooting missing dashboard data **User says:** "My dashboard shows no data for service X" **Actions:** 1. Verify datasource exists: `gcx datasources get ` 2. Check if service is being monitored: - Prometheus: `gcx metrics query -d 'up{job="service-x"}'` - Look for service in scrape results 3. Verify labels exist: `gcx metrics labels -d --label job` 4. Test simple query: `gcx metrics query -d 'up{job="service-x"}'` **Result:** Identifies whether datasource is misconfigured, service isn't being scraped, or label selectors are wrong. ### Example 4: Finding logs for a specific namespace **User says:** "Show me all log streams from the production namespace" **Actions:** 1. Get Loki datasource UID: `gcx datasources list --type loki` 2. Verify namespace label exists: `gcx logs labels -d --label namespace` 3. List all streams in namespace: `gcx logs series -d -M '{namespace="production"}'` **Result:** Table showing all label combinations for log streams in the production namespace. ## Troubleshooting ### Error: "datasource UID is required" **Cause:** The `-d` flag was omitted and no default datasource is configured. **Solution:** ```bash # Option 1: Pass the UID explicitly gcx metrics labels -d # Option 2: Set a default datasource gcx config set contexts..default-prometheus-datasource ``` ### Error: "at least one --match selector is required" **Cause:** The `loki series` command was called without a `-M` flag. **Solution:** Loki series requires at least one LogQL selector: ```bash # Correct gcx logs series -d -M '{job="varlogs"}' # Wrong - will fail gcx logs series -d ``` ### Error: "parse error on line 1, column X: bare \" in non-quoted-field" **Cause:** Shell is interpreting quotes in the LogQL selector incorrectly. **Solution:** Use single quotes around the entire selector: ```bash # Correct - single quotes outside gcx logs series -d -M '{name="value", cluster="prod"}' # Wrong - shell interprets quotes incorrectly gcx logs series -d -M {name="value"} ``` ### Error: "datasource.prometheus.datasource.grafana.app \"\" not found" **Cause:** The datasource UID doesn't exist or you don't have access to it. **Solution:** 1. List datasources to verify UID: `gcx datasources list` 2. Check you're using the correct context: `gcx config current-context` 3. Verify datasource exists: `gcx datasources get ` ### No output from labels/series commands **Cause:** Datasource has no data or hasn't scraped/ingested anything yet. **Solution:** 1. For Prometheus: Check targets are active: `gcx metrics query -d 'up'` 2. For Loki: Verify labels exist: `gcx logs labels -d ` 3. Check datasource URL is reachable: `gcx datasources get ` ## Advanced Usage For detailed patterns, LogQL syntax guide, and advanced discovery workflows, see: - [`references/discovery-patterns.md`](references/discovery-patterns.md) - Common discovery patterns and workflows - [`references/logql-syntax.md`](references/logql-syntax.md) - LogQL selector syntax guide ## Output Formats All commands support `-o json` or `-o yaml` for programmatic use: ```bash # Get JSON output for piping to jq gcx metrics labels -d -o json # Example: Count total metrics gcx metrics metadata -d -o json 2>/dev/null | python3 -c "import json,sys; print(len(json.load(sys.stdin)['data']))" ``` Default output is `table` format for human readability.