Db2 on Cloud, IBM Cloud Databases (PostgreSQL/MongoDB/Redis/etc via ibmcloud cdb), Cloud Object Storage (S3-compatible, HMAC, Aspera), Cloudant (CouchDB), Event Streams (Kafka)
Expert-level Apache Kafka, event streaming, Kafka Streams, and distributed messaging
Queues, workers, retries, and safe asynchronous execution models. Use when: async processing, message queue design, worker design, retry strategy, job queue, background…
Flow control and system stability under load through backpressure mechanisms. Use when: backpressure, flow control, system stability under load, producer consumer, channel buffer,…
Architecture-level design review evaluating 15 Tier 2 best practices drawn from DDD (Evans), POEAA (Fowler), Release It! (Nygard), DDIA (Kleppmann), Team Topologies…
Design ETL/ELT pipeline architectures with data flow diagrams and transformation specs for Supabase and BigQuery
Import data into the AWS data lake from S3 files, local uploads, JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora), Amazon Redshift, Snowflake, BigQuery,…
Review Kafka Connect connector configurations for common misconfigurations using the Lenses MCP server.
Analyse Kafka consumer group lag using the Lenses MCP server. Diagnoses lag causes (throughput bottlenecks, rebalancing, partition skew, stalled consumers) and suggests…
Review dead letter queue implementations for completeness using the Lenses MCP server. Checks DLQ topic existence, configuration, monitoring, metadata preservation, retry logic,…
Review Kafka producer and consumer performance configurations in both the live cluster (via Lenses MCP) and the codebase.
Scaffold a production-ready Python Kafka producer and consumer using `confluent-kafka-python`, with Schema Registry, graceful shutdown, idempotent producer, tests and a complete…
Review Kafka schema changes (Avro, Protobuf, JSON Schema) for compatibility and evolution best practices using the Lenses MCP server.
Generate a ShadowTraffic configuration to populate a Kafka topic with realistic synthetic data. Discovers the target topic, its key and value schemas, and the correct serializers…
Generate a TestContainers Java test class that spins up ShadowTraffic in-process to populate a Kafka topic with synthetic data during tests.
Audit all Kafka topic configurations against production best practices using the Lenses MCP server. Checks replication factor, retention, partitions, compaction, naming…
Use when the user asks to "monitor brand mentions", "set up social listening", "did anything spike about us this week", or "watch these accounts for buying triggers"; runs…
Write optimized SQL for your dialect with best practices. Use when translating a natural-language data need into SQL, building a multi-CTE query with joins and aggregations,…
Designs and optimizes Apache Kafka topics and configurations
Understanding the Osprey moderation infrastructure — system architecture, ClickHouse data access, schema reference, and relationship to Ozone labelling.
Use when working on Acuantia's BigQuery Dataform pipeline (acuantia-gcp-dataform project) - adds Acuantia-specific patterns on top of dataform-engineering-fundamentals: ODS…
Add KafkaFlow consumer handlers for processing Kafka/Redpanda messages (project)
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment.
Expert data engineer specializing in building reliable data pipelines, lakehouse architectures, and scalable data infrastructure.
Expert data engineer specializing in building scalable data pipelines, ETL/ELT processes, and data infrastructure.
Evaluate a multi-step AI agent's whole run — tool calls, intermediate steps, and final result — not just final-answer correctness, so you can pinpoint WHERE it went wrong.
Data pipelines, feature stores, and embedding generation for AI/ML systems. Use when building RAG pipelines, ML feature serving, or data transformations.
Python DAG workflow orchestration using Apache Airflow for data pipelines, ETL processes, and scheduled task automation
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment.
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment.
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment.
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment.
Complete guide for Apache Airflow orchestration including DAGs, operators, sensors, XComs, task dependencies, dynamic workflows, and production deployment
Diagnoses Kafka consumer group lag using the Kafka AdminClient API and JMX metrics exposed via the Confluent Metrics API.
Production-grade guidance for Apache Kafka event streaming, partition strategies, consumer group management, transactional messaging, and Confluent Schema Registry integration…
Extracts and transforms Avro, Protobuf, and JSON Schema definitions from Confluent Schema Registry. Generates typed data models and validates schema compatibility using the Schema…
Apache Kafka Stream Processor is built around Apache Kafka event streaming platform. The underlying ecosystem is represented by tulios/kafkajs (3,987+ GitHub stars).
Processes real-time event streams using KafkaJS consumer groups and transforms messages with configurable schemas.
Complete guide for Apache Spark data processing including RDDs, DataFrames, Spark SQL, streaming, MLlib, and production deployment
Strategic guidance for designing modern data platforms, covering storage paradigms (data lake, warehouse, lakehouse), modeling approaches (dimensional, normalized, data vault,…
Constructs Kubernetes-native workflow DAGs using Argo Workflows CRDs with configurable retry strategies, artifact passing via S3/MinIO, and template composition through…
Konstruiert gut strukturierte Argumente mithilfe der Hypothese-Argument-Beispiel- Triade. Behandelt die Formulierung falsifizierbarer Hypothesen, den Aufbau logischer Argumente…
Inspect and debug Airflow on Astronomer (Astro) deployments — fetch DAG runs, task instance logs, container logs, env vars, and deployment state without installing an MCP plugin.
Workflow and best practices for writing Apache Airflow DAGs. Use when the user wants to create a new DAG, write pipeline code, or asks about DAG patterns and conventions.
Use when a user asks to generate or improve unit tests with measurable coverage gains using a strict staged workflow (build -> baseline -> plan -> generate -> verify -> coverage…
Run BFC (cost basis computation) locally using production user data dumped from BigQuery. Use when: (1) profiling BFC memory/performance for a specific user, (2) debugging a…
Generates Python code using BigQuery DataFrames (BigFrames), the pandas/scikit-learn-style API over BigQuery.
Use when asking about BigQuery costs, pricing, bytes billed, slot usage, reducing query costs, choosing between on-demand and editions pricing, managing reservations, optimizing…
Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.
Analyzes species-environment relationships with constrained ordination (CCA, RDA, db-RDA), variance partitioning, indicator species (indicspecies IndVal.g group-equalized),…
Build and modify EdgeSpark apps. Use when a project has edgespark.toml, the user mentions EdgeSpark, or work involves the edgespark CLI, server SDK types, storage/auth/database…
Use when building distributed microservices with Dapr sidecar architecture. Triggers include Dapr components, service invocation, state management, pub/sub, secrets, bindings,…
Use when building event-driven systems with Apache Kafka on Kubernetes. Triggers include EDA patterns, Kafka producers/consumers, Strimzi operator deployment, Schema Registry,…
Change Data Capture (CDC) from a Rails Postgres database via Debezium into Kafka topics — logical replication setup, wal_level=logical, REPLICA IDENTITY FULL, publication /…
Specifies bugs, upgrades, refactors, and behavioral changes as explicit deltas against existing feature specifications, preserving original intent while defining what changes,…
Use when deciding whether a project fits Critical Inquiry (CI) and which format to target. CI is the leading interdisciplinary journal of criticism and theory in the arts and…
Conception de pipelines CI/CD pour tout type de plateforme. Se déclenche avec "CI/CD", "pipeline", "GitHub Actions", "Azure DevOps", "GitLab CI", "déploiement automatique — from…
MUST USE when designing ClickHouse architectures, selecting between ingestion or modeling patterns, or translating best practices into workload-specific system designs.
Use when syncing data FROM relational databases (PostgreSQL, MySQL, MongoDB) TO ClickHouse. Covers change data capture using Debezium, Airbyte, or custom triggers.
Run ClickHouse integration tests in CI with GitHub Actions and Docker containers. Use when setting up automated testing against a real ClickHouse instance, configuring CI…