---
name: benchsemiauto
description: jalanin benchmark aja tanpa generate ke chatgpt — sample.json harus udah ada manual dari BOSS
user-invocable: true
---

## Trigger Command

**BOSS ketik:**
```
/pull-input <capability> [replace]
```

Contoh:
- `/pull-input categorize` → append ke existing sample.json
- `/pull-input categorize replace` → hapus sample.json lama, pakai yang baru

## Workflow /pull-input

1. **BOSS** manual kirim prompt ke chatGPT "GENERATOR INPUT" sendiri
2. **BOSS** ketik `/pull-input <capability> [replace]`
3. **Goblin** jalanin `scripts/pull-input.sh`:
   - Buka ChatGPT → cari chat "GENERATOR INPUT"
   - Ambil message terakhir dari assistant
   - Parse ke format `[{testcase, input_text, expected}]`
   - **Mode replace** → overwrite `sample.json`
   - **Mode append** (default) → gabungin dengan yang lama
4. **Lapor** total cases di `sample.json`

## Format Output (sample.json)

```json
[
  {
    "testcase": "testcase_1",
    "input_text": "...",
    "expected": ""
  },
  {
    "testcase": "testcase_2",
    "input_text": "...",
    "expected": ""
  }
]
```

## Parsing Logic

- **Kalo output ChatGPT valid JSON** → pakai langsung
- **Kalo numbered list** (`1. ...`, `2. ...`) → split per nomor
- **Kalo plain text** → treat sebagai 1 case

## Script Lokasi

```
~/.openclaw/workspace/scripts/pull-input.sh
```

Manual run:
```bash
bash ~/.openclaw/workspace/scripts/pull-input.sh "GENERATOR INPUT" categorize [replace]
```

## Setelah sample.json Ready

```bash
cd ~/CIVIL_GROUP/Projects/API_CIVIL_GROUP
python benchmark_runner.py
```