Fine-Tuning Pipeline

apitree automatically collects agent interaction data from MCP tool calls and provides a 6-step pipeline to produce fine-tuning datasets for LLMs.

6-Step Pipeline

1. Collect — Every MCP tool call is logged via POST /v1/agent-log with session grouping, latency, and success status.
2. Filter — Interactions are auto-scored 1-5 based on latency, output richness, error patterns, and tool-specific heuristics. Only quality ≥ 4 interactions pass.
3. Format — Exported as system/user/assistant JSONL matching Anthropic and OpenAI fine-tuning format.
4. Validate — Schema conformity checks + PII scrubbing (Korean RRN, SSN, emails, credit cards, phone numbers).
5. Export — Versioned datasets with metadata (quality distribution, tool distribution, avg quality). Download as JSONL.
6. Monitor — Track freshness (24h/7d/30d), per-tool readiness, and trained vs. untrained counts.

Quality Scoring

Each interaction starts at score 3 (OK). Bonuses: fast latency (<300ms: +1), rich output (>500 chars: +0.5), multi-result search (+0.5). Penalties: errors (-1), empty output (-1), slow (>5s: -0.5). Tool-specific rules apply for search, call, batch, and details tools.

Human-Edit Ground Truth

Admins can correct low-quality auto-generated outputs via PUT /v1/agent-log/{id}/human-edit. The corrected version becomes the ground-truth label (quality automatically set to 5). Human-edited examples are preferred during dataset builds.

API Endpoints

• • GET /v1/agent-log/training/stats — Pipeline statistics (admin)
• • POST /v1/agent-log/training/build — Build validated dataset (admin)
• • PUT /v1/agent-log/{id}/human-edit — Submit correction (admin)
• • POST /v1/agent-log/mark-trained — Mark as used for training (admin)

SDK + MCP

• • SDK: apitree.getTrainingStats(), apitree.buildTrainingDataset(options)
• • MCP tool: get_training_stats — pipeline stats for agents
• • Admin page: /admin/fine-tuning