Fine-Tuning Pipeline
apitree automatically collects agent interaction data from MCP tool calls and provides a 6-step pipeline to produce fine-tuning datasets for LLMs.
6-Step Pipeline
- Collect — Every MCP tool call is logged via
POST /v1/agent-logwith session grouping, latency, and success status. - Filter — Interactions are auto-scored 1-5 based on latency, output richness, error patterns, and tool-specific heuristics. Only quality ≥ 4 interactions pass.
- Format — Exported as system/user/assistant JSONL matching Anthropic and OpenAI fine-tuning format.
- Validate — Schema conformity checks + PII scrubbing (Korean RRN, SSN, emails, credit cards, phone numbers).
- Export — Versioned datasets with metadata (quality distribution, tool distribution, avg quality). Download as JSONL.
- Monitor — Track freshness (24h/7d/30d), per-tool readiness, and trained vs. untrained counts.
Quality Scoring
Each interaction starts at score 3 (OK). Bonuses: fast latency (<300ms: +1), rich output (>500 chars: +0.5), multi-result search (+0.5). Penalties: errors (-1), empty output (-1), slow (>5s: -0.5). Tool-specific rules apply for search, call, batch, and details tools.
Human-Edit Ground Truth
Admins can correct low-quality auto-generated outputs via PUT /v1/agent-log/{id}/human-edit. The corrected version becomes the ground-truth label (quality automatically set to 5). Human-edited examples are preferred during dataset builds.
API Endpoints
- •
GET /v1/agent-log/training/stats— Pipeline statistics (admin) - •
POST /v1/agent-log/training/build— Build validated dataset (admin) - •
PUT /v1/agent-log/{id}/human-edit— Submit correction (admin) - •
POST /v1/agent-log/mark-trained— Mark as used for training (admin)
SDK + MCP
- • SDK:
apitree.getTrainingStats(),apitree.buildTrainingDataset(options) - • MCP tool:
get_training_stats— pipeline stats for agents - • Admin page:
/admin/fine-tuning