Agent Builder
Build investigation agents without prompt engineering
An agent in ManySignal is a typed question set: each question names a signal, describes how to gather it, defines the evidence format, and specifies how it contributes to the final verdict. The AI runs the question set — you define the logic.
Agent question set — YAML
Structure defines behavior, not the prompt
Each question names what the agent should check, where to look for evidence, and how much weight that answer carries. The AI fills in the reasoning; you define the decision framework.
agent:
name: credential-abuse-triage
version: "1.2"
verdict_schema:
type: enum
values: [true_positive, false_positive, needs_investigation]
questions:
- id: intel_match
prompt: "Does the source IP appear in any threat intel feed?"
evidence_source: threat_intel
weight: 0.30
answer_type: bool
- id: geo_anomaly
prompt: "Is this login from a geography outside the user's baseline?"
evidence_source: entity_graph.user.geo_baseline
weight: 0.25
answer_type: bool
- id: prior_frequency
prompt: "How many times has this user triggered this detection in 30 days?"
evidence_source: findings_history
weight: 0.20
answer_type: integer
- id: mfa_outcome
prompt: "Did MFA succeed or fail during this session?"
evidence_source: auth_events
weight: 0.25
answer_type: enum
enum_values: [success, failure, bypass, absent]Agent capabilities you configure
Typed evidence gathering
Each question specifies an evidence source — threat intel, entity graph property, historical findings, raw log query, or external API call via a Universal Action connector. The agent resolves each source at run time and stores the raw evidence against the question answer.
- Evidence stored verbatim alongside the verdict
- Missing evidence flagged explicitly, not silently ignored
- External API calls subject to the same rate-limit and retry controls as workflow actions
Typed evidence gathering
Weighted verdict synthesis
Question weights sum to 1.0. The agent computes a weighted score, applies a configurable verdict threshold, and emits a structured verdict with a confidence score and per-question contribution breakdown. No hidden prompt instructions alter the verdict logic.
- Verdict threshold configurable per alert type
- Per-question contribution visible in the case timeline
- Tie-breaking rules for inconclusive scores
Weighted verdict synthesis
Version control and A/B testing
Agent question sets are versioned objects. You can run two versions in parallel on the same alert stream, compare verdict distribution, false-positive rate, and escalation rate before promoting the new version. Rollback to a prior version is a one-click operation.
- Git-compatible version history
- Side-by-side verdict comparison dashboard
- Promotion gated on configurable accuracy threshold
Version control and A/B testing
Agent Builder — FAQ
Do I need to write prompts to build an agent?
No. The agent question set is a structured YAML definition. The platform constructs the model prompts from the question definitions, evidence payloads, and entity context. You define what to check; the platform handles how to ask the model.
Can agents call external APIs for evidence?
Yes. Any Universal Action connector can be used as an evidence source within a question. The agent calls the connector, receives the structured response, and uses it to answer the question. The raw API response is stored as evidence.
What is the maximum number of questions per agent?
There is no hard limit, but question sets longer than 12 questions begin to exceed the optimal context window for most models. The recommended pattern is to chain two agents — a fast pre-filter agent (3-5 questions) followed by a deep-dive agent for escalated cases.
Can I share agents across tenants in an MSSP setup?
Yes. Agents can be published to a shared library at the MSSP tier and made available to child tenants. Child tenants can inherit and extend a shared agent or override specific questions without forking the entire definition.
How do custom agents differ from the five built-in ManySignal agents?
The five built-in agents (detect, triage, investigate, respond, report) run on every alert automatically. Custom agents are purpose-built for specific scenarios — industry-specific triage protocols, customer-facing investigation reports, or integration with specialist tools — and are triggered by rules or manual invocation.
How is agent output quality evaluated before going to production?
The agent sandbox runs a candidate agent against a set of historical findings with known-good verdicts. You see precision, recall, and a per-finding diff of the candidate verdict versus the reference. Agents that perform worse than a configurable threshold on the test set are blocked from production promotion.
Which AI models can custom agents use?
Custom agents can be configured to use any of the models available on ManySignal's multi-model AI layer: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and locally hosted open-weights models in self-hosted deployments. Model selection is per-agent and version-pinnable.
How does the autonomy ladder apply to custom agent response recommendations?
Custom agent verdicts that recommend response actions are subject to the same autonomy ladder as built-in triage agent verdicts. The action class of each recommended action determines whether it executes autonomously, queues for approval, or surfaces as a recommendation only.
Can custom agents be tested on synthetic data without live telemetry?
Yes. The agent builder includes a synthetic event generator that produces realistic event sequences for common scenarios. Analysts can build and refine custom agents entirely in the sandbox before connecting them to any live data source.
Build your first agent in an afternoon
The agent builder sandbox includes test data and a verdict preview panel — no live environment required.