Log Management & Retention
Retain what compliance requires. Query everything, fast.
ManySignal's three-tier log store gives you sub-second search on recent data, cost-efficient compressed storage for older events, and on-demand columnar queries for forensic investigation — with per-source retention policies you declare in YAML.
Retention policy — YAML
Retention policy as code, not a UI setting you forget
Retention policies live in your detection-as-code repository alongside detection rules. A policy change goes through the same review and deploy process as a rule change — not an ad-hoc UI click that leaves no audit trail.
retention:
defaults:
hot_days: 90 # indexed, sub-second search
warm_days: 365 # compressed parquet, queryable on demand
cold_days: 2555 # 7 years total; archived, query cost applies
overrides:
- source: aws.cloudtrail
hot_days: 90
warm_days: 730 # 2 years hot+warm for compliance
cold_days: 2555
- source: crowdstrike.endpoint
hot_days: 30 # high volume — shorter hot window
warm_days: 365
cold_days: 1825 # 5 years total
- source: github.audit_log
hot_days: 90
warm_days: 365
cold_days: 2555
pii_fields: # redact before warm transition
- actor.email
- actor.ip
- source: pagerduty.incidents
hot_days: 180
warm_days: 365
cold_days: 0 # delete after 18 monthsThree tiers, one query surface
Hot tier: indexed for speed
Events in the hot tier are indexed in ClickHouse with full field-level indexing. Any field on any normalized event is searchable in sub-second latency. Hot-tier queries support aggregations, time-series analysis, and join-style lookups against the entity graph.
- Default 90-day window, configurable per source
- Sub-second p99 latency on most query patterns
- Full-text search on unstructured message fields
Hot tier: indexed for speed
Warm tier: compressed for cost
Events older than the hot window are converted to Parquet format and written to object storage. Warm-tier queries run as columnar scans — no per-row indexing overhead. Queries on the warm tier take seconds to minutes depending on data volume, and are billed as compute cost, not storage cost.
- 10-20x compression vs. raw JSON storage
- On-demand query — no warm-up, no pre-partitioning required
- Results stream incrementally for large time-range queries
Warm tier: compressed for cost
Cold tier: archived for compliance
Data beyond the warm window is re-compressed with higher-ratio algorithms and moved to a cold-tier object store (Glacier-compatible). Cold-tier queries require a restore step that typically takes 2-8 hours. Cold data can be selectively restored to warm tier for investigation without restoring the entire archive.
- Per-record restore: pull only the events you need
- Cold tier query billed separately — no surprise costs for infrequent access
- Cold data deletion at configurable policy expiry
Cold tier: archived for compliance
Log Management — FAQ
Can I query across hot and warm tiers in a single search?
Yes. The natural language search interface and the query API transparently query both tiers. Warm-tier results stream in after hot-tier results complete. You see a progress indicator and partial results rather than waiting for the full warm scan.
What formats are supported for log ingestion?
Normalized JSON (the standard ManySignal event schema), raw JSON (parsed by vendor-specific parsers), CEF, LEEF, and syslog RFC5424. Raw syslog with custom parsers is supported via the connector SDK field mapping configuration.
How are retention policies applied to existing data?
Retention policy changes apply to new data immediately. Existing data is governed by the policy that was in effect when it was written. You can explicitly backfill a policy change to apply it retroactively — with a confirmation prompt showing the volume of records affected.
Is there a legal hold mechanism?
Yes. Individual cases or time windows can be placed under legal hold, which pauses the retention policy for the affected data regardless of what the policy would otherwise do. Legal holds are logged to the audit trail and require an administrator to release.
How does ManySignal avoid the per-GB pricing model that makes SIEM costs unpredictable?
ManySignal is priced by protected assets and autonomy tier, not by log volume. You can ingest full-fidelity telemetry from all sources without financial penalty for higher event rates. Hot/warm/cold tiering manages storage costs automatically based on access frequency, not billing.
What is the query latency for hot-tier versus warm-tier searches?
Hot-tier queries (data up to 30 days old) typically return results in under 5 seconds for most query patterns. Warm-tier queries (31–90 days) may take 10–30 seconds for large result sets. Cold-tier (archived) queries are asynchronous and typically complete in 2–10 minutes depending on data volume.
Can we ingest logs from on-premises sources without a cloud hop?
Yes. ManySignal supports PrivateLink ingestion from AWS environments and VPC peering for on-premises sources. For fully on-premises environments, the self-hosted deployment receives logs directly without any internet transit.
How does log deduplication work to prevent double-ingestion?
Each connector manifest specifies a deduplication key (typically a source-specific event ID or hash). The ingestion pipeline checks incoming events against a dedup bloom filter with a configurable window (default 24 hours). Duplicate events are counted but not stored twice.
What is included in the default retention window and how do we extend it?
The default retention is 90 days for hot/warm-tier events and 180 days for entity graph state. Extended retention is available in 90-day increments up to 7 years for raw events, configurable per source. Compliance-driven retention requirements (PCI 12-month, HIPAA 6-year) are covered by the available extensions.
Retain what you need. Search it all.
Per-source retention policies, three-tier storage, and on-demand cold queries — included in all ManySignal plans.