episilia

logs

Terabytes of logs? Manage them for FREE with episilia

"Episilia has been helping us in a big way to
manage 2TB of logs per day in production"
- VP Devops, FreightTiger

Solutions |

Full value from every log line

Episilia enables high value solutions on streaming logs to eliminate security threats, monitor user experience and troubleshoot applications.

Eliminate Security Threats

With high speed indexing at 15MBps per core and PCRE regex out of the box, incoming logs can be analyzed as they arrive and eliminate every security threat to the business.

Monitor User Experience

High velocity data coming in from streaming media, logistics carriers and payment gateways can be monitored to provide for the right user experience and rectified in realtime.

Troubleshoot Applications

Detailed application logs can be stored for months to get an accurate picture of the entire application stack and troubleshoot issues across versions without any guesswork.

Teams with high volume logs rely on Episilia to cut costs by 3x or more

Testimonial

"At nurture.farm we generate terabytes of logs daily and we use them to identify and fix production issues. Episilia operates at high throughput and low resource utilization while integrating seamlessly with the K8S cluster. Centralized logging with Episilia helps our business IT and security teams with root cause analysis."

Abhinasha Karana

Technology Architect @ nurture farm

Under the hood |

Fast and Affordable

Episilia costs less because it is fast

Zero Bottlenecks

Peer to Peer collaborative architecture; no SPOF
Stateless components to scale horizontally
No central database; S3 is the primary datastore for logs
Indexer and search instances scale independent of each other

Minimal Data Footprint

Data structures exploit natural order and repeatable values in logs
Double optimized bloom filters with index size at 1.5% of log bytes
Logs and index LZ4 compressed. 10TB of logs zips to 1TB on disk
Flatbuffer format to de-serialize data with less cpu cycles

CPU and RAM Optimized

C++ with SIMD indexes logs at 10-20MBps/core or 1TB/core/day
10 concurrent search queries with just 4 Cores + 4GB RAM
3 Tier cache in RAM, Rocksdb and Local FS
Tail has a guaranteed latency of 3-5 seconds from log write to display

"We did pre-mature optimization in every step in building Episilia, for each cpu cycle and each byte."
- Episilia Team

The features |

Built for dev and ops teams

Logging without limits, Queries in realtime, and Simple to operate

Collecting logs

Sources: Kafka, HTTP, S3

Supports Open Telemetry and generic JSON message format for logs
Logs can be transferred via Kafka, HTTP, S3 to Episilia
Timestamp derived from logs or stamped on arrival
Independent of log collectors - Fluent Bit, Vector, Filebeat

Indexing

10-20 MBps per core

Indexing at 10-20 MBps per core; roughly 1TB per core per day
Index is 1.5% of data size, covers keyword + regex
Indexer instances can be started/stopped to scale with no data loss
Group related logs together by virtual app IDs

Tailing and Alerting

K8s to console ~ 3 seconds

Logs tailed to Episilia console in under 3 seconds from origin
Tail logs support keyword and regex filters
Alerts on logs typically delivered within 5 seconds from origin
Alerts delivered to Slack, Pagerduty, email etc

Searching

1-5 seconds, 1M results

All logs available to search any time; no need to pre-load
Search queries support keyword + PCRE compliant regex queries
A search query needs a max of 350 millicores and 300MB RAM
98% of search queries return in between 1-5 seconds
Large search results > 1M downloaded to a csv file for analysis
Logs fetched from S3 on-demand; no hard dependency on disk
Frequently read logs are cached to local disk and RocksDB
Search instances can be started/stopped to scale
Grafana Loki browser supported besides Episilia native console
Save queries; share queries and search results with your team

Storage and archival

No DB, only S3

Data and index files are stored only in S3
All metadata stored in S3; no dependency on a live database
Any S3 store - MinIO, Azure, GCP, DigitalOcean, Alibaba, OCP
S3 files in date/hour folders; simple copy to archive/restore

Open and extensible

Open file formats and APIs

Log data files in Hadoop sequence file open format
Files in S3 can be processed by Hive or any HDFS engine
Search, tail, alerts available as APIs to integrate into third party apps
Supports Loki query format for search queries

Simple ops

Self-throttling, auto scaling

Episilia cluster is stateless; no state in any live database or disk
Deployed via Helm charts or docker manifests
Standard K8s/container nodes with AVX compatible cores for SIMD
Any service can be started/restarted/stopped any time
Distributed cluster with co-operating nodes; no leader election
Local redpanda for service co-ordination and communication
Prometheus-compatible for monitoring
All services support pause/resume control to avoid stop/restart
In-built throttling on memory usage; no crashes on high loads
Fine grained metrics collected to analyze and tune throughput

"We reduced our logging infrastructure cost by 4X with Episilia. There is no comparison."
- Architect, Nurture Farm

Deploy it today |

Episilia is FREE

Fill the form to get a FREE license
Install using Helm charts, instructions here
Send logs to Episilia cluster and view them