"Episilia has been indeed helping us in a big way to
manage 2TB per day of logs in production".
- VP Devops @
Sunil Guttula
Hi, I’m Sunil Guttula, architect of Episilia and CEO of 91social. I’ve been a hands-on developer for the last 25 years, building and supporting systems in production. We built Episilia so you can log as much as you want without worrying about latency and cost.
Here you’ll find a simple, technical and detailed overview of Episilia and all that it can do for your business as a tech solution, so you can make an informed choice quickly.
-
Why did we build Episilia?
-
How is Episilia fast and cheap for high volume logs?
-
What does it cost for X TBs of daily logs?
-
What features does Episilia support and what does it not?
-
How can you try out Episilia?
I do hope you find Episilia useful. Happy logging!
Sunil Guttula
The reason |
Why build Episilia?
Why one more log management system?
Logs are quite distinct from business transaction data in that they are:
-
High volume; 10x-100x more than transaction data
-
Low budget; logs are expected to be managed with lower budgets
-
Of varying value; critical during incidents and low value at other times
Current log management systems share these common traits
-
They process logs using s/w built for business data - Lucene, Elastic, Clickhouse
-
They store logs in the same way as business data
-
They end up being slow and costly at high volumes
So we engineered Episilia from the ground up to handle high volume logs faster and cheaper.
-
We designed new data structures optimized for log storage and retrieval
-
Grouped, indexed and compressed logs for low storage footprint
-
Storage separated from compute; logs loaded on-demand for search
-
Used the most compute efficient language, C++ with SIMD
"Get the developers to log less, we got to bring down the cost"
- CTO of a unicorn
Teams with high volume logs rely on Episilia to cut costs by 3x or more
Testimonial
"At nurture.farm we generate a huge amount of logs to discover and solve the bugs on the fast evolving tech stack. With high throughput ingestion and seamless integration with our Kubernetes cluster, Episilia is very efficient in resource utilization and performance. It helps us analyze our logs from the centralized Grafana dashboard. At the same time, centralized logging with Episilia helps our business technologies and security team for root cause analysis."
Abhinasha Karana
Technology Architect @ nurture farm
The design |
Fast and affordable
Episilia is cost-effective because it is fast
-
Episilia is fast because of its data structures, algorithms, high performance design and coding.
-
C++ with SIMD gets the most out of a CPU, which costs 4X of RAM. So we coded Episilia in C++ with SIMD.
-
New data structures exploit repeatability of values in logs and data affinity to apps.
-
Indexing uses highly tuned bloom filters with index size @ 1.5% of data size.
-
LZ4 used for fast (de)compression. 1 TB of logs with index compresses to 100GB storage at rest.
-
SIMD Indexer ingests logs @ 10MBps per core; 3TB of daily log ingestion needs a maximum of 4 cores.
-
Logs and all associated metadata are stored only in S3, no database needed. This keeps clusters stateless and scales well.
-
Indexing and search operate independently; search loads logs on-demand keeping compute requirements low.
-
A search query runs at 350 mcores and 300MB mem max; 10 concurrent queries need 4 Cores + 4GB RAM.
-
98% of search queries return in 1-5 seconds, while reading logs from S3 block storage.
-
Log tail has a latency of 3-5 seconds from log capture to user console.
-
Alerts on logs and tail execute on the same consumer, keeping cost of real-time processing low.
"We did pre-mature optimization every step of the way in building Episilia, and broke a few more rules along the way."
- Architect, Episilia
The cost |
$100 per TB per year
All inclusive - compute, block storage, 12 months retention and license
Estimated price for 1TB - 25TB logs per day
Price includes :
Features included :
Ingestion, Indexing, Search, Tail logs, Alerts on logs
Users:
Unlimited. Concurrent users are assumed to be 20 for the calculation.
Logs retention :
12 months logs are stored in S3
Deployment :
On-premise or client's cloud account
Cost includes :
Compute infra, block storage, License cost
Cost excludes :
Log collectors, log transfer to Episilia.
Cost of Episilia compared to its peers
The features |
Log without limits
Debug issues, secure business, understand users
Features aligned to logs flow
Collecting logs
Sources: Kafka, HTTP, S3
-
Supports Open Telemetry and generic JSON message format for logs
-
Logs can be transferred via Kafka, HTTP, S3 to Episilia
-
Timestamp derived from logs or stamped on arrival
-
Independent of log collectors - Fluent Bit, Vector, Filebeat
Indexing
10-20 MBps per core
-
Indexing at 10-20 MBps per core; roughly 1TB per core per day
-
Index is 1.5% of data size, covers keyword + regex
-
Indexer instances can be started/stopped to scale with no data loss
-
Group related logs together by virtual app IDs
Tailing and Alerting
K8s to console ~ 3 seconds
-
Logs tailed to Episilia console in under 3 seconds from origin
-
Tail logs support keyword and regex filters
-
Alerts on logs typically delivered within 5 seconds from origin
-
Alerts delivered to Slack, Pagerduty, email etc
Searching
1-5 seconds, 1M results
-
All logs available to search any time; no need to pre-load
-
Search queries support keyword + PCRE compliant regex queries
-
A search query needs a max of 350 millicores and 300MB RAM
-
98% of search queries return in between 1-5 seconds
-
Large search results > 1M downloaded to a csv file for analysis
-
Logs fetched from S3 on-demand; no hard dependency on disk
-
Frequently read logs are cached to local disk and embedded RocksDB
-
Search instances can be started/stopped to scale
-
Grafana Loki browser supported besides Episilia native console
-
Save queries; share queries and search results with your team
Storage and archival
No DB, only S3
-
Data and index files are stored only in S3
-
All metadata stored in S3; no dependency on a live database
-
Supports any S3 service - MinIO, Azure, GCP, DigitalOcean, Alibaba, OCP
-
S3 files in date/hour folders; simple copy to archive/restore
Open and extensible
Open file formats and APIs
-
Log data files in Hadoop sequence file open format
-
Files in S3 can be processed by Hive or any HDFS engine
-
Search, tail, alerts available as APIs to integrate into third party apps
-
Supports Loki query format for search queries
Easy ops
Self-throttling, auto scaling
-
Episilia cluster is stateless; no state in any live database or disk
-
Deployed via Helm charts or docker manifests
-
Standard K8s/container nodes with AVX compatible cores for SIMD
-
Any service can be started/restarted/stopped any time
-
Distributed cluster with co-operating nodes; no leader election
-
Local redpanda for service co-ordination and communication
-
Prometheus-compatible for monitoring
-
All services support pause/resume control to avoid stop/restart
-
In-built throttling on memory usage; no crashes on high loads
-
Fine grained metrics collected to analyze and tune throughput
Try it out |
1TB/month free
Try it in your cloud
-
Contact us for a trial account
-
Install using Helm charts, instructions here
-
Send logs to Episilia cluster and view them