logs
logs > 1TB per month?
welcome, episilia is built for you!
logs < 1TB? please come back when you hit 1TB
"Episilia has been indeed helping us in a big way for managing 2TB per day of logs in production".
- VP Devops @ Logistics Company
Sunil Guttula
Hi, I am Sunil Guttula, architect and ceo of episilia, have been a hands-on developer for the last 25 years building and supporting systems in production. We built episilia so you can log as much as you want without worrying about latency and cost.
​
Regarding the website, we tried to keep it simple, technical and detailed so you can make an informed choice as fast as possible. The website covers
​
-
Why did we build episilia?
-
How is episilia fast and cheap for high volume logs?
-
What does it cost for X TBs of daily logs?
-
Features it supports, and does not
-
How to try out episilia?
​
I do hope you find episilia useful, happy logging.
​
Sunil Guttula
the reason |
why build episilia?
why one more log mgmt system?
Logs are quite distinct from business transaction data
-
High volume; 10x-100x more than transaction data
-
Low budget; logs are expected to be managed with lower budgets
-
Varying value; critical during incidents and low value at other times
​
Current log mgmt systems share these common traits
-
Process logs using s/w built for business data - lucene, elastic, clickhouse
-
Store logs in the same way as business data
-
And end up being slow and costly at high volumes
​
So we engineered episilia from the ground up to handle high volume logs faster and cheaper.
-
Designed new data structures optimized for log storage and retrieval
-
Grouped, indexed and compressed logs for low storage footprint
-
Storage separated from compute; logs loaded on-demand for search
-
Used the most compute efficient language, C++ with SIMD
"tell the developers to log less, we got to bring down the cost"
- CTO of a unicorn
Teams with high volume logs rely on episilia to cut costs by 3x or more
Testimonial
"At nurture.farm we generate a huge amount of logs to discover and solve the bugs on the fast evolving tech stack. With high throughput ingestion and seamless integration with our Kubernetes cluster, Episilia is very efficient in resource utilization and performance. It helps us analyze our logs from the centralized Grafana dashboard.. At the same time, centralized logging with Episilia helps our business technologies and security team for root cause analysis"
Abhinasha Karana
Technology Architect @ nurture farm
the design |
fast and cheap
episilia is cheap because it is fast
-
Episilia is fast because of its data structures, algorithms, high performance design & coding
-
C++ with SIMD gets the most out of a CPU; CPU costs 4X of RAM. So we coded episilia in C++ with SIMD
-
New data structures exploit repeatability of values in logs and data affinity to apps
-
Indexing uses highly tuned bloom filters with index size @ 1.5% of data size
-
LZ4 used for fast (de)compression. 1 TB of logs with index compresses to 100GB storage at rest
-
SIMD Indexer ingests logs @ 10MBps per core; 3TB of daily log ingestion needs max of 4 cores
-
Logs and all associated metadata stored only in S3, no database needed. Keeps cluster stateless and scales well
-
Indexing and Search operate independently; Search loads logs on-demand keeping compute requirements low
-
A search query runs at 350 mcores and 300MB mem max; 10 concurrent queries need 4 Cores + 4GB RAM
-
98% of search queries return in 1-5 seconds, while reading logs from S3 block storage
-
Log tail has a latency of 3-5 seconds from log capture to user console.
-
Alerts on logs and Tail execute on the same consumer, keeping cost of real-time processing low
"we did pre-mature optimization every step of the way in building episilia, and broke a few more rules along the way"
- architect, episilia
the cost |
$100 per TB per year
all inclusive - compute, block storage, 12 months retention & license
Estimated price for 1TB - 25TB logs per day
Price Includes :
Features included :
Ingestion, Indexing, Search, Tail logs, Alerts on logs
Users:
Unlimited. Concurrent users are assumed to be 20 for the calculation.
Logs retention :
12 months Logs are stored in S3
Deployment :
On-premise or Client's cloud account
Cost includes :
Compute infra, Block storage, License cost
Cost excludes :
Log collectors, Log transfer to Episilia.
Cost of episilia compared to its peers
the features |
log w/o limits
debug issues, secure business, understand users
features aligned to logs flow
collecting logs
sources: kafka, http, s3
-
supports opentelemetry and generic json message format for logs
-
logs can be transferred via kafka, http, S3 to episilia
-
timestamp derived from logs or stamped on arrival
-
independent of log collectors - fluentbit, vector, filebeat
indexing
10-20 MBps per core
-
indexing at 10-20 MBps per core; roughly 1TB per core per day
-
index is 1.5% of data size, covers keyword + regex
-
indexer instances can be started/stopped to scale with no data loss
-
group related logs together by virtual app ids
tailing & alerting
k8s to console ~ 3 seconds
-
logs tailed to episilia console in under 3 seconds from origin
-
tail logs support keyword & regex filters
-
alerts on logs typically delivered within 5 seconds from origin
-
alerts delivered to slack, pagerduty, email etc
searching
1-5 seconds, 1M results
-
all logs available to search any time; no need to pre-load
-
search queries support keyword + PCRE compliant regex queries
-
a search query needs a max of 350 millicores and 300MB RAM
-
98% of search queries return in between 1-5 seconds
-
large search results > 1M downloaded to a csv file for analysis
-
logs fectched from S3 on-demand; no hard dependency on disk
-
frequently read logs are cached to local disk and embedded rocksdb
-
search instances can be started/stopped to scale
-
grafana loki browser supported besides episilia native console
-
save queries; share queries and search results with your team
storage & archival
no db, only s3
-
data and index files are stored only in S3
-
all metadata stored in S3; no dependency on a live database
-
supports any S3 service; minio, azure, gcp, digitalocean, alibaba, ocp
-
S3 files in date/hour folders; simple copy to archive/restore
open & extensible
open file formats & APIs
-
log data files in hadoop sequence file open format
-
files in S3 can be processed by hive or any hdfs engine
-
search, tail, alerts available as APIs to integrate into 3rd party apps
-
supports loki query format for search queries
easy ops
self-throttling, auto scaling
-
episilia cluster is stateless; no state in any live database or disk
-
deployed via helm charts or docker manifests
-
standard k8s/container nodes with AVX compatible cores for SIMD
-
any service can be started/restarted/stopped any time
-
distributed cluster with co-operating nodes; no leader election
-
local redpanda for service co-ordination and communication
-
is prometheus compatible for monitoring
-
all services support pause/resume control to avoid stop/restart
-
in-built throttling on memory usage; no crashes on high loads
-
fine grained metrics collected to analyze and tune throughput
try it out |
1TB/month free
try it in your cloud
-
Contact us for a trial account
-
Install using helm charts, instructions here
-
Send logs to episilia cluster and view them
​