top of page
episilia main page creative-02.png
logo_edited.png
logo_edited.png
logo_edited.png
logo.png

logs

logs > 1TB per month?

welcome, episilia is built for you!

logo_edited.png
logo_edited.png
logo_edited.png

logs < 1TB? please come back when you hit 1TB

"Episilia has been indeed helping us in a big way for managing 2TB per day of logs in production".
- VP Devops @ Logistics Company

Sunil.png

Sunil Guttula

Hi, I am Sunil Guttula, architect and ceo of episilia, have been a hands-on developer for the last 25 years building and supporting systems in production. We built episilia so you can log as much as you want without worrying about latency and cost.

​

Regarding the website, we tried to keep it simple, technical and detailed so you can make an informed choice as fast as possible. The website covers

​

  1. Why did we build episilia?

  2. How is episilia fast and cheap for high volume logs?

  3. What does it cost for X TBs of daily logs?

  4. Features it supports, and does not

  5. How to try out episilia?

​

I do hope you find episilia useful, happy logging.

​

Sunil Guttula

episilia main page design-01.png
logo_edited.png
logo_edited.png
logo_edited.png
the reason  | 
logo.png

why build episilia?

why one more log mgmt system?

Logs are quite distinct from business transaction data 

  1. High volume; 10x-100x more than transaction data

  2. Low budget; logs are expected to be managed with lower budgets

  3. Varying value; critical during incidents and low value at other times 

​

Current log mgmt systems share these common traits

  1. Process logs using s/w built for business data - lucene, elastic, clickhouse

  2. Store logs in the same way as business data

  3. And end up being slow and costly at high volumes

​

So we engineered episilia from the ground up to handle high volume logs faster and cheaper.

  1. Designed new data structures optimized for log storage and retrieval 

  2. Grouped, indexed and compressed logs for low storage footprint

  3. Storage separated from compute; logs loaded on-demand for search

  4. Used the most compute efficient language, C++ with SIMD

"tell the developers to log less, we got to bring down the cost"
-
CTO of a unicorn
logo_edited.png
logo_edited.png
logo_edited.png
Teams with high volume logs rely on episilia to cut costs by 3x or more
freight-tiger-logo.png
Nurture_Master_Logo_Landscape_OnWhite.png

Testimonial

"At nurture.farm we generate a huge amount of logs to discover and solve the bugs on the fast evolving tech stack. With high throughput ingestion and seamless integration with our Kubernetes cluster, Episilia is very efficient in resource utilization and performance. It helps us analyze our logs from the centralized Grafana dashboard.. At the same time, centralized logging with Episilia helps our business technologies and security team for root cause analysis"

Abinasha-Karana-1.jpg

Abhinasha Karana

Technology Architect @ nurture farm

the design  | 
logo.png

fast and cheap

episilia is cheap because it is fast

  • Episilia is fast because of its data structures, algorithms, high performance design & coding

  • C++ with SIMD gets the most out of a CPU; CPU costs 4X of RAM. So we coded episilia in C++ with SIMD

  • New data structures exploit repeatability of values in logs and data affinity to apps

  • Indexing uses highly tuned bloom filters with index size @ 1.5% of data size

  • LZ4 used for fast (de)compression. 1 TB of logs with index compresses to 100GB storage at rest

  • SIMD Indexer ingests logs @ 10MBps per core; 3TB of daily log ingestion needs max of 4 cores

  • Logs and all associated metadata stored only in S3, no database needed. Keeps cluster stateless and scales well

  • Indexing and Search operate independently; Search loads logs on-demand keeping compute requirements low

  • A search query runs at 350 mcores and 300MB mem max; 10 concurrent queries need 4 Cores + 4GB RAM

  • 98% of search queries return in 1-5 seconds, while reading logs from S3 block storage

  • Log tail has a latency of 3-5 seconds from log capture to user console.

  • Alerts on logs and Tail execute on the same consumer, keeping cost of real-time processing low

episilia-flow.gif

"we did pre-mature optimization every step of the way in building episilia, and broke a few more rules along the way" 
- architect, episilia

the cost  | 
logo.png

$100 per TB per year

all inclusive - compute, block storage, 12 months retention & license

Estimated price for 1TB - 25TB logs per day

Price Includes :

Features included.png

Features included :

Ingestion, Indexing, Search, Tail logs, Alerts on logs

Users.png

Users:

Unlimited. Concurrent users are assumed to be 20 for the calculation.

Logs.png

Logs retention :

12 months Logs are stored in S3

Deployment.png
Cost.png
Cost.png

Deployment :

On-premise or Client's cloud account

Cost includes :

Compute infra, Block storage, License cost

Cost excludes :

Log collectors, Log transfer to Episilia. 

Cost of episilia compared to its peers
Get the detailed price calculator

Thanks for submitting!

the features  | 
logo.png

log w/o limits

debug issues, secure business, understand users

features aligned to logs flow
collecting logs.png
collecting logs

sources: kafka, http, s3

  • supports opentelemetry and generic json message format for logs

  • logs can be transferred via kafka, http, S3 to episilia

  • timestamp derived from logs or stamped on arrival

  • independent of log collectors - fluentbit, vector, filebeat

indexing
indexing.png

10-20 MBps per core

  • indexing at 10-20 MBps per core; roughly 1TB per core per day

  • index is 1.5% of data size, covers keyword + regex

  • indexer instances can be started/stopped to scale with no data loss 

  • group related logs together by virtual app ids

tailing & alerting.png
tailing & alerting

k8s to console ~ 3 seconds

  • logs tailed to episilia console in under 3 seconds from origin

  • tail logs support keyword & regex filters 

  • alerts on logs typically delivered within 5 seconds from origin

  • alerts delivered to slack, pagerduty, email etc

searching.png
searching

1-5 seconds, 1M results

  • all logs available to search any time; no need to pre-load

  • search queries support keyword + PCRE compliant regex queries

  • a search query needs a max of 350 millicores and 300MB RAM

  • 98% of search queries return in between 1-5 seconds

  • large search results > 1M downloaded to a csv file for analysis 

  • logs fectched from S3 on-demand; no hard dependency on disk

  • frequently read logs are cached to local disk and embedded rocksdb

  • search instances can be started/stopped to scale

  • grafana loki browser supported besides episilia native console

  • save queries; share queries and search results with your team

storage & archival.png
storage & archival

no db, only s3

  • data and index files are stored only in S3

  • all metadata stored in S3; no dependency on a live database

  • supports any S3 service; minio, azure, gcp, digitalocean, alibaba, ocp  

  • S3 files in date/hour folders; simple copy to archive/restore

open-file.png
open & extensible

open file formats & APIs

  • log data files in hadoop sequence file open format

  • files in S3 can be processed by hive or any hdfs engine 

  • search, tail, alerts available as APIs to integrate into 3rd party apps 

  • supports loki query format for search queries

easy ops.png
easy ops

self-throttling, auto scaling

  • episilia cluster is stateless; no state in any live database or disk

  • deployed via helm charts or docker manifests 

  • standard k8s/container nodes with AVX compatible cores for SIMD

  • any service can be started/restarted/stopped any time

  • distributed cluster with co-operating nodes; no leader election

  • local redpanda for service co-ordination and communication

  • is prometheus compatible for monitoring

  • all services support pause/resume control to avoid stop/restart

  •  in-built throttling on memory usage; no crashes on high loads 

  • fine grained metrics collected to analyze and tune throughput 

try it out  | 
logo.png
logo_edited.png

1TB/month free

try it in your cloud

  1. Contact us for a trial account

  2. Install using helm charts, instructions here

  3. Send logs to episilia cluster and view them

​

Contact us

Thanks for contacting us, we will get back to you within 24 hours.

bottom of page