Goals
🎯 Primary Goals
The main objective is to design and implement a Highly Available Centralized Logging System for our on-premise Kubernetes infrastructure that ensures:
- Consistent log collection across multiple services and nodes.
- High Availability (HA) during pod, node, or network failures.
- Scalability, both in terms of log volume and system components.
- Structured logs with rich context for better observability.
- Search and visualization capabilities through Grafana/Kibana.
- Secure and auditable access for internal teams.
- Storage and retention using on-premise solutions like MinIO or NFS.
✅ Success Criteria
A solution will be considered successful if it meets the following:
- ✅ Logs from Node.js, Go, Python services are collected consistently.
- ✅ System continues operating when one or more Kubernetes nodes go down.
- ✅ Structured JSON logs with fields like
request_id,timestamp,service,severityare supported. - ✅ Query performance remains acceptable under load.
- ✅ Helm-based deployment is available and repeatable for on-prem K8s.
- ✅ No single point of failure exists in ingestion or query pipeline.
📋 Acceptance Criteria
| Criteria | Requirement |
|---|---|
| 🔌 Logging Stack | Support for Loki, ELK, or Graylog |
| ☸️ Kubernetes Native | Components deployed using Helm, StatefulSet, DaemonSet, Ingress |
| 📦 Storage | Compatible with MinIO, NFS, or CephFS |
| 🔄 HA & Scalability | Ingesters and query services must scale horizontally |
| 📄 Log Format | Must support JSON format with standardized fields |
| 🔍 Search & Alerting | Must integrate with Grafana or Kibana for dashboards and alert rules |
| 🔐 Security | Access control via RBAC or reverse proxy auth |
| 📁 Persistence | Each stateful component has persistent volumes configured |
| 🧪 Fault Tolerance | System tolerates node or pod crashes without data loss |
📌 Out of Scope
- Managed logging solutions like Datadog, CloudWatch, or GCP Logging
- Multi-cloud or hybrid-cloud scenarios
- Full SIEM integration (only optional)
🧑💻 Stakeholders
- Backend Team — for log visibility and debugging
- DevOps / Infra Team — for deployment, scaling, and resilience
- Security / Compliance — for audit logs and access control