member-1
Alloy push
member-n
Alloy push
Mimir
metrics
Loki
logs
Tempo
traces
Pyroscope
profiles
Central
LGTM+ control room
Huginn Agent
LangGraphquery loop
Telegram thread
#incident-auth-service
Alert fired
auth-service p95 latency crossed 2.4s
Hypothesis: latency aligns with Redis saturation after deploy canary-42.
Suggested action
Scale redis-session to two replicas, then watch p95 for 15 minutes.
A self-hosted agent layer for the LGTM+ stack.
Huginn keeps the familiar Grafana data plane and adds a reviewable agent loop on top. Alerts become cited RCA threads, chats can ask follow-up questions, and remediation stays behind explicit approval until trust is earned.
Reactive RCA
Alertmanager webhooks trigger an agent that queries metrics, logs, traces, profiles and topology before posting a hypothesis.
Conversational on-call
Telegram threads keep incident context alive so operators can ask why a service is slow without switching across query languages.
Approval-gated remediation
Runbooks become action cards with approve or reject decisions, audit logs and an opt-in path to automation.
From signal to cited hypothesis.
The product shape follows the main operational pain: correlate four observability pillars under pressure, without hiding the evidence behind a black box.
- 01
Collect
Alloy ships metrics, logs, traces and profiles from member nodes into the central LGTM+ stack.
- 02
Investigate
The LangGraph agent iterates over PromQL, LogQL, TraceQL, Pyroscope and topology tools.
- 03
Cite
Every hypothesis links back to Grafana Explore with the exact query and time window that produced it.
- 04
Decide
Telegram action cards capture approve, reject and false-positive decisions before any command can run.
Centralized data, reviewable control.
Huginn separates the data plane from the agent control plane. The agent never reads raw object storage; it uses the same APIs an operator would verify in Grafana.
Data plane
Mimir, Loki, Tempo, Pyroscope and MinIO store the four signals behind one tenant header.
Control plane
Alertmanager triggers the Python agent. Telegram carries the thread, citations, action card and approvals.
BYOK runtime
OpenAI-compatible, Anthropic, Ollama or local endpoints. No bundled model and no phone-home telemetry by default.
Two modes, one config schema.
The same system can start as a single-instance homelab deployment and grow into central plus member nodes without changing its mental model.
Single-instance
One host runs collectors, LGTM+ backends, MinIO, Grafana, Alertmanager and the agent for local tenant operations.
Central + members
Member nodes push through HTTPS with bearer token and X-Scope-OrgID, so NAT and dynamic IP environments stay workable.
Trust boundary
PII redaction, local-only KB files, sandboxed commands and command whitelists keep automation reviewable.
Huginn chooses ownership over outsourced intelligence.
The project accepts operational weight so incident context, model choice, and remediation decisions stay inside the owner's boundary.
- 01
Self-hosted LGTM+
Optimizes
Optimized for data ownership, cost control, and a stack operators can verify directly in Grafana.
Accepted cost
The owner must run more infrastructure than a cloud AI monitoring add-on.
- 02
BYOK model runtime
Optimizes
Optimized for provider freedom, local model options, and privacy-conscious deployments.
Accepted cost
Model quality, latency, pricing, and key management become deployment decisions instead of bundled defaults.
- 03
Approval before action
Optimizes
Optimized for trust: remediation starts as a reviewed action card with citations and audit trail.
Accepted cost
It intentionally moves slower than full autopilot until repeated approvals justify per-action automation.
A design-phase system with concrete implementation rails.
Huginn is not presented as a finished AI SRE product. The useful evidence today is the architecture, ADR set, deployment assets, and execution plan that define what should be built next.
Decision record is explicit
README, Vision, Architecture, ADRs, and active exec plans define the problem, stack choices, agent scope, delivery path, and security boundary.
Deployment shape is mapped
Compose/member directories, Grafana stack config, scripts, and hardening notes outline the central plus member topology.
Next proof is a live RCA loop
The next milestone is firing a sample alert, running the LangGraph tool loop, and posting a cited hypothesis into Telegram.