Monitoring Hub Reference on Cozystack

Monitoring Parameters

Mon, 01 Jan 0001 00:00:00 +0000

Parameters

Common parameters

Name	Description	Type	Value
`host`	The hostname used to access Grafana externally (defaults to ‘grafana’ subdomain for the tenant host).	`string`	`""`

Metrics storage configuration

Name	Description	Type	Value
`metricsStorages`	Configuration of metrics storage instances.	`[]object`	`[...]`
`metricsStorages[i].name`	Name of the storage instance.	`string`	`""`
`metricsStorages[i].retentionPeriod`	Retention period for metrics.	`string`	`""`
`metricsStorages[i].deduplicationInterval`	Deduplication interval for metrics.	`string`	`""`
`metricsStorages[i].storage`	Persistent volume size.	`string`	`10Gi`
`metricsStorages[i].storageClassName`	StorageClass used for the data.	`string`	`""`
`metricsStorages[i].vminsert`	Configuration for vminsert.	`object`	`{}`
`metricsStorages[i].vminsert.minAllowed`	Minimum guaranteed resources.	`object`	`{}`
`metricsStorages[i].vminsert.minAllowed.cpu`	CPU request.	`quantity`	`""`
`metricsStorages[i].vminsert.minAllowed.memory`	Memory request.	`quantity`	`""`
`metricsStorages[i].vminsert.maxAllowed`	Maximum allowed resources.	`object`	`{}`
`metricsStorages[i].vminsert.maxAllowed.cpu`	CPU limit.	`quantity`	`""`
`metricsStorages[i].vminsert.maxAllowed.memory`	Memory limit.	`quantity`	`""`
`metricsStorages[i].vmselect`	Configuration for vmselect.	`object`	`{}`
`metricsStorages[i].vmselect.minAllowed`	Minimum guaranteed resources.	`object`	`{}`
`metricsStorages[i].vmselect.minAllowed.cpu`	CPU request.	`quantity`	`""`
`metricsStorages[i].vmselect.minAllowed.memory`	Memory request.	`quantity`	`""`
`metricsStorages[i].vmselect.maxAllowed`	Maximum allowed resources.	`object`	`{}`
`metricsStorages[i].vmselect.maxAllowed.cpu`	CPU limit.	`quantity`	`""`
`metricsStorages[i].vmselect.maxAllowed.memory`	Memory limit.	`quantity`	`""`
`metricsStorages[i].vmstorage`	Configuration for vmstorage.	`object`	`{}`
`metricsStorages[i].vmstorage.minAllowed`	Minimum guaranteed resources.	`object`	`{}`
`metricsStorages[i].vmstorage.minAllowed.cpu`	CPU request.	`quantity`	`""`
`metricsStorages[i].vmstorage.minAllowed.memory`	Memory request.	`quantity`	`""`
`metricsStorages[i].vmstorage.maxAllowed`	Maximum allowed resources.	`object`	`{}`
`metricsStorages[i].vmstorage.maxAllowed.cpu`	CPU limit.	`quantity`	`""`
`metricsStorages[i].vmstorage.maxAllowed.memory`	Memory limit.	`quantity`	`""`

Logs storage configuration

Name	Description	Type	Value
`logsStorages`	Configuration of logs storage instances.	`[]object`	`[...]`
`logsStorages[i].name`	Name of the storage instance.	`string`	`""`
`logsStorages[i].retentionPeriod`	Retention period for logs.	`string`	`1`
`logsStorages[i].storage`	Persistent volume size.	`string`	`10Gi`
`logsStorages[i].storageClassName`	StorageClass used to store the data.	`string`	`replicated`

Alerta configuration

Name	Description	Type	Value
`alerta`	Configuration for the Alerta service.	`object`	`{}`
`alerta.storage`	Persistent volume size for the database.	`string`	`10Gi`
`alerta.storageClassName`	StorageClass used for the database.	`string`	`""`
`alerta.resources`	Resource configuration.	`object`	`{}`
`alerta.resources.requests`	Resource requests.	`object`	`{}`
`alerta.resources.requests.cpu`	CPU request.	`quantity`	`100m`
`alerta.resources.requests.memory`	Memory request.	`quantity`	`256Mi`
`alerta.resources.limits`	Resource limits.	`object`	`{}`
`alerta.resources.limits.cpu`	CPU limit.	`quantity`	`1`
`alerta.resources.limits.memory`	Memory limit.	`quantity`	`1Gi`
`alerta.alerts`	Alert routing configuration.	`object`	`{}`
`alerta.alerts.telegram`	Configuration for Telegram alerts.	`object`	`{}`
`alerta.alerts.telegram.token`	Telegram bot token.	`string`	`""`
`alerta.alerts.telegram.chatID`	Telegram chat ID(s), separated by commas.	`string`	`""`
`alerta.alerts.telegram.disabledSeverity`	List of severities without alerts (e.g. [“informational”,“warning”]).	`[]string`	`[]`
`alerta.alerts.slack`	Configuration for Slack alerts.	`object`	`{}`
`alerta.alerts.slack.url`	Configuration uri for Slack alerts.	`string`	`""`
`alerta.alerts.slack.disabledSeverity`	List of severities without alerts (e.g. [“informational”,“warning”]).	`[]string`	`[]`
`alerta.alerts.email`	Configuration for email alerts. Alerts carrying the label notify_email with the string value true are emailed when their severity is in the severities list (default critical and major). A bare YAML boolean does not match. Alerts listed in alertnames are emailed unless matched by an earlier platform route.	`object`	`{}`
`alerta.alerts.email.to`	Recipient address, or several separated by commas. A display name is accepted, such as `Oncall <oncall@example.org>`, as long as it carries no comma. Quote display names containing punctuation such as `:`, `@`, parentheses or `?`. RFC 2047 encoded words are not accepted. IP domain literals accept dotted-quad IPv4 only. IPv6 literals are rejected because Go mail parsers disagree on their syntax across versions. ASCII control characters are rejected. Addresses must be literal, so Alertmanager template actions are rejected. The accepted form is deliberately narrower than RFC 5322. The local part must be a dot-atom, with no leading, trailing or doubled dot, and quoted local parts, comments and a `mailto:` prefix are rejected. An address outside that form fails the render, because Alertmanager parses addresses only when it sends and drops the notification with no record a tenant can read.	`string`	`""`
`alerta.alerts.email.from`	Sender address, one only. A display name is accepted. It takes the same narrower form as `to`, and an address outside it fails the render.	`string`	`""`
`alerta.alerts.email.smarthost`	SMTP server as `host:port` (e.g. “smtp.example.com:465”). A value without a port, an invalid bracketed IP address or a port with leading zeros fails the render. Port 465 uses implicit TLS, other ports use STARTTLS.	`string`	`""`
`alerta.alerts.email.authUsername`	Username for SMTP LOGIN/PLAIN authentication. The password is read from the `password` key of a Secret named `alertmanager-email-password` in the release namespace, which the platform operator creates. The name is fixed and the Secret is mounted into Alertmanager, so a missing Secret blocks the whole Alertmanager rollout, not only email delivery. The credential in it must belong to this tenant alone. `smarthost` is chosen by whoever writes the Monitoring, so one relay credential provisioned under the same name in every namespace can be collected by pointing `smarthost` at a host the tenant controls.	`string`	`""`
`alerta.alerts.email.repeatInterval`	Minimum time before a still-firing alert is emailed again, as an Alertmanager duration with units `y`, `w`, `d`, `h`, `m`, `s`, `ms` in descending order, each at most once, such as `5m`, `12h` or `1h30m`. Delivery is evaluated every 5m, so shorter intervals take effect at 5m and other intervals round up to a multiple of 5m. Zero, a unit out of order and a total beyond what Alertmanager accepts fail the render. Empty keeps the 12h default.	`string`	`""`
`alerta.alerts.email.alertnames`	Alert names emailed whether or not the rule carries the notify_email label, with no severity filter. Earlier platform routes still take precedence. Alerts with `event="Heartbeat"`, including `Watchdog`, go only to Alerta, and alerts with severity `none`, `CDIDefaultStorageClassDegraded` and `CDINoDefaultStorageClass` are blackholed and never reach email. This is the way to opt in alerts the platform ships, since their helm-owned rule definitions cannot be labelled per cluster. Entries must be plain Prometheus alert names such as `KubeAPIDown`, and anything else fails the render. Empty adds no extra route.	`[]string`	`[]`
`alerta.alerts.email.severities`	Severities the notify_email route accepts (e.g. [“critical”,“major”,“warning”]). Empty keeps the default of critical and major. List every severity in use to email opted-in alerts regardless of severity. Entries must be plain severity names, and anything else fails the render.	`[]string`	`[]`

Grafana configuration

Name	Description	Type	Value
`grafana`	Configuration for Grafana.	`object`	`{}`
`grafana.db`	Database configuration.	`object`	`{}`
`grafana.db.size`	Persistent volume size for the database.	`string`	`10Gi`
`grafana.resources`	Resource configuration.	`object`	`{}`
`grafana.resources.requests`	Resource requests.	`object`	`{}`
`grafana.resources.requests.cpu`	CPU request.	`quantity`	`100m`
`grafana.resources.requests.memory`	Memory request.	`quantity`	`256Mi`
`grafana.resources.limits`	Resource limits.	`object`	`{}`
`grafana.resources.limits.cpu`	CPU limit.	`quantity`	`1`
`grafana.resources.limits.memory`	Memory limit.	`quantity`	`1Gi`

Vmagent configuration

Name	Description	Type	Value
`vmagent`	Configuration for VictoriaMetrics Agent.	`object`	`{}`
`vmagent.externalLabels`	External labels applied to all metrics.	`object`	`{}`
`vmagent.remoteWrite`	Remote write configuration.	`object`	`{}`

OIDC authentication

Name	Description	Type	Value
`oidc`	OIDC authentication for the Grafana instance. See docs/oidc-grafana.md for the operator guide.	`object`	`{}`
`oidc.mode`	Identity mode. `None`: no OIDC, only the `grafana-admin-password` Secret works. `System`: trust the platform `cozy` realm via a per-instance confidential client and audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path.	`string`	`None`
`oidc.customConfig`	Tenant-supplied `[auth.generic_oauth]` payload; consumed only when `mode: CustomConfig`.	`object`	`{}`
`oidc.customConfig.config`	Inline map of `[auth.generic_oauth]` keys. Written verbatim into the Grafana CR `spec.config`. Mutually exclusive with `secretRef.name`.	`object`	`{}`
`oidc.customConfig.secretRef`	Reference to a Secret holding an `auth.ini` fragment; mounted under `/etc/grafana/oidc` and loaded via `GF_PATHS_CUSTOM_INI`.	`object`	`{}`
`oidc.customConfig.secretRef.name`	Name of an existing Secret in the release namespace whose `auth.ini` key holds a ready-made `[auth.generic_oauth]` block. Mutually exclusive with `customConfig.config`.	`string`	`""`
`oidc.users`	Users granted access to this Grafana instance’s org. Each entry is reconciled into Grafana org membership + role by a chart-owned post-install/post-upgrade Job. Works for both `System` and `CustomConfig` modes; ignored when `mode: None`. Users not listed here get nothing (no default org role from group mapping).	`[]object`	`[]`
`oidc.users[i].email`	Email address matched against the `email` claim from the issuer. The chart pre-provisions a Grafana account with this email so the users-Job can set the org role before the operator’s first login (`oauth_allow_insecure_email_lookup = true` glues the two together). Must match a conservative pattern: local part is `[A-Za-z0-9._%+-]+`, domain is `[A-Za-z0-9.-]+\.[A-Za-z]{2,}`. Whitespace, quoted literals, and unusual URL-reserved characters are rejected on schema level so the reconcile-Job’s shell handling never has to defend against malformed input.	`string`	`""`
`oidc.users[i].role`	Grafana org role to bind: `Admin`, `Editor`, or `Viewer`. Applied to the release’s Grafana org (`Main Org.`); server-level `GrafanaAdmin` promotion is out of scope for Phase 1. Required — the users-Job POSTs `{loginOrEmail, role}` to `/api/orgs/1/users` and PATCHes `{role}` to converge across re-runs; omitting the field would render `role: "null"` into the body and fail with HTTP 400.	`string`	`{}`

Monitoring Setup

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Cozystack provides a comprehensive monitoring and alerting system for Kubernetes clusters and applications. The system is based on Victoria Metrics for storing metrics and logs, Grafana for visualization, Alerta for alerting, and WorkloadMonitor for monitoring application states.

The architecture is divided into two levels:

System level: Monitoring of cluster infrastructure
Tenant-specific level: Isolated monitoring for each tenant

Installation of Monitoring

System Monitoring

System monitoring is installed automatically during platform deployment through the HelmRelease monitoring-agents in packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml.

Monitoring Dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Cozystack integrates Grafana as the primary visualization tool for monitoring metrics collected by VictoriaMetrics (VM). This section covers accessing pre-built dashboards, creating custom visualizations, and integrating external data sources to provide comprehensive observability for your Cozystack clusters and applications.

Accessing Grafana

To access Grafana and explore dashboards:

Navigate to the Grafana URL: https://grafana.<tenant-domain>, where <tenant-domain> is your tenant’s domain.
Log in using your tenant credentials (OIDC or token-based authentication).
Once logged in, you can browse pre-configured dashboards in the “Dashboards” section.

For initial setup and configuration details, refer to Monitoring Setup.

Monitoring Logs

Mon, 01 Jan 0001 00:00:00 +0000

Collecting and Storing Logs

Cozystack uses Fluent Bit for log collection and VictoriaLogs for log storage and querying. Logs are collected from various sources within the cluster and stored in dedicated log storages configured per tenant.

Configuring Logs Storages

Log storages are configured through the monitoring hub parameters. Each tenant can have multiple log storage instances with customizable retention periods and storage sizes.

Parameter	Description	Type	Default
`logsStorages`	Array of log storage configurations	`[]object`	`[]`
`logsStorages[i].name`	Name of the storage instance	`string`	`""`
`logsStorages[i].retentionPeriod`	Retention period for logs (e.g., “30d”)	`string`	`"1"`
`logsStorages[i].storage`	Persistent volume size	`string`	`"10Gi"`
`logsStorages[i].storageClassName`	StorageClass for data persistence	`string`	`"replicated"`

For detailed configuration options, see Monitoring Hub Reference.

Custom Metrics Collection

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Cozystack tenant monitoring supports scraping custom metrics from your own applications and exporters. The tenant VMAgent discovers scrape targets through Kubernetes namespace labels, allowing you to connect any application that exposes a Prometheus-compatible /metrics endpoint.

This guide explains how to create VMServiceScrape and VMPodScrape resources so that the tenant VMAgent collects your custom metrics and makes them available in Grafana.

Prerequisites

Monitoring is enabled for your tenant (see Monitoring Setup)
Your application or exporter is deployed and exposes a Prometheus-compatible /metrics endpoint
You have kubectl access to the cluster

Using VMServiceScrape

A VMServiceScrape tells the tenant VMAgent to scrape metrics from endpoints behind a Kubernetes Service.

Monitoring Alerting

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The alerting system in Cozystack integrates Prometheus, Alertmanager, and Alerta to provide comprehensive monitoring and notification capabilities. Alerts are generated based on metrics collected by VMAgent and stored in VMCluster, then routed through Alertmanager for grouping and deduplication, and finally managed by Alerta for notifications via various channels like Telegram and Slack.

Alerting Flow

sequenceDiagram
 participant P as Prometheus
 participant AM as Alertmanager
 participant A as Alerta
 participant T as Telegram
 participant S as Slack
 P->>AM: Send Alert
 AM->>A: Forward Alert
 A->>T: Send Notification
 A->>S: Send Notification

Configuring Alerts in Alerta

Alerta is the alerting system integrated into Cozystack’s monitoring stack. It processes alerts from various sources and provides notifications through multiple channels.