mirror of
https://github.com/lukaszraczylo/kubemirror.git
synced 2026-07-08 16:34:36 +00:00
initial commit
This commit is contained in:
@@ -0,0 +1,267 @@
|
||||
# KubeMirror Monitoring
|
||||
|
||||
This directory contains observability resources for monitoring KubeMirror in production.
|
||||
|
||||
## Overview
|
||||
|
||||
KubeMirror exposes Prometheus metrics on port 8080 at `/metrics`. The monitoring stack includes:
|
||||
|
||||
- **ServiceMonitor**: Prometheus Operator resource for automatic metric scraping
|
||||
- **PrometheusRule**: Alert rules for common operational issues
|
||||
- **Grafana Dashboard**: Comprehensive visualization of controller metrics
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Prometheus Operator installed in your cluster
|
||||
- Grafana (optional, for dashboards)
|
||||
|
||||
```bash
|
||||
# Install Prometheus Operator (if not already installed)
|
||||
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
||||
helm repo update
|
||||
helm install prometheus prometheus-community/kube-prometheus-stack \
|
||||
--namespace monitoring \
|
||||
--create-namespace
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Deploy Monitoring Resources
|
||||
|
||||
```bash
|
||||
# Apply ServiceMonitor and PrometheusRule
|
||||
kubectl apply -f monitoring/servicemonitor.yaml
|
||||
kubectl apply -f monitoring/prometheusrule.yaml
|
||||
```
|
||||
|
||||
### Import Grafana Dashboard
|
||||
|
||||
1. **Via UI:**
|
||||
- Open Grafana
|
||||
- Go to Dashboards → Import
|
||||
- Upload `grafana-dashboard.json`
|
||||
- Select your Prometheus datasource
|
||||
|
||||
2. **Via ConfigMap (GitOps):**
|
||||
```bash
|
||||
kubectl create configmap kubemirror-dashboard \
|
||||
--from-file=dashboard.json=monitoring/grafana-dashboard.json \
|
||||
-n monitoring \
|
||||
--dry-run=client -o yaml | kubectl apply -f -
|
||||
|
||||
# Label for automatic discovery by Grafana
|
||||
kubectl label configmap kubemirror-dashboard \
|
||||
grafana_dashboard=1 \
|
||||
-n monitoring
|
||||
```
|
||||
|
||||
## Available Metrics
|
||||
|
||||
### Controller Runtime Metrics
|
||||
|
||||
These metrics are provided by the controller-runtime framework:
|
||||
|
||||
- `controller_runtime_reconcile_total` - Total reconciliations (by controller, result)
|
||||
- `controller_runtime_reconcile_errors_total` - Failed reconciliations
|
||||
- `controller_runtime_reconcile_time_seconds` - Reconciliation duration histogram
|
||||
- `workqueue_depth` - Current workqueue depth
|
||||
- `workqueue_adds_total` - Total items added to workqueue
|
||||
- `workqueue_retries_total` - Workqueue retry count
|
||||
|
||||
### Leader Election Metrics
|
||||
|
||||
- `leader_election_master_status` - Leader election status (1 = leader, 0 = follower)
|
||||
|
||||
### Go Runtime Metrics
|
||||
|
||||
- `go_goroutines` - Current goroutine count
|
||||
- `go_memstats_alloc_bytes` - Allocated memory
|
||||
- `process_open_fds` - Open file descriptors
|
||||
- `process_cpu_seconds_total` - CPU time
|
||||
|
||||
## Alert Rules
|
||||
|
||||
The PrometheusRule defines alerts for:
|
||||
|
||||
### Critical Alerts
|
||||
|
||||
- **KubeMirrorControllerDown**: Controller pod is not running
|
||||
- Severity: `critical`
|
||||
- Fires after: 5 minutes
|
||||
|
||||
### Warning Alerts
|
||||
|
||||
- **KubeMirrorHighReconcileErrors**: High error rate in reconciliation
|
||||
- Threshold: >10% error rate
|
||||
- Fires after: 10 minutes
|
||||
|
||||
- **KubeMirrorReconcileLatencyHigh**: Slow reconciliation loops
|
||||
- Threshold: p99 latency > 5 seconds
|
||||
- Fires after: 10 minutes
|
||||
|
||||
- **KubeMirrorWorkqueueDepthHigh**: Work items piling up
|
||||
- Threshold: >100 items in queue
|
||||
- Fires after: 15 minutes
|
||||
|
||||
- **KubeMirrorLeaderElectionLost**: Controller is not the leader
|
||||
- Fires after: 2 minutes
|
||||
|
||||
- **KubeMirrorHighFailureRate**: Overall operation failure rate high
|
||||
- Threshold: >5% failure rate
|
||||
- Fires after: 10 minutes
|
||||
|
||||
- **KubeMirrorMemoryHigh**: High memory usage
|
||||
- Threshold: >90% of memory limit
|
||||
- Fires after: 5 minutes
|
||||
|
||||
- **KubeMirrorCPUThrottling**: CPU throttling detected
|
||||
- Fires after: 10 minutes
|
||||
|
||||
## Recording Rules
|
||||
|
||||
Recording rules pre-compute expensive queries for better dashboard performance:
|
||||
|
||||
- `kubemirror:reconcile_duration_seconds:p99` - P99 reconciliation latency
|
||||
- `kubemirror:reconcile_duration_seconds:p95` - P95 reconciliation latency
|
||||
- `kubemirror:reconcile_duration_seconds:p50` - P50 reconciliation latency
|
||||
- `kubemirror:reconcile_rate:5m` - Reconciliation rate (5m window)
|
||||
- `kubemirror:reconcile_errors:rate5m` - Error rate (5m window)
|
||||
- `kubemirror:workqueue_depth:max` - Max workqueue depth
|
||||
|
||||
## Grafana Dashboard
|
||||
|
||||
The dashboard includes the following panels:
|
||||
|
||||
1. **Controller Status** - Up/down status
|
||||
2. **Reconciliation Rate** - Operations per second by type and result
|
||||
3. **Total Workqueue Depth** - Combined queue depth across controllers
|
||||
4. **Reconciliation Latency** - P99 and P95 latency trends
|
||||
5. **Workqueue Depth** - Per-controller queue depth
|
||||
6. **Memory Usage** - Working set vs limits
|
||||
7. **CPU Usage** - CPU utilization percentage
|
||||
8. **Error Rate** - Percentage of failed reconciliations
|
||||
9. **Process Stats** - Goroutines and file descriptors
|
||||
|
||||
## Querying Metrics
|
||||
|
||||
### Using Prometheus UI
|
||||
|
||||
```promql
|
||||
# Total reconciliation rate
|
||||
sum(rate(controller_runtime_reconcile_total[5m])) by (controller, result)
|
||||
|
||||
# Error rate
|
||||
sum(rate(controller_runtime_reconcile_errors_total[5m])) by (controller)
|
||||
|
||||
# P99 latency
|
||||
histogram_quantile(0.99,
|
||||
sum(rate(controller_runtime_reconcile_time_seconds_bucket[5m])) by (le, controller)
|
||||
)
|
||||
|
||||
# Current workqueue depth
|
||||
workqueue_depth{name=~"secret|configmap"}
|
||||
```
|
||||
|
||||
### Using kubectl
|
||||
|
||||
```bash
|
||||
# Port-forward to metrics endpoint
|
||||
kubectl port-forward -n kubemirror-system svc/kubemirror-controller-metrics 8080:8080
|
||||
|
||||
# Curl metrics (raw Prometheus format)
|
||||
curl http://localhost:8080/metrics
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### ServiceMonitor Not Scraping
|
||||
|
||||
Check if Prometheus Operator is configured to discover ServiceMonitors in the kubemirror-system namespace:
|
||||
|
||||
```bash
|
||||
# Check ServiceMonitor status
|
||||
kubectl get servicemonitor -n kubemirror-system
|
||||
|
||||
# Check Prometheus targets
|
||||
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090
|
||||
# Open http://localhost:9090/targets
|
||||
```
|
||||
|
||||
### Alerts Not Firing
|
||||
|
||||
Verify PrometheusRule is loaded:
|
||||
|
||||
```bash
|
||||
# Check PrometheusRule
|
||||
kubectl get prometheusrule -n kubemirror-system
|
||||
|
||||
# Check Prometheus rules
|
||||
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090
|
||||
# Open http://localhost:9090/rules
|
||||
```
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
If alerts fire for high memory:
|
||||
|
||||
1. Check for memory leaks in controller logs
|
||||
2. Increase memory limits in Helm values:
|
||||
```yaml
|
||||
resources:
|
||||
limits:
|
||||
memory: 1Gi
|
||||
```
|
||||
3. Reduce worker threads or max targets if necessary
|
||||
|
||||
### High Reconciliation Latency
|
||||
|
||||
If reconciliation is slow:
|
||||
|
||||
1. Check API server latency: `kubectl get --raw /metrics | grep apiserver_request_duration`
|
||||
2. Increase worker threads in Helm values:
|
||||
```yaml
|
||||
controller:
|
||||
workerThreads: 10
|
||||
```
|
||||
3. Review rate limiting settings if hitting API limits
|
||||
|
||||
## Integration with Alertmanager
|
||||
|
||||
To route KubeMirror alerts to specific channels:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: alertmanager-config
|
||||
namespace: monitoring
|
||||
data:
|
||||
alertmanager.yml: |
|
||||
route:
|
||||
routes:
|
||||
- match:
|
||||
component: kubemirror
|
||||
receiver: kubemirror-team
|
||||
continue: true
|
||||
|
||||
receivers:
|
||||
- name: kubemirror-team
|
||||
slack_configs:
|
||||
- channel: '#kubemirror-alerts'
|
||||
api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Set up alerts** - Deploy PrometheusRule to catch issues early
|
||||
2. **Monitor trends** - Use Grafana dashboard to spot degradation over time
|
||||
3. **Baseline metrics** - Understand normal behavior during low/high load
|
||||
4. **Tune resources** - Adjust CPU/memory based on actual usage patterns
|
||||
5. **Alert fatigue** - Tune alert thresholds to reduce false positives
|
||||
6. **Retention** - Ensure Prometheus retains metrics for at least 7 days
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Prometheus Operator Documentation](https://prometheus-operator.dev/)
|
||||
- [Grafana Dashboard Best Practices](https://grafana.com/docs/grafana/latest/best-practices/best-practices-for-creating-dashboards/)
|
||||
- [Controller Runtime Metrics](https://book.kubebuilder.io/reference/metrics.html)
|
||||
@@ -0,0 +1,678 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": "-- Grafana --",
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "KubeMirror Controller Metrics Dashboard",
|
||||
"editable": true,
|
||||
"gnetId": null,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"text": {},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "up{service=\"kubemirror-controller-metrics\"}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Controller Status",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": true
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "ops"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": ["mean", "last"],
|
||||
"displayMode": "table",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(controller_runtime_reconcile_total{controller=\"secret\"}[5m])) by (result)",
|
||||
"legendFormat": "Secret - {{result}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum(rate(controller_runtime_reconcile_total{controller=\"configmap\"}[5m])) by (result)",
|
||||
"legendFormat": "ConfigMap - {{result}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Reconciliation Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 5
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 10
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"text": {},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(workqueue_depth{name=~\"secret|configmap\"})",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Total Workqueue Depth",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": true
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": ["mean", "max"],
|
||||
"displayMode": "table",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, sum(rate(controller_runtime_reconcile_time_seconds_bucket{controller=\"secret\"}[5m])) by (le))",
|
||||
"legendFormat": "Secret p99",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum(rate(controller_runtime_reconcile_time_seconds_bucket{controller=\"secret\"}[5m])) by (le))",
|
||||
"legendFormat": "Secret p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, sum(rate(controller_runtime_reconcile_time_seconds_bucket{controller=\"configmap\"}[5m])) by (le))",
|
||||
"legendFormat": "ConfigMap p99",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum(rate(controller_runtime_reconcile_time_seconds_bucket{controller=\"configmap\"}[5m])) by (le))",
|
||||
"legendFormat": "ConfigMap p95",
|
||||
"refId": "D"
|
||||
}
|
||||
],
|
||||
"title": "Reconciliation Latency",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": true
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": ["mean", "max"],
|
||||
"displayMode": "table",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "workqueue_depth{name=\"secret\"}",
|
||||
"legendFormat": "Secret",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "workqueue_depth{name=\"configmap\"}",
|
||||
"legendFormat": "ConfigMap",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Workqueue Depth",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": true
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 12
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": ["mean", "max"],
|
||||
"displayMode": "table",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "container_memory_working_set_bytes{pod=~\"kubemirror-.*\",container=\"controller\"}",
|
||||
"legendFormat": "Memory Usage",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "container_spec_memory_limit_bytes{pod=~\"kubemirror-.*\",container=\"controller\"}",
|
||||
"legendFormat": "Memory Limit",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Memory Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": true
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percent"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 12
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": ["mean", "max"],
|
||||
"displayMode": "table",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(container_cpu_usage_seconds_total{pod=~\"kubemirror-.*\",container=\"controller\"}[5m]) * 100",
|
||||
"legendFormat": "CPU Usage",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": true
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.05
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 20
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": ["mean", "max"],
|
||||
"displayMode": "table",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(controller_runtime_reconcile_errors_total{controller=\"secret\"}[5m])) / sum(rate(controller_runtime_reconcile_total{controller=\"secret\"}[5m]))",
|
||||
"legendFormat": "Secret Error Rate",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "sum(rate(controller_runtime_reconcile_errors_total{controller=\"configmap\"}[5m])) / sum(rate(controller_runtime_reconcile_total{controller=\"configmap\"}[5m]))",
|
||||
"legendFormat": "ConfigMap Error Rate",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Error Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": "Prometheus",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"tooltip": false,
|
||||
"viz": false,
|
||||
"legend": false
|
||||
},
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": true
|
||||
},
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 20
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": ["mean", "last"],
|
||||
"displayMode": "table",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "8.0.0",
|
||||
"targets": [
|
||||
{
|
||||
"expr": "process_open_fds{job=\"kubemirror-controller-metrics\"}",
|
||||
"legendFormat": "Open File Descriptors",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "go_goroutines{job=\"kubemirror-controller-metrics\"}",
|
||||
"legendFormat": "Goroutines",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Process Stats",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 27,
|
||||
"style": "dark",
|
||||
"tags": ["kubernetes", "kubemirror", "controller"],
|
||||
"templating": {
|
||||
"list": []
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "KubeMirror Controller",
|
||||
"uid": "kubemirror-controller",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: kubemirror-alerts
|
||||
namespace: kubemirror-system
|
||||
labels:
|
||||
app.kubernetes.io/name: kubemirror
|
||||
app.kubernetes.io/component: monitoring
|
||||
spec:
|
||||
groups:
|
||||
- name: kubemirror.rules
|
||||
interval: 30s
|
||||
rules:
|
||||
# Controller health alerts
|
||||
- alert: KubeMirrorControllerDown
|
||||
expr: up{service="kubemirror-controller-metrics"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "KubeMirror controller is down"
|
||||
description: "KubeMirror controller in namespace {{ $labels.namespace }} has been down for more than 5 minutes."
|
||||
|
||||
- alert: KubeMirrorHighReconcileErrors
|
||||
expr: |
|
||||
rate(controller_runtime_reconcile_errors_total{controller="secret"}[5m]) > 0.1
|
||||
or
|
||||
rate(controller_runtime_reconcile_errors_total{controller="configmap"}[5m]) > 0.1
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "High reconciliation error rate in KubeMirror"
|
||||
description: "KubeMirror controller {{ $labels.controller }} is experiencing high error rate: {{ $value | humanizePercentage }} errors/sec"
|
||||
|
||||
- alert: KubeMirrorReconcileLatencyHigh
|
||||
expr: |
|
||||
histogram_quantile(0.99,
|
||||
rate(controller_runtime_reconcile_time_seconds_bucket{controller=~"secret|configmap"}[5m])
|
||||
) > 5
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "High reconciliation latency in KubeMirror"
|
||||
description: "KubeMirror {{ $labels.controller }} controller p99 latency is {{ $value | humanizeDuration }}"
|
||||
|
||||
- alert: KubeMirrorWorkqueueDepthHigh
|
||||
expr: |
|
||||
workqueue_depth{name=~"secret|configmap"} > 100
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "High workqueue depth in KubeMirror"
|
||||
description: "KubeMirror {{ $labels.name }} workqueue has {{ $value }} items pending for more than 15 minutes"
|
||||
|
||||
- alert: KubeMirrorLeaderElectionLost
|
||||
expr: |
|
||||
leader_election_master_status{name="kubemirror-controller-leader"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: warning
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "KubeMirror lost leader election"
|
||||
description: "KubeMirror controller on pod {{ $labels.pod }} is not the leader"
|
||||
|
||||
# Resource mirror alerts
|
||||
- alert: KubeMirrorHighFailureRate
|
||||
expr: |
|
||||
sum(rate(controller_runtime_reconcile_errors_total{controller=~"secret|configmap"}[5m]))
|
||||
/
|
||||
sum(rate(controller_runtime_reconcile_total{controller=~"secret|configmap"}[5m]))
|
||||
> 0.05
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "High mirror operation failure rate"
|
||||
description: "KubeMirror has {{ $value | humanizePercentage }} failure rate over the last 10 minutes"
|
||||
|
||||
- alert: KubeMirrorMemoryHigh
|
||||
expr: |
|
||||
container_memory_working_set_bytes{pod=~"kubemirror-.*",container="controller"}
|
||||
/
|
||||
container_spec_memory_limit_bytes{pod=~"kubemirror-.*",container="controller"}
|
||||
> 0.9
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "KubeMirror controller high memory usage"
|
||||
description: "KubeMirror controller {{ $labels.pod }} is using {{ $value | humanizePercentage }} of its memory limit"
|
||||
|
||||
- alert: KubeMirrorCPUThrottling
|
||||
expr: |
|
||||
rate(container_cpu_cfs_throttled_seconds_total{pod=~"kubemirror-.*",container="controller"}[5m]) > 0.5
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: kubemirror
|
||||
annotations:
|
||||
summary: "KubeMirror controller is being CPU throttled"
|
||||
description: "KubeMirror controller {{ $labels.pod }} is experiencing CPU throttling: {{ $value | humanizeDuration }}/sec"
|
||||
|
||||
- name: kubemirror.recording
|
||||
interval: 30s
|
||||
rules:
|
||||
# Recording rules for better query performance
|
||||
- record: kubemirror:reconcile_duration_seconds:p99
|
||||
expr: |
|
||||
histogram_quantile(0.99,
|
||||
rate(controller_runtime_reconcile_time_seconds_bucket{controller=~"secret|configmap"}[5m])
|
||||
)
|
||||
|
||||
- record: kubemirror:reconcile_duration_seconds:p95
|
||||
expr: |
|
||||
histogram_quantile(0.95,
|
||||
rate(controller_runtime_reconcile_time_seconds_bucket{controller=~"secret|configmap"}[5m])
|
||||
)
|
||||
|
||||
- record: kubemirror:reconcile_duration_seconds:p50
|
||||
expr: |
|
||||
histogram_quantile(0.50,
|
||||
rate(controller_runtime_reconcile_time_seconds_bucket{controller=~"secret|configmap"}[5m])
|
||||
)
|
||||
|
||||
- record: kubemirror:reconcile_rate:5m
|
||||
expr: |
|
||||
sum(rate(controller_runtime_reconcile_total{controller=~"secret|configmap"}[5m])) by (controller, result)
|
||||
|
||||
- record: kubemirror:reconcile_errors:rate5m
|
||||
expr: |
|
||||
sum(rate(controller_runtime_reconcile_errors_total{controller=~"secret|configmap"}[5m])) by (controller)
|
||||
|
||||
- record: kubemirror:workqueue_depth:max
|
||||
expr: |
|
||||
max(workqueue_depth{name=~"secret|configmap"}) by (name)
|
||||
@@ -0,0 +1,30 @@
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: kubemirror-controller
|
||||
namespace: kubemirror-system
|
||||
labels:
|
||||
app.kubernetes.io/name: kubemirror
|
||||
app.kubernetes.io/component: controller
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kubemirror
|
||||
app.kubernetes.io/component: controller
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
path: /metrics
|
||||
scheme: http
|
||||
scrapeTimeout: 10s
|
||||
relabelings:
|
||||
# Add namespace label
|
||||
- sourceLabels: [__meta_kubernetes_namespace]
|
||||
targetLabel: namespace
|
||||
# Add pod label
|
||||
- sourceLabels: [__meta_kubernetes_pod_name]
|
||||
targetLabel: pod
|
||||
# Add service label
|
||||
- sourceLabels: [__meta_kubernetes_service_name]
|
||||
targetLabel: service
|
||||
Reference in New Issue
Block a user