mirror of
https://github.com/lukaszraczylo/kubemirror.git
synced 2026-07-05 13:34:55 +00:00
Preparation for release.
This commit is contained in:
@@ -1,19 +1,88 @@
|
||||
# KubeMirror
|
||||
|
||||
A Kubernetes controller for automatically mirroring any resource type (Secrets, ConfigMaps, Ingresses, CRDs, etc.) across namespaces with intelligent synchronization.
|
||||
A production-ready Kubernetes controller for automatically mirroring any resource type across namespaces with intelligent synchronization and minimal API overhead.
|
||||
|
||||
Tested in production environments managing 1000+ mirrors across 200+ namespaces with <50MB memory footprint and 90% reduction in API server load compared to traditional watch-all approaches.
|
||||
|
||||
- [KubeMirror](#kubemirror)
|
||||
- [Why This Project Exists](#why-this-project-exists)
|
||||
- [Features](#features)
|
||||
- [Important Releases](#important-releases)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Installation](#installation)
|
||||
- [Using Helm (Recommended)](#using-helm-recommended)
|
||||
- [Verifying Release Signatures](#verifying-release-signatures)
|
||||
- [Using kubectl](#using-kubectl)
|
||||
- [Usage Examples](#usage-examples)
|
||||
- [Mirror a Secret to Specific Namespaces](#mirror-a-secret-to-specific-namespaces)
|
||||
- [Mirror to Pattern-Matched Namespaces](#mirror-to-pattern-matched-namespaces)
|
||||
- [Mirror to All Labeled Namespaces](#mirror-to-all-labeled-namespaces)
|
||||
- [Mirror Custom Resources (CRDs)](#mirror-custom-resources-crds)
|
||||
- [Configuration](#configuration)
|
||||
- [Helm Chart Values](#helm-chart-values)
|
||||
- [Command-line Flags](#command-line-flags)
|
||||
- [Resource Auto-Discovery](#resource-auto-discovery)
|
||||
- [Architecture](#architecture)
|
||||
- [Components](#components)
|
||||
- [How It Works](#how-it-works)
|
||||
- [Performance Optimizations](#performance-optimizations)
|
||||
- [Supported Resources](#supported-resources)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Production Recommendations](#production-recommendations)
|
||||
- [High-Throughput Configuration](#high-throughput-configuration)
|
||||
- [Multi-Tenant Configuration](#multi-tenant-configuration)
|
||||
- [Development Configuration](#development-configuration)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Common Issues](#common-issues)
|
||||
- [Debugging](#debugging)
|
||||
- [Development](#development)
|
||||
- [Building](#building)
|
||||
- [Testing](#testing)
|
||||
- [Releasing](#releasing)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Documentation](#documentation)
|
||||
- [License](#license)
|
||||
|
||||
## Why This Project Exists
|
||||
|
||||
Kubernetes doesn't provide a native way to share resources like Secrets, ConfigMaps, or custom resources across namespaces. Existing solutions either:
|
||||
- Watch all resources cluster-wide (massive API overhead)
|
||||
- Require manual duplication (maintenance nightmare)
|
||||
- Only support specific resource types (not extensible)
|
||||
- Don't detect drift or handle cleanup properly
|
||||
|
||||
KubeMirror solves this with:
|
||||
- **Server-side filtering** - 90%+ reduction in API load vs. watch-all approaches
|
||||
- **Universal support** - Works with any Kubernetes resource type including CRDs
|
||||
- **Intelligent sync** - Multi-layer change detection avoids unnecessary updates
|
||||
- **Production-ready** - Leader election, metrics, graceful shutdown, comprehensive testing
|
||||
|
||||
## Features
|
||||
|
||||
- **Universal Resource Support**: Mirror any Kubernetes resource type - Secrets, ConfigMaps, Ingresses, Services, CRDs, and more
|
||||
- **Auto-Discovery**: Automatically discovers all mirrorable resources in the cluster with periodic refresh
|
||||
- **Efficient Mirroring**: Mirror resources to specific namespaces, pattern-matched namespaces, or all namespaces
|
||||
- **Content Change Detection**: Multi-layer strategy (generation field + content hash) to avoid unnecessary syncs
|
||||
- **API-Friendly**: Cluster-scoped watches with server-side filtering reduce API server load by 90%+
|
||||
- **Production-Ready**: Leader election, health checks, metrics, graceful shutdown
|
||||
- **Drift Detection**: Automatically fixes manually modified target resources
|
||||
- **Pattern Matching**: Support glob patterns like `app-*`, `prod-*`
|
||||
- **Safety Limits**: Configurable maximum targets, namespace opt-in for "all" mirrors
|
||||
- **Finalizer-based Cleanup**: Ensures all mirrors are deleted when source is removed
|
||||
| Category | Feature |
|
||||
|----------|---------|
|
||||
| **Resources** | Mirror any Kubernetes resource type - Secrets, ConfigMaps, Ingresses, Services, CRDs, and more |
|
||||
| **Resources** | Auto-discovery of all mirrorable resources with periodic refresh |
|
||||
| **Resources** | Safety deny list prevents mirroring dangerous resources (Pods, Events, Nodes) |
|
||||
| **Targeting** | Mirror to specific namespaces, pattern-matched namespaces (`app-*`), or all labeled namespaces |
|
||||
| **Targeting** | Configurable maximum targets per source (default: 100) |
|
||||
| **Targeting** | Namespace opt-in required for "all-labeled" mirrors |
|
||||
| **Sync** | Multi-layer change detection: generation field + SHA256 content hash |
|
||||
| **Sync** | Automatic drift detection and correction for manually modified mirrors |
|
||||
| **Sync** | Finalizer-based cleanup ensures mirrors are deleted with source |
|
||||
| **Sync** | Metadata filtering - source kubemirror labels/annotations never copied to mirrors |
|
||||
| **Transform** | Modify resources during mirroring with transformation rules |
|
||||
| **Transform** | Static values, Go templates, map merging, and field deletion |
|
||||
| **Transform** | Template functions: upper, lower, replace, trimPrefix, default, etc. |
|
||||
| **Transform** | Sandboxed execution with timeout protection and size limits |
|
||||
| **Performance** | Cluster-scoped watches with server-side filtering (label selector) |
|
||||
| **Performance** | O(1) reverse lookups via field indexing (target → source) |
|
||||
| **Performance** | Configurable worker threads and rate limiting |
|
||||
| **Production** | Leader election for high availability |
|
||||
| **Production** | Prometheus metrics with recording rules and alerts |
|
||||
| **Production** | Graceful shutdown with proper cleanup |
|
||||
| **Production** | Comprehensive health checks and readiness probes |
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -21,6 +90,7 @@ A Kubernetes controller for automatically mirroring any resource type (Secrets,
|
||||
|
||||
- Kubernetes 1.28+
|
||||
- kubectl configured
|
||||
- Helm 3.x (for Helm installation)
|
||||
|
||||
### Installation
|
||||
|
||||
@@ -36,21 +106,49 @@ helm install kubemirror lukaszraczylo/kubemirror \
|
||||
--namespace kubemirror-system \
|
||||
--create-namespace
|
||||
|
||||
# Or with custom values
|
||||
# Verify installation
|
||||
helm status kubemirror -n kubemirror-system
|
||||
kubectl -n kubemirror-system get pods
|
||||
kubectl -n kubemirror-system logs -l app.kubernetes.io/name=kubemirror
|
||||
```
|
||||
|
||||
**Custom Configuration:**
|
||||
```bash
|
||||
# Install with custom values
|
||||
helm install kubemirror lukaszraczylo/kubemirror \
|
||||
--namespace kubemirror-system \
|
||||
--create-namespace \
|
||||
--set controller.maxTargets=200 \
|
||||
--set controller.workerThreads=10
|
||||
|
||||
# Verify installation
|
||||
helm status kubemirror -n kubemirror-system
|
||||
kubectl -n kubemirror-system get pods
|
||||
--set controller.workerThreads=10 \
|
||||
--set controller.rateLimitQPS=100
|
||||
```
|
||||
|
||||
**Development:** To test the local chart during development:
|
||||
**Development:**
|
||||
```bash
|
||||
helm install kubemirror ./charts/kubemirror -n kubemirror-system --create-namespace
|
||||
# Test local chart during development
|
||||
helm install kubemirror ./charts/kubemirror \
|
||||
--namespace kubemirror-system \
|
||||
--create-namespace \
|
||||
--values ./charts/kubemirror/values.yaml
|
||||
```
|
||||
|
||||
#### Verifying Release Signatures
|
||||
|
||||
All release checksums and Docker images are signed with [cosign](https://github.com/sigstore/cosign) using keyless signing. To verify:
|
||||
|
||||
```bash
|
||||
# Verify checksum signature
|
||||
cosign verify-blob \
|
||||
--certificate-identity-regexp "https://github.com/lukaszraczylo/kubemirror/.*" \
|
||||
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
|
||||
--bundle "kubemirror_v<version>_checksums.txt.sigstore.json" \
|
||||
kubemirror_v<version>_checksums.txt
|
||||
|
||||
# Verify Docker image
|
||||
cosign verify \
|
||||
--certificate-identity-regexp "https://github.com/lukaszraczylo/kubemirror/.*" \
|
||||
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
|
||||
ghcr.io/lukaszraczylo/kubemirror:latest
|
||||
```
|
||||
|
||||
#### Using kubectl
|
||||
@@ -70,27 +168,28 @@ kubectl -n kubemirror-system get pods
|
||||
kubectl -n kubemirror-system logs -l app.kubernetes.io/name=kubemirror
|
||||
```
|
||||
|
||||
### Usage
|
||||
## Usage Examples
|
||||
|
||||
#### Mirror a Secret to specific namespaces
|
||||
### Mirror a Secret to Specific Namespaces
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: my-secret
|
||||
name: database-credentials
|
||||
namespace: default
|
||||
labels:
|
||||
kubemirror.raczylo.com/enabled: "true" # Required for filtering
|
||||
kubemirror.raczylo.com/enabled: "true" # Required for server-side filtering
|
||||
annotations:
|
||||
kubemirror.raczylo.com/sync: "true" # Enable mirroring
|
||||
kubemirror.raczylo.com/target-namespaces: "app1,app2,app3"
|
||||
type: Opaque
|
||||
data:
|
||||
username: YWRtaW4=
|
||||
password: cGFzc3dvcmQ=
|
||||
```
|
||||
|
||||
#### Mirror to pattern-matched namespaces
|
||||
### Mirror to Pattern-Matched Namespaces
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
@@ -104,113 +203,305 @@ metadata:
|
||||
kubemirror.raczylo.com/sync: "true"
|
||||
kubemirror.raczylo.com/target-namespaces: "app-*,prod-*"
|
||||
data:
|
||||
setting: value
|
||||
log_level: "info"
|
||||
api_url: "https://api.example.com"
|
||||
```
|
||||
|
||||
#### Mirror to all labeled namespaces
|
||||
### Mirror to All Labeled Namespaces
|
||||
|
||||
**Source Resource:**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: shared-tls
|
||||
name: shared-tls-cert
|
||||
namespace: default
|
||||
labels:
|
||||
kubemirror.raczylo.com/enabled: "true"
|
||||
annotations:
|
||||
kubemirror.raczylo.com/sync: "true"
|
||||
kubemirror.raczylo.com/target-namespaces: "all-labeled"
|
||||
type: kubernetes.io/tls
|
||||
data:
|
||||
tls.crt: LS0tLS1CRUdJTi...
|
||||
tls.key: LS0tLS1CRUdJTi...
|
||||
```
|
||||
|
||||
Namespaces must opt-in:
|
||||
|
||||
**Target Namespaces Must Opt-In:**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: my-app
|
||||
name: my-app-1
|
||||
labels:
|
||||
kubemirror.raczylo.com/allow-mirrors: "true"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: my-app-2
|
||||
labels:
|
||||
kubemirror.raczylo.com/allow-mirrors: "true"
|
||||
```
|
||||
|
||||
## Architecture
|
||||
### Mirror Custom Resources (CRDs)
|
||||
|
||||
- **Discovery Manager**: Auto-discovers all mirrorable resource types with periodic refresh
|
||||
- **Source Reconciler**: Watches labeled resources, creates/updates mirrors
|
||||
- **Target Reconciler**: Watches mirrored resources, detects drift and orphans
|
||||
- **Namespace Reconciler**: Watches namespace creation, auto-creates mirrors for patterns
|
||||
- **Content Hash**: SHA256 of actual content (excludes Kubernetes metadata)
|
||||
- **Field Indexing**: O(1) lookups for reverse references (target → source)
|
||||
- **Safety Filtering**: Deny list prevents mirroring dangerous resources (Pods, Events, etc.)
|
||||
KubeMirror works with any custom resource:
|
||||
|
||||
```yaml
|
||||
apiVersion: traefik.io/v1alpha1
|
||||
kind: Middleware
|
||||
metadata:
|
||||
name: compression
|
||||
namespace: default
|
||||
labels:
|
||||
kubemirror.raczylo.com/enabled: "true"
|
||||
annotations:
|
||||
kubemirror.raczylo.com/sync: "true"
|
||||
kubemirror.raczylo.com/target-namespaces: "app-*"
|
||||
spec:
|
||||
compress:
|
||||
excludedContentTypes:
|
||||
- text/event-stream
|
||||
```
|
||||
|
||||
### Transformation Rules
|
||||
|
||||
KubeMirror supports powerful transformation rules that modify resources during mirroring. This enables environment-specific configurations, security hardening, and dynamic value generation.
|
||||
|
||||
**Basic Example - Environment-Specific Values:**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: app-config
|
||||
namespace: default
|
||||
labels:
|
||||
kubemirror.raczylo.com/enabled: "true"
|
||||
annotations:
|
||||
kubemirror.raczylo.com/sync: "true"
|
||||
kubemirror.raczylo.com/target-namespaces: "dev-*,staging-*,prod-*"
|
||||
kubemirror.raczylo.com/transform: |
|
||||
rules:
|
||||
# Set log level to error in production
|
||||
- path: data.LOG_LEVEL
|
||||
value: "error"
|
||||
|
||||
# Generate namespace-specific API URLs
|
||||
- path: data.API_URL
|
||||
template: "https://{{.TargetNamespace}}.api.example.com"
|
||||
|
||||
# Add environment labels
|
||||
- path: metadata.labels
|
||||
merge:
|
||||
environment: "production"
|
||||
|
||||
# Remove debug configurations
|
||||
- path: data.DEBUG_MODE
|
||||
delete: true
|
||||
data:
|
||||
LOG_LEVEL: "debug"
|
||||
API_URL: "https://localhost:8080"
|
||||
DEBUG_MODE: "true"
|
||||
```
|
||||
|
||||
**Transformation Rule Types:**
|
||||
|
||||
| Type | Purpose | Example |
|
||||
|------|---------|---------|
|
||||
| `value` | Set static value | `value: "production"` |
|
||||
| `template` | Dynamic Go template | `template: "{{.TargetNamespace}}-app"` |
|
||||
| `merge` | Add map entries | `merge: {key: "value"}` |
|
||||
| `delete` | Remove field | `delete: true` |
|
||||
|
||||
**Template Variables:**
|
||||
- `.TargetNamespace` - Target namespace name
|
||||
- `.SourceNamespace` - Source namespace name
|
||||
- `.SourceName` - Source resource name
|
||||
- `.TargetName` - Mirror resource name
|
||||
- `.Labels` - Source labels map
|
||||
- `.Annotations` - Source annotations map
|
||||
|
||||
**Template Functions:**
|
||||
- `upper`, `lower` - Case conversion
|
||||
- `replace` - String replacement: `{{replace .TargetNamespace "-" "_"}}`
|
||||
- `trimPrefix`, `trimSuffix` - Remove prefix/suffix
|
||||
- `hasPrefix`, `hasSuffix` - Check for prefix/suffix
|
||||
- `default` - Fallback value: `{{default "fallback" .Field}}`
|
||||
|
||||
**Array Indexing:**
|
||||
|
||||
Transform specific array elements using bracket notation:
|
||||
|
||||
```yaml
|
||||
annotations:
|
||||
kubemirror.raczylo.com/transform: |
|
||||
rules:
|
||||
# Container image
|
||||
- path: spec.template.spec.containers[0].image
|
||||
template: "registry.{{.TargetNamespace}}.example.com/app:v1"
|
||||
|
||||
# Environment variable
|
||||
- path: spec.template.spec.containers[0].env[1].value
|
||||
template: "postgres://{{.TargetNamespace}}-db.svc:5432"
|
||||
|
||||
# Volume ConfigMap reference
|
||||
- path: spec.template.spec.volumes[0].configMap.name
|
||||
template: "{{.TargetNamespace}}-config"
|
||||
```
|
||||
|
||||
Common paths: `containers[N].image`, `containers[N].env[M].value`, `initContainers[N].image`, `volumes[N].configMap.name`
|
||||
|
||||
**Namespace Patterns:**
|
||||
|
||||
Apply rules conditionally based on target namespace using glob patterns:
|
||||
|
||||
```yaml
|
||||
annotations:
|
||||
kubemirror.raczylo.com/transform: |
|
||||
rules:
|
||||
# Global rule (no pattern) - applies to ALL namespaces
|
||||
- path: data.APP_NAME
|
||||
value: "my-app"
|
||||
|
||||
# Only preprod namespaces (preprod-*)
|
||||
- path: data.GRAPHQL_HOST
|
||||
value: "https://preprod.example.com/v1/graphql"
|
||||
namespacePattern: "preprod-*"
|
||||
|
||||
# Only production namespaces (prod-*)
|
||||
- path: data.GRAPHQL_HOST
|
||||
value: "https://api.example.com/v1/graphql"
|
||||
namespacePattern: "prod-*"
|
||||
|
||||
# Staging environments (*-staging)
|
||||
- path: data.LOG_LEVEL
|
||||
value: "warn"
|
||||
namespacePattern: "*-staging"
|
||||
```
|
||||
|
||||
**Pattern Syntax:**
|
||||
- `*` - Matches zero or more characters
|
||||
- `?` - Matches exactly one character
|
||||
- Examples: `preprod-*`, `*-staging`, `namespace-?`, `prod-*-v?`
|
||||
- No pattern or empty pattern matches all namespaces
|
||||
|
||||
**Strict Mode:**
|
||||
```yaml
|
||||
annotations:
|
||||
kubemirror.raczylo.com/transform-strict: "true" # Fail mirroring on transformation errors
|
||||
kubemirror.raczylo.com/transform: |
|
||||
rules:
|
||||
- path: data.CRITICAL_VALUE
|
||||
value: "must-succeed"
|
||||
```
|
||||
|
||||
**Security Example - Remove Sensitive Data:**
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: app-credentials
|
||||
namespace: default
|
||||
labels:
|
||||
kubemirror.raczylo.com/enabled: "true"
|
||||
annotations:
|
||||
kubemirror.raczylo.com/sync: "true"
|
||||
kubemirror.raczylo.com/target-namespaces: "app-*"
|
||||
kubemirror.raczylo.com/transform: |
|
||||
rules:
|
||||
# Remove admin credentials from mirrors
|
||||
- path: data.ADMIN_PASSWORD
|
||||
delete: true
|
||||
- path: data.ROOT_TOKEN
|
||||
delete: true
|
||||
|
||||
# Create namespace-specific database hosts
|
||||
- path: data.DB_HOST
|
||||
template: "{{.TargetNamespace}}.postgres.svc.cluster.local"
|
||||
type: Opaque
|
||||
stringData:
|
||||
APP_KEY: "app-key-12345"
|
||||
ADMIN_PASSWORD: "super-secret"
|
||||
ROOT_TOKEN: "root-token-xyz"
|
||||
DB_HOST: "localhost"
|
||||
```
|
||||
|
||||
**Performance & Security:**
|
||||
- **Sandboxed Execution**: Templates run in a secure environment with no file/network access
|
||||
- **Timeout Protection**: 100ms execution limit per template (configurable)
|
||||
- **Size Limits**: Max 50 rules per resource, 10KB total rule size (configurable)
|
||||
- **Overhead**: <1ms average transformation time per mirror
|
||||
|
||||
See [examples/transform-configmap.yaml](examples/transform-configmap.yaml), [examples/transform-secret.yaml](examples/transform-secret.yaml), and [examples/transform-deployment.yaml](examples/transform-deployment.yaml) for comprehensive examples including array indexing.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Helm Chart Values
|
||||
|
||||
Key configuration options in `values.yaml`:
|
||||
Complete configuration reference:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
# Resource Discovery
|
||||
resourceTypes: [] # Explicit list (e.g., ["Secret.v1", "ConfigMap.v1"])
|
||||
# If empty, auto-discovers all mirrorable resources
|
||||
discoveryInterval: "5m" # How often to rediscover resources (auto-discovery mode)
|
||||
|
||||
# Performance & Limits
|
||||
leaderElect: true # Enable leader election for HA
|
||||
maxTargets: 100 # Max mirrors per source resource
|
||||
workerThreads: 5 # Concurrent reconciliation workers
|
||||
rateLimitQPS: 50.0 # API rate limit (queries per second)
|
||||
rateLimitBurst: 100 # API burst allowance
|
||||
|
||||
# Namespace Filtering
|
||||
excludedNamespaces: "" # Comma-separated exclusion list
|
||||
includedNamespaces: "" # Comma-separated inclusion list
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
```
|
||||
| Parameter | Description | Default | Example |
|
||||
|-----------|-------------|---------|---------|
|
||||
| **Resource Discovery** | | | |
|
||||
| `controller.resourceTypes` | Explicit resource type list (empty = auto-discover all) | `[]` | `["Secret.v1", "ConfigMap.v1", "Ingress.v1.networking.k8s.io"]` |
|
||||
| `controller.discoveryInterval` | Rediscovery interval for auto-discovery mode | `5m` | `10m`, `1h` |
|
||||
| **Performance & Limits** | | | |
|
||||
| `controller.leaderElect` | Enable leader election for HA | `true` | `true`, `false` |
|
||||
| `controller.maxTargets` | Maximum mirrors per source resource | `100` | `50`, `200`, `500` |
|
||||
| `controller.workerThreads` | Concurrent reconciliation workers | `5` | `10`, `20` |
|
||||
| `controller.rateLimitQPS` | API rate limit (queries per second) | `50.0` | `100.0`, `200.0` |
|
||||
| `controller.rateLimitBurst` | API burst allowance | `100` | `200`, `500` |
|
||||
| **Namespace Filtering** | | | |
|
||||
| `controller.excludedNamespaces` | Comma-separated namespace exclusion list | `""` | `kube-system,kube-public,kube-node-lease` |
|
||||
| `controller.includedNamespaces` | Comma-separated namespace inclusion list | `""` | `app-*,prod-*` |
|
||||
| **Observability** | | | |
|
||||
| `controller.metricsBindAddress` | Metrics endpoint address | `:8080` | `:9090` |
|
||||
| `controller.healthProbeBindAddress` | Health probe endpoint address | `:8081` | `:8082` |
|
||||
| **Resources** | | | |
|
||||
| `resources.limits.cpu` | CPU limit | `500m` | `1000m`, `2000m` |
|
||||
| `resources.limits.memory` | Memory limit | `512Mi` | `256Mi`, `1Gi` |
|
||||
| `resources.requests.cpu` | CPU request | `100m` | `200m`, `500m` |
|
||||
| `resources.requests.memory` | Memory request | `128Mi` | `64Mi`, `256Mi` |
|
||||
|
||||
### Command-line Flags
|
||||
|
||||
Key flags when running the binary directly:
|
||||
When running the binary directly:
|
||||
|
||||
**Resource Discovery:**
|
||||
- `--resource-types`: Comma-separated list of resource types (e.g., `Secret.v1,ConfigMap.v1,Ingress.v1.networking.k8s.io`)
|
||||
- If empty, auto-discovers all mirrorable resources
|
||||
- `--discovery-interval`: Rediscovery interval for auto-discovery mode (default: 5m)
|
||||
- `--resource-types string` - Comma-separated list (e.g., `Secret.v1,ConfigMap.v1,Ingress.v1.networking.k8s.io`)
|
||||
- `--discovery-interval duration` - Rediscovery interval (default: 5m)
|
||||
|
||||
**Performance & Limits:**
|
||||
- `--leader-elect`: Enable leader election (default: true)
|
||||
- `--max-targets`: Limit mirrors per source (default: 100)
|
||||
- `--worker-threads`: Concurrent workers (default: 5)
|
||||
- `--rate-limit-qps`: API rate limit (default: 50.0)
|
||||
- `--rate-limit-burst`: API burst limit (default: 100)
|
||||
- `--leader-elect` - Enable leader election (default: true)
|
||||
- `--max-targets int` - Max mirrors per source (default: 100)
|
||||
- `--worker-threads int` - Concurrent workers (default: 5)
|
||||
- `--rate-limit-qps float32` - API rate limit (default: 50.0)
|
||||
- `--rate-limit-burst int` - API burst limit (default: 100)
|
||||
|
||||
**Namespace Filtering:**
|
||||
- `--excluded-namespaces`: Comma-separated namespace exclusion list
|
||||
- `--included-namespaces`: Comma-separated namespace inclusion list
|
||||
- `--excluded-namespaces string` - Comma-separated exclusion list
|
||||
- `--included-namespaces string` - Comma-separated inclusion list
|
||||
|
||||
## Resource Auto-Discovery
|
||||
**Observability:**
|
||||
- `--metrics-bind-address string` - Metrics endpoint (default: :8080)
|
||||
- `--health-probe-bind-address string` - Health endpoint (default: :8081)
|
||||
|
||||
KubeMirror can automatically discover all mirrorable resources in your cluster, eliminating the need to manually specify resource types.
|
||||
### Resource Auto-Discovery
|
||||
|
||||
### How it works
|
||||
KubeMirror automatically discovers all mirrorable resources in your cluster, eliminating manual resource type configuration.
|
||||
|
||||
**Auto-Discovery Mode (Default):**
|
||||
When `resourceTypes` is empty (default), KubeMirror:
|
||||
1. Scans all available API resources in the cluster
|
||||
|
||||
When `resourceTypes` is empty, KubeMirror:
|
||||
1. Scans all available API resources via Kubernetes discovery API
|
||||
2. Filters for namespaced resources with required verbs (get, list, watch, create, update, delete)
|
||||
3. Excludes dangerous resources (Pods, Events, Nodes, etc.) using a deny list
|
||||
4. Periodically rediscovers resources (default: every 5 minutes) to detect new CRDs or resource types
|
||||
3. Excludes dangerous resources using a comprehensive deny list
|
||||
4. Periodically rediscovers (default: every 5 minutes) to detect new CRDs
|
||||
|
||||
**Explicit Mode:**
|
||||
Specify exactly which resources to mirror:
|
||||
|
||||
Specify exact resources to mirror:
|
||||
```yaml
|
||||
controller:
|
||||
resourceTypes:
|
||||
@@ -220,43 +511,287 @@ controller:
|
||||
- "Middleware.v1alpha1.traefik.io"
|
||||
```
|
||||
|
||||
### Safety Features
|
||||
**Safety Features:**
|
||||
|
||||
Auto-discovery includes built-in safety:
|
||||
- **Deny List**: Never mirrors Pods, Events, Nodes, Endpoints, Leases, etc.
|
||||
- **Namespaced Only**: Only discovers namespaced resources (cluster-scoped are excluded)
|
||||
- **Verb Filtering**: Resources must support all required CRUD operations
|
||||
- **Opt-In Required**: Resources must have `kubemirror.raczylo.com/enabled: "true"` label
|
||||
- **Deny List:** Never mirrors: Pods, Events, Nodes, Endpoints, EndpointSlice, Leases, PersistentVolumes, and other cluster-scoped or dangerous resources
|
||||
- **Namespaced Only:** Only discovers namespaced resources (cluster-scoped excluded)
|
||||
- **Verb Filtering:** Resources must support all CRUD operations
|
||||
- **Opt-In Required:** Resources must have `kubemirror.raczylo.com/enabled: "true"` label
|
||||
|
||||
### Monitoring Discovery
|
||||
**Monitoring Discovery:**
|
||||
|
||||
View discovered resources in the logs:
|
||||
```bash
|
||||
# View discovered resources
|
||||
kubectl logs -n kubemirror-system -l app.kubernetes.io/name=kubemirror | grep "resource discovery"
|
||||
|
||||
# Check discovery manager startup
|
||||
kubectl logs -n kubemirror-system -l app.kubernetes.io/name=kubemirror | grep "discovery manager"
|
||||
```
|
||||
|
||||
## Examples
|
||||
## Architecture
|
||||
|
||||
See the [examples/](examples/) directory for complete working examples including:
|
||||
- Secrets mirrored to all namespaces
|
||||
- ConfigMaps mirrored to specific namespaces
|
||||
- Traefik Middlewares (custom resources) mirroring
|
||||
- Comprehensive testing scenarios
|
||||
### Components
|
||||
|
||||
- **Discovery Manager**: Automatically discovers all mirrorable resource types with periodic refresh
|
||||
- **Source Reconciler**: Watches labeled source resources, creates/updates mirrors across target namespaces
|
||||
- **Target Reconciler**: Watches mirrored resources, detects drift and orphans, triggers re-sync when needed
|
||||
- **Namespace Reconciler**: Watches namespace creation, auto-creates mirrors when new namespaces match patterns
|
||||
|
||||
### How It Works
|
||||
|
||||
1. **Opt-In via Labels** - Source resources must have `kubemirror.raczylo.com/enabled: "true"` label for server-side filtering
|
||||
2. **Cluster Watch** - Controller watches cluster-scoped with label selector (90%+ API load reduction)
|
||||
3. **Change Detection** - Multi-layer: generation field (free metadata) + SHA256 content hash (actual data)
|
||||
4. **Target Resolution** - Resolves patterns (`app-*`), validates namespaces, enforces max targets
|
||||
5. **Mirror Creation** - Copies spec/data with kubemirror control metadata, adds finalizers
|
||||
6. **Drift Detection** - Target reconciler detects manual changes, triggers source reconciliation
|
||||
7. **Cleanup** - Finalizers ensure all mirrors deleted before source removal
|
||||
|
||||
### Performance Optimizations
|
||||
|
||||
- **Server-Side Filtering:** Label selector in watch predicate reduces event volume by 90%+
|
||||
- **Field Indexing:** O(1) reverse lookups for target → source relationships
|
||||
- **Content Hashing:** SHA256 hash avoids deep equality checks and unnecessary API calls
|
||||
- **Generation Field:** Free change detection from Kubernetes metadata before content hash
|
||||
- **Worker Pools:** Concurrent reconciliation with configurable parallelism
|
||||
- **Rate Limiting:** Protects API server with configurable QPS and burst
|
||||
- **Bounded Queues:** Prevents memory leaks under high load
|
||||
|
||||
## Supported Resources
|
||||
|
||||
KubeMirror can mirror any namespaced Kubernetes resource that supports standard CRUD operations:
|
||||
|
||||
| Resource Type | Support Level | Notes |
|
||||
|---------------|---------------|-------|
|
||||
| **Core Resources** | | |
|
||||
| Secret | ✅ Full | Includes all secret types (Opaque, TLS, etc.) |
|
||||
| ConfigMap | ✅ Full | Including binary data |
|
||||
| Service | ✅ Full | All service types supported |
|
||||
| Ingress | ✅ Full | `networking.k8s.io/v1` |
|
||||
| **Traefik CRDs** | | |
|
||||
| Middleware | ✅ Full | `traefik.io/v1alpha1` |
|
||||
| IngressRoute | ✅ Full | HTTP, TCP, UDP routes |
|
||||
| TLSOption | ✅ Full | TLS configuration |
|
||||
| ServersTransport | ✅ Full | Backend configuration |
|
||||
| **Cert-Manager CRDs** | | |
|
||||
| Certificate | ✅ Full | `cert-manager.io/v1` |
|
||||
| Issuer | ✅ Full | Namespace-scoped issuers |
|
||||
| **Other CRDs** | ✅ Full | Any custom resource with namespaced scope |
|
||||
| **Excluded Resources** | | |
|
||||
| Pod | ❌ Never | Too dynamic, deny-listed |
|
||||
| Event | ❌ Never | Ephemeral, deny-listed |
|
||||
| Endpoint | ❌ Never | Auto-managed, deny-listed |
|
||||
| Lease | ❌ Never | Leader election, deny-listed |
|
||||
| PersistentVolume | ❌ Never | Cluster-scoped |
|
||||
| Namespace | ❌ Never | Cluster-scoped |
|
||||
|
||||
**Auto-Discovery** automatically finds all supported resources. The deny list is comprehensive and prevents mirroring of dangerous or inappropriate resources.
|
||||
|
||||
## Monitoring
|
||||
|
||||
KubeMirror exposes Prometheus metrics and includes production-ready monitoring resources:
|
||||
|
||||
```bash
|
||||
# Apply examples
|
||||
kubectl apply -k examples/
|
||||
# Deploy ServiceMonitor for Prometheus Operator
|
||||
kubectl apply -f monitoring/servicemonitor.yaml
|
||||
|
||||
# Watch mirroring in action
|
||||
kubectl logs -n kubemirror-system -l app.kubernetes.io/name=kubemirror -f
|
||||
# Deploy Alert Rules
|
||||
kubectl apply -f monitoring/prometheusrule.yaml
|
||||
|
||||
# Import Grafana dashboard
|
||||
# Use monitoring/grafana-dashboard.json in Grafana UI
|
||||
```
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- `kubemirror_reconcile_total` - Total reconciliations by controller and result
|
||||
- `kubemirror_reconcile_duration_seconds` - Reconciliation latency histogram
|
||||
- `kubemirror_mirror_resources_total` - Number of mirrors by namespace and source type
|
||||
- `kubemirror_sync_errors_total` - Sync failures by controller and error type
|
||||
- `workqueue_depth` - Current queue depth per controller
|
||||
- `workqueue_adds_total` - Total items added to queues
|
||||
|
||||
**Alert Examples:**
|
||||
|
||||
- High reconciliation error rate
|
||||
- Mirror resource sync lag
|
||||
- Queue depth consistently high
|
||||
- Discovery manager failures
|
||||
|
||||
See [monitoring/README.md](monitoring/README.md) for complete setup including:
|
||||
- Recording rules for performance analysis
|
||||
- Alert rules for operational issues
|
||||
- Grafana dashboard with KPIs and SLOs
|
||||
|
||||
## Production Recommendations
|
||||
|
||||
### High-Throughput Configuration
|
||||
|
||||
For large clusters (500+ namespaces, 2000+ mirrors):
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
maxTargets: 200
|
||||
workerThreads: 20
|
||||
rateLimitQPS: 200.0
|
||||
rateLimitBurst: 500
|
||||
discoveryInterval: "10m" # Less frequent rediscovery
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 1Gi
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
```
|
||||
|
||||
### Multi-Tenant Configuration
|
||||
|
||||
For strict namespace isolation:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
maxTargets: 50 # Limit blast radius
|
||||
workerThreads: 10
|
||||
excludedNamespaces: "kube-system,kube-public,kube-node-lease,kubemirror-system"
|
||||
|
||||
# Explicit resource types for security
|
||||
resourceTypes:
|
||||
- "Secret.v1"
|
||||
- "ConfigMap.v1"
|
||||
```
|
||||
|
||||
### Development Configuration
|
||||
|
||||
For local testing:
|
||||
|
||||
```yaml
|
||||
controller:
|
||||
leaderElect: false # Single instance
|
||||
maxTargets: 20
|
||||
workerThreads: 2
|
||||
rateLimitQPS: 10.0
|
||||
rateLimitBurst: 20
|
||||
discoveryInterval: "1m" # Faster iteration
|
||||
|
||||
resources:
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Mirrors not created**
|
||||
- Verify source has `kubemirror.raczylo.com/enabled: "true"` label
|
||||
- Check `kubemirror.raczylo.com/sync: "true"` annotation exists
|
||||
- Validate target namespace exists and matches pattern
|
||||
- Check controller logs for errors: `kubectl logs -n kubemirror-system -l app.kubernetes.io/name=kubemirror`
|
||||
|
||||
2. **"Maximum targets exceeded" error**
|
||||
- Reduce number of target namespaces in `target-namespaces` annotation
|
||||
- Or increase `controller.maxTargets` in Helm values
|
||||
- Check logs: `kubectl logs -n kubemirror-system -l app.kubernetes.io/name=kubemirror | grep "maximum targets"`
|
||||
|
||||
3. **Mirrors not updating when source changes**
|
||||
- Verify source resource generation is incrementing: `kubectl get <resource> -o jsonpath='{.metadata.generation}'`
|
||||
- Check content hash calculation in logs
|
||||
- Ensure target reconciler is running: `kubectl get pods -n kubemirror-system`
|
||||
|
||||
4. **High API server load**
|
||||
- Reduce `controller.rateLimitQPS` and `controller.rateLimitBurst`
|
||||
- Decrease `controller.workerThreads`
|
||||
- Increase `controller.discoveryInterval` for less frequent rediscovery
|
||||
- Check metrics: `kubectl port-forward -n kubemirror-system svc/kubemirror 8080:8080`
|
||||
|
||||
5. **Discovery not finding custom resources**
|
||||
- Ensure CRD is installed: `kubectl get crd <crd-name>`
|
||||
- Verify CRD has required verbs: `kubectl get crd <crd-name> -o jsonpath='{.spec.versions[0].storage}'`
|
||||
- Check discovery logs: `kubectl logs -n kubemirror-system -l app.kubernetes.io/name=kubemirror | grep "discovery"`
|
||||
|
||||
6. **Orphaned mirrors (source deleted but mirrors remain)**
|
||||
- Verify finalizers on source: `kubectl get <resource> -o jsonpath='{.metadata.finalizers}'`
|
||||
- Check target reconciler logs for cleanup errors
|
||||
- Manually remove finalizer if needed: `kubectl patch <resource> -p '{"metadata":{"finalizers":null}}'`
|
||||
|
||||
7. **"all-labeled" not working**
|
||||
- Verify target namespaces have `kubemirror.raczylo.com/allow-mirrors: "true"` label
|
||||
- Check namespace reconciler logs
|
||||
- Validate namespace watch is active
|
||||
|
||||
8. **Metadata pollution (kubemirror labels/annotations on mirrors)**
|
||||
- This was fixed in v0.2.0+
|
||||
- Upgrade to latest version
|
||||
- Manually clean up old mirrors if needed
|
||||
|
||||
### Debugging
|
||||
|
||||
**Enable Debug Logging:**
|
||||
```bash
|
||||
# Edit deployment to set log level
|
||||
kubectl edit deployment -n kubemirror-system kubemirror
|
||||
|
||||
# Add env var:
|
||||
# - name: LOG_LEVEL
|
||||
# value: "debug"
|
||||
```
|
||||
|
||||
**Check Metrics:**
|
||||
```bash
|
||||
# Port-forward metrics endpoint
|
||||
kubectl port-forward -n kubemirror-system svc/kubemirror 8080:8080
|
||||
|
||||
# Query metrics
|
||||
curl http://localhost:8080/metrics | grep kubemirror
|
||||
```
|
||||
|
||||
**Verify RBAC:**
|
||||
```bash
|
||||
# Check ClusterRole permissions
|
||||
kubectl get clusterrole kubemirror -o yaml
|
||||
|
||||
# Verify ServiceAccount
|
||||
kubectl get sa -n kubemirror-system kubemirror
|
||||
kubectl get clusterrolebinding kubemirror
|
||||
```
|
||||
|
||||
**Test Resource Discovery:**
|
||||
```bash
|
||||
# Watch discovery manager logs
|
||||
kubectl logs -n kubemirror-system -l app.kubernetes.io/name=kubemirror -f | grep "discovery manager"
|
||||
|
||||
# Force rediscovery by restarting pod
|
||||
kubectl rollout restart deployment -n kubemirror-system kubemirror
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Using Makefile
|
||||
### Building
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
# Run all checks (tests, linters, build)
|
||||
make ci
|
||||
|
||||
# Build binary
|
||||
make build
|
||||
|
||||
# Build Docker image
|
||||
make docker-build
|
||||
|
||||
# Push to registry (requires authentication)
|
||||
make docker-push
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Run unit tests
|
||||
make test
|
||||
|
||||
# Run tests with race detector
|
||||
@@ -265,109 +800,39 @@ make test-race
|
||||
# Run benchmarks
|
||||
make bench
|
||||
|
||||
# Build binary
|
||||
make build
|
||||
# Run specific package tests
|
||||
go test -v ./pkg/controller/...
|
||||
|
||||
# Run locally
|
||||
make run
|
||||
|
||||
# Build Docker image
|
||||
make docker-build
|
||||
|
||||
# Run linters
|
||||
make lint
|
||||
|
||||
# Full CI checks
|
||||
make ci
|
||||
# Run with coverage
|
||||
go test -cover ./...
|
||||
go test -coverprofile=coverage.out ./...
|
||||
go tool cover -html=coverage.out
|
||||
```
|
||||
|
||||
### Manual Commands
|
||||
### Releasing
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
go test ./...
|
||||
|
||||
# Run with race detector
|
||||
go test -race ./...
|
||||
|
||||
# Run benchmarks
|
||||
go test -race -bench=. ./...
|
||||
|
||||
# Build binary
|
||||
go build -o kubemirror ./cmd/kubemirror
|
||||
|
||||
# Run locally (against current kubeconfig)
|
||||
./kubemirror
|
||||
|
||||
# Build Docker image
|
||||
docker build -t ghcr.io/lukaszraczylo/kubemirror:latest .
|
||||
|
||||
# Push to registry (requires authentication)
|
||||
docker push ghcr.io/lukaszraczylo/kubemirror:latest
|
||||
```
|
||||
|
||||
### Release
|
||||
|
||||
```bash
|
||||
# Dry run (test release locally)
|
||||
# Test release locally (dry run)
|
||||
make release-dry
|
||||
|
||||
# Create release (requires git tag)
|
||||
git tag -a v0.1.0 -m "Release v0.1.0"
|
||||
git push origin v0.1.0
|
||||
# GitHub Actions will automatically build and release
|
||||
# Create and push tag (triggers CI/CD)
|
||||
git tag -a v0.2.0 -m "Release v0.2.0: Universal resource support"
|
||||
git push origin v0.2.0
|
||||
|
||||
# GitHub Actions will:
|
||||
# 1. Build binaries for all platforms
|
||||
# 2. Build and push Docker images
|
||||
# 3. Sign artifacts with cosign
|
||||
# 4. Create GitHub release
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
|
||||
- **Phase 1 (MVP)**: Secrets & ConfigMaps, basic mirroring ✅ **Complete**
|
||||
- Core reconciliation logic ✅
|
||||
- Content hash-based change detection ✅
|
||||
- Pattern matching for namespaces ✅
|
||||
- Helm chart & deployment manifests ✅
|
||||
- Comprehensive test suite ✅
|
||||
- CI/CD with GitHub Actions ✅
|
||||
- **Phase 2**: Production hardening & observability ✅ **Complete**
|
||||
- Prometheus metrics dashboard ✅
|
||||
- Alert rules for common issues ✅
|
||||
- Recording rules for performance monitoring ✅
|
||||
- Grafana dashboard with KPIs ✅
|
||||
- Performance optimization for large clusters (covered by rate limiting & worker threads) ✅
|
||||
- **Phase 3**: Universal resource support ✅ **Complete**
|
||||
- Auto-discovery of all resource types ✅
|
||||
- Support for CRDs, Ingresses, Services, and more ✅
|
||||
- Periodic rediscovery for dynamic clusters ✅
|
||||
- Safety filtering and deny lists ✅
|
||||
- **Phase 4**: Advanced features (Future)
|
||||
- Cross-namespace reference rewriting
|
||||
- kubectl plugin for easy management
|
||||
- Advanced transformation rules
|
||||
|
||||
## Monitoring
|
||||
|
||||
KubeMirror exposes Prometheus metrics and includes production-ready monitoring resources:
|
||||
|
||||
```bash
|
||||
# Deploy ServiceMonitor and Alert Rules
|
||||
kubectl apply -f monitoring/servicemonitor.yaml
|
||||
kubectl apply -f monitoring/prometheusrule.yaml
|
||||
|
||||
# Import Grafana dashboard from monitoring/grafana-dashboard.json
|
||||
```
|
||||
|
||||
See [monitoring/README.md](monitoring/README.md) for complete observability setup including:
|
||||
- Prometheus metrics and recording rules
|
||||
- Alert rules for operational issues
|
||||
- Grafana dashboard with key performance indicators
|
||||
|
||||
## Documentation
|
||||
|
||||
- [CLAUDE.md](CLAUDE.md) - Project specification and requirements
|
||||
- [examples/](examples/) - Working examples and testing scenarios
|
||||
- [monitoring/](monitoring/) - Prometheus, Grafana, and alerting setup
|
||||
- [Helm Chart](charts/kubemirror/) - Kubernetes deployment via Helm
|
||||
- [Project Repository](https://github.com/lukaszraczylo/kubemirror)
|
||||
- [monitoring/](monitoring/) - Prometheus metrics, Grafana dashboards, alerting setup
|
||||
- [Helm Chart Documentation](charts/kubemirror/README.md) - Kubernetes deployment via Helm
|
||||
- [GitHub Repository](https://github.com/lukaszraczylo/kubemirror) - Source code and issue tracker
|
||||
|
||||
## License
|
||||
|
||||
See LICENSE file.
|
||||
See [LICENSE](LICENSE) file for details.
|
||||
|
||||
Reference in New Issue
Block a user