The test suite uses a data-driven framework approach where test scenarios are systematically defined and executed using reusable functions. This ensures comprehensive coverage of all edge cases without code duplication.

Prerequisites

Kubernetes cluster running (tested with docker-desktop)
kubectl configured and pointing to docker-desktop context
Go 1.21+ installed
curl (for health checks)

Test Architecture

Test Framework Components

common.sh: Base utilities (logging, assertions, cleanup)
test-framework.sh: DRY test framework functions (resource creation, updates, verification)
test-comprehensive.sh: Comprehensive test scenarios using the framework (supports selective execution)
test-parallel.sh: Parallel test runner for faster execution (batches independent tests)
run-all-tests.sh: Main test runner (builds binary, starts controller, runs tests)

Test Framework Functions

The framework provides reusable functions for all operations:

# Resource lifecycle
create_source <type> <name> <namespace> <has_label> <has_annotation> <targets> <data>
update_source_labels <type> <name> <namespace> <enabled_value>
update_source_annotations <type> <name> <namespace> <sync_value> <targets>
update_source_data <type> <name> <namespace> <new_data>

# Namespace operations
create_test_namespace <name> <allow_mirrors_label>
update_namespace_labels <namespace> <allow_mirrors_value>
delete_namespace <namespace>

# Verification functions
verify_mirrors_exist <type> <name> <namespace1> <namespace2> ...
verify_mirrors_not_exist <type> <name> <namespace1> <namespace2> ...
verify_mirror_data <type> <source_name> <source_ns> <target_ns> <expected_data>
verify_orphan_cleanup <type> <name> <namespace1> <namespace2> ...

Comprehensive Test Suite

The comprehensive test suite (test-comprehensive.sh) covers 30 systematic scenarios:

Source Lifecycle Scenarios

Source without labels/annotations: No mirrors created
Add enabled label: Still no mirrors (sync annotation required)
Add sync annotation: Mirrors created in targets
Modify source data: Mirrors updated
Set sync to false: All mirrors deleted
Set enabled to false: All mirrors deleted

Target Namespace Management

Add namespace to list: New mirror created
Remove namespace from list: Orphaned mirror deleted
Change list to pattern: Old mirrors deleted, new pattern mirrors created
Multiple patterns: Mirrors in all matching namespaces

Pattern Matching

Create namespace matching pattern: Automatic mirror creation
Mix explicit + pattern: Both types work together
Change pattern: Orphaned mirrors cleaned up

'all' Keyword with Opt-in

'all' without namespace label: No mirror created
Add allow-mirrors label: Mirror created
Remove allow-mirrors label: Mirror deleted
Change label true→false: Mirror deleted

Edge Cases

Target namespace deleted: Other mirrors unaffected
Recreate deleted namespace: Mirror recreated
Source deleted: Cascade deletion of all mirrors
Target manually deleted: Automatic recreation
Remove sync annotation: All mirrors deleted

Resource Types

Mixed resource types: ConfigMaps alongside Secrets
Custom Resource (Traefik Middleware): CRD mirroring

Transformation Scenarios (24-30)

Static value transformation: Replace data values with static strings
Template transformation: Use Go templates with context variables
Merge transformation: Merge new data into existing fields
Delete transformation: Remove specific fields
Multiple transformations: Combine multiple rules
Strict mode: Fail on transformation errors vs skip

All basic scenarios tested with both Secrets and ConfigMaps.

Running Tests

Run Complete Test Suite (Sequential)

cd e2e
./run-all-tests.sh

This will:

Check you're on docker-desktop context
Build the KubeMirror binary
Start the controller in background
Run comprehensive test scenarios (all 30 scenarios sequentially)
Report detailed results with pass/fail for each
Clean up all resources automatically

Performance: ~5-7 minutes for all 30 scenarios

Run Complete Test Suite (Parallel) - FASTER ⚡

cd e2e
# Start controller first
../kubemirror --max-targets=100 --worker-threads=5 > /tmp/kubemirror-test.log 2>&1 &

# Run tests in parallel batches
./test-parallel.sh

This runs independent tests in parallel batches:

Sequential: Scenarios 1-11 (core lifecycle - must run sequentially)
Parallel Batch 1: Scenarios 12-15 (namespace labels)
Parallel Batch 2: Scenarios 16-19 (deletion scenarios)
Parallel Batch 3: Scenarios 20-23 (mixed resources)
Parallel Batch 4: Scenarios 24-27 (transformations part 1)
Parallel Batch 5: Scenarios 28-30 (transformations part 2)

Performance: ~3-4 minutes (40-50% faster than sequential)

Run Selective Scenarios

Run only specific scenarios for faster iteration during development:

# Must have KubeMirror controller running first
cd /Users/nvm/Documents/projects/private/kube-mirror
./kubemirror --max-targets=100 --worker-threads=5 > /tmp/kubemirror-test.log 2>&1 &

# Run only transformation tests (scenarios 24-30)
cd e2e
./test-comprehensive.sh 24 25 26 27 28 29 30

# Run only specific scenarios
./test-comprehensive.sh 1 2 3

# Run single scenario for debugging
./test-comprehensive.sh 24

Performance: <1 minute for a few scenarios

Test Output

Each test produces colored output:

🔵 [INFO]: Informational messages
✅ [PASS]: Test passed
❌ [FAIL]: Test failed
⚠️ [WARN]: Warning messages

Example output:

======================================
KubeMirror E2E Test Suite
======================================

[INFO] Step 1: Checking Kubernetes context
[PASS] Running on docker-desktop context
[INFO] Step 2: Building KubeMirror binary
[PASS] KubeMirror binary built successfully
[INFO] Step 3: Starting KubeMirror controller
[INFO] KubeMirror started with PID: 12345
[PASS] Controller is healthy

======================================
Running Test Suite 1: Basic Mirroring
======================================
[INFO] Starting Basic Mirroring tests
[INFO] Test 1: Mirror Secret to explicit namespace list
[PASS] Resource secret/test-explicit-list-secret exists in namespace e2e-target-1
[PASS] Resource secret/test-explicit-list-secret exists in namespace e2e-target-2
...

======================================
Test Summary
======================================
Total Tests: 45
Passed:      45
Failed:      0
======================================
All tests passed!

Test Resources

Tests create temporary resources with clear naming for isolation:

Source Namespace: kubemirror-e2e-source (dedicated namespace for all test source resources)
Target Namespaces: kubemirror-e2e-* prefixed (ns-1, ns-2, app-1, db-1, etc.)
Secrets: test-* prefixed in source namespace
ConfigMaps: test-* prefixed in source namespace
CRDs: Traefik Middleware resources for CRD testing

All resources are cleaned up automatically on test completion, including:

Automatic finalizer removal from source resources (prevents hanging deletions)
Cascade deletion of all target namespaces
Cleanup on test interruption (SIGINT/SIGTERM)

Troubleshooting

Tests fail with "context not docker-desktop"

Switch to docker-desktop context:

kubectl config use-context docker-desktop

Tests timeout waiting for resources

Controller may not be running or not reconciling. Check:

# Check if controller is running
ps aux | grep kubemirror

# Check controller logs
tail -f /tmp/kubemirror-e2e-test.log

# Check controller health
curl http://localhost:8081/healthz

Cleanup hanging

If tests get interrupted, manually clean up:

# Delete all e2e test namespaces
kubectl delete namespace -l kubemirror-e2e-test=true

# Delete test resources in default namespace
kubectl delete secret,configmap -n default -l kubemirror.raczylo.com/enabled=true

# Kill controller if still running
pkill kubemirror

Individual test fails

Run test with verbose output to see which assertion failed:

bash -x ./test-basic-mirroring.sh

Check controller logs for errors:

grep -i error /tmp/kubemirror-e2e-test.log

Adding New Test Scenarios

The DRY framework makes it easy to add new test scenarios. Here's how:

Example: Add a new scenario

# In test-comprehensive.sh, add a new scenario block:

run_test_scenario "23: Your new scenario description"

# Use framework functions to set up test conditions
create_test_namespace e2e-new-ns
create_source secret test-new default true true "e2e-new-ns" "test-data"

# Perform the action you want to test
update_source_annotations secret test-new default true "e2e-new-ns,e2e-new-ns-2"

# Verify expected results
verify_mirrors_exist secret test-new e2e-new-ns e2e-new-ns-2

complete_test_scenario "23" "pass"

Framework Functions Reference

Resource Creation:

create_source secret my-secret default true true "ns1,ns2" "data"
#             ↑      ↑         ↑       ↑    ↑    ↑         ↑
#             type   name      ns      lbl  ann  targets   data

Resource Updates:

update_source_labels secret my-secret default true     # Set enabled=true
update_source_labels secret my-secret default false    # Set enabled=false
update_source_labels secret my-secret default ""       # Remove label

update_source_annotations secret my-secret default true "ns1,ns2"  # Enable sync
update_source_annotations secret my-secret default false ""        # Set sync=false
update_source_annotations secret my-secret default "" ""           # Remove annotation

update_source_data secret my-secret default "new-data-v2"

Namespace Operations:

create_test_namespace my-ns true     # Create with allow-mirrors=true
create_test_namespace my-ns false    # Create with allow-mirrors=false
create_test_namespace my-ns ""       # Create with no label

update_namespace_labels my-ns true   # Set allow-mirrors=true
update_namespace_labels my-ns false  # Set allow-mirrors=false
update_namespace_labels my-ns ""     # Remove label

Verification:

verify_mirrors_exist secret my-secret ns1 ns2 ns3
verify_mirrors_not_exist secret my-secret ns4 ns5
verify_mirror_data secret my-secret default target-ns "expected-data"
verify_orphan_cleanup secret my-secret orphan-ns1 orphan-ns2

Test Coverage Summary

Category	Scenarios	Details
Source lifecycle	6	No labels → add label → add annotation → modify → disable
Target management	4	Add/remove namespaces, change list to pattern, multiple patterns
Pattern matching	3	New namespace creation, pattern changes, mixed explicit+pattern
'all' keyword opt-in	4	No label, add label, remove label, change true→false
Edge cases	5	Namespace deletion, recreation, source deletion, target recreation
Resource types	2	Mixed ConfigMaps, Custom Resource (Traefik Middleware)
Transformations	7	Static value, template, merge, delete, multiple, strict mode
Total	30	Comprehensive coverage with multiple resource types

Test Methodology

Systematic Approach

The test framework follows a systematic approach:

State Setup: Create namespaces and resources in known state
Action: Perform the operation being tested (create, update, delete, label change)
Verification: Assert expected outcomes using verification functions
Cleanup: Automatic cleanup via trap handlers

DRY Principles

Reusable functions: All operations abstracted into framework functions
Data-driven: Test scenarios are data, not code
Composable: Combine framework functions to create complex scenarios
Maintainable: Add new scenarios without duplicating code

Coverage Strategy

Tests systematically cover:

Happy path: Expected behavior under normal conditions
Edge cases: Boundary conditions and unusual states
Error conditions: Invalid inputs, missing resources, conflicts
State transitions: All possible state changes (no labels → labels → annotations, etc.)
Concurrent operations: Namespace creation during reconciliation, multiple updates

Test Utilities Reference

Common Utilities (`common.sh`)

Logging:

log_info <message>: Blue informational message
log_success <message>: Green success message (increments pass count)
log_fail <message>: Red failure message (increments fail count)
log_warn <message>: Yellow warning message

Assertions:

assert_resource_exists <type> <name> <namespace>
assert_resource_not_exists <type> <name> <namespace>
assert_annotation_exists <type> <name> <namespace> <annotation_key>
assert_label_exists <type> <name> <namespace> <label_key> <expected_value>
assert_data_matches <type> <source_name> <source_ns> <target_name> <target_ns> <data_key>

Waiting:

wait_for_resource <type> <name> <namespace> [timeout]
wait_for_resource_deletion <type> <name> <namespace> [timeout]

Utilities:

cleanup_namespace <namespace>
cleanup_resource <type> <name> <namespace>
check_context: Verify running on docker-desktop
print_summary: Print test results summary

CI/CD Integration

To run tests in CI:

#!/bin/bash
set -e

# Start kind cluster or use existing k8s
kind create cluster --name kubemirror-test

# Switch context
kubectl config use-context kind-kubemirror-test

# Run tests
cd e2e
./run-all-tests.sh

# Cleanup
kind delete cluster --name kubemirror-test

Performance Notes

Test Execution Times

Sequential execution (test-comprehensive.sh): ~5-7 minutes for all 30 scenarios
Parallel execution (test-parallel.sh): ~3-4 minutes for all 30 scenarios (40-50% faster)
Selective execution (few scenarios): <1 minute
Controller startup: ~10 seconds
Resource reconciliation: typically <5 seconds per operation

Test Coverage

Total scenarios: 30 comprehensive scenarios
Total assertions: 100+ across all scenarios
Resource types tested: Secrets, ConfigMaps, Traefik Middlewares (CRDs)
Each scenario includes: Setup, action, verification, and cleanup phases

Optimization Tips

Use test-parallel.sh for full test runs (40-50% faster)
Use selective execution during development: ./test-comprehensive.sh 24 25 26
Run only affected scenarios after code changes
Parallel execution is safe - batches ensure test independence

Test Isolation and Cleanup

Automatic cleanup: All resources cleaned up via trap handlers
Namespace isolation:
- Dedicated source namespace: kubemirror-e2e-source
- Target namespaces: kubemirror-e2e-* prefixed
- No pollution of default namespace
Execution modes:
- Sequential: All scenarios run in order (test-comprehensive.sh with no args)
- Parallel: Independent scenarios batched (test-parallel.sh)
- Selective: Run specific scenarios (test-comprehensive.sh 24 25 26)
Idempotent: Tests can be re-run without manual cleanup
Resource labeling: Test resources labeled test-resource: e2e for easy identification
Finalizer handling: Automatic finalizer removal prevents stuck resource deletions

Known Limitations

Tests assume clean docker-desktop cluster (or equivalent local cluster)
Some scenarios require waiting for reconciliation (30s default timeout)
Controller must be stopped between runs if running manually (run-all-tests.sh handles this)
Parallel execution requires sufficient cluster resources (5-6 tests may run concurrently)
Some scenarios depend on previous state (scenarios 1-11 must run sequentially)

README.md

KubeMirror E2E Tests

Overview