kubemirror/e2e/README.md

KubeMirror E2E Tests

Comprehensive, DRY (Don't Repeat Yourself) test framework for KubeMirror functionality.

Overview

The test suite uses a data-driven framework approach where test scenarios are systematically defined and executed using reusable functions. This ensures comprehensive coverage of all edge cases without code duplication.

Prerequisites

• Kubernetes cluster running (tested with docker-desktop)
• kubectl configured and pointing to docker-desktop context
• Go 1.21+ installed
• curl (for health checks)

Test Architecture

Test Framework Components

1. common.sh: Base utilities (logging, assertions, cleanup)
2. test-framework.sh: DRY test framework functions (resource creation, updates, verification)
3. test-comprehensive.sh: Comprehensive test scenarios using the framework
4. run-all-tests.sh: Main test runner (builds binary, starts controller, runs tests)

Test Framework Functions

The framework provides reusable functions for all operations:

# Resource lifecycle
create_source <type> <name> <namespace> <has_label> <has_annotation> <targets> <data>
update_source_labels <type> <name> <namespace> <enabled_value>
update_source_annotations <type> <name> <namespace> <sync_value> <targets>
update_source_data <type> <name> <namespace> <new_data>

# Namespace operations
create_test_namespace <name> <allow_mirrors_label>
update_namespace_labels <namespace> <allow_mirrors_value>
delete_namespace <namespace>

# Verification functions
verify_mirrors_exist <type> <name> <namespace1> <namespace2> ...
verify_mirrors_not_exist <type> <name> <namespace1> <namespace2> ...
verify_mirror_data <type> <source_name> <source_ns> <target_ns> <expected_data>
verify_orphan_cleanup <type> <name> <namespace1> <namespace2> ...

Comprehensive Test Suite

The comprehensive test suite (test-comprehensive.sh) covers 22 systematic scenarios:

Source Lifecycle Scenarios

1. Source without labels/annotations: No mirrors created
2. Add enabled label: Still no mirrors (sync annotation required)
3. Add sync annotation: Mirrors created in targets
4. Modify source data: Mirrors updated
5. Set sync to false: All mirrors deleted
6. Set enabled to false: All mirrors deleted

Target Namespace Management

7. Add namespace to list: New mirror created
8. Remove namespace from list: Orphaned mirror deleted
9. Change list to pattern: Old mirrors deleted, new pattern mirrors created
10. Multiple patterns: Mirrors in all matching namespaces

Pattern Matching

11. Create namespace matching pattern: Automatic mirror creation
12. Mix explicit + pattern: Both types work together
13. Change pattern: Orphaned mirrors cleaned up

'all' Keyword with Opt-in

14. 'all' without namespace label: No mirror created
15. Add allow-mirrors label: Mirror created
16. Remove allow-mirrors label: Mirror deleted
17. Change label true→false: Mirror deleted

Edge Cases

18. Target namespace deleted: Other mirrors unaffected
19. Recreate deleted namespace: Mirror recreated
20. Source deleted: Cascade deletion of all mirrors
21. Target manually deleted: Automatic recreation
22. Remove sync annotation: All mirrors deleted

Resource Types

All scenarios tested with both Secrets and ConfigMaps.

Running Tests

Run Complete Test Suite

cd e2e
./run-all-tests.sh

This will:

1. Check you're on docker-desktop context
2. Build the KubeMirror binary
3. Start the controller in background
4. Run comprehensive test scenarios (22+ scenarios)
5. Report detailed results with pass/fail for each
6. Clean up all resources automatically

Run Individual Test Scenarios

# Must have KubeMirror controller running first
cd /Users/nvm/Documents/projects/private/kube-mirror
./kubemirror --max-targets=100 --worker-threads=5 > /tmp/kubemirror-test.log 2>&1 &

# Then run the test
cd e2e
./test-comprehensive.sh

Test Output

Each test produces colored output:

• 🔵 [INFO]: Informational messages
• ✅ [PASS]: Test passed
• ❌ [FAIL]: Test failed
• ⚠️ [WARN]: Warning messages

Example output:

======================================
KubeMirror E2E Test Suite
======================================

[INFO] Step 1: Checking Kubernetes context
[PASS] Running on docker-desktop context
[INFO] Step 2: Building KubeMirror binary
[PASS] KubeMirror binary built successfully
[INFO] Step 3: Starting KubeMirror controller
[INFO] KubeMirror started with PID: 12345
[PASS] Controller is healthy

======================================
Running Test Suite 1: Basic Mirroring
======================================
[INFO] Starting Basic Mirroring tests
[INFO] Test 1: Mirror Secret to explicit namespace list
[PASS] Resource secret/test-explicit-list-secret exists in namespace e2e-target-1
[PASS] Resource secret/test-explicit-list-secret exists in namespace e2e-target-2
...

======================================
Test Summary
======================================
Total Tests: 45
Passed:      45
Failed:      0
======================================
All tests passed!

Test Resources

Tests create temporary resources:

• Namespaces: e2e-* prefixed
• Secrets: test-* prefixed in default namespace
• ConfigMaps: test-* prefixed in default namespace

All resources are cleaned up automatically on test completion.

Troubleshooting

Tests fail with "context not docker-desktop"

Switch to docker-desktop context:

kubectl config use-context docker-desktop

Tests timeout waiting for resources

Controller may not be running or not reconciling. Check:

# Check if controller is running
ps aux | grep kubemirror

# Check controller logs
tail -f /tmp/kubemirror-e2e-test.log

# Check controller health
curl http://localhost:8081/healthz

Cleanup hanging

If tests get interrupted, manually clean up:

# Delete all e2e test namespaces
kubectl delete namespace -l kubemirror-e2e-test=true

# Delete test resources in default namespace
kubectl delete secret,configmap -n default -l kubemirror.raczylo.com/enabled=true

# Kill controller if still running
pkill kubemirror

Individual test fails

Run test with verbose output to see which assertion failed:

bash -x ./test-basic-mirroring.sh

Check controller logs for errors:

grep -i error /tmp/kubemirror-e2e-test.log

Adding New Test Scenarios

The DRY framework makes it easy to add new test scenarios. Here's how:

Example: Add a new scenario

# In test-comprehensive.sh, add a new scenario block:

run_test_scenario "23: Your new scenario description"

# Use framework functions to set up test conditions
create_test_namespace e2e-new-ns
create_source secret test-new default true true "e2e-new-ns" "test-data"

# Perform the action you want to test
update_source_annotations secret test-new default true "e2e-new-ns,e2e-new-ns-2"

# Verify expected results
verify_mirrors_exist secret test-new e2e-new-ns e2e-new-ns-2

complete_test_scenario "23" "pass"

Framework Functions Reference

Resource Creation:

create_source secret my-secret default true true "ns1,ns2" "data"
#             ↑      ↑         ↑       ↑    ↑    ↑         ↑
#             type   name      ns      lbl  ann  targets   data

Resource Updates:

update_source_labels secret my-secret default true     # Set enabled=true
update_source_labels secret my-secret default false    # Set enabled=false
update_source_labels secret my-secret default ""       # Remove label

update_source_annotations secret my-secret default true "ns1,ns2"  # Enable sync
update_source_annotations secret my-secret default false ""        # Set sync=false
update_source_annotations secret my-secret default "" ""           # Remove annotation

update_source_data secret my-secret default "new-data-v2"

Namespace Operations:

create_test_namespace my-ns true     # Create with allow-mirrors=true
create_test_namespace my-ns false    # Create with allow-mirrors=false
create_test_namespace my-ns ""       # Create with no label

update_namespace_labels my-ns true   # Set allow-mirrors=true
update_namespace_labels my-ns false  # Set allow-mirrors=false
update_namespace_labels my-ns ""     # Remove label

Verification:

verify_mirrors_exist secret my-secret ns1 ns2 ns3
verify_mirrors_not_exist secret my-secret ns4 ns5
verify_mirror_data secret my-secret default target-ns "expected-data"
verify_orphan_cleanup secret my-secret orphan-ns1 orphan-ns2

Test Coverage Summary

Category	Scenarios	Details
Source lifecycle	6	No labels → add label → add annotation → modify → disable
Target management	4	Add/remove namespaces, change list to pattern, multiple patterns
Pattern matching	3	New namespace creation, pattern changes, mixed explicit+pattern
'all' keyword opt-in	4	No label, add label, remove label, change true→false
Edge cases	5	Namespace deletion, recreation, source deletion, target recreation
Total	22	All with Secrets and ConfigMaps

Test Methodology

Systematic Approach

The test framework follows a systematic approach:

1. State Setup: Create namespaces and resources in known state
2. Action: Perform the operation being tested (create, update, delete, label change)
3. Verification: Assert expected outcomes using verification functions
4. Cleanup: Automatic cleanup via trap handlers

DRY Principles

• Reusable functions: All operations abstracted into framework functions
• Data-driven: Test scenarios are data, not code
• Composable: Combine framework functions to create complex scenarios
• Maintainable: Add new scenarios without duplicating code

Coverage Strategy

Tests systematically cover:

• Happy path: Expected behavior under normal conditions
• Edge cases: Boundary conditions and unusual states
• Error conditions: Invalid inputs, missing resources, conflicts
• State transitions: All possible state changes (no labels → labels → annotations, etc.)
• Concurrent operations: Namespace creation during reconciliation, multiple updates

Test Utilities Reference

Common Utilities (common.sh)

Logging:

• log_info <message>: Blue informational message
• log_success <message>: Green success message (increments pass count)
• log_fail <message>: Red failure message (increments fail count)
• log_warn <message>: Yellow warning message

Assertions:

• assert_resource_exists <type> <name> <namespace>
• assert_resource_not_exists <type> <name> <namespace>
• assert_annotation_exists <type> <name> <namespace> <annotation_key>
• assert_label_exists <type> <name> <namespace> <label_key> <expected_value>
• assert_data_matches <type> <source_name> <source_ns> <target_name> <target_ns> <data_key>

Waiting:

• wait_for_resource <type> <name> <namespace> [timeout]
• wait_for_resource_deletion <type> <name> <namespace> [timeout]

Utilities:

• cleanup_namespace <namespace>
• cleanup_resource <type> <name> <namespace>
• check_context: Verify running on docker-desktop
• print_summary: Print test results summary

CI/CD Integration

To run tests in CI:

#!/bin/bash
set -e

# Start kind cluster or use existing k8s
kind create cluster --name kubemirror-test

# Switch context
kubectl config use-context kind-kubemirror-test

# Run tests
cd e2e
./run-all-tests.sh

# Cleanup
kind delete cluster --name kubemirror-test

Performance Notes

• Comprehensive test suite: ~3-5 minutes
• Controller startup: ~10 seconds
• Resource reconciliation: typically <5 seconds per operation
• Total assertions: 60+ across all scenarios
• Each scenario includes setup, action, verification, and cleanup phases

Test Isolation and Cleanup

• Automatic cleanup: All resources cleaned up via trap handlers
• Namespace isolation: Tests use e2e-* prefixed namespaces
• Sequential execution: Tests run sequentially to avoid race conditions
• Idempotent: Tests can be re-run without manual cleanup
• Resource labeling: Test resources labeled for easy identification

Known Limitations

• Tests assume clean docker-desktop cluster (or equivalent local cluster)
• Some scenarios require waiting for reconciliation (30s default timeout)
• Tests are sequential (not parallel) to ensure deterministic behavior
• Controller must be stopped between runs if running manually (run-all-tests.sh handles this)