kubemirror/e2e/README.md

# KubeMirror E2E Tests

Comprehensive, DRY (Don't Repeat Yourself) test framework for KubeMirror functionality.

## Overview

The test suite uses a **data-driven framework** approach where test scenarios are systematically defined and executed using reusable functions. This ensures comprehensive coverage of all edge cases without code duplication.

## Prerequisites

- Kubernetes cluster running (tested with docker-desktop)
- kubectl configured and pointing to docker-desktop context
- Go 1.21+ installed
- curl (for health checks)

## Test Architecture

### Test Framework Components

1. **common.sh**: Base utilities (logging, assertions, cleanup)
2. **test-framework.sh**: DRY test framework functions (resource creation, updates, verification)
3. **test-comprehensive.sh**: Comprehensive test scenarios using the framework (supports selective execution)
4. **test-parallel.sh**: Parallel test runner for faster execution (batches independent tests)
5. **run-all-tests.sh**: Main test runner (builds binary, starts controller, runs tests)

### Test Framework Functions

The framework provides reusable functions for all operations:

```bash
# Resource lifecycle
create_source <type> <name> <namespace> <has_label> <has_annotation> <targets> <data>
update_source_labels <type> <name> <namespace> <enabled_value>
update_source_annotations <type> <name> <namespace> <sync_value> <targets>
update_source_data <type> <name> <namespace> <new_data>

# Namespace operations
create_test_namespace <name> <allow_mirrors_label>
update_namespace_labels <namespace> <allow_mirrors_value>
delete_namespace <namespace>

# Verification functions
verify_mirrors_exist <type> <name> <namespace1> <namespace2> ...
verify_mirrors_not_exist <type> <name> <namespace1> <namespace2> ...
verify_mirror_data <type> <source_name> <source_ns> <target_ns> <expected_data>
verify_orphan_cleanup <type> <name> <namespace1> <namespace2> ...
```

## Comprehensive Test Suite

The comprehensive test suite (`test-comprehensive.sh`) covers **30 systematic scenarios**:

### Source Lifecycle Scenarios

1. **Source without labels/annotations**: No mirrors created
2. **Add enabled label**: Still no mirrors (sync annotation required)
3. **Add sync annotation**: Mirrors created in targets
4. **Modify source data**: Mirrors updated
5. **Set sync to false**: All mirrors deleted
6. **Set enabled to false**: All mirrors deleted

### Target Namespace Management

7. **Add namespace to list**: New mirror created
8. **Remove namespace from list**: Orphaned mirror deleted
9. **Change list to pattern**: Old mirrors deleted, new pattern mirrors created
10. **Multiple patterns**: Mirrors in all matching namespaces

### Pattern Matching

11. **Create namespace matching pattern**: Automatic mirror creation
12. **Mix explicit + pattern**: Both types work together
13. **Change pattern**: Orphaned mirrors cleaned up

### 'all' Keyword with Opt-in

14. **'all' without namespace label**: No mirror created
15. **Add allow-mirrors label**: Mirror created
16. **Remove allow-mirrors label**: Mirror deleted
17. **Change label true→false**: Mirror deleted

### Edge Cases

18. **Target namespace deleted**: Other mirrors unaffected
19. **Recreate deleted namespace**: Mirror recreated
20. **Source deleted**: Cascade deletion of all mirrors
21. **Target manually deleted**: Automatic recreation
22. **Remove sync annotation**: All mirrors deleted

### Resource Types

23. **Mixed resource types**: ConfigMaps alongside Secrets
24. **Custom Resource (Traefik Middleware)**: CRD mirroring

### Transformation Scenarios (24-30)

25. **Static value transformation**: Replace data values with static strings
26. **Template transformation**: Use Go templates with context variables
27. **Merge transformation**: Merge new data into existing fields
28. **Delete transformation**: Remove specific fields
29. **Multiple transformations**: Combine multiple rules
30. **Strict mode**: Fail on transformation errors vs skip

All basic scenarios tested with both **Secrets** and **ConfigMaps**.

## Running Tests

### Run Complete Test Suite (Sequential)

```bash
cd e2e
./run-all-tests.sh
```

This will:
1. Check you're on docker-desktop context
2. Build the KubeMirror binary
3. Start the controller in background
4. Run comprehensive test scenarios (all 30 scenarios sequentially)
5. Report detailed results with pass/fail for each
6. Clean up all resources automatically

**Performance**: ~5-7 minutes for all 30 scenarios

### Run Complete Test Suite (Parallel) - **FASTER** ⚡

```bash
cd e2e
# Start controller first
../kubemirror --max-targets=100 --worker-threads=5 > /tmp/kubemirror-test.log 2>&1 &

# Run tests in parallel batches
./test-parallel.sh
```

This runs independent tests in parallel batches:
- **Sequential**: Scenarios 1-11 (core lifecycle - must run sequentially)
- **Parallel Batch 1**: Scenarios 12-15 (namespace labels)
- **Parallel Batch 2**: Scenarios 16-19 (deletion scenarios)
- **Parallel Batch 3**: Scenarios 20-23 (mixed resources)
- **Parallel Batch 4**: Scenarios 24-27 (transformations part 1)
- **Parallel Batch 5**: Scenarios 28-30 (transformations part 2)

**Performance**: ~3-4 minutes (40-50% faster than sequential)

### Run Selective Scenarios

Run only specific scenarios for faster iteration during development:

```bash
# Must have KubeMirror controller running first
cd /Users/nvm/Documents/projects/private/kube-mirror
./kubemirror --max-targets=100 --worker-threads=5 > /tmp/kubemirror-test.log 2>&1 &

# Run only transformation tests (scenarios 24-30)
cd e2e
./test-comprehensive.sh 24 25 26 27 28 29 30

# Run only specific scenarios
./test-comprehensive.sh 1 2 3

# Run single scenario for debugging
./test-comprehensive.sh 24
```

**Performance**: <1 minute for a few scenarios

## Test Output

Each test produces colored output:
- 🔵 **[INFO]**: Informational messages
- ✅ **[PASS]**: Test passed
- ❌ **[FAIL]**: Test failed
- ⚠️  **[WARN]**: Warning messages

Example output:
```
======================================
KubeMirror E2E Test Suite
======================================

[INFO] Step 1: Checking Kubernetes context
[PASS] Running on docker-desktop context
[INFO] Step 2: Building KubeMirror binary
[PASS] KubeMirror binary built successfully
[INFO] Step 3: Starting KubeMirror controller
[INFO] KubeMirror started with PID: 12345
[PASS] Controller is healthy

======================================
Running Test Suite 1: Basic Mirroring
======================================
[INFO] Starting Basic Mirroring tests
[INFO] Test 1: Mirror Secret to explicit namespace list
[PASS] Resource secret/test-explicit-list-secret exists in namespace e2e-target-1
[PASS] Resource secret/test-explicit-list-secret exists in namespace e2e-target-2
...

======================================
Test Summary
======================================
Total Tests: 45
Passed:      45
Failed:      0
======================================
All tests passed!
```

## Test Resources

Tests create temporary resources with clear naming for isolation:
- **Source Namespace**: `kubemirror-e2e-source` (dedicated namespace for all test source resources)
- **Target Namespaces**: `kubemirror-e2e-*` prefixed (ns-1, ns-2, app-1, db-1, etc.)
- **Secrets**: `test-*` prefixed in source namespace
- **ConfigMaps**: `test-*` prefixed in source namespace
- **CRDs**: Traefik Middleware resources for CRD testing

All resources are cleaned up automatically on test completion, including:
- Automatic finalizer removal from source resources (prevents hanging deletions)
- Cascade deletion of all target namespaces
- Cleanup on test interruption (SIGINT/SIGTERM)

## Troubleshooting

### Tests fail with "context not docker-desktop"

Switch to docker-desktop context:
```bash
kubectl config use-context docker-desktop
```

### Tests timeout waiting for resources

Controller may not be running or not reconciling. Check:
```bash
# Check if controller is running
ps aux | grep kubemirror

# Check controller logs
tail -f /tmp/kubemirror-e2e-test.log

# Check controller health
curl http://localhost:8081/healthz
```

### Cleanup hanging

If tests get interrupted, manually clean up:
```bash
# Delete all e2e test namespaces
kubectl delete namespace -l kubemirror-e2e-test=true

# Delete test resources in default namespace
kubectl delete secret,configmap -n default -l kubemirror.raczylo.com/enabled=true

# Kill controller if still running
pkill kubemirror
```

### Individual test fails

Run test with verbose output to see which assertion failed:
```bash
bash -x ./test-basic-mirroring.sh
```

Check controller logs for errors:
```bash
grep -i error /tmp/kubemirror-e2e-test.log
```

## Adding New Test Scenarios

The DRY framework makes it easy to add new test scenarios. Here's how:

### Example: Add a new scenario

```bash
# In test-comprehensive.sh, add a new scenario block:

run_test_scenario "23: Your new scenario description"

# Use framework functions to set up test conditions
create_test_namespace e2e-new-ns
create_source secret test-new default true true "e2e-new-ns" "test-data"

# Perform the action you want to test
update_source_annotations secret test-new default true "e2e-new-ns,e2e-new-ns-2"

# Verify expected results
verify_mirrors_exist secret test-new e2e-new-ns e2e-new-ns-2

complete_test_scenario "23" "pass"
```

### Framework Functions Reference

**Resource Creation:**
```bash
create_source secret my-secret default true true "ns1,ns2" "data"
#             ↑      ↑         ↑       ↑    ↑    ↑         ↑
#             type   name      ns      lbl  ann  targets   data
```

**Resource Updates:**
```bash
update_source_labels secret my-secret default true     # Set enabled=true
update_source_labels secret my-secret default false    # Set enabled=false
update_source_labels secret my-secret default ""       # Remove label

update_source_annotations secret my-secret default true "ns1,ns2"  # Enable sync
update_source_annotations secret my-secret default false ""        # Set sync=false
update_source_annotations secret my-secret default "" ""           # Remove annotation

update_source_data secret my-secret default "new-data-v2"
```

**Namespace Operations:**
```bash
create_test_namespace my-ns true     # Create with allow-mirrors=true
create_test_namespace my-ns false    # Create with allow-mirrors=false
create_test_namespace my-ns ""       # Create with no label

update_namespace_labels my-ns true   # Set allow-mirrors=true
update_namespace_labels my-ns false  # Set allow-mirrors=false
update_namespace_labels my-ns ""     # Remove label
```

**Verification:**
```bash
verify_mirrors_exist secret my-secret ns1 ns2 ns3
verify_mirrors_not_exist secret my-secret ns4 ns5
verify_mirror_data secret my-secret default target-ns "expected-data"
verify_orphan_cleanup secret my-secret orphan-ns1 orphan-ns2
```

## Test Coverage Summary

| Category | Scenarios | Details |
|----------|-----------|---------|
| Source lifecycle | 6 | No labels → add label → add annotation → modify → disable |
| Target management | 4 | Add/remove namespaces, change list to pattern, multiple patterns |
| Pattern matching | 3 | New namespace creation, pattern changes, mixed explicit+pattern |
| 'all' keyword opt-in | 4 | No label, add label, remove label, change true→false |
| Edge cases | 5 | Namespace deletion, recreation, source deletion, target recreation |
| Resource types | 2 | Mixed ConfigMaps, Custom Resource (Traefik Middleware) |
| Transformations | 7 | Static value, template, merge, delete, multiple, strict mode |
| **Total** | **30** | **Comprehensive coverage with multiple resource types** |

## Test Methodology

### Systematic Approach

The test framework follows a systematic approach:

1. **State Setup**: Create namespaces and resources in known state
2. **Action**: Perform the operation being tested (create, update, delete, label change)
3. **Verification**: Assert expected outcomes using verification functions
4. **Cleanup**: Automatic cleanup via trap handlers

### DRY Principles

- **Reusable functions**: All operations abstracted into framework functions
- **Data-driven**: Test scenarios are data, not code
- **Composable**: Combine framework functions to create complex scenarios
- **Maintainable**: Add new scenarios without duplicating code

### Coverage Strategy

Tests systematically cover:
- **Happy path**: Expected behavior under normal conditions
- **Edge cases**: Boundary conditions and unusual states
- **Error conditions**: Invalid inputs, missing resources, conflicts
- **State transitions**: All possible state changes (no labels → labels → annotations, etc.)
- **Concurrent operations**: Namespace creation during reconciliation, multiple updates

## Test Utilities Reference

### Common Utilities (`common.sh`)

**Logging:**
- `log_info <message>`: Blue informational message
- `log_success <message>`: Green success message (increments pass count)
- `log_fail <message>`: Red failure message (increments fail count)
- `log_warn <message>`: Yellow warning message

**Assertions:**
- `assert_resource_exists <type> <name> <namespace>`
- `assert_resource_not_exists <type> <name> <namespace>`
- `assert_annotation_exists <type> <name> <namespace> <annotation_key>`
- `assert_label_exists <type> <name> <namespace> <label_key> <expected_value>`
- `assert_data_matches <type> <source_name> <source_ns> <target_name> <target_ns> <data_key>`

**Waiting:**
- `wait_for_resource <type> <name> <namespace> [timeout]`
- `wait_for_resource_deletion <type> <name> <namespace> [timeout]`

**Utilities:**
- `cleanup_namespace <namespace>`
- `cleanup_resource <type> <name> <namespace>`
- `check_context`: Verify running on docker-desktop
- `print_summary`: Print test results summary

## CI/CD Integration

To run tests in CI:

```bash
#!/bin/bash
set -e

# Start kind cluster or use existing k8s
kind create cluster --name kubemirror-test

# Switch context
kubectl config use-context kind-kubemirror-test

# Run tests
cd e2e
./run-all-tests.sh

# Cleanup
kind delete cluster --name kubemirror-test
```

## Performance Notes

### Test Execution Times
- **Sequential execution** (test-comprehensive.sh): ~5-7 minutes for all 30 scenarios
- **Parallel execution** (test-parallel.sh): ~3-4 minutes for all 30 scenarios (40-50% faster)
- **Selective execution** (few scenarios): <1 minute
- **Controller startup**: ~10 seconds
- **Resource reconciliation**: typically <5 seconds per operation

### Test Coverage
- **Total scenarios**: 30 comprehensive scenarios
- **Total assertions**: 100+ across all scenarios
- **Resource types tested**: Secrets, ConfigMaps, Traefik Middlewares (CRDs)
- **Each scenario includes**: Setup, action, verification, and cleanup phases

### Optimization Tips
- Use `test-parallel.sh` for full test runs (40-50% faster)
- Use selective execution during development: `./test-comprehensive.sh 24 25 26`
- Run only affected scenarios after code changes
- Parallel execution is safe - batches ensure test independence

## Test Isolation and Cleanup

- **Automatic cleanup**: All resources cleaned up via trap handlers
- **Namespace isolation**:
  - Dedicated source namespace: `kubemirror-e2e-source`
  - Target namespaces: `kubemirror-e2e-*` prefixed
  - No pollution of `default` namespace
- **Execution modes**:
  - Sequential: All scenarios run in order (test-comprehensive.sh with no args)
  - Parallel: Independent scenarios batched (test-parallel.sh)
  - Selective: Run specific scenarios (test-comprehensive.sh 24 25 26)
- **Idempotent**: Tests can be re-run without manual cleanup
- **Resource labeling**: Test resources labeled `test-resource: e2e` for easy identification
- **Finalizer handling**: Automatic finalizer removal prevents stuck resource deletions

## Known Limitations

- Tests assume clean docker-desktop cluster (or equivalent local cluster)
- Some scenarios require waiting for reconciliation (30s default timeout)
- Controller must be stopped between runs if running manually (run-all-tests.sh handles this)
- Parallel execution requires sufficient cluster resources (5-6 tests may run concurrently)
- Some scenarios depend on previous state (scenarios 1-11 must run sequentially)