Compare commits

..
60 Commits
Author SHA1 Message Date
github-actions[bot]andGitHub 40332d8c5f chore(deps): update dependencies
Automated dependency update. All tests passed.
2026-10-10 03:29:17 +00:00
github-actions[bot]andGitHub f85b583ded chore(deps): update dependencies
Automated dependency update. All tests passed.
2026-10-09 03:29:06 +00:00
github-actions[bot]andGitHub 4093fe8151 chore(deps): update module github.com/tidwall/gjson to v1.20.0
Automated dependency update. All tests passed.
2026-10-07 03:28:32 +00:00
github-actions[bot]andGitHub 0865bdfcce chore(deps): update dependencies
Automated dependency update. All tests passed.
2026-10-05 03:32:39 +00:00
github-actions[bot]andGitHub 46373c94a5 chore(deps): update dependencies to v1.47.0
Automated dependency update. All tests passed.
2026-10-04 10:38:12 +00:00
lukaszraczylo d18269e376 chore(deps): hold cel-go below 0.32.0
cel-go 0.32.0 moved to the cel.dev/cel-go module path. The k8s.io/apiserver
module still imports the old path, so go mod tidy fails. This repository has
no direct cel-go imports. Remove the cap when k8s.io/apiserver moves to the
new path.
2026-10-04 10:14:10 +01:00
lukaszraczylo e4b95e08c2 fix(deps): bump Masterminds/semver to v3.5.0
The update only touched go.mod. This commit tidies go.sum so the build resolves.
2026-10-04 10:13:43 +01:00
lukaszraczylo a8bee99211 chore: remove dependabot config 2026-10-04 01:04:56 +01:00
lukaszraczylo 13b113b86f ci(deps): update dependencies with Renovate [skip ci] 2026-10-03 18:44:40 +01:00
lukaszraczyloandGitHub d146ece447 Update go.mod and go.sum (#56) 2026-10-02 04:25:37 +01:00
lukaszraczylo 15e71cffce fix(ci): require Go 1.26 instead of 1.27
The go directive moved to 1.27.0 during the dependency update. No dependency needs more than Go 1.26, and the CI runner has Go 1.25. Set the go directive, workflows and Dockerfile to 1.26.
2026-10-02 04:03:22 +01:00
lukaszraczylo 1b2c2b1a3d fix(controller): detect existing job with AlreadyExists
The job creation check matched the substring 'exists' in the error text.
It now uses apierrors.IsAlreadyExists, so unrelated errors no longer match.
2026-10-02 04:01:01 +01:00
lukaszraczylo bdfc781044 fix(controller): report Failed when a group fails
checkOverallStatus overwrote the Failed status with Running. A failed workflow
stayed Running and kept requeueing. It now stays Failed and the failure event
fires once. Also remove err shadowing in the visualization client.
2026-10-02 03:59:44 +01:00
lukaszraczylo 5ebe8d8c72 ci: use Go 1.25 in workflows and Dockerfile
The module requires Go 1.25. The workflows and the Dockerfile used older versions.
2026-10-02 03:58:20 +01:00
lukaszraczylo f3f3301368 chore(deps): update Go modules to latest
Update k8s libraries to v0.37.1 and controller-runtime to v0.25.2.
Pin kube-openapi to the version that k8s v0.37.1 requires.
Add the new Apply method to the test status writer mock.
2026-10-02 03:58:15 +01:00
lukaszraczyloandGitHub 49f068edcd Update go.mod and go.sum (#46) 2026-01-15 03:30:56 +00:00
lukaszraczyloandGitHub 5b40e991c5 Update go.mod and go.sum (#45) 2026-01-13 03:30:53 +00:00
lukaszraczyloandGitHub 7516fa9d4f Update go.mod and go.sum (#42) 2026-01-10 03:30:46 +00:00
lukaszraczyloandGitHub 19a193ebe0 Update go.mod and go.sum (#41) 2026-01-09 03:31:35 +00:00
lukaszraczyloandGitHub 9c768d1ae1 Update go.mod and go.sum (#40) 2026-01-07 03:31:47 +00:00
lukaszraczyloandGitHub 04816a7524 Update go.mod and go.sum (#39) 2026-01-06 03:30:14 +00:00
lukaszraczylo f2db3238c0 fix: change invalid 'improved' changelog kind to 'changed'
Artifact Hub only accepts these changelog kinds:
- added, changed, deprecated, removed, fixed, security

Changed 'improved' to 'changed' for performance optimization entry.

Resolves: "error preparing package: error enriching package from
annotations: invalid change: invalid kind: improved"
2026-01-04 14:36:07 +00:00
lukaszraczyloandGitHub 711ff6f309 Update go.mod and go.sum (#38) 2025-12-27 03:29:34 +00:00
lukaszraczyloandGitHub f015636a9e Update go.mod and go.sum (#37) 2025-12-24 03:31:23 +00:00
lukaszraczyloandGitHub baf0b32047 Update go.mod and go.sum (#36) 2025-12-23 03:31:38 +00:00
lukaszraczyloandGitHub 9a3b7e0cbf Update go.mod and go.sum (#35) 2025-12-22 14:35:26 +00:00
lukaszraczylo 7dd53d8530 fixup! Add rather comprehensive examples and plugin screenshot. 2025-12-22 14:32:27 +00:00
lukaszraczylo 5d066c1349 Add rather comprehensive examples and plugin screenshot. 2025-12-17 23:34:44 +00:00
lukaszraczylo 794e2d487a Remove dependency on kube-rbac-proxy. 2025-12-17 23:06:21 +00:00
lukaszraczylo fee9f74aad release-minor trigger. 2025-12-17 22:35:25 +00:00
lukaszraczylo 005df83478 Update release. 2025-12-17 22:33:24 +00:00
lukaszraczylo 836be577eb Update README.md 2025-12-17 22:33:24 +00:00
lukaszraczylo f96fd7843e new year update (#30)
* Multiple fixes

- add goreleaser to the build / release process
- add kubectl plugin for job graphs visualization
- add installation scripts
- update dependencies

* Update the release & CRD content.

* Next set of improvements.

  Code Quality

  - Label constants: Added LabelWorkflowName, LabelGroupName, LabelJobName, LabelJobID in controllers/definitions.go
  - Removed commented debug code: Cleaned up dead code from multiple files
  - Removed unused dependencyTree field: Cleaned connPackage struct
  - Fixed snake_case variables: Changed to camelCase (runGroup, groupDep, runJob, jobDep, k8sJob)

  Kubernetes Best Practices

  - Finalizers: Implemented handleDeletion() and deleteChildJobs() for proper cleanup
  - Status enum validation: Added +kubebuilder:validation:Enum=pending;running;succeeded;failed;aborted
  - ImagePullPolicy default: Created getImagePullPolicy() helper that defaults to IfNotPresent
  - Resource limits support: Added Resources *corev1.ResourceRequirements to ManagedJobParameters

  Observability

  - Prometheus metrics: Created controllers/metrics.go with counters (jobs created/succeeded/failed), histogram (reconciliation duration), and gauge (active jobs)
  - Structured logging: Added logger field to connPackage, used context-based logging throughout

  Configuration

  - Leader election ID: Made configurable via --leader-election-id flag
  - Development mode: Made configurable via --dev-mode flag and LOG_LEVEL env var

  Performance

  - Dependency lookup optimization: Changed from O(n*m) to O(1) using lookup maps (jobDepMap, groupDepMap)
  - Reconciliation backoff: Added RequeueAfter: 30*time.Second when workflow is running

  Documentation & Testing

  - Godoc documentation: Added comprehensive comments to API types and controller
  - Unit tests: Added helpers_test.go with tests for all helper functions
  - Integration tests: Added managedjob_controller_test.go with Ginkgo/Gomega tests

* Add the helm chart release.

* Add reasonable test coverage.

* fixup! Add reasonable test coverage.

* fixup! fixup! Add reasonable test coverage.

* fixup! fixup! fixup! Add reasonable test coverage.
2025-12-17 22:33:23 +00:00
lukaszraczylo 2b36071647 Multiple fixes (#29)
* Multiple fixes

- add goreleaser to the build / release process
- add kubectl plugin for job graphs visualization
- add installation scripts
- update dependencies

* Update the release & CRD content.

* Next set of improvements.

  Code Quality

  - Label constants: Added LabelWorkflowName, LabelGroupName, LabelJobName, LabelJobID in controllers/definitions.go
  - Removed commented debug code: Cleaned up dead code from multiple files
  - Removed unused dependencyTree field: Cleaned connPackage struct
  - Fixed snake_case variables: Changed to camelCase (runGroup, groupDep, runJob, jobDep, k8sJob)

  Kubernetes Best Practices

  - Finalizers: Implemented handleDeletion() and deleteChildJobs() for proper cleanup
  - Status enum validation: Added +kubebuilder:validation:Enum=pending;running;succeeded;failed;aborted
  - ImagePullPolicy default: Created getImagePullPolicy() helper that defaults to IfNotPresent
  - Resource limits support: Added Resources *corev1.ResourceRequirements to ManagedJobParameters

  Observability

  - Prometheus metrics: Created controllers/metrics.go with counters (jobs created/succeeded/failed), histogram (reconciliation duration), and gauge (active jobs)
  - Structured logging: Added logger field to connPackage, used context-based logging throughout

  Configuration

  - Leader election ID: Made configurable via --leader-election-id flag
  - Development mode: Made configurable via --dev-mode flag and LOG_LEVEL env var

  Performance

  - Dependency lookup optimization: Changed from O(n*m) to O(1) using lookup maps (jobDepMap, groupDepMap)
  - Reconciliation backoff: Added RequeueAfter: 30*time.Second when workflow is running

  Documentation & Testing

  - Godoc documentation: Added comprehensive comments to API types and controller
  - Unit tests: Added helpers_test.go with tests for all helper functions
  - Integration tests: Added managedjob_controller_test.go with Ginkgo/Gomega tests

* Add the helm chart release.

* Add reasonable test coverage.
2025-12-17 22:33:23 +00:00
lukaszraczylo b6ce5b7c98 Fix CVEs. 2025-12-17 22:33:23 +00:00
lukaszraczylo ba6e43acfe Update TOC. 2025-12-17 22:33:23 +00:00
lukaszraczylo 6f9644bdfa Fix chart 2025-12-17 22:33:22 +00:00
lukaszraczylo c604c16aa0 Add missing ImagePullPolicy to the newly created jobs. 2025-12-17 22:33:22 +00:00
lukaszraczylo 925f8a1592 Create dependabot.yml 2025-12-17 22:33:22 +00:00
lukaszraczylo 7f82327e1b Update golang to 1.21 2025-12-17 22:33:22 +00:00
lukaszraczylo 97f4007a6a Update dependencies. 2025-12-17 22:33:21 +00:00
lukaszraczylo 79c124816e Hotifx: Missing event generation permissions. 2025-12-17 22:33:21 +00:00
lukaszraczylo cb4a008e48 Add events generation rbac policy. 2025-12-17 22:33:21 +00:00
lukaszraczylo 10648997ad Fix issue with the annotations and labels. 2025-12-17 22:33:21 +00:00
lukaszraczylo bded5b0250 Add annotations support. 2025-12-17 22:33:20 +00:00
lukaszraczylo 7500a71056 Update README with examples of parameters. 2025-12-17 22:33:20 +00:00
lukaszraczylo 0a062b6ad7 Fix bug where parameters were not propagated. 2025-12-17 22:33:20 +00:00
lukaszraczylo 8799456499 Add codeowners. 2025-12-17 22:33:20 +00:00
lukaszraczylo d3c5cc6ccd Bugfix - chicken or egg situation. 2025-12-17 22:33:19 +00:00
lukaszraczylo 8bdccb4b8d Update chart information. 2025-12-17 22:33:19 +00:00
lukaszraczylo 554019b1cc Remove duplicated attempts to re-run job and add minor improvements. 2025-12-17 22:33:19 +00:00
lukaszraczylo 41421ea889 Add codeql 2025-12-17 22:33:19 +00:00
lukaszraczylo 7e4ea52abc Build docker image only on the master branch. 2025-12-17 22:33:19 +00:00
lukaszraczylo 76f25e0746 Update dependencies. 2025-12-17 22:33:18 +00:00
lukaszraczylo a20157681e Update CRD accordingly. 2025-12-17 22:33:18 +00:00
lukaszraczylo e037788677 Completely rebuild the tree generation logic. 2025-12-17 22:33:18 +00:00
lukaszraczylo e39c93e9ea Update operator build. 2025-12-17 22:33:18 +00:00
lukaszraczylo 785a6adbd4 Add RBAC access to events. 2025-12-17 22:33:17 +00:00
lukaszraczylo 35b9efc0e1 Update documentation on configmMapGenerators and kustomization. 2025-12-17 22:33:17 +00:00
lukaszraczylo 70033ec15b Add forgotten Volume refrence 2025-12-17 22:33:17 +00:00
60 changed files with 17011 additions and 8835 deletions
+20
View File
@@ -0,0 +1,20 @@
name: Update dependencies
on:
workflow_dispatch:
schedule:
- cron: "0 3 * * *"
permissions:
actions: write
contents: write
issues: write
pull-requests: write
jobs:
update:
uses: lukaszraczylo/shared-actions/.github/workflows/renovate-autoupdate.yaml@main
with:
go-version: ">=1.24"
node: false
release-workflow: release.yaml
@@ -77,7 +77,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v3
with:
go-version: 1.18
go-version: 1.21
- name: Configure git for private modules
uses: extractions/netrc@v1
with:
+22
View File
@@ -0,0 +1,22 @@
name: Pull Request
on:
pull_request:
branches:
- main
push:
branches:
- "**"
- "!main"
permissions:
contents: read
pull-requests: write
security-events: write
jobs:
pr-checks:
uses: lukaszraczylo/shared-actions/.github/workflows/go-pr.yaml@main
with:
go-version: "1.26"
secrets: inherit
+52
View File
@@ -0,0 +1,52 @@
name: Release
on:
workflow_dispatch:
push:
paths-ignore:
- "**.md"
- "docs/**"
- "config/samples/**"
branches:
- main
permissions:
id-token: write
contents: write
packages: write
deployments: write
jobs:
release:
uses: lukaszraczylo/shared-actions/.github/workflows/go-release.yaml@main
with:
go-version: "1.26"
docker-enabled: true
secrets: inherit
publish-helm-chart:
needs: release
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Get release version
id: version
run: |
VERSION=$(git describe --tags --abbrev=0 2>/dev/null || echo "0.0.0")
VERSION=${VERSION#v}
echo "version=$VERSION" >> $GITHUB_OUTPUT
- name: Trigger helm-charts release
env:
GH_TOKEN: ${{ secrets.HOMEBREW_TAP_TOKEN }}
run: |
gh api repos/lukaszraczylo/helm-charts/dispatches \
-f event_type=release-chart \
-f client_payload[chart_name]=jobs-manager \
-f client_payload[version]=${{ steps.version.outputs.version }} \
-f client_payload[source_repo]=lukaszraczylo/jobs-manager-operator \
-f client_payload[chart_path]=charts/jobs-manager-operator
+1
View File
@@ -25,3 +25,4 @@ chart-releaser.yaml
*.swp
*.swo
*~
dist/
+183
View File
@@ -0,0 +1,183 @@
# yaml-language-server: $schema=https://goreleaser.com/static/schema.json
# vim: set ts=2 sw=2 tw=0 fo=cnqoj
version: 2
project_name: jobs-manager-operator
before:
hooks:
- go mod tidy
# Update Helm chart version to match release version
- 'sed -i.bak ''s/^version:.*/version: {{ .Version }}/'' charts/jobs-manager-operator/Chart.yaml'
- 'sed -i.bak ''s/^appVersion:.*/appVersion: "{{ .Version }}"/'' charts/jobs-manager-operator/Chart.yaml'
- 'sed -i.bak ''s/tag:.*/tag: "{{ .Version }}"/'' charts/jobs-manager-operator/values.yaml'
- rm -f charts/jobs-manager-operator/Chart.yaml.bak charts/jobs-manager-operator/values.yaml.bak
builds:
# Kubernetes operator manager
- id: manager
main: .
binary: manager
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -s -w
- -X main.Version={{.Version}}
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64
ignore:
- goos: windows
goarch: arm64
# kubectl plugin for workflow visualization
- id: kubectl-managedjob
main: ./cmd/kubectl-managedjob
binary: kubectl-managedjob
env:
- CGO_ENABLED=0
flags:
- -trimpath
ldflags:
- -s -w
- -X main.Version={{.Version}}
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64
ignore:
- goos: windows
goarch: arm64
archives:
- id: default
formats:
- tar.gz
name_template: >-
{{ .ProjectName }}_
{{- .Version }}_
{{- .Os }}_
{{- .Arch }}
files:
- README.md
format_overrides:
- goos: windows
formats:
- zip
checksum:
name_template: 'checksums.txt'
algorithm: sha256
changelog:
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
- '^chore:'
- Merge pull request
- Merge branch
release:
github:
owner: lukaszraczylo
name: jobs-manager-operator
draft: false
prerelease: auto
name_template: "v{{.Version}}"
header: |
## Jobs Manager Operator v{{.Version}}
Kubernetes operator for managing complex multi-job workflows.
### Operator Installation
**Helm (from repository):**
```bash
helm repo add jobs-manager https://lukaszraczylo.github.io/helm-charts
helm repo update
helm install jobs-manager jobs-manager/jobs-manager --version {{.Version}}
```
**Helm (from release asset):**
```bash
helm install jobs-manager https://github.com/lukaszraczylo/jobs-manager-operator/releases/download/v{{.Version}}/jobs-manager-{{.Version}}.tgz
```
**Docker:**
```bash
docker pull ghcr.io/lukaszraczylo/jobs-manager-operator:{{.Version}}
```
### kubectl Plugin Installation
**One-liner (recommended):**
```bash
curl -sSL https://raw.githubusercontent.com/lukaszraczylo/jobs-manager-operator/main/scripts/install-plugin.sh | bash
```
**Manual:**
```bash
# Download the archive for your platform from the assets below
tar -xzf jobs-manager-operator_{{.Version}}_<os>_<arch>.tar.gz
mv kubectl-managedjob /usr/local/bin/
chmod +x /usr/local/bin/kubectl-managedjob
```
### Plugin Usage
```bash
kubectl managedjob visualize <workflow-name> -n <namespace>
kubectl managedjob visualize <workflow-name> -w # Watch mode
kubectl managedjob list -n <namespace>
kubectl managedjob status <workflow-name> -n <namespace>
```
dockers_v2:
- ids:
- manager
images:
- "ghcr.io/lukaszraczylo/jobs-manager-operator"
tags:
- "{{ .Version }}"
- "latest"
platforms:
- linux/amd64
- linux/arm64
dockerfile: Dockerfile.goreleaser
labels:
"org.opencontainers.image.title": "{{ .ProjectName }}"
"org.opencontainers.image.version": "{{ .Version }}"
"org.opencontainers.image.source": "https://github.com/lukaszraczylo/jobs-manager-operator"
"org.opencontainers.image.description": "Kubernetes operator for managing complex multi-job workflows"
signs:
- cmd: cosign
signature: "${artifact}.sigstore.json"
args:
- sign-blob
- "--bundle=${signature}"
- "${artifact}"
- "--yes"
artifacts: checksum
output: true
docker_signs:
- cmd: cosign
artifacts: images
output: true
args:
- sign
- "${artifact}@${digest}"
- "--yes"
+1 -1
View File
@@ -1,5 +1,5 @@
# Build the manager binary
FROM golang:1.19 as builder
FROM golang:1.26 AS builder
ARG TARGETOS
ARG TARGETARCH
+8
View File
@@ -0,0 +1,8 @@
# Dockerfile for GoReleaser dockers_v2
# GoReleaser organizes binaries by platform: linux/amd64/manager, linux/arm64/manager
FROM gcr.io/distroless/static:nonroot
ARG TARGETPLATFORM
WORKDIR /
COPY ${TARGETPLATFORM}/manager /manager
USER 65532:65532
ENTRYPOINT ["/manager"]
+13 -2
View File
@@ -66,6 +66,17 @@ test: manifests generate fmt vet envtest ## Run tests.
build: manifests generate fmt vet ## Build manager binary.
go build -o bin/manager main.go
.PHONY: build-plugin
build-plugin: fmt vet ## Build kubectl-managedjob plugin binary.
go build -o bin/kubectl-managedjob ./cmd/kubectl-managedjob
.PHONY: install-plugin
install-plugin: build-plugin ## Install kubectl-managedjob plugin to GOPATH/bin.
cp bin/kubectl-managedjob $(GOBIN)/kubectl-managedjob
.PHONY: build-all
build-all: build build-plugin ## Build both manager and plugin binaries.
.PHONY: run
run: manifests generate fmt vet ## Run a controller from your host.
go run ./main.go
@@ -139,7 +150,7 @@ ENVTEST ?= $(LOCALBIN)/setup-envtest
## Tool Versions
KUSTOMIZE_VERSION ?= v3.8.7
CONTROLLER_TOOLS_VERSION ?= v0.11.1
CONTROLLER_TOOLS_VERSION ?= v0.17.1
KUSTOMIZE_INSTALL_SCRIPT ?= "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"
.PHONY: kustomize
@@ -163,7 +174,7 @@ $(ENVTEST): $(LOCALBIN)
test -s $(LOCALBIN)/setup-envtest || GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
helmify:
$(call go-get-tool,$(HELMIFY),github.com/arttor/helmify/cmd/helmify@v0.3.7)
$(call go-get-tool,$(HELMIFY),github.com/arttor/helmify/cmd/helmify@v0.4.10)
helm: manifests kustomize helmify
$(KUSTOMIZE) build config/default | $(HELMIFY)
+184 -2
View File
@@ -1,8 +1,45 @@
# Kubernetes Jobs Manager Operator
- [Kubernetes Jobs Manager Operator](#kubernetes-jobs-manager-operator)
- [Description](#description)
- [Features](#features)
- [Getting Started](#getting-started)
- [Installation with helm](#installation-with-helm)
- [Prerequisites for local runs](#prerequisites-for-local-runs)
- [Jobs configuration](#jobs-configuration)
- [Examples](#examples)
- [How does it look in practice?](#how-does-it-look-in-practice)
- [Things to remember](#things-to-remember)
- [Available params](#available-params)
- [Resource limits](#resource-limits)
- [Kustomization and references](#kustomization-and-references)
- [Running on the cluster](#running-on-the-cluster)
- [Manual installation](#manual-installation)
- [Manually uninstall CRDs](#manually-uninstall-crds)
- [Manually undeploy controller](#manually-undeploy-controller)
- [How it works](#how-it-works)
- [kubectl Plugin](#kubectl-plugin)
- [Observability](#observability)
- [Prometheus Metrics](#prometheus-metrics)
- [ServiceMonitor](#servicemonitor)
- [Configuration](#configuration)
- [License](#license)
## Description
This operator is responsible for managing the lifecycle of complicated workflows which consist of multiple jobs and making their management easy, without need for dozens of yaml files and doing magic with ordering.
## Features
- **Workflow orchestration** - Define complex job workflows with dependency management
- **Parameter inheritance** - DRY configuration with params merged from workflow → group → job levels
- **Parallel execution** - Run jobs in parallel within groups or sequentially with dependencies
- **Prometheus metrics** - Built-in observability with metrics for jobs created/succeeded/failed, active jobs count, and reconciliation duration
- **Resource limits** - Configure CPU and memory limits for job containers
- **Finalizers** - Proper cleanup of child jobs when ManagedJob resources are deleted
- **kubectl plugin** - Visualize job dependency graphs directly from the command line
- **O(1) dependency lookup** - Optimized performance for large workflows
## Getting Started
### Installation with helm
@@ -94,6 +131,16 @@ spec:
parallel: true
```
### Examples
More example manifests are available in the [`config/samples/`](config/samples/) directory:
| Example | Description |
|---------|-------------|
| [Quick Start](config/samples/managedjob_quickstart.yaml) | Simple "Hello World" with two sequential jobs |
| [Parallel Processing](config/samples/managedjob_parallel_example.yaml) | Fan-out/fan-in pattern with parallel job execution |
| [Comprehensive Demo](config/samples/managedjob_comprehensive_example.yaml) | Full-featured example showcasing all capabilities: ConfigMaps, Secrets, volumes, resource limits, multi-group dependencies, and parameter inheritance |
### How does it look in practice?
```yaml
@@ -145,6 +192,64 @@ In this case - result for the first job will look like this:
status: succeeded
```
### Available params
There's quite a lot of of flexibility with parameters. On every level where params are allowed, you can define:
```yaml
params:
fromEnv:
- configMapRef:
name: "configmap-name"
key: "key-name"
env:
- name: "FOO"
value: "bar"
volumes:
- name: secrets-store-api
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: api-secrets-provider
volumeMount:
- name: secrets-store-api
mountPath: "/mnt/secrets-api"
readOnly: true
serviceAccount: "service-account-name"
restartPolicy: "Never"
imagePullSecrets:
- "ghcr-token"
imagePullPolicy:
- "Always"
labels:
this/works: "true"
annotations:
this/works/aswell: "true"
resources:
limits:
cpu: "500m"
memory: "256Mi"
requests:
cpu: "100m"
memory: "128Mi"
```
### Resource limits
You can configure resource limits for job containers at any level (workflow, group, or job). These follow the standard Kubernetes resource requirements:
```yaml
params:
resources:
limits:
cpu: "1"
memory: "512Mi"
requests:
cpu: "250m"
memory: "128Mi"
```
### Kustomization and references
In case of any issues with `configmapGenerator` or `secretGenerator`, please add following to your `kustomization.yaml`:
@@ -211,9 +316,87 @@ This project aims to follow the Kubernetes [Operator pattern](https://kubernetes
It uses [Controllers](https://kubernetes.io/docs/concepts/architecture/controller/),
which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.
## kubectl Plugin
The operator includes a kubectl plugin for visualizing job dependency graphs. Install it using the provided script:
```sh
curl -sSL https://raw.githubusercontent.com/lukaszraczylo/jobs-manager-operator/main/scripts/install-plugin.sh | bash
```
Once installed, you can visualize ManagedJob workflows:
```sh
# Visualize a ManagedJob workflow as an ASCII tree with status colors
kubectl managedjob visualize <managedjob-name> -n <namespace>
# Watch mode - continuously update the visualization
kubectl managedjob visualize <managedjob-name> -w
# Show all ManagedJobs in a namespace
kubectl managedjob list -n <namespace>
# Show status summary for a specific workflow
kubectl managedjob status <managedjob-name> -n <namespace>
```
Status colors in the visualization:
- **Green**: succeeded
- **Yellow**: running
- **Red**: failed
- **Gray**: pending
![kubectl plugin screenshot](public/plugin-screenshot.png)
## Observability
### Prometheus Metrics
The operator exposes the following Prometheus metrics:
| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `managedjob_jobs_created_total` | Counter | namespace, workflow, group | Total number of jobs created |
| `managedjob_jobs_succeeded_total` | Counter | namespace, workflow, group | Total number of jobs that succeeded |
| `managedjob_jobs_failed_total` | Counter | namespace, workflow, group | Total number of jobs that failed |
| `managedjob_active_jobs` | Gauge | namespace, workflow | Current number of active jobs |
| `managedjob_reconciliation_duration_seconds` | Histogram | namespace, workflow | Time taken for reconciliation |
### ServiceMonitor
If you're using the Prometheus Operator, a ServiceMonitor is included in the Helm chart. Enable it in your values:
```yaml
serviceMonitor:
enabled: true
interval: 30s
labels: {}
```
## Configuration
The operator supports the following configuration options:
| Flag | Environment Variable | Description | Default |
|------|---------------------|-------------|---------|
| `--leader-election-id` | - | Custom leader election ID | `jobs-manager-operator` |
| `--dev-mode` | - | Enable development logging mode | `false` |
| - | `LOG_LEVEL` | Logging level (debug, info, warn, error) | `info` |
Example Helm values for configuration:
```yaml
controllerManager:
manager:
leaderElectionId: "my-custom-id"
devMode: true
env:
LOG_LEVEL: "debug"
```
## License
Copyright 2023.
Copyright 2023-2025.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
@@ -226,4 +409,3 @@ distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
+55 -3
View File
@@ -21,104 +21,156 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// ManagedJobDependencies defines a dependency relationship between jobs or groups
type ManagedJobDependencies struct {
// Name is the identifier of the dependency (job or group name)
// +kubebuilder:validation:Optional
// +kubebuilder:default=""
Name string `json:"name"`
Name string `json:"name"`
// Status tracks the execution status of the dependency
// +kubebuilder:validation:Enum=pending;running;succeeded;failed;aborted
Status string `json:"status"`
}
// ManagedJobDefinition defines a single job within a group
type ManagedJobDefinition struct {
// Name is the unique identifier for this job within the group
// +kubebuilder:validation:Required
// +kubebuilder:validation:MaxLength=40
// +kubebuilder:validation:Pattern=[a-z0-9-]+
Name string `json:"name"`
// Parallel indicates if this job can run in parallel with others in the group
// +kubebuilder:validation:Optional
// +kubebuilder:default=false
Parallel bool `json:"parallel"`
// Image is the container image to run for this job
// +kubebuilder:validation:Required
// +kubebuilder:validation:MinLength=5
Image string `json:"image"`
// Args are the command-line arguments to pass to the container
// +kubebuilder:validation:Optional
Args []string `json:"args,omitempty"`
// Params contains job-specific parameters that override group and spec-level params
// +kubebuilder:validation:Optional
Params ManagedJobParameters `json:"params"`
// Status tracks the execution status of this job
// +kubebuilder:validation:Optional
// +kubebuilder:default=pending
// +kubebuilder:validation:Enum=pending;running;succeeded;failed;aborted
Status string `json:"status"`
// Dependencies lists the jobs that must complete before this job can run
// +kubebuilder:validation:Optional
// +optional
Dependencies []*ManagedJobDependencies `json:"dependencies"`
// CompiledParams contains the merged parameters from spec, group, and job levels
// +optional
CompiledParams ManagedJobParameters `json:"compiledParams"`
}
// ManagedJobGroup defines a group of jobs that can be executed together
type ManagedJobGroup struct {
// Name is the unique identifier for this group within the workflow
// +kubebuilder:validation:Required
// +kubebuilder:validation:MaxLength=40
// +kubebuilder:validation:Pattern=[a-z0-9-]+
Name string `json:"name"`
// Parallel indicates if this group can run in parallel with other groups
// +kubebuilder:validation:Optional
// +kubebuilder:default=false
Parallel bool `json:"parallel"`
// Jobs is the list of jobs to execute within this group
// +kubebuilder:validation:Required
// +kubebuilder:validation:MinItems=1
Jobs []*ManagedJobDefinition `json:"jobs"`
// Params contains group-level parameters that override spec-level params
// +kubebuilder:validation:Optional
Params ManagedJobParameters `json:"params"`
// Dependencies lists the groups that must complete before this group can run
// +kubebuilder:validation:Optional
// +optional
Dependencies []*ManagedJobDependencies `json:"dependencies"`
// Status tracks the execution status of this group
// +kubebuilder:validation:Optional
// +kubebuilder:default=pending
// +kubebuilder:validation:Enum=pending;running;succeeded;failed;aborted
Status string `json:"status"`
}
// ManagedJobParameters defines common parameters that can be set at spec, group, or job level.
// Parameters at lower levels override those at higher levels.
type ManagedJobParameters struct {
// FromEnv specifies environment variable sources (ConfigMaps, Secrets)
// +kubebuilder:validation:Optional
FromEnv []corev1.EnvFromSource `json:"fromEnv,omitempty"`
// Env specifies individual environment variables
// +kubebuilder:validation:Optional
Env []corev1.EnvVar `json:"env,omitempty"`
// Volumes specifies volumes to mount in job pods
// +kubebuilder:validation:Optional
Volumes []corev1.Volume `json:"volumes,omitempty"`
// VolumeMounts specifies where to mount volumes in containers
// +kubebuilder:validation:Optional
VolumeMounts []corev1.VolumeMount `json:"volumeMount,omitempty"`
// ServiceAccount is the Kubernetes service account to use for job pods
// +kubebuilder:validation:Optional
ServiceAccount string `json:"serviceAccount,omitempty"`
// RestartPolicy defines the pod restart policy (Never, OnFailure)
// +kubebuilder:validation:Optional
// +kubebuilder:default=OnFailure
// +kubebuilder:validation:Enum=Never;OnFailure
RestartPolicy string `json:"restartPolicy,omitempty"`
// ImagePullSecrets are references to secrets for pulling private images
// +kubebuilder:validation:Optional
ImagePullSecrets []corev1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
// ImagePullPolicy defines when to pull the container image
// +kubebuilder:validation:Optional
// +kubebuilder:validation:Enum=Always;Never;IfNotPresent
ImagePullPolicy string `json:"imagePullPolicy,omitempty"`
// Labels are additional labels to apply to job pods
// +kubebuilder:validation:Optional
Labels map[string]string `json:"labels,omitempty"`
// Annotations are additional annotations to apply to job pods
// +kubebuilder:validation:Optional
Annotations map[string]string `json:"annotations,omitempty"`
// Resources specifies compute resources for the job container
// +kubebuilder:validation:Optional
Resources *corev1.ResourceRequirements `json:"resources,omitempty"`
}
// ManagedJobSpec defines the desired state of ManagedJob
type ManagedJobSpec struct {
// Retries is the number of times to retry failed jobs
// +kubebuilder:validation:Required
// +kubebuilder:default=1
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Minimum=0
// +kubebuilder:validation:Maximum=100
Retries int `json:"retries"`
// Groups is the list of job groups to execute in this workflow
// +kubebuilder:validation:Required
// +kubebuilder:validation:MinItems=1
Groups []*ManagedJobGroup `json:"groups"`
// Params contains spec-level parameters that apply to all jobs
// +kubebuilder:validation:Optional
Params ManagedJobParameters `json:"params"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// ManagedJob is the Schema for the managedjobs API
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
// ManagedJob is the Schema for the managedjobs API.
// It defines a workflow consisting of groups of jobs with dependencies.
type ManagedJob struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
// Spec defines the desired workflow configuration
Spec ManagedJobSpec `json:"spec,omitempty"`
// Status tracks the overall execution status of the workflow
// +kubebuilder:validation:Optional
// +kubebuilder:default=pending
// +kubebuilder:validation:Enum=pending;running;succeeded;failed;aborted
Status string `json:"status"`
}
+12 -1
View File
@@ -1,5 +1,4 @@
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
/*
Copyright 2023.
@@ -213,6 +212,18 @@ func (in *ManagedJobParameters) DeepCopyInto(out *ManagedJobParameters) {
(*out)[key] = val
}
}
if in.Annotations != nil {
in, out := &in.Annotations, &out.Annotations
*out = make(map[string]string, len(*in))
for key, val := range *in {
(*out)[key] = val
}
}
if in.Resources != nil {
in, out := &in.Resources, &out.Resources
*out = new(v1.ResourceRequirements)
(*in).DeepCopyInto(*out)
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ManagedJobParameters.
+26 -18
View File
@@ -1,29 +1,37 @@
apiVersion: v2
name: jobs-manager
description: Kubernetes jobs manager operator
# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
description: Kubernetes jobs manager operator for orchestrating workflow-based job execution with dependency management
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.0.20
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "0.0.20"
version: 0.0.34
appVersion: "0.0.34"
keywords:
- operator
- jobs
- tasks
- workflow
- kubernetes
- batch
home: https://raczylo.com
sources:
- https://github.com/lukaszraczylo/jobs-manager-operator
maintainers:
- name: lukaszraczylo
email: [email protected]
annotations:
artifacthub.io/changes: |
- kind: added
description: Prometheus metrics support (jobs created/succeeded/failed, active jobs, reconciliation duration)
- kind: added
description: Configurable leader election ID via --leader-election-id flag
- kind: added
description: Configurable development logging mode via --dev-mode flag
- kind: added
description: LOG_LEVEL environment variable support
- kind: added
description: Finalizers for proper resource cleanup
- kind: added
description: Resource limits support for job containers
- kind: added
description: Reconciliation backoff/requeue logic
- kind: changed
description: O(1) dependency lookup performance optimization
@@ -1,13 +1,3 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "chart.fullname" . }}-controller-manager
labels:
app.kubernetes.io/component: rbac
app.kubernetes.io/created-by: jobs-manager-operator
app.kubernetes.io/part-of: jobs-manager-operator
{{- include "chart.labels" . | nindent 4 }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -50,38 +40,23 @@ spec:
- linux
containers:
- args:
- --secure-listen-address=0.0.0.0:8443
- --upstream=http://127.0.0.1:8080/
- --logtostderr=true
- --v=0
env:
- name: KUBERNETES_CLUSTER_DOMAIN
value: {{ .Values.kubernetesClusterDomain }}
image: {{ .Values.controllerManager.kubeRbacProxy.image.repository }}:{{ .Values.controllerManager.kubeRbacProxy.image.tag
| default .Chart.AppVersion }}
name: kube-rbac-proxy
ports:
- containerPort: 8443
name: https
protocol: TCP
resources: {{- toYaml .Values.controllerManager.kubeRbacProxy.resources | nindent
10 }}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
- args:
- --health-probe-bind-address=:8081
- --metrics-bind-address=127.0.0.1:8080
- --leader-elect
{{- toYaml .Values.controllerManager.manager.args | nindent 8 }}
{{- if .Values.controllerManager.manager.leaderElectionId }}
- --leader-election-id={{ .Values.controllerManager.manager.leaderElectionId }}
{{- end }}
{{- if .Values.controllerManager.manager.devMode }}
- --dev-mode
{{- end }}
command:
- /manager
env:
- name: KUBERNETES_CLUSTER_DOMAIN
value: {{ .Values.kubernetesClusterDomain }}
image: {{ .Values.controllerManager.manager.image.repository }}:{{ .Values.controllerManager.manager.image.tag
| default .Chart.AppVersion }}
value: {{ quote .Values.kubernetesClusterDomain }}
{{- if .Values.controllerManager.manager.env.LOG_LEVEL }}
- name: LOG_LEVEL
value: {{ quote .Values.controllerManager.manager.env.LOG_LEVEL }}
{{- end }}
image: {{ .Values.controllerManager.manager.image.repository }}:{{ .Values.controllerManager.manager.image.tag | default .Chart.AppVersion }}
livenessProbe:
httpGet:
path: /healthz
@@ -89,20 +64,19 @@ spec:
initialDelaySeconds: 15
periodSeconds: 20
name: manager
ports:
- containerPort: 8443
name: https
protocol: TCP
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
resources: {{- toYaml .Values.controllerManager.manager.resources | nindent 10
}}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
resources: {{- toYaml .Values.controllerManager.manager.resources | nindent 10 }}
securityContext: {{- toYaml .Values.controllerManager.manager.containerSecurityContext | nindent 10 }}
securityContext:
runAsNonRoot: true
serviceAccountName: {{ include "chart.fullname" . }}-controller-manager
terminationGracePeriodSeconds: 10
terminationGracePeriodSeconds: 10
File diff suppressed because it is too large Load Diff
@@ -5,6 +5,30 @@ metadata:
labels:
{{- include "chart.labels" . | nindent 4 }}
rules:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- batch
resources:
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- jobsmanager.raczylo.com
resources:
@@ -31,28 +55,6 @@ rules:
- get
- patch
- update
# Add ability to manage jobs
- apiGroups:
- batch
resources:
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
# Add access to events
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
@@ -3,7 +3,7 @@ kind: ClusterRole
metadata:
name: {{ include "chart.fullname" . }}-metrics-reader
labels:
app.kubernetes.io/component: kube-rbac-proxy
app.kubernetes.io/component: manager
app.kubernetes.io/created-by: jobs-manager-operator
app.kubernetes.io/part-of: jobs-manager-operator
{{- include "chart.labels" . | nindent 4 }}
@@ -3,7 +3,7 @@ kind: Service
metadata:
name: {{ include "chart.fullname" . }}-controller-manager-metrics-service
labels:
app.kubernetes.io/component: kube-rbac-proxy
app.kubernetes.io/component: manager
app.kubernetes.io/created-by: jobs-manager-operator
app.kubernetes.io/part-of: jobs-manager-operator
control-plane: controller-manager
@@ -3,7 +3,7 @@ kind: ClusterRole
metadata:
name: {{ include "chart.fullname" . }}-proxy-role
labels:
app.kubernetes.io/component: kube-rbac-proxy
app.kubernetes.io/component: manager
app.kubernetes.io/created-by: jobs-manager-operator
app.kubernetes.io/part-of: jobs-manager-operator
{{- include "chart.labels" . | nindent 4 }}
@@ -26,7 +26,7 @@ kind: ClusterRoleBinding
metadata:
name: {{ include "chart.fullname" . }}-proxy-rolebinding
labels:
app.kubernetes.io/component: kube-rbac-proxy
app.kubernetes.io/component: manager
app.kubernetes.io/created-by: jobs-manager-operator
app.kubernetes.io/part-of: jobs-manager-operator
{{- include "chart.labels" . | nindent 4 }}
@@ -0,0 +1,11 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "chart.fullname" . }}-controller-manager
labels:
app.kubernetes.io/component: rbac
app.kubernetes.io/created-by: jobs-manager-operator
app.kubernetes.io/part-of: jobs-manager-operator
{{- include "chart.labels" . | nindent 4 }}
annotations:
{{- toYaml .Values.controllerManager.serviceAccount.annotations | nindent 4 }}
@@ -0,0 +1,31 @@
{{- if .Values.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "chart.fullname" . }}-metrics
{{- if .Values.serviceMonitor.namespace }}
namespace: {{ .Values.serviceMonitor.namespace }}
{{- end }}
labels:
{{- include "chart.labels" . | nindent 4 }}
{{- with .Values.serviceMonitor.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
endpoints:
- path: /metrics
port: https
scheme: https
interval: {{ .Values.serviceMonitor.interval }}
scrapeTimeout: {{ .Values.serviceMonitor.scrapeTimeout }}
tlsConfig:
insecureSkipVerify: true
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
namespaceSelector:
matchNames:
- {{ .Release.Namespace }}
selector:
matchLabels:
control-plane: controller-manager
{{- include "chart.selectorLabels" . | nindent 6 }}
{{- end }}
+37 -12
View File
@@ -1,19 +1,27 @@
controllerManager:
kubeRbacProxy:
image:
repository: gcr.io/kubebuilder/kube-rbac-proxy
tag: v0.13.1
resources:
limits:
cpu: 500m
memory: 128Mi
requests:
cpu: 5m
memory: 64Mi
manager:
# Command line arguments for the manager
args:
- --health-probe-bind-address=:8081
- --metrics-bind-address=:8443
- --metrics-secure
- --leader-elect
# Leader election ID - customize for multi-tenant clusters
leaderElectionId: "jobsmanager.raczylo.com"
# Enable development mode with verbose logging (console format)
devMode: false
# Environment variables for the manager container
env:
# Set to "debug" to enable verbose logging
LOG_LEVEL: ""
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
image:
repository: ghcr.io/lukaszraczylo/jobs-manager-operator
tag: {{ .Chart.AppVersion }}
tag: ""
resources:
limits:
cpu: 500m
@@ -22,7 +30,12 @@ controllerManager:
cpu: 10m
memory: 64Mi
replicas: 1
serviceAccount:
annotations: {}
kubernetesClusterDomain: cluster.local
# Metrics service configuration
metricsService:
ports:
- name: https
@@ -30,3 +43,15 @@ metricsService:
protocol: TCP
targetPort: https
type: ClusterIP
# ServiceMonitor for Prometheus Operator integration
serviceMonitor:
enabled: false
# Namespace where ServiceMonitor will be created (defaults to release namespace)
namespace: ""
# Additional labels for ServiceMonitor
labels: {}
# Scrape interval
interval: 30s
# Scrape timeout
scrapeTimeout: 10s
+260
View File
@@ -0,0 +1,260 @@
package main
import (
"context"
"fmt"
"os"
"os/signal"
"syscall"
"time"
"github.com/fatih/color"
"github.com/spf13/cobra"
"raczylo.com/jobs-manager-operator/pkg/visualization"
)
var (
namespace string
watch bool
interval time.Duration
noColor bool
kubeconfig string
)
func main() {
if err := rootCmd.Execute(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
var rootCmd = &cobra.Command{
Use: "kubectl-managedjob",
Short: "Visualize and manage ManagedJob workflows",
Long: `A kubectl plugin for visualizing ManagedJob workflows.
This plugin helps you understand the structure and execution status
of your ManagedJob workflows with ASCII tree visualization.`,
}
var visualizeCmd = &cobra.Command{
Use: "visualize <name>",
Short: "Visualize a ManagedJob workflow tree",
Long: `Display a ManagedJob workflow as an ASCII tree with status colors.
Status colors:
- Green: succeeded
- Yellow: running
- Red: failed
- Gray: pending
- Magenta: aborted`,
Args: cobra.ExactArgs(1),
RunE: runVisualize,
}
var listCmd = &cobra.Command{
Use: "list",
Short: "List all ManagedJobs in a namespace",
RunE: runList,
}
var statusCmd = &cobra.Command{
Use: "status <name>",
Short: "Show status summary of a ManagedJob",
Args: cobra.ExactArgs(1),
RunE: runStatus,
}
func init() {
// Global flags
rootCmd.PersistentFlags().StringVarP(&namespace, "namespace", "n", "default", "Kubernetes namespace")
rootCmd.PersistentFlags().StringVar(&kubeconfig, "kubeconfig", "", "Path to kubeconfig file")
rootCmd.PersistentFlags().BoolVar(&noColor, "no-color", false, "Disable colored output")
// Visualize flags
visualizeCmd.Flags().BoolVarP(&watch, "watch", "w", false, "Watch for changes and refresh")
visualizeCmd.Flags().DurationVar(&interval, "interval", 2*time.Second, "Watch refresh interval")
// Add commands
rootCmd.AddCommand(visualizeCmd)
rootCmd.AddCommand(listCmd)
rootCmd.AddCommand(statusCmd)
}
func runVisualize(cmd *cobra.Command, args []string) error {
name := args[0]
// Handle no-color flag
if noColor {
color.NoColor = true
}
// Set KUBECONFIG if provided
if kubeconfig != "" {
_ = os.Setenv("KUBECONFIG", kubeconfig) // #nosec G104 - env var set failure is extremely rare
}
client, err := visualization.NewClient()
if err != nil {
return fmt.Errorf("failed to create client: %w", err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Handle interrupt
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-sigCh
cancel()
}()
if watch {
return watchLoop(ctx, client, name)
}
return visualizeOnce(ctx, client, name)
}
func visualizeOnce(ctx context.Context, client *visualization.Client, name string) error {
mj, err := client.GetManagedJob(ctx, name, namespace)
if err != nil {
return err
}
tree := visualization.BuildTree(mj)
renderer := visualization.NewRenderer(!noColor)
fmt.Print(renderer.Render(tree))
return nil
}
func watchLoop(ctx context.Context, client *visualization.Client, name string) error {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
// Clear screen and move cursor to top
fmt.Print("\033[H\033[2J")
if err := visualizeOnce(ctx, client, name); err != nil {
fmt.Fprintf(os.Stderr, "Error: %v\n", err)
}
fmt.Printf("\nWatching %s/%s (Ctrl+C to exit, refreshing every %s)\n", namespace, name, interval)
select {
case <-ctx.Done():
return nil
case <-ticker.C:
continue
}
}
}
func runList(cmd *cobra.Command, args []string) error {
// Handle no-color flag
if noColor {
color.NoColor = true
}
// Set KUBECONFIG if provided
if kubeconfig != "" {
_ = os.Setenv("KUBECONFIG", kubeconfig) // #nosec G104 - env var set failure is extremely rare
}
client, err := visualization.NewClient()
if err != nil {
return fmt.Errorf("failed to create client: %w", err)
}
ctx := context.Background()
mjList, err := client.ListManagedJobs(ctx, namespace)
if err != nil {
return err
}
if len(mjList.Items) == 0 {
fmt.Printf("No ManagedJobs found in namespace %s\n", namespace)
return nil
}
// Print table header
fmt.Printf("%-30s %-12s %-8s %-8s\n", "NAME", "STATUS", "GROUPS", "JOBS")
fmt.Printf("%-30s %-12s %-8s %-8s\n", "----", "------", "------", "----")
renderer := visualization.NewRenderer(!noColor)
_ = renderer // For potential future color support in list
for i := range mjList.Items {
summary := visualization.GetStatusSummary(&mjList.Items[i])
statusStr := formatStatus(summary.Status, !noColor)
fmt.Printf("%-30s %-12s %-8d %-8d\n", summary.Name, statusStr, summary.Groups, summary.Jobs)
}
return nil
}
func runStatus(cmd *cobra.Command, args []string) error {
name := args[0]
// Handle no-color flag
if noColor {
color.NoColor = true
}
// Set KUBECONFIG if provided
if kubeconfig != "" {
_ = os.Setenv("KUBECONFIG", kubeconfig) // #nosec G104 - env var set failure is extremely rare
}
client, err := visualization.NewClient()
if err != nil {
return fmt.Errorf("failed to create client: %w", err)
}
ctx := context.Background()
mj, err := client.GetManagedJob(ctx, name, namespace)
if err != nil {
return err
}
summary := visualization.GetStatusSummary(mj)
fmt.Printf("Name: %s\n", summary.Name)
fmt.Printf("Namespace: %s\n", summary.Namespace)
fmt.Printf("Status: %s\n", formatStatus(summary.Status, !noColor))
fmt.Printf("Groups: %d\n", summary.Groups)
fmt.Printf("Jobs: %d\n", summary.Jobs)
fmt.Println()
fmt.Printf("Job Status:\n")
fmt.Printf(" Pending: %d\n", summary.Pending)
fmt.Printf(" Running: %d\n", summary.Running)
fmt.Printf(" Succeeded: %d\n", summary.Succeeded)
fmt.Printf(" Failed: %d\n", summary.Failed)
fmt.Printf(" Aborted: %d\n", summary.Aborted)
return nil
}
func formatStatus(status string, useColor bool) string {
if !useColor {
return status
}
switch status {
case visualization.StatusSucceeded:
return color.GreenString(status)
case visualization.StatusRunning:
return color.YellowString(status)
case visualization.StatusFailed:
return color.RedString(status)
case visualization.StatusPending:
return color.HiBlackString(status)
case visualization.StatusAborted:
return color.MagentaString(status)
default:
return status
}
}
File diff suppressed because it is too large Load Diff
+7 -25
View File
@@ -1,5 +1,5 @@
# This patch inject a sidecar container which is a HTTP proxy for the
# controller manager, it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
# This patch configures the controller manager to expose metrics securely
# using controller-runtime's built-in authentication and authorization.
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -25,31 +25,13 @@ spec:
values:
- linux
containers:
- name: kube-rbac-proxy
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- "ALL"
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1
- name: manager
args:
- "--secure-listen-address=0.0.0.0:8443"
- "--upstream=http://127.0.0.1:8080/"
- "--logtostderr=true"
- "--v=0"
- "--health-probe-bind-address=:8081"
- "--metrics-bind-address=:8443"
- "--metrics-secure"
- "--leader-elect"
ports:
- containerPort: 8443
protocol: TCP
name: https
resources:
limits:
cpu: 500m
memory: 128Mi
requests:
cpu: 5m
memory: 64Mi
- name: manager
args:
- "--health-probe-bind-address=:8081"
- "--metrics-bind-address=127.0.0.1:8080"
- "--leader-elect"
+3
View File
@@ -1,2 +1,5 @@
resources:
- manager.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
+1 -1
View File
@@ -5,7 +5,7 @@ metadata:
control-plane: controller-manager
app.kubernetes.io/name: service
app.kubernetes.io/instance: controller-manager-metrics-service
app.kubernetes.io/component: kube-rbac-proxy
app.kubernetes.io/component: manager
app.kubernetes.io/created-by: jobs-manager-operator
app.kubernetes.io/part-of: jobs-manager-operator
app.kubernetes.io/managed-by: kustomize
+2 -3
View File
@@ -9,9 +9,8 @@ resources:
- role_binding.yaml
- leader_election_role.yaml
- leader_election_role_binding.yaml
# Comment the following 4 lines if you want to disable
# the auth proxy (https://github.com/brancz/kube-rbac-proxy)
# which protects your /metrics endpoint.
# The following resources configure RBAC for secure metrics
# using controller-runtime's built-in authentication/authorization.
- auth_proxy_service.yaml
- auth_proxy_role.yaml
- auth_proxy_role_binding.yaml
+24 -1
View File
@@ -2,9 +2,32 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: null
name: manager-role
rules:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- batch
resources:
- jobs
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- jobsmanager.raczylo.com
resources:
@@ -0,0 +1,253 @@
# Comprehensive ManagedJob Example
# This example demonstrates all capabilities of the jobs-manager-operator
#
# Workflow Overview:
# 1. setup-group: Creates initial configuration (runs first, sequential)
# 2. parallel-workers: Multiple parallel jobs that process data
# 3. aggregation-group: Depends on parallel-workers completion
# 4. notification-group: Final notification job
---
apiVersion: v1
kind: ConfigMap
metadata:
name: job-config
data:
APP_NAME: "hello-world-demo"
LOG_LEVEL: "info"
greeting.txt: |
Hello from the jobs-manager-operator!
This file was mounted from a ConfigMap.
---
apiVersion: v1
kind: Secret
metadata:
name: job-secrets
type: Opaque
stringData:
API_KEY: "demo-secret-key-12345"
DATABASE_URL: "postgres://user:pass@localhost/demo"
---
apiVersion: jobsmanager.raczylo.com/v1beta1
kind: ManagedJob
metadata:
name: comprehensive-demo
labels:
app.kubernetes.io/name: comprehensive-demo
app.kubernetes.io/component: demo
spec:
# Retry failed jobs up to 3 times
retries: 3
# Spec-level parameters - applied to ALL jobs (can be overridden)
params:
# Environment variables applied globally
env:
- name: GLOBAL_ENV
value: "available-in-all-jobs"
- name: WORKFLOW_ID
value: "demo-workflow-001"
# Environment variables from ConfigMap (applied to all jobs)
fromEnv:
- configMapRef:
name: job-config
# Resource defaults for all jobs
resources:
requests:
memory: "64Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "100m"
# Labels and annotations for all job pods
labels:
managed-by: jobs-manager-operator
environment: demo
annotations:
prometheus.io/scrape: "false"
# Default restart policy
restartPolicy: OnFailure
# Default image pull policy
imagePullPolicy: IfNotPresent
groups:
# =========================================================================
# GROUP 1: Setup (Sequential, runs first)
# =========================================================================
- name: setup-group
parallel: false # Jobs in this group run sequentially
jobs:
- name: init-job
image: busybox:1.36
args:
- /bin/sh
- -c
- |
echo "=== Initialization Job ==="
echo "Workflow ID: $WORKFLOW_ID"
echo "Global Env: $GLOBAL_ENV"
echo "App Name from ConfigMap: $APP_NAME"
echo "Setup complete!"
- name: validate-config
image: busybox:1.36
args:
- /bin/sh
- -c
- |
echo "=== Validating Configuration ==="
echo "Checking environment variables..."
test -n "$APP_NAME" && echo "✓ APP_NAME is set"
test -n "$LOG_LEVEL" && echo "✓ LOG_LEVEL is set"
echo "Configuration validated!"
# This job depends on init-job completing successfully
dependencies:
- name: init-job
status: succeeded
# =========================================================================
# GROUP 2: Parallel Workers (runs after setup-group)
# =========================================================================
- name: parallel-workers
parallel: true # Jobs in this group run in parallel
dependencies:
- name: setup-group
status: succeeded
# Group-level params - override spec-level for this group
params:
env:
- name: GROUP_NAME
value: "parallel-workers"
- name: WORKER_MODE
value: "parallel"
jobs:
- name: worker-alpha
image: busybox:1.36
parallel: true
args:
- /bin/sh
- -c
- |
echo "=== Worker Alpha ==="
echo "Processing task A..."
echo "Group: $GROUP_NAME"
echo "Mode: $WORKER_MODE"
sleep 2
echo "Worker Alpha completed!"
- name: worker-beta
image: busybox:1.36
parallel: true
args:
- /bin/sh
- -c
- |
echo "=== Worker Beta ==="
echo "Processing task B..."
sleep 2
echo "Worker Beta completed!"
# Job-level params - override group and spec level
params:
env:
- name: BETA_SPECIFIC
value: "only-in-worker-beta"
# Custom resource requirements for this job
resources:
requests:
memory: "32Mi"
cpu: "25m"
limits:
memory: "64Mi"
cpu: "50m"
- name: worker-gamma
image: busybox:1.36
parallel: true
args:
- /bin/sh
- -c
- |
echo "=== Worker Gamma ==="
echo "Processing task C..."
sleep 2
echo "Worker Gamma completed!"
# =========================================================================
# GROUP 3: Aggregation (depends on parallel-workers)
# =========================================================================
- name: aggregation-group
parallel: false
dependencies:
- name: parallel-workers
status: succeeded
# Group-level params with volume mounts
params:
env:
- name: AGGREGATION_MODE
value: "combine-results"
# Mount ConfigMap as a volume
volumes:
- name: config-volume
configMap:
name: job-config
volumeMounts:
- name: config-volume
mountPath: /config
readOnly: true
jobs:
- name: aggregate-results
image: busybox:1.36
args:
- /bin/sh
- -c
- |
echo "=== Aggregating Results ==="
echo "Mode: $AGGREGATION_MODE"
echo ""
echo "Reading mounted ConfigMap file:"
cat /config/greeting.txt
echo ""
echo "All worker results aggregated!"
# =========================================================================
# GROUP 4: Notification (final step)
# =========================================================================
- name: notification-group
parallel: false
dependencies:
- name: aggregation-group
status: succeeded
# Group-level params with secrets
params:
fromEnv:
- secretRef:
name: job-secrets
jobs:
- name: send-notification
image: busybox:1.36
args:
- /bin/sh
- -c
- |
echo "=== Sending Notification ==="
echo "Workflow completed successfully!"
echo ""
echo "API Key available: $(test -n "$API_KEY" && echo 'yes' || echo 'no')"
echo "Database URL available: $(test -n "$DATABASE_URL" && echo 'yes' || echo 'no')"
echo ""
echo "Final Summary:"
echo "- Workflow ID: $WORKFLOW_ID"
echo "- App Name: $APP_NAME"
echo "- All groups completed!"
echo ""
echo "🎉 Demo workflow finished successfully!"
@@ -0,0 +1,86 @@
# Parallel Execution ManagedJob Example
# Demonstrates fan-out/fan-in pattern with parallel jobs
#
# Workflow:
# 1. prepare (single job)
# 2. process-1, process-2, process-3 (run in parallel)
# 3. finalize (waits for all parallel jobs)
---
apiVersion: jobsmanager.raczylo.com/v1beta1
kind: ManagedJob
metadata:
name: parallel-demo
spec:
retries: 2
params:
env:
- name: BATCH_ID
value: "batch-001"
groups:
# Stage 1: Preparation
- name: preparation
jobs:
- name: prepare
image: busybox:1.36
args:
- /bin/sh
- -c
- |
echo "Preparing batch: $BATCH_ID"
echo "Starting parallel processing..."
# Stage 2: Parallel Processing (fan-out)
- name: processing
parallel: true
dependencies:
- name: preparation
status: succeeded
jobs:
- name: process-1
image: busybox:1.36
parallel: true
args:
- /bin/sh
- -c
- |
echo "Processing chunk 1..."
sleep 3
echo "Chunk 1 done!"
- name: process-2
image: busybox:1.36
parallel: true
args:
- /bin/sh
- -c
- |
echo "Processing chunk 2..."
sleep 2
echo "Chunk 2 done!"
- name: process-3
image: busybox:1.36
parallel: true
args:
- /bin/sh
- -c
- |
echo "Processing chunk 3..."
sleep 4
echo "Chunk 3 done!"
# Stage 3: Finalization (fan-in)
- name: finalization
dependencies:
- name: processing
status: succeeded
jobs:
- name: finalize
image: busybox:1.36
args:
- /bin/sh
- -c
- |
echo "All parallel jobs completed!"
echo "Batch $BATCH_ID finished successfully."
+32
View File
@@ -0,0 +1,32 @@
# Quick Start ManagedJob Example
# A simple example to get started with the jobs-manager-operator
#
# This demonstrates:
# - Basic sequential job execution
# - Simple environment variables
# - Job dependencies within a group
---
apiVersion: jobsmanager.raczylo.com/v1beta1
kind: ManagedJob
metadata:
name: hello-world
spec:
retries: 1
groups:
- name: hello-group
jobs:
- name: say-hello
image: busybox:1.36
args:
- echo
- "Hello, World!"
- name: say-goodbye
image: busybox:1.36
args:
- echo
- "Goodbye, World!"
dependencies:
- name: say-hello
status: succeeded
+3 -3
View File
@@ -153,8 +153,8 @@ func (cp *connPackage) generateDependencyTree() {
_, theSame, _ := pandati.CompareStructsReplaced(originalMainJobDefinition, cp.mj)
if !theSame {
cp.updateCRDStatusDirectly()
if err := cp.updateCRDStatusDirectly(); err != nil {
cp.logger.Error(err, "Failed to update CRD status in dependency tree")
}
}
// fmt.Print(mainTree.Print())
// fmt.Printf("Dependency tree: %# v", pretty.Formatter(mainTree))
}
+368
View File
@@ -0,0 +1,368 @@
package controllers
import (
"context"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
// TreeTestSuite tests the Tree implementation
type TreeTestSuite struct {
suite.Suite
}
func TestTreeInterfaceSuite(t *testing.T) {
suite.Run(t, new(TreeTestSuite))
}
func (s *TreeTestSuite) TestNew() {
tree := New("root")
s.Equal("root", tree.Text())
s.Empty(tree.Items())
}
func (s *TreeTestSuite) TestAdd() {
root := New("root")
child := root.Add("child")
s.Equal("child", child.Text())
s.Len(root.Items(), 1)
}
func (s *TreeTestSuite) TestAddTree() {
root := New("root")
subtree := New("subtree")
subtree.Add("subtree-child")
root.AddTree(subtree)
s.Len(root.Items(), 1)
s.Equal("subtree", root.Items()[0].Text())
}
func (s *TreeTestSuite) TestItems() {
root := New("root")
root.Add("child1")
root.Add("child2")
root.Add("child3")
items := root.Items()
s.Len(items, 3)
}
func (s *TreeTestSuite) TestText() {
tree := New("test-text")
s.Equal("test-text", tree.Text())
}
func (s *TreeTestSuite) TestPrint_Simple() {
tree := New("root")
output := tree.Print()
s.Contains(output, "root")
}
func (s *TreeTestSuite) TestPrint_WithChildren() {
tree := New("root")
tree.Add("child1")
tree.Add("child2")
output := tree.Print()
s.Contains(output, "root")
s.Contains(output, "child1")
s.Contains(output, "child2")
s.Contains(output, "├──")
s.Contains(output, "└──")
}
func (s *TreeTestSuite) TestPrint_NestedChildren() {
tree := New("root")
child := tree.Add("child")
child.Add("grandchild1")
child.Add("grandchild2")
output := tree.Print()
s.Contains(output, "root")
s.Contains(output, "child")
s.Contains(output, "grandchild1")
s.Contains(output, "grandchild2")
}
func (s *TreeTestSuite) TestPrint_WorkflowStructure() {
workflow := New("my-workflow")
group1 := workflow.Add("init-group")
group1.Add("setup-job")
group1.Add("config-job")
group2 := workflow.Add("build-group")
group2.Add("Depends on group: init-group")
group2.Add("compile-job")
output := workflow.Print()
s.Contains(output, "my-workflow")
s.Contains(output, "init-group")
s.Contains(output, "setup-job")
s.Contains(output, "build-group")
s.Contains(output, "Depends on group: init-group")
}
// ==================== GENERATE DEPENDENCY TREE TESTS ====================
type DependencyTreeTestSuite struct {
suite.Suite
ctx context.Context
cancel context.CancelFunc
client *MockClient
reconciler *ManagedJobReconciler
}
func (s *DependencyTreeTestSuite) SetupTest() {
s.ctx, s.cancel = context.WithCancel(context.Background())
s.client = NewMockClient()
s.reconciler = &ManagedJobReconciler{
Client: s.client,
Scheme: s.client.Scheme(),
Recorder: NewFakeRecorder(),
}
}
func (s *DependencyTreeTestSuite) TearDownTest() {
s.cancel()
}
func TestDependencyTreeSuite(t *testing.T) {
suite.Run(t, new(DependencyTreeTestSuite))
}
func (s *DependencyTreeTestSuite) newConnPackage(mj *jobsmanagerv1beta1.ManagedJob) *connPackage {
cp := &connPackage{
ctx: s.ctx,
r: s.reconciler,
mj: mj,
req: ctrl.Request{
NamespacedName: types.NamespacedName{
Name: mj.Name,
Namespace: mj.Namespace,
},
},
logger: zap.New(),
}
cp.buildDependencyMaps()
return cp
}
func (s *DependencyTreeTestSuite) TestGenerateDependencyTree_SingleGroup() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
NewTestJobDef("job2", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.generateDependencyTree()
// Sequential jobs should have dependencies
s.NotEmpty(mj.Spec.Groups[0].Jobs[1].Dependencies)
}
func (s *DependencyTreeTestSuite) TestGenerateDependencyTree_ParallelJobs() {
mj := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: NewTestManagedJob("workflow", "default", nil).ObjectMeta,
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "group1",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{Name: "job1", Image: "busybox", Parallel: true, Status: ExecutionStatusPending},
{Name: "job2", Image: "busybox", Parallel: true, Status: ExecutionStatusPending},
},
Status: ExecutionStatusPending,
},
},
},
}
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.generateDependencyTree()
// Parallel jobs should not have dependencies on each other
s.Empty(mj.Spec.Groups[0].Jobs[0].Dependencies)
s.Empty(mj.Spec.Groups[0].Jobs[1].Dependencies)
}
func (s *DependencyTreeTestSuite) TestGenerateDependencyTree_MultipleGroups() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
NewTestGroup("group2", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job2", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.generateDependencyTree()
// Sequential groups should have dependencies
s.NotEmpty(mj.Spec.Groups[1].Dependencies)
}
func (s *DependencyTreeTestSuite) TestGenerateDependencyTree_ParallelGroups() {
mj := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: NewTestManagedJob("workflow", "default", nil).ObjectMeta,
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "group1",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{{Name: "job1", Image: "busybox", Status: ExecutionStatusPending}},
Parallel: true,
Status: ExecutionStatusPending,
},
{
Name: "group2",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{{Name: "job2", Image: "busybox", Status: ExecutionStatusPending}},
Parallel: true,
Status: ExecutionStatusPending,
},
},
},
}
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.generateDependencyTree()
// Parallel groups should not have dependencies on each other
s.Empty(mj.Spec.Groups[0].Dependencies)
s.Empty(mj.Spec.Groups[1].Dependencies)
}
func (s *DependencyTreeTestSuite) TestCheckIfPresentInDependencies_Found() {
mj := NewTestManagedJob("workflow", "default", nil)
cp := s.newConnPackage(mj)
deps := []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "dep1", Status: ExecutionStatusPending},
{Name: "dep2", Status: ExecutionStatusPending},
}
s.True(cp.checkIfPresentInDependencies(deps, "dep1"))
s.True(cp.checkIfPresentInDependencies(deps, "dep2"))
}
func (s *DependencyTreeTestSuite) TestCheckIfPresentInDependencies_NotFound() {
mj := NewTestManagedJob("workflow", "default", nil)
cp := s.newConnPackage(mj)
deps := []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "dep1", Status: ExecutionStatusPending},
}
s.False(cp.checkIfPresentInDependencies(deps, "dep3"))
}
func (s *DependencyTreeTestSuite) TestCheckIfPresentInDependencies_Empty() {
mj := NewTestManagedJob("workflow", "default", nil)
cp := s.newConnPackage(mj)
var deps []*jobsmanagerv1beta1.ManagedJobDependencies
s.False(cp.checkIfPresentInDependencies(deps, "any"))
}
// ==================== MATRIX TEST: TREE PRINTING ====================
func TestTreePrint_Matrix(t *testing.T) {
tests := []struct {
name string
buildTree func() Tree
contains []string
}{
{
name: "single_node",
buildTree: func() Tree {
return New("root")
},
contains: []string{"root"},
},
{
name: "two_children",
buildTree: func() Tree {
tree := New("parent")
tree.Add("child1")
tree.Add("child2")
return tree
},
contains: []string{"parent", "├──", "└──", "child1", "child2"},
},
{
name: "deep_nesting",
buildTree: func() Tree {
tree := New("l1")
l2 := tree.Add("l2")
l3 := l2.Add("l3")
l3.Add("l4")
return tree
},
contains: []string{"l1", "l2", "l3", "l4"},
},
{
name: "workflow_example",
buildTree: func() Tree {
wf := New("my-workflow")
g1 := wf.Add("init")
g1.Add("setup-db")
g1.Add("setup-cache")
g2 := wf.Add("build")
g2.Add("Depends on group: init")
g2.Add("compile")
return wf
},
contains: []string{"my-workflow", "init", "setup-db", "build", "Depends on group: init"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tree := tt.buildTree()
output := tree.Print()
for _, expected := range tt.contains {
assert.Contains(t, output, expected)
}
})
}
}
func TestTreePrint_MultilineText(t *testing.T) {
tree := New("root")
tree.Add("line1\nline2\nline3")
output := tree.Print()
// Should have all three lines
assert.True(t, strings.Contains(output, "line1"))
assert.True(t, strings.Contains(output, "line2"))
assert.True(t, strings.Contains(output, "line3"))
}
+94 -72
View File
@@ -1,31 +1,16 @@
package controllers
import (
"strings"
"github.com/lukaszraczylo/pandati"
kbatch "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/log"
)
/* Compile parameters from top to the job level */
type compiledParams struct {
FromEnv []corev1.EnvFromSource
Env []corev1.EnvVar
Volumes []corev1.Volume
VolumeMounts []corev1.VolumeMount
ServiceAccount string
RestartPolicy string
ImagePullSecrets []corev1.LocalObjectReference
ImagePullPolicy string
Labels map[string]string
}
func (cp *connPackage) compileParameters(params ...jobsmanagerv1beta1.ManagedJobParameters) jobsmanagerv1beta1.ManagedJobParameters {
cparams := jobsmanagerv1beta1.ManagedJobParameters{}
for _, params := range params {
@@ -51,33 +36,48 @@ func (cp *connPackage) compileParameters(params ...jobsmanagerv1beta1.ManagedJob
if params.ImagePullSecrets != nil {
cparams.ImagePullSecrets = append(cparams.ImagePullSecrets, params.ImagePullSecrets...)
}
if params.ImagePullPolicy != "" {
cparams.ImagePullPolicy = params.ImagePullPolicy
}
if params.Labels != nil {
cparams.Labels = make(map[string]string)
for k, v := range params.Labels {
cparams.Labels[k] = v
}
}
if params.Annotations != nil {
cparams.Annotations = make(map[string]string)
for k, v := range params.Annotations {
cparams.Annotations[k] = v
}
}
if params.Resources != nil {
cparams.Resources = params.Resources
}
}
}
return cparams
}
// updateDependentJobs updates the status of all dependencies that reference the completed job.
// Uses the pre-built jobDepMap for O(1) lookup instead of iterating through all jobs.
func (cp *connPackage) updateDependentJobs(completedJob string, jobStatus string) {
for _, group := range cp.mj.Spec.Groups {
for _, job := range group.Jobs {
for _, dependency := range job.Dependencies {
if dependency.Name == completedJob && dependency.Status != jobStatus {
dependency.Status = jobStatus
}
if deps, exists := cp.jobDepMap[completedJob]; exists {
for _, dep := range deps {
if dep.Status != jobStatus {
dep.Status = jobStatus
}
}
}
}
// updateDependentGroups updates the status of all dependencies that reference the completed group.
// Uses the pre-built groupDepMap for O(1) lookup instead of iterating through all groups.
func (cp *connPackage) updateDependentGroups(completedGroup string, jobStatus string) {
for _, group := range cp.mj.Spec.Groups {
for _, dependency := range group.Dependencies {
if dependency.Name == completedGroup && dependency.Status != jobStatus {
dependency.Status = jobStatus
if deps, exists := cp.groupDepMap[completedGroup]; exists {
for _, dep := range deps {
if dep.Status != jobStatus {
dep.Status = jobStatus
}
}
}
@@ -86,16 +86,20 @@ func (cp *connPackage) updateDependentGroups(completedGroup string, jobStatus st
func (cp *connPackage) checkRunningJobsStatus() {
var childJobs kbatch.JobList
labelSelector := labels.SelectorFromSet(labels.Set{
"jobmanager.raczylo.com/workflow-name": cp.mj.Name,
LabelWorkflowName: cp.mj.Name,
})
listOptions := &client.ListOptions{LabelSelector: labelSelector, Namespace: cp.mj.Namespace}
err := cp.r.Client.List(cp.ctx, &childJobs, listOptions)
if err != nil {
log.Log.Info("Unable to list child jobs", "error", err.Error())
cp.logger.Error(err, "Unable to list child jobs")
return
}
activeJobCount := 0
for _, childJob := range childJobs.Items {
if childJob.Status.Active > 0 {
activeJobCount++
}
for _, group := range cp.mj.Spec.Groups {
for _, job := range group.Jobs {
generatedJobName := jobNameGenerator(cp.mj.Name, group.Name, job.Name)
@@ -103,9 +107,11 @@ func (cp *connPackage) checkRunningJobsStatus() {
if childJob.Status.Succeeded > 0 && job.Status != ExecutionStatusSucceeded {
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeNormal, "Completed", "Job %s completed [prev: %s]", childJob.Name, job.Status)
job.Status = ExecutionStatusSucceeded
RecordJobSucceeded(cp.mj.Namespace, cp.mj.Name, group.Name)
} else if childJob.Status.Failed > 0 && job.Status != ExecutionStatusFailed {
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeWarning, "Failed", "Job %s failed [prev: %s]", childJob.Name, job.Status)
job.Status = ExecutionStatusFailed
RecordJobFailed(cp.mj.Namespace, cp.mj.Name, group.Name)
} else if childJob.Status.Active > 0 && job.Status != ExecutionStatusRunning {
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeNormal, "Running", "Job %s running [prev: %s]", childJob.Name, job.Status)
job.Status = ExecutionStatusRunning
@@ -116,12 +122,12 @@ func (cp *connPackage) checkRunningJobsStatus() {
}
}
}
SetActiveJobs(cp.mj.Namespace, cp.mj.Name, float64(activeJobCount))
}
func (cp *connPackage) runPendingJobs() {
// originalMainJobDefinition := cp.mj.DeepCopy()
for _, group := range cp.mj.Spec.Groups {
run_group := false
runGroup := false
groupJobsCompleted := 0
for _, job := range group.Jobs {
@@ -144,60 +150,59 @@ func (cp *connPackage) runPendingJobs() {
if pandati.ExistsInSlice(approvedStatuses, group.Status) {
if len(group.Dependencies) > 0 {
groupsCompleted := 0
for _, group_dependency := range group.Dependencies {
if group_dependency.Status == ExecutionStatusSucceeded {
for _, groupDep := range group.Dependencies {
if groupDep.Status == ExecutionStatusSucceeded {
groupsCompleted++
}
if group_dependency.Status == ExecutionStatusFailed {
if groupDep.Status == ExecutionStatusFailed {
group.Status = ExecutionStatusAborted
cp.updateDependentGroups(group.Name, ExecutionStatusFailed)
}
}
if groupsCompleted == len(group.Dependencies) {
run_group = true
runGroup = true
}
} else {
run_group = true
runGroup = true
}
if !run_group {
// fmt.Println("Group "+group.Name+" is not running as dependencies were not met", group.Dependencies)
if !runGroup {
continue // not running the group as dependencies were not met
} else {
group.Status = ExecutionStatusRunning
cp.updateDependentGroups(group.Name, ExecutionStatusRunning)
for _, job := range group.Jobs {
run_job := false
runJob := false
if job.Status == ExecutionStatusPending {
if len(job.Dependencies) > 0 {
jobsCompleted := 0
for _, job_dependency := range job.Dependencies {
if job_dependency.Status == ExecutionStatusSucceeded {
for _, jobDep := range job.Dependencies {
if jobDep.Status == ExecutionStatusSucceeded {
jobsCompleted++
}
if job_dependency.Status == ExecutionStatusFailed {
if jobDep.Status == ExecutionStatusFailed {
job.Status = ExecutionStatusAborted
cp.updateDependentJobs(job.Name, ExecutionStatusFailed)
}
}
if jobsCompleted == len(job.Dependencies) {
run_job = true
runJob = true
}
} else {
run_job = true
runJob = true
}
}
if !run_job {
if !runJob {
continue // job is not ready as dependencies were not met
} else {
approvedStatuses = []string{ExecutionStatusRunning, ExecutionStatusFailed, ExecutionStatusAborted}
if !pandati.ExistsInSlice(approvedStatuses, job.Status) {
err := cp.executeJob(job, group)
if err != nil {
log.Log.Info("Unable to execute job", "error", err.Error())
if !strings.Contains(err.Error(), "exists") {
cp.logger.Error(err, "Unable to execute job", "job", job.Name, "group", group.Name)
if !apierrors.IsAlreadyExists(err) {
job.Status = ExecutionStatusFailed
group.Status = ExecutionStatusFailed
cp.updateDependentJobs(job.Name, ExecutionStatusFailed)
@@ -213,8 +218,6 @@ func (cp *connPackage) runPendingJobs() {
}
}
}
// fmt.Println("Running group: ", group.Name, " with status: ", group.Status, " accepted: ", run_group)
}
}
}
@@ -225,16 +228,20 @@ func (cp *connPackage) executeJob(j *jobsmanagerv1beta1.ManagedJobDefinition, g
if retries == 0 {
return nil
}
retries32 := int32(retries)
// Ensure retries is within int32 bounds (max reasonable value for k8s backoff limit)
if retries < 0 || retries > 100 {
retries = 1 // default to 1 for invalid values
}
retries32 := int32(retries) // #nosec G115 - bounds checked above
return &retries32
}
// compile labels
labels := map[string]string{
"jobmanager.raczylo.com/workflow-name": cp.mj.Name,
"jobmanager.raczylo.com/group-name": g.Name,
"jobmanager.raczylo.com/job-name": generatedJobName,
"jobmanager.raczylo.com/job-id": j.Name,
LabelWorkflowName: cp.mj.Name,
LabelGroupName: g.Name,
LabelJobName: generatedJobName,
LabelJobID: j.Name,
}
// merge labels with j.Parameters.Labels
@@ -242,7 +249,13 @@ func (cp *connPackage) executeJob(j *jobsmanagerv1beta1.ManagedJobDefinition, g
labels[k] = v
}
job_handler := kbatch.Job{
annotations := map[string]string{}
for k, v := range j.CompiledParams.Annotations {
annotations[k] = v
}
k8sJob := kbatch.Job{
ObjectMeta: metav1.ObjectMeta{
Name: generatedJobName,
Namespace: cp.mj.Namespace,
@@ -250,9 +263,10 @@ func (cp *connPackage) executeJob(j *jobsmanagerv1beta1.ManagedJobDefinition, g
Spec: kbatch.JobSpec{
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Name: generatedJobName,
Namespace: cp.mj.Namespace,
Labels: labels,
Name: generatedJobName,
Namespace: cp.mj.Namespace,
Labels: labels,
Annotations: annotations,
},
Spec: corev1.PodSpec{
Volumes: j.CompiledParams.Volumes,
@@ -263,10 +277,11 @@ func (cp *connPackage) executeJob(j *jobsmanagerv1beta1.ManagedJobDefinition, g
Name: generatedJobName,
Image: j.Image,
Args: j.Args,
ImagePullPolicy: corev1.PullPolicy(j.CompiledParams.ImagePullPolicy),
ImagePullPolicy: getImagePullPolicy(j.CompiledParams.ImagePullPolicy),
EnvFrom: j.CompiledParams.FromEnv,
Env: j.CompiledParams.Env,
VolumeMounts: j.CompiledParams.VolumeMounts,
Resources: getResources(j.CompiledParams.Resources),
},
},
RestartPolicy: corev1.RestartPolicy(j.CompiledParams.RestartPolicy),
@@ -276,43 +291,50 @@ func (cp *connPackage) executeJob(j *jobsmanagerv1beta1.ManagedJobDefinition, g
},
}
getMetaRefForWorkflowData, err := cp.getOwnerReference()
ownerRef, err := cp.getOwnerReference()
if err != nil {
return err
}
job_handler.SetOwnerReferences([]metav1.OwnerReference{getMetaRefForWorkflowData})
k8sJob.SetOwnerReferences([]metav1.OwnerReference{ownerRef})
err = cp.r.Client.Create(cp.ctx, &job_handler)
if err != nil || pandati.IsZero(job_handler) {
err = cp.r.Client.Create(cp.ctx, &k8sJob)
if err != nil || pandati.IsZero(k8sJob) {
return err
}
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeNormal, "Created", "Created job %s", job_handler.Name)
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeNormal, "Created", "Created job %s", k8sJob.Name)
RecordJobCreated(cp.mj.Namespace, cp.mj.Name, g.Name)
return nil
}
func (cp *connPackage) checkOverallStatus() {
groupsCompleted := 0
failedGroup := ""
negativeStatuses := []string{ExecutionStatusFailed, ExecutionStatusAborted}
for _, group := range cp.mj.Spec.Groups {
if group.Status == ExecutionStatusSucceeded {
groupsCompleted++
} else if pandati.ExistsInSlice(negativeStatuses, group.Status) {
cp.mj.Status = ExecutionStatusFailed
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeWarning, "Failure", "Run failed in group %s", group.Name)
} else {
continue
} else if failedGroup == "" && pandati.ExistsInSlice(negativeStatuses, group.Status) {
failedGroup = group.Name
}
}
if groupsCompleted == len(cp.mj.Spec.Groups) {
switch {
case failedGroup != "":
if cp.mj.Status != ExecutionStatusFailed {
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeWarning, "Failure", "Run failed in group %s", failedGroup)
}
cp.mj.Status = ExecutionStatusFailed
case groupsCompleted == len(cp.mj.Spec.Groups):
if cp.mj.Status != ExecutionStatusSucceeded {
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeNormal, "Success", "Run completed successfuly")
cp.r.Recorder.Eventf(cp.mj, corev1.EventTypeNormal, "Success", "Run completed successfully")
}
cp.mj.Status = ExecutionStatusSucceeded
} else {
default:
cp.mj.Status = ExecutionStatusRunning
}
cp.r.Status().Update(cp.ctx, cp.mj)
if err := cp.r.Status().Update(cp.ctx, cp.mj); err != nil {
cp.logger.Error(err, "Failed to update overall status")
}
}
+522
View File
@@ -0,0 +1,522 @@
package controllers
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
kbatch "k8s.io/api/batch/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
// ScrapperTestSuite tests the CRD scrapper functionality
type ScrapperTestSuite struct {
suite.Suite
ctx context.Context
cancel context.CancelFunc
client *MockClient
reconciler *ManagedJobReconciler
}
func (s *ScrapperTestSuite) SetupTest() {
s.ctx, s.cancel = context.WithCancel(context.Background())
s.client = NewMockClient()
s.reconciler = &ManagedJobReconciler{
Client: s.client,
Scheme: s.client.Scheme(),
Recorder: NewFakeRecorder(),
}
}
func (s *ScrapperTestSuite) TearDownTest() {
s.cancel()
}
func TestScrapperSuite(t *testing.T) {
suite.Run(t, new(ScrapperTestSuite))
}
// Helper to create connPackage with proper request setup
func (s *ScrapperTestSuite) newConnPackage(mj *jobsmanagerv1beta1.ManagedJob) *connPackage {
cp := &connPackage{
ctx: s.ctx,
r: s.reconciler,
mj: mj,
req: ctrl.Request{
NamespacedName: types.NamespacedName{
Name: mj.Name,
Namespace: mj.Namespace,
},
},
logger: zap.New(),
}
cp.buildDependencyMaps()
return cp
}
// ==================== CHECK RUNNING JOBS STATUS TESTS ====================
func (s *ScrapperTestSuite) TestCheckRunningJobsStatus_NoJobs() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.checkRunningJobsStatus()
s.Equal(ExecutionStatusPending, mj.Spec.Groups[0].Jobs[0].Status)
}
func (s *ScrapperTestSuite) TestCheckRunningJobsStatus_JobSucceeded() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusRunning
s.client.AddManagedJob(mj)
k8sJob := NewTestK8sJob("workflow-group1-job1", "default", "workflow", "group1", kbatch.JobStatus{Succeeded: 1})
s.client.AddJob(k8sJob)
cp := s.newConnPackage(mj)
cp.checkRunningJobsStatus()
s.Equal(ExecutionStatusSucceeded, mj.Spec.Groups[0].Jobs[0].Status)
}
func (s *ScrapperTestSuite) TestCheckRunningJobsStatus_JobFailed() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusRunning
s.client.AddManagedJob(mj)
k8sJob := NewTestK8sJob("workflow-group1-job1", "default", "workflow", "group1", kbatch.JobStatus{Failed: 1})
s.client.AddJob(k8sJob)
cp := s.newConnPackage(mj)
cp.checkRunningJobsStatus()
s.Equal(ExecutionStatusFailed, mj.Spec.Groups[0].Jobs[0].Status)
}
func (s *ScrapperTestSuite) TestCheckRunningJobsStatus_JobActive() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusPending
s.client.AddManagedJob(mj)
k8sJob := NewTestK8sJob("workflow-group1-job1", "default", "workflow", "group1", kbatch.JobStatus{Active: 1})
s.client.AddJob(k8sJob)
cp := s.newConnPackage(mj)
cp.checkRunningJobsStatus()
s.Equal(ExecutionStatusRunning, mj.Spec.Groups[0].Jobs[0].Status)
}
func (s *ScrapperTestSuite) TestCheckRunningJobsStatus_ListError() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
s.client.ListError = ErrServerError
cp := s.newConnPackage(mj)
cp.checkRunningJobsStatus() // should not panic
s.Equal(ExecutionStatusPending, mj.Spec.Groups[0].Jobs[0].Status)
}
// ==================== RUN PENDING JOBS TESTS ====================
func (s *ScrapperTestSuite) TestRunPendingJobs_NoDependencies_StartsJob() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.runPendingJobs()
s.Equal(ExecutionStatusRunning, mj.Spec.Groups[0].Jobs[0].Status)
s.Equal(ExecutionStatusRunning, mj.Spec.Groups[0].Status)
s.GreaterOrEqual(s.client.CreateCalls, 1)
}
func (s *ScrapperTestSuite) TestRunPendingJobs_AllJobsCompleted_GroupSucceeds() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusSucceeded
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.runPendingJobs()
s.Equal(ExecutionStatusSucceeded, mj.Spec.Groups[0].Status)
}
func (s *ScrapperTestSuite) TestRunPendingJobs_GroupDependencyNotMet_Waits() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
NewTestGroup("group2", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job2", "busybox"),
}, NewTestDependency("group1", ExecutionStatusPending)),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.runPendingJobs()
s.Equal(ExecutionStatusRunning, mj.Spec.Groups[0].Status)
s.Equal(ExecutionStatusPending, mj.Spec.Groups[1].Status)
}
func (s *ScrapperTestSuite) TestRunPendingJobs_GroupDependencyMet_Starts() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
NewTestGroup("group2", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job2", "busybox"),
}, NewTestDependency("group1", ExecutionStatusSucceeded)),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusSucceeded
mj.Spec.Groups[0].Status = ExecutionStatusSucceeded
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.runPendingJobs()
s.Equal(ExecutionStatusRunning, mj.Spec.Groups[1].Status)
}
func (s *ScrapperTestSuite) TestRunPendingJobs_GroupDependencyFailed_Aborts() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
NewTestGroup("group2", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job2", "busybox"),
}, NewTestDependency("group1", ExecutionStatusFailed)),
})
controllerutil.AddFinalizer(mj, FinalizerName)
// Set group1 as already completed so it doesn't start running
mj.Spec.Groups[0].Status = ExecutionStatusFailed
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusFailed
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.runPendingJobs()
s.Equal(ExecutionStatusAborted, mj.Spec.Groups[1].Status)
}
func (s *ScrapperTestSuite) TestRunPendingJobs_JobDependencyFailed_Aborts() {
job1 := NewTestJobDef("job1", "busybox")
job2 := NewTestJobDef("job2", "busybox", NewTestDependency("job1", ExecutionStatusFailed))
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{job1, job2}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
// Set job1 as already failed so dependency check sees failed status
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusFailed
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.runPendingJobs()
// Job2 should be aborted because its dependency (job1) is failed
s.Equal(ExecutionStatusAborted, mj.Spec.Groups[0].Jobs[1].Status)
}
func (s *ScrapperTestSuite) TestRunPendingJobs_CreateJobError_FailsGroup() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
s.client.CreateError = ErrForbidden
cp := s.newConnPackage(mj)
cp.runPendingJobs()
s.Equal(ExecutionStatusFailed, mj.Spec.Groups[0].Jobs[0].Status)
s.Equal(ExecutionStatusFailed, mj.Spec.Groups[0].Status)
}
// ==================== CHECK OVERALL STATUS TESTS ====================
func (s *ScrapperTestSuite) TestCheckOverallStatus_AllGroupsSucceeded() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Status = ExecutionStatusSucceeded
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusSucceeded
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.checkOverallStatus()
s.Equal(ExecutionStatusSucceeded, mj.Status)
}
func (s *ScrapperTestSuite) TestCheckOverallStatus_GroupFailed() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Status = ExecutionStatusFailed
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.checkOverallStatus()
s.Equal(ExecutionStatusFailed, mj.Status)
}
func (s *ScrapperTestSuite) TestCheckOverallStatus_GroupRunning() {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Status = ExecutionStatusRunning
s.client.AddManagedJob(mj)
cp := s.newConnPackage(mj)
cp.checkOverallStatus()
s.Equal(ExecutionStatusRunning, mj.Status)
}
// ==================== MATRIX TEST: JOB STATUS TRANSITIONS ====================
func TestScrapper_JobStatusTransitions(t *testing.T) {
tests := []struct {
name string
scenario TestScenario
initialStatus string
k8sJobStatus kbatch.JobStatus
expectedStatus string
}{
{"good_pending_to_running", ScenarioGood, ExecutionStatusPending, kbatch.JobStatus{Active: 1}, ExecutionStatusRunning},
{"good_running_to_succeeded", ScenarioGood, ExecutionStatusRunning, kbatch.JobStatus{Succeeded: 1}, ExecutionStatusSucceeded},
{"notgood_running_to_failed", ScenarioNotGood, ExecutionStatusRunning, kbatch.JobStatus{Failed: 1}, ExecutionStatusFailed},
{"edge_already_succeeded", ScenarioGood, ExecutionStatusSucceeded, kbatch.JobStatus{Succeeded: 1}, ExecutionStatusSucceeded},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ctx := context.Background()
client := NewMockClient()
reconciler := &ManagedJobReconciler{
Client: client,
Scheme: client.Scheme(),
Recorder: NewFakeRecorder(),
}
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = tt.initialStatus
client.AddManagedJob(mj)
k8sJob := NewTestK8sJob("workflow-group1-job1", "default", "workflow", "group1", tt.k8sJobStatus)
client.AddJob(k8sJob)
cp := &connPackage{
ctx: ctx,
r: reconciler,
mj: mj,
req: ctrl.Request{NamespacedName: types.NamespacedName{Name: "workflow", Namespace: "default"}},
logger: zap.New(),
}
cp.buildDependencyMaps()
cp.checkRunningJobsStatus()
assert.Equal(t, tt.expectedStatus, mj.Spec.Groups[0].Jobs[0].Status)
})
}
}
// ==================== EXECUTE JOB TESTS ====================
func TestExecuteJob_CreatesK8sJob(t *testing.T) {
ctx := context.Background()
client := NewMockClient()
reconciler := &ManagedJobReconciler{
Client: client,
Scheme: client.Scheme(),
Recorder: NewFakeRecorder(),
}
mj := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: metav1.ObjectMeta{
Name: "workflow",
Namespace: "default",
UID: "test-uid",
},
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "group1",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{Name: "job1", Image: "busybox:latest", Args: []string{"echo", "hello"}},
},
},
},
},
}
controllerutil.AddFinalizer(mj, FinalizerName)
client.AddManagedJob(mj)
cp := &connPackage{
ctx: ctx,
r: reconciler,
mj: mj,
req: ctrl.Request{NamespacedName: types.NamespacedName{Name: "workflow", Namespace: "default"}},
logger: zap.New(),
}
cp.buildDependencyMaps()
err := cp.executeJob(mj.Spec.Groups[0].Jobs[0], mj.Spec.Groups[0])
require.NoError(t, err)
assert.Equal(t, 1, client.CreateCalls)
}
func TestExecuteJob_WithRetries(t *testing.T) {
ctx := context.Background()
client := NewMockClient()
reconciler := &ManagedJobReconciler{
Client: client,
Scheme: client.Scheme(),
Recorder: NewFakeRecorder(),
}
mj := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: metav1.ObjectMeta{
Name: "workflow",
Namespace: "default",
UID: "test-uid",
},
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Retries: 3,
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "group1",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{Name: "job1", Image: "busybox:latest"},
},
},
},
},
}
controllerutil.AddFinalizer(mj, FinalizerName)
client.AddManagedJob(mj)
cp := &connPackage{
ctx: ctx,
r: reconciler,
mj: mj,
req: ctrl.Request{NamespacedName: types.NamespacedName{Name: "workflow", Namespace: "default"}},
logger: zap.New(),
}
cp.buildDependencyMaps()
err := cp.executeJob(mj.Spec.Groups[0].Jobs[0], mj.Spec.Groups[0])
require.NoError(t, err)
}
func TestExecuteJob_CreateError(t *testing.T) {
ctx := context.Background()
client := NewMockClient()
client.CreateError = ErrForbidden
reconciler := &ManagedJobReconciler{
Client: client,
Scheme: client.Scheme(),
Recorder: NewFakeRecorder(),
}
mj := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: metav1.ObjectMeta{
Name: "workflow",
Namespace: "default",
UID: "test-uid",
},
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "group1",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{Name: "job1", Image: "busybox:latest"},
},
},
},
},
}
controllerutil.AddFinalizer(mj, FinalizerName)
client.AddManagedJob(mj)
cp := &connPackage{
ctx: ctx,
r: reconciler,
mj: mj,
req: ctrl.Request{NamespacedName: types.NamespacedName{Name: "workflow", Namespace: "default"}},
logger: zap.New(),
}
cp.buildDependencyMaps()
err := cp.executeJob(mj.Spec.Groups[0].Jobs[0], mj.Spec.Groups[0])
require.Error(t, err)
}
+9 -2
View File
@@ -10,10 +10,17 @@ const (
ExecutionStatusUnknown string = "unknown"
)
var (
jobOwnerKey = ".metadata.controller"
// Label keys used for job tracking and identification
const (
LabelWorkflowName = "jobmanager.raczylo.com/workflow-name"
LabelGroupName = "jobmanager.raczylo.com/group-name"
LabelJobName = "jobmanager.raczylo.com/job-name"
LabelJobID = "jobmanager.raczylo.com/job-id"
)
// FinalizerName is the finalizer used to ensure cleanup of child resources
const FinalizerName = "jobmanager.raczylo.com/finalizer"
type (
ExecutionStatus string
+59 -23
View File
@@ -5,11 +5,11 @@ import (
"strings"
"sync"
"github.com/go-logr/logr"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"raczylo.com/jobs-manager-operator/api/v1beta1"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/log"
)
func jobNameGenerator(name ...string) string {
@@ -17,19 +17,37 @@ func jobNameGenerator(name ...string) string {
return strings.ToLower(strings.Join(name, "-"))
}
type jobStatusUpdate struct {
Job *jobsmanagerv1beta1.ManagedJob
PatchedResource string
Status string
type connPackage struct {
r *ManagedJobReconciler
ctx context.Context
req ctrl.Request
mtx sync.Mutex
mj *jobsmanagerv1beta1.ManagedJob
logger logr.Logger
// jobDepMap maps job names to dependencies that reference them (for O(1) lookup)
jobDepMap map[string][]*jobsmanagerv1beta1.ManagedJobDependencies
// groupDepMap maps group names to dependencies that reference them (for O(1) lookup)
groupDepMap map[string][]*jobsmanagerv1beta1.ManagedJobDependencies
}
type connPackage struct {
r *ManagedJobReconciler
ctx context.Context
req ctrl.Request
mtx sync.Mutex
mj *jobsmanagerv1beta1.ManagedJob
dependencyTree Tree
// buildDependencyMaps constructs lookup maps for efficient dependency status updates.
// This converts O(n*m) lookups to O(1) by mapping job/group names to their dependents.
func (cp *connPackage) buildDependencyMaps() {
cp.jobDepMap = make(map[string][]*jobsmanagerv1beta1.ManagedJobDependencies)
cp.groupDepMap = make(map[string][]*jobsmanagerv1beta1.ManagedJobDependencies)
for _, group := range cp.mj.Spec.Groups {
// Map group dependencies
for _, dep := range group.Dependencies {
cp.groupDepMap[dep.Name] = append(cp.groupDepMap[dep.Name], dep)
}
// Map job dependencies
for _, job := range group.Jobs {
for _, dep := range job.Dependencies {
cp.jobDepMap[dep.Name] = append(cp.jobDepMap[dep.Name], dep)
}
}
}
}
func (cp *connPackage) getOwnerReference() (metav1.OwnerReference, error) {
@@ -40,7 +58,7 @@ func (cp *connPackage) getOwnerReference() (metav1.OwnerReference, error) {
}
t := true
return metav1.OwnerReference{
APIVersion: v1beta1.GroupVersion.String(),
APIVersion: jobsmanagerv1beta1.GroupVersion.String(),
Kind: "ManagedJob",
Name: mj.Name,
UID: mj.UID,
@@ -50,15 +68,33 @@ func (cp *connPackage) getOwnerReference() (metav1.OwnerReference, error) {
func (cp *connPackage) updateCRDStatusDirectly() error {
cp.mtx.Lock()
err := cp.r.Update(cp.ctx, cp.mj)
if err != nil {
// log.Log.Info("Error", err.Error(), "more", "Unable to update ManagedJob status directly")
defer cp.mtx.Unlock()
if err := cp.r.Update(cp.ctx, cp.mj); err != nil {
cp.logger.Error(err, "Unable to update ManagedJob status directly")
return err
}
// get updated ManagedJob
err = cp.r.Client.Get(cp.ctx, cp.req.NamespacedName, cp.mj)
if err != nil {
log.Log.Error(err, "Unable to get updated ManagedJob")
if err := cp.r.Client.Get(cp.ctx, cp.req.NamespacedName, cp.mj); err != nil {
cp.logger.Error(err, "Unable to get updated ManagedJob")
return err
}
cp.mtx.Unlock()
return err
return nil
}
// getImagePullPolicy returns the specified pull policy or IfNotPresent as default
func getImagePullPolicy(policy string) corev1.PullPolicy {
if policy == "" {
return corev1.PullIfNotPresent
}
return corev1.PullPolicy(policy)
}
// getResources returns the resource requirements or empty requirements if nil
func getResources(resources *corev1.ResourceRequirements) corev1.ResourceRequirements {
if resources == nil {
return corev1.ResourceRequirements{}
}
return *resources
}
+353
View File
@@ -0,0 +1,353 @@
/*
Copyright 2023.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package controllers
import (
"testing"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
func TestJobNameGenerator(t *testing.T) {
tests := []struct {
name string
parts []string
expected string
}{
{
name: "single part",
parts: []string{"workflow"},
expected: "workflow",
},
{
name: "multiple parts",
parts: []string{"workflow", "group", "job"},
expected: "workflow-group-job",
},
{
name: "uppercase conversion",
parts: []string{"Workflow", "GROUP", "Job"},
expected: "workflow-group-job",
},
{
name: "mixed case",
parts: []string{"MyWorkflow", "TestGroup", "BuildJob"},
expected: "myworkflow-testgroup-buildjob",
},
{
name: "empty parts",
parts: []string{},
expected: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := jobNameGenerator(tt.parts...)
if result != tt.expected {
t.Errorf("jobNameGenerator(%v) = %v, want %v", tt.parts, result, tt.expected)
}
})
}
}
func TestGetImagePullPolicy(t *testing.T) {
tests := []struct {
name string
policy string
expected corev1.PullPolicy
}{
{
name: "empty policy defaults to IfNotPresent",
policy: "",
expected: corev1.PullIfNotPresent,
},
{
name: "Always policy",
policy: "Always",
expected: corev1.PullAlways,
},
{
name: "Never policy",
policy: "Never",
expected: corev1.PullNever,
},
{
name: "IfNotPresent policy",
policy: "IfNotPresent",
expected: corev1.PullIfNotPresent,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := getImagePullPolicy(tt.policy)
if result != tt.expected {
t.Errorf("getImagePullPolicy(%v) = %v, want %v", tt.policy, result, tt.expected)
}
})
}
}
func TestGetResources(t *testing.T) {
tests := []struct {
name string
resources *corev1.ResourceRequirements
expectNil bool
}{
{
name: "nil resources returns empty",
resources: nil,
expectNil: false,
},
{
name: "non-nil resources returned as-is",
resources: &corev1.ResourceRequirements{
Limits: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("100m"),
corev1.ResourceMemory: resource.MustParse("128Mi"),
},
},
expectNil: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := getResources(tt.resources)
if tt.resources == nil {
// Should return empty struct
if result.Limits != nil || result.Requests != nil {
t.Errorf("getResources(nil) should return empty ResourceRequirements")
}
} else {
// Should return the same values
if len(result.Limits) != len(tt.resources.Limits) {
t.Errorf("getResources() limits mismatch")
}
}
})
}
}
func TestBuildDependencyMaps(t *testing.T) {
// Create a mock ManagedJob with dependencies
mj := &jobsmanagerv1beta1.ManagedJob{
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "group1",
Dependencies: []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "init-group", Status: "pending"},
},
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{
Name: "job1",
Dependencies: []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "setup-job", Status: "pending"},
},
},
{
Name: "job2",
Dependencies: []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "setup-job", Status: "pending"},
{Name: "job1", Status: "pending"},
},
},
},
},
{
Name: "group2",
Dependencies: []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "group1", Status: "pending"},
},
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{
Name: "job3",
Dependencies: nil,
},
},
},
},
},
}
cp := &connPackage{mj: mj}
cp.buildDependencyMaps()
// Verify job dependency map
t.Run("job dependency map - setup-job has 2 dependents", func(t *testing.T) {
deps, exists := cp.jobDepMap["setup-job"]
if !exists {
t.Fatal("setup-job should exist in jobDepMap")
}
if len(deps) != 2 {
t.Errorf("setup-job should have 2 dependents, got %d", len(deps))
}
})
t.Run("job dependency map - job1 has 1 dependent", func(t *testing.T) {
deps, exists := cp.jobDepMap["job1"]
if !exists {
t.Fatal("job1 should exist in jobDepMap")
}
if len(deps) != 1 {
t.Errorf("job1 should have 1 dependent, got %d", len(deps))
}
})
t.Run("job dependency map - non-existent job", func(t *testing.T) {
_, exists := cp.jobDepMap["non-existent"]
if exists {
t.Error("non-existent job should not be in jobDepMap")
}
})
// Verify group dependency map
t.Run("group dependency map - init-group has 1 dependent", func(t *testing.T) {
deps, exists := cp.groupDepMap["init-group"]
if !exists {
t.Fatal("init-group should exist in groupDepMap")
}
if len(deps) != 1 {
t.Errorf("init-group should have 1 dependent, got %d", len(deps))
}
})
t.Run("group dependency map - group1 has 1 dependent", func(t *testing.T) {
deps, exists := cp.groupDepMap["group1"]
if !exists {
t.Fatal("group1 should exist in groupDepMap")
}
if len(deps) != 1 {
t.Errorf("group1 should have 1 dependent, got %d", len(deps))
}
})
}
func TestUpdateDependentJobs(t *testing.T) {
dep1 := &jobsmanagerv1beta1.ManagedJobDependencies{Name: "target-job", Status: "pending"}
dep2 := &jobsmanagerv1beta1.ManagedJobDependencies{Name: "target-job", Status: "pending"}
dep3 := &jobsmanagerv1beta1.ManagedJobDependencies{Name: "other-job", Status: "pending"}
cp := &connPackage{
jobDepMap: map[string][]*jobsmanagerv1beta1.ManagedJobDependencies{
"target-job": {dep1, dep2},
"other-job": {dep3},
},
}
// Update target-job status
cp.updateDependentJobs("target-job", "succeeded")
t.Run("target-job dependents updated", func(t *testing.T) {
if dep1.Status != "succeeded" {
t.Errorf("dep1.Status = %v, want succeeded", dep1.Status)
}
if dep2.Status != "succeeded" {
t.Errorf("dep2.Status = %v, want succeeded", dep2.Status)
}
})
t.Run("other-job dependents unchanged", func(t *testing.T) {
if dep3.Status != "pending" {
t.Errorf("dep3.Status = %v, want pending (should be unchanged)", dep3.Status)
}
})
t.Run("non-existent job is safe", func(t *testing.T) {
// Should not panic
cp.updateDependentJobs("non-existent-job", "succeeded")
})
}
func TestUpdateDependentGroups(t *testing.T) {
dep1 := &jobsmanagerv1beta1.ManagedJobDependencies{Name: "target-group", Status: "pending"}
dep2 := &jobsmanagerv1beta1.ManagedJobDependencies{Name: "other-group", Status: "pending"}
cp := &connPackage{
groupDepMap: map[string][]*jobsmanagerv1beta1.ManagedJobDependencies{
"target-group": {dep1},
"other-group": {dep2},
},
}
cp.updateDependentGroups("target-group", "succeeded")
t.Run("target-group dependents updated", func(t *testing.T) {
if dep1.Status != "succeeded" {
t.Errorf("dep1.Status = %v, want succeeded", dep1.Status)
}
})
t.Run("other-group dependents unchanged", func(t *testing.T) {
if dep2.Status != "pending" {
t.Errorf("dep2.Status = %v, want pending (should be unchanged)", dep2.Status)
}
})
}
func TestCompileParameters(t *testing.T) {
cp := &connPackage{}
t.Run("merges multiple parameter sets", func(t *testing.T) {
params1 := jobsmanagerv1beta1.ManagedJobParameters{
ServiceAccount: "sa1",
RestartPolicy: "Never",
}
params2 := jobsmanagerv1beta1.ManagedJobParameters{
ServiceAccount: "sa2", // should override
ImagePullPolicy: "Always",
}
result := cp.compileParameters(params1, params2)
if result.ServiceAccount != "sa2" {
t.Errorf("ServiceAccount = %v, want sa2", result.ServiceAccount)
}
if result.RestartPolicy != "Never" {
t.Errorf("RestartPolicy = %v, want Never", result.RestartPolicy)
}
if result.ImagePullPolicy != "Always" {
t.Errorf("ImagePullPolicy = %v, want Always", result.ImagePullPolicy)
}
})
t.Run("merges env vars", func(t *testing.T) {
params1 := jobsmanagerv1beta1.ManagedJobParameters{
Env: []corev1.EnvVar{{Name: "VAR1", Value: "val1"}},
}
params2 := jobsmanagerv1beta1.ManagedJobParameters{
Env: []corev1.EnvVar{{Name: "VAR2", Value: "val2"}},
}
result := cp.compileParameters(params1, params2)
if len(result.Env) != 2 {
t.Errorf("len(Env) = %v, want 2", len(result.Env))
}
})
t.Run("handles empty parameters", func(t *testing.T) {
result := cp.compileParameters(jobsmanagerv1beta1.ManagedJobParameters{})
if result.ServiceAccount != "" {
t.Errorf("ServiceAccount should be empty")
}
})
}
+102 -13
View File
@@ -18,18 +18,26 @@ package controllers
import (
"context"
"time"
"github.com/lukaszraczylo/pandati"
kbatch "k8s.io/api/batch/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/tools/record"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
const (
// RequeueDelay is the time to wait before requeuing when jobs are running
RequeueDelay = 30 * time.Second
)
// ManagedJobReconciler reconciles a ManagedJob object
type ManagedJobReconciler struct {
client.Client
@@ -40,47 +48,128 @@ type ManagedJobReconciler struct {
//+kubebuilder:rbac:groups=jobsmanager.raczylo.com,resources=managedjobs,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=jobsmanager.raczylo.com,resources=managedjobs/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=jobsmanager.raczylo.com,resources=managedjobs/finalizers,verbs=update
//+kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups="",resources=events,verbs=create;update;patch;delete;get;list;watch
// Reconcile ensures ManagedJob workflows progress toward completion.
// It orchestrates job execution respecting dependencies, manages retries,
// and tracks overall workflow status.
func (r *ManagedJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)
cp := &connPackage{
r: r,
ctx: ctx,
req: req,
dependencyTree: nil,
}
logger := log.FromContext(ctx).WithValues("managedJob", req.NamespacedName)
var managedJob jobsmanagerv1beta1.ManagedJob
if err := r.Get(ctx, req.NamespacedName, &managedJob); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
cp.mj = &managedJob
cp := &connPackage{
r: r,
ctx: ctx,
req: req,
logger: logger,
mj: &managedJob,
}
// Handle deletion with finalizer
if !managedJob.DeletionTimestamp.IsZero() {
return r.handleDeletion(ctx, cp)
}
// Add finalizer if not present
if !controllerutil.ContainsFinalizer(&managedJob, FinalizerName) {
controllerutil.AddFinalizer(&managedJob, FinalizerName)
if err := r.Update(ctx, &managedJob); err != nil {
logger.Error(err, "Failed to add finalizer")
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: time.Second}, nil
}
originalMainJobDefinition := cp.mj.DeepCopy()
cp.generateDependencyTree()
cp.buildDependencyMaps() // Build lookup maps for O(1) dependency updates
_, theSame, _ := pandati.CompareStructsReplaced(originalMainJobDefinition, cp.mj)
if !theSame {
cp.updateCRDStatusDirectly()
if err := cp.updateCRDStatusDirectly(); err != nil {
logger.Error(err, "Failed to update CRD status after dependency tree generation")
}
return ctrl.Result{}, nil
}
originalMainJobDefinition = cp.mj.DeepCopy()
// TODO: Re-enable after testing
cp.checkRunningJobsStatus()
cp.runPendingJobs()
_, theSame, _ = pandati.CompareStructsReplaced(originalMainJobDefinition, cp.mj)
if !theSame {
cp.updateCRDStatusDirectly()
if err := cp.updateCRDStatusDirectly(); err != nil {
logger.Error(err, "Failed to update CRD status after job processing")
}
}
cp.checkOverallStatus()
// fmt.Printf("Reconcile: %# v", pretty.Formatter(r.Updater))
// If workflow is still running, requeue after a delay to check status
if cp.mj.Status == ExecutionStatusRunning {
return ctrl.Result{RequeueAfter: RequeueDelay}, nil
}
return ctrl.Result{}, nil
}
// handleDeletion cleans up child jobs before removing the finalizer
func (r *ManagedJobReconciler) handleDeletion(ctx context.Context, cp *connPackage) (ctrl.Result, error) {
if !controllerutil.ContainsFinalizer(cp.mj, FinalizerName) {
return ctrl.Result{}, nil
}
cp.logger.Info("Cleaning up child jobs before deletion")
// Delete all child jobs
if err := r.deleteChildJobs(ctx, cp); err != nil {
cp.logger.Error(err, "Failed to delete child jobs")
return ctrl.Result{}, err
}
// Remove finalizer
controllerutil.RemoveFinalizer(cp.mj, FinalizerName)
if err := r.Update(ctx, cp.mj); err != nil {
cp.logger.Error(err, "Failed to remove finalizer")
return ctrl.Result{}, err
}
cp.logger.Info("Successfully cleaned up ManagedJob")
return ctrl.Result{}, nil
}
// deleteChildJobs removes all jobs owned by this ManagedJob
func (r *ManagedJobReconciler) deleteChildJobs(ctx context.Context, cp *connPackage) error {
var childJobs kbatch.JobList
labelSelector := labels.SelectorFromSet(labels.Set{
LabelWorkflowName: cp.mj.Name,
})
listOptions := &client.ListOptions{
LabelSelector: labelSelector,
Namespace: cp.mj.Namespace,
}
if err := r.Client.List(ctx, &childJobs, listOptions); err != nil {
return err
}
for i := range childJobs.Items {
job := &childJobs.Items[i]
if err := r.Client.Delete(ctx, job, client.PropagationPolicy("Background")); err != nil {
cp.logger.Error(err, "Failed to delete child job", "job", job.Name)
// Continue trying to delete other jobs
} else {
cp.logger.Info("Deleted child job", "job", job.Name)
}
}
return nil
}
// SetupWithManager sets up the controller with the Manager.
func (r *ManagedJobReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
+277
View File
@@ -0,0 +1,277 @@
//go:build envtest
/*
Copyright 2023.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package controllers
import (
"context"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
kbatch "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
var _ = Describe("ManagedJob Controller", func() {
const (
ManagedJobName = "test-workflow"
ManagedJobNamespace = "default"
timeout = time.Second * 10
interval = time.Millisecond * 250
)
Context("When creating a ManagedJob", func() {
It("Should add finalizer to new ManagedJob", func() {
ctx := context.Background()
managedJob := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: metav1.ObjectMeta{
Name: ManagedJobName + "-finalizer",
Namespace: ManagedJobNamespace,
},
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "group1",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{
Name: "job1",
Image: "busybox:latest",
Args: []string{"echo", "hello"},
},
},
},
},
},
}
Expect(k8sClient.Create(ctx, managedJob)).Should(Succeed())
lookupKey := types.NamespacedName{Name: ManagedJobName + "-finalizer", Namespace: ManagedJobNamespace}
createdManagedJob := &jobsmanagerv1beta1.ManagedJob{}
Eventually(func() bool {
err := k8sClient.Get(ctx, lookupKey, createdManagedJob)
return err == nil
}, timeout, interval).Should(BeTrue())
// Trigger reconciliation manually since we don't have the controller running
reconciler := &ManagedJobReconciler{
Client: k8sClient,
Scheme: k8sClient.Scheme(),
}
_, err := reconciler.Reconcile(ctx, reconcile.Request{NamespacedName: lookupKey})
Expect(err).NotTo(HaveOccurred())
// Verify finalizer was added
Eventually(func() bool {
err := k8sClient.Get(ctx, lookupKey, createdManagedJob)
if err != nil {
return false
}
for _, f := range createdManagedJob.Finalizers {
if f == FinalizerName {
return true
}
}
return false
}, timeout, interval).Should(BeTrue())
// Cleanup
Expect(k8sClient.Delete(ctx, createdManagedJob)).Should(Succeed())
})
It("Should initialize job statuses to pending", func() {
ctx := context.Background()
managedJob := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: metav1.ObjectMeta{
Name: ManagedJobName + "-status",
Namespace: ManagedJobNamespace,
},
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "init-group",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{
Name: "init-job",
Image: "busybox:latest",
},
},
},
{
Name: "main-group",
Dependencies: []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "init-group"},
},
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{
Name: "main-job",
Image: "busybox:latest",
Dependencies: []*jobsmanagerv1beta1.ManagedJobDependencies{
{Name: "init-job"},
},
},
},
},
},
},
}
Expect(k8sClient.Create(ctx, managedJob)).Should(Succeed())
lookupKey := types.NamespacedName{Name: ManagedJobName + "-status", Namespace: ManagedJobNamespace}
createdManagedJob := &jobsmanagerv1beta1.ManagedJob{}
Eventually(func() bool {
err := k8sClient.Get(ctx, lookupKey, createdManagedJob)
return err == nil
}, timeout, interval).Should(BeTrue())
// Trigger reconciliation
reconciler := &ManagedJobReconciler{
Client: k8sClient,
Scheme: k8sClient.Scheme(),
}
// First reconcile adds finalizer
_, err := reconciler.Reconcile(ctx, reconcile.Request{NamespacedName: lookupKey})
Expect(err).NotTo(HaveOccurred())
// Second reconcile processes jobs
_, err = reconciler.Reconcile(ctx, reconcile.Request{NamespacedName: lookupKey})
Expect(err).NotTo(HaveOccurred())
// Verify statuses
Eventually(func() bool {
err := k8sClient.Get(ctx, lookupKey, createdManagedJob)
if err != nil {
return false
}
// Check that groups have jobs with pending status initially
for _, g := range createdManagedJob.Spec.Groups {
for _, j := range g.Jobs {
if j.Status == "" {
return false
}
}
}
return true
}, timeout, interval).Should(BeTrue())
// Cleanup
Expect(k8sClient.Delete(ctx, createdManagedJob)).Should(Succeed())
})
})
Context("When a ManagedJob is deleted", func() {
It("Should clean up child jobs", func() {
ctx := context.Background()
// Create a ManagedJob
managedJob := &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: metav1.ObjectMeta{
Name: ManagedJobName + "-cleanup",
Namespace: ManagedJobNamespace,
},
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: []*jobsmanagerv1beta1.ManagedJobGroup{
{
Name: "cleanup-group",
Jobs: []*jobsmanagerv1beta1.ManagedJobDefinition{
{
Name: "cleanup-job",
Image: "busybox:latest",
},
},
},
},
},
}
Expect(k8sClient.Create(ctx, managedJob)).Should(Succeed())
lookupKey := types.NamespacedName{Name: ManagedJobName + "-cleanup", Namespace: ManagedJobNamespace}
// Create a child job with the workflow label
childJob := &kbatch.Job{
ObjectMeta: metav1.ObjectMeta{
Name: "test-workflow-cleanup-cleanup-group-cleanup-job",
Namespace: ManagedJobNamespace,
Labels: map[string]string{
LabelWorkflowName: ManagedJobName + "-cleanup",
},
},
Spec: kbatch.JobSpec{
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: "test",
Image: "busybox:latest",
},
},
RestartPolicy: corev1.RestartPolicyNever,
},
},
},
}
Expect(k8sClient.Create(ctx, childJob)).Should(Succeed())
// Verify child job exists
childJobKey := types.NamespacedName{
Name: "test-workflow-cleanup-cleanup-group-cleanup-job",
Namespace: ManagedJobNamespace,
}
Eventually(func() bool {
err := k8sClient.Get(ctx, childJobKey, &kbatch.Job{})
return err == nil
}, timeout, interval).Should(BeTrue())
// Delete the ManagedJob
createdManagedJob := &jobsmanagerv1beta1.ManagedJob{}
Expect(k8sClient.Get(ctx, lookupKey, createdManagedJob)).Should(Succeed())
Expect(k8sClient.Delete(ctx, createdManagedJob)).Should(Succeed())
})
})
})
var _ = Describe("Execution Status Constants", func() {
It("Should have correct status values", func() {
Expect(ExecutionStatusPending).To(Equal("pending"))
Expect(ExecutionStatusRunning).To(Equal("running"))
Expect(ExecutionStatusSucceeded).To(Equal("succeeded"))
Expect(ExecutionStatusFailed).To(Equal("failed"))
Expect(ExecutionStatusAborted).To(Equal("aborted"))
})
It("Should have correct label values", func() {
Expect(LabelWorkflowName).To(Equal("jobmanager.raczylo.com/workflow-name"))
Expect(LabelGroupName).To(Equal("jobmanager.raczylo.com/group-name"))
Expect(LabelJobName).To(Equal("jobmanager.raczylo.com/job-name"))
Expect(LabelJobID).To(Equal("jobmanager.raczylo.com/job-id"))
})
})
+85
View File
@@ -0,0 +1,85 @@
package controllers
import (
"github.com/prometheus/client_golang/prometheus"
"sigs.k8s.io/controller-runtime/pkg/metrics"
)
var (
// JobsCreatedTotal tracks the total number of jobs created
JobsCreatedTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "managedjob_jobs_created_total",
Help: "Total number of Kubernetes jobs created by the operator",
},
[]string{"namespace", "workflow", "group"},
)
// JobsSucceededTotal tracks the total number of jobs that succeeded
JobsSucceededTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "managedjob_jobs_succeeded_total",
Help: "Total number of jobs that completed successfully",
},
[]string{"namespace", "workflow", "group"},
)
// JobsFailedTotal tracks the total number of jobs that failed
JobsFailedTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "managedjob_jobs_failed_total",
Help: "Total number of jobs that failed",
},
[]string{"namespace", "workflow", "group"},
)
// ReconciliationDuration tracks how long reconciliations take
ReconciliationDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Name: "managedjob_reconciliation_duration_seconds",
Help: "Time spent reconciling ManagedJob resources",
Buckets: prometheus.ExponentialBuckets(0.001, 2, 15), // 1ms to ~16s
},
[]string{"namespace", "workflow"},
)
// ActiveJobs tracks the number of currently running jobs per workflow
ActiveJobs = prometheus.NewGaugeVec(
prometheus.GaugeOpts{
Name: "managedjob_active_jobs",
Help: "Number of currently active (running) jobs per workflow",
},
[]string{"namespace", "workflow"},
)
)
func init() {
// Register custom metrics with the controller-runtime metrics registry
metrics.Registry.MustRegister(
JobsCreatedTotal,
JobsSucceededTotal,
JobsFailedTotal,
ReconciliationDuration,
ActiveJobs,
)
}
// RecordJobCreated increments the job created counter
func RecordJobCreated(namespace, workflow, group string) {
JobsCreatedTotal.WithLabelValues(namespace, workflow, group).Inc()
}
// RecordJobSucceeded increments the job succeeded counter
func RecordJobSucceeded(namespace, workflow, group string) {
JobsSucceededTotal.WithLabelValues(namespace, workflow, group).Inc()
}
// RecordJobFailed increments the job failed counter
func RecordJobFailed(namespace, workflow, group string) {
JobsFailedTotal.WithLabelValues(namespace, workflow, group).Inc()
}
// SetActiveJobs sets the number of active jobs for a workflow
func SetActiveJobs(namespace, workflow string, count float64) {
ActiveJobs.WithLabelValues(namespace, workflow).Set(count)
}
+152
View File
@@ -0,0 +1,152 @@
package controllers
import (
"testing"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
"github.com/stretchr/testify/assert"
)
func TestRecordJobCreated(t *testing.T) {
// Reset counter for testing
JobsCreatedTotal.Reset()
// Record job creation
RecordJobCreated("default", "test-workflow", "group1")
RecordJobCreated("default", "test-workflow", "group1")
RecordJobCreated("production", "other-workflow", "group2")
// Verify counts
assert.Equal(t, float64(2), testutil.ToFloat64(JobsCreatedTotal.WithLabelValues("default", "test-workflow", "group1")))
assert.Equal(t, float64(1), testutil.ToFloat64(JobsCreatedTotal.WithLabelValues("production", "other-workflow", "group2")))
}
func TestRecordJobSucceeded(t *testing.T) {
// Reset counter for testing
JobsSucceededTotal.Reset()
// Record job success
RecordJobSucceeded("default", "workflow1", "group1")
RecordJobSucceeded("default", "workflow1", "group1")
RecordJobSucceeded("default", "workflow1", "group2")
// Verify counts
assert.Equal(t, float64(2), testutil.ToFloat64(JobsSucceededTotal.WithLabelValues("default", "workflow1", "group1")))
assert.Equal(t, float64(1), testutil.ToFloat64(JobsSucceededTotal.WithLabelValues("default", "workflow1", "group2")))
}
func TestRecordJobFailed(t *testing.T) {
// Reset counter for testing
JobsFailedTotal.Reset()
// Record job failure
RecordJobFailed("production", "critical-workflow", "init")
RecordJobFailed("production", "critical-workflow", "init")
RecordJobFailed("production", "critical-workflow", "cleanup")
// Verify counts
assert.Equal(t, float64(2), testutil.ToFloat64(JobsFailedTotal.WithLabelValues("production", "critical-workflow", "init")))
assert.Equal(t, float64(1), testutil.ToFloat64(JobsFailedTotal.WithLabelValues("production", "critical-workflow", "cleanup")))
}
func TestSetActiveJobs(t *testing.T) {
// Reset gauge for testing
ActiveJobs.Reset()
// Set active jobs
SetActiveJobs("default", "workflow1", 5)
SetActiveJobs("production", "workflow2", 3)
SetActiveJobs("default", "workflow1", 2) // Update to lower value
// Verify values
assert.Equal(t, float64(2), testutil.ToFloat64(ActiveJobs.WithLabelValues("default", "workflow1")))
assert.Equal(t, float64(3), testutil.ToFloat64(ActiveJobs.WithLabelValues("production", "workflow2")))
}
func TestMetricsLabels(t *testing.T) {
tests := []struct {
name string
namespace string
workflow string
group string
metric *prometheus.CounterVec
recorder func(ns, wf, grp string)
}{
{
name: "created_metric_labels",
namespace: "ns1",
workflow: "wf1",
group: "grp1",
metric: JobsCreatedTotal,
recorder: RecordJobCreated,
},
{
name: "succeeded_metric_labels",
namespace: "ns2",
workflow: "wf2",
group: "grp2",
metric: JobsSucceededTotal,
recorder: RecordJobSucceeded,
},
{
name: "failed_metric_labels",
namespace: "ns3",
workflow: "wf3",
group: "grp3",
metric: JobsFailedTotal,
recorder: RecordJobFailed,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
tt.metric.Reset()
tt.recorder(tt.namespace, tt.workflow, tt.group)
value := testutil.ToFloat64(tt.metric.WithLabelValues(tt.namespace, tt.workflow, tt.group))
assert.Equal(t, float64(1), value)
})
}
}
func TestMetricsMultipleNamespaces(t *testing.T) {
// Reset all metrics
JobsCreatedTotal.Reset()
JobsSucceededTotal.Reset()
JobsFailedTotal.Reset()
ActiveJobs.Reset()
namespaces := []string{"dev", "staging", "production"}
for _, ns := range namespaces {
RecordJobCreated(ns, "workflow", "group")
RecordJobSucceeded(ns, "workflow", "group")
RecordJobFailed(ns, "workflow", "group")
SetActiveJobs(ns, "workflow", 1)
}
for _, ns := range namespaces {
assert.Equal(t, float64(1), testutil.ToFloat64(JobsCreatedTotal.WithLabelValues(ns, "workflow", "group")))
assert.Equal(t, float64(1), testutil.ToFloat64(JobsSucceededTotal.WithLabelValues(ns, "workflow", "group")))
assert.Equal(t, float64(1), testutil.ToFloat64(JobsFailedTotal.WithLabelValues(ns, "workflow", "group")))
assert.Equal(t, float64(1), testutil.ToFloat64(ActiveJobs.WithLabelValues(ns, "workflow")))
}
}
func TestActiveJobsGaugeDecreases(t *testing.T) {
ActiveJobs.Reset()
// Simulate job lifecycle
SetActiveJobs("default", "workflow", 0)
assert.Equal(t, float64(0), testutil.ToFloat64(ActiveJobs.WithLabelValues("default", "workflow")))
SetActiveJobs("default", "workflow", 5)
assert.Equal(t, float64(5), testutil.ToFloat64(ActiveJobs.WithLabelValues("default", "workflow")))
SetActiveJobs("default", "workflow", 3)
assert.Equal(t, float64(3), testutil.ToFloat64(ActiveJobs.WithLabelValues("default", "workflow")))
SetActiveJobs("default", "workflow", 0)
assert.Equal(t, float64(0), testutil.ToFloat64(ActiveJobs.WithLabelValues("default", "workflow")))
}
+732
View File
@@ -0,0 +1,732 @@
package controllers
import (
"context"
"errors"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/stretchr/testify/suite"
kbatch "k8s.io/api/batch/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
// ReconcilerTestSuite contains all reconciler tests
type ReconcilerTestSuite struct {
suite.Suite
ctx context.Context
cancel context.CancelFunc
client *MockClient
reconciler *ManagedJobReconciler
}
func (s *ReconcilerTestSuite) SetupTest() {
s.ctx, s.cancel = context.WithTimeout(context.Background(), 30*time.Second)
s.client = NewMockClient()
s.reconciler = &ManagedJobReconciler{
Client: s.client,
Scheme: s.client.Scheme(),
Recorder: NewFakeRecorder(),
}
}
func (s *ReconcilerTestSuite) TearDownTest() {
s.cancel()
}
func TestReconcilerSuite(t *testing.T) {
suite.Run(t, new(ReconcilerTestSuite))
}
// ==================== GOOD SCENARIOS ====================
func (s *ReconcilerTestSuite) TestReconcile_Good_NewManagedJob_AddsFinalizer() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
s.client.AddManagedJob(mj)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
result, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.NoError(err)
s.True(result.RequeueAfter > 0, "should requeue after adding finalizer")
updated := s.client.GetManagedJobByKey("default/test-workflow")
s.True(controllerutil.ContainsFinalizer(updated, FinalizerName), "finalizer should be added")
}
func (s *ReconcilerTestSuite) TestReconcile_Good_SingleJobWorkflow_CreatesJob() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act - first reconcile generates dependency tree
_, err := s.reconciler.Reconcile(s.ctx, req)
s.NoError(err)
// Act - second reconcile runs jobs
_, err = s.reconciler.Reconcile(s.ctx, req)
s.NoError(err)
// Assert
s.GreaterOrEqual(s.client.CreateCalls, 1, "should have created at least one job")
}
func (s *ReconcilerTestSuite) TestReconcile_Good_CompletedJob_UpdatesStatus() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusRunning
mj.Spec.Groups[0].Status = ExecutionStatusRunning
s.client.AddManagedJob(mj)
// Add a completed K8s job
k8sJob := NewTestK8sJob("test-workflow-group1-job1", "default", "test-workflow", "group1", kbatch.JobStatus{
Succeeded: 1,
})
s.client.AddJob(k8sJob)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
_, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.NoError(err)
}
func (s *ReconcilerTestSuite) TestReconcile_Good_RunningWorkflow_RequeuesAfterDelay() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Status = ExecutionStatusRunning
mj.Spec.Groups[0].Status = ExecutionStatusRunning
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusRunning
s.client.AddManagedJob(mj)
// Add running K8s job
k8sJob := NewTestK8sJob("test-workflow-group1-job1", "default", "test-workflow", "group1", kbatch.JobStatus{
Active: 1,
})
s.client.AddJob(k8sJob)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
result, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.NoError(err)
s.Equal(RequeueDelay, result.RequeueAfter, "should requeue after delay for running workflow")
}
// ==================== NOT GOOD SCENARIOS ====================
func (s *ReconcilerTestSuite) TestReconcile_NotGood_ManagedJobNotFound_ReturnsNoError() {
// Arrange - no ManagedJob added to client
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "nonexistent", Namespace: "default"},
}
// Act
result, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.NoError(err, "should not return error for not found")
s.Zero(result.RequeueAfter, "should not requeue for not found")
}
func (s *ReconcilerTestSuite) TestReconcile_NotGood_FailedJob_UpdatesStatusToFailed() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusRunning
mj.Spec.Groups[0].Status = ExecutionStatusRunning
s.client.AddManagedJob(mj)
// Add a failed K8s job
k8sJob := NewTestK8sJob("test-workflow-group1-job1", "default", "test-workflow", "group1", kbatch.JobStatus{
Failed: 1,
})
s.client.AddJob(k8sJob)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
_, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.NoError(err)
}
func (s *ReconcilerTestSuite) TestReconcile_NotGood_DeletionInProgress_CleansUpJobs() {
// Arrange
now := metav1.Now()
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.DeletionTimestamp = &now
s.client.AddManagedJob(mj)
// Add child job to be deleted
k8sJob := NewTestK8sJob("test-workflow-group1-job1", "default", "test-workflow", "group1", kbatch.JobStatus{})
s.client.AddJob(k8sJob)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
_, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.NoError(err)
s.GreaterOrEqual(s.client.DeleteCalls, 1, "should have deleted child jobs")
}
// ==================== REALLY BAD SCENARIOS ====================
func (s *ReconcilerTestSuite) TestReconcile_ReallyBad_GetError_ReturnsError() {
// Arrange
s.client.GetError = ErrServerError
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
_, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.Error(err)
}
func (s *ReconcilerTestSuite) TestReconcile_ReallyBad_UpdateConflict_ReturnsError() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
s.client.AddManagedJob(mj)
s.client.UpdateError = ErrConflict
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
_, err := s.reconciler.Reconcile(s.ctx, req)
// Assert
s.Error(err, "should return error on update conflict")
}
func (s *ReconcilerTestSuite) TestReconcile_ReallyBad_ContextTimeout_ReturnsError() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
s.client.AddManagedJob(mj)
// Cancel context immediately
ctx, cancel := context.WithCancel(context.Background())
cancel()
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act
_, err := s.reconciler.Reconcile(ctx, req)
// Assert
s.Error(err, "should return error on context timeout")
}
func (s *ReconcilerTestSuite) TestReconcile_ReallyBad_ListJobsError_Continues() {
// Arrange
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
s.client.AddManagedJob(mj)
s.client.ListError = ErrServerError
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "test-workflow", Namespace: "default"},
}
// Act - should not panic, should handle gracefully
_, err := s.reconciler.Reconcile(s.ctx, req)
// Assert - list error is logged but doesn't fail reconciliation
s.NoError(err)
}
// ==================== MATRIX TEST: WORKFLOW SCENARIOS ====================
func TestReconcile_WorkflowScenarios(t *testing.T) {
tests := []struct {
name string
scenario TestScenario
setupMJ func() *jobsmanagerv1beta1.ManagedJob
setupJobs func() []*kbatch.Job
clientSetup func(*MockClient)
expectError bool
expectRequeue bool
expectRequeueAfter time.Duration
validateResult func(*testing.T, *MockClient, ctrl.Result)
}{
// GOOD SCENARIOS
{
name: "good_simple_workflow_starts",
scenario: ScenarioGood,
setupMJ: func() *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("init", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("setup", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
setupJobs: func() []*kbatch.Job { return nil },
expectError: false,
expectRequeue: false,
},
{
name: "good_multi_group_workflow",
scenario: ScenarioGood,
setupMJ: func() *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("init", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("setup", "busybox:latest"),
}),
NewTestGroup("main", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("process", "busybox:latest"),
}, NewTestDependency("init", ExecutionStatusPending)),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
setupJobs: func() []*kbatch.Job { return nil },
expectError: false,
expectRequeue: false,
},
{
name: "good_all_jobs_completed",
scenario: ScenarioGood,
setupMJ: func() *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("init", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("setup", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusSucceeded
mj.Spec.Groups[0].Status = ExecutionStatusSucceeded
return mj
},
setupJobs: func() []*kbatch.Job {
return []*kbatch.Job{
NewTestK8sJob("workflow-init-setup", "default", "workflow", "init", kbatch.JobStatus{Succeeded: 1}),
}
},
expectError: false,
expectRequeue: false,
},
// NOT GOOD SCENARIOS
{
name: "notgood_job_failed_workflow_continues",
scenario: ScenarioNotGood,
setupMJ: func() *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("init", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("setup", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
mj.Spec.Groups[0].Jobs[0].Status = ExecutionStatusRunning
return mj
},
setupJobs: func() []*kbatch.Job {
return []*kbatch.Job{
NewTestK8sJob("workflow-init-setup", "default", "workflow", "init", kbatch.JobStatus{Failed: 1}),
}
},
expectError: false,
expectRequeue: false,
},
{
name: "notgood_dependent_job_aborted",
scenario: ScenarioNotGood,
setupMJ: func() *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("init", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("setup", "busybox:latest"),
NewTestJobDef("verify", "busybox:latest", NewTestDependency("setup", ExecutionStatusFailed)),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
setupJobs: func() []*kbatch.Job {
return []*kbatch.Job{
NewTestK8sJob("workflow-init-setup", "default", "workflow", "init", kbatch.JobStatus{Failed: 1}),
}
},
expectError: false,
expectRequeue: false,
},
// REALLY BAD SCENARIOS
{
name: "reallybad_api_server_unavailable",
scenario: ScenarioReallyBad,
setupMJ: func() *jobsmanagerv1beta1.ManagedJob {
return NewTestManagedJob("workflow", "default", nil)
},
setupJobs: func() []*kbatch.Job { return nil },
clientSetup: func(c *MockClient) {
c.GetError = ErrNetworkFailure
},
expectError: true,
expectRequeue: false,
},
{
name: "reallybad_create_job_fails",
scenario: ScenarioReallyBad,
setupMJ: func() *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("init", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("setup", "busybox:latest"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
setupJobs: func() []*kbatch.Job { return nil },
clientSetup: func(c *MockClient) {
c.CreateError = ErrForbidden
},
expectError: false, // Job creation failure doesn't stop reconciliation
expectRequeue: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ctx := context.Background()
client := NewMockClient()
reconciler := &ManagedJobReconciler{
Client: client,
Scheme: client.Scheme(),
Recorder: NewFakeRecorder(),
}
// Setup client behavior
if tt.clientSetup != nil {
tt.clientSetup(client)
}
// Add ManagedJob
mj := tt.setupMJ()
if mj != nil {
client.AddManagedJob(mj)
}
// Add Jobs
for _, job := range tt.setupJobs() {
client.AddJob(job)
}
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "workflow", Namespace: "default"},
}
// Act
result, err := reconciler.Reconcile(ctx, req)
// Assert
if tt.expectError {
assert.Error(t, err, "expected error for scenario: %s", tt.scenario)
} else {
assert.NoError(t, err, "expected no error for scenario: %s", tt.scenario)
}
if tt.expectRequeue {
assert.True(t, result.RequeueAfter > 0, "expected requeue")
}
if tt.expectRequeueAfter > 0 {
assert.Equal(t, tt.expectRequeueAfter, result.RequeueAfter)
}
if tt.validateResult != nil {
tt.validateResult(t, client, result)
}
})
}
}
// ==================== EDGE CASES ====================
func TestReconcile_EdgeCases(t *testing.T) {
tests := []struct {
name string
description string
setup func(*MockClient) *jobsmanagerv1beta1.ManagedJob
validate func(*testing.T, *MockClient, ctrl.Result, error)
}{
{
name: "empty_groups",
description: "ManagedJob with no groups should complete immediately",
setup: func(c *MockClient) *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("empty", "default", []*jobsmanagerv1beta1.ManagedJobGroup{})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
validate: func(t *testing.T, c *MockClient, r ctrl.Result, err error) {
assert.NoError(t, err)
},
},
{
name: "group_with_no_jobs",
description: "Group with no jobs should be marked as succeeded",
setup: func(c *MockClient) *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("no-jobs", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("empty-group", []*jobsmanagerv1beta1.ManagedJobDefinition{}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
validate: func(t *testing.T, c *MockClient, r ctrl.Result, err error) {
assert.NoError(t, err)
},
},
{
name: "circular_dependency_protection",
description: "Jobs with circular dependencies should not cause infinite loop",
setup: func(c *MockClient) *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("circular", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job-a", "busybox", NewTestDependency("job-b", ExecutionStatusPending)),
NewTestJobDef("job-b", "busybox", NewTestDependency("job-a", ExecutionStatusPending)),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
validate: func(t *testing.T, c *MockClient, r ctrl.Result, err error) {
assert.NoError(t, err, "should handle circular deps gracefully")
},
},
{
name: "rapid_status_changes",
description: "Multiple rapid reconciliations should be idempotent",
setup: func(c *MockClient) *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("rapid", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
validate: func(t *testing.T, c *MockClient, r ctrl.Result, err error) {
assert.NoError(t, err)
},
},
{
name: "very_long_job_name",
description: "Long names should be handled (K8s has 63 char limit)",
setup: func(c *MockClient) *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("very-long-group-name-that-exceeds-normal", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("extremely-long-job-name-here", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
validate: func(t *testing.T, c *MockClient, r ctrl.Result, err error) {
assert.NoError(t, err)
},
},
{
name: "special_characters_in_name",
description: "Names with special characters",
setup: func(c *MockClient) *jobsmanagerv1beta1.ManagedJob {
mj := NewTestManagedJob("test-workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group-1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job-1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
return mj
},
validate: func(t *testing.T, c *MockClient, r ctrl.Result, err error) {
assert.NoError(t, err)
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
client := NewMockClient()
reconciler := &ManagedJobReconciler{
Client: client,
Scheme: client.Scheme(),
Recorder: NewFakeRecorder(),
}
mj := tt.setup(client)
client.AddManagedJob(mj)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: mj.Name, Namespace: mj.Namespace},
}
result, err := reconciler.Reconcile(context.Background(), req)
tt.validate(t, client, result, err)
})
}
}
// ==================== KUBERNETES VOLATILITY TESTS ====================
func TestReconcile_KubernetesVolatility(t *testing.T) {
tests := []struct {
name string
description string
clientSetup func(*MockClient)
expectError bool
}{
{
name: "intermittent_api_failure",
description: "API fails on first call but succeeds on retry",
clientSetup: func(c *MockClient) {
c.FailOnNthCall["get"] = 1
},
expectError: true,
},
{
name: "list_timeout",
description: "List operation times out",
clientSetup: func(c *MockClient) {
c.ListError = ErrTimeout
},
expectError: false, // List error is handled gracefully
},
{
name: "create_conflict",
description: "Job already exists when creating",
clientSetup: func(c *MockClient) {
c.CreateError = errors.New("already exists")
},
expectError: false, // Already exists is handled
},
{
name: "update_resource_version_conflict",
description: "Optimistic locking conflict on update",
clientSetup: func(c *MockClient) {
c.SimulateConflictOnUpdate = true
},
expectError: false, // First update should succeed
},
{
name: "forbidden_permission",
description: "Insufficient RBAC permissions",
clientSetup: func(c *MockClient) {
c.CreateError = ErrForbidden
},
expectError: false, // Logged but doesn't fail reconciliation
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
client := NewMockClient()
tt.clientSetup(client)
reconciler := &ManagedJobReconciler{
Client: client,
Scheme: client.Scheme(),
Recorder: NewFakeRecorder(),
}
mj := NewTestManagedJob("workflow", "default", []*jobsmanagerv1beta1.ManagedJobGroup{
NewTestGroup("group1", []*jobsmanagerv1beta1.ManagedJobDefinition{
NewTestJobDef("job1", "busybox"),
}),
})
controllerutil.AddFinalizer(mj, FinalizerName)
client.AddManagedJob(mj)
req := reconcile.Request{
NamespacedName: types.NamespacedName{Name: "workflow", Namespace: "default"},
}
_, err := reconciler.Reconcile(context.Background(), req)
if tt.expectError {
require.Error(t, err, tt.description)
} else {
require.NoError(t, err, tt.description)
}
})
}
}
+2
View File
@@ -1,3 +1,5 @@
//go:build envtest
/*
Copyright 2023.
+505
View File
@@ -0,0 +1,505 @@
package controllers
import (
"context"
"errors"
"sync"
kbatch "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/tools/record"
"sigs.k8s.io/controller-runtime/pkg/client"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
// Common test errors for simulating Kubernetes API failures
var (
ErrNotFound = errors.New("not found")
ErrConflict = errors.New("conflict: object has been modified")
ErrTimeout = errors.New("context deadline exceeded")
ErrServerError = errors.New("internal server error")
ErrForbidden = errors.New("forbidden: insufficient permissions")
ErrNetworkFailure = errors.New("network is unreachable")
)
// TestScenario defines the type of test scenario
type TestScenario string
const (
ScenarioGood TestScenario = "good"
ScenarioNotGood TestScenario = "not_good"
ScenarioReallyBad TestScenario = "really_bad"
)
// MockClient implements a mock Kubernetes client for testing
type MockClient struct {
mu sync.RWMutex
// Storage for mock objects
managedJobs map[string]*jobsmanagerv1beta1.ManagedJob
jobs map[string]*kbatch.Job
// Error injection for different operations
GetError error
ListError error
CreateError error
UpdateError error
DeleteError error
// Call counters for verification
GetCalls int
ListCalls int
CreateCalls int
UpdateCalls int
DeleteCalls int
// Behavior modifiers
SimulateConflictOnUpdate bool
SimulateSlowResponse bool
FailOnNthCall map[string]int // operation -> fail on nth call
}
// NewMockClient creates a new mock client
func NewMockClient() *MockClient {
return &MockClient{
managedJobs: make(map[string]*jobsmanagerv1beta1.ManagedJob),
jobs: make(map[string]*kbatch.Job),
FailOnNthCall: make(map[string]int),
}
}
// Get implements client.Client
func (m *MockClient) Get(ctx context.Context, key types.NamespacedName, obj client.Object, opts ...client.GetOption) error {
m.mu.Lock()
defer m.mu.Unlock()
m.GetCalls++
// Check for context cancellation (simulates timeout)
select {
case <-ctx.Done():
return ErrTimeout
default:
}
// Check for injected error
if m.GetError != nil {
return m.GetError
}
// Check for nth call failure
if n, ok := m.FailOnNthCall["get"]; ok && m.GetCalls == n {
return ErrServerError
}
keyStr := key.String()
switch v := obj.(type) {
case *jobsmanagerv1beta1.ManagedJob:
if mj, ok := m.managedJobs[keyStr]; ok {
*v = *mj.DeepCopy()
return nil
}
return apierrors.NewNotFound(schema.GroupResource{Group: "jobsmanager.raczylo.com", Resource: "managedjobs"}, key.Name)
case *kbatch.Job:
if j, ok := m.jobs[keyStr]; ok {
*v = *j.DeepCopy()
return nil
}
return apierrors.NewNotFound(schema.GroupResource{Group: "batch", Resource: "jobs"}, key.Name)
}
return apierrors.NewNotFound(schema.GroupResource{Resource: "unknown"}, key.Name)
}
// List implements client.Client
func (m *MockClient) List(ctx context.Context, list client.ObjectList, opts ...client.ListOption) error {
m.mu.Lock()
defer m.mu.Unlock()
m.ListCalls++
select {
case <-ctx.Done():
return ErrTimeout
default:
}
if m.ListError != nil {
return m.ListError
}
if n, ok := m.FailOnNthCall["list"]; ok && m.ListCalls == n {
return ErrServerError
}
// Extract namespace from options
listOpts := &client.ListOptions{}
for _, opt := range opts {
opt.ApplyToList(listOpts)
}
switch v := list.(type) {
case *kbatch.JobList:
items := []kbatch.Job{}
for _, job := range m.jobs {
if listOpts.Namespace == "" || job.Namespace == listOpts.Namespace {
// Check label selector if present
if listOpts.LabelSelector != nil {
if !listOpts.LabelSelector.Matches(labelSetFromMap(job.Labels)) {
continue
}
}
items = append(items, *job.DeepCopy())
}
}
v.Items = items
return nil
case *jobsmanagerv1beta1.ManagedJobList:
items := []jobsmanagerv1beta1.ManagedJob{}
for _, mj := range m.managedJobs {
if listOpts.Namespace == "" || mj.Namespace == listOpts.Namespace {
items = append(items, *mj.DeepCopy())
}
}
v.Items = items
return nil
}
return nil
}
// Create implements client.Client
func (m *MockClient) Create(ctx context.Context, obj client.Object, opts ...client.CreateOption) error {
m.mu.Lock()
defer m.mu.Unlock()
m.CreateCalls++
select {
case <-ctx.Done():
return ErrTimeout
default:
}
if m.CreateError != nil {
return m.CreateError
}
if n, ok := m.FailOnNthCall["create"]; ok && m.CreateCalls == n {
return ErrServerError
}
key := types.NamespacedName{
Namespace: obj.GetNamespace(),
Name: obj.GetName(),
}.String()
switch v := obj.(type) {
case *jobsmanagerv1beta1.ManagedJob:
if _, exists := m.managedJobs[key]; exists {
return errors.New("already exists")
}
m.managedJobs[key] = v.DeepCopy()
case *kbatch.Job:
if _, exists := m.jobs[key]; exists {
return errors.New("already exists")
}
m.jobs[key] = v.DeepCopy()
}
return nil
}
// Update implements client.Client
func (m *MockClient) Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error {
m.mu.Lock()
defer m.mu.Unlock()
m.UpdateCalls++
select {
case <-ctx.Done():
return ErrTimeout
default:
}
if m.UpdateError != nil {
return m.UpdateError
}
if m.SimulateConflictOnUpdate && m.UpdateCalls > 1 {
return ErrConflict
}
if n, ok := m.FailOnNthCall["update"]; ok && m.UpdateCalls == n {
return ErrServerError
}
key := types.NamespacedName{
Namespace: obj.GetNamespace(),
Name: obj.GetName(),
}.String()
switch v := obj.(type) {
case *jobsmanagerv1beta1.ManagedJob:
if _, exists := m.managedJobs[key]; !exists {
return ErrNotFound
}
m.managedJobs[key] = v.DeepCopy()
case *kbatch.Job:
if _, exists := m.jobs[key]; !exists {
return ErrNotFound
}
m.jobs[key] = v.DeepCopy()
}
return nil
}
// Delete implements client.Client
func (m *MockClient) Delete(ctx context.Context, obj client.Object, opts ...client.DeleteOption) error {
m.mu.Lock()
defer m.mu.Unlock()
m.DeleteCalls++
select {
case <-ctx.Done():
return ErrTimeout
default:
}
if m.DeleteError != nil {
return m.DeleteError
}
if n, ok := m.FailOnNthCall["delete"]; ok && m.DeleteCalls == n {
return ErrServerError
}
key := types.NamespacedName{
Namespace: obj.GetNamespace(),
Name: obj.GetName(),
}.String()
switch obj.(type) {
case *jobsmanagerv1beta1.ManagedJob:
delete(m.managedJobs, key)
case *kbatch.Job:
delete(m.jobs, key)
}
return nil
}
// Patch implements client.Client
func (m *MockClient) Patch(ctx context.Context, obj client.Object, patch client.Patch, opts ...client.PatchOption) error {
return m.Update(ctx, obj)
}
// DeleteAllOf implements client.Client
func (m *MockClient) DeleteAllOf(ctx context.Context, obj client.Object, opts ...client.DeleteAllOfOption) error {
return nil
}
// Status implements client.Client
func (m *MockClient) Status() client.SubResourceWriter {
return &MockStatusWriter{client: m}
}
// SubResource implements client.Client
func (m *MockClient) SubResource(subResource string) client.SubResourceClient {
return nil
}
// Apply implements client.Client
func (m *MockClient) Apply(ctx context.Context, obj runtime.ApplyConfiguration, opts ...client.ApplyOption) error {
return nil
}
// Scheme implements client.Client
func (m *MockClient) Scheme() *runtime.Scheme {
scheme := runtime.NewScheme()
_ = jobsmanagerv1beta1.AddToScheme(scheme)
_ = kbatch.AddToScheme(scheme)
_ = corev1.AddToScheme(scheme)
return scheme
}
// RESTMapper implements client.Client
func (m *MockClient) RESTMapper() meta.RESTMapper {
return nil
}
// GroupVersionKindFor implements client.Client
func (m *MockClient) GroupVersionKindFor(obj runtime.Object) (schema.GroupVersionKind, error) {
return schema.GroupVersionKind{}, nil
}
// IsObjectNamespaced implements client.Client
func (m *MockClient) IsObjectNamespaced(obj runtime.Object) (bool, error) {
return true, nil
}
// MockStatusWriter implements client.StatusWriter
type MockStatusWriter struct {
client *MockClient
}
func (m *MockStatusWriter) Create(ctx context.Context, obj client.Object, subResource client.Object, opts ...client.SubResourceCreateOption) error {
return nil
}
func (m *MockStatusWriter) Update(ctx context.Context, obj client.Object, opts ...client.SubResourceUpdateOption) error {
return m.client.Update(ctx, obj)
}
func (m *MockStatusWriter) Patch(ctx context.Context, obj client.Object, patch client.Patch, opts ...client.SubResourcePatchOption) error {
return m.client.Patch(ctx, obj, patch)
}
func (m *MockStatusWriter) Apply(ctx context.Context, obj runtime.ApplyConfiguration, opts ...client.SubResourceApplyOption) error {
return nil
}
// Helper to add test data
func (m *MockClient) AddManagedJob(mj *jobsmanagerv1beta1.ManagedJob) {
m.mu.Lock()
defer m.mu.Unlock()
key := types.NamespacedName{
Namespace: mj.Namespace,
Name: mj.Name,
}.String()
m.managedJobs[key] = mj.DeepCopy()
}
func (m *MockClient) AddJob(job *kbatch.Job) {
m.mu.Lock()
defer m.mu.Unlock()
key := types.NamespacedName{
Namespace: job.Namespace,
Name: job.Name,
}.String()
m.jobs[key] = job.DeepCopy()
}
func (m *MockClient) GetManagedJobByKey(key string) *jobsmanagerv1beta1.ManagedJob {
m.mu.RLock()
defer m.mu.RUnlock()
return m.managedJobs[key]
}
func (m *MockClient) GetJobByKey(key string) *kbatch.Job {
m.mu.RLock()
defer m.mu.RUnlock()
return m.jobs[key]
}
// labelSetFromMap creates a label set from a map for selector matching
type labelSet map[string]string
func labelSetFromMap(m map[string]string) labelSet {
return labelSet(m)
}
func (ls labelSet) Has(key string) bool {
_, ok := ls[key]
return ok
}
func (ls labelSet) Get(key string) string {
return ls[key]
}
// Lookup implements labels.Labels interface
func (ls labelSet) Lookup(key string) (string, bool) {
v, ok := ls[key]
return v, ok
}
// Test fixtures and factory functions
// NewTestManagedJob creates a ManagedJob for testing
func NewTestManagedJob(name, namespace string, groups []*jobsmanagerv1beta1.ManagedJobGroup) *jobsmanagerv1beta1.ManagedJob {
return &jobsmanagerv1beta1.ManagedJob{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: namespace,
UID: types.UID("test-uid-" + name),
},
Spec: jobsmanagerv1beta1.ManagedJobSpec{
Groups: groups,
},
Status: ExecutionStatusPending,
}
}
// NewTestGroup creates a ManagedJobGroup for testing
func NewTestGroup(name string, jobs []*jobsmanagerv1beta1.ManagedJobDefinition, deps ...*jobsmanagerv1beta1.ManagedJobDependencies) *jobsmanagerv1beta1.ManagedJobGroup {
return &jobsmanagerv1beta1.ManagedJobGroup{
Name: name,
Jobs: jobs,
Dependencies: deps,
Status: ExecutionStatusPending,
}
}
// NewTestJobDef creates a ManagedJobDefinition for testing
func NewTestJobDef(name, image string, deps ...*jobsmanagerv1beta1.ManagedJobDependencies) *jobsmanagerv1beta1.ManagedJobDefinition {
return &jobsmanagerv1beta1.ManagedJobDefinition{
Name: name,
Image: image,
Args: []string{"echo", "test"},
Dependencies: deps,
Status: ExecutionStatusPending,
}
}
// NewTestDependency creates a ManagedJobDependencies for testing
func NewTestDependency(name, status string) *jobsmanagerv1beta1.ManagedJobDependencies {
return &jobsmanagerv1beta1.ManagedJobDependencies{
Name: name,
Status: status,
}
}
// NewTestK8sJob creates a Kubernetes Job for testing
func NewTestK8sJob(name, namespace, workflowName, groupName string, status kbatch.JobStatus) *kbatch.Job {
return &kbatch.Job{
ObjectMeta: metav1.ObjectMeta{
Name: name,
Namespace: namespace,
Labels: map[string]string{
LabelWorkflowName: workflowName,
LabelGroupName: groupName,
LabelJobName: name,
},
},
Spec: kbatch.JobSpec{
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{Name: "test", Image: "busybox"},
},
RestartPolicy: corev1.RestartPolicyNever,
},
},
},
Status: status,
}
}
// NewFakeRecorder creates a fake event recorder for testing
func NewFakeRecorder() record.EventRecorder {
return record.NewFakeRecorder(100)
}
+108 -69
View File
@@ -1,82 +1,121 @@
module raczylo.com/jobs-manager-operator
go 1.19
go 1.26.0
require (
github.com/kr/pretty v0.3.1
github.com/lukaszraczylo/pandati v0.0.18
github.com/onsi/ginkgo/v2 v2.6.0
github.com/onsi/gomega v1.24.1
k8s.io/api v0.26.1
k8s.io/apimachinery v0.26.1
k8s.io/client-go v0.26.1
sigs.k8s.io/controller-runtime v0.14.4
github.com/fatih/color v1.19.0
github.com/go-logr/logr v1.4.4
github.com/lukaszraczylo/pandati v0.0.29
github.com/onsi/ginkgo/v2 v2.33.0
github.com/onsi/gomega v1.44.0
github.com/prometheus/client_golang v1.24.1
github.com/spf13/cobra v1.10.2
github.com/stretchr/testify v1.12.1
k8s.io/api v0.37.1
k8s.io/apimachinery v0.37.1
k8s.io/client-go v0.37.1
sigs.k8s.io/controller-runtime v0.25.2
)
require (
cel.dev/expr v0.25.3 // indirect
dario.cat/mergo v1.0.2 // indirect
github.com/Masterminds/semver/v3 v3.5.0 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/emicklei/go-restful/v3 v3.10.1 // indirect
github.com/evanphx/json-patch v5.6.0+incompatible // indirect
github.com/evanphx/json-patch/v5 v5.6.0 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-logr/zapr v1.2.3 // indirect
github.com/go-openapi/jsonpointer v0.19.6 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.22.3 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/gnostic v0.6.9 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/imdario/mergo v0.3.13 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/emicklei/go-restful/v3 v3.13.0 // indirect
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/fxamacker/cbor/v2 v2.9.6 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
github.com/go-openapi/jsonpointer v1.0.2 // indirect
github.com/go-openapi/jsonreference v1.0.3 // indirect
github.com/go-openapi/swag v0.29.2 // indirect
github.com/go-openapi/swag/cmdutils v0.29.2 // indirect
github.com/go-openapi/swag/conv v0.29.2 // indirect
github.com/go-openapi/swag/fileutils v0.29.2 // indirect
github.com/go-openapi/swag/jsonutils v0.29.2 // indirect
github.com/go-openapi/swag/loading v0.29.2 // indirect
github.com/go-openapi/swag/mangling v0.29.2 // indirect
github.com/go-openapi/swag/netutils v0.29.2 // indirect
github.com/go-openapi/swag/pools v0.29.2 // indirect
github.com/go-openapi/swag/stringutils v0.29.2 // indirect
github.com/go-openapi/swag/typeutils v0.29.2 // indirect
github.com/go-openapi/swag/yamlutils v0.29.2 // indirect
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
github.com/google/cel-go v0.31.0 // indirect
github.com/google/gnostic-models v0.7.1 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/pprof v0.0.0-20260402051712-545e8a4df936 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.31.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.17 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/mattn/go-colorable v0.1.16 // indirect
github.com/mattn/go-isatty v0.0.24 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.14.0 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.40.0 // indirect
github.com/prometheus/procfs v0.9.0 // indirect
github.com/rogpeppe/go-internal v1.9.0 // indirect
github.com/rs/zerolog v1.29.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/tidwall/gjson v1.14.4 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/wI2L/jsondiff v0.3.0 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.9.0 // indirect
go.uber.org/zap v1.24.0 // indirect
golang.org/x/net v0.7.0 // indirect
golang.org/x/oauth2 v0.5.0 // indirect
golang.org/x/sys v0.5.0 // indirect
golang.org/x/term v0.5.0 // indirect
golang.org/x/text v0.7.0 // indirect
golang.org/x/time v0.3.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/protobuf v1.28.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.3 // indirect
github.com/prometheus/common v0.72.0 // indirect
github.com/prometheus/procfs v0.22.0 // indirect
github.com/rs/zerolog v1.35.1 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/tidwall/gjson v1.20.0 // indirect
github.com/tidwall/match v1.2.0 // indirect
github.com/tidwall/pretty v1.2.2 // indirect
github.com/tidwall/sjson v1.2.5 // indirect
github.com/wI2L/jsondiff v0.7.1 // indirect
github.com/x448/float16 v0.8.4 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.72.0 // indirect
go.opentelemetry.io/otel v1.47.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.47.0 // indirect
go.opentelemetry.io/otel/log v1.47.0 // indirect
go.opentelemetry.io/otel/metric v1.47.0 // indirect
go.opentelemetry.io/otel/sdk v1.47.0 // indirect
go.opentelemetry.io/otel/trace v1.47.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.28.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/exp v0.0.0-20260908205506-85c1c2202aba // indirect
golang.org/x/mod v0.41.0 // indirect
golang.org/x/net v0.59.0 // indirect
golang.org/x/oauth2 v0.37.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/term v0.46.0 // indirect
golang.org/x/text v0.42.0 // indirect
golang.org/x/time v0.16.0 // indirect
golang.org/x/tools v0.51.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.5.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc // indirect
google.golang.org/grpc v1.84.0 // indirect
google.golang.org/protobuf v1.36.12 // indirect
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/apiextensions-apiserver v0.26.1 // indirect
k8s.io/component-base v0.26.1 // indirect
k8s.io/klog/v2 v2.90.0 // indirect
k8s.io/kube-openapi v0.0.0-20230217203603-ff9a8e8fa21d // indirect
k8s.io/utils v0.0.0-20230220204549-a5ecb0141aa5 // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.3 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
k8s.io/apiextensions-apiserver v0.37.1 // indirect
k8s.io/apiserver v0.37.1 // indirect
k8s.io/component-base v0.37.1 // indirect
k8s.io/klog/v2 v2.140.0 // indirect
k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad // indirect
k8s.io/streaming v0.37.1 // indirect
k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3 // indirect
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.37.0 // indirect
sigs.k8s.io/json v0.0.0-20260909141634-11ed52e25bc5 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
sigs.k8s.io/structured-merge-diff/v6 v6.4.2 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
)
+254 -713
View File
File diff suppressed because it is too large Load Diff
+34 -5
View File
@@ -30,6 +30,8 @@ import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/healthz"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
"sigs.k8s.io/controller-runtime/pkg/metrics/filters"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
"raczylo.com/jobs-manager-operator/controllers"
@@ -52,26 +54,53 @@ func main() {
var metricsAddr string
var enableLeaderElection bool
var probeAddr string
flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
var leaderElectionID string
var devMode bool
var secureMetrics bool
flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. Use :8443 for HTTPS with authentication, or 0 to disable.")
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
"Enable leader election for controller manager. "+
"Enabling this will ensure there is only one active controller manager.")
flag.StringVar(&leaderElectionID, "leader-election-id", "jobsmanager.raczylo.com",
"The name of the leader election resource.")
flag.BoolVar(&devMode, "dev-mode", false,
"Enable development mode with verbose logging (console format).")
flag.BoolVar(&secureMetrics, "metrics-secure", true,
"If set, the metrics endpoint is served securely via HTTPS with authentication.")
opts := zap.Options{
Development: true,
Development: devMode,
}
opts.BindFlags(flag.CommandLine)
flag.Parse()
// Allow environment variable to override dev mode
if os.Getenv("LOG_LEVEL") == "debug" {
devMode = true
opts.Development = true
}
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
// Configure metrics options
metricsOpts := metricsserver.Options{
BindAddress: metricsAddr,
SecureServing: secureMetrics,
}
// If secure metrics are enabled, use authentication and authorization filters
if secureMetrics {
metricsOpts.FilterProvider = filters.WithAuthenticationAndAuthorization
}
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
Metrics: metricsOpts,
HealthProbeBindAddress: probeAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "b86e0f00.raczylo.com",
LeaderElectionID: leaderElectionID,
// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
// when the Manager ends. This requires the binary to immediately end when the
// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
+135
View File
@@ -0,0 +1,135 @@
package visualization
import (
"context"
"fmt"
"k8s.io/apimachinery/pkg/runtime"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/config"
jobsmanagerv1beta1 "raczylo.com/jobs-manager-operator/api/v1beta1"
)
// Client wraps the Kubernetes client for ManagedJob operations
type Client struct {
client client.Client
}
// NewClient creates a new Client for ManagedJob operations
func NewClient() (*Client, error) {
cfg, err := config.GetConfig()
if err != nil {
return nil, fmt.Errorf("failed to get kubeconfig: %w", err)
}
scheme := runtime.NewScheme()
if err = clientgoscheme.AddToScheme(scheme); err != nil {
return nil, fmt.Errorf("failed to add client-go scheme: %w", err)
}
if err = jobsmanagerv1beta1.AddToScheme(scheme); err != nil {
return nil, fmt.Errorf("failed to add jobsmanager scheme: %w", err)
}
cl, err := client.New(cfg, client.Options{Scheme: scheme})
if err != nil {
return nil, fmt.Errorf("failed to create client: %w", err)
}
return &Client{client: cl}, nil
}
// GetManagedJob retrieves a ManagedJob by name and namespace
func (c *Client) GetManagedJob(ctx context.Context, name, namespace string) (*jobsmanagerv1beta1.ManagedJob, error) {
mj := &jobsmanagerv1beta1.ManagedJob{}
err := c.client.Get(ctx, client.ObjectKey{Name: name, Namespace: namespace}, mj)
if err != nil {
return nil, fmt.Errorf("failed to get ManagedJob %s/%s: %w", namespace, name, err)
}
return mj, nil
}
// ListManagedJobs lists all ManagedJobs in a namespace
func (c *Client) ListManagedJobs(ctx context.Context, namespace string) (*jobsmanagerv1beta1.ManagedJobList, error) {
mjList := &jobsmanagerv1beta1.ManagedJobList{}
opts := []client.ListOption{}
if namespace != "" {
opts = append(opts, client.InNamespace(namespace))
}
err := c.client.List(ctx, mjList, opts...)
if err != nil {
return nil, fmt.Errorf("failed to list ManagedJobs: %w", err)
}
return mjList, nil
}
// BuildTree builds a StatusTree from a ManagedJob
func BuildTree(mj *jobsmanagerv1beta1.ManagedJob) *StatusTree {
root := NewStatusTreeWithStatus(mj.Name, mj.Status)
for _, group := range mj.Spec.Groups {
groupNode := root.AddWithStatus(group.Name, group.Status)
// Add group dependencies
for _, dep := range group.Dependencies {
groupNode.Add(RenderDependency(dep.Name, true))
}
// Add jobs
for _, job := range group.Jobs {
jobNode := groupNode.AddWithStatus(job.Name, job.Status)
// Add job dependencies
for _, dep := range job.Dependencies {
jobNode.Add(RenderDependency(dep.Name, false))
}
}
}
return root
}
// GetStatusSummary returns a summary of job statuses
type StatusSummary struct {
Name string
Namespace string
Status string
Groups int
Jobs int
Pending int
Running int
Succeeded int
Failed int
Aborted int
}
// GetStatusSummary builds a summary of the ManagedJob status
func GetStatusSummary(mj *jobsmanagerv1beta1.ManagedJob) StatusSummary {
summary := StatusSummary{
Name: mj.Name,
Namespace: mj.Namespace,
Status: mj.Status,
Groups: len(mj.Spec.Groups),
}
for _, group := range mj.Spec.Groups {
for _, job := range group.Jobs {
summary.Jobs++
switch job.Status {
case StatusPending:
summary.Pending++
case StatusRunning:
summary.Running++
case StatusSucceeded:
summary.Succeeded++
case StatusFailed:
summary.Failed++
case StatusAborted:
summary.Aborted++
}
}
}
return summary
}
+150
View File
@@ -0,0 +1,150 @@
package visualization
import (
"strings"
"github.com/fatih/color"
)
// Box-drawing characters for tree rendering
const (
newLine = "\n"
emptySpace = " "
middleItem = "\u251c\u2500\u2500 " // ├──
continueItem = "\u2502 " // │
lastItem = "\u2514\u2500\u2500 " // └──
)
// Status constants
const (
StatusPending = "pending"
StatusRunning = "running"
StatusSucceeded = "succeeded"
StatusFailed = "failed"
StatusAborted = "aborted"
StatusUnknown = "unknown"
)
// Renderer handles tree rendering with optional color support
type Renderer struct {
useColor bool
green *color.Color
yellow *color.Color
red *color.Color
gray *color.Color
magenta *color.Color
cyan *color.Color
}
// NewRenderer creates a new Renderer
func NewRenderer(useColor bool) *Renderer {
return &Renderer{
useColor: useColor,
green: color.New(color.FgGreen),
yellow: color.New(color.FgYellow),
red: color.New(color.FgRed),
gray: color.New(color.FgHiBlack),
magenta: color.New(color.FgMagenta),
cyan: color.New(color.FgCyan),
}
}
// Render renders a StatusTree to a string
func (r *Renderer) Render(t *StatusTree) string {
var sb strings.Builder
r.renderNode(&sb, t, []bool{})
return sb.String()
}
// renderNode renders a single node and its children
func (r *Renderer) renderNode(sb *strings.Builder, t *StatusTree, spaces []bool) {
// Render current node
r.renderText(sb, t.Text(), t.Status(), spaces, true)
// Render children
items := t.Items()
for i, child := range items {
isLast := i == len(items)-1
r.renderChild(sb, child, spaces, isLast)
}
}
// renderChild renders a child node with proper indentation
func (r *Renderer) renderChild(sb *strings.Builder, t *StatusTree, spaces []bool, isLast bool) {
// Add prefix based on whether this is the last item
for _, space := range spaces {
if space {
sb.WriteString(emptySpace)
} else {
sb.WriteString(continueItem)
}
}
if isLast {
sb.WriteString(lastItem)
} else {
sb.WriteString(middleItem)
}
// Render the text with status
r.renderTextInline(sb, t.Text(), t.Status())
sb.WriteString(newLine)
// Render children with updated spaces
newSpaces := append(spaces, isLast)
items := t.Items()
for i, child := range items {
childIsLast := i == len(items)-1
r.renderChild(sb, child, newSpaces, childIsLast)
}
}
// renderText renders the root node text
func (r *Renderer) renderText(sb *strings.Builder, text, status string, spaces []bool, isRoot bool) {
if isRoot {
r.renderTextInline(sb, text, status)
sb.WriteString(newLine)
}
}
// renderTextInline renders text with status inline
func (r *Renderer) renderTextInline(sb *strings.Builder, text, status string) {
sb.WriteString(text)
if status != "" {
sb.WriteString(" ")
r.renderStatus(sb, status)
}
}
// renderStatus renders the status with appropriate color
func (r *Renderer) renderStatus(sb *strings.Builder, status string) {
statusText := "[" + status + "]"
if !r.useColor {
sb.WriteString(statusText)
return
}
switch status {
case StatusSucceeded:
sb.WriteString(r.green.Sprint(statusText))
case StatusRunning:
sb.WriteString(r.yellow.Sprint(statusText))
case StatusFailed:
sb.WriteString(r.red.Sprint(statusText))
case StatusPending:
sb.WriteString(r.gray.Sprint(statusText))
case StatusAborted:
sb.WriteString(r.magenta.Sprint(statusText))
default:
sb.WriteString(r.cyan.Sprint(statusText))
}
}
// RenderDependency formats a dependency reference
func RenderDependency(name string, isGroup bool) string {
if isGroup {
return "depends on group: " + name
}
return "depends on: " + name
}
+421
View File
@@ -0,0 +1,421 @@
package visualization
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
)
type RendererTestSuite struct {
suite.Suite
}
func TestRendererSuite(t *testing.T) {
suite.Run(t, new(RendererTestSuite))
}
func (s *RendererTestSuite) TestNewRenderer() {
r := NewRenderer(true)
s.NotNil(r)
s.True(r.useColor)
r2 := NewRenderer(false)
s.NotNil(r2)
s.False(r2.useColor)
}
func (s *RendererTestSuite) TestRender_SimpleTree() {
r := NewRenderer(false)
tree := NewStatusTree("root")
output := r.Render(tree)
s.Contains(output, "root")
}
func (s *RendererTestSuite) TestRender_WithStatus() {
r := NewRenderer(false)
tree := NewStatusTreeWithStatus("workflow", StatusRunning)
output := r.Render(tree)
s.Contains(output, "workflow")
s.Contains(output, "[running]")
}
func (s *RendererTestSuite) TestRender_WithChildren() {
r := NewRenderer(false)
tree := NewStatusTree("root")
tree.Add("child1")
tree.Add("child2")
output := r.Render(tree)
s.Contains(output, "root")
s.Contains(output, "child1")
s.Contains(output, "child2")
// Check for tree characters
s.Contains(output, "├──")
s.Contains(output, "└──")
}
func (s *RendererTestSuite) TestRender_NestedChildren() {
r := NewRenderer(false)
tree := NewStatusTree("workflow")
group := tree.Add("group1")
group.Add("job1")
group.Add("job2")
output := r.Render(tree)
s.Contains(output, "workflow")
s.Contains(output, "group1")
s.Contains(output, "job1")
s.Contains(output, "job2")
}
func (s *RendererTestSuite) TestRender_AllStatusColors() {
r := NewRenderer(false)
tree := NewStatusTree("workflow")
tree.AddWithStatus("pending-job", StatusPending)
tree.AddWithStatus("running-job", StatusRunning)
tree.AddWithStatus("succeeded-job", StatusSucceeded)
tree.AddWithStatus("failed-job", StatusFailed)
tree.AddWithStatus("aborted-job", StatusAborted)
tree.AddWithStatus("unknown-job", StatusUnknown)
output := r.Render(tree)
s.Contains(output, "[pending]")
s.Contains(output, "[running]")
s.Contains(output, "[succeeded]")
s.Contains(output, "[failed]")
s.Contains(output, "[aborted]")
s.Contains(output, "[unknown]")
}
func (s *RendererTestSuite) TestRenderDependency_Job() {
result := RenderDependency("init-job", false)
s.Equal("depends on: init-job", result)
}
func (s *RendererTestSuite) TestRenderDependency_Group() {
result := RenderDependency("init-group", true)
s.Equal("depends on group: init-group", result)
}
// ==================== MATRIX TEST: RENDER SCENARIOS ====================
func TestRenderer_StatusRendering(t *testing.T) {
tests := []struct {
name string
status string
expected string
}{
{
name: "pending_status",
status: StatusPending,
expected: "[pending]",
},
{
name: "running_status",
status: StatusRunning,
expected: "[running]",
},
{
name: "succeeded_status",
status: StatusSucceeded,
expected: "[succeeded]",
},
{
name: "failed_status",
status: StatusFailed,
expected: "[failed]",
},
{
name: "aborted_status",
status: StatusAborted,
expected: "[aborted]",
},
{
name: "unknown_status",
status: StatusUnknown,
expected: "[unknown]",
},
{
name: "custom_status",
status: "custom",
expected: "[custom]",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := NewRenderer(false)
tree := NewStatusTreeWithStatus("node", tt.status)
output := r.Render(tree)
assert.Contains(t, output, tt.expected)
})
}
}
func TestRenderer_TreeStructure(t *testing.T) {
tests := []struct {
name string
buildTree func() *StatusTree
contains []string
}{
{
name: "single_node",
buildTree: func() *StatusTree {
return NewStatusTree("root")
},
contains: []string{"root"},
},
{
name: "parent_with_children",
buildTree: func() *StatusTree {
tree := NewStatusTree("parent")
tree.Add("child1")
tree.Add("child2")
return tree
},
contains: []string{"parent", "child1", "child2", "├──", "└──"},
},
{
name: "deep_nesting",
buildTree: func() *StatusTree {
tree := NewStatusTree("level1")
l2 := tree.Add("level2")
l3 := l2.Add("level3")
l3.Add("level4")
return tree
},
contains: []string{"level1", "level2", "level3", "level4"},
},
{
name: "workflow_structure",
buildTree: func() *StatusTree {
tree := NewStatusTreeWithStatus("my-workflow", StatusRunning)
g1 := tree.AddWithStatus("group1", StatusSucceeded)
g1.AddWithStatus("job1", StatusSucceeded)
g2 := tree.AddWithStatus("group2", StatusRunning)
g2.Add(RenderDependency("group1", true))
g2.AddWithStatus("job2", StatusRunning)
return tree
},
contains: []string{
"my-workflow", "[running]",
"group1", "[succeeded]",
"job1",
"group2",
"depends on group: group1",
"job2",
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := NewRenderer(false)
tree := tt.buildTree()
output := r.Render(tree)
for _, expected := range tt.contains {
assert.Contains(t, output, expected)
}
})
}
}
func TestRenderer_ColorMode(t *testing.T) {
tree := NewStatusTreeWithStatus("workflow", StatusSucceeded)
// Without color
rNoColor := NewRenderer(false)
outputNoColor := rNoColor.Render(tree)
assert.Contains(t, outputNoColor, "[succeeded]")
// With color - output should still contain the text (ANSI codes are transparent to Contains)
rColor := NewRenderer(true)
outputColor := rColor.Render(tree)
assert.NotEmpty(t, outputColor)
}
func TestRenderer_ComplexWorkflow(t *testing.T) {
r := NewRenderer(false)
// Build a complex workflow tree
workflow := NewStatusTreeWithStatus("production-deploy", StatusRunning)
// Setup phase
setup := workflow.AddWithStatus("setup", StatusSucceeded)
setup.AddWithStatus("configure-aws", StatusSucceeded)
setup.AddWithStatus("validate-config", StatusSucceeded)
// Build phase
build := workflow.AddWithStatus("build", StatusSucceeded)
build.Add(RenderDependency("setup", true))
build.AddWithStatus("compile-frontend", StatusSucceeded)
build.AddWithStatus("compile-backend", StatusSucceeded)
build.AddWithStatus("run-unit-tests", StatusSucceeded)
// Deploy phase
deploy := workflow.AddWithStatus("deploy", StatusRunning)
deploy.Add(RenderDependency("build", true))
staging := deploy.AddWithStatus("deploy-staging", StatusSucceeded)
staging.Add(RenderDependency("compile-frontend", false))
integration := deploy.AddWithStatus("run-integration-tests", StatusRunning)
integration.Add(RenderDependency("deploy-staging", false))
prod := deploy.AddWithStatus("deploy-production", StatusPending)
prod.Add(RenderDependency("run-integration-tests", false))
// Cleanup phase
cleanup := workflow.AddWithStatus("cleanup", StatusPending)
cleanup.Add(RenderDependency("deploy", true))
cleanup.AddWithStatus("remove-staging", StatusPending)
output := r.Render(workflow)
// Verify structure
expectedPhrases := []string{
"production-deploy",
"setup",
"configure-aws",
"build",
"compile-frontend",
"deploy",
"deploy-staging",
"run-integration-tests",
"deploy-production",
"cleanup",
"depends on group: setup",
"depends on group: build",
"depends on: deploy-staging",
}
for _, phrase := range expectedPhrases {
assert.Contains(t, output, phrase, "output should contain: %s", phrase)
}
// Verify line structure
lines := strings.Split(output, "\n")
assert.Greater(t, len(lines), 10, "should have multiple lines")
}
func TestRenderDependency_Matrix(t *testing.T) {
tests := []struct {
name string
depName string
isGroup bool
expected string
}{
{
name: "job_dependency",
depName: "init-job",
isGroup: false,
expected: "depends on: init-job",
},
{
name: "group_dependency",
depName: "setup-group",
isGroup: true,
expected: "depends on group: setup-group",
},
{
name: "long_job_name",
depName: "very-long-job-name-with-many-parts",
isGroup: false,
expected: "depends on: very-long-job-name-with-many-parts",
},
{
name: "empty_name",
depName: "",
isGroup: false,
expected: "depends on: ",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
result := RenderDependency(tt.depName, tt.isGroup)
assert.Equal(t, tt.expected, result)
})
}
}
// Test that tree rendering handles edge cases
func TestRenderer_EdgeCases(t *testing.T) {
tests := []struct {
name string
buildTree func() *StatusTree
verify func(*testing.T, string)
}{
{
name: "empty_text_node",
buildTree: func() *StatusTree {
return NewStatusTree("")
},
verify: func(t *testing.T, output string) {
assert.NotEmpty(t, output)
},
},
{
name: "special_characters_in_text",
buildTree: func() *StatusTree {
tree := NewStatusTree("workflow-v1.2.3")
tree.Add("job_with_underscore")
tree.Add("job.with.dots")
return tree
},
verify: func(t *testing.T, output string) {
assert.Contains(t, output, "workflow-v1.2.3")
assert.Contains(t, output, "job_with_underscore")
assert.Contains(t, output, "job.with.dots")
},
},
{
name: "many_siblings",
buildTree: func() *StatusTree {
tree := NewStatusTree("parent")
for i := 0; i < 20; i++ {
tree.Add("child")
}
return tree
},
verify: func(t *testing.T, output string) {
// Should have 19 middle items and 1 last item
assert.Equal(t, 19, strings.Count(output, "├──"))
assert.Equal(t, 1, strings.Count(output, "└──"))
},
},
{
name: "unicode_in_text",
buildTree: func() *StatusTree {
tree := NewStatusTree("🚀 deployment")
tree.Add("✅ verified")
tree.Add("⏳ pending")
return tree
},
verify: func(t *testing.T, output string) {
assert.Contains(t, output, "🚀")
assert.Contains(t, output, "✅")
assert.Contains(t, output, "⏳")
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := NewRenderer(false)
tree := tt.buildTree()
output := r.Render(tree)
tt.verify(t, output)
})
}
}
+54
View File
@@ -0,0 +1,54 @@
package visualization
// StatusTree represents a tree node with text and execution status
type StatusTree struct {
text string
status string
items []*StatusTree
}
// NewStatusTree creates a new StatusTree node
func NewStatusTree(text string) *StatusTree {
return &StatusTree{
text: text,
items: make([]*StatusTree, 0),
}
}
// NewStatusTreeWithStatus creates a new StatusTree node with a status
func NewStatusTreeWithStatus(text, status string) *StatusTree {
return &StatusTree{
text: text,
status: status,
items: make([]*StatusTree, 0),
}
}
// Add creates and appends a child node, returning it for chaining
func (t *StatusTree) Add(text string) *StatusTree {
child := NewStatusTree(text)
t.items = append(t.items, child)
return child
}
// AddWithStatus creates and appends a child node with status
func (t *StatusTree) AddWithStatus(text, status string) *StatusTree {
child := NewStatusTreeWithStatus(text, status)
t.items = append(t.items, child)
return child
}
// Items returns all child nodes
func (t *StatusTree) Items() []*StatusTree {
return t.items
}
// Text returns the node's text value
func (t *StatusTree) Text() string {
return t.text
}
// Status returns the node's status value
func (t *StatusTree) Status() string {
return t.status
}
+165
View File
@@ -0,0 +1,165 @@
package visualization
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/suite"
)
type TreeTestSuite struct {
suite.Suite
}
func TestTreeSuite(t *testing.T) {
suite.Run(t, new(TreeTestSuite))
}
func (s *TreeTestSuite) TestNewStatusTree() {
tree := NewStatusTree("root")
s.Equal("root", tree.Text())
s.Equal("", tree.Status())
s.Empty(tree.Items())
}
func (s *TreeTestSuite) TestNewStatusTreeWithStatus() {
tree := NewStatusTreeWithStatus("workflow", StatusRunning)
s.Equal("workflow", tree.Text())
s.Equal(StatusRunning, tree.Status())
s.Empty(tree.Items())
}
func (s *TreeTestSuite) TestAdd() {
root := NewStatusTree("root")
child := root.Add("child")
s.Equal("child", child.Text())
s.Len(root.Items(), 1)
s.Equal(child, root.Items()[0])
}
func (s *TreeTestSuite) TestAddWithStatus() {
root := NewStatusTree("root")
child := root.AddWithStatus("job", StatusSucceeded)
s.Equal("job", child.Text())
s.Equal(StatusSucceeded, child.Status())
s.Len(root.Items(), 1)
}
func (s *TreeTestSuite) TestChaining() {
root := NewStatusTree("workflow")
group := root.Add("group1")
job := group.AddWithStatus("job1", StatusRunning)
job.Add("depends on: init-job")
s.Len(root.Items(), 1)
s.Len(root.Items()[0].Items(), 1)
s.Len(root.Items()[0].Items()[0].Items(), 1)
}
func (s *TreeTestSuite) TestItems() {
root := NewStatusTree("root")
root.Add("child1")
root.Add("child2")
root.Add("child3")
items := root.Items()
s.Len(items, 3)
s.Equal("child1", items[0].Text())
s.Equal("child2", items[1].Text())
s.Equal("child3", items[2].Text())
}
// ==================== MATRIX TEST: TREE BUILDING ====================
func TestStatusTree_StatusValues(t *testing.T) {
statuses := []string{
StatusPending,
StatusRunning,
StatusSucceeded,
StatusFailed,
StatusAborted,
StatusUnknown,
}
for _, status := range statuses {
t.Run(status, func(t *testing.T) {
tree := NewStatusTreeWithStatus("node", status)
assert.Equal(t, status, tree.Status())
})
}
}
func TestStatusTree_DeepNesting(t *testing.T) {
depth := 10
root := NewStatusTree("level-0")
current := root
for i := 1; i < depth; i++ {
current = current.Add("level-" + string(rune('0'+i)))
}
// Traverse back to verify
node := root
for i := 0; i < depth-1; i++ {
assert.Len(t, node.Items(), 1)
node = node.Items()[0]
}
assert.Empty(t, node.Items())
}
func TestStatusTree_MultipleChildren(t *testing.T) {
root := NewStatusTree("workflow")
// Add multiple groups
for i := 0; i < 5; i++ {
group := root.AddWithStatus("group"+string(rune('0'+i)), StatusPending)
// Add jobs to each group
for j := 0; j < 3; j++ {
group.AddWithStatus("job"+string(rune('0'+j)), StatusPending)
}
}
assert.Len(t, root.Items(), 5)
for _, group := range root.Items() {
assert.Len(t, group.Items(), 3)
}
}
func TestStatusTree_EmptyTree(t *testing.T) {
tree := NewStatusTree("")
assert.Equal(t, "", tree.Text())
assert.Equal(t, "", tree.Status())
assert.Empty(t, tree.Items())
}
func TestStatusTree_ComplexWorkflow(t *testing.T) {
// Simulate a real workflow structure
workflow := NewStatusTreeWithStatus("my-workflow", StatusRunning)
// First group - succeeded
group1 := workflow.AddWithStatus("init-group", StatusSucceeded)
group1.AddWithStatus("setup-database", StatusSucceeded)
group1.AddWithStatus("setup-cache", StatusSucceeded)
// Second group - running
group2 := workflow.AddWithStatus("build-group", StatusRunning)
group2.Add("depends on group: init-group")
build := group2.AddWithStatus("build-app", StatusRunning)
build.Add("depends on: setup-database")
group2.AddWithStatus("run-tests", StatusPending).Add("depends on: build-app")
// Third group - pending
group3 := workflow.AddWithStatus("deploy-group", StatusPending)
group3.Add("depends on group: build-group")
group3.AddWithStatus("deploy-staging", StatusPending)
group3.AddWithStatus("deploy-production", StatusPending).Add("depends on: deploy-staging")
assert.Len(t, workflow.Items(), 3)
assert.Equal(t, StatusSucceeded, workflow.Items()[0].Status())
assert.Equal(t, StatusRunning, workflow.Items()[1].Status())
assert.Equal(t, StatusPending, workflow.Items()[2].Status())
}
+71
View File
@@ -0,0 +1,71 @@
apiVersion: krew.googlecontainertools.github.com/v1alpha2
kind: Plugin
metadata:
name: managedjob
spec:
version: v0.0.33
homepage: https://github.com/lukaszraczylo/jobs-manager-operator
shortDescription: Visualize ManagedJob workflows
description: |
A kubectl plugin for visualizing ManagedJob workflows from the
Jobs Manager Operator. It provides ASCII tree visualization of
workflow structure and execution status with colored output.
Features:
- Visualize workflow structure as ASCII tree
- Color-coded status indicators (green=succeeded, yellow=running, red=failed)
- Watch mode for real-time status updates
- List all ManagedJobs in a namespace
- Quick status summary view
Usage:
kubectl managedjob visualize <name> -n <namespace>
kubectl managedjob visualize <name> -w # Watch mode
kubectl managedjob list -n <namespace>
kubectl managedjob status <name> -n <namespace>
caveats: |
This plugin requires the Jobs Manager Operator CRD to be installed
in your cluster. See https://github.com/lukaszraczylo/jobs-manager-operator
for installation instructions.
platforms:
- selector:
matchLabels:
os: darwin
arch: amd64
uri: https://github.com/lukaszraczylo/jobs-manager-operator/releases/download/v0.0.33/jobs-manager-operator_0.0.33_darwin_amd64.tar.gz
sha256: REPLACE_WITH_ACTUAL_SHA256
bin: kubectl-managedjob
- selector:
matchLabels:
os: darwin
arch: arm64
uri: https://github.com/lukaszraczylo/jobs-manager-operator/releases/download/v0.0.33/jobs-manager-operator_0.0.33_darwin_arm64.tar.gz
sha256: REPLACE_WITH_ACTUAL_SHA256
bin: kubectl-managedjob
- selector:
matchLabels:
os: linux
arch: amd64
uri: https://github.com/lukaszraczylo/jobs-manager-operator/releases/download/v0.0.33/jobs-manager-operator_0.0.33_linux_amd64.tar.gz
sha256: REPLACE_WITH_ACTUAL_SHA256
bin: kubectl-managedjob
- selector:
matchLabels:
os: linux
arch: arm64
uri: https://github.com/lukaszraczylo/jobs-manager-operator/releases/download/v0.0.33/jobs-manager-operator_0.0.33_linux_arm64.tar.gz
sha256: REPLACE_WITH_ACTUAL_SHA256
bin: kubectl-managedjob
- selector:
matchLabels:
os: windows
arch: amd64
uri: https://github.com/lukaszraczylo/jobs-manager-operator/releases/download/v0.0.33/jobs-manager-operator_0.0.33_windows_amd64.zip
sha256: REPLACE_WITH_ACTUAL_SHA256
bin: kubectl-managedjob.exe
Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

+9
View File
@@ -0,0 +1,9 @@
{
"packageRules": [
{
"matchManagers": ["gomod"],
"matchPackageNames": ["github.com/google/cel-go"],
"allowedVersions": "<0.32.0"
}
]
}
+293
View File
@@ -0,0 +1,293 @@
#!/bin/bash
# kubectl-managedjob - Installation Script
# Usage: curl -sSL https://raw.githubusercontent.com/lukaszraczylo/jobs-manager-operator/main/scripts/install-plugin.sh | bash
#
# Or with a specific version:
# curl -sSL https://raw.githubusercontent.com/lukaszraczylo/jobs-manager-operator/main/scripts/install-plugin.sh | bash -s -- v1.0.0
set -e
# Configuration
GITHUB_REPO="lukaszraczylo/jobs-manager-operator"
BINARY_NAME="kubectl-managedjob"
INSTALL_DIR="${INSTALL_DIR:-/usr/local/bin}"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
info() {
echo -e "${BLUE}[INFO]${NC} $1"
}
success() {
echo -e "${GREEN}[OK]${NC} $1"
}
warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
error() {
echo -e "${RED}[ERROR]${NC} $1"
exit 1
}
# Detect OS and architecture
detect_platform() {
local os arch
case "$(uname -s)" in
Darwin)
os="darwin"
;;
Linux)
os="linux"
;;
MINGW*|MSYS*|CYGWIN*)
os="windows"
;;
*)
error "Unsupported operating system: $(uname -s)"
;;
esac
case "$(uname -m)" in
x86_64|amd64)
arch="amd64"
;;
arm64|aarch64)
arch="arm64"
;;
*)
error "Unsupported architecture: $(uname -m)"
;;
esac
echo "${os}_${arch}"
}
# Get the latest release version from GitHub
get_latest_version() {
local response version curl_opts
# Use GitHub token if available (higher rate limit)
curl_opts=(-sS)
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_opts+=(-H "Authorization: token ${GITHUB_TOKEN}")
fi
# Fetch with error handling
response=$(curl "${curl_opts[@]}" "https://api.github.com/repos/${GITHUB_REPO}/releases/latest" 2>&1)
# Check for rate limiting
if echo "$response" | grep -q "API rate limit exceeded"; then
error "GitHub API rate limit exceeded.
You have a few options:
1. Wait ~1 hour for the rate limit to reset
2. Specify a version manually:
curl -sSL https://raw.githubusercontent.com/${GITHUB_REPO}/main/scripts/install-plugin.sh | bash -s -- v0.0.33
3. Use a GitHub token (set GITHUB_TOKEN environment variable)"
fi
# Check for other API errors
if echo "$response" | grep -q '"message":'; then
local msg
msg=$(echo "$response" | grep '"message":' | sed -E 's/.*"message": *"([^"]+)".*/\1/')
error "GitHub API error: $msg"
fi
# Extract version
version=$(echo "$response" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
if [[ -z "$version" ]]; then
error "Failed to fetch latest version from GitHub. Response: $response"
fi
echo "$version"
}
# Download and install the plugin
download_and_install() {
local version="$1"
local platform="$2"
local tmp_dir
tmp_dir=$(mktemp -d)
trap "rm -rf $tmp_dir" EXIT
# Construct download URL
local archive_ext="tar.gz"
if [[ "$platform" == windows_* ]]; then
archive_ext="zip"
fi
# Archive name format from goreleaser: jobs-manager-operator_VERSION_OS_ARCH.tar.gz
local archive_name="jobs-manager-operator_${version#v}_${platform}.${archive_ext}"
local download_url="https://github.com/${GITHUB_REPO}/releases/download/${version}/${archive_name}"
info "Downloading ${archive_name}..."
if ! curl -sSL -o "$tmp_dir/release.${archive_ext}" "$download_url"; then
error "Failed to download release from: $download_url"
fi
info "Extracting archive..."
if [[ "$archive_ext" == "zip" ]]; then
if ! unzip -q "$tmp_dir/release.zip" -d "$tmp_dir"; then
error "Failed to extract archive"
fi
else
if ! tar -xzf "$tmp_dir/release.tar.gz" -C "$tmp_dir"; then
error "Failed to extract archive"
fi
fi
# Find the binary
local binary_path
binary_path=$(find "$tmp_dir" -name "$BINARY_NAME" -type f | head -1)
if [[ -z "$binary_path" ]]; then
error "Could not find $BINARY_NAME in the archive"
fi
# Determine install location
local install_path="$INSTALL_DIR/$BINARY_NAME"
# Check if we need sudo
if [[ -w "$INSTALL_DIR" ]]; then
info "Installing to ${install_path}..."
cp "$binary_path" "$install_path"
chmod +x "$install_path"
else
info "Installing to ${install_path} (requires sudo)..."
sudo cp "$binary_path" "$install_path"
sudo chmod +x "$install_path"
fi
success "Installed $BINARY_NAME to $install_path"
}
# Verify installation
verify_installation() {
if command -v "$BINARY_NAME" &> /dev/null; then
success "Installation verified!"
info "Run '$BINARY_NAME --help' to get started"
echo ""
"$BINARY_NAME" --help 2>/dev/null | head -10 || true
else
warn "Installation complete, but $BINARY_NAME is not in PATH"
warn "You may need to add $INSTALL_DIR to your PATH"
echo ""
echo "Add this to your shell profile (~/.bashrc, ~/.zshrc, etc.):"
echo " export PATH=\"\$PATH:$INSTALL_DIR\""
fi
}
# Handle --uninstall flag
uninstall() {
local install_path="$INSTALL_DIR/$BINARY_NAME"
echo ""
echo "Uninstalling $BINARY_NAME..."
if [[ -f "$install_path" ]]; then
if [[ -w "$install_path" ]]; then
rm "$install_path"
else
sudo rm "$install_path"
fi
success "Removed $install_path"
else
warn "$BINARY_NAME not found at $install_path"
fi
success "Uninstallation complete"
}
# Main installation flow
main() {
local version="${1:-}"
echo ""
echo "╔═══════════════════════════════════════════════════════════╗"
echo "║ kubectl-managedjob - Installation Script ║"
echo "║ Kubernetes workflow visualization plugin for kubectl ║"
echo "╚═══════════════════════════════════════════════════════════╝"
echo ""
# Check required dependencies
if ! command -v curl &> /dev/null; then
error "curl is required but not installed"
fi
if ! command -v tar &> /dev/null; then
error "tar is required but not installed"
fi
# Detect platform
local platform
platform=$(detect_platform)
info "Detected platform: $platform"
# Get version
if [[ -z "$version" ]]; then
info "Fetching latest release..."
version=$(get_latest_version)
fi
info "Installing version: $version"
# Download and install
download_and_install "$version" "$platform"
# Verify
verify_installation
echo ""
echo "╔═══════════════════════════════════════════════════════════╗"
echo "║ Installation Complete! ║"
echo "╠═══════════════════════════════════════════════════════════╣"
echo "║ Usage: ║"
echo "║ kubectl managedjob visualize <name> -n <namespace> ║"
echo "║ kubectl managedjob list -n <namespace> ║"
echo "║ kubectl managedjob status <name> -n <namespace> ║"
echo "╚═══════════════════════════════════════════════════════════╝"
echo ""
}
# Handle flags
case "${1:-}" in
--uninstall)
uninstall
exit 0
;;
--help|-h)
echo "kubectl-managedjob Installation Script"
echo ""
echo "Usage:"
echo " install-plugin.sh [VERSION] Install kubectl-managedjob (latest or specific version)"
echo " install-plugin.sh --uninstall Remove kubectl-managedjob"
echo " install-plugin.sh --help Show this help"
echo ""
echo "Environment Variables:"
echo " INSTALL_DIR Installation directory (default: /usr/local/bin)"
echo " GITHUB_TOKEN GitHub token for higher API rate limits"
echo ""
echo "Examples:"
echo " # Install latest version"
echo " curl -sSL https://raw.githubusercontent.com/${GITHUB_REPO}/main/scripts/install-plugin.sh | bash"
echo ""
echo " # Install specific version"
echo " curl -sSL https://raw.githubusercontent.com/${GITHUB_REPO}/main/scripts/install-plugin.sh | bash -s -- v0.0.33"
echo ""
echo " # Install to custom directory"
echo " INSTALL_DIR=~/bin curl -sSL https://raw.githubusercontent.com/${GITHUB_REPO}/main/scripts/install-plugin.sh | bash"
exit 0
;;
esac
main "$@"
+72
View File
@@ -0,0 +1,72 @@
#!/bin/bash
# Update Krew manifest with correct SHA256 checksums
# Usage: ./scripts/update-krew-manifest.sh v0.0.33
set -e
VERSION="${1:-}"
GITHUB_REPO="lukaszraczylo/jobs-manager-operator"
MANIFEST_FILE="plugins/krew/managedjob.yaml"
if [[ -z "$VERSION" ]]; then
echo "Usage: $0 <version>"
echo "Example: $0 v0.0.33"
exit 1
fi
# Remove 'v' prefix for archive names
VERSION_NUM="${VERSION#v}"
echo "Updating Krew manifest for version $VERSION..."
# Platforms to update
PLATFORMS=(
"darwin_amd64:tar.gz"
"darwin_arm64:tar.gz"
"linux_amd64:tar.gz"
"linux_arm64:tar.gz"
"windows_amd64:zip"
)
# Download checksums file
CHECKSUMS_URL="https://github.com/${GITHUB_REPO}/releases/download/${VERSION}/checksums.txt"
echo "Downloading checksums from $CHECKSUMS_URL..."
CHECKSUMS=$(curl -sSL "$CHECKSUMS_URL")
if [[ -z "$CHECKSUMS" ]]; then
echo "Error: Failed to download checksums"
exit 1
fi
echo "Checksums:"
echo "$CHECKSUMS"
echo ""
# Update manifest
for platform_ext in "${PLATFORMS[@]}"; do
platform="${platform_ext%:*}"
ext="${platform_ext#*:}"
archive_name="jobs-manager-operator_${VERSION_NUM}_${platform}.${ext}"
sha256=$(echo "$CHECKSUMS" | grep "$archive_name" | awk '{print $1}')
if [[ -z "$sha256" ]]; then
echo "Warning: No checksum found for $archive_name"
continue
fi
echo " $platform: $sha256"
done
# Update version in manifest
sed -i.bak "s/version: v.*/version: ${VERSION}/" "$MANIFEST_FILE"
# Update URIs and find/replace SHA256 placeholders
# This is a simplified approach - for production, use yq or similar
echo ""
echo "Manifest updated with version $VERSION"
echo "NOTE: SHA256 checksums must be manually updated in $MANIFEST_FILE"
echo ""
echo "Copy the checksums above and replace REPLACE_WITH_ACTUAL_SHA256 in the manifest."
rm -f "${MANIFEST_FILE}.bak"
+1 -6
View File
@@ -1,11 +1,6 @@
version: 1
force:
major: 0
minor: 0
patch: 0
wording:
patch:
minor:
- release-minor
major:
- release-major
- release-major