Skip to content

HYPERFLEET-569 - feat: Implement maestro DSL loader #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rh-amarin wants to merge 1 commit into
base: main
Choose a base branch
from
+702 −1,229
Open

HYPERFLEET-569 - feat: Implement maestro DSL loader #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ A HyperFleet Adapter requires several files for configuration:
To see all configuration options read [configuration.md](configuration.md) file

#### Adapter configuration

The adapter deployment configuration (`AdapterConfig`) controls runtime and infrastructure
settings for the adapter process, such as client connections, retries, and broker
subscription details. It is loaded with Viper, so values can be overridden by CLI flags
Expand All @@ -167,10 +168,11 @@ and environment variables in this priority order: CLI flags > env vars > file >
(HyperFleet API, Maestro, broker, Kubernetes)

Reference examples:
- `configs/adapter-deployment-config.yaml` (full reference with env/flag notes)
- `charts/examples/adapter-config.yaml` (minimal deployment example)

- `charts/examples/adapter-config.yaml`

#### Adapter task configuration

The adapter task configuration (`AdapterTaskConfig`) defines the **business logic** for
processing events: parameters, preconditions, resources to create, and post-actions.
This file is loaded as **static YAML** (no Viper overrides) and is required at runtime.
Expand All @@ -180,13 +182,13 @@ This file is loaded as **static YAML** (no Viper overrides) and is required at r
- **Resource manifests**: inline YAML or external file via `manifest.ref`

Reference examples:
- `charts/examples/adapter-task-config.yaml` (worked example)
- `configs/adapter-task-config-template.yaml` (complete schema reference)

- `charts/examples/adapter-task-config.yaml` (worked example)

### Broker Configuration

Broker configuration is particular since responsibility is split between:

- **Hyperfleet broker library**: configures the connection to a concrete broker (google pubsub, rabbitmq, ...)
- Configured using a YAML file specified by the `BROKER_CONFIG_FILE` environment variable
- **Adapter**: configures which topic/subscriptions to use on the broker
Expand Down
51 changes: 48 additions & 3 deletions charts/examples/adapter-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@ spec:
version: "0.1.0"

# Log the full merged configuration after load (default: false)
debugConfig: false
debugConfig: true
log:
level: debug

clients:
hyperfleetApi:
Expand All @@ -22,8 +24,51 @@ spec:
retryBackoff: exponential

broker:
subscriptionId: "example-clusters-subscription"
topic: "example-clusters"
subscriptionId: "CHANGE_ME"
topic: "CHANGE_ME"

kubernetes:
apiVersion: "v1"
#kubeConfigPath: PATH_TO_KUBECONFIG # for local development

maestro:
grpcServerAddress: "maestro-grpc.maestro.svc.cluster.local:8090"

# HTTPS server address for REST API operations (optional)
# Environment variable: HYPERFLEET_MAESTRO_HTTP_SERVER_ADDRESS
httpServerAddress: "http://maestro.maestro.svc.cluster.local:8000"

# Source identifier for CloudEvents routing (must be unique across adapters)
# Environment variable: HYPERFLEET_MAESTRO_SOURCE_ID
sourceId: "hyperfleet-adapter"

# Client identifier (defaults to sourceId if not specified)
# Environment variable: HYPERFLEET_MAESTRO_CLIENT_ID
clientId: "hyperfleet-adapter-client"
insecure: true

# Authentication configuration
#auth:
# type: "tls" # TLS certificate-based mTLS
#
# tlsConfig:
# # gRPC TLS configuration
# # Certificate paths (mounted from Kubernetes secrets)
# # Environment variable: HYPERFLEET_MAESTRO_CA_FILE
# caFile: "/etc/maestro/certs/grpc/ca.crt"
#
# # Environment variable: HYPERFLEET_MAESTRO_CERT_FILE
# certFile: "/etc/maestro/certs/grpc/client.crt"
#
# # Environment variable: HYPERFLEET_MAESTRO_KEY_FILE
# keyFile: "/etc/maestro/certs/grpc/client.key"
#
# # Server name for TLS verification
# # Environment variable: HYPERFLEET_MAESTRO_SERVER_NAME
# serverName: "maestro-grpc.maestro.svc.cluster.local"
#
# # HTTP API TLS configuration (may use different CA than gRPC)
# # If not set, falls back to caFile for backwards compatibility
# # Environment variable: HYPERFLEET_MAESTRO_HTTP_CA_FILE
# httpCaFile: "/etc/maestro/certs/https/ca.crt"

34 changes: 34 additions & 0 deletions charts/examples/adapter-task-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,31 @@ spec:
# Resources with valid K8s manifests
resources:
- name: "maestro"
Copy link
Contributor

@xueli181114 xueli181114 Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This kind of configuration lose the manifestwork ref. And for there should be multiple manifests in the manifestwork.

Copy link
Contributor

@xueli181114 xueli181114 Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And we should put manifests in same manifestwork together. Manifests number cannot be changed once the manifestwork created.

transport:
client: "maestro"
maestro:
targetCluster: cluster1
Copy link
Contributor

@xueli181114 xueli181114 Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

manifestwork.ref here should be something required and have a validation

manifest:
apiVersion: v1
kind: Namespace
metadata:
name: "maestro-{{ .clusterId }}"
labels:
hyperfleet.io/cluster-id: "{{ .clusterId }}"
hyperfleet.io/cluster-name: "{{ .clusterName }}"
annotations:
hyperfleet.io/generation: "{{ .generationSpec }}"
discovery:
Copy link
Contributor

@xueli181114 xueli181114 Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discovery is manifest level not manifestwork.

Copy link
Contributor

@xueli181114 xueli181114 Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the discovery can help us locate the manifest attributes with jsonPath

namespace: "*" # Cluster-scoped resource (Namespace)
bySelectors:
labelSelector:
hyperfleet.io/cluster-id: "{{ .clusterId }}"
hyperfleet.io/cluster-name: "{{ .clusterName }}"

- name: "clusterNamespace"
transport:
client: "kubernetes"
manifest:
apiVersion: v1
kind: Namespace
Expand All @@ -98,6 +122,8 @@ spec:
# in the namespace created above
# it will require a service account to be created in that namespace as well as a role and rolebinding
- name: "jobServiceAccount"
transport:
client: "kubernetes"
manifest:
ref: "/etc/adapter/job-serviceaccount.yaml"
discovery:
Expand All @@ -107,6 +133,8 @@ spec:
hyperfleet.io/cluster-id: "{{ .clusterId }}"

- name: "job_role"
transport:
client: "kubernetes"
manifest:
ref: "/etc/adapter/job-role.yaml"
discovery:
Expand All @@ -118,6 +146,8 @@ spec:
hyperfleet.io/resource-type: "role"

- name: "job_rolebinding"
transport:
client: "kubernetes"
manifest:
ref: "/etc/adapter/job-rolebinding.yaml"
discovery:
Expand All @@ -129,6 +159,8 @@ spec:
hyperfleet.io/resource-type: "rolebinding"

- name: "jobNamespace"
transport:
client: "kubernetes"
manifest:
ref: "/etc/adapter/job.yaml"
discovery:
Expand All @@ -143,6 +175,8 @@ spec:
# and using the same service account as the adapter

- name: "deploymentNamespace"
transport:
client: "kubernetes"
manifest:
ref: "/etc/adapter/deployment.yaml"
discovery:
Expand Down
53 changes: 50 additions & 3 deletions cmd/adapter/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"github.com/openshift-hyperfleet/hyperfleet-adapter/internal/executor"
"github.com/openshift-hyperfleet/hyperfleet-adapter/internal/hyperfleet_api"
"github.com/openshift-hyperfleet/hyperfleet-adapter/internal/k8s_client"
"github.com/openshift-hyperfleet/hyperfleet-adapter/internal/maestro_client"
"github.com/openshift-hyperfleet/hyperfleet-adapter/pkg/health"
"github.com/openshift-hyperfleet/hyperfleet-adapter/pkg/logger"
"github.com/openshift-hyperfleet/hyperfleet-adapter/pkg/otel"
Expand Down Expand Up @@ -285,14 +286,32 @@ func runServe() error {
return fmt.Errorf("failed to create Kubernetes client: %w", err)
}

// Create Maestro client if configured
var maestroClient maestro_client.ManifestWorkClient
if config.Spec.Clients.Maestro != nil {
log.Info(ctx, "Creating Maestro client...")
maestroClient, err = createMaestroClient(ctx, config.Spec.Clients.Maestro, log)
if err != nil {
errCtx := logger.WithErrorField(ctx, err)
log.Errorf(errCtx, "Failed to create Maestro client")
return fmt.Errorf("failed to create Maestro client: %w", err)
}
log.Info(ctx, "Maestro client created successfully")
}

// Create the executor using the builder pattern
log.Info(ctx, "Creating event executor...")
exec, err := executor.NewBuilder().
execBuilder := executor.NewBuilder().
WithConfig(config).
WithAPIClient(apiClient).
WithK8sClient(k8sClient).
WithLogger(log).
Build()
WithLogger(log)

if maestroClient != nil {
execBuilder = execBuilder.WithMaestroClient(maestroClient)
}

exec, err := execBuilder.Build()
if err != nil {
errCtx := logger.WithErrorField(ctx, err)
log.Errorf(errCtx, "Failed to create executor")
Expand Down Expand Up @@ -494,3 +513,31 @@ func createK8sClient(ctx context.Context, k8sConfig config_loader.KubernetesConf
}
return k8s_client.NewClient(ctx, clientConfig, log)
}

// createMaestroClient creates a Maestro client from the config
func createMaestroClient(ctx context.Context, maestroConfig *config_loader.MaestroClientConfig, log logger.Logger) (*maestro_client.Client, error) {
Copy link
Contributor

@xueli181114 xueli181114 Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about move the createXXXClient functions to another file to simplify main.go file?

clientConfig := &maestro_client.Config{
MaestroServerAddr: maestroConfig.HTTPServerAddress,
GRPCServerAddr: maestroConfig.GRPCServerAddress,
SourceID: maestroConfig.SourceID,
Insecure: maestroConfig.Insecure,
}

// Parse timeout if specified
if maestroConfig.Timeout != "" {
timeout, err := time.ParseDuration(maestroConfig.Timeout)
if err != nil {
return nil, fmt.Errorf("invalid maestro timeout %q: %w", maestroConfig.Timeout, err)
}
clientConfig.HTTPTimeout = timeout
}

// Configure TLS if auth type is "tls"
if maestroConfig.Auth.Type == "tls" && maestroConfig.Auth.TLSConfig != nil {
Copy link
Contributor

@rafabene rafabene Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could lead to a confusing scenario where a user
configures auth.type: tls but forgets to provide tlsConfig — the client would silently
connect without mTLS, potentially exposing traffic.

Consider returning an explicit error when the configuration is inconsistent:

  if maestroConfig.Auth.Type == "tls" {
      if maestroConfig.Auth.TLSConfig == nil {
          return nil, fmt.Errorf("maestro auth type is 'tls' but tlsConfig is not provided")
      }
      clientConfig.CAFile = maestroConfig.Auth.TLSConfig.CAFile
      clientConfig.ClientCertFile = maestroConfig.Auth.TLSConfig.CertFile
      clientConfig.ClientKeyFile = maestroConfig.Auth.TLSConfig.KeyFile
  }

clientConfig.CAFile = maestroConfig.Auth.TLSConfig.CAFile
clientConfig.ClientCertFile = maestroConfig.Auth.TLSConfig.CertFile
clientConfig.ClientKeyFile = maestroConfig.Auth.TLSConfig.KeyFile
}

return maestro_client.NewMaestroClient(ctx, clientConfig, log)
}
Loading