Deploying Hosted Execution with DAI

When deploying DAI into a Kubernetes environment you can optionally choose to deploy Hosted Execution alongside it. Hosted Execution provides ephemeral SUTs and EEs. These are launched as containers, on demand to run tests. This page describes how to enable Hosted Execution and the associated configuration options.

Enabling Hosted Execution

Enabling Hosted Execution can be achieved by setting the global.eggplantCloud.enableDaiCloudSutV2 to true in your DAI values. Additionally you will need to provide an RLM license for EPF along with the licensed MAC address. If you are not sure what to put for these values then please contact Eggplant Customer Support.

dai.yaml (Additional Values)

global:
  eggplantCloud:
    enableDaiCloudSutV2: true

hosted-execution:
  rlm-hosted-execution:
    license: |-
       Your EPF license multiline string
    licensedMacAddress: aa:bb:cc:dd:ee:ff

The above should be added to the dai.yaml file created in Deploying Eggplant DAI in Containers. You can then proceed to deploy the helm chart as normal (and described in Deploy Eggplant DAI with Kubernetes).

ヒント

The above describes the minimal configuration changes needed to enable Hosted Execution. It is strongly recommended to review the Supported Customizations section below to further tailor the installation to your environment.

Supported Customizations

There are a number of additional configuration options that can be used to optimise the behaviour of Hosted Execution for your environment. All of the configuration described below is via helm values. The values should be added to your standard dai.yaml file used for the main installation.

Overriding Container Images

You can override the container images used by the SUT and EE if desired. This would, for example, be needed for an air-gapped install. There is a single image for the SUT and two for the EE (one for agent and one for fusion engine). These can be set in the helm values:

dai.yaml (Additional Values)

hosted-execution:
  provisioner:
    sutImage: example.com/dai-hosted-execution-sut:latest
    agentImage: example.com/dai-run-agent:latest
    epfImage: example.com/fusion-engine:latest

Managing the scheduling of SUT and EE pods

To control where the SUT and EE Pods, which are launched by the provisioner, are scheduled, you can add a node selector for these under hosted-execution.provisioner.sutEeNodeSelector.

Additionally, you can optionally, provide a list of tolerations for the scheduling. This can be used to allow the SUT and EE pods to schedule to nodes with taints, if further control of scheduling is desired.

hosted-execution:
  provisioner:
    sutEeNodeSelector:
      node-pool: sut-ee-nodepool
    sutEeTolerations:
      - effect: NoSchedule
        key: node-pool/hosted-execution
        operator: Exists

ヒント

The SUT and EE pods are quite resource heavy. It may be desirable to run these on dedicated nodes for better isolation from other cluster workloads.

ノート

Hosted Execution introduces a number of services such as the provisioner, rlm and traefik. These will all respect the standard DAI field of .global.dai.nodeSelector to set the node selector labels. The above configuration only applies to the on demand SUT and EE pods launched by the provisioner.

Image Pre-Pull

The containers for the SUTs and EEs are launched on demand so that cluster resources are only consumed when needed. The nature of the SUT and EE images means they are larger than typical container images and while the containers themselves start up quickly, there can be a delay caused by waiting for the images to pull to the nodes.

For this reason a "Pre-Pull DaemonSet" is deployed by default. As the name suggests this service runs on each node and ensures that the images are already present before any tests start. Once the images are pulled, the "pause" container is run so that no resources are consumed.

This can be customised in the following ways:

1. If you do not wish to run the Pre-Pull DaemonSet, you can disable it completely by setting hosted-execution.prePullDaemonSet.enabled=false.

2. If you need to override the pause container image (e.g. for an air-gapped install), you can do this by providing the desired image under hosted-execution.prePullDaemonSet.pauseContainer. The default image is registry.k8s.io/pause:3.10.

ノート

The Pre-Pull DaemonSet will use the same images as defined in Overriding Container Images and run with the same node selector and tolerations as described in Managing the scheduling of SUT and EE pods

RLM

Eggplant Fusion Engine uses the Reprise License Manager (RLM) for licensing. By default, we deploy an RLM instance with the helm chart so Fusion Engine can obtain licenses. This requires supplying the RLM license and MAC address with the helm values. See Using RLM for Team License Management for more information about RLM.

Existing Secret

If desired, you can supply the name of an existing Kubernetes secret that contains the RLM license under the key eggPlantFunctional.lic and the MAC address under the key licensedMacAddress. The name of the existing secret must then be supplied under hosted-execution.rlm-hosted-execution.existingSecretName and the license and licensedMacAddress can be left unset.

dai.yaml (Additional Values)

hosted-execution:
  rlm-hosted-execution:
    license:
    licensedMacAddress:
    existingSecretName: my-rlm-secret

External RLM

If you already have an existing RLM installation you can choose to use this, instead of deploying RLM as part of Hosted Execution. This allows you to manage all your Fusion Engine licenses in one place.

To use an external RLM instance, set hosted-execution.rlm-hosted-execution.enabled to false and supply the name of the RLM host under hosted-execution.provisioner.rlmHost

dai.yaml (Additional Values)

hosted-execution:
  provisioner:
    rlmHost: rlm.example.com
  rlm-hosted-execution:
    enabled: false

Design SUT Access

The Hosted Execution SUTs can be used in one of two modes:

• Run Mode: where they are paired with an EE pod and used to run tests directly from DAI.
• Design Mode: where they are requested from the desktop version of Eggplant Studio for the purpose of developing the tests.

From an infrastructure perspective the key difference is that Design SUTs need to accept connections from outside the Kubernetes cluster (i.e., from a user's desktop). Hosted Execution supports two methods of supporting Design SUTs. These are described below, so you may choose the most suitable for your environment.

lb-service

This is the default option and also the simplest to set up, in that it uses the standard Kubernetes Load Balancer service. It will create one Load Balancer service for each Design SUT and pass the external IP for that service back to the Studio application.

This works best for environments where Load Balancers are automatically provisioned quickly and without limitations on numbers (e.g., when using metalLB or similar).

It is not ideal for:

• Cloud environments - where Load Balancer provisioning can take a few minutes, and there may be quotas on the quantity that can be provisioned.
• Environments where new Load Balancers require manual setup / routing config to make the service accessible to users.
• Environments where the External IP that is set on the Load Balancer service is not accessible by the user (i.e. if routing is via firewalls etc.).

traefik

Due to the limitations with the lb-service described above, a second option for Design Mode SUT access is provided through Traefik. Under this option, one or more Load Balancers are pre-provisioned, each with multiple listeners to support multiple Design Mode SUTs. Additionally, a Traefik pod is deployed to support the routing between the Load Balancer and the Cloud-SUTs. Each time a Design Mode SUT is created, an IngressRouteTCP resource is also created. This allows Traefik to dynamically route the connection, while keeping the Load Balancers static.

The minimal configuration to use Traefik is to set hosted-execution.provisioner.designModeType=traefik and enable the Traefik container and Load Balancer with hosted-execution.traefik.enabled=true:

dai.yaml (Additional Values)

hosted-execution:
  provisioner:
    designModeType: traefik
  traefik:
    enabled: true

Further options to customise your installation are shown in the table below:

Key	Description	Default
hosted-execution.traefik.replicaCount	Number of traefik replicas to run	1
hosted-execution.traefik.designModeLoadBalancer.count	Number of Load Balancer service resources to create	1
hosted-execution.traefik.designModeLoadBalancer.loadBalancerClass	The Load Balancer class to set in the service	null
hosted-execution.traefik.designModeLoadBalancer.annotations	Map of annotations to be configured on the Load Balancer service. (useful for passing configuration options to cloud service controllers)	{}
hosted-execution.traefik.designModeLoadBalancer.loadBalancerSourceRanges	List of CIDRs that are allowed to connect to the Load Balancer	[]
hosted-execution.traefik.designModeLoadBalancer.ports.start	Integer for the first port on the Load Balancer that will be used for Design SUTs	5000
hosted-execution.traefik.designModeLoadBalancer.ports.end	Integer for the last port on the Load Balancer that will be used for Design SUTs	5020

Resources

The default CPU and Memory resources for the SUT and EEs that are created are sufficient for a typical test execution. It is possible that these will need to be tuned, based on the actual tests that are being executed. The resources of both the SUT and the EE can be set via the helm values. The table below shows the values:

Key	Description	Default
hosted-execution.provisioner.sutResources	Resource requests and limits for a SUT	{ cpuRequest: "1" cpuLimit: memoryRequest: "4Gi" memoryLimit: "8Gi" storageRequest: "1200Mi" storageLimit: "1200Mi" }
hosted-execution.provisioner.eeResources	Resource requests and limits for an EE	{ cpuRequest: "1" cpuLimit: memoryRequest: "1Gi" memoryLimit: "2Gi" storageRequest: "6Gi" storageLimit: "6Gi" }
hosted-execution.provisioner.sutHomeDirSizeLimit	Storage Limit for the SUT home directory	"1Gi"
hosted-execution.provisioner.sutTmpDirSizeLimit	Storage Limit for the SUT /tmp directory	"200Mi"
hosted-execution.provisioner.eeHomeDirSizeLimit	Storage Limit for the EE home directory	"1Gi"
hosted-execution.provisioner.eeTmpDirSizeLimit	Storage Limit for the EE /tmp directory	"200Mi"

Network Policies

The nature of Hosted Execution is that users of DAI can run arbitrary commands on the SUTs and EEs, which could potentially be used to access cluster or other internal resources not normally accessible by users. We, therefore, encourage the use of Network Policies to limit the access at a network level from the SUT and EE pods.

Network Policies are enabled by default for the SUT and EEs. They can be disabled by setting hosted-execution.networkPolicies.enabled=false in the helm values. This may be useful for debugging, but is not recommended for normal use.

The default rules of the Network Policies block outbound access to private address ranges, while allowing outbound access to public internet ranges. Additionally, access is permitted for required functionality such as DNS and RLM access for the EE.

If your environment requires access to private address ranges, or services running in the cluster, then you can create additional Network Policies and apply them to the same namespace with a matching pod selector. SUT pods will always have the label cloud-sut=True and EE Pods will always have the label cloud-ee=True.

Below are example network policies you could add.

Example 1: Allows the EE to access the 10.99.1.2 IP address on port 443.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cloud-ee-supplemental
  namespace: dai
spec:
  egress:
  - to:
    - ipBlock:
        cidr: 10.99.1.2/32
    ports:
      - port: 443
        protocol: TCP
  podSelector:
    matchLabels:
      cloud-ee: "True"
  policyTypes:
  - Egress

Example 2: Allows the SUT to access a pod in the test-app namespace which has the label app=test-app, on port 443.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cloud-sut-supplemental
  namespace: dai
spec:
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: test-app
      podSelector:
        matchLabels:
          app: test-app
    ports:
      - port: 443
        protocol: TCP
  podSelector:
    matchLabels:
      cloud-sut: "True"
  policyTypes:
  - Egress

Cleanup

Under normal operation SUTs and EEs created by Hosted Execution are cleaned up immediately after use. That is on completion of a test run, for tests triggered from the DAI runner, and when the user disconnects the SUT for Design mode SUTs requested from the Desktop app.

Occasionally, this process can be interrupted and leave behind SUTs and EEs that would be using unnecessary cluster resources. As a fallback, the Hosted Execution provisioner runs a periodic cleanup job to remove SUT and EEs after a defined period, even if no other instruction to remove them has been received.

The table below shows helm values related to this:

Key	Description	Default
hosted-execution.provisioner.cleanup.cronExpression	A standard cron expression that dictates how often the cleanup runs.	0,15,30,45 * * * *
hosted-execution.provisioner.cleanup.eeMaxHoursBeforeCleanup	The number of hours an EE will be left running before it is cleaned up.	12
hosted-execution.provisioner.cleanup.runSutMaxHoursBeforeCleanup	The number of hours a Run mode SUT will be left running before it is cleaned up.	12
hosted-execution.provisioner.cleanup.designSutMaxHoursBeforeCleanup	The number of hours a Design Mode SUT will be left running before it is cleaned up.	8