diff --git a/content/kubevirt/livemigration/cclm.md b/content/kubevirt/livemigration/cclm.md new file mode 100644 index 00000000..1fb385cc --- /dev/null +++ b/content/kubevirt/livemigration/cclm.md @@ -0,0 +1,283 @@ +--- +title: Cross Cluster Live Migration +linktitle: Cross Cluster Live Migration +description: Cross Cluster Live Migration +tags: ['cnv','kubevirt','ocp-v','v4.12'] +--- +# Cross cluster live migration + +Official documentation: [12.5. Configuring a cross-cluster live migration network](https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/virtualization/live-migration#virt-configuring-cross-cluster-live-migration-network) + +Tested with: + +|Component|Version| +|---|---| +|OpenShift|v4.20.4| +|OpenShift Virt|v4.20.1| +|MTV|v2.10.0| + +Without ACM, just a pure cross cluster live migration with two OpenShift clusters. + +## Cluster overview + +We have two identicial clusters in terms of + +* OpenShift Version +* CPU Type and Model + +Cluster one called OCP1 is the target cluster with mtv. +Cluster two called OCP7 is the source cluster. + +This cluster are running on bare OpenShift Cluster called ISAR. + +![](cclm/overview.drawio) + +### Details about the OCP1 & OCP7 adjustments at ISAR + +OCP1 and OCP7 are provided via our [stormshift automation](https://github.com/stormshift/automation) + +??? quote "OCP1 & OCP7 Infrastructure details" + + #### Patch the cpu model + + === "Command" + + ```shell + oc get vm -o name | xargs oc patch --type=merge -p '{"spec":{"template":{"spec":{"domain":{"cpu":{"model":"Haswell-v4"}}}}}}' + ``` + + #### Enable VT-X/vmx feature + + === "Command" + + ```shell + oc get vm -o name | xargs oc patch --type=merge -p '{"spec":{"template":{"spec":{"domain":{"cpu":{"features":[{"name":"vmx","policy":"require"}]}}}}}}' + ``` + + #### Restart all VM's + + === "Command" + + ```shell + oc get vm --no-headers -o custom-columns="NAME:.metadata.name" | xargs -n1 virtctl restart + ``` + + #### Check setttings as ISAR + + === "Command" + + ```shell + oc get vm -o custom-columns=NAME:.metadata.name,CPU:.spec.template.spec.domain.cpu + ``` + + === "Example output" + + ```shell + oc get vm -o custom-columns=NAME:.metadata.name,CPU:.spec.template.spec.domain.cpu + NAME CPU + ocp1-cp-0 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1] + ocp1-cp-1 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1] + ocp1-cp-2 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1] + ocp1-worker-0 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1] + ocp1-worker-1 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1] + ocp1-worker-2 map[cores:8 features:[map[name:vmx policy:require]] model:Haswell-v4 sockets:1 threads:1] + ``` + + #### Add second interface into vlan 2001 for the VM's/nodes + + === "oc apply -f ...." + + ```bash + oc apply -n stormshift-ocp1-infra -f {{ page.canonical_url }}cclm/isar-2001-net-attach-def.yaml + oc apply -n stormshift-ocp7-infra -f {{ page.canonical_url }}cclm/isar-2001-net-attach-def.yaml + + ``` + + === "isar-2001-net-attach-def.yaml" + + ```yaml + --8<-- "content/kubevirt/livemigration/cclm/isar-2001-net-attach-def.yaml" + ``` + + #### Adjust VM's to add second interface to worker nodes: + + ![](cclm/isar-second-interface.png) + +## OCP1 and OCP7 cluster preperation + +### Install following operators + +* Nmstate Operator (instantiate the operator now) +* OpenShift Virtualization (instantiate the operator **LATER!**) +* Migration Toolkit for Virtualization (instantiate the operator **LATER!**) + +### Prepare required live migration network + +Both clusters have to be connected via an L2 network. +In my case it's vlan 2001 with `192.168.201.0/24 subnet + +Here an high level overview: + +![](cclm/live-migration-network.drawio) + +???+ bug "There is an documetion bug in the offical docs" + + + +??? example "NodeNetworkConfigurationPolicy for linux bridge into VLAN 2001" + + Apply this to OCP1 and OCP7 + + === "coe-bridge-via-enp2s0.yaml" + + ```yaml + --8<-- "content/kubevirt/livemigration/cclm/coe-bridge-via-enp2s0.yaml" + ``` + + === "oc apply -f ...." + + ```bash + oc apply -f {{ page.canonical_url }}cclm/coe-bridge-via-enp2s0.yaml + ``` + +??? example "NetworkAttachmentDefinition for OCP1 and OCP7" + + Little helper for find out the interfaces on the nodes: + + ```shell + oc get nodes -l node-role.kubernetes.io/worker -o name | while read line ; do echo "# $line";oc debug -q $line -- ip -br l | grep enp ; done + ``` + + Apply this to OCP1 + + === "ocp1.net-attach-def.yaml" + + ```yaml + --8<-- "content/kubevirt/livemigration/cclm/ocp1.net-attach-def.yaml" + ``` + + === "oc apply -f ...." + + ```bash + oc apply -f {{ page.canonical_url }}cclm/ocp1.net-attach-def.yaml + ``` + + Apply this to OCP7 + + === "ocp7.net-attach-def.yaml" + + ```yaml + --8<-- "content/kubevirt/livemigration/cclm/ocp7.net-attach-def.yaml" + ``` + + === "oc apply -f ...." + + ```bash + oc apply -f {{ page.canonical_url }}cclm/ocp7.net-attach-def.yaml + ``` + +### Instantiate the operator + +#### OpenShift Virtualization on OCP1 and OCP7 + +Instantiate with following changes: + +```yaml +spec: + liveMigrationConfig: + network: livemigration-network + featureGates: + decentralizedLiveMigration: true +``` + +Wait until `virt-synchronization-controller-xxx` pods are running: + +```shell +oc get pods -n openshift-cnv -l kubevirt.io=virt-synchronization-controller +``` + +```shell +% oc get pods -n openshift-cnv -l kubevirt.io=virt-synchronization-controller +NAME READY STATUS RESTARTS AGE +virt-synchronization-controller-5b58bd4478-5l25n 1/1 Running 0 3d21h +virt-synchronization-controller-5b58bd4478-zwpn4 1/1 Running 0 3d21h +``` + +Optional: Check the virt-handler migration network configuration: + +```shell +% oc project openshift-cnv +% oc get pods -l kubevirt.io=virt-handler -o name | while read line ; do oc exec -q $line -- /bin/sh -c 'echo -n "$HOSTNAME $NODE_NAME "; ip -4 -br a show dev migration0' ; done +virt-handler-dm5mh ocp1-worker-2 migration0@if9 UP 192.168.201.129/24 +virt-handler-h6bn9 ocp1-worker-1 migration0@if9 UP 192.168.201.131/24 +virt-handler-nndkq ocp1-worker-0 migration0@if9 UP 192.168.201.130/24 +``` + +```shell +% oc project openshift-cnv +% oc get pods -l kubevirt.io=virt-handler -o name | while read line ; do oc exec -q $line -- /bin/sh -c 'echo -n "$HOSTNAME $NODE_NAME "; ip -4 -br a show dev migration0' ; done +virt-handler-bjwvt ocp7-worker-1 migration0@if9 UP 192.168.201.3/24 +virt-handler-gl8gs ocp7-worker-0 migration0@if9 UP 192.168.201.1/24 +virt-handler-kxg7k ocp7-worker-2 migration0@if8 UP 192.168.201.4/24 +``` + +#### Migration toolkit for Virtualization on OCP1 + +Instantiate with following change: + +```yaml +spec: + feature_ocp_live_migration: 'true' +``` + +## Migration toolkit for Virtualization + +### Create provide at OCP1 + +#### Create service account and token at OCP7 + +Create clusterrole `live-migration-role` + +```shell +oc apply -f {{ page.canonical_url }}cclm/clusterrole.yaml +``` + +```shell +oc create namespace openshift-mtv +oc create serviceaccount cclm -n openshift-mtv +oc create clusterrolebinding cclm --clusterrole=live-migration-role --serviceaccount=openshift-mtv:cclm +oc apply -f - < + +## Resources + +* +* +* +* +* diff --git a/content/kubevirt/livemigration/cclm/clusterrole.yaml b/content/kubevirt/livemigration/cclm/clusterrole.yaml new file mode 100644 index 00000000..b2fb6006 --- /dev/null +++ b/content/kubevirt/livemigration/cclm/clusterrole.yaml @@ -0,0 +1,115 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: live-migration-role +rules: + - apiGroups: + - forklift.konveyor.io + resources: + - '*' + verbs: + - get + - list + - watch + - apiGroups: + - "" + resources: + - secrets + - namespaces + - configmaps + - persistentvolumes + - persistentvolumeclaims + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - k8s.cni.cncf.io + resources: + - network-attachment-definitions + verbs: + - get + - list + - watch + - apiGroups: + - storage.k8s.io + resources: + - storageclasses + verbs: + - get + - list + - watch + - apiGroups: + - kubevirt.io + resources: + - virtualmachines + - virtualmachines/finalizers + - virtualmachineinstancemigrations + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - kubevirt.io + resources: + - kubevirts + - virtualmachineinstances + verbs: + - get + - list + - watch + - apiGroups: + - cdi.kubevirt.io + resources: + - datavolumes + - datavolumes/finalizers + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - apps + resources: + - deployments + verbs: + - get + - list + - watch + - create + - update + - patch + - delete + - apiGroups: + - instancetype.kubevirt.io + resources: + - virtualmachineclusterpreferences + - virtualmachineclusterinstancetypes + verbs: + - get + - list + - watch + - apiGroups: + - instancetype.kubevirt.io + resources: + - virtualmachinepreferences + - virtualmachineinstancetypes + verbs: + - get + - list + - watch + - create + - update + - patch + - delete diff --git a/content/kubevirt/livemigration/cclm/coe-bridge-via-enp2s0.yaml b/content/kubevirt/livemigration/cclm/coe-bridge-via-enp2s0.yaml new file mode 100644 index 00000000..ab42c9be --- /dev/null +++ b/content/kubevirt/livemigration/cclm/coe-bridge-via-enp2s0.yaml @@ -0,0 +1,21 @@ +apiVersion: nmstate.io/v1 +kind: NodeNetworkConfigurationPolicy +metadata: + name: br-2001-via-enp2s0 +spec: + desiredState: + interfaces: + - bridge: + options: + stp: + enabled: false + port: + - name: enp2s0 + description: Linux Brige info COE Network via enp2s0 + ipv4: + enabled: false + name: br-2001 + state: up + type: linux-bridge + nodeSelector: + node-role.kubernetes.io/worker: "" diff --git a/content/kubevirt/livemigration/cclm/create-mtv-provider.png b/content/kubevirt/livemigration/cclm/create-mtv-provider.png new file mode 100644 index 00000000..af862512 Binary files /dev/null and b/content/kubevirt/livemigration/cclm/create-mtv-provider.png differ diff --git a/content/kubevirt/livemigration/cclm/isar-2001-net-attach-def.yaml b/content/kubevirt/livemigration/cclm/isar-2001-net-attach-def.yaml new file mode 100644 index 00000000..a10224c6 --- /dev/null +++ b/content/kubevirt/livemigration/cclm/isar-2001-net-attach-def.yaml @@ -0,0 +1,18 @@ +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + annotations: + k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/coe-bridge + name: coe-bridge-2001 +spec: + config: | + { + "name": "coe-bridge", + "type": "bridge", + "cniVersion": "0.3.1", + "bridge": "coe-bridge", + "macspoofchk": false, + "ipam": {}, + "vlan": 2001, + "preserveDefaultVlan": false + } diff --git a/content/kubevirt/livemigration/cclm/isar-second-interface.png b/content/kubevirt/livemigration/cclm/isar-second-interface.png new file mode 100644 index 00000000..96d773d1 Binary files /dev/null and b/content/kubevirt/livemigration/cclm/isar-second-interface.png differ diff --git a/content/kubevirt/livemigration/cclm/live-migration-network.drawio b/content/kubevirt/livemigration/cclm/live-migration-network.drawio new file mode 100644 index 00000000..87ac7965 --- /dev/null +++ b/content/kubevirt/livemigration/cclm/live-migration-network.drawio @@ -0,0 +1,67 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/kubevirt/livemigration/cclm/localnet-net-attach-def.yaml b/content/kubevirt/livemigration/cclm/localnet-net-attach-def.yaml new file mode 100644 index 00000000..ee6c2e23 --- /dev/null +++ b/content/kubevirt/livemigration/cclm/localnet-net-attach-def.yaml @@ -0,0 +1,17 @@ +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + annotations: + k8s.ovn.org/network-id: "1" + k8s.ovn.org/network-name: localnet-via-br-ex + name: localnet-via-br-ex + namespace: default +spec: + config: |- + { + "cniVersion": "0.4.0", + "name": "localnet-via-br-ex", + "type": "ovn-k8s-cni-overlay", + "netAttachDefName": "default/localnet-via-br-ex", + "topology": "localnet" + } diff --git a/content/kubevirt/livemigration/cclm/localnet-via-br-ex.yaml b/content/kubevirt/livemigration/cclm/localnet-via-br-ex.yaml new file mode 100644 index 00000000..054a4111 --- /dev/null +++ b/content/kubevirt/livemigration/cclm/localnet-via-br-ex.yaml @@ -0,0 +1,13 @@ +apiVersion: nmstate.io/v1 +kind: NodeNetworkConfigurationPolicy +metadata: + name: localnet-via-br-ex +spec: + desiredState: + ovn: + bridge-mappings: + - bridge: br-ex + localnet: localnet-via-br-ex + state: present + nodeSelector: + node-role.kubernetes.io/worker: "" diff --git a/content/kubevirt/livemigration/cclm/ocp1.net-attach-def.yaml b/content/kubevirt/livemigration/cclm/ocp1.net-attach-def.yaml new file mode 100644 index 00000000..3c500296 --- /dev/null +++ b/content/kubevirt/livemigration/cclm/ocp1.net-attach-def.yaml @@ -0,0 +1,21 @@ +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: livemigration-network + namespace: openshift-cnv +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "migration-bridge", + "type": "macvlan", + "bridge": "br-2001", + "mode": "bridge", + "ipam": { + "type": "whereabouts", + "range": "192.168.201.0/24", + "exclude": [ + "192.168.201.0/25" + ] + } + } diff --git a/content/kubevirt/livemigration/cclm/ocp7.net-attach-def.yaml b/content/kubevirt/livemigration/cclm/ocp7.net-attach-def.yaml new file mode 100644 index 00000000..5eb15972 --- /dev/null +++ b/content/kubevirt/livemigration/cclm/ocp7.net-attach-def.yaml @@ -0,0 +1,21 @@ +apiVersion: k8s.cni.cncf.io/v1 +kind: NetworkAttachmentDefinition +metadata: + name: livemigration-network + namespace: openshift-cnv +spec: + config: | + { + "cniVersion": "0.3.1", + "name": "migration-bridge", + "type": "macvlan", + "bridge": "br-2001", + "mode": "bridge", + "ipam": { + "type": "whereabouts", + "range": "192.168.201.0/24", + "exclude": [ + "192.168.201.128/25" + ] + } + } diff --git a/content/kubevirt/livemigration/cclm/overview.drawio b/content/kubevirt/livemigration/cclm/overview.drawio new file mode 100644 index 00000000..d51de53a --- /dev/null +++ b/content/kubevirt/livemigration/cclm/overview.drawio @@ -0,0 +1,19 @@ + + + + + + + + + + + + + + + + + + + diff --git a/content/kubevirt/livemigration/index.md b/content/kubevirt/livemigration/index.md new file mode 100644 index 00000000..040c1231 --- /dev/null +++ b/content/kubevirt/livemigration/index.md @@ -0,0 +1,24 @@ +--- +title: Live Migration +linktitle: Live Migration +description: Live Migration +tags: ['tagA','tagB','v4.17'] +--- +# Live Migration + +## Content + +{% set current_page_title = page.title %} +{% for n in navigation if n.title == current_page_title %} +{% for c in n.children if c.title != current_page_title %} +{% if c.abs_url is string %} + +- [{{ c.title }}]({{c.canonical_url}}) + +{% else %} + +- **[{{ c.title }}]({{ c.children[0].canonical_url }})** + +{% endif %} +{% endfor %} +{% endfor %} diff --git a/mkdocs.yml b/mkdocs.yml index f73d783e..8cbacf13 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -266,6 +266,9 @@ nav: - Virtualization: - kubevirt/index.md - Application Aware Quota: kubevirt/application-aware-quota/index.md + - LiveMigration: + - kubevirt/livemigration/index.md + - Cross Cluster: kubevirt/livemigration/cclm.md - PCI passthrough: kubevirt/pci-passthrough.md - Node Health Check: kubevirt/node-health-check.md - Descheduler: kubevirt/descheduler/index.md