diff --git a/_cloudproviders/9.4.0/L2-networking-management.md b/_cloudproviders/9.4.0/L2-networking-management.md new file mode 100644 index 0000000..070c6b7 --- /dev/null +++ b/_cloudproviders/9.4.0/L2-networking-management.md @@ -0,0 +1,106 @@ +--- +layout: page +title: "L2 Network Connectivity" +order: 21 +comments: true +version: + - 9.4.0 +--- + +Now that we've set up the cloud provider shell's automation, let's learn how to implement network connectivity. CloudShell supports two networking modes, L2 for VLAN-level management and L3 for subnet-level management. This article discusses layer 2 connectivity. If you're developing an L3 cloud provider, skip to the next article. + +## ApplyConnectivityChanges method + +To add support for L2 VLAN connectivity in a custom cloud provider, we need to implement the *ApplyConnectivityChanges* method. This method is used to connect the VMs in the sandbox to the network elements. + +The VLAN IDs are allocated by Quali Server, according to the settings of the VLAN service. These IDs are sent to the command as parameters. The implementation of this method needs to be able to support Access mode, Trunk mode, VLAN and VXLAN ranges, if supported by your cloud provider. Additionally, the implementation needs to support a range of VLAN IDs. + +The *ApplyConnectivityChanges* method can receive a list of actions of type _**setVlan**_ or _**removeVlan**_. The method receives an action for each connection that needs to be created or disconnected. In case of P2P connections, the method receives two requests, one for each App. + +Like other methods, this method needs to return an action result in the response per connection. If the method's execution fails, CloudShell needs to indicate the failure in the returned action or raise an exception. + +### Called when +It is run automatically when reserving the sandbox, as part of CloudShell's default sandbox setup script, and is also called in an active sandbox when a deployed App is connected or disconnected from a VLAN service or from another deployed App in a P2P connection. + +### Error handling + +If *ApplyConnectivityChanges* fails, CloudShell needs to indicate the failure in the returned action or raise an exception. + +### Signature + +{% highlight python %} +def ApplyConnectivityChanges(self, context, request): +{% endhighlight %} + +### Inputs + +**context**: *context* is a *ResourceCommandContext* object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered when creating the Cloud Provider resource in the **Inventory** dashboard +3. reservation - current reservation details +4. connectors – details of any visual connectors between the Cloud Provider App and other endppoints in the sandbox. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 8 18 %} +{% endhighlight %} + + +#### Handle Request + +**setVlan action request:** + +* The requested VLAN mode is indicated in a property called *mode* under *connectionParams*. The VLAN id array can receive many VLAN id permutations. Please refer to the online help for all support permutations. +* If the cloud provider supports VXLAN or both VLAN and VXLAN, it is required to expose a discoverable attribute on the cloud provider's **VLAN Type** call. The default value is **VLAN** and cloudshell permits VLAN ids in the 2-4096 range. If the **VLAN Type** attribute is set to **VXLAN**, CloudShell will permit VLAN ids to be allocated in the 2-16,000,000 range. +* *customAttributes* is a list of special attributes for a specific action. The *setVlan* action can get a custom attribute called **Vnic Name**. When this attribute exists, we can use it in custom logic that will allocate a specific VNIC to the App's VM. For example, the **Vnic Name** value is 1 and the cloud provider shell needs to create the VLAN connection on eth1. Code example. + +**removeVlan action request:** + +* The *actionTarget* property is an object that indicates on which compute resource we need to apply the *removeVlan* action request. +* The target resource might have more than one network interface so to determine on which interface to perform the *removeVlan* action, you need to find the *interface* attribute value. The *removeVlan* request contains an array of all connector attributes. The unique identifier of the network interface to disconnect is the value of the *Interface* attribute. + +#### Method result + +**setVlan action result:** + +If the action is successful, you need to set *updatedInterface* property. The value of this property is set on an attribute on the relevant connector. Each connector in CloudShell has a source and a target component. Cloudshell automatically determines if the action result is for source or target of the connector and sets this value on the appropriate attribute - **Source Interface** or **Target Interface**. + +Code example + +**removeVlan action result:** + +You only need to indicate if the action is successful or not. + +Code example + +### ApplyConnectivityChanges method implementation + +The *ApplyConnectivityChanges* method should perform the following steps: + + 1. Retrieve cloud provider resource connection credentials. + + 2. Retrieve actions from request. + + 3. Handle Remove Vlan actions. + +     a. Retrieve requested interface to disconnect. + +     b. Retrieve requested VM id to disconnect. + +     c. Disconnect. + +     d. Return the result. + + 4. Handle *setVlan* actions. + +     a. Retrieve VLAN parameters (VNIC name, VLAN mode, VLAN id). + +     b. Connect. + +     c. Add a new interface id to the result. + + 5. Return the appropriate result. + + + diff --git a/_cloudproviders/9.4.0/L3-networking-management.md b/_cloudproviders/9.4.0/L3-networking-management.md new file mode 100644 index 0000000..4fdfc58 --- /dev/null +++ b/_cloudproviders/9.4.0/L3-networking-management.md @@ -0,0 +1,192 @@ +--- +layout: page +title: "L3 Network Connectivity" +order: 23 +comments: true +version: + - 9.4.0 +--- + +In this article, we'll learn how to implement L3 network connectivity. Cloudshell recognizes a cloud provider shell as an L3 networking cloud provider if the method *PrepareSandboxInfra* exists in the shell driver. If the method doesn’t exist, CloudShell considers the shell as an L2 networking shell. + +To add support for L3 subnet connectivity in a custom cloud provider, we'll need to implement 3 methods: + +* *PrepareSandboxInfra* is used to prepare the infrastructure required for a sandbox operating with L3 connectivity. +* *Deploy* should be extended to support *connetToSubnet* action requests. +* *CleanupSandboxInfra* cleans any sandbox-level entities created in the cloud provider, usualy entities created by the *PrepareSandboxInfra* command. This is the last command to be called in the orchestration flow. + +## PrepareSandboxInfra method + +The *PrepareSandboxInfra* method is the 1st method to be called in the sandbox setup. It creates the required infrastructure for the sandbox. The network address space of the sandbox is allocated automatically by CloudShell and represented using CIDR notation and provided in the actions request. + +It must support these three action types: + +_**prepareCloudInfra**_ + +When handling this action, you need to create/allocate the requested CIDR to the sandbox in your cloud provider. This is also the place to create/allocate cloud resource that will be used by the entire sandbox. For example, an AWS shell might create an S3 bucket that will be used by the entire sandbox, and a shared Security Group, which will be attached to all AWS VM instances, to allow inbound traffic from specific QualiX and Execution Server VM instances for secure connectivity to the VMs and configuration management. + +Generally this is the 1st action you should handle before proceeding to other actions. There is always one instance of this type of action. + +_**prepareSubnet**_ + +*prepareSubnet* gets one or more actions from this type. When handling this action, you need to create/allocate a subnet for each action according to the CIDR and **Subnet** service attributes in the blueprint. A subnet can be declared as public or private by the blueprint designer. A subnet is considered private if it is not accessible from outside the sandbox, and is considered public if it is possible to access the subnet from outside the sandbox. As a general best practice, it is recommended to allow outbound traffic from private subnets to elements outside of the sandbox. + +CloudShell sends a *prepareSubnet* action for each **Subnet** service in the blueprint. If there are no **Subnet** services in the blueprint, CloudShell will create a single subnet for the sandbox, by sending a *prepareSubnet* action with the default values. In a "single subnet" scenario, it is assumed that the sandbox components are all located in the same default subnet. Note that the driver still needs to create/allocate a subnet in "single subnet" mode. + +The result for this action must include a *subnetId* property. This must be an id that can be used to uniquely identify the subnet in the cloud provider. The "subnet id" will be passed as part of *connectToSubnet* actions in the *Deploy* method and should be used to identify the subnets that the App wants to connect to. + +_**createKey**_ + +There is always one instance of this type of action. This action type has no metadata. To handle this action, you return an ssh key that will be used by CloudShell to connect to Linux VMs in the sandbox. If you don’t want to use ssh keys, you can simply ignore this action without returning any *createKeys* action results. + +Cloudshell stores the ssh key securely and if the deployed App has an empty password attribute or no password attribute, it will try to authenticate with the VM using the value in the **Username** attribute and the ssh key when using QualiX to ssh to a deployed App. + +The *Deploy* method is responsible to create the VM with the sandbox ssh key that was generated in the *prepareSandboxInfra* method. Cloudshell does not pass the ssh key to the *Deploy* method and it’s the responsibility of the shell developer to store this ssh key in a secure place that is accessible during the *Deploy* method. An example for this implementation for an aws shell is to use an S3 bucket that is used only by the sandbox. + +**Note:** *PrepareSandboxInfra* can be called multiple times in a sandbox. Setup can be called multiple times in the sandbox, and every time setup is called, the *PrepareSandboxInfra* method will be called again. So your implementation needs to support this use case and take under consideration that the cloud resource might already exist. It's recommended to follow the “get or create” pattern when implementing this method. + +### Called when + +This command is called for L3 Networking type implementations in the beginning of the orchestration flow (preparation stage), +even before *Deploy* is called. + +### Error handling + +If failure occurs, return a "success false" action result. + +### Signature + +{% highlight python %} +def PrepareSandboxInfra(self, context, request, cancellation_context) +{% endhighlight %} + +### Inputs + +**context:** *context* is a *ResourceCommandContext* object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered when creating the Cloud Provider resource in the **Inventory** dashboard +3. reservation - current reservation details +4. connectors – details of any visual connectors between the Cloud Provider App and other endppoints in the sandbox. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 8 18 %} +{% endhighlight %} + +Here's a code sample that extracts the cloud provider data from the *context*: + +{% highlight python %} +cloud_provider_resource = HeavenlyCloudsShell.create_from_context(context) +{% endhighlight %} + +**Request** + +JSON string that contains the requested actions. + +**Cancelation context** + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 127 130 %} +{% endhighlight %} + +### PrepareSandboxInfra method implementation + + 1. Retrieve the cloud provider resource connection credentials. + + 2. Retrieve requested actions (1 PrepareCloudInfra, 1 CreateKeys, >=1 PrepareSubnet). + +     a. Handle *PrepareCloudInfra*. + +     b. Optionally, handle *CreateKeys*. + +     c. Handle the *PrepareSubnet* action(s). + +     d. Return the action results. + + 3. Return the driver's response. + +## Deploy method with L3 connectivity + +When working with L3 connectivity, the subnets and shared sandbox resources are created in the *PrepareSandboxInfra* method. The *Deploy* method is executed only after *PrepareSandboxInfra* completes successfully. *Deploy* can receive 0 or more *connectToSubnet* actions in addition to the single *deployApp* action. The *connectToSubnet* actions tell the VM that we are creating the subnets that the VM should be connected to. The **subnetId** property in the *connectToSubnet* action should be used to uniquely identify the subnet in the cloud provider. + +The *connectToSubnet* action contains a special property called **vnicName**. When this property contains a value, it means that the blueprint designer specified a request from the cloud provider shell to create a connection on a specific network interface. For example, if the **vnicName** value is 1, we need to create the network interface for the relevant subnet from the action on eth1. + +**Note:** In case the shell supports ssh key authentications for linux vms it’s the responsibility of the deploy method implementation to get the ssh key created during prepareSandboxInfra. Cloudshell will not send the ssh key to the deploy method in the request. + +### L3 Deploy method implementation + +In L3, the *Deploy* method should perform the following steps: + + 1. Retrieve the cloud provider resource connection credentials. + + 2. Retrieve the *Deploy* action. + + 3. Retrieve *CononectToSubnetActions*. + + 4. Call the *Deploy* logic of the selected deployment type. + +     a. (Steps 4-a to -4-e are performed by the *Deploy* logic) Generate a unique App name. + +     b. Create the network configuration plan to use when calling the cloud provider SDK VM instance deploy. + +     c. Create the VM instance using the deployment path attributes and network plan. + +     d. If *Deploy* succeeds, do the following: + +          i. Collect the VM's details (operating system, specifications, networking information). + +          ii. Create *ConnectToSubnet* results. + +     e. Return results. + + 5. Return the driver's response. + +## CleanupSandboxInfra + +This method is responsible for cleaning any sandbox-level entities created in the cloud provider, usually entities created in the *PrepareSandboxInfra* command. + +### Called When + +This method is the last method to be called during the sandbox's teardown (before the sandbox is completed). + +### Error handling + +If a failure occurs, return a "success false" action result. + +### Signature + +{% highlight python %} +def CleanupSandboxInfra(self, context, request): +{% endhighlight %} + +### Inputs + +**context:** *context* is a *ResourceCommandContext* object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered when creating the Cloud Provider resource in the **Inventory** dashboard +3. reservation - current reservation details +4. connectors – details of any visual connectors between the Cloud Provider App and other endppoints in the sandbox. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 8 18 %} +{% endhighlight %} + +**Request** + +JSON string contains single *CleanupSandboxInfra* action. + +### CleanupSandboxInfra method implementation + + 1. Retrieve the cloud provider resource connection credentials. + + 2. Retrieve the *CleanupSandboxInfra* action. + +     * Clean the sandbx-created infrastructure resources and return a success result. + + 3. Return result. + + diff --git a/_cloudproviders/9.4.0/app-deployment.md b/_cloudproviders/9.4.0/app-deployment.md new file mode 100644 index 0000000..139e515 --- /dev/null +++ b/_cloudproviders/9.4.0/app-deployment.md @@ -0,0 +1,459 @@ +--- +layout: page +title: "App Deployment" +order: 17 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we'll learn how to implement the App's deployment. + +To deploy an App successfully, you need to implement the following 4 methods: + +1. [Deploy](#Deploy) creates the App's VM instance. +2. [PowerOn](#PowerOn) spins up the VM. +3. [remote_refresh_ip](#RemoteRefreshIp) updates the deployed App's IP address. +4. [GetVmDetails](#GetVmDetails) gets information about the VM itself, its operating system, specifications and networking information. + +These methods are executed in the above order during the deployment of an App in the sandbox (either automatically as part of the default sandbox setup script that runs when reserving a sandbox or manually by the user after adding an App to an active sandbox). Once the App is deployed, these methods can be run as individual commands from the deployed App's commands pane, with the exception of the *Deploy* command which is no longer needed once the App is deployed.. + +## Deploy method + +Creates the App's VM instance. + +### Signature + +The *deploy* method accepts three inputs: *context*, *request*, and *cancellation_context*. + +{% github_sample_ref /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/865f356f4aec14e170cd9e5f30b575c48f2dc865/src/driver.py %} +{% highlight python %} +{% github_sample /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/865f356f4aec14e170cd9e5f30b575c48f2dc865/src/driver.py 75 %} +{% endhighlight %} + +### Inputs + +**context** + +**context:** *context* is a *ResourceCommandContext* object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered when creating the Cloud Provider resource in the **Inventory** dashboard +3. reservation - current reservation details +4. connectors – details of any visual connectors between the Cloud Provider App and other endpoints in the sandbox + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 8 18 %} +{% endhighlight %} + +**Request** + +*Request* object that contains the resource deployment path and deployed App configuration (in the App template's **App Resource** page). + +**cancellation_context** + +CloudShell supports the canceling of App command executions. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 127 130 %} +{% endhighlight %} + +To allow the cancellation of a command on the Cloud Provider's Apps, we need to: + +1. Check for cancellation before each operation. If cancelled, delete cloud objects created by operation. +3. Return the appropriate result. + +Usage example + +{% github_sample_ref /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/865f356f4aec14e170cd9e5f30b575c48f2dc865/src/heavenly_cloud_service_wrapper.py %} +{% highlight python %} +{% github_sample /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/865f356f4aec14e170cd9e5f30b575c48f2dc865/src/heavenly_cloud_service_wrapper.py 15 19 %} +{% endhighlight %} + +### Output + +*DriverResponse* object that contains a list of action results. + +### Error handling + +If App deployment fails, return a "success false" action result. + +### Deploy method implementation + +The deploy method should perform the following steps: + +1. Retrieve the cloud provider resource's connection credentials. +2. Retrieve the *Deploy* action. +3. Call the *Deploy* logic of the selected deployment type. +4. *(Steps 4 - 8 are performed within the deploy logic)* Generate a unique name for the App. For example, "My-App_968e-a950". Deployed Apps are classified as resources in CloudShell and therefore must have a unique name. +5. Create a VM instance using the deployment path attributes (the HeavenlyCloud service represents your custom cloud SDK). +6. If VM deployment is successful: +* Collect VM details (operating system, specifications, networking information). +* Optionally, override App resource attribute values. For example, if we generate a unique password for each VM instance, we will also want to update this password in the Password attribute on the Deployed App Resource for future use (to allow the sandbox end-user to connect to the VM). +* If needed, add additional data to the action result. This key-value data will be available from all API resource queries. It can be useful for implementing custom logic during the lifecycle of the sandbox. Example. +7. If VM deployment fails, return a "fail" result. +8. Return *DeployAppResult*. +9. Return *DriverResponse*. + +Note that the links in the above workflow pertain to a driver of an L2 implementation. However, the only difference between L2 and L3 driver implementations is that L2 implements *ApplyConnectivityChanges* while L3 uses the *PrepareSandboxInfra* and *CleanSandboxInfra* methods. + + +### DeployAppResult JSON example +
Click here +

+ +{% prism javascript %} +{ + "actionId": "aac7cc0c-a215-4aee-8fc1-f79.4.034423", + "deployedAppAdditionalData": {}, + "deployedAppAddress": "192.168.0.5", + "deployedAppAttributes": [ + { + "attributeName": "Password", + "attributeValue": "123456" + }, + { + "attributeName": "User", + "attributeValue": "super user" + } + ], + "errorMessage": "", + "infoMessage": "", + "success": true, + "type": "DeployApp", + "vmDetailsData": { + "appName": "", + "errorMessage": "", + "vmInstanceData": [ + { + "hidden": false, + "key": "Cloud Name", + "value": "white" + }, + { + "hidden": false, + "key": "Cloud Index", + "value": "0" + }, + { + "hidden": false, + "key": "Cloud Size", + "value": "not so big" + }, + { + "hidden": false, + "key": "Instance Name", + "value": "angel vm__ca11f5" + }, + { + "hidden": true, + "key": "Hidden stuff", + "value": "something not for UI" + } + ], + "vmNetworkData": [ + { + "interfaceId": 0, + "isPredefined": false, + "isPrimary": true, + "networkData": [ + { + "hidden": false, + "key": "MaxSpeed", + "value": "1KB" + }, + { + "hidden": false, + "key": "Network Type", + "value": "Ethernet" + } + ], + "networkId": 0, + "privateIpAddress": "10.0.0.0", + "publicIpAddress": "8.8.8.0" + }, + { + "interfaceId": 1, + "isPredefined": false, + "isPrimary": false, + "networkData": [ + { + "hidden": false, + "key": "MaxSpeed", + "value": "1KB" + }, + { + "hidden": false, + "key": "Network Type", + "value": "Ethernet" + } + ], + "networkId": 1, + "privateIpAddress": "10.0.0.1", + "publicIpAddress": "8.8.8.1" + } + ] + }, + "vmName": "angel vm__ca11f5", + "vmUuid": "027ad770-9ecb-4936-a7df-aeaf526dfc34" +} +} +{% endprism %} + + +

+
+ + + +#### DeployAppResult properties + +| **Name** |**Type** | **Description** | +| actionId | string | (Mandatory) The action GUID as received (deploy_app_action.actionId) result must include the action id it results for, so server can match result to action. | +|deployedAppAddress | string | (Mandatory) The primary ip address of the VM instance. This value will be set as the deployed App’s resource address in CloudShell. | +|errorMessage | string | (Optional) Error message to be displayed to the sandbox end-user if VM deployment fails. | +| infoMessage | string | (Optional) Info message to be displayed to the sandbox end-user if VM deployment succeeds. | +| success | bool | (Mandatory) | +| type | string | (Read only) *DeployApp* object type. It is automatically set in *DeployAppResult* object type (in cloudshell-cp-core). | +| vmName | string | Unique name of the resource in CloudShell. | +| vmUuid | string | Unique resource id. Populate *vmUuid* with the unique id of the resource in your custom cloud provider. Cloudshell does not use this id, but will keep it for other method calls. | +| deployedAppAdditionalData | dictionary | Container used to persist custom data in resource, similar to AWS Tags. Included in all resource API query results. For example, reading the custom data and returning it in the VM Details.| +| deployedAppAttributes | array | Contains data describing the deployed app attributes, and are displayed in the App's **Attributes** pane in the sandbox. It should be used to change default attribute values on the deployed App resource. For example User & Password attributes exist as part of the default deployed App model. If your custom cloud provider generates a password in runtime for the VM, you should update the *deployedAppAttributes* property accordingly. | +| vmDetailsData | object | Contains vmNetworkData and vmInstanceData. Displayed in the App's VM Details pane. For details about the return data, see the [GetVmDetails method](#GetVmDetails)'s **Return value** section below. | + + + +## PowerOn method + +The *PowerOn* method spins up the VM. It is run automatically when reserving the sandbox, as part of CloudShell's default sandbox setup script, and can also be run manually by the sandbox end-user from the deployed App's commands pane. During *PowerOn*, the VM's IP address and a green live status icon are displayed on the App in sandbox. + +You don't have to implement this method if the *deploy* method has been configured to spin up the VM. +If *PowerOn* does not fail, CloudShell will set resource state to "online" once the VM is up. + +### Signature + +{% highlight python %} +def PowerOn(self, context, ports) +{% endhighlight %} + +### Inputs + +**context:** *context* is a ResourceRemoteCommandContext object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered by the user when creating the Cloud Provider resource in the **Inventory** dashboard +3. remote_reservation – reservation details +4. remote_endpoints- will contain a single ResourceContextDetails object which provides data for the operation. + +**Ports** + +Legacy argument. Obsolete for custom cloud providers. + +### PowerOn method implementation + +The *PowerOn* method should perform the following steps: + +1) Retrieve the cloud provider resource's connection credentials. + +2) Convert the *deployed_app_json* context from string to object. + +     The json contains information about the CloudShell server, the deployed App and reservation. + +     For details, copy the json contents into your preferred JSON editor. For example: + + ![PowerOn JSON]({{site.baseurl}}/assets/PowerOn-json-example.png) + +3) Power on the deployed App resource. + +### PowerOn implementation example + +{% github_sample_ref /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/ea88bab874c42d69508ba5b2542e867c5b375d5f/src/driver.py %} +{% highlight python %} +{% github_sample /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/ea88bab874c42d69508ba5b2542e867c5b375d5f/src/driver.py 113 128 %} +{% endhighlight %} + +### Return value + +None + +### Error handling + +In case of an error, the command should raise an exception. + + + +## remote_refresh_ip + +The *remote_refresh_ip* method retrieves the VM's updated IP address from the cloud provider and sets it on the deployed App resource. The IP of the main network interface also needs to be retrieved from the cloud provider. Both private and public IPs are retrieved, as appropriate. + +*remote_refresh_ip* is run automatically during the sandbox's setup, after the VM is created and connected to networks, and can also be run manually by the sandbox end-user by running the **Refresh IP** command in the sandbox. + +**Note:** This method is mandatory. However, you can choose to disable the call to this method during setup using the **Wait for IP** attribute. For details, see [Controlling App Deployment Orchestration]({{site.baseurl}}/cloudproviders/{{pageVersion}}/controlling-app-deployment-orchestration.html). + +### Signature + +{% highlight python %} +def remote_refresh_ip(self, context, ports, cancellation_context): +{% endhighlight %} + +### Inputs + +**context**: *context* is a ResourceRemoteCommandContext object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered when creating the Cloud Provider resource in the **Inventory** dashboard +3. remote_reservation – reservation details +4. remote_endpoints - will contain a single ResourceContextDetails object which provides data for the operation. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 171 180 %} +{% endhighlight %} + +**Ports** + +Legacy argument. Obsolete for custom cloud providers. + +**Cancellation context** + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 127 130 %} +{% endhighlight %} + +### Return value + +None. + +Unlike other methods that update data using the result, *remote_refresh_ip* updates the deployed App resource with the IP by calling *cloudshell-automation-api*. However, if you implemented a return output, make sure to convert the *remote_refresh_ip* object to a string and pass the value to the "IP" attribute on the deployed App. + +### Error handling + +If the operation fails, the command should raise an exception. + +### remote_refresh_ip method implementation + +This method should perform the following steps: + +1. Retrieve the Cloud Provider resource's connection credentials. +2. Convert the *deployed_app_json* context from string to object. +3. Retrieve previously known private/public IPs (if there are any), VM instance id. +4. Verify that the deployed App's private IP is the same as the ip in the cloud provider. If it's different, update the deployed App ip with the IP on the cloud provider. +* If the IPs are different, update the deployed App IP with the IP on the cloud provider by calling *UpdateResourceAddress*. +* If the operation fails, display an error to the sandbox end-user. +5. If needed, verify that the deployed App's public IP is the same as the ip in the cloud provider. +* If the IPs are different, update the deployed App ip with the ip on the cloud provider by calling *SetAttributeValue* and setting the *Public IP* attribute. +* If the operation fails, display an error to the sandbox end-user. + + + +## GetVmDetails method + +The *GetVmDetails* method gets information about the App's VM, operating system, specifications and networking information. It is called by the default setup script when reserving the sandbox, after the *RefreshIp* method is called, and can also be run manually by the sandbox end-user on deployed Apps from the App's **VM Details** pane. + +**Note:** The implementation is expected to query the cloud provider for the details, but not return any cached or stored data. + +### Signature + +{% highlight python %} +def GetVmDetails(self, context, requests, cancellation_context): +{% endhighlight %} + +### Inputs + +**context**: *context* is a *ResourceCommandContext* object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered when creating the Cloud Provider resource in the **Inventory** dashboard +3. reservation – reservation details +4. connectors – details of any visual connectors between the Cloud Provider App and other endppoints in the sandbox + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 8 18 %} +{% endhighlight %} + +**Requests** + +JSON string that contains a list of items containing App requests and deployed App data. This method can be called for a set of VMs. + +**Cancellation request** + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 127 130 %} +{% endhighlight %} + +### Return value + +*VmDetailsData* object in a serialized JSON. + +{% github_sample_ref /QualiSystems/cloudshell-cp-core/blob/d58c094d9600b5a6232da16dada1d3a408a88ac9/package/cloudshell/cp/core/models.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-cp-core/blob/d58c094d9600b5a6232da16dada1d3a408a88ac9/package/cloudshell/cp/core/models.py 281 293 %} +{% endhighlight %} + +#### vmDetailsData properties + +*vmDetailsData* is used to describe the App's VM. All properties are optional. + +| **Name** |**Type** | **Description** | +| appName | String | The App's name. No need to assign it in the deploy operation. Must be assigned in *getVmDetails* method. | +| errorMessage | string | Indication message to be displayed to the sandbox end-user when getting the vmDetails. | +| vmNetworkData | array | Array of cloudshell-cp-core VmDetailsNetworkInterface object. Create a *vmNetworkData* object for each VM NIC you wish to associate with resource. See the VmDetailsNetworkInterface table below. | +| vmInstanceData | array | Array of cloudshell-cp-core's *VmDetailsProperty*. Contains data about the VM instance attributes. It should be used to change persist values of the VM resource. For example to persist Storage and operating system data. See the VmDetailsProperty table below. | + +#### VmDetailsNetworkInterface + +| **Name** |**Type** | **Description** | +| interfaceId | String | The network interface id with which the address is associated. | +| networkId | string | The unique id of the network associated with the network interface. | +| isPrimary | bool | Determines if NIC is primary. Primary affects the default selected network in VmDetailsTab in cloudshell | +|isPredefined | bool | Determines if NIC is predefined. Predefined means that the network existed before the sandbox reservation. for example, a Static Management network that is not modeled in the blueprint. | +| networkData | array | Array of cloudshell-cp-core VmDetailsProperty. Contains data describing the NIC. Examples of network properties include Device Index and MAC Address. | +| privateIpAddress | string | NIC address. | +| publicIpAddress | string | The public ip associated with the NIC's private ip. + +#### VmDetailsProperty + +| **Name** |**Type** | **Description** | +| key | string | | +| value | string | | +| hidden | bool | Determines if the property is displayed to the sandbox end-user. | + +### GetVmDetails method implementation + +This method should perform the following steps: + +1. Retrieve cloud provider resource connection credentials. +2. Convert the JSON string to object +3. For each request, do the following: +* Retrieve identifiers. +* Query the cloud provider for the VM's configuration and networks. +* Populate *vmInstanceData* with the data (key-value) you wish to persist on the VM resource. For example to persist the VM's Storage & operating system data. +* Create *vmNetworkData*. *vmNetworkData* is a list of *VmDetailsNetworkInterface*, one for each VM NIC you wish to associate with the resource. +* Collect as *VmDetailsData* result. +4. Convert to JSON and return the result. + + diff --git a/_cloudproviders/9.4.0/configuring-deployment-paths.md b/_cloudproviders/9.4.0/configuring-deployment-paths.md new file mode 100644 index 0000000..63d1c35 --- /dev/null +++ b/_cloudproviders/9.4.0/configuring-deployment-paths.md @@ -0,0 +1,106 @@ +--- +layout: page +title: Configuring Deployment Paths +order: 9 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we'll learn how to set up the App's deployment paths. + +The shell's deployment paths are defined in the shell project's **Deployments** folder. The folder includes a *deployment-path.yaml* file, which represents a single deployment path. You can create multiple deployment paths for your shell by creating additional copies of the file and modifying their settings. + +Note that since Cloud Provider is a 2nd Gen shell, the deployment paths are not available in Resource Manager Client's **Resource Families** explorer. + +### Setting up the deployment type and image + +Let's start by configuring a new deployment path. + +In the **Deployments** folder, rename the *deployment-path.yaml* to the name of the deployment option. For example, "my-deployment-path.yaml". + +Open the yaml and locate the following line: + +{% highlight yaml %} +vendor.resource.MyDeploymentPath: +{% endhighlight %} + +Replace "MyDeploymentPath" with the new display name of the deployment path (spaces are supported). For example: "My Test Path". + +{% highlight yaml %} +vendor.resource.My Test Path: +{% endhighlight %} + +Install the shell on CloudShell and in the **Manage>Apps** page, create an App template. You should be able to see the new deployment option in the **Select Deployment Type** area: + +![Resource information]({{site.baseurl}}/assets/cp-new-deployment-path.png){:class="img-responsive"} + +We can also change the icon of the deployment path by placing the new image file in the **Deployments** folder and replacing *shell-icon.png* in the yaml's `artifacts:` section with the new file name. + +For example, setting image file "my-icon.png": + +{% highlight yaml %} +artifacts: + icon: + file: my-icon.png + type: tosca.artifacts.File +{% endhighlight %} + +### Adding attributes + +Next, add the required attributes. + +_**Notes:**_ +_You cannot modify attribute properties **type** and **name**, and any attributes that are associated with the shell’s family as this will affect other shells that use this family. The family attributes are listed in the Cloud Provider Shell Standard._ + +_CloudShell does not allow upgrading shells with deleted/modified attributes. Therefore, if you need to make an unsupported change to an attribute (for example, deleting an attribute or changing its type), make sure to remove the shell from CloudShell before you install the updated version._ + +Place the cursor at the end of the `derived_from:` line and press the **[Enter]** key. Type "properties:" and press **[Enter]** again. Press the **[Tab]** key and add the attribute. _**To make the attribute visible to the user in CloudShell, make sure to include the "tags: [user_input]" line.**_ For example, adding a string attribute called "My attribute": + +{% highlight yaml %} +node_types: + + vendor.resource.My Test Path: + derived_from: cloudshell.nodes.CustomDeploymentOption + properties: + My attribute: + type: string + tags: [user_input] # supported types are: string, integer, float, boolean, cloudshell.datatypes.Password +{% endhighlight %} + +The deployment path should look something like this: + +![Resource information]({{site.baseurl}}/assets/cp-deployment-path-attribute-1.png){:class="img-responsive"} + +Same as with attributes in the *shell-definition.yaml*, you can also specify additional details, like default value, description and possible values (`constraints` property). For example: + +{% highlight yaml %} +My attribute: + type: string + default: value 1 + description: "This is my my attribute." + constraints: value 1, value 2, value 3 + tags: [user_input] +{% endhighlight %} + +### Setting attributes as read only in the blueprint + +In some cases, you may want a specific deployment attribute to be unavailable for editing from the blueprint, possibly because it defines critical VM properties, like the image ID. If this is the case, you can set the attribute to only be editable by the admin in the App template. To do so, add the `editable_only_in_app_template` rule to the attribute. For example: + +{% highlight yaml %} +My attribute: + type: string + default: value 1 + description: "This is my attribute." + constraints: value 1, value 2, value 3 + tags: [user_input, editable_only_in_app_template] +{% endhighlight %} + +*Note that the `editable_only_in_app_template` rule only blocks admins from editing the attribute value in the blueprint but not in the sandbox, where the attribute is available for editing by design. Regular users cannot edit the attribute in blueprints and sandboxes.* + +The attribute will be read only in the blueprint. + + diff --git a/_cloudproviders/9.4.0/controlling-app-deployment-orchestration.md b/_cloudproviders/9.4.0/controlling-app-deployment-orchestration.md new file mode 100644 index 0000000..9f9f32a --- /dev/null +++ b/_cloudproviders/9.4.0/controlling-app-deployment-orchestration.md @@ -0,0 +1,20 @@ +--- +layout: page +title: "Controlling App Deployment Orchestration" +order: 11 +comments: true +version: + - 9.4.0 +--- + +In this article, we'll learn how to customize the App's behavior during the sandbox's out-of-the-box Setup process. This article does not apply to Teardown, which by design powers off and deletes the App VMs from the cloud provider. + +Customizing the App's orchestration is done through the use of attributes, which are included with the App. + +* **Auto Power On** powers on the VM at setup. Note that this attributes will affect the live status icon, so setting "Auto Power On = False" will result in the App not being powered on in the sandbox, and the App will have an offline live status icon. + +* **Autoload** discovers the deployed App. This includes attribute values and the VM's structure, if the VM represents a virtual device that has a resource structure like blades and ports. For example, vRouters and vSwitches. + +* **Wait for IP** determines if CloudShell will refresh the VM's IP after it is powered on. The decision depends on the cloud provider's deployment behavior. In other words, does the cloud provider refresh the IP after the VM is powered on (for example VMware vCenter), or is the IP immediately available once the VM is created (like on AWS EC2). + + diff --git a/_cloudproviders/9.4.0/creating-cloud-provider-shell.md b/_cloudproviders/9.4.0/creating-cloud-provider-shell.md new file mode 100644 index 0000000..1f71937 --- /dev/null +++ b/_cloudproviders/9.4.0/creating-cloud-provider-shell.md @@ -0,0 +1,53 @@ +--- +layout: page +title: Creating the Cloud Provider Shell +order: 3 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Before you start, make sure to set up your machine for shell development and implementations, as explained in [Getting Started with Cloud Providers]({{site.baseurl}}/cloudproviders/{{pageVersion}}/getting-started-with-cloud-providers.html). + +## Creating the cloud provider shell + +The first step is to create the Cloud Provider shell. As with other shells, this is done using the shellfoundry command-line tool to create and install the shell on CloudShell. + +Since the Cloud Provider shell only works with CloudShell 9.0 and above, make sure to associate shellfoundry with a 9.0 installation by running the `shellfoundry config` command in command-line. For example, associating shellfoundry to CloudShell Server 192.168.85.13: + +{% highlight bash %} +shellfoundry config host 192.168.85.13 +{% endhighlight %} + +Navigate to the folder that will contain the Cloud Provider shell and create the shell. For example, creating a shell called CLPShell: + +{% highlight bash %} +shellfoundry new clp-shell --template gen2/cloud-provider +{% endhighlight %} + +The shell is created in the folder. + +Note that the shell project comprises the same files and folders as any other shell, with one exception, the Cloud Provider shell also includes a **Deployments** folder, which contains the deployment paths for the Cloud Provider’s App templates. More on this in [Configuring Deployment Paths]({{site.baseurl}}/cloudproviders/{{pageVersion}}/configuring-deployment-paths.html). For details about the shell project structure, see [The Shell Project Guide]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html). + +Let's make sure the shell was imported into CloudShell. In command-line, navigate to the shell's root folder and run: + +{% highlight bash %} +shellfoundry install +{% endhighlight %} + +In CloudShell Portal, open the **Manage>Shells** page to see the Cloud Provider shell. + +## Setting the shell's metadata and image + +Now that we have created the Cloud Provider, we need to set its general details, including the Cloud Provider's author, version and image. + +* **template_name** is the Cloud Provider name that is displayed to the CloudShell administrator in CloudShell Portal’s **Shells** management page. +* **template_author** is the Cloud Provider developer’s name. By default, the template author is the author defined in the `shellfoundry config` command. +* **template_version** defines the version number of the Cloud Provider. When extending Cloud Providers, make sure you update the version number. It is also best practice to version any code/project you write according to semantic versioning, to avoid breaking changes. This also allows us to better support customers that will encounter issues with their Cloud Providers. +* **template_icon**: The Cloud Provider includes a default image. This image is displayed on the Cloud Provider’s resources in CloudShell Portal, and resides in the Cloud Provider shell’s root folder. To change the image, simply replace the image in the file. + +Install the Cloud Provider shell again. This will replace the old version of the shell with the new one. + diff --git a/_cloudproviders/9.4.0/extending-cloud-provider-shell-data-model.md b/_cloudproviders/9.4.0/extending-cloud-provider-shell-data-model.md new file mode 100644 index 0000000..f7feef6 --- /dev/null +++ b/_cloudproviders/9.4.0/extending-cloud-provider-shell-data-model.md @@ -0,0 +1,121 @@ +--- +layout: page +title: "Extending the Shell's Data Model" +order: 7 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we will learn how to add attributes to our shell. For information about modifying or deleting attributes from the shell, scroll down to the bottom of this article. + +Attributes can be added to the cloud provider shell or to the deployment path on the App template. Attributes on the shell are for general authentication/authorization purposes and for setting general configurations for the cloud provider integration, while attributes on the deployment path are typically settings that apply to the App's VM. For example, Region applies to the cloud provider shell and Image ID applies to the deployment path. We will cover deployment path attributes in the next article. + +CloudShell does not allow upgrading shells with deleted/modified attributes. Therefore, if you need to make an unsupported change to an attribute (for example, deleting an attribute or changing its type), you will need to remove the shell from CloudShell before you install the updated version. + +### Adding an attribute to the shell + +Adding attributes to the shell is done in the shell's *shell-definition.yaml* file. + +Let's start by adding the **VLAN Type** attribute from the standard. Attributes that are included on the shell's standard, like this attribute, need to be added to the `capabilities` section, under `properties`: + +{% highlight yaml %} +node_types: + vendor.resource.ClpShell: + derived_from: cloudshell.nodes.CustomCloudProvider + capabilities: + concurrent_execution: + type: cloudshell.capabilities.SupportConcurrentCommands + auto_discovery_capability: + type: cloudshell.capabilities.AutoDiscovery + properties: + VLAN Type: + type: string # supported types are: string, integer, float, boolean, cloudshell.datatypes.Password +{% endhighlight %} + +Let's see how it looks on CloudShell. Install the shell: + +{% highlight bash %} +shellfoundry install +{% endhighlight %} + +Log in to CloudShell Portal, and open the required domain to enable the Apps based on this cloud provider resource to be usable in this domain. In the **Inventory** dashboard, create a resource from the shell. + +The attribute is displayed in the resource's **Validation & Discovery** page: + +![Resource information]({{site.baseurl}}/assets/cp-discovery-attribute.png){:class="img-responsive"} + +Note that since the **VLAN Type** attribute is defined on the family, the attribute's settings (possible values in this case) are inherited from the standard itself. The attribute's name and type are required, but you can also set the attribute's default value, description, and [rules]({{site.baseurl}}/shells/{{pageVersion}}/modeling-the-shell.html#determine-the-usage-of-custom-shell-attributes) (`tags` property). For example: + +{% highlight yaml %} +properties: + VLAN Type: + type: string + default: VLAN + description: "Select the VLAN type to use - VLAN or VXLAN" + tags: [setting, configuration] # supported tags are: configuration, setting, search_filter, abstract_filter, include_in_insight, readonly_to_users, display_in_diagram, connection_attribute, read_only +{% endhighlight %} + +However, if the attribute is not included in the shell's family, you will need to set it both in the `properties:` section, and in the `capabilities:` section’s `properties`. We’ll add an attribute called “my discovery attribute”. + +First we'll add it to the `capabilities` section: + +{% highlight yaml %} +node_types: + vendor.resource.ClpShell: + derived_from: cloudshell.nodes.CustomCloudProvider + capabilities: + concurrent_execution: + type: cloudshell.capabilities.SupportConcurrentCommands + auto_discovery_capability: + type: cloudshell.capabilities.AutoDiscovery + properties: + my discovery attribute: + type: string +{% endhighlight %} + +And then we'll add it to the `properties:` section as well (note that this section is missing, so you'll need to add it directly under the `derived_from:` line: + +{% highlight yaml %} +node_types: + vendor.resource.MyCustomClp1: + derived_from: cloudshell.nodes.CustomCloudProvider + properties: + my discovery attribute: + type: string +{% endhighlight %} + +Install the shell and return to CloudShell Portal, in the **Inventory** dashboard, click the resource's more actions button and select **Discover**: + +![Resource information]({{site.baseurl}}/assets/cp-open-resource-discovery-page.png){:class="img-responsive"} + +The resource's **Validation & Discovery** page is displayed, showing the new discovery attribute we created. + +![Resource information]({{site.baseurl}}/assets/cp-discovery-attribute-2.png){:class="img-responsive"} + +You can also set additional settings. Since this attribute is not included in the family, you can also set possible values (`constraints` property). + +For example: + +{% highlight yaml %} +properties: + my discovery attribute: + type: string + default: value 3 + description: "This is my discovery attribute." + constraints: [value 1, value 2, value 3] + tags: [setting, configuration] +{% endhighlight %} + +### Modifying an attribute on a shell + +You cannot modify attribute properties **type** and **name**, and any attributes that are associated with the shell’s family as this will affect other shells that use this family. The family attributes are listed in the shell’s standard. To find the attributes defined in the shell’s standard, see the documentation page of your shell’s standard. + +### Deleting an attribute from a shell + +Deleting any of the shell's default attributes (those that come with the standard) is not supported. It is also not possible to customize a 2nd Gen shell’s data model (families and models) and its structure, which is as defined in the Cloud Provider Shell Standard. + + diff --git a/_cloudproviders/9.4.0/getting-started-with-cloud-providers.md b/_cloudproviders/9.4.0/getting-started-with-cloud-providers.md new file mode 100644 index 0000000..c1dd20d --- /dev/null +++ b/_cloudproviders/9.4.0/getting-started-with-cloud-providers.md @@ -0,0 +1,53 @@ +--- +layout: page +title: "Getting Started with Cloud Providers" +order: 1 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this chapter, we'll learn how to create a Cloud Provider shell. The goal is to demonstrate the end-to-end cycle, from generating a new shell project to implementing the cloud provider interface and automation processes, as well as testing the shell in CloudShell. + +### What is a Cloud Provider shell? + +The Cloud Provider shell standard is a project used to define a new cloud provider in CloudShell. The Cloud Provider shell is used to extend the system, allowing deployment of applications to an additional L2 or L3 cloud. The Cloud Provider Standard may also be used to implement deployment of applications as part of deployment containers. + +A Cloud Provider shell may include more than one deployment type, allowing a variety of options to base the virtual instance on (for example, selecting the image from the marketplace or loading a custom one). + + +### How is a Cloud Provider shell created? + +The basic creation and implementation process is as follows: + +1) Create a new Cloud Provider shell. + +2) Set up the Cloud Provider interface. In other words, implement the cloud provider driver that will be called by the server when interaction with the cloud provider is needed. + +3) If the shell requires the use of python dependencies, which aren't available in the public PyPi repository, add them to the local PyPi Server repository. See CloudShell Help's PyPi Server - Managing Python Driver and Script Dependencies. + +4) Debug the shell. + +5) Install the shell on CloudShell. + +6) In CloudShell, create a resource based on the new shell and make sure it works. + +*Before developing your shell, please watch the following video to determine whether you need to create a new shell or customize an existing one:* + + + +## Supported versions - CloudShell v9.0 and up + +As of version 9.0, CloudShell supports the ability to define custom cloud providers (using the `cloud_provider` shell template), as well as the out-of-the-box cloud providers VMware vCenter, AWS EC2, Microsoft Azure and OpenStack. Note that the out-of-the-box cloud provider shells cannot be modified. + +## Prerequisites + +* [Get CloudShell](http://info.quali.com/cloudshell-developer-edition-download): Download the latest CloudShell SDK and run it on your machine. +* [Python](https://www.python.org/downloads/): Make sure Python 2.7.x (latest recommended) is installed on your machine. +* **IDE/Text Editor:** Your preferred IDE editor. We recommend using PyCharm (which offers a free community edition) because of the tooling we’ve already created for that IDE, including a CloudShell developer plugin. +* **Shellfoundry:** Shellfoundry is our CLI tool that allows you to quickly and easily generate and distribute Shells. Make sure to install it on your machine. See [Installing or Updating Shellfoundry]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html#installing-or-updating-shellfoundry) for details. + + diff --git a/_cloudproviders/9.4.0/power-off-and-delete.md b/_cloudproviders/9.4.0/power-off-and-delete.md new file mode 100644 index 0000000..7c8c458 --- /dev/null +++ b/_cloudproviders/9.4.0/power-off-and-delete.md @@ -0,0 +1,111 @@ +--- +layout: page +title: "Power off and Delete VM" +order: 19 +comments: true +version: + - 9.4.0 +--- + +In this article, we'll learn how to implement the PowerOff and DeleteInstance commands, which shut down and delete the VM from the cloud provider, respectively. + +## PowerOff method + +The *PowerOff* method shuts down (or powers off) the VM instance. It is run automatically as part of the sandbox's teardown, and can also be run manually by the sandbox end-user from the deployed App's commands pane. When *PowerOff* completes, the green 'online' live status icon is replaced with a grey one on the App resource, indicating it is offline. + +**Note:** CloudShell sets the resource state to 'offline' if *PowerOff* completed successfully. + +### Signature + +{% highlight python %} +def PowerOff(self, context, ports) +{% endhighlight %} + +### Inputs + +**context**: *context* is a ResourceRemoteCommandContext object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered by the user when creating the Cloud Provider resource in the **Inventory** dashboard +3. remote_reservation – reservation details +4. remote_endpoints - will contain a single ResourceContextDetails object which provides data for the operation. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 171 180 %} +{% endhighlight %} + +**Ports** + +Legacy argument. Obsolete for custom cloud providers. + +### Error handling + +If an error occurs during the *PowerOff* operation, the command should raise an exception. + +### PowerOff method implementation + +The *PowerOff* method should perform the following steps: + +1. Retrieve the cloud provider resource's connection credentials +2. Convert context deployed_app_json string to object +3. Power off the deployed App resource + +### PowerOff implementation example + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/ea88bab874c42d69508ba5b2542e867c5b375d5f/src/driver.py 129 144 %} +{% endhighlight %} + +### Return value + +None + +## DeleteInstance method + +The *DeleteInstance* method powers off the VM, deletes the VM from the cloud provider and removes the App from the sandbox. It is run when removing the deployed App from the sandbox or during the sandbox's teardown. + +### Signature + +def DeleteInstance(self, context, ports) + +### Inputs + +**context**: *context* is a ResourceRemoteCommandContext object that contains: + +1. connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +2. resource - resource configuration settings entered by the user when creating the Cloud Provider resource in the **Inventory** dashboard +3. remote_reservation – reservation details +4. remote_endpoints- will contain a single ResourceContextDetails object which provides data for the operation. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 171 180 %} +{% endhighlight %} + +**Ports** + +Legacy argument. Obsolete for custom cloud providers. + +### DeleteInstance method implementation + +The *DeleteInstance* method should perform the following steps: + +1. Retrieve the cloud provider resource's connection credentials. +2. Convert the *deployed_app_json* context string to object. +3. Delete the VM instance from the cloud provider. + +### DeleteInstance implementation example + +{% github_sample_ref /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/ea88bab874c42d69508ba5b2542e867c5b375d5f/src/driver.py %} +{% highlight python %} +{% github_sample /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/ea88bab874c42d69508ba5b2542e867c5b375d5f/src/driver.py 148 163 %} +{% endhighlight %} + +### Return value + +None + + + diff --git a/_cloudproviders/9.4.0/removing-the-address-field.md b/_cloudproviders/9.4.0/removing-the-address-field.md new file mode 100644 index 0000000..495a782 --- /dev/null +++ b/_cloudproviders/9.4.0/removing-the-address-field.md @@ -0,0 +1,42 @@ +--- +layout: page +title: Removing the Address Field +order: 8 +comments: true +version: + - 9.4.0 +--- + +For some cloud providers, like AWS EC2 and Azure, the **Address** field is irrelevant. If this is the case, you can easily remove the field from the shell by setting the **hide_address** property in the shell-definition.yaml. In the `capabilities` section, under `properties`, uncomment the property and set it to *true*: + +{% highlight yaml %} +node_types: + vendor.resource.ClpShell: + derived_from: cloudshell.nodes.CustomCloudProvider + properties: + my discovery attribute: + type: string + capabilities: + concurrent_execution: + type: cloudshell.capabilities.SupportConcurrentCommands + auto_discovery_capability: + type: cloudshell.capabilities.AutoDiscovery + properties: + my discovery attribute: + type: string + VLAN Type: + type: string + enable_auto_discovery: + type: boolean + default: true + auto_discovery_description: + type: string + default: Describe the auto discovery + inventory_description: + type: string + default: Describe the resource shell template +# hide_address: +# type:string +# default: false +{% endhighlight %} + diff --git a/_cloudproviders/9.4.0/resource-discovery.md b/_cloudproviders/9.4.0/resource-discovery.md new file mode 100644 index 0000000..18b7f64 --- /dev/null +++ b/_cloudproviders/9.4.0/resource-discovery.md @@ -0,0 +1,62 @@ +--- +layout: page +title: "Resource Discovery" +order: 15 +comments: true +version: + - 9.4.0 +--- + +The *get_inventory* command “discovers” the resource in CloudShell, or in other words, validates the values of the cloud provider attributes that were entered by the user. It is executed when creating the resource in CloudShell, and can be manually run later on, for example, if you change some of the resource’s attribute values. + +For example, in a vCenter cloud provider, *get_inventory* would check the value provided in the **Default DataCenter** attribute +to validate that such a datacenter exists in the vCenter Server. + +In addition, this is the place to assign values to optional attributes that were not given a value by the CloudShell admin. + +### Signature + +{% highlight python %} +def get_inventory(self, context) +{% endhighlight %} + +### Inputs + +**context**: *context* is an *AutoLoadCommandContext* object that contains: + +* connectivity - CloudShell server connectivity data for authentication with CloudShell Automation API +* resource - resource configuration settings entered by the user when creating the new resource in the **Inventory** dashboard + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 133 138 %} +{% endhighlight %} + +     **Note:** The convention for specifying *context.resource.attributes* keys in the driver files(s) is: `my_shell_name.attribute_name`. + +### Return value + +The *AutoloaDetails* class that represents details discovered by the *get_inventory* function. + +{% github_sample_ref /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py %} +{% highlight python %} +{% github_sample /QualiSystems/cloudshell-shell-core/blob/36009fdec45134ae38cb9273328b7686be66e553/cloudshell/shell/core/driver_context.py 141 146 %} +{% endhighlight %} + +### Error handling +If one of the validations failed, an error indication will be displayed in CloudShell and the resource will be marked as excluded. + +### get_inventory method implementation example + +{% github_sample_ref /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/865f356f4aec14e170cd9e5f30b575c48f2dc865/src/driver.py %} +{% highlight python %} +{% github_sample /QualiSystems/Custom-L2-Cloud-Provider-Shell-Example/blob/865f356f4aec14e170cd9e5f30b575c48f2dc865/src/driver.py 42 70 %} +{% endhighlight %} + + + + + + + + diff --git a/_cloudproviders/9.4.0/the-cloud-provider-interface.md b/_cloudproviders/9.4.0/the-cloud-provider-interface.md new file mode 100644 index 0000000..fc7a1ae --- /dev/null +++ b/_cloudproviders/9.4.0/the-cloud-provider-interface.md @@ -0,0 +1,45 @@ +--- +layout: page +title: "The Cloud Provider Interface" +order: 13 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +The rest of this chapter is dedicated to implementing the cloud provider driver that will be called by the server when interaction with the cloud provider is needed. This includes setting up communication between CloudShell and the cloud provider of choice, implementing the driver commands required from the resource, such as Deploy App, Power On and Refresh IP, and setting the resource’s live status icon (for example, "online" and "offline"). + +The articles are: + +* [Resource Discovery (get_inventory)]({{site.baseurl}}/cloudproviders/{{pageVersion}}/resource-discovery.html) +* [App Deployment (Deploy)]({{site.baseurl}}/cloudproviders/{{pageVersion}}/app-deployment.html) +* [Power off and Delete VM]({{site.baseurl}}/cloudproviders/{{pageVersion}}/power-off-and-delete.html) +* [L2 Network Connectivity]({{site.baseurl}}/cloudproviders/{{pageVersion}}/L2-networking-management.html) +* [L3 Network Connectivity]({{site.baseurl}}/cloudproviders/{{pageVersion}}/L3-networking-management.html) + +*For illustration purposes, we will use a mock shell called HeavenlyCloud, which demonstrates the use and implementation of a custom cloud provider. We've created two versions of the shell, **L2HeavenlyCloudShell** for L2 clouds (download here) and **L3HeavenlyCloudShell** for L3 clouds (download here). The difference between the two is in the networking connectivity implementation.* + +To see how the HeavenlyCloud cloud provider works, let’s create a resource using the HeavenlyCloud shell in CloudShell Portal. First, download the appropriate mock shell .zip file from GitHub and extract it to your computer. + +![Shell Commands]({{ site.baseurl}}/assets/cp-heavenly-cloud-project.png) + +Install the shell on CloudShell by running this command-line from the extracted shell project folder: + +{% highlight yaml %} +shellfoundry install +{% endhighlight %} + +In CloudShell Portal, open the **Inventory** dashboard and create a resource from the **HeavenlyCloud** shell. + +In the **Manage>Apps** page, create a new App template. Note that two new deployment types have been added, HeavenlyCloudAngelDeployment and HeavenlyCloudManDeployment. Select a deployment type and carry on setting the App template. + +![Shell Commands]({{ site.baseurl}}/assets/cp-heavenly-cloud-deployment-paths.png) + +In the **Deployment Paths** page of the dialog box, make sure to select the new HeavenlyCloud cloud provider resource you created. + +Next, let’s implement the *get_inventory* function, which discovers and validates the resource against the cloud provider of choice. + + diff --git a/_cloudproviders/9.4.0/the-cloud-provider-model.md b/_cloudproviders/9.4.0/the-cloud-provider-model.md new file mode 100644 index 0000000..3d1b81d --- /dev/null +++ b/_cloudproviders/9.4.0/the-cloud-provider-model.md @@ -0,0 +1,23 @@ +--- +layout: page +title: "The Cloud Provider Model" +order: 5 +comments: true +version: + - 9.4.0 +--- + +In this article, we'll learn about the Cloud Provider shell's model. + +The Cloud Provider Standard defines two elements: the Cloud Provider shell and the Deployment Type service. The Cloud Provider shell is responsible for accessing the cloud provider and executing the App's automation commands (like deploy VM, power on, power off, refresh IP). And the deployment type service shell sets the deployment path on the App template, including the VM's definition. The service contains the attributes of the deployment type, but has no driver since the automation commands are defined in the Cloud Provider shell's driver. + +* The cloud provider model extension includes the attributes that control the behavior of the cloud provider as a whole. For example the region name and default values for elements created in the cloud. + +* The deployment type extension needs to include the attributes that are needed for every deployment method in this cloud. You can set these attributes to be visible in the **Deployment Paths** tab of the App template dialog box (we'll learn how to do this later on this chapter). + +The Cloud Provider shell is created using the **gen2/cloud-provider** shellfoundry template. The project folder includes a shell-definition.yaml file for the cloud provider model definition, and a skeleton of the cloud provider model. Inside the **Deployments** folder, there is a deployment-path.yaml file for the deployment type model. If more +than one deployment type is needed, additional yaml files may be added to this folder. + +The association between the cloud provider and its relevant deployment types is implicit. + + diff --git a/_configmanagement/9.4.0/cf-add-custom-script-to-app.md b/_configmanagement/9.4.0/cf-add-custom-script-to-app.md new file mode 100644 index 0000000..9023ca9 --- /dev/null +++ b/_configmanagement/9.4.0/cf-add-custom-script-to-app.md @@ -0,0 +1,47 @@ +--- +layout: page +title: Adding Your Script to an App +category: cf +order: 3 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Now that we have tested and debugged our script, the next step is to add it to an App template. + +1) In CloudShell Portal, open the **Manage>Apps** page. + +2) Edit or create an App. + +3) Open the **Configuration Management** tab. + +4) From the **Select Tool** drop-down list, select **Script**. + +5) Select the appropriate **Connection Method**, depending on the VM’s operating system. Select **Windows Remote Management** for Windows machines, or **SSH** for Linux. + +6) Specify the script’s **URL**. Make sure you specify the raw version of the URL. + +     If the selected **Connection Method** is WinRM, the script file should have a .ps1 extension. Similarly, if **SSH** is selected, the script file extension can be either .sh or .bash. + +If the URL is protected by Basic Authentication, specify the **Username** and **Password**. + +7) To pass parameters to the script (as environment variables), click **Add Parameter** and enter each parameter’s name and value. + +You can add parameters to the App template in the following ways: +* Provide the value as part of the App template, making it the default value for every instance of this App template +* Specify a static value in the App in the blueprint +* Specify a dynamic value in the App in the blueprint, linking the parameter to one of the blueprint's Global Inputs. To do so, just enter the global input’s name in curly brackets as the parameter value, or click the “plus” button to select an available Global Input. +* Pass a value using the API, as illustrated in this [example]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-custom-scripts.html#CustomScriptParams). This will replace any value provided in the App template or in the blueprint + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-custom-script-Configuration-Management-page.png){:class="img-responsive"} + +8) Open the **App Resource** tab, and enter the VM's access credentials. + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-custom-script-App-Resource-page.png){:class="img-responsive"} + +9) Click **Done**. + diff --git a/_configmanagement/9.4.0/cf-add-playbook-to-app.md b/_configmanagement/9.4.0/cf-add-playbook-to-app.md new file mode 100644 index 0000000..405d315 --- /dev/null +++ b/_configmanagement/9.4.0/cf-add-playbook-to-app.md @@ -0,0 +1,49 @@ +--- +layout: page +title: Adding the Playbook to an App +category: cf +order: 7 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Now that we have tested and debugged our playbook, the next step is to add it to an App template. + +1) In CloudShell Portal, open the **Manage>Apps** page. + +2) Edit or create an App. + +3) Open the **Configuration Management** tab. + +4) From the **Select Tool** drop-down list, select **Ansible**. + +5) Select the **Connection** Method, depending on the Vm’s operating system. Select **Windows Remote Management** for Windows machines, or **SSH** for Linux. + +6) Specify the playbook’s **URL**. Make sure you specify the raw version of the URL. + +     The playbook can be a yml file for a single playbook or a zip file containing several playbooks. For simple tasks, a single yml file should be enough, but for more complex configuration logic, and for using roles, a zip file may be required (for a sample zip file, click [here](https://github.com/QualiSystems/app-starter-pack/blob/dev/Playbooks/wordpress-rhel7.zip?raw=true)). The zip file must contain a least one yml (if there are several yml files, the main one must be named *site.yml*). In order to use roles, include in the zip a folder named *roles*, and add the roles to it. + +     If the URL is protected, specify the **Username** and **Password**. + +7) In the **Inventory Groups** field, specify the inventory groups, separated by semicolons “;”. For details, see the [Inventory Groups]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-ansible-examples.html#InventoryGroups) example. + +8) To add parameters to the playbook, click **Add Parameter**, and enter each parameter’s name and value. Note that the parameters are added to the App template. + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-Configuration-Management-page.png){:class="img-responsive"} + +You can add parameters to the App template in the following ways: +* Provide the value as part of the App template, making it the default value for every instance of this App template +* Specify a static value in the App in the blueprint +* Specify a dynamic value in the App in the blueprint, linking the parameter to one of the blueprint's Global Inputs. To do so, just enter the global input’s name in curly brackets as the parameter value, or click the “plus” button to select an available Global Input. +* Pass a value using the API, as illustrated in this [example]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-ansible-examples.html#ConfigureApps). This will replace any value provided in the App template or in the blueprint. + +9) Open the **App Resource** tab, and enter the VM's access credentials. + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-custom-script-App-Resource-page.png){:class="img-responsive"} + +10) Click **Done**. + diff --git a/_configmanagement/9.4.0/cf-ansible-examples.md b/_configmanagement/9.4.0/cf-ansible-examples.md new file mode 100644 index 0000000..a8f471d --- /dev/null +++ b/_configmanagement/9.4.0/cf-ansible-examples.md @@ -0,0 +1,121 @@ +--- +layout: page +title: Ansible Playbook Examples +category: cf +order: 6 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +### Examples +We’ve put together some basic playbook examples to help you get started. If you are new to Ansible development, feel free to test them out. + +#### Hello World +A basic playbook that prints “Hello World”, just to make sure we’re communicating with the VM, and are able to run the playbook. When run as part of an App’s deployment in CloudShell, the message will be displayed in the sandbox diagram's **Output** window. + +For example, the *site.yml* file may look something like this: + +{% highlight bash %} + +--- +- hosts: all + tasks: + - name: Print Hello World + debug: msg="Hello World" +{% endhighlight %} + + + +#### Parameters +A playbook that prints the parameter defined in an App template or API call (see below). Such playbooks are useful for debugging and making sure parameter variables are set with the correct values. Note that the parameters are stored as environment variables on the App instance in the sandbox. + +The *site.yml* file will look something like this: + +{% assign special = "{{P1|default('No Message')}}" %} +{% highlight bash %} +--- +- hosts: all + vars: + - msg: "{{special}}" + tasks: + - name: Print P1 + debug: var=msg +{% endhighlight %} + +* Parameter defined in the App template: + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-params.png){:class="img-responsive"} + +* Parameter defined in the `ConfigureApps` API method (python file): + +{% highlight bash %} +Python +from cloudshell.api.cloudshell_api import * + +session = CloudShellAPISession('localhost', 'admin', 'admin', 'Global') +session.ConfigureApps( + reservationId='dfb2df41-334e-4630-8bc6-ec846eb072d6', + appConfigurations=[AppConfiguration('LinuxVmApp_9cb2-72d6', [ConfigParam('P1', 'Hello World From Here!')])], + printOutput=True +) +{% endhighlight %} + +**Note:** Since this script uses parameters defined on the App template, it will not work when run manually outside of CloudShell. For this to work, you will need to edit the python code as follows: +* Update the CloudShell settings passed to the `session` variable, if needed. +* Specify the ID of an active sandbox (in the `reservationId` field of the python code). +* Replace 'LinuxVmApp_9cb2-72d6' with the App’s name. + +#### Inventory Groups +In some cases, a playbook contains plays that target many different VMs that require configuration. In order to have the playbook run only the plays that are relevant to a specific VM, or to a group of hosts (VMs) that your VM belongs to, you can use the **Inventory Groups** field in the App template. +In this field, specify the groups that your VM belongs to (more than one group can be provided). Use semicolons ";" to separate multiple groups. + +For example , specifying groups "servers/http" and servers/sql": + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-inventory-groups.png){:class="img-responsive"} + +This example shows how these groups entered in the App’s **Configuration Management** page should be written in the Ansible *hosts* file. + +{% highlight bash %} +hosts +[servers:children] +http +sql + +[http] +192.168.1.2 + +[sql] +192.168.1.12 +{% endhighlight %} + +#### Privilege escalation +Now let’s say you want Ansible to dynamically get the user and password from the App when running a task on a particular App's VM. This is useful if the VM credentials on the App are not strong enough for the task you wish to run. To achieve this, you will need to add the following two parameters to your App’s **Configuration Management** tab: +* **ansible_become_user** - the user with the stronger permissions (probably a root user) +* **ansible_become_pass** - the password to that user + +For example: + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-dynamic-creds.png){:class="img-responsive"} + +And modify the *site.yml* file to get the relevant VM’s user and password from these parameters for each task: + +{% assign special = "{{ cmd_var.stdout }}" %} +{% highlight bash %} +--- +- hosts: all + tasks: + - name: Get sudoers file content + become: yes + become_method: su + command: cat /etc/sudoers + register: cmd_var + + - name: Print output + debug: msg= "{{ special }}" + +{% endhighlight %} + diff --git a/_configmanagement/9.4.0/cf-ansible-tips-and-tricks.md b/_configmanagement/9.4.0/cf-ansible-tips-and-tricks.md new file mode 100644 index 0000000..d098616 --- /dev/null +++ b/_configmanagement/9.4.0/cf-ansible-tips-and-tricks.md @@ -0,0 +1,21 @@ +--- +layout: page +title: Tips and Tricks for Ansible Playbooks +category: cf +order: 8 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Please take the following under consideration when developing your playbook: + +* Make sure the Execution Server can access the script’s raw URL. +* When setting the VM user’s credentials in the App, make sure you provide the credentials of a user that has the necessary permissions to successfully execute all the playbook's tasks. +* If the App’s VM takes a long time to fully load, you may want to adjust the maximum time for the machine to respond, by setting the **Timeout Minutes** attribute on the **Custom Script Configuration** resource model (in Resource Manager Client’s **Resource Families** explorer). +* Use the **Ansible Additional Arguments** attribute to specify more parameters that should run along with the `ansible-playbook` command (e.g. `-vvv` for easier debugging). The attribute resides in Resource Manager Client>**Resource Families** explorer>**Configuration Services**. For details about supported arguments, see the official Ansible documentation. +* To use an Ansible playbook on a Windows VM, make sure WinRM is configured and loaded automatically to allow the playbook to communicate with that VM. A script for this is provided in CloudShell’s online help. + diff --git a/_configmanagement/9.4.0/cf-ansible.md b/_configmanagement/9.4.0/cf-ansible.md new file mode 100644 index 0000000..675ce39 --- /dev/null +++ b/_configmanagement/9.4.0/cf-ansible.md @@ -0,0 +1,56 @@ +--- +layout: page +title: Ansible +category: cf +order: 5 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +This article will take you through the development process of an Ansible playbook for CloudShell App deployment. + +### Prerequisites +* Execution Server installed on Linux machine, with the `ansible` attribute enabled. +* Ansible 2.2 installed (not provided – should be installed manually by the user). + +This can be easily done by deploying the Linux Virtual Appliance, which installs the Execution Server and Ansible on a Linux machine. + +#### Ansible playbooks for Apps +Before running the playbook in an App, it is recommended to test it manually. To do that, follow these steps: + +1) On a Linux machine, access the `/etc/ansible/ansible.cfg` file and uncomment the line: + +{% highlight bash %} +host_key_checking = False +{% endhighlight %} + +2) Create a folder that will be the root of your test. This folder will contain the Ansible playbook as well as any additional folders and files required by the playbook. + +3) Add an inventory file with hosts and their groups (optional) for testing. + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-hosts-file.png){:class="img-responsive"} + +4) Add one or more playbook files. + +     **Note:** If you plan on using several playbook files in the App, make sure the main one is named *site.yml*. + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-site-yml-file.png){:class="img-responsive"} + +5) [Optional] Add a “roles” folder. + + ![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-add-roles-folder.png){:class="img-responsive"} + +6) [Optional] And populate it with the desired roles. + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-roles-definitions-folder.png){:class="img-responsive"} + +6) Test the playbook by running `ansible-playbook –i `. For some sample playbooks, see [Ansible Playbook Examples]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-ansible-examples.html). + +![Discovery Dialog]({{ site.baseurl}}/assets/cf-ansible-run-playbook.png){:class="img-responsive"} + +7) Once you are done developing your playbook, zip the playbook files along with the *roles* folder, upload it to a repository, and set the URL in the App template, as explained in [Adding the Playbook to an App]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-add-playbook-to-app.html). + diff --git a/_configmanagement/9.4.0/cf-custom-scripts-tips-and-tricks.md b/_configmanagement/9.4.0/cf-custom-scripts-tips-and-tricks.md new file mode 100644 index 0000000..6c6e2ae --- /dev/null +++ b/_configmanagement/9.4.0/cf-custom-scripts-tips-and-tricks.md @@ -0,0 +1,21 @@ +--- +layout: page +title: Tips and Tricks for Custom Scripts +category: cf +order: 4 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Please take the following under consideration when developing your script: + +* Make sure the Execution Server can access the script’s raw URL. +* If the VM is loaded with a user that does not have the necessary permissions but you have a stronger user, pass that user to the App as a parameter to use it in the script as an environment variable (see the [Parameters]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-custom-scripts.html#CustomScriptParams) example). +* If the App’s VM takes a long time to fully load, you may want to adjust the maximum time for the machine to respond by setting the **Timeout Minutes** attribute on the **Custom Script Configuration** resource model (in Resource Manager Client’s **Resource Families** explorer). +* To use a bash or sh script on a Windows VM, make sure WinRM is configured and loaded automatically to allow the custom script to communicate with that VM. A script for this is provided in CloudShell’s online help. + + diff --git a/_configmanagement/9.4.0/cf-custom-scripts.md b/_configmanagement/9.4.0/cf-custom-scripts.md new file mode 100644 index 0000000..92371f0 --- /dev/null +++ b/_configmanagement/9.4.0/cf-custom-scripts.md @@ -0,0 +1,76 @@ +--- +layout: page +title: Custom Scripts +category: cf +order: 2 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +This article will take you through the development process of a custom script for CloudShell App deployment. CloudShell supports PowerShell scripts for Windows VMs, and bash or sh scripts for Linux VMs. When developing your script, we recommend you first simulate/debug it by running it manually on a similar machine. + +### Examples + +Here are several examples of custom scripts to help you get started. If you are new to script development, feel free to test them out. + +#### Hello World + +A basic script, just to make sure we’re communicating with the VM and are able to run a script. When run with an App, the output should be shown in the sandbox diagram's **Output** window. + +{% highlight bash %} +Windows / Linux +echo “Hello World“ +{% endhighlight %} + +#### Parameters + +A basic script that prints out the parameter defined in an App template or API call (see below). This is useful for debugging the script and making sure the parameter is received with the correct value. Note that the parameters are stored as environment variables on the App instance in the sandbox. + +{% highlight bash %} +Windows +echo $env:P1 +Linux +echo $P1 +{% endhighlight %} + +* Specified in the App template: +![Discovery Dialog]({{ site.baseurl}}/assets/cf-custom-script-params.png){:class="img-responsive"} + +* Specified in the ConfigureApps API method: + +{% highlight bash %} +Python +from cloudshell.api.cloudshell_api import * + +session = CloudShellAPISession('localhost', 'admin', 'admin', 'Global') +session.ConfigureApps( + reservationId='dfb2df41-334e-4630-8bc6-ec846eb072d6', + appConfigurations=[AppConfiguration('LinuxVmApp_9cb2-72d6', [ConfigParam('P1', 'Hello World From Here!')])], + printOutput=True +) +{% endhighlight %} + +**Note:** Since this script uses parameters defined on the App template, it will not work when run manually outside of CloudShell. For this to work, you will need to edit the python code as follows: +* Update the CloudShell settings passed to the `session` variable, if needed. +* Specify the ID of an active sandbox (in the `reservationId` field of the python code). +* Replace 'LinuxVmApp_9cb2-72d6' with the App’s name. + +#### Linux privilege escalation + +If the deployed App resource is set with the credentials of a “regular” user, you may specify the root credentials in the parameters (see the Parameters example above), to use them in the script. + +{% highlight bash %} +Linux +echo $pass | su $user –c ‘ls /root’ +or +Linux +su - $user <CloudShell API Guide. + +### CloudShell Sandbox API + +**CloudShell Sandbox API** is a RESTful API that allows you to use CloudShell sandboxes as part of your CI/CD process. For example, you can start, extend and stop sandboxes, run sandbox orchestration and automation commands, and get information about your sandboxes and execution activity. For additional information, see the CloudShell API Guide's CloudShell Sandbox API Overview. + +#### Examples + +For implementation examples, visit our TeamCity or Jenkins plugin documentation, which leverages CloudShell Sandbox API to automate the consumption of sandboxes as part of the devops cycle. + +### CloudShell Automation API + +**CloudShell Automation API** is a Python open source package you can use to develop CloudShell orchestration and automation capabilities. Using the CloudShell Automation API, you can design orchestration scripts that communicate and run operations on CloudShell, from administrative operations like adding users, to sandbox-level operations like provisioning resources, resolving connectivity and running health check on the sandbox's elements. For additional information, see the CloudShell Automation API online help page, and the CloudShell Automation API Reference Guide. + +shellfoundry extendNote that since CloudShell Automation API can perform sandbox and CloudShell-level operations, it mostly applies to orchestration scripts and is not recommended to be used in shells. Having said that, there are two methods in the API that apply directly to shells: *WriteMessageToReservationOutput* allows the driver to print messages in real time to the output console, and *SetResourceLiveStatus* allows the driver to indicate the resource’s state with an icon. For example, online and offline. + +#### Examples + +* Adding resources to the sandbox, creating resource connections and setting 'online' live status icon on the resources: + +{% highlight python %} +import time + + +def execute(): + from cloudshell.workflow.orchestration.sandbox import Sandbox + Sandbox = Sandbox() + Reservation_Id = Sandbox.reservationContextDetails.id + + Resource_List = ['PUT Mock', 'PUT Mock/Port 1', 'Traffic Mock', 'Traffic Mock/Port 1'] + + time.sleep(3) + + Sandbox.automation_api.AddResourcesToReservation(reservationId=Reservation_Id, resourcesFullPath=Resource_List) + time.sleep(1) + print "" + Sandbox.automation_api.WriteMessageToReservationOutput(Reservation_Id, "resources added to sandbox") + + time.sleep(3) + Sandbox.automation_api.SetResourceLiveStatus('PUT Mock', 'Online', "Active") + Sandbox.automation_api.SetResourceLiveStatus('Traffic Mock', 'Online', "Active") + + time.sleep(1) + Sandbox.automation_api.WriteMessageToReservationOutput(Reservation_Id, "resources are online") + + time.sleep(3) + Sandbox.automation_api.CreateRouteInReservation(reservationId=Reservation_Id, + sourceResourceFullPath='PUT Mock/Port 1', + targetResourceFullPath='Traffic Mock/Port 1', + mappingType='bi', routeAlias='API-created route') + time.sleep(1) + Sandbox.automation_api.WriteMessageToReservationOutput(Reservation_Id, "resources connected") +{% endhighlight %} + +### TestShell API + +The **TestShell API** allows designing orchestration and automation using C#, TCL API and XML RPC. It is intended for performing a wide variety of operations within CloudShell; from administrative tasks, such as managing inventory or users, to sandbox operations, such as executing commands and controlling both resource and sandbox live statuses. It is especially useful for writing tests, and obtaining information about resources, blueprints and sandboxes. TestShell API and CloudShell Automation API provide the same capabilities and functionality. + +For additional information, see the appropriate TestShell API Reference Guide: +* TestShell API Library Reference Guide +* TestShell API C# Reference Guide +* TestShell API TCL Reference Guide +* TestShell API XML RPC Reference Guide + + +### Quali API + +Quali API allows you to automate the scheduling and queuing of test automation suites. It can be used in C# and TCL, and is also used for getting sandbox attachments and execution server details. + +For additional information, see the appropriate Quali API Reference Guide: +* Quali API Library Reference Guide +* Quali API C# Reference Guide +* Quali API REST Reference Guide + + +### Packaging API + +**CloudShell Packaging API** allows you to automate the creation configuration of CloudShell blueprint packages, which can be used for backup purposes and for sharing blueprints between different CloudShell deployments. For additional information and implementation examples, see the CloudShell Packaging API documentation. + + diff --git a/_devops/9.4.0/devops-integration.md b/_devops/9.4.0/devops-integration.md new file mode 100644 index 0000000..5c5c266 --- /dev/null +++ b/_devops/9.4.0/devops-integration.md @@ -0,0 +1,111 @@ +--- +layout: page +title: Automating CloudShell Sandboxes for DevOps +comments: true +order: 2 +version: + - 9.4.0 +--- +In this section, we'll look at different recipes for creating and using CloudShell Sandboxes as a part of your CI/CD +or other DevOps pipelines. In this context, we'll concentrate on the use case of automating CloudShell externally from scripts or other platforms. + +### Available APIs + +CloudShell currently supports two different sets of APIs which can be used to automate and integrate Sandboxes with DevOps. + +* The CloudShell Automation API (_cloudshell-automation-api_ package) - This python package contains a comprehensive set of APIs for anything from managing inventory and connections to administrating users and groups as well as managing sandboxes and blueprints. The package allows developers to use the TestShell API in shell drivers and python scripts. +* The CloudShell Sandbox APIs - This RESTful API is intended for automating the consumption of sandboxes oustide of CloudShell. As such, it provides a specific set of commands focused on the workflow of starting and ending sandboxes, and running automation. + +#### Which API to choose? +If the functionality you're looking for is covered in the Sandbox APIs, it is recommended to use it over the CloudShell Automation API package. +If not, you can consider using the Sandbox API for whatever subset of the required functionality it does offer and complement that with CloudShell Automation API calls. + +#### Where are the APIs documented? + +**CloudShell Sandbox API** + +The Sandbox API live documentation page is installed with CloudShell. To access it, simply browse to the following default address: +http://[CloudShellPortalAddress]:82/api/v2/explore/. If you're accessing the link from the CloudShell Portal machine itself, or from the CloudShell SDK edition machine, you can simply use "localhost" or "127.0.0.1". The API documentation page allows you to test and experiment with the APIs as well as provide information on the different operations and parameters. + +**CloudShell Automation API** + +The latest 8.3 Automation API online help is available here. + +When using the _cloudshell-automation-api_ package, make sure to install the version of the API which is compatible with your CloudShell version. To make the task of finding the right version easier, this package follows a versioning schema different from other CloudShell packages. The _major_ and _minor_ version of the _cloudshell-automation-api_ package will always match the CloudShell release its compatible with. Therefore, to install the latest compatible version you need to add these version requirements when installing from pip. For example, to install the latest _cloudshell-automation-api_ compatible with CloudShell 8.1, run: +{% highlight bash %} +python -m pip install "cloudshell-automation-api>=8.1,<8.2" +{% endhighlight %} + +To install the latest _cloudshell-automation-api_ compatible with CloudShell 8.0, run: +{% highlight bash %} +python -m pip install "cloudshell-automation-api>=7.1,<8.0" +{% endhighlight %} + +### Starting and stopping a sandbox + +The simplest and recommended way to start or stop a new sandbox instance from a blueprint is using the Sandbox API. +The Sandbox API live documentation page contains the details of the response and request format. The basic URLs of these operationsare _/blueprints/{blueprint_identifier}/start_ and _/sandboxes/{sandbox_id}/stop_. + +#### Using the Sandbox API from python + +The following code sample demonstrates how to use the Sandbox APIs for a simple flow - creating a sandbox from a blueprint and then ensuring setup completes successfully after running some tests. + +{% github_sample_ref /QualiSystems/devguide_examples/blob/master/devops_integration/sandbox_api_python_2_and_3/sandbox_api_example.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/master/devops_integration/sandbox_api_python_2_and_3/sandbox_api_example.py 13 25 %} +{% endhighlight %} + +This code uses a simple wrapper around the Sandbox API which uses the _requests_ package. You can find the source code for that implementation [here](https://github.com/QualiSystems/devguide_examples/blob/master/devops_integration/sandbox_api_python_2_and_3/sandbox_api/sandbox_apis.py): + +Since sandboxes used this way are really a scope for testing. It can be beneficial to use python's _with_ statement to ensure a sandbox is always set up and torn down properly with every usage. The following code wraps the Sandbox API as a context, ensuring proper cleanup and providing convenient access to the sandbox details. + +{% github_sample_ref /QualiSystems/devguide_examples/blob/master/devops_integration/sandbox_api_python_2_and_3/sandbox_context_example.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/master/devops_integration/sandbox_api_python_2_and_3/sandbox_context_example.py 15 23 %} +{% endhighlight %} + + +**Both of the above examples are python 2/3 compatible.** + +#### Using the Automation API from python + +The following code demonstrates implementing the same basic flow using the CloudShell Automation API package: + +{% github_sample_ref /QualiSystems/devguide_examples/blob/master/devops_integration/python_api/python_api_example.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/master/devops_integration/python_api/python_api_example.py 18 34 %} +{% endhighlight %} + +Similar to the Sandbox API example, we can wrap the setup and teardown of the sandbox in a context object and take advantage of the python _with_ operator to simplify the flow. + +{% github_sample_ref /QualiSystems/devguide_examples/blob/master/devops_integration/python_api/python_api_context_example.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/master/devops_integration/python_api/python_api_context_example.py 18 31 %} +{% endhighlight %} + +### Sandbox API Missing and Upcoming Features + +Currently, the Sandbox API does not support the creation of future (pending) sandboxes. If your CloudShell environment requires this feature, you may need to fall back to the _cloudshell-automation-api_ package API for sandbox reservation (using the `CreateReservation` method). + +### Executing Orchestration Commands + +Executing orchestration commands from the Sandbox API is supported using the _/sandboxes/{sandbox_identifier}/commands/{command_name}/start_ method. +In addition, the Automation API package provides the necessary APIs to execute orchestration scripts in the sandbox: + +{% github_sample_ref /QualiSystems/devguide_examples/blob/master/devops_integration/python_api/python_api_example.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/master/devops_integration/python_api/python_api_example.py 9 16 %} +{% endhighlight %} + +### Integrating with CI tools + +The same basic workflow demonstrated in the above examples can be used to implement a plugin for CI frameworks +such as Jenkins, Travis or TeamCity. + +Several Quali community projects already provide readymade solutions for some CI tools. + +The [Sandbox-Jenkins-Plugin](https://github.com/jenkinsci/cloudshell-sandbox-plugin) provides build steps for creating and ending sandboxes for Jenkins, as well as integration with the new PipeLines feature for continuous delivery and DevOps automation. + +The [Sandbox-Teamcity-Plugin](https://github.com/QualiSystems/Sandbox-TeamCIty-Plugin) provides similar build steps and integration for TeamCity. + + diff --git a/_introduction/9.4.0/setting-up-the-development-ide.md b/_introduction/9.4.0/setting-up-the-development-ide.md new file mode 100644 index 0000000..8c62b56 --- /dev/null +++ b/_introduction/9.4.0/setting-up-the-development-ide.md @@ -0,0 +1,80 @@ +--- +layout: page +title: Setting up the Development Environment +date: "2016-04-30 13:02:32 +0300" +order: 2 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this section we'll go over the recommended steps for setting up a development environment for developing CloudShell Shells and orchestration scripts. + +### Get the latest Python 2.7.x +![Python]({{ site.baseurl}}/assets/python-logo.png){:class="img-responsive"} + +Download and install the latest version of Python 2.7.x from the [official website](https://www.python.org/downloads/). +We also recommend installing pip (even though technically its included in the latest versions of Python). Follow the instructions on this website: [https://pip.pypa.io/en/stable/installing/](https://pip.pypa.io/en/stable/installing/) + +### Pick and install an IDE + +In the scope of this guide we'll discuss developing shells and scripts using standard +Python, which means that there are many great options for an IDE. +Some free IDEs to consider: + +* [Microsoft Visual Studio Code](https://code.visualstudio.com/) +* [JetBrains PyCharm](https://www.jetbrains.com/pycharm/) +* [GitHub Atom ](https://atom.io/) +* [Sublime Text ](https://www.sublimetext.com/) + +These are all great IDEs. At this point we recommend using PyCharm simply because you'll be able to use +Quali's developer plugin developed for that IDE in the community. This plugin is not a must but will make +your life a little easier by automating some steps required to upload your driver or set it up for debugging. + +### Install Git + +![Git]({{ site.baseurl }}/assets/git-logo.png){:class="img-responsive"} + +Some CloudShell developer tools require Git to be installed. +Follow the installation instructions on the [official website](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). + +### Install ShellFoundry (for Shells development) + +To shorten the Shell development cycle and reduce much of the boilerplate of setting up new +shell projects we recommend installing the [ShellFoundry](https://github.com/QualiSystems/shellfoundry) package. +To install ShellFoundry, simply open a new command line window and type in the following: + +{% highlight bash %} python -m pip install shellfoundry {% endhighlight %} + +For additional information about Shellfoundry, see [Shellfoundry]({{site.baseurl}}/reference/{{pageVersion}}/shellfoundry-intro.html). + +### Install and setup the CloudShell SDK + +![CloudShell]({{ site.baseurl}}/assets/cloudshell-logo.png){:class="img-responsive"} + +To deploy, test and debug your automation you'll need to have a working CloudShell SDK. +The CloudShell SDK is free for developers and can be downloaded from our +[community website](http://community.quali.com/spaces/12/index.html). +Follow the instructions on the [download page](http://info.quali.com/cloudshell-developer-edition-download) to get a local development installation of CloudShell up and running. +Please make sure you use an SDK version that matches your CloudShell production version for which you are developing Shells and Orchestration. + +### (Optional) Install the CloudShell PyCharm plugin (PyCharm) + +If you've selected to use PyCharm as your IDE, you can take advantage of the community contributed CloudShell plugin. +Follow the installation instructions on the [project repo](https://github.com/QualiSystemsLab/CloudShell-PyCharm-Plugin). +From the _Step-by-step installation guide_ section perform steps 1-6 only. Don't continue to configure the plugin usage further yet, we'll get to that part later in this guide. + +### Where to next? + +This guide contains three main areas depending on the type of development or integration you wish to do with CloudShell: + +* [Extending cloudShell with Shells]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html) +* [Orchestration Scripts]({{site.baseurl}}/orchestration/{{pageVersion}}/getting-started.html) +* [Configuration Management]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-overview.html) +* [Extending CloudShell with Cloud Providers]({{site.baseurl}}/cloudproviders/{{pageVersion}}/getting-started-with-cloud-providers.html) +* [CloudShell APIs]({{site.baseurl}}/devops/{{pageVersion}}/available-apis.html) + + diff --git a/_introduction/9.4.0/the-cloudshell-devguide.md b/_introduction/9.4.0/the-cloudshell-devguide.md new file mode 100644 index 0000000..580e017 --- /dev/null +++ b/_introduction/9.4.0/the-cloudshell-devguide.md @@ -0,0 +1,60 @@ +--- +layout: page +title: "The CloudShell DevGuide" +date: "2016-04-30 13:02:32 +0300" +order: 1 +comments: false +version: + - 9.4.0 + +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Welcome! In the following pages you will learn all you need to know to become an expert CloudShell developer. The guide is intended both for developers taking their first steps with the platform and seasoned CloudShell developers. + +_**Before developing shells and scripts, please familiarize yourself with CloudShell by taking [Quali U courses](http://courses.quali.com). These courses also include installation instructions for the CloudShell SDK package that deploys a developer edition of CloudShell on which you can perform your training and development activities.**_ + +### How this guide is organized + +The CloudShell developer guide is comprised of several distinct content areas. Each one describes a different branch of development or possible integration options with CloudShell. Each area of the guide is independent of the others and includes its own ‘getting started’ tutorials, examples, instructional videos and articles. You don’t have to follow a specific order so feel free to explore the areas that are relevant to your development. The areas of the DevGuide are: + +#### [Shells: Extending CloudShell’s functionality]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html) + +Developers can extend CloudShell’s capabilities to provide additional functionality for Apps or physical devices by creating Shells. This includes better modeling for these components as well as custom commands that can be used in the sandbox to integrate the component in CloudShell. This area of the guide covers the end-to-end flow of creating new Shells and importing them into CloudShell. + + +#### [Orchestration Scripts: Implementing sandbox workflows]({{site.baseurl}}/orchestration/{{pageVersion}}/getting-started.html) + +Orchestration scripts are a feature in CloudShell that enables the implementation of sandbox workflows. CloudShell has built-in flows for setup and teardown, which deploy and tear down Apps, resources and connections between components in the sandbox without having to add additional code. Users can extend or fork these scripts, as well as implement additional workflows for save/restore operations, scaling, or more use case-specific workflows for anything from performance testing to failover simulation or traffic generation. + + +#### [Configuration Management: Developing configuration scripts for App VMs]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-overview.html) +Configuration management extends CloudShell Apps by allowing the running of scripts, which install applications on virtual machines (VMs) deployed in the sandbox or run post-deployment configuration. This configuration can be executed as part of the sandbox setup, or later on in a live sandbox. Configuration management operations can be performed using custom scripts or Ansible playbooks. + + +#### [Custom Cloud Providers: Implementing support for cloud providers]({{site.baseurl}}/cloudproviders/{{pageVersion}}/getting-started-with-cloud-providers.html) + +CloudShell 9.0 provides support for the Cloud Provider shell, which enables you to integrate CloudShell with the cloud provider of your choice. CloudShell provides out-of-the-box support for cloud providers AWS EC2, Microsoft Azure, VMware vCenter and OpenStack. However, to deploy VMs on other cloud providers, such as Kubernetes or Oracle Cloud, or create a modified version of one of our out-of-the-box cloud providers, you will need to create a shell that allows this to happen. + + +#### [CloudShell APIs]({{site.baseurl}}/devops/{{pageVersion}}/available-apis.html) + +This chapter describes the different APIs CloudShell offers, the uses for each as well as useful links and implementation examples. + +In addition, this area examines how CloudShell can be used in conjunction with other DevOps tools and within the DevOps pipeline. This includes CloudShell’s APIs as well as existing Open Source projects for integrating CloudShell with other products. + +### Where to go next + +The _[Setting Up the Development Environment]({{site.baseurl}}/introduction/{{pageVersion}}/setting-up-the-development-ide.html)_ section contains important information for tooling and infrastructure you should install before starting development. + +You can then proceed to any one of the development areas described above: + +* [Extending cloudShell with Shells]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html) +* [Orchestration Scripts]({{site.baseurl}}/orchestration/{{pageVersion}}/getting-started.html) +* [Configuration Management]({{site.baseurl}}/configmanagement/{{pageVersion}}/cf-overview.html) +* [Extending CloudShell with Cloud Providers]({{site.baseurl}}/cloudproviders/{{pageVersion}}/getting-started-with-cloud-providers.html) +* [CloudShell APIs]({{site.baseurl}}/devops/{{pageVersion}}/available-apis.html) + + diff --git a/_orchestration/9.4.0/common-orchestration-recipes.md b/_orchestration/9.4.0/common-orchestration-recipes.md new file mode 100644 index 0000000..e62ec58 --- /dev/null +++ b/_orchestration/9.4.0/common-orchestration-recipes.md @@ -0,0 +1,57 @@ +--- +layout: page +title: Common Orchestration Script Recipes +category: orch +comments: true +order: 9 +version: + - 9.4.0 +--- + +In this section, we’ll provide a few handy examples of common script operations. The intention is to grow this into a good source to copy paste common code from. All of the examples are available in the +[DevGuide Examples](https://github.com/QualiSystems/devguide_examples) repository under the orchestration_scripts_examples folder. + +#### Executing commands on sandbox resources + +The following script attempts to execute a command only on resources that support it. If a resource does not support the command, the script will simply ignore it and move on to the next resource. + +{% github_sample_ref /QualiSystems/devguide_examples/blob/master/orchestration_scripts_examples/try_execute_commands/try_execute_commands.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/master/orchestration_scripts_examples/try_execute_commands/try_execute_commands.py 8 28 %} +{% endhighlight %} + +#### Configuring Apps in a Sandbox + +App configuration in a sandbox, initiated either by setup orchestration or a dedicated orchestration script, can be performed in parallel or ordered by custom logic using the app_configuration methods. +In the following example, we will configure all the ‘web servers’ Apps after configuring the ‘application server’ App; Also, to enable connection between the deployed Apps,we will pass the application server’s address to the web servers configuration: + +{% highlight python %} + +from cloudshell.workflow.orchestration.sandbox import Sandbox + +sandbox = Sandbox() + +## configure Application server +application_server = sandbox.components.get_apps_by_name_contains('application server')[0] + +sandbox.apps_configuration.apply_apps_configurations(application_server) + +application_server_address = sandbox.components.get_apps_by_name_contains('application server')[0].deployed_app.FullAddress + +web_servers = sandbox.components.get_apps_by_name_contains('web server') + +for server in web_servers: + ## set application server as app param (application_server_address is pre-configured on the app) + sandbox.apps_configuration.set_config_param(server, 'application_server_address', application_server_address) + +## configure web servers +sandbox.apps_configuration.apply_apps_configurations(web_servers) + +{% endhighlight %} + +Make sure to add a requirements.txt file that will include the cloudshell-orch-core package to use this example. + +Note the code in the components helper’s method to get the correct Apps from the sandbox and the usage in the App object rather than the name of the App for other methods like _apps_configuration.set_config_param_. + +Configuration of a sandbox’s Apps can be streamlined by using the OOB setup logic, as described in the CloudShell's OOB Orchestration section. + diff --git a/_orchestration/9.4.0/getting-information-from-cloudshell.md b/_orchestration/9.4.0/getting-information-from-cloudshell.md new file mode 100644 index 0000000..7ffeb29 --- /dev/null +++ b/_orchestration/9.4.0/getting-information-from-cloudshell.md @@ -0,0 +1,166 @@ +--- +layout: page +title: Getting Information from CloudShell +category: orch +comments: true +order: 2 +version: + - 9.4.0 +tags: + - api +--- + + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Information about the sandbox on which your script is running and its components is available in your script as an environment variable. The standard way to get the information is using the **Sandbox** object. + +**To use the *Sandbox* object:** + +* Import the cloudshell-orch-core python package and add it to your script, as illustrated in the example below. Note that the package is automatically imported when your sandbox starts. +In this example, the following code gets an object that contains all of the sandbox’s information: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +Sandbox = Sandbox() +reservation_context_details = Sandbox.reservationContextDetails +{% endhighlight %} + +Note that to execute this code, you will need to include a requirements.txt file in your script, see [Orchestration: Scripts Deep Dive]({{site.baseurl}}/orchestration/{{pageVersion}}/scripts-deep-dive.html) for more details. + +To facilitate writing and debugging activities, it is recommended to use advanced IDEs such as PyCharm, which provide autocomplete functionality, as illustrated below. + +![Sandbox information]({{site.baseurl}}/assets/reservation_context_8_1.png){:class="img-responsive"} + +Note that if you plan on using methods in your script, and want the IDE to autocomplete the *sandbox* object's class properties from within the method, you will need to include a docstring referencing the object. For details, see [Docstrings in orchestration scripts]({{site.baseurl}}/reference/{{pageVersion}}/intellisense-with-docstrings.html#docstrings-in-orchestration-scripts). + +### Accessing the sandbox components + +Use the **Sandbox** class to access and use the components of a sandbox in your orchestration scripts to implement custom logic. + +For example, let’s assume we want to get the names of the resources and Apps in a sandbox. To do so, we will use **Sandbox.component**. The following code will iterate over the resources and Apps in the sandbox and print out their names: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +Sandbox = Sandbox() + +for resource_name, resource in Sandbox.components.resources.iteritems(): + print 'Found resource: {0}, with address: {0}'.format(resource_name, resource.FullAddress) + +for app_name, app in Sandbox.components.apps.iteritems(): + print app_name +{% endhighlight %} + +The components in the sandbox are stored in a dictionary object, from which a specific resource can be retrieved using a simple syntax. For example: + +{% highlight python %} +Sandbox = Sandbox() +resource_details = Sandbox.components.resources['my_resource'] +{% endhighlight %} + +It’s also possible to get the sandbox components using helpers methods located under Sandbox.component such as get_resources_by_model, get_apps_by_name_contains and others. For example: + +{% highlight python %} +Sandbox = Sandbox() +services = Sandbox.components.get_services_by_alias('my-service-alias') +for service in services: + print service.Alias +{% endhighlight %} + +To refresh the components information at any time during the sandbox’s lifecycle, use the **Sandbox.components.refresh_components** method. + + +### Accessing the sandbox’s user inputs + +User inputs provided by the user when they reserved the blueprint can be accessed by your script, as contextual information. This data is stored in several environment variables based on the input type: + +* **Global inputs** - These inputs are a part of the reservation form and can represent general data you wish to collect from the user for your automation. They can also be used to group together multiple other inputs as one data entry. You can access these using the GLOBALINPUTS environment variable. + +* **Resource requirements** - These are inputs related to abstract resources. An abstract resource in CloudShell allows you to declare a generic spec or criteria for a resource rather than explicitly using a specific one. When customizing such an abstract resource, you can choose to make some of its properties available for the user to select, so as to make it more flexible. For example, for a physical device, instead of specifying the model in the blueprint, you can set the model as a parameter with a dropdown list for the user to select from when reserving it. Resource requirements are accessed using the RESOURCEREQUIREMENTS environment variable. + +* **Resource additional info** - When customizing an abstract resource, you can also choose to add some parameters to the resource that are not requirements but rather instructions on what to do with it. An example would be specifying an OS version to install on it. In this case, this parameter is not used to select the resource but rather to operate on the selected resource in the active sandbox. Additional info parameters are accessed using the RESOURCEADDITIONALINFO environment variable. + + As with sandboxes, we can use some helper modules to get the resource information in Python using the same object we used to get the reservation's details: + + {% highlight python %} +Sandbox = Sandbox() + +global_value = Sandbox.global_inputs['input name'] +requirement_value = Sandbox.requirement_inputs['resource1']['input_name'] +additiona_info_value = Sandbox.additional_info_inputs['resource1']['input_name'] +{% endhighlight %} + + +### Getting script input parameters + +You can add input parameters to a script by editing the it from the Script Management dashboard. +The input parameter values are also provided automatically to your script. CloudShell sets up an environment variable with +the same name as the parameter. + +This means that if your script looks like this, with a parameter called ‘Param1’ defined: + +![Sandbox information]({{site.baseurl}}/assets/script_param.png){:class="img-responsive"} + +You’ll be able to access it using an environment variable by that name: + +{% highlight python %} +import os +os.environ['PARAM1'] +{% endhighlight %} + +You can also use the Sandbox class: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +Sandbox = Sandbox() +sandbox.get_user_param('Param1') + +{% endhighlight %} + +Note that when using the *get_user_param* helper function, the input name is case insensitive since the function will uppercase it. + +### Getting sandbox information using the API + +A common use case for a script is to get a list of the different Apps and resources in the sandbox, to be able to call additional commands or API functions on them. To get that information, we can use the CloudShellAPI. + +**To start a CloudShell API session:** + +1. Obtain the Quali Server’s connectivity details. These details are also available as an environment variable in your script called ‘qualiConnectivityContext’. As with the sandbox information, you can use the Sandbox class to quickly get the connectivity information in a more convenient object form and initialize a CloudShellAPISession object by calling **Sandbox.connectivityContextDetails**. + +2. Create a CloudShell API session object. Since initializing a CloudShell API session object is a very common operation, you can use the **Sandbox** class to directly create an object. The **Sandbox** class will handle the passing of all of the required connectivity information for you. The **Sandbox** class provides a shortcut which makes accessing the CloudShell API from your script much easier. Simply use the following code: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +session = Sandbox.automation_api +{% endhighlight %} + +**Sandbox.automation_api** is a CloudShell API session object. You can use the IDE’s autocomplete capabilities to explore the available functions: + +![Sandbox information]({{site.baseurl}}/assets/sandbox_automation_api.png){:class="img-responsive"} + +### Getting custom sandbox metadata + +Starting with CloudShell 9.2, it is possible to store and retrieve custom key-value data from the sandbox. For details, see [Custom Sandbox Metadata]({{site.baseurl}}/reference/{{pageVersion}}/working-with-sandbox-metadata.html). + +### Getting saved sandbox information + +Starting with CloudShell 9.0, the *cloudshell-orch-core* python package includes a new class called *reservationLifecycleDetails*, which allows you to get the following details about your sandbox: saved sandbox name and description, and the current sandbox user name. + +![Sandbox information]({{site.baseurl}}/assets/reservationLifecycleDetails.png){:class="img-responsive"} + +Note that depending on the sandbox, the information may be partial. For example, if the sandbox is not a saved sandbox, the saved sandbox name and description will be missing. For details about our OOB saved sandbox orchestration scripts, see [CloudShell's OOB Orchestration]({{site.baseurl}}/orchestration/{{pageVersion}}/the-oob-orchestration.html). + +### Getting the user context + +Starting with CloudShell 9.2, you can get the CloudShell user who ran the blueprint/orchestration command in the **Sandbox** class. + +For example: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +Sandbox = Sandbox() +user = Sandbox.reservationContextDetails.running_user +{% endhighlight %} + + diff --git a/_orchestration/9.4.0/getting-started.md b/_orchestration/9.4.0/getting-started.md new file mode 100644 index 0000000..001913c --- /dev/null +++ b/_orchestration/9.4.0/getting-started.md @@ -0,0 +1,140 @@ +--- +layout: page +title: Getting Started +category: orch +order: 1 +comments: true +version: + - 9.4.0 +tags: + - orchestration +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Orchestration scripts can enable automating sandbox workflows. You can use orchestration scripts to create setup +and teardown procedures as well as other custom workflows that can be made available in the sandbox. Examples would include +saving and restoring state, starting test traffic, running a failover scenarios and more. *Please note that sandbox environment automation and enhanced orchestration is available with CloudShell Premium Tier.* + +_**Before developing scripts, please familiarize yourself with CloudShell by taking [Quali U courses](http://courses.quali.com). These courses also include installation instructions for the CloudShell SDK package that deploys a developer edition of CloudShell on which you can perform your training and development activities.**_ + + +### Prerequisites +* [Get CloudShell](http://info.quali.com/cloudshell-developer-edition-download): Download the latest CloudShell SDK and run it on your machine. +* [Python](https://www.python.org/downloads/): Make sure the appropriate Python version - 2.7.x and/or 3.x - (latest recommended) is installed on your machine. +
Starting with CloudShell 9.2, CloudShell comes with out-of-the-box support for python 3 for orchestration scripts. +* **IDE/Text Editor:** Your preferred IDE editor. We recommend using PyCharm (which offers a free community edition) because of the tooling we’ve already created for that IDE, including a CloudShell developer plugin. + + +### Creating and using orchestration scripts in CloudShell + +This procedure shows the basic steps for creating and using orchestration scripts in CloudShell. + +1) Create a python script. You can create a single python script, or a more complex orchestration that uses dependencies, as explained in [Scripts Deep Dive]({{site.baseurl}}/orchestration/{{pageVersion}}/scripts-deep-dive.html). + +2) If the script requires the use of python dependencies, which aren’t available in the public PyPi repository, add them to the local PyPi Server. See CloudShell Help's Updating Python Dependencies for Shells, Drivers and Scripts. + +3) Upload the script to CloudShell. When uploading the script, you can set it as a setup or teardown script, to have it run automatically in the sandbox, or leave it as a manually launched orchestration script. + +4) Attach the script to the blueprint. If it’s a setup or teardown script, remove the blueprint’s existing script first. + +### Creating a simple blueprint script + +In your preferred IDE create a new Python file. For now we'll want to keep its functionality very basic. +Simply add some Python code to print 'Hello CloudShell'. +As a side note, in CloudShell the output of a script is displayed in the output widget in the sandbox workspace, +so whatever you print in your script will find its way there. + +{% highlight python %} +print 'hello CloudShell' +{% endhighlight %} + +Save the file and give it any name, for example 'hello.py'. + +### Uploading the orchestration script to CloudShell + +In CloudShell Portal, open the **Manage** dashboard. In the left sidebar, click **Scripts** and select the **Blueprint** option. +The page should be similar to this: + +![Scripts Management Page]({{ site.baseurl}}/assets/environment_scripts_9.2.png){:class="img-responsive"} + +Click the **Add New Script** button and browse for the script. Once the script is uploaded, click **Edit**. +For now, just provide a name for the script (the script's file name is the default) and select the script's python version. + +![Scripts Management Page]({{ site.baseurl}}/assets/environment_script_hello_9.2.png){:class="img-responsive"} + +And click **Save** to seal the deal. Your script is now in CloudShell, all we need now is a blueprint where it can work its magic. + +### Attaching the script to a blueprint + +Open the **Blueprints** dashboard. Click **Add New**. A new blueprint will be created and you’ll be taken to the blueprint's workspace. + +Take a second to name your blueprint, you can do that by clicking the name next to the pencil icon. + +The last stop is the properties page. This is where we’ll connect our new script. Click the **Blueprint** drop down menu and select **Properties**. + +Here we’ll simply click the **Add Script** button to assign our new script to the blueprint +and click the **Update** button at the bottom of the page. That’s it! Our script is ready to be used. However, in order to run it, we’ll need to make our blueprint go live by creating a sandbox. Click the **Reserve** button to create a sandbox +from the blueprint. Next, let's run the script. + +### Executing the script in the sandbox + +Click the **Commands** button on the toolbar to open the **Blueprint Commands** side-pane. + +Click the **Run** icon next to the command to launch it! + +If the command executed successfully you should see a checkmark appear next to the command name and the **Output** pane +will display the command output. + +![Scripts Management Page]({{ site.baseurl}}/assets/run_script.png){:class="img-responsive"} + +In the scope of this simple tutorial, we’ve seen how to link a trivial Python script with a CloudShell blueprint. +We'll look into more concrete examples and nuances in later sections of the guide. + +### Best Practices for working with orchestration scripts + +When developing orchestration scripts, we recommend to download the latest default setup or teardown script from CloudShell Portal's **Manage>Scripts** page, make a copy and modify the contents. This will ensure you are using the right packages and structure. + +If you wish to develop an orchestration script from scratch or develop a script for a different CloudShell version, you will need to import the *cloudshell-orch-core* package, which provides the basic features and capabilities needed for CloudShell orchestration scripts. Use this table to decide which version to use: + + + +| CloudShell version | cloudshell-orch-core version | +| :--------- | :--------- | +| 8.3 GA | >=1.7.5.0,<1.7.6.0 | +| 9.0 GA | >=2.0.0.0,<2.1.0.0 | +| 9.1 GA | >=2.1.0.0,<2.2.0.0 | +| 9.2 GA | >=3.0.0.0,<3.1.0.0 | +| 9.3 GA | >=3.2.0.0,<3.3.0.0 | + +To set a specific version of the package, add a line to the script's requirements.txt file. For example: + +{% highlight bash %} +cloudshell-orch-core>=2.1.0.0,<2.2.0.0 +{% endhighlight %} + +For details about the requirements.txt file, see [Scripts Deep Dive]({{site.baseurl}}/orchestration/{{pageVersion}}/scripts-deep-dive.html). + +### Setting the default python version for new orchestration scripts + +Starting with CloudShell 9.2, the *DefaultPythonVersion* admin key allows you to control the python version in which all new orchestration scripts are created. For details, see CloudShell help's Advanced CloudShell Customizations. + +Note that this key also applies to new shells. + diff --git a/_orchestration/9.4.0/script-commands-customization.md b/_orchestration/9.4.0/script-commands-customization.md new file mode 100644 index 0000000..2f43f66 --- /dev/null +++ b/_orchestration/9.4.0/script-commands-customization.md @@ -0,0 +1,110 @@ +--- +layout: page +title: Script commands Visibility and Usability +category: orch +date: "2016-04-30 13:02:32 +0300" +order: 6 +comments: true +version: + - 9.4.0 +tags: + - orchestration +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +You can control many aspects of how the orchestration commands appear and behave in CloudShell by editing the script from the Scripts management page. + +To demonstrate these capabilities, we’ll create a simple script, which we’ll later customize. The script essentially prints out the parameters it receives and then sets the status of the sandbox to ‘Downloading’. + +1) Create a new Python file and name it customization_test.py. Add the following code: + +{% highlight python %} + +from cloudshell.workflow.orchestration.sandbox import Sandbox +import cloudshell.helpers.scripts.cloudshell_dev_helpers as dev_helpers + +def print_parameters_values(sandbox): + """ + :param Sandbox sandbox: + :return: + """ + print sandbox.get_user_param('first_param') + print sandbox.get_user_param('second_param') + +def change_reservation_status_to_online(sandbox): + """ + :param Sandbox sandbox: + :return: + """ + sandbox.automation_api.SetReservationLiveStatus(sandbox.id, "Downloading") + +def main(): + dev_helpers.attach_to_cloudshell() + sandbox = Sandbox() + print_parameters_values(sandbox) + change_reservation_status_to_online(sandbox) + +if __name__ == "__main__": + main() + +{% endhighlight %} + +     Note that you can get the Sandbox id by using **sandbox.id**. The Sandbox name is also available via the **sandbox.name** property. + +     Since this script has imports, you'll need a *requirements.txt* file and a *main.py* function. For details, see [Scripts Deep Dive]({{site.baseurl}}/orchestration/{{pageVersion}}/scripts-deep-dive.html). + +2) In CloudShell Portal, open the **Manage** dashboard. + +3) Navigate to the _Scripts_ sub section and select **Blueprint**. + +![Scripts Management Page]({{ site.baseurl}}/assets/environment_scripts_9.2.png){:class="img-responsive"} + +4) Add the script into CloudShell by dragging the script into the **Scripts – Blueprint** page or add the customization_test.py script. + +5) To customize the script, click the **Edit** button. +![Scripts Management Page]({{ site.baseurl}}/assets/orch_script_edit_9.2.png){:class="img-responsive"} + + +### Setting display name, descriptions and category + +* You can set the script’s display name or alias by editing the **Alias** field in the edit form. Note that if you try to execute this script as a command from the API, you’ll still need to reference it by its name. +* **Script Type** allows you to set the script as an orchestration script that CloudShell will run accordingly. +* Use the **Type** field to select the python version of the script. When the script is executed, CloudShell will create a virtual environment using the selected python version. +* **Visibility** controls who can access this command in CloudShell. Options are **Everyone** or **Admin Only** +* The **Description** is also easy to set via the edit form. Enter any text here, it will be displayed as a reference under the script name in the commands pane. This does not apply to orchestration scripts. +* Categories help improve usability by grouping commands with a similar role or domain under a folder in the sandbox’s command pane. This grouping is for visual purposes only and does not affect API calls. For this example, let’s set the category name to ‘Customization’. + + + +### Adding parameters + +Our script expects two parameters, but CloudShell has no way of knowing that. We need to add these parameters in this same form. + +1. Click the **Add Parameter** link at the bottom of the form. + +2. Set the **Name** as ‘first_param’ and the **Default Value** as ‘None’. +Note that if the command parameters don’t have default values, they will become mandatory and the user won’t be able to execute the command without filling in values for them. + +3. Then, enter a meaningful **Description**. + +4. Repeat the process, this time for the second param, which the script expects to be named “second_param”. + + +### Test the new look and feel + +To test the script, we need to add it to a blueprint. +1. In CloudShell Portal, click **Blueprints**. + + The **Blueprint Catalog** is displayed. + +2. Create a new blueprint or select an existing one. + +3. In the blueprint’s **Properties** page, associate the script with the blueprint. + +4. Reserve the blueprint and open the **Commands** pane in the sandbox. + +![Scripts Management Page]({{ site.baseurl}}/assets/scripts_customization.png){:class="img-responsive"} + + diff --git a/_orchestration/9.4.0/scripts-deep-dive.md b/_orchestration/9.4.0/scripts-deep-dive.md new file mode 100644 index 0000000..8ff19f3 --- /dev/null +++ b/_orchestration/9.4.0/scripts-deep-dive.md @@ -0,0 +1,181 @@ +--- +layout: page +title: Scripts Deep Dive +category: orch +comments: true +version: + - 9.4.0 +order: 6 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this section we’ll take a more in-depth view at scripts and learn how they can be used most effectively for CloudShell orchestration. + +#### How CloudShell handles scripts + +CloudShell executes a Python script by creating a temporary virtual environment for the script’s execution, downloading all required packages from the script’s requirements.txt file and executing the script using the Execution Server python engine. + +To send information to the script, CloudShell sets environment variables in the scope of the script process. These environment variables include information about the sandbox reservation, as well as the script’s parameters. The script standard output is returned as the command result. If an exception is raised, or if a non-zero process result code is returned, the execution will be considered a failure. + + +#### Using requirements.txt with CloudShell orchestration scripts + +It’s now possible to attach a requirements.txt file to make sure your script will have all dependencies installed on the virtual environment before the script’s execution starts. For details about the *cloudshell-orch-core* package, see this [Getting Started]({{site.baseurl}}/orchestration/{{pageVersion}}/getting-started.html#best-practices-for-working-with-orchestration-scripts) section. + +*To use the requirements.txt:* +* Archive this file with the orchestration script in one ZIP file. It will then be possible to upload this ZIP file to CloudShell using the CloudShell Portal. + +#### Using a main function and packaging multiple files + +As scripts become more complex, instead of structuring them as one big function, it is advisable to create a main function and separate the rest of the logic into different functions. Python requires some boilerplate code in addition to the main function to make this work. Here is an example code demonstrating how to use main functions with scripts: + +{% highlight python %} + +from cloudshell.workflow.orchestration.sandbox import Sandbox +import os + +def print_keys(): + for key in os.environ: + print key + " : " + os.environ[key] + +def print_app_names(sandbox): + """ + :param Sandbox sandbox: + :return: + """ + reservation_details = sandbox.automation_api.GetReservationDetails(sandbox.id).ReservationDescription + for app in reservation_details.Apps: + print app.Name + +def main(): + sandbox = Sandbox() + print_keys() + print_app_names(sandbox) + +if __name__ == "__main__": + main() + +{% endhighlight %} + +Depending on the complexity of the script, it may be wise to also separate the code into multiple files. To do that, we can take advantage of Python’s ability to support executing .zip archives containing multiple scripts. The only requirement is that one of the files is named ____main____.py, which is how the entry point of the Python process is determined. + +#### Setup and teardown scripts + +Setup and teardown are special types of orchestration scripts. There are two things that make them special: +1. They can’t have any inputs as they are launched automatically +2. If you use CloudShell’s default *Python Setup & Teardown* driver, then simply including a teardown or setup script in the sandbox and setting a duration for the setup/teardown is enough for CloudShell to launch it. + +*To set a script as a teardown or setup script:* + +1. In the *Scripts - Blueprint* management page, edit the script. +2. From the script’s *Script Type* dropdown list, select *Setup* or *Teardown*, as appropriate. +The script will take on that special behavior. Note that the script can only run as part of the sandbox Setup or Teardown process. In addition, you won’t be able to add any inputs to it. + + +#### Getting the script's python version + +To get the script's python version (2.7.x/3.x), use the following: + +{% highlight python %} +import sys +print ("You are using python: " + str(sys.version_info[0])) +{% endhighlight %} + + +#### Logging in orchestration scripts +*To implement logging in your orchestration script:* + +* Use Sandbox.logger, which is based on the CloudShell standard logging format. +The logger lists all the data that is needed for debugging the script, including the Sandbox id, function names, error level, time stamp and more. + + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox + +Sandbox = Sandbox() + +resource_name = 'My_Resource' +try: + Sandbox.automation_api.IncludeResource(resource_name) +except: + Sandbox.logger.error('Failed to include resource {0}'.format(resource_name)) +{% endhighlight %} + +Orchestration script logs are saved on the execution server's *%programdata%\QualiSystems\logs\* folder in a dedicated sub-folder with the reservation ID as the name. For example: + +![Log Structure]({{site.baseurl}}/assets/logging-reservation-id.png){:class="img-responsive"} + +#### Debugging scripts + +CloudShell includes some helper functions to make it easier to debug a script by running it on real sandbox reservation data. The helper functions allow the script to be “attached” to a CloudShell sandbox, by filling in all of the script’s environment variables so that the same information is available to it as if it were launched by CloudShell. + +**To attach a script to a CloudShell sandbox:** + +1) Create a sandbox reservation. + +2) In the script's project folder, create a python script. We'll use this script as our debugger. + +3) Paste the following code into the script: + +{% highlight python %} +from cloudshell.helpers.scripts.cloudshell_dev_helpers import attach_to_cloudshell_as +attach_to_cloudshell_as('CLOUDSHELL_USER', 'CLOUDSHELL_PWD', 'DOMAIN', 'RESERVATION_ID', 'SERVER_ADDRESS') +{% endhighlight %} + +3) To debug a specific function in your script, you will need to call the function from the debugger script. + +     For example, calling the *connect_layer1_routes* function from a script called *Setup_Script*: + +{% highlight python %} +from cloudshell.helpers.scripts.cloudshell_dev_helpers import attach_to_cloudshell_as +attach_to_cloudshell_as('admin', 'admin', 'Global', 'c425c01c-2cc6-4a3a-bb45-d357be6c294f', '192.168.42.125') + +from Setup_Script import connect_layer1_routes +connect_layer1_routes() +{% endhighlight %} + +4) Set the relevant details and debug the new script. + +     You can add additional functions to the script, from the same file or other ones in the project folder. + +As you write more complex orchestration scripts, you may want to separate the code to multiple files. To do that, we can take advantage of Python's ability to support executing _.zip_ archives containing multiple scripts. The only requirement, is that one of the files is named _\_\_main\_\_.py_, which is how the entry point of the Python process is determined. + +#### Referencing other packages + +1. Create a Sandbox reservation. +2. Add the following code and fill in the required data for the function parameters. + +{% highlight python %} +import cloudshell.helpers.scripts.cloudshell_dev_helpers as dev_helpers +dev_helpers.attach_to_cloudshell_as(user="CLOUDSHELL_USER", password="CLOUDSHELL_PWD", domain="DOMAIN", + reservation_id="RESERVATION_ID", server_address="ADDRESS", command_parameters={"NAME":"VALUE"}) +{% endhighlight %} + + +Note that if we include the above code in the example script we provided earlier, we’ll be able to run it locally as well as from the CloudShell Sandbox. The attach_to_cloudshell_as function will populate all of the blueprint data as CloudShell would, so from the code’s perspective, it doesn’t make a different where it is being run from. Furthermore, the code will ignore the special attach_to_cloudshell_as function if you run it from CloudShell so that there is no adverse effect to leaving the statement there. + + +However, using this strategy will expose your CloudShell credentials in the code. To avoid this, we recommend you use a similar function which takes the same information from a file. Make sure to add that file to the .gitignore list so that it doesn’t get on source control. The following code will have the same effect as the lines above, but will look for the information in a JSON file named quali_config.json, which should be in the project root. + +{% highlight python %} +import cloudshell.helpers.scripts.cloudshell_dev_helpers as dev_helpers +dev_helpers.attach_to_cloudshell() +{% endhighlight %} + +The _quali_config.json_ should have the following structure: + +{% highlight JSON %} +{ + "user" : "USER", + "password" : "PASSWORD", + "domain" : "DOMAIN", + "server_address" : "SERVER_ADDRESS", + "cloudshell_api_port" :"CLOUDSHELL_API_PORT", + "reservation_id" : "reservation_id", + "command_parameters" : { "PARAM_NAME" : "PARAM_VALUE" } +} +{% endhighlight %} + + diff --git a/_orchestration/9.4.0/the-oob-orchestration.md b/_orchestration/9.4.0/the-oob-orchestration.md new file mode 100644 index 0000000..4ba96e7 --- /dev/null +++ b/_orchestration/9.4.0/the-oob-orchestration.md @@ -0,0 +1,374 @@ +--- +layout: page +title: CloudShell's OOB Orchestration +category: orch +comments: true +version: + - 9.4.0 +order: 9 +--- +Every CloudShell installation includes out of the box workflows. These reflect some common workflows we see across many of our customers that we’ve decided to integrate as default behavior. The OOB setup and teardown processes handle App deployment and startup, connectivity, App discovery and installation. The OOB Save and Restore processes are used for saving the sandbox state and restoring it as a new sandbox. The setup and teardown OOB scripts are included as part of the default blueprint template as of CloudShell 7.1, while the Save and Restore OOB scripts are included starting with CloudShell 9.0. + +In this article: + +* [Setup and Teardown Orchestration](#setup) +* [Save and Restore Orchestration](#SaveRestore) + + +### Setup and Teardown Orchestration + +The following diagram describes the OOB setup and teardown flow: + + +![Setup Workflow]({{site.baseurl}}/assets/orchestration_workflow_9.2.png){: .center-image } + +These OOB setup and teardown scripts can be found in the Scripts – Blueprint management page. You can review their source code in the [cloudshell-orch-sandbox repository](https://github.com/QualiSystems/cloudshell-orch-sandbox/tree/v8.1/SandboxOrchestration/environment_scripts). + +As of CloudShell 8.1, the default setup and teardown logic moved to a python package called cloudshell-orch-core for ease of use. The default blueprint template includes a reference to the cloudshell-orch-core package using the requirments.txt mechanism, which is supported for orchestration scripts. + +Here is the implementation of the OOB setup script: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +from cloudshell.workflow.orchestration.setup.default_setup_orchestrator import DefaultSetupWorkflow + +sandbox = Sandbox() + +DefaultSetupWorkflow().register(sandbox) + +sandbox.execute_setup() +{% endhighlight %} + +As you can see, to use the default orchestration logic, we instantiated the *DefaultSetupWorkflow* class and registered the sandbox to use the default behavior. +Starting in CloudShell 8.1, sandbox setup is divided into 4 stages: preparation, provisioning, connectivity and configuration. It’s possible to disable the default implementation of each stage by setting enable_stageName=False, as illustrated in this example: + +{% highlight python %} +DefaultSetupWorkflow().register(Sandbox, enable_connectivity=False) +{% endhighlight %} + +The OOB setup and teardown scripts can easily be customized or extended. Click [here](https://github.com/QualiSystems/cloudshell-orch-sandbox/blob/develop/Samples/Setup/ordered_configuration_example.py) for an example on how to customize the app configuration order in the setup stage, or see [other samples](https://github.com/QualiSystems/cloudshell-orch-sandbox/tree/develop/Samples) to learn how to extend the OOB orchestration scripts. + +#### Extending the OOB Setup Orchestration Scripts + +Setup script logic is divided into 4 stages – Preparation, Provisioning, Connectivity and Configuration. Each Setup stage has a specific logic functionality. + +* **Preparation** is empty in the default Setup script. This is the place to enter any code that logically has to be executed before Setup logic is initiated. +* **Provisioning** deploys and discovers all apps. +* **Connectivity** connects all layer 1/layer 2/subnet connections, powers on and refreshes IPs on deployed apps. +* **Configuration** applies any additional configuration on deployed apps + +Note that the OOB setup already includes some default logic in each of the stages as depicted in the diagram at the top of this page. However, the OOB setup can easily be extended using the following extension methods: + +* add_to_preparation +* on_preparation_ended +* add_to_provisioning +* on_provisioning_ended +* add_to_connectivity +* on_connectivity_ended +* add_to_configuration +* on_configuration_ended + + +Each stage has an interim `on__[stage]_ended` step which allows the execution of any code that has to run between stages. Note that all the functions you add to a stage (using `add_to_configuration`, for example) run in parallel, while `on__[stage]_ended` functions run sequentially. + +You can extend the OOB setup and teardown scripts by adding additional steps, or controlling the order of execution. In this section we will focus on the setup script, for examples about how to extend the teardown, see [Extending the OOB Teardown Orchestration Scripts](#OOB-Teardown-scripts) below. An example for extending the OOB Setup script can be calling additional commands to validate Apps or resource states, launching additional orchestration, or controlling the order in which the sandbox is provisioned. + +1. Create a copy of the appropriate script, (see below for extension options), and upload the updated version separately into CloudShell Portal as a Setup script. DO NOT remove any step in the setup workflow. However, you can add your own steps or change the order of execution. + +2. Make sure not to name your extended script ‘setup’ but give it a more specific name. The name ‘setup’ is a reserved name, which may cause unexpected behavior when used on a setup script. + +You can extend the Setup script to implement the required logic in one of the setup stages: preparation, provisioning, connectivity and configuration or as a post stage for validation. Make sure to add a requirements.txt file that will include the cloudshell-orch-core package. For example, adding some logic to the configuration stage can be made in the following way: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +from cloudshell.workflow.orchestration.setup.default_setup_orchestrator import DefaultSetupWorkflow + +Sandbox = Sandbox() + +DefaultSetupWorkflow().register(Sandbox) + +sandbox.workflow.add_to_configuration(my_custom_login, components) +{% endhighlight %} + + +Each of the following methods gets a custom function and list of components to use in the function. For example, executing some custom logic to validate resource configuration: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +from cloudshell.workflow.orchestration.setup.default_setup_orchestrator import DefaultSetupWorkflow + +def custom_function(sandbox, components): + pass + +sandbox = Sandbox() + +DefaultSetupWorkflow().register(sandbox) + +resources = sandbox.components.get_resources_by_model('Generic Resource Model') + +sandbox.workflow.on_configuration_ended(custom_function, resources) +{% endhighlight %} + +Note that all methods of the OOB setup logic in the same stage are executed in parallel. +The custom function should get arrays of the sandbox and its components as inputs. It’s recommended to use this function template as a starting point: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox + +def custom_func(sandbox, components): + """ + :param Sandbox sandbox: + :return: + """ + pass +{% endhighlight %} + + +Here is an implementation example of custom configuration logic for a 3 tier application where each type of App is configured consecutively while passing some global inputs and configuration parameters between the Apps: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +from cloudshell.workflow.orchestration.setup.default_setup_orchestrator import DefaultSetupWorkflow + + +def main(): + sandbox = Sandbox() + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Starting to execute the cool stuff!') + + DefaultSetupWorkflow().register(sandbox, enable_configuration=False) # Disable OOB configuration + sandbox.workflow.add_to_configuration(function=configure_apps, + components=sandbox.components.apps) + sandbox.execute_setup() + + +def configure_apps(sandbox, components): + """ + :param Sandbox sandbox: + :return: + """ + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='configure_apps started') + build_id = sandbox.global_inputs['build_id'] + + # Configure databases + databases = sandbox.components.get_apps_by_name_contains('Database') + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Configuring Databases') + for app in databases: + sandbox.apps_configuration.set_config_param(app=app, + key='build_id', + value=build_id) + + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Database configured with build_id {0}'.format(str(build_id))) + + sandbox.apps_configuration.apply_apps_configurations(databases) + + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Finished to configure databases') + + # Configure Application Servers + app_servers = sandbox.components.get_apps_by_name_contains('Application') + + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Configuring Application Servers') + + database = sandbox.components.get_apps_by_name_contains('Database')[0].deployed_app.FullAddress + + for app_server in app_servers: + sandbox.apps_configuration.set_config_param(app=app_server, + key='build_id', + value=build_id) + + + sandbox.apps_configuration.set_config_param(app=app_server, + key='DB_address', + value=database) + + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Application Server configured with build_id {0} , and DB address {1}' + .format(str(build_id), str(database))) + + sandbox.apps_configuration.apply_apps_configurations(app_servers) + + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Finished to configure Application Servers') + + # Configure web servers + application_server_address = sandbox.components.get_apps_by_name_contains('Application')[0].deployed_app.FullAddress + + web_servers = sandbox.components.get_apps_by_name_contains('Web') + + for app in web_servers: + sandbox.apps_configuration.set_config_param(app=app, + key='Application Server', + value=application_server_address) + + sandbox.apps_configuration.set_config_param(app=app, + key='build_id', + value=build_id) + + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Web Server configured with build_id {0}, and Application Server address {1}' + .format(str(build_id), str(application_server_address))) + + + sandbox.apps_configuration.apply_apps_configurations(web_servers) + sandbox.automation_api.WriteMessageToReservationOutput(reservationId=sandbox.id, + message='Finished to configure Web Servers') + +main() +{% endhighlight %} + + +Here is another implementation that shows a scenario where some physical devices need to be loaded while few applications are deployed: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +from cloudshell.workflow.orchestration.setup.default_setup_orchestrator import DefaultSetupWorkflow + + +def load_firmware_sequential(sandbox, components): + """ + :param Sandbox sandbox: + :param components: + :return: + """ + for component in components: + sandbox.automation_api.ExecuteCommand(reservationId=sandbox.id, + targetName=component.Name, + targetType='Resource', + commandName='load_configuration') + + +sandbox = Sandbox() +DefaultSetupWorkflow().register(sandbox) + +chassis = sandbox.components.get_resources_by_model('Generic Chassis Model') +sandbox.workflow.add_to_provisioning(function=load_firmware_sequential, + components=chassis) + +sandbox.execute_setup() +{% endhighlight %} + + +#### Extending the OOB Teardown Orchestration Scripts + +You can extend the OOB Teardown script to execute custom steps prior to the out-of-the-box teardown orchestration, or to execute custom steps in parallel to the OOB teardown. This is done using the following extension methods, which are included in the workflow property in the **Sandbox** class: +* add_to_teardown +* before_teardown_started + +Each of the above methods gets a custom function and list of components to use in the function. All steps configured using the before_teardown_started method will be executed in a sequential manner, and all steps configured using the add_to_teardown method will be executed in parallel. + +Here is an example of how to execute a command on a resource prior to the default teardown orchestration, note that a requirements.txt file containing cloudshell-orch-core should be attached to the script: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +from cloudshell.workflow.orchestration.teardown.default_teardown_orchestrator import DefaultTeardownWorkflow + + +def execute_resource_cleanup(sandbox, components): + """ + :param Sandbox sandbox: + :param components: + :return: + """ + for component in components: + sandbox.automation_api.ExecuteCommand(reservationId=sandbox.id, + targetName=component.Name, + targetType='Resource', + commandName='cleanup') +sandbox = Sandbox() + +DefaultTeardownWorkflow().register(sandbox) + +chassis = sandbox.components.get_resources_by_model("Generic Chassis Model") +sandbox.workflow.before_teardown_started(execute_resource_cleanup, chassis) + +sandbox.execute_teardown() +{% endhighlight %} + +Make sure to follow these steps when implementing a custom teardown orchestration: +1. Create a copy of the appropriate script, (see below for extension options), and upload the updated version separately into CloudShell Portal as a Teardown script. DO NOT remove steps from the teardown workflow. However, you can add your own steps or change the order of execution. + +2. Make sure not to name your extended script ‘teardown’ but give it a more specific name. The name ‘teardown’ is a reserved name, which may cause unexpected behavior when used on a setup script. + + +### Save and Restore Orchestration + +*Note that these orchestration scripts apply to customers who have purchased the **Save and Restore** paid add-on. Contact your account manager to obtain a license.* + +Starting with CloudShell 9.0, Save and Restore scripts are provided to support the capability to save and restore sandboxes. They reside in a python package called *cloudshell-orch-core*. The OOB default blueprint template includes these orchestration scripts and a reference to the *cloudshell-orch-core* package (required by these scripts) using the requirements.txt mechanism. Here is the implementation of the OOB Save script: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox + +sandbox = Sandbox() + +sandbox.execute_save() +{% endhighlight %} + +By running the `execute_save` method on a sandbox, the script will call a server logic that will create a saved sandbox. For details about the saving process, see the CloudShell Help. + +#### Extending the OOB Save Orchestration Script + +You can extend the OOB Save script to execute custom steps before or after the default sandbox save process takes place. + +To do this, simply add your custom code before or after the line that executes the Save operation. For example, a Save orchestration script that sends a simple notification email when the Save operation completes: + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +import smtplib + +sandbox = Sandbox() + +sandbox.execute_save() + +# code for sending email notification: +server = smtplib.SMTP('smtp.gmail.com', 587) + +server.ehlo() +server.starttls() +server.ehlo() + +#Next, log in to the server +server.login("", "") + +#Send the mail +msg = "Sandbox was saved successfully" +server.sendmail("", "", msg) +{% endhighlight %} + +#### Extending the OOB Restore Orchestration Script + +You can also extend the OOB Restore script to execute custom functionality at any point during the default sandbox restore process. The Restore script is a part of the sandbox setup process, and actually replaces the setup. Out of the box, the setup and restore logic are identical. However, if you customized the Setup script and you want the same customized script to be launched when restoring a sandbox, you should customize the Restore script as well, as the Restore script is the one that is being launched in a restored sandbox’s setup phase. It is also possible to customize the Restore script to have a different logic than the Setup script, to create a logic that is relevant only for restored sandboxes. For detailed explanations on how to extend the script's stages and use its extension methods, see the [Setup and Teardown Orchestration](#setup-and-teardown-orchestration) section above. + +*Note that CloudShell does not support executing the Setup command directly via the Restore script using APIs, such as ExecuteEnvironmentCommand or EnqueueEnvironmentCommand.* + +For example, a Restore script that writes a message to the **Output** console before the Restore workflow operation (to extend the workflow operation itself, use the above extension methods in the [Extending the OOB Setup Orchestration Scripts](#extending-the-oob-setup-orchestration-scripts) section above): + +{% highlight python %} +from cloudshell.workflow.orchestration.sandbox import Sandbox +from cloudshell.workflow.orchestration.setup.default_setup_orchestrator import DefaultSetupWorkflow + +sandbox = Sandbox() + +def func(sandbox, components): + sandbox.automation_api.WriteMessageToReservationOutput(sandbox.id, "my custom message") + +DefaultSetupWorkflow().register(sandbox) + +sandbox.workflow.add_to_configuration(func, None) + +sandbox.execute_restore() + +{% endhighlight %} + +As you can see, to use the default orchestration logic, we instantiated the DefaultSetupWorkflow class and registered the sandbox to use the default Setup orchestration behavior. Starting in CloudShell 8.1, sandbox setup is divided into 4 stages: preparation, provisioning, connectivity and configuration. It’s possible to disable the default implementation of each stage by setting enable_stageName=False, as illustrated in this example: + +{% highlight python %} +DefaultSetupWorkflow().register(sandbox, enable_connectivity=False) +{% endhighlight %} + + diff --git a/_posts/9.4.0/deployed-app-shells.md b/_posts/9.4.0/deployed-app-shells.md new file mode 100644 index 0000000..d4d578c --- /dev/null +++ b/_posts/9.4.0/deployed-app-shells.md @@ -0,0 +1,67 @@ +--- +layout: page +title: Authoring a Deployed App Shell +comments: true +order: 4 +version: + - 9.4.0 +--- + +As we've discussed previously, there are two base types of Shells in CloudShell: Inventory Resources and Apps. +As opposed to Inventory Resources, which are cataloged and exist in the CloudShell inventory regardless of their +usage in sandboxes, Apps are deployed and exist within the Sandbox. They also follow a different lifecycle - each App is first +defined as a blueprint in the Apps catalog, the deployed app Shell is only created once the app blueprint is deployed and finally the deployed app is deleted when the sandbox ends. Before we discuss Deployed App Shells development, let's cover that process in more depth. + +### The App blueprint + +The App Blueprint is the spec or plan describing how to create and configure the Deployed App. Viewed schematically, +its comprised of the following layers: + +![App Architecture]({{ site.baseurl }}/assets/app_architecture.png){: .center-image } + +The App Blueprint is created in the App Dashboard. In CloudShell portal, open the Manage dashboard. In the left sidebar, click _Apps_. + +* **Deployment Type** - The app deployment type includes the technology which should be used to deploy the App as well as the relevant parameters. When you create a new App Blueprint in the catalog, in addition to naming the app, specifying the deployment type and parameters are the initial steps. For example, these are the options for a vCenter Linked Clone deployment type: + +![App Architecture]({{ site.baseurl }}/assets/deployment_type.png){: .center-image } + +* **Installation Script** - The installation script is run after the deployment is complete. A prerequisite to running +the script is that the OS is up and connected to the network. The role of the installation script is to complete any +configuration/installation of the app following the deployment of the image. The installation script may also have parameters +and these are provided when it is selected. + +* **Deployed App Shell** - The last stage is selecting the specific Deployed App Shell. This step is optional as by default the App will just use the 'Generic App' Shell. This will provide no special App commands or attributes, but can provide the basic functions like remote connectivity and power. To select the Deployed App Shell, click on the 'Advanced' link and select the Family/Model of the Deployed App you wish to use. + +### The App lifecycle + +The following diagram illustrates the App lifecycle: +![App Architecture]({{ site.baseurl }}/assets/app_lifecycle.png){: .center-image } + +App Blueprints are created in the Catalog. They can then be added to Environment blueprints and customized there, or directly to a sandbox. The App becomes a a deployed app as a result of two processes only: The Sandbox setup, or when the user manually select to deploy it. + +The Deployed App Shell becomes relevant at that stage. It shouldn't assume anything about the App Blueprint or the deployment process that preceded its creation. The main function of the Deployed App Shell is to manage the App and provide support for controlling its basic functionality, running health checks, configuration tasks etc. + +### A simple example + +We can create a deployed app Shell to test its behavior. +From the Command-line, run the following: + +{% highlight bash %} +shellfoundry new example-couchbase-app --template=deployed-app +cd example_deployed_app +{% endhighlight %} + +We've just created a new Shell project for the deployed app. Before we can proceed, we need to configure the App blueprint +to make sure we have a viable deployment option for this App to experiment on. + +### Create the CouchBase App blueprint + +There are various deployment options we can use for the App Blueprint. We'll present two options, if you'd rather +configure Couchbase in another way that makes sense in your environment. + +#### Option 1: AWS Deployment Type (Requires CloudShell 7.1 or newer) + +If you have an AWS account, you can use it to easily start a Couchbase server. +First you'll need to configure + + diff --git a/_reference/9.3.0/common-cloudshell-packages.md b/_reference/9.3.0/common-cloudshell-packages.md new file mode 100644 index 0000000..e1b2915 --- /dev/null +++ b/_reference/9.3.0/common-cloudshell-packages.md @@ -0,0 +1,48 @@ +--- +layout: page +title: Common CloudShell Packages +category: ref +order: 27 +comments: true +version: + - 9.3.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +This article lists the most common Python packages. These packages were developed by Quali and are available on public PyPi. + +**Notes:** +* *cloudshell-orch-core* is the only package you need for orchestration scripts. +* *cloudshell-orch-core* contains *cloudshell-automation-api* +* While *cloudshell-automation-api* includes several methods that apply to shells, it provides unnecessary access to CloudShell administrative areas and is therefore not recommended to be used in shells. + + + + +| **Package** | **Python version** |**Description** | **Links** | +| cloudshell-orch-core | Python 2/3 | Package for writing CloudShell orchestration scripts. For additional information, see [Best Practices for working with orchestration scripts]({{site.baseurl}}/orchestration/{{pageVersion}}/getting-started.html). | [PyPi](https://pypi.org/project/cloudshell-orch-core/)/[GitHub](https://github.com/QualiSystems/cloudshell-orch-core) | +| cloudshell-automation-api | Python 2/3 | Package for working with CloudShell Automation API. | [PyPi](https://pypi.org/project/cloudshell-automation-api/)/[Documentation](https://help.quali.com/Online%20Help/0.0/Python-API/) | +| cloudshell-logging | Python 2/3 | Package for creating shell loggers. | [PyPi](https://pypi.org/project/cloudshell-logging)/[GitHub](https://github.com/QualiSystems/cloudshell-logging) | +| cloudshell-core | Python 2 | Legacy logging package (replaced with *cloudshell-logging* in version 9.3). | [PyPi](https://pypi.org/project/cloudshell-core/)/[GitHub](https://github.com/QualiSystems/cloudshell-core) | +| cloudshell-snmp | Python 2/3 | Package for developing shell communication with devices via SNMP. | [PyPi](https://pypi.org/project/cloudshell-snmp/)/[GitHub](https://github.com/QualiSystems/cloudshell-snmp) | +| cloudshell-cli | Python 2/3 | Package for developing shell communication with devices via CLI. | [PyPi](https://pypi.org/project/cloudshell-cli/)/[GitHub](https://github.com/QualiSystems/cloudshell-cli) | +| cloudshell-shell-core | Python 2/3 | Package containing base shell functionality and interfaces. | [PyPi](https://pypi.org/project/cloudshell-shell-core/)/[GitHub](https://github.com/QualiSystems/cloudshell-shell-core) | + diff --git a/_reference/9.3.0/migrating-shells-to-python-3.md b/_reference/9.3.0/migrating-shells-to-python-3.md new file mode 100644 index 0000000..b41c180 --- /dev/null +++ b/_reference/9.3.0/migrating-shells-to-python-3.md @@ -0,0 +1,48 @@ +--- +layout: page +title: Migrating Shells to Python 3 +category: ref +order: 29 +comments: true +version: + - 9.3.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we’ll explain the general process for upgrading a Python 2-based shell to Python 3. + +_**Note:** Quali is currently working on updating all Quali-certified Python 2 shells to Python 3. If you need a Quali-certified shell, please contact your Quali Customer Success representative._ + +1) Open the *shelldefinition.yaml* file and promote the shell version. For example: + +{% highlight yaml %} +template_version: 2.0.0 +{% endhighlight %} + +3) Open the *drivermetadata.xml* file and add the **PythonVersion="3"** tag: + +{% highlight xml %} + +{% endhighlight %} + +4) In the *requirements.txt* file: + +     a. Replace `cloudshell-core` with `cloudshell-logging`. + +     b. Update the rest of the CloudShell packages to the latest versions. + +5) In every .py file in the project: + +     a. Update the code syntax to Python 3 format. + +     b. Update the import of `cloudshell-core` to `cloudshell-logging` (no need to update the code). + +{% highlight python %} +from cloudshell.logging.qs_logger import get_qs_logger +{% endhighlight %} + +5) Make sure all non-CloudShell packages support Python 3. + +6) Pack and install the shell on CloudShell. diff --git a/_reference/9.4.0/L1-switch-shell.md b/_reference/9.4.0/L1-switch-shell.md new file mode 100644 index 0000000..6efe751 --- /dev/null +++ b/_reference/9.4.0/L1-switch-shell.md @@ -0,0 +1,17 @@ +--- +layout: page +title: L1 Switch Shells +category: ref +order: 13 +comments: true +version: + - 9.4.0 +--- + +This article guides you on how to create and develop an L1 switch shell. + +The **L1 Switch Shell template** is a standardized python-based template that contains the structure and common predefined commands required from an L1 switch shell. Note that previous versions of the L1 shell were based on TCL. However, to simplify the development and maintenance, we changed the infrastructure to be python-based, which provides a standardized development experience similar to our other shells. Due to the implementation of the L1 switch infrastructure in CloudShell, importing this type of shell follows a unique procedure. + +For step-by-step instructions on how to create, implement and use the L1 switch shell, see the readme file on the shell's GitHub repository. + + diff --git a/_reference/9.4.0/associating-service-categories.md b/_reference/9.4.0/associating-service-categories.md new file mode 100644 index 0000000..70939e7 --- /dev/null +++ b/_reference/9.4.0/associating-service-categories.md @@ -0,0 +1,60 @@ +--- +layout: page +title: Categories in 1st Gen Service Shells +category: ref +order: 11 +comments: true +version: + - 9.4.0 +--- + +This article explains how to associate service categories to a service Shell. CloudShell exposes services to CloudShell users via service categories, which are associated to the CloudShell domains in which the services are required. The service categories of a specific domain constitute that domain’s services catalog. By default, each service Shell template is associated to a category in the Global domain. + +There are two ways to associate service Shells to categories: + +* Add the desired categories to the service’s family in Resource Manager Client +* Define the association in the service Shell’s data model + +In this article, we’ll learn how to associate categories via the Shell’s data model. + +**To associate a category to a service Shell:** + +1) Make sure you are running Shellfoundry with the Global admin user. Run `shellfoundry config` to see which user you are using. + +2) Create a new Shell from a template using `shellfoundry new`. + +3) Open the Shell’s root folder. + +4) Edit the categories in the _datamodel.xml_ and _category.xml_ files. + +     **To edit the datamodel.xml:** + +     a) In the _/datamodel_ folder, edit the _datamodel.xml_. + +     b) Under ``, create a duplicate of the commented `` line. + +     c) Change the category name as appropriate. + +     d) Uncomment the line. + +     e) Repeat to add additional categories. + +     f) Save the file. + +     **To edit the categories.xml (required if the categories do not exist in CloudShell):** + +     a) In the _/categories_ folder, edit the _categories.xml_. + +     b) Under ``, create a copy of the lines that start with the ``. + +          For example: + +{% highlight bash %} {% endhighlight %} + +     c) Specify the category name. Optionally specify child categories, which are nested under parent categories. + +     d) Save the file. + + diff --git a/_reference/9.4.0/aws-traffic-mirroring.md b/_reference/9.4.0/aws-traffic-mirroring.md new file mode 100644 index 0000000..53ee8db --- /dev/null +++ b/_reference/9.4.0/aws-traffic-mirroring.md @@ -0,0 +1,168 @@ +--- +layout: page +title: AWS Traffic Mirroring +category: ref +order: 25 +comments: true +version: + - 9.4.0 +--- + +This article explains how to copy network traffic running through a deployed AWS App to another App. For additional information about traffic mirroring, see AWS Documentation. + +This is done by calling two hidden commands on the AWS cloud provider via the CloudShell Automation API: +* `CreateTrafficMirroring`: Deploys traffic mirror sessions, traffic targets and filters, and associates them with the sandbox VPC +* `RemoveTrafficMirroring`: Tears down traffic mirroring sessions and related AWS resources. + +## Prerequisites + +* Source EC2 instance must be a Nitro-based instance. +* Target EC2 instance must have UDP port 4789 opened for traffic from the source instance. You can do this via the `Set AppSecurityGroups` API method, setting the target instance to accept all sandbox traffic or using the **Inbound Ports** attribute on the App template. +* Source and target NICs are required for the `CreateTrafficMirroring` call. Starting with *cloudshell-cp-aws* versions 2.4.3.x, NIC has been added to VM Details, to facilitate calling the resource command. + +## Limitations + +* A source network interface may be tapped up to 3 sessions. +* A target may have open sessions with up to 10 different sources (some dedicated instance types can have more). + +For details, see this AWS Documentation page: Traffic Mirroring Limits and Considerations. + +## Command interface + +Mandatory parameters are indicated. + +### CreateTrafficMirroring + ++ **DriverRequest**: + - **Actions**: List containing the following: + - **actionId**: (Str) + - **type**: (Mandatory, Str) Must be "CreateTrafficMirroring" + - **actionParams**: List containing the following: + - **type**: (Mandatory, Str) Must be "CreateTrafficMirroringParams" + - **sourceNicId**: (Mandatory, Str) Network interface ID of the source EC2 instance + - **targetNicId**: (Mandatory, Str) Network interface ID of the target EC2 instance (the traffic mirror target) + - **sessionNumber**: (Str) Traffic mirror session number that determines the order in which sessions are evaluated when an interface is used by multiple sessions (smallest number takes priority). Every traffic mirror session requires one. If left empty, CloudShell will allocate a number. + - **filterRules**: List of the following: + - **type**: (Mandatory, Str) Must be "TrafficFilterRule" + - **direction**: (Mandatory, Str) Defines the traffic direction on the source NIC. Valid values are **ingress** (inbound) and **egress** (outbound) + - **sourcePortRange**: (Type) Port range (**fromPort** and **toPort**) of the source EC2 instance (from which the traffic is sent). + - **destinationPortRange**: (Type) Port range (**fromPort** and **toPort**) of the target EC2 instance (to which the traffic is sent). + - **protocol**: (Mandatory, Str) Port protocol (tcp, udp, etc.) + - **sourceCidr**: (Str) CIDR of the source EC2 instance (from which the traffic is sent) + - **destinationCidr**: (Str) CIDR of the target EC2 instance (to which the traffic is sent) + +### RemoveTrafficMirroring + ++ **DriverRequest**: + - **Actions**: List containing the following: + - **actionId**: (Str) + - **type**: (Mandatory, Str) Must be "RemoveTrafficMirroring" + - **targetNicId**: (Mandatory, Str) Network interface ID of the target EC2 instance (the traffic mirror target) + - **sessionId**: (Mandatory, Str) Traffic mirror session ID (this ID is automatically assigned by AWS and returned in the *CreateTrafficMirroring* output). + + **Notes:** + * Make sure to include both the `sessionId` and `TargetNicId` parameters, but provide a value only for one of them, as illustrated in the [RemoveTrafficMirroring](#removetrafficmirroring-1) example. + * You cannot use the `sourceNic` parameter to remove traffic mirroring sessions. + +## Examples + +#### Using `reservationId` to get NICs + +If the EC2 instance has a single NIC: + +{% highlight python %} +reservation = session.GetReservationDetails('b9818bde-fc86-49f2-beae-74e238a3ad07').ReservationDescription + +resource_interface = next(next(p.Value for p in r.VmDetails.NetworkData[0].AdditionalData if p.Name=='nic') for r in reservation.Resources if 'resource_name' in r.Name) + +{% endhighlight %} + + +If the EC2 instance has more than one NICs: + +{% highlight python %} +from cloudshell.api.cloudshell_api import CloudShellAPISession + + +session = CloudShellAPISession('localhost', "admin", "admin", "Global") +reservation = session.GetReservationDetails('b9818bde-fc86-49f2-beae-74e238a3ad07').ReservationDescription + +for resource in reservation.Resources: + if resource.VmDetails and len(resource.VmDetails.NetworkData)>0: + for network_data in resource.VmDetails.NetworkData: + for additional_data in network_data.AdditionalData: + if additional_data.Name=="nic": + nic = additional_data.Value + +{% endhighlight %} + +#### CreateTrafficMirroring + +{% highlight python %} + +import cloudshell.api.cloudshell_api as api + +session = api.CloudShellAPISession('localhost', 'admin', 'admin', 'Global') + +result = session.ExecuteCommand(reservationId='9a8e81ac-5dd1-483a-9937-c17e477f455d', + targetName='AWS', + targetType='Resource', + commandName='CreateTrafficMirroring', + commandInputs=[api.InputNameValue(Name='request', Value=''' + { + "driverRequest": { + "actions": [ + { + "actionId": "41f206a1-a6c0-4603-90f7-6492cd0fb00d", + "type": "CreateTrafficMirroring", + "actionParams": {"type": "CreateTrafficMirroringParams", + "sourceNicId": "eni-008322622e675cd80", + "targetNicId": "eni-05fc8d0b34b762305", + "sessionNumber": "5", + "filterRules": [{"type": "TrafficFilterRule", + "direction": "ingress", + "protocol": "tcp", + "sourcePortRange": {"type": "PortRange", "fromPort": "50000", "toPort": "65535"}, + "sourceCidr": "192.168.0.124/24" + }] + } + } + ] + } + } + ''')] + ) +print result.Output + +{% endhighlight %} + +#### RemoveTrafficMirroring + +{% highlight python %} + +import cloudshell.api.cloudshell_api as api + +session = api.CloudShellAPISession('localhost', 'admin', 'admin', 'Global') + +result = session.ExecuteCommand(reservationId='514d18dc-cd6c-4725-ac60-9e281d0b8e27', + targetName='AWS', + targetType='Resource', + commandName='RemoveTrafficMirroring', + commandInputs=[api.InputNameValue(Name='request', Value=''' + { + "driverRequest": { + "actions": [ + { + "actionId": "514d18dc-cd6c-4725-ac60-9e281d0b8e27", + "type": "RemoveTrafficMirroring", + "sessionId": "", + "targetNicId": "eni-08e44d9a954444e3f" + } + ] + } + } + ''')] + ) +print result.Output + +{% endhighlight %} diff --git a/_reference/9.4.0/creating-static-vm.md b/_reference/9.4.0/creating-static-vm.md new file mode 100644 index 0000000..76a818e --- /dev/null +++ b/_reference/9.4.0/creating-static-vm.md @@ -0,0 +1,139 @@ +--- +layout: page +title: Shells that Load a Static VM +category: ref +order: 21 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we will learn how to customize a shell template to load a static VM into CloudShell. A static VM is a VM whose lifecycle is not managed through CloudShell sandboxes. For example, a VM that provides critical services or data, like a database, switch or bridge. For additional information about static VMs, see the CloudShell Help's Static VMs Overview article. + +Static VMs are viewed as resources by CloudShell. The only difference between a static VM resource and a regular resource is that the static VM needs to find the VM in the cloud provider and create a link between the CloudShell resource and the cloud provider resource, thus giving it the cloud provider shell's capabilities. This is done by modifying the shell's *get_inventory* command to load the VM's details into CloudShell, using a CloudShell cloud provider resource to access the cloud provider server. + +Let's start by creating a new shell project. Static VMs can only be modeled in shells of type 'Application', so we'll use the `deployed_app` shell template. In command-line, navigate to the folder in which you want to create the shell project and run the following command: + +{% highlight bash %} +shellfoundry new my-static-vm --template gen2/deployed-app +{% endhighlight %} + +In the shell's driver, we'll need to import the `ApiVMDetails` class and the `jsonpickle` package: + +{% highlight python %} +from cloudshell.shell.core.driver_context import ApiVmDetails, ApiVmCustomParam +import jsonpickle +{% endhighlight %} + +*You can remove ApiVmCustomParam from the import if you don't plan on setting custom parameters directly in the driver.* + +Note that `ApiVMDetails` is included in the *cloudshell-shell-core* package version 3.1.x and *jsonpickle* is not included in the python standard library, so we'll need to add them both to the *requirement.txt* file as well: + +{% highlight bash %} +cloudshell-shell-core>=3.1.0,<3.2.0 +jsonpickle==1.1.0 +{% endhighlight %} + +As we said before, to load a static VM, the shell needs to return the *ApiVmDetails* in the *get_inventory* command's response. Here's an example implementation for loading a vCenter VM as a static VM: + +{% highlight python %} +def get_inventory(self, context): + """ + Will locate vm in vcenter and fill its uuid + :type context: cloudshell.shell.core.context.ResourceCommandContext + """ + # get stuff from my cloud provider + # ... + uuid = '42415d31-bc19-d317-2319-b52s55e8b542' # unique identifier of the VM + my_clp_name = 'vCenter resource' # cloudshell cloud provider resource name + + vm_details = ApiVmDetails() + + vm_details.UID = uuid + vm_details.CloudProviderName = my_clp_name + + param1 = ApiVmCustomParam() # remove the "param1" lines if no custom params + param1.Name = 'param1' + param1.Value = 'value1' + vm_details.VmCustomParams = [param1] + str_vm_details = jsonpickle.encode(vm_details, unpicklable=False) + + # return vm_details + autoload_atts = [AutoLoadAttribute('', 'VmDetails', str_vm_details)] + return AutoLoadDetails([], autoload_atts) +{% endhighlight %} + +Make sure you pass the right details to the *get_inventory* command. Generally speaking, these are the CloudShell cloud provider resource (`my_clp_name` variable) that will run the process and the VM's identification details (`uuid` variable in our case). Note that for other cloud providers, different details may need to be passed in order to uniquely identify the VM. For example, a similar implementation for Azure might need the VM name and the resource group name. + +Install the shell on CloudShell by running the following command-line from the shell project's root folder: + +{% highlight bash %} +shellfoundry install +{% endhighlight %} + +When creating the resource in CloudShell Portal's **Inventory** dashboard, specify "na" as the **Address** if you don't know the VM's address or if the VM has a dynamic one: + +![Shell Commands]({{site.baseurl}}/assets/static-vm-resource-IP.png) + +In the CloudShell sandbox, the static VM will look like any other resource, with the correct live status icon ('online' in this case): + +![Shell Commands]({{site.baseurl}}/assets/static-vm-resource.png) + +### Advanced: Prompting the user for inputs in the Resource Discovery page + +If you want to allow the admin to provide the details during resource discovery, you will need to dynamically pull them from the context. For illustration purposes, we will set an attribute called "vCenter Name" that will define the vCenter cloud provider resource to be used to load and power on/off the VM. + +First, in the *shell-definition.yaml*, add the attribute as a discovery attribute. For example: + +{% highlight yaml %} +node_types: + + vendor.resource.MyStaticVm: + derived_from: cloudshell.nodes.DeployedApp + properties: + vCenter Name: + type: string + tags: [setting, configuration] + capabilities: + auto_discovery_capability: + type: cloudshell.capabilities.AutoDiscovery + properties: + vCenter Name: + type: string + tags: [setting, configuration] +{% endhighlight %} + +Now, let's generate the shell's data model by running the following command-line from the shell's root folder: + +{% highlight bash %} +shellfoundry generate +{% endhighlight %} + +The data model file is created in the shell project's *src* folder and lists the shell's attributes and functions, including those that come with the shell's standard and custom ones, like our **vCenter Name** attribute. + +In the driver, replace the following line: + +{% highlight python %} +my_clp_name = 'my_clp_resource_name' +{% endhighlight %} + +With this: + +{% highlight python %} +my_clp_name = context.resource.attributes['MyStaticVm.vCenter Name'] +{% endhighlight %} + +And install the shell on CloudShell: + +{% highlight bash %} +shellfoundry install +{% endhighlight %} + +The resource's discovery page will look something like this: + +![Shell Commands]({{site.baseurl}}/assets/static-vm-resource-discovery_9.2.png) + + diff --git a/_reference/9.4.0/defining-Azure-custom-routing.md b/_reference/9.4.0/defining-Azure-custom-routing.md new file mode 100644 index 0000000..c7d9ba3 --- /dev/null +++ b/_reference/9.4.0/defining-Azure-custom-routing.md @@ -0,0 +1,116 @@ +--- +layout: page +title: Custom Routing for Azure Apps +category: ref +order: 17 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we'll learn how to set up custom routing for Azure VMs deployed in the sandbox. + +CloudShell supports the creation of custom routes in Azure sandboxes, allowing you to force communication from any subnet(s) to go through any specific IP(s). For example, to have traffic go through a firewall or VPN connection. + +To do this, you need to use a blueprint or custom setup script that will tell the Azure cloud provider resource to define the custom routing (using the resource's CreateRouteTable hidden command). + +The custom routing needs to be supplied in JSON format. Let's create a JSON file and add the following code, which routes incoming communications from CIDR "10.0.1.0/28" to a VM that has the address "10.0.1.15". + +{% highlight python %} +{ "route_tables": { + "name": "myRouteTable1", + "subnets": ["subnetId1", "subnetId2"], + "routes": [{ + "name": "myRoute1", + "address_prefix": "10.0.1.0/28", + "next_hop_type": "VirtualAppliance", + "next_hop_address": "10.0.1.15" + }] + } +} +{% endhighlight %} + +Next, let's define the custom routing. + ++ **route_tables**: + - **name**: Provide a display name for the route table. + - **subnets**: Specify a comma-separated list of the subnet IDs of the source VMs. This will affect all VMs that have a NIC in that subnet. + Note that for VMs connected to more than one subnet, you will need to specify all connected subnets to ensure that all communication from such VMs use the custom routing. + + - **routes**: The route's settings: + + - **name**: Provide a display name for the route. + - **address_prefix**: Specify the target CIDR. Communication to an address in this CIDR will be diverted to our route. + - **next_hop_type**: Specify "VirtualAppliance" as the value. + - **next_hop_address**: The IP that the traffic will be directed to. For example, to connect VM 1 to VM 2 through VM 5, set VM 5 as the next_hop. + +To set additional hops, duplicate the routes section and edit the `next_hop_address`. For example, setting the traffic to hop through 10.0.1.15 and then thrugh 10.23.1.25: + +{% highlight python %} +{ "route_tables": { + "name": "myRouteTable1", + "subnets": ["subnetId1", "subnetId2"], + "routes": [{ + "name": "myRoute1", + "address_prefix": "10.0.1.0/28", + "next_hop_type": "VirtualAppliance", + "next_hop_address": "10.0.1.15" + }], + "name": "myRouteTable2", + "subnets": ["subnetId3", "subnetId4"], + "routes": [{ + "name": "myRoute2", + "address_prefix": "10.0.1.0/28", + "next_hop_type": "VirtualAppliance", + "next_hop_address": "10.23.1.25" + }] + } +} +{% endhighlight %} + +Lastly, make sure your script calls the JSON file. Here's an example script that uses our first JSON example (in this example, we embedded the JSON code in the script): + +{% highlight python %} +from cloudshell.api.cloudshell_api import InputNameValue +from cloudshell.workflow.orchestration.sandbox import Sandbox + +import json + +Sandbox = Sandbox() + +reservation_id = Sandbox.reservationContextDetails.id +azure_cloud_provider_name = 'Azure resource' + +request_json_string = ''' + { "name": "myRouteTable1", + "subnets": ["subnet1", "subnet2"], + "routes": [{ + "name": "myRoute1", + "address_prefix": "10.0.1.0/28", + "next_hop_type": "VirtualAppliance", + "next_hop_address": "10.0.1.15"}] + } +''' + +request_json = json.loads(request_json_string) + +session = Sandbox.automation_api + +result = session.ExecuteCommand(reservationId=reservation_id, + targetName="Azure resource", + targetType="Resource", + commandName="create_route_table", + commandInputs=[InputNameValue("request", request_json_string)], + printOutput=False) +{% endhighlight %} + + +Add the script as a blueprint script to CloudShell and attach it to your blueprint. If this is a setup script, set the script's **Script Type** to **Setup** to ensure that CloudShell will execute it automatically when a user reserves the blueprint. For additional information, see CloudShell help's Configure Blueprint Orchestration. + +*Note that you can have only one setup script attached to a blueprint and this will be the default for Azure that provisions and configures the resources. By overriding this script with another of type setup, you will potentially lose this functionality. Alternatively, you could create your own Setup script, which is based on the out-of-the-box one and includes the custom routing code - see [CloudShell's OOB Orchestration]({{site.baseurl}}/orchestration/{{pageVersion}}/the-oob-orchestration.html)* + +Future sandboxes based on this blueprint will use the defined custom routing for Azure VMs in the specified IPs/subnets. + diff --git a/_reference/9.4.0/intellisense-with-docstrings.md b/_reference/9.4.0/intellisense-with-docstrings.md new file mode 100644 index 0000000..2e8c519 --- /dev/null +++ b/_reference/9.4.0/intellisense-with-docstrings.md @@ -0,0 +1,69 @@ +--- +layout: page +title: Intellisense in Shells and Scripts (using Docstrings) +category: ref +order: 23 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we'll learn how to allow the IDE to recognize the CloudShell elements we want to use. This way, whenever we type in these elements in our methods, the IDE will provide useful code editing features, including code completion, parameter info and member lists. + +This is done by including Python documentation strings in our methods. Python documentation strings (or docstrings) allow you to associate documentation with python modules, functions, classes and methods. For additional information about docstrings, check out this Python documentation page. + +*To make use of intellisense, we recommend to use an IDE that supports such capabilities, such as PyCharm.* + +### Docstrings in shells + +Let's start by creating a shell: + +{% highlight bash %} +shellfoundry new docstrings-test +{% endhighlight %} + +Since we created our shell from a shell template, in this case the generic resource template, we already have docstrings defined for the out of the box method: + +![docstrings]({{site.baseurl}}/assets/docstrings.png) + +The initialize command's docstring defines the *InitCommandContext* object. Therefore, if we want to use this context in the method, intellisense will recognize the object and display its contents: + +![docstrings-context]({{site.baseurl}}/assets/docstrings-context.png) + +Note that if you create your own methods in the driver, you will need to manually create the docstrings and fill in the parameter types. First, create a method that uses the shell's *context* object and type six quotes in the method. Note that in Pycharm, when you type in three quotes, the IDE will add an additional three automatically. Make sure the cursor is in the middle (3 quotes before and 3 after) and press the **[Enter]** key. The docstrings section is created, listing the method's referenced parameters (in our case, *context*): + +![docstrings-manual-2]({{site.baseurl}}/assets/docstrings-manual-2.png) + +At this point, intellisense still doesn't recognize the object, as context can be any one of the driver's imported "context" objects: + +![docstrings-context-2]({{site.baseurl}}/assets/docstrings-context-2.png) + +So let's fill in the object's type (in our case, *ResourceCommandContext*) and optionally provide an informative description: + +![docstrings-manual-3]({{site.baseurl}}/assets/docstrings-manual-3.png) + +Now when you write "context" in the method, intellisense will identify it as a *ResourceCommandContext* object and display the appropriate information: + +![docstrings-manual-4]({{site.baseurl}}/assets/docstrings-manual-4.png) + +Note that the type doesn't have to be a class. It can also be a string or integer and it's a good practice to define your method's parameters this way. For example, if you'd like to define a custom parameter that is a string, you'd write something like this: + +![docstrings-manual-5]({{site.baseurl}}/assets/docstrings-manual-5.png) + +### Docstrings in orchestration scripts + +*sandbox* is the main object we use in CloudShell blueprint and orchestration scripts. The *sandbox* object is provided with the *cloudshell-orch-core* python package so include it in your *requirements.txt* file, import the *sandbox* object into the __main__.py file, and set the docstring in your method as follows: + +![docstrings-orchestration-scripts]({{site.baseurl}}/assets/docstrings-orchestration-scripts.png) + +### Docstrings in resource scripts + +In CloudShell resource scripts, we mainly use the *cloudshell_script_helpers* object. Since *cloudshell_script_helpers* is a module of functions and variables, you can access its contents directly from your method without having to create a docstring. The object is provided with the *cloudshell-automation-api* python package so include it in your *requirements.txt* file and import the object as "script_help" into the __main__.py file and you're good to go. + +For example: + +![docstrings-resource-scripts]({{site.baseurl}}/assets/docstrings-resource-scripts.png) + diff --git a/_reference/9.4.0/mapping-sub-resource-connections.md b/_reference/9.4.0/mapping-sub-resource-connections.md new file mode 100644 index 0000000..1ff857a --- /dev/null +++ b/_reference/9.4.0/mapping-sub-resource-connections.md @@ -0,0 +1,73 @@ +--- +layout: page +title: Mapping Connections using App Sub-resources +category: ref +order: 12 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +CloudShell allows developers to map connections between sub-resources residing on deployed Apps. This applies to scenarios where you want to map the port connections between virtual devices residing in App VMs. For example, to map the connection between port 1 residing on a virtual switch and port 2 residing on another virtual switch. + +The port mapping is done during the deployment of the App. This requires creating a shell and specifying the port's vNIC name in an attribute on the `Get_Inventory` command of the deployed App’s shell driver, and associating that shell to the desired App. Then, to map that App's vNIC, the blueprint designer will need to specify the vNIC name on the App. + +This is supported for vCenter, AWS EC2 and OpenStack Apps. + +### Configuration + +In this procedure, we will guide you on how to enable sub-resource mapping between Apps. + +1) Download the driver of the App's cloud provider from CloudShell Portal's **Manage>Drivers>Resource** page. + +2) Edit the *driver.py* file in your preferred editor. + +3) Modify the `get_inventory` command to include the sub-resources you want to support and the vNIC names. + +     For example, 2 sub-resources with vNIC names "Port 1" and "Port 2": + +{% highlight xml %} + def get_inventory(self, context): + """ + :type context: models.QualiDriverModels.AutoLoadCommandContext + """ + sub_resources = [AutoLoadResource(model='Generic Ethernet Port', name='Port 1', relative_address='port1'), + AutoLoadResource(model='Generic Ethernet Port', name='Port 2', relative_address='port2')] + + attributes = [AutoLoadAttribute('port1', 'Requested vNIC Name', '0'), AutoLoadAttribute('port2', 'Requested vNIC Name', '1')] + + result = AutoLoadDetails(sub_resources, attributes) + + return result + #return AutoLoadDetails([],[]) + {% endhighlight %} + +     Note that for AWS EC2 Apps, the vNICs must be sequential and start with "0". For example, 0, 1, 2. + +4) Create a shell model for the App in **Resource Manager Client>Resource Families>Generic App Family**. + +5) Create an attribute called **Requested vNIC Name** and add it to the new shell. + +6) Associate the port model defined in the command to the new shell in **Resource Manager Client>Resource Structure**. + +     In the above example, we used a port model called Generic Ethernet Port. + +7) Add the driver to CloudShell Portal's **Manage>Drivers>Resource** page and associate the new shell model. + +### Configuring the App + +1) In CloudShell Portal's **Manage>Apps** page, create or edit an App template. + +2) In the App's **App Resource** page, select the shell you created. + +3) Add the App to a blueprint. + +4) Create a connector from this App to another endpoint in the blueprint. + +5) Edit the connector line and in the **Requested Source vNIC Name** attribute, enter the vNIC name to use. + +     **Note:** The vNIC name must be defined in the driver's `get_inventory` command. In our case, "Port 1" or "Port 2". + diff --git a/_reference/9.4.0/migrating_1st_gen_shells.md b/_reference/9.4.0/migrating_1st_gen_shells.md new file mode 100644 index 0000000..7d2c1bf --- /dev/null +++ b/_reference/9.4.0/migrating_1st_gen_shells.md @@ -0,0 +1,253 @@ +--- +layout: page +title: Converting 1st Gen Shells into 2nd Gen +category: ref +order: 10 +comments: true +version: + - 9.4.0 +tags: + - migration + - 1st gen shells +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +As of version 8.0, CloudShell supports 2nd Generation Shells. In this topic we’ll go through the steps needed to convert a 1st Generation Shell into a 2nd Generation Shell. + + +### Overview +CloudShell Shells can be thought of as either 1st Generation Shells (1st Gen) or 2nd Generation Shells (2nd Gen). Both types of Shells can coexist inside the same CloudShell sandbox, but they differ in their structure and in the way that they are managed. + +* **1st Gen Shells** are imported as CloudShell packages that contain the data model and driver for the intended sandbox element. 1st Gen Shells allow extensive control of the family and model, and therefore are not standardized. While they allow maximal flexibility, when using them, some Shell management capabilities may not be available. + +* **2nd Gen Shells** are imported through CloudShell Portal's **Shells** management page. 2nd Gen Shells are based on standardized models and attributes, which streamlines the creation, maintenance and sharing of Shells. + + + +### Prerequisites +Before we start, it is important to understand the requirements: + +* To convert a 1st Gen Shell you need access to the Shell’s source code. +* You can only convert a Shell that implements the latest standard version. + +It is highly recommended to first learn how to create and model a 2nd Gen Shell before trying to convert from 1st Gen. This is described in detail in previous chapters of this guide. + +**To convert a 1st Gen Shell to a 2nd Gen Shell:** + +1. Create a 1st Gen Shell +2. Create a 2nd Generation Shell +3. Edit the Shell’s data model +4. Convert the Driver +5. Test the conversion + + + +### Create a 1st Gen Shell +In the context of this example, we will create a 1st Gen Shell, enable AutoLoad and add additional custom functions. + +**To prepare the 1st Gen Shell for conversion:** + +1) Create a 1st Gen switch Shell by running the following command in your Command Line: + +{% highlight bash %} +shellfoundry new my-switch --template gen1/networking/switch +cd my-switch +{% endhighlight %} + + +This Shell implements the networking standard v 4.0.1. + + +2) Enable the _autoload_ by updating the following line in the _shellconfig.xml_ file in the _datamodel_ folder: + +{% highlight xml %} + + +{% endhighlight %} + +3) Replace the _get_inventory_ function in the _driver.py_ file with this sample of _get_inventory_: + +{% highlight python %} +def get_inventory(self, context): + sub_resources = [ AutoLoadResource(model ='Generic Chassis',name= 'Chassis 1', relative_address='1'), + AutoLoadResource(model='Generic Module',name= 'Module 1',relative_address= '1/1'), + AutoLoadResource(model='Generic Port',name= 'Port 1', relative_address='1/1/1'), + AutoLoadResource(model='Generic Port', name='Port 2', relative_address='1/1/2'), + AutoLoadResource(model='Generic Power Port', name='Power Port', relative_address='1/PP1')] + + attributes = [ AutoLoadAttribute(relative_address='', attribute_name='Location', attribute_value='Santa Clara Lab'), + AutoLoadAttribute('', 'Model', 'Catalyst 3850'), + AutoLoadAttribute('', 'Vendor', 'Cisco'), + AutoLoadAttribute('1', 'Serial Number', 'JAE053002JD'), + AutoLoadAttribute('1', 'Model', 'WS-X4232-GB-RJ'), + AutoLoadAttribute('1/1', 'Model', 'WS-X4233-GB-EJ'), + AutoLoadAttribute('1/1', 'Serial Number', 'RVE056702UD'), + AutoLoadAttribute('1/1/1', 'MAC Address', 'fe80::e10c:f055:f7f1:bb7t16'), + AutoLoadAttribute('1/1/1', 'IPv4 Address', '192.168.10.7'), + AutoLoadAttribute('1/1/2', 'MAC Address', 'te67::e40c:g755:f55y:gh7w36'), + AutoLoadAttribute('1/1/2', 'IPv4 Address', '192.168.10.9'), + AutoLoadAttribute('1/PP1', 'Model', 'WS-X4232-GB-RJ'), + AutoLoadAttribute('1/PP1', 'Port Description', 'Power'), + AutoLoadAttribute('1/PP1', 'Serial Number', 'RVE056702UD')] + + return AutoLoadDetails(sub_resources,attributes) +{% endhighlight %} + + +4) Add an additional function that prints the ‘vendor’ attribute value: + +{% highlight python %} +def sample_command(self, context): + """ + Restores a saved artifact previously saved by this Shell driver using the orchestration_save function + :param ResourceCommandContext context: The context object for the command with resource and reservation info + """ + return context.resource.attributes["Vendor"] +{% endhighlight %} + + + +### Create a 2nd Gen Shell + +To convert the Shell, we will first create a new 2nd Gen Shell that implements the same standard version. Then we can edit the data model and update the driver. + +**To prepare the 2nd Gen Shell for conversion:** +* Create a 2nd Gen switch Shell by running the following in your Command Line: + +{% highlight bash %} +shellfoundry new my-switch-g2 --template gen2/networking/switch +cd my-switch-g2 +{% endhighlight %} + +It is recommended to change the name of the Shell to enable the 1st Gen Shell and the 2nd Gen Shell to work side by side on the same CloudShell database. + + +### Edit the Shell’s data model + +**To edit the Shell’s data model:** +* In the _shell-definition.yaml_ file, update the metadata section and make sure that the Shell imports the same CloudShell standard version as the 1st Gen shell. + +{% highlight yaml %} +metadata: + template_name: MySwitch + template_author: Anonymous + template_version: 0.1.0 + template_icon: shell-icon.png + +description: > + Sample TOSCA based shell + +imports: + - cloudshell_standard: cloudshell_networking_standard_4_0_1.yaml +{% endhighlight %} + +* If the Shell includes custom attributes, see [Modeling the Shell]({{site.baseurl}}/shells/{{pageVersion}}/modeling-the-shell.html) to learn how to model them in 2nd Gen Shell format. + +* If the Shell includes custom attributes in the discovery process, see [Auto Discovery For Inventory Shells]({{site.baseurl}}/shells/{{pageVersion}}/implementing-discovery-for-inventory-shells.html) to learn how to customize the Auto-discovery process. + +### Convert the Driver +To convert the driver, we need to update the driver files and then update the code. + +#### Updating the driver +To update the driver, we need to copy the 1st Gen driver into the 2nd Gen Shell project. However, since the name of the Shell may be different, we need to copy the files and keep all the references of the Shell’s name. + +**To update the driver:** + +1) In a text editor, open the 2nd Gen driver files in the **src** folder and save the lines that include references to the Shell’s name: + * In the _drivermetadata.xml_ file, copy line #1 (_MainClass_ and _Name_) +{% highlight xml %} + +{% endhighlight %} + +* In the _driver.py_ file, copy the _class name_ +{% highlight python %} +class MySwitchG2Driver (ResourceDriverInterface): +{% endhighlight %} + +2) Replace the files in the 2nd Gen Shell’s _src_ folder with the files from the 1st Gen Shell’s _src_ folder. + +3) To ensure that the original Shell’s name is used, open the files (_drivermetadata.xml_ and _driver.py_) and replace the relevant lines with the lines we saved. + + +At this point, it is recommended to install the Shell and make sure that we don’t get any error message. If the installation fails, make sure that the name references match the new Shell’s name and that you copied all the driver files properly. + +4) To test the modeling of the 2nd Gen Shell, run the following command in your Command Line to install the Shell: + +{% highlight bash %} +shellfoundry install +{% endhighlight %} + + +#### Updating the Code – The Shell’s data model + +To convert the 1st Gen Shell code to 2nd Gen, we need to modify it to match the 2nd Gen Shell’s data model. + +**To update the driver data model code:** + +1) Generate the Shell’s data model by running the following command in your Command Line: +{% highlight bash %} +shellfoundry generate +{% endhighlight %} + +2) Add the following to the driver.py file to import the Shell date model into the new driver: +{% highlight python %} +from data_model import * +{% endhighlight %} + +The Shell’s data model should be used in all the places in the driver where we refer to an attribute by name. +* For our example, replace the _sample_command_ with the code below. + +{% github_sample_ref QualiSystems/devguide_examples/blob/master/2nd%20gen%20shells%20-%20migration/src/driver.py %} +{% highlight python %} +{% github_sample QualiSystems/devguide_examples/blob/master/2nd%20gen%20shells%20-%20migration/src/driver.py 198 201 %} +{% endhighlight %} + +This code converts the _context_ object that CloudShell provides to an instance of the Shell’s data model, which is saved in the resource variable, then retrieves the value of the vendor attribute by referring to the _resource.vendor_ property. + + +#### Updating the code – Auto-Discovery +To simplify the conversion process, CloudShell provides a special python class that transforms 1st Gen discovery code to the 2nd Gen discovery’s structure without having to rewrite the existing code. + +The class is called _LegacyUtils_ and it is automatically generated with the Shell’s data model. + +The example below shows how to use the _LegacyUtils_ class: + + +{% github_sample_ref QualiSystems/devguide_examples/blob/master/2nd%20gen%20shells%20-%20migration/src/driver.py %} +{% highlight python %} +{% github_sample QualiSystems/devguide_examples/blob/master/2nd%20gen%20shells%20-%20migration/src/driver.py 204 229 %} +{% endhighlight %} + + +In this example, you can see that the code creates two arrays: _sub_resources_ and _attributes_, which are saved in the _autoload_details_ object. This 1st Gen structure is using explicit attribute names and model names in string format and needs to be transformed to 2nd Gen format. + +With the _LegacyUtils().migrate_autoload_details_ method, we convert the _autoload_details_ object into a new object _migrated_details_. Then the function returns the new formatted structure by calling the _migrated_details.create_autoload_details()_ object. + + +{% highlight python %} +migrated_details = LegacyUtils().migrate_autoload_details(autoload_details, context) +return migrated_details.create_autoload_details() +{% endhighlight %} + + +By using _LegacyUtils_ we can avoid rewriting the _get_invontory_ function. However, in the long term, it is recommended to refactor the code to use the Shell’s data model, as defined in [Auto Discovery For Inventory Shells]({{site.baseurl}}/shells/{{pageVersion}}/implementing-discovery-for-inventory-shells.html) + + +### Test the conversion +**To test the conversion:** + +1) Install the Shell by running the following in Command-Line: +{% highlight bash %} +shellfoundry install +{% endhighlight %} + +2) Log in as administrator to CloudShell Portal, create a Shell resource. +Make sure that the Auto-discovery successfully creates the resource with all of its sub-resources. + +3) Add the resource to a sandbox and run the _sample_command_. + +4) Check the **Output** console to see that the command successfully printed the value of the vendor attribute. + + diff --git a/_reference/9.4.0/python-coding-standards.md b/_reference/9.4.0/python-coding-standards.md new file mode 100644 index 0000000..7317c02 --- /dev/null +++ b/_reference/9.4.0/python-coding-standards.md @@ -0,0 +1,69 @@ +--- +layout: page +title: Python Coding Standards +category: ref +order: 3 +comments: true +version: + - 9.4.0 +--- +In this article we will describe the coding style for all scripts and drivers. Adhering to the same coding style makes our code more readable, +easier to understand for other teams and easier to maintain. +Quali Python Standard derives from [PEP 0008 -- Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/) and extends it with a few clarifications: + +- 4 spaces are used for indentation +- Class names should use the CapWords convention +- Use one leading underscore only for non-public methods and instance variables. +- Function names should be lowercase, with words separated by underscores as necessary, + started with a strong action verb (started with lower case) +- Variables should be lower_case_with_underscores. Avoid single character variable names +- Module names - lower_case_with_underscores. +- Constants - UPPER_CASE_WITH_UNDERSCORES +- Inline comments should start with # and a leading space +- Code should be documented with doc string according to Sphynx. + For details guideline see: [Spinx Guideline](http://www.sphinx-doc.org/) + +## Sphinx documentation example + +```Python +def Deploy(self, context, Name=None): + """ + Deploys app from template + :type context: cloudshell.shell.core.driver_context.ResourceCommandContext + :param context: The CloudShell execution context for the command + :type Name: str + :param Name: Name of the app to Deploy + :rtype: str + :return The VM details JSON of the deployed VM + """ +``` + +- Unit tests should be located under similar location as the class they test, for example: + +``` +vcenter_shell +|-- tests +| +-- test_commands +| +-- test_virtual_switch_to_machine_connector.py ++-- vcenter_shell + +-- commands + +-- virtual_switch_to_machine_connector.py +``` + +[PEP 20 -- The Zen of Python](https://www.python.org/dev/peps/pep-0020/) is a set of sentences that express the spirit behind Python. + +**We would like to emphasize the following:** + +- Beautiful is better than ugly. + +- Explicit is better than implicit. + +- Simple is better than complex. + +- Readability counts. + +- Errors should never pass silently. + +- Unless explicitly silenced. + + diff --git a/_reference/9.4.0/quali-shell-framework-python-3.md b/_reference/9.4.0/quali-shell-framework-python-3.md new file mode 100644 index 0000000..3743e6b --- /dev/null +++ b/_reference/9.4.0/quali-shell-framework-python-3.md @@ -0,0 +1,184 @@ +--- +layout: page +title: Quali’s Shell Framework (Python 3) +category: ref +order: 9 +comments: true +version: + - 9.4.0 +--- + + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we will familiarize ourselves with the CloudShell shell framework and learn how to leverage it to develop and customize commands in shells based on Python 3. Note that this applies to 1st Gen and 2nd Gen shells. + +## Introduction + +Every CloudShell shell consists of a data model and a driver. The driver is written in python and can have python package dependencies. Quali’s officially released shells use a common set of python packages developed by Quali, which contain most of the logic of Quali shells, while the driver itself (the “.py” file inside the shell) is the thin layer that defines the interface with CloudShell along with the driver’s python requirements. + +Quali’s official shells have a granularity level of Vendor and OS. This means that each official shell supports all devices of a specific vendor and OS. The exact functionality that is exposed by the shell is defined in the relevant [shell standard](https://github.com/QualiSystems/cloudshell-standards/tree/master/Documentation). The structure of the python packages reflects this granularity – for example, any logic that is common to all shells resides in cloudshell-shell-flows, while any Cisco-specific logic resides in cloudshell-networking-cisco, and any Cisco NXOS-specific logic resides in cloudshell-networking-cisco-nxos. It is possible to use Quali’s shell framework when creating your own shells or customizing existing ones. + +Note that using the framework is optional. + +To work with one or more of Quali framework’s python packages, you need to list them in your shell project’s requirements.txt file. Then, you can either write the code that uses the packages directly in the shell’s driver or create your own python packages and add them to the shell's requirements file. You can also load such custom python packages into your local [PyPi server repository](http://help.quali.com/Online%20Help/9.3/Portal/Content/Admn/Cnfgr-Pyth-Env-Wrk-Offln.htm?Highlight=pypi) on the Quali Server machine to make them available to your entire CloudShell deployment. + +**Important:** We don't recommend modifying Quali python packages as CloudShell will overwrite them if a newer package that has the same file name is published on the public PyPi repository. Alternatively, you’re welcome to create your own packages, using our python packages as a reference. + + +## Python package structure + +The following diagram shows the python classes used by the shell commands and their dependencies: + +![Python Package Structure Diagram]({{site.baseurl}}/assets/python-3-package-structure.jpg) + +## Architecture + +The architecture of a Quali python shell comprises three inter-dependent elements: +* [Flows](#Flows) +* [Command Templates](#CommandTemplates) +* [Command Actions](#CommandActions) + +Flows sequentially execute several Command Actions, while each Command Action runs a specific Command Template. + +An additional element that is used by the flows is the communication handler, which allows you to communicate with the device. For details about communication handlers, see this [section](#CommunicationHandlers). +  +## Key Entities + +There are several objects that must be initialized in the python driver, to allow you to work with Quali's infrastructure: +* **Communication Handlers** – These entities handle the communication between the shell and the device. The most common handlers are cli (`cloudshell-cli`) and snmp (`cloudshell-snmp`). These handlers must be initialized whenever calling a shell command and passed to the flows. +* **cli** - A Python package that resides in the driver and is used to create the CLI configurator. This package provides an easy abstraction interface for CLI access and communication (Telnet, TCP, SSH etc.) for network devices. The CLI class instance is provided by `cloudshell.cli.cli`. It must be created when the driver is initializing, since it allows the shell to designate a single session pool for the shell’s commands. You are welcome to use the *_get_cli* helper from *driver_helper* mentioned above. *_get_cli* allows you to define the session pool’s size and idle timeout. +* **api** is an instance of the *cloudshell-automation-API*’s *CloudShellAPISession* class. It must be created on every command execution. This class has a helper named *_get_api*, which is also provided by the *driver_helper* mentioned above. +* **logger** is a logger object from *cloudshell-core*. It is recommended to use the *driver_helper’s get_logger_with_thread_id* function. +* **resource config** – Python implementation of the relevant Quali standard, which defines the shell’s attributes and default values. For example, a *NetworkingResourceConfig* class that contains all the attributes required by the networking standard. It can be easily created using the `NetworkingResourceConfig.from_context` method, which is provided with the [cloudshell-shell-networking-standard](https://pypi.org/project/cloudshell-shell-networking-standard/) python package. + +For reference, see this [example](https://github.com/QualiSystems/Cisco-IOS-Switch-Shell-2G/blob/dev/src/driver.py). + +### Communication Handlers + +The most common ways to communicate with the device are via: +* CLI – cloudshell-cli +* SNMP – cloudshell-snmp + +These handlers need to be initialized and passed to the flows. + +#### CLI Configurator + +The CLI configurator resides in the cloudshell-cli python package and includes all the typical CloudShell CLI attributes you would need in order to communicate and work with a device modeled in CloudShell. For example, it knows how to retrieve the device’s username and password, create a session, determine what kind of session to initiate, etc. For reference, see [cisco_cli_handler.py](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/cli/cisco_cli_handler.py) and [cisco_command_modes.py](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/cli/cisco_command_modes.py). + +There is a nice guide on how to use *cloudshell-cli* [here](https://github.com/QualiSystems/cloudshell-cli/blob/dev/README.md). To simplify the usage of CloudShell CLI with shells, we added a *CliServiceConfigurator* base class that allows you to quickly set up communication with the device. + +To work with CloudShell CLI from the shell driver, you need to create a *CommandMode* and initialize the *CliServiceConfigurator*. + +Example: + +{% highlight python %} +command_mode = CommandMode(".*") +with CliServiceConfigurator(resource_config=resource_config, logger=logger).get_cli_service(command_mode) as cli_service: + cli_service.send_command("") +{% endhighlight %} + + + +#### SNMP Configurator + +The SNMP configurator provides SNMP communication with the device and resides in the cloudshell-snmp python package. It includes two classes for working with SNMP: +* **SnmpConfigurator** is used for basic SNMP initialization +* **EnableDisableSnmpConfigurator** allows you to implement Enable and Disable SNMP flows, which manage SNMP sessions on the device + +Like the CLI configurator, the main responsibilities of the SNMP configurator are: +* Initialize *Snmp*. It does this by extracting and preparing the *Snmp* parameters provided by the resource driver’s context +* Analyze Enable and Disable SNMP attributes from the command context +* Trigger the appropriate flow. +
To initialize the `SNMPConfigurator` object, you need to pass the following objects: `resource_config` and `logger`. To use the SNMP configurator, you must implement the following properties in your child class: + +          • *_create_enable_flow* + +          • *_create_disable_flow* + +For reference, see [cisco_snmp_handler](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/snmp/cisco_snmp_handler.py). For more information, see the [Flows](#Flows) section. +  +## Flows + +A flow is an organized sequence of Command Actions. Each flow has interfaces, which contain commands that are triggered by the resource driver. All the base flows are located in the [cloudshell-shell-flows](https://github.com/QualiSystems/cloudshell-shell-flows/tree/dev/cloudshell/shell/flows) python package. They are based on the *BaseFlow* interface, which is also located in the package. + +This is an abstract class that includes generic implementations for preparing and validating the required parameters. For example, when running the Save command in CloudShell Portal, the flow must validate the folder path provided by the sandbox end-user. + +The necessary interfaces are already implemented in the base flows. + +* **[Connectivity Flow](https://github.com/QualiSystems/cloudshell-shell-connectivity-flow/blob/dev/cloudshell/shell/flows/connectivity/basic_flow.py)** – Uses multithread logic to speed up the VLAN configuration on the device, especially when the resource needs to undergo a huge request that involves multiple, concurrently run actions. To initialize this flow, you have to provide the *logger* and *cli_handler* objects (described in [Key Entities](#KeyEntities)). Use the `apply_connectivity_changes` method to start. The following methods must be implemented: + +          • *_add_vlan_flow* + +          • *_remove_vlan_flow* + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/flows/cisco_connectivity_flow.py). + +* **[Configuration Flow](https://github.com/QualiSystems/cloudshell-shell-flows/blob/dev/cloudshell/shell/flows/configuration/basic_flow.py)** – Prepares and validates the provided path for the `save`, `restore`, `orchestration save` and `orchestration restore` commands. To initialize this flow, you have to pass the *logger*, *resource_config*, *cli_handler* and *api* objects (described in [Key Entities](#KeyEntities)). The following methods must be implemented: + +          • *_save_flow* + +          • *_restore_flow* + +          • *_file_system* – a default filesystem value, see example below. + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/flows/cisco_configuration_flow.py). + +* **[Firmware Flow](https://github.com/QualiSystems/cloudshell-shell-flows/blob/dev/cloudshell/shell/flows/firmware/basic_flow.py)** – This flow serves as a configuration flow and also validates the firmware’s file path. To initialize this flow, you need to pass the *logger* object. The following method must be implemented: + +          • *_load_firmware_flow* + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/flows/cisco_load_firmware_flow.py). + +* **[Run Command Flow](https://github.com/QualiSystems/cloudshell-shell-flows/blob/dev/cloudshell/shell/flows/command/basic_flow.py)** – As you can see from the name, this flow handles the `Run Custom Command` and `Run Custom Config Command` driver methods and doesn’t require anything else to implement. However, if you want to customize the `run_command_flow` property, you are welcome to override it. To initialize this flow, just pass the *logger* and *cli_handler* objects. + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/flows/cisco_run_command_flow.py) on how to create the run command flow. + +* **[State Flow](https://github.com/QualiSystems/cloudshell-shell-flows/blob/dev/cloudshell/shell/flows/state/basic_flow.py)** – This flow is very similar to the Run Custom Command flow as it doesn’t require any additional implementations. It contains implementations for the `Health Check` and `Shutdown` commands. To initialize this flow, you need to pass the *logger*, *api*, *cli_handler* and *resource_config* objects mentioned in [Key Entities](#KeyEntities). + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/flows/cisco_state_flow.py) on how to create the state flow. + +* **[Autoload Flow](https://github.com/QualiSystems/cloudshell-shell-flows/blob/dev/cloudshell/shell/flows/autoload/basic_flow.py)** – Discovers the device’s hardware structure and general details, such as the firmware version and model. This flow requires the *autoload_flow* method to be implemented. To initialize this flow, just pass the *logger* object. The following method must be implemented: + +          • *_autoload_flow* + +For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/flows/cisco_autoload_flow.py) on how to create the autoload flow. + +## Command Templates + +A Command Template is a constant object of the *CommandTemplate* class, which is located in [cloudshell-cli](https://github.com/QualiSystems/cloudshell-cli/blob/dev/cloudshell/cli/command_template/command_template.py). It contains three elements: +* **command** – A command you want to send to the device. Supports formatting parameters. For example: + +{% highlight bash %}copy {src} {dst} [vrf {vrf}]{% endhighlight %} + +* **action map** – A dictionary of regex patterns (as keys) and lambda functions (as values). To illustrate this, when you get a “(yes/no)” prompt, you can send an appropriate command by specifying the required lambda function. For example: + +{% highlight bash %} +r'\(y/n\)': lambda session, logger: session.send_line('y', logger) +{% endhighlight %} + +* **error map** – Similar to action map but for errors. It is a dictionary of regexp pattern and error message test pairs. For example: + +{% highlight bash %} +r"[Ii]nvalid\s*([Ii]nput|[Cc]ommand ": “Override mode is not supported” +{% endhighlight %} + +     For reference, see this command template [file](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/command_templates/configuration.py). + +## Command Actions + +Command Action is a method or function (depending on your implementation) that runs one or more Command Templates. The [CommandTemplateExecutor](https://github.com/QualiSystems/cloudshell-cli/blob/dev/cloudshell/cli/command_template/command_template_executor.py) is provided for running command templates. For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/command_actions/add_remove_vlan_actions.py). + +For example, if we want to execute the CONFIGURE_VLAN command template: + +{% highlight bash %} +CONFIGURE_VLAN = CommandTemplate("vlan {vlan_id}", error_map=OrderedDict({"%.*\.": "Error creating vlan")})) +{% endhighlight %} + +We need to (1) pass the `session` object, command template and action/error maps to the *CommandTemplateExecutor*, and (2) `vlan_id` to the `execute_command` method. As illustrated below: + +{% highlight bash %} +CommandTemplateExecutor(session, CONFIGURE_VLAN, action_map=action_map, error_map=error_map).execute_command(vlan_id=vlan_id) +{% endhighlight %} + diff --git a/_reference/9.4.0/quali-shell-framework.md b/_reference/9.4.0/quali-shell-framework.md new file mode 100644 index 0000000..8addeae --- /dev/null +++ b/_reference/9.4.0/quali-shell-framework.md @@ -0,0 +1,204 @@ +--- +layout: page +title: Quali’s Shell Framework (Python 2) +category: ref +order: 7 +comments: true +version: + - 9.4.0 +--- + + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this article, we will familiarize ourselves with the CloudShell shell framework and learn how to leverage it to develop and customize commands in shells based on Python 2. Note that this applies to 1st Gen and 2nd Gen shells. + +## Introduction + +Every CloudShell shell consists of a data model and a driver. The driver is written in python and can have python package dependencies. Quali’s officially released shells use a common set of python packages developed by Quali, which contain most of the logic of Quali shells, while the driver itself (the “.py” file inside the shell) is the thin layer that defines the interface with CloudShell along with the driver’s python requirements. + +Quali’s official shells have a granularity level of Vendor and OS. This means that each official shell supports all devices of a specific vendor and OS. The exact functionality that is exposed by the shell is defined in the relevant [shell standard](https://github.com/QualiSystems/cloudshell-standards/tree/master/Documentation). The structure of the python packages reflects this granularity – for example, any logic that is common to all networking devices resides in cloudshell-networking, while any Cisco-specific logic resides in cloudshell-networking-cisco, and any Cisco IOS-specific logic resides in cloudshell-networking-cisco-ios. It is possible to use Quali’s shell framework when creating your own shells or customizing existing ones. + +Note that using the framework is optional. To work with one or more of Quali framework’s python packages, you need to list them in your shell project’s requirements.txt file. Then, you can either write the code, which uses the packages, directly in the shell’s driver or create your own python packages and add them to the shell's requirements file. You can also load such custom python packages into your local [PyPi server repository](http://help.quali.com/Online%20Help/9.1/Portal/Content/Admn/Cnfgr-Pyth-Env-Wrk-Offln.htm?Highlight=pypi) on the Quali Server machine to make them available to your entire CloudShell deployment. + +**Important:** We don't recommend to modify Quali python packages as CloudShell may overwrite them if a newer package that has the same file name is published on the public PyPi repository. Alternatively, you’re welcome to create your own packages, using our python packages as a reference. + + +## Python package structure + +The following diagram shows the python classes used by the shell commands and their dependencies: + +![Python Package Structure Diagram]({{ site.baseurl}}/assets/python-package-structure.png) + +## Architecture + +The architecture of a Quali python shell comprises four inter-dependent elements: +* [Runners](#Runners) +* [Flows](#Flows) +* [Command Templates](#CommandTemplates) +* [Command Actions](#CommandActions) + +Runners execute Flows and process user inputs, and also define CLI and/or SNMP handlers, which are used in the Flows. Flows sequentially execute a number of Command Actions, while each Command Action runs a specific Command Template. + +An additional element that is used by the runners is the communication handler, which allows you to communicate with the device. For details about communication handlers, see this [section](#CommunicationHandlers). +  +## Key Entities + +There are several objects that must be initialized in the python driver, to allow you to work with Quali's infrastructure: +* **Communication Handlers** – These entities handle the communication between the shell and the device. The most common handlers are cli (`cloudshell-cli`) and snmp (`cloudshell-snmp`). These handlers must be initialized whenever calling a shell command, and passed to the runners. +* **cli** - A Python package that resides in the driver and is used to create the cli handler. This package provides an easy abstraction interface for CLI access and communication (Telnet, TCP, SSH etc.) for network devices. The CLI class instance is provided by `cloudshell.cli.cli`. It must be created when the driver is initializing, since it allows the shell to designate a single session pool for all of the shell’s commands. You are welcome to use the *_get_cli* helper from *driver_helper* mentioned above. *_get_cli* allows you to define the session pool’s size and idle timeout. +* **api** is an instance of the *cloudshell-automation-API*’s *CloudShellAPISession* class. It must be created on every command execution. This class has a helper named *_get_api*, which is also provided by the *driver_helper* mentioned above. +* **logger** is a logger object from *cloudshell-core*. It is recommended to use the *driver_helper’s get_logger_with_thread_id* function. +* **resource config** – Python implementation of the relevant Quali standard, which defines the shell’s attributes and default values. For example, a *GenericNetworkingResource* class that contains all attributes required by the networking standard. It can be easily created using the `create_networking_resource_from_context` method from +[cloudshell.devices.standards.networking.configuration_attributes_structure](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/standards/networking/configuration_attributes_structure.py). + + +Note that there is a helper method for each of these objects. For detailed information about each helper method, see [cloudshell.devices.driver_helper](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/driver_helper.py). + +For reference, see this [example](https://github.com/QualiSystems/Cisco-NXOS-Switch-Shell-2G/blob/dev/src/driver.py). + +### Communication Handlers + +The most common ways to communicate with the device are via: +* CLI – cloudshell-cli +* SNMP – cloudshell-snmp + +These handlers need to be initialized and passed to the runners. + +#### CLI Handler + +The CLI handler includes all the typical CloudShell CLI attributes you would need in order to communicate and work with a device modeled in CloudShell. For example, it knows how to retrieve the device’s username and password, create a session, determine what kind of session to initiate, etc. For reference, see [cisco_cli_handler.py](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/cli/cisco_cli_handler.py) and [cisco_command_modes.py](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/dev/cloudshell/networking/cisco/cli/cisco_command_modes.py). + +There is a nice guide on how to use *cloudshell-cli* [here](https://github.com/QualiSystems/cloudshell-cli/blob/dev/README.md). However, to simplify the usage of the CloudShell CLI, we implemented a *CliHandlerImpl* base class located [here](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/cli_handler_impl.py). + +In the child class, you only need to implement these methods and properties: + +* **enable_mode** – is a property that returns the *CommandMode* class, which enables you to use the **Enable** mode, a mode that grants you root admin access to a Linux device. [Example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/cli/cisco_command_modes.py#L46). + +     For details and implementation examples of *CommandMode*, click [here](https://github.com/QualiSystems/cloudshell-cli/blob/dev/README.md). + +* **config_mode** – is a property that returns the *CommandMode* class, which enables you to use the **Configuration** mode, which allows you to configure the device’s settings. [Example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/cli/cisco_command_modes.py#L83). +* **on_session_start** – is a method that contains the commands you want to automatically execute at the start of each session. [Example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/cli/cisco_cli_handler.py#L80). + +Example: Cli handler that requires the parameters `cli`, `resource_config`, `logger` and `api`: + +![CLI Handler Code Example]({{ site.baseurl}}/assets/cli-handler-code-example.png) + +Note that the first parameter, `cli`, needs a CLI instance from *cloudshell.cli.cli* to be initiated. For details about these parameters, see the [Key Entities](#KeyEntities) section. + +#### SNMP Handler + +The SNMP handler provides SNMP communication with the device. Like the CLI handler, the main responsibilities of the SNMP handler are: +* Initialize *QualiSnmp*. It does this by extracting and preparing the *QualiSnmp* parameters provided by the resource driver’s context +* Analyze Enable and Disable SNMP attributes from the command context +* Trigger the appropriate flow. +To initialize the `SNMPHandler` object, you need to pass the following objects: `resource_config`, `logger`, `api`. To use the handler, you must implement the following properties in your child class: + +          • _create_enable_flow + +          • _create_disable_flow + +For reference, see [cisco_snmp_handler](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/snmp/cisco_snmp_handler.py). For more information, see the [Flows](#Flows) section. +  +## Runners + This is an abstract class that includes generic implementations for preparing and validating the required parameters. For example, when running the Save command in CloudShell Portal, the runner must validate the folder path provided by the sandbox end-user. Typically, the shell extracts the data for the required parameters from the resource command’s context and user inputs. + +The necessary interfaces are already implemented in the base Runners. However, you can implement your own base runner, e.g. *ConnectivityRunner* implements the *ConnectivityOperationsInterface* interface. The runner’s interfaces contain commands that are triggered by the resource driver. + +Overall, we have six Runners, all base classes and their interfaces are located in [cloudshell-networking-devices](https://github.com/QualiSystems/cloudshell-networking-devices/tree/dev/cloudshell/devices/runners): + +**Note:** All runners except for Autoload Runner require the cli-handler parameter to be passed to the runner while it is being initialized. For example, see [this](https://github.com/QualiSystems/Cisco-IOS-Shell/blob/5.0.2/src/cisco_ios_resource_driver.py#L86-L87). + +* **[Connectivity Runner](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/runners/connectivity_runner.py)** – Uses multithread logic to speed up the VLAN configuration on the device, especially when the resource needs to undergo a huge request that involves multiple, concurrently run actions. To initialize this runner, you have to provide the logger and cli_handler objects (described in [Key Entities](#KeyEntities)). Use the `apply_connectivity_changes` method to start. The following properties have to be implemented: + +          • add_vlan_flow + +          • remove_vlan_flow + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/runners/cisco_connectivity_runner.py). + +* **[Configuration Runner](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/runners/configuration_runner.py)** – Prepares and validates the provided path for the `save`, `restore`, `orchestration save` and `orchestration restore` commands. To initialize this runner, you have to pass the *logger*, *resource_config*, *cli_handler* and *api* objects (described in Key Entities). The following properties have to be implemented: + +          • save_flow + +          • restore_flow + +          • file_system – a default filesystem value, see example below. + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/runners/cisco_configuration_runner.py). + +* **[Firmware Runner](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/runners/firmware_runner.py)** – This runner serves as a Configuration Runner, and also validates the firmware’s file path. To initialize this runner, you need to pass the *logger* and *cli_handler* objects. The following property has to be implemented: + +          • load_firmware_flow + +     For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/runners/cisco_firmware_runner.py). + +* **[Run Command Runner](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/runners/run_command_runner.py)** – As you can see from the name, this Runner handles the `Run Custom Command` and `Run Custom Config Command` driver methods, and doesn’t require anything else to implement. However, if you want to customize the `run_command_flow` property, you are welcome to override it. To initialize this runner, just pass the *logger* and *cli_handler* objects. + +     For reference, see this [example](https://github.com/QualiSystems/Cisco-IOS-Shell/blob/5.0.2/src/cisco_ios_resource_driver.py#L79-L87) on how to create the run command runner. + +* **[State Runner](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/runners/state_runner.py)** – This runner is very similar to the Run Custom Command Runner as it doesn’t require any additional implementations. It contains implementations for the `Health Check` and `Shutdown` commands. To initialize this runner, you need to pass the *logger*, *api*, *cli_handler* and *resource_config* objects mentioned in [Key Entities](#KeyEntities). + +     For reference, see this [example](https://github.com/QualiSystems/Cisco-IOS-Shell/blob/5.0.2/src/cisco_ios_resource_driver.py#L371-L379) on how to create the state runner. + +* **[Autoload Runner](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/runners/autoload_runner.py)** – Discovers the device’s hardware structure and general details, such as the firmware version and model. This runner requires the *autoload_flow* property to be implemented. + +     To initialize this runner, just pass the `resource_config` and `snmp_handler` objects. For reference, see this [example](https://github.com/QualiSystems/Cisco-IOS-Shell/blob/5.0.2/src/cisco_ios_resource_driver.py#L60-L65) on how to create the autoload runner. + +## Flows + +Flow is an organized sequence of Command Actions. All the base Flows are located in the [cloudshell-networking-devices](https://github.com/QualiSystems/cloudshell-networking-devices/blob/dev/cloudshell/devices/flows/action_flows.py) Python package. They are based on the *BaseFlow* interface located in the same place. + +Most shells include the following flows: +* Save Configuration Flow +* Restore Configuration Flow +* Add Vlan Flow +* Remove Vlan Flow +* Load Firmware Flow +* Run Command Flow +* Shutdown Flow +* Enable Snmp Flow +* Disable Snmp Flow + +Note that *Run Command Flow* doesn’t require you to implement the `execute_flow` method as it’s already implemented there. The only difference between the *Run Command Flow* and the rest is a set of parameters. Run Command Flow has a generic approach to all devices, and doesn’t require any specific implementation. + +For reference, see these [files](https://github.com/QualiSystems/cloudshell-networking-cisco/tree/5.2.16/cloudshell/networking/cisco/flows). + +## Command Templates + +A Command Template is a constant object of the *CommandTemplate* class, which is located in [cloudshell-cli](https://github.com/QualiSystems/cloudshell-cli/blob/dev/cloudshell/cli/command_template/command_template.py). It contains three elements: +* **command** – A command you want to send to the device. Supports formatting parameters. For example: + +{% highlight bash %}copy {src} {dst} [vrf {vrf}]{% endhighlight %} + +* **action map** – A dictionary of regex patterns (as keys) and lambda functions (as values). To illustrate this, when you get a “(yes/no)” prompt, you can send an appropriate command by specifying the required lambda function. For example: + +{% highlight bash %} +r'\(y/n\)': lambda session, logger: session.send_line('y', logger) +{% endhighlight %} + +* **error map** – Similar to action map but for errors. It is a dictionary of regexp pattern and error message test pairs. For example: + +{% highlight bash %} +r"[Ii]nvalid\s*([Ii]nput|[Cc]ommand ": “Override mode is not supported” +{% endhighlight %} + +     For reference, see this command template [file](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/command_templates/configuration.py). + +## Command Actions + +Command Action is a method or function (depending on your implementation) that runs one or more Command Templates. The [CommandTemplateExecutor](https://github.com/QualiSystems/cloudshell-cli/blob/dev/cloudshell/cli/command_template/command_template_executor.py) is provided for running command templates. For reference, see this [example](https://github.com/QualiSystems/cloudshell-networking-cisco/blob/5.2.16/cloudshell/networking/cisco/command_actions/add_remove_vlan_actions.py). + +For example, if we want to execute the CONFIGURE_VLAN command template: + +{% highlight bash %} +CONFIGURE_VLAN = CommandTemplate("vlan {vlan_id}", error_map=OrderedDict({"%.*\.": "Error creating vlan")})) +{% endhighlight %} + +We need to (1) pass the `session` object, command template and action/error maps to the *CommandTemplateExecutor*, and (2) `vlan_id` to the `execute_command` method. As illustrated below: + +{% highlight bash %} +CommandTemplateExecutor(session, CONFIGURE_VLAN, action_map=action_map, error_map=error_map).execute_command(vlan_id=vlan_id) +{% endhighlight %} + diff --git a/_reference/9.4.0/resource-scripts.md b/_reference/9.4.0/resource-scripts.md new file mode 100644 index 0000000..0449ce8 --- /dev/null +++ b/_reference/9.4.0/resource-scripts.md @@ -0,0 +1,150 @@ +--- +layout: page +title: Resource Scripts +category: ref +order: 15 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Resource scripts allow you to add automation to specific sandbox components. These scripts are intended to add simple functionality, or to be used for testing and debugging activities. Note that in order to add automation to a shell, the best practice is to use the component's driver. + +### Using the script_helper + +Resource scripts get information from the sandbox component using the script_helper. + +**To use the script helper:** + +Import the cloudshell-automation-api python package and add it to your script, as illustrated in the example below. Note that the package is automatically imported when your sandbox starts. In this example, the following code gets an object that contains all of the sandbox’s information: + +{% highlight python %} +import cloudshell.helpers.scripts.cloudshell_scripts_helpers as script_help +{% endhighlight %} + +Note that to execute this code, you will need to include a *requirements.txt* file in your script. If you want to write your own packages and use them in your script, make sure to place them in the local PyPi Server repository on the Quali Server machine. For details, see CloudShell Help's PyPi Server - Managing Python Driver and Script Dependencies. + +To facilitate writing and debugging activities, it is recommended to use advanced IDEs such as PyCharm, which provide autocomplete functionality, as illustrated below. + +![Resource information]({{site.baseurl}}/assets/resource_context.png){:class="img-responsive"} + +### Accessing the sandbox component + +Use the *get_resource_context* method to access and use the sandbox component in your resource script. + +For example, let's assume we want to get metadata information from the component, such as name and address: + +{% highlight python %} +resource_name = script_help.get_resource_context_details().name +resource_address = script_help.get_resource_context_details().address +{% endhighlight %} + +Or to get information from attributes on the component: + +* For global attributes, use the attributes element. For example, "Region" and "Execution Server Selector": + +{% highlight python %} +resource_region = script_help.get_resource_context_details().attributes.Region +resource_ess = script_help.get_resource_context_details().attributes["Execution Server Selector"] +{% endhighlight %} + +* For namespaced attributes (i.e. attributes that exist on a 2nd Gen shell only), provide the full attribute name, including the namespace. For example, "Vendor" and "OS Version": + +{% highlight python %} +resource_vendor = script_help.get_resource_context_details().attributes['CS_Switch.Vendor'] +resource_os_version = script_help.get_resource_context_details().attributes['CS_Switch.OS Version'] +{% endhighlight %} + +### Using API from the resource script + +To use the API, create a session variable that uses the helper's get_api_session method: + +{% highlight python %} +session = script_help.get_api_session() +{% endhighlight %} + +### Getting reservation context details + +The *get_reservation_context_details* helper provides the reservation context. + +![get_reservation_context_from_resource_scripts]({{ site.baseurl}}/assets/resource_reservation_context_details.png) + +To get this object, include this line in your script: + +{% highlight python %} +from cloudshell.helpers.scripts.cloudshell_scripts_helpers import get_reservation_context_details +{% endhighlight %} + +Note that starting with CloudShell 9.2, you can also get the CloudShell user who ran the command using the *get_reservation_context_details* helper. + +For example: + +{% highlight python %} +user = get_reservation_context_details().running_user +{% endhighlight %} + +### Returning outputs from a resource script + +In order to return outputs, use `print`. This applies to flat scripts and methods nested within resource scripts. + +The script standard output is returned as the command result. However, if an exception is raised, or if a non-zero process result code is returned, the execution will be considered a failure. As a side note, in CloudShell, the output of a script is displayed in the **Output** console in the sandbox workspace, so whatever you print in your script will find its way there. + +For example: + +{% highlight python %} +import cloudshell.helpers.scripts.cloudshell_scripts_helpers as script_help + +def print_output(): + resource_address = script_help.get_resource_context_details().address + print(resource_address) + {% endhighlight %} + +### Associating a resource script to a CloudShell resource + +1) Place the python script(s) and requirements.txt files in a folder. + +2) From within the folder, select the two files and zip. + +3) In CloudShell Portal's **Scripts** management page, open the **Resource Scripts** page and add the zip file. + +4) **Edit** the script and from the **Models** drop-down list, select the models of the resources and services. + +![Resource information]({{site.baseurl}}/assets/resource_script-add-to-cloudshell.png){:class="img-responsive"} + +5) Click **Save**. + +### Example + +In this example, we use the CloudShell Automation API to get the resource's vendor and OS version: + +**requirements.txt** + +{% highlight bash %} +cloudshell-automation-api>=9.3,<9.4 +{% endhighlight %} + +**__main__.py** + +{% highlight python %} +import cloudshell.helpers.scripts.cloudshell_scripts_helpers as script_help + +session = script_help.get_api_session() + +resource_vendor = script_help.get_resource_context_details().attributes['CS_Switch.Vendor'] +resource_os_version = script_help.get_resource_context_details().attributes['CS_Switch.OS Version'] + +session.WriteMessageToReservationOutput( + reservationId=script_help.get_reservation_context_details().id, + message='the resource vendor is {}'.format(resource_vendor) +) + +session.WriteMessageToReservationOutput( + reservationId=script_help.get_reservation_context_details().id, + message='the resource OS version is {}'.format(resource_os_version) +) +{% endhighlight %} + + diff --git a/_reference/9.4.0/shellfoundry-intro.md b/_reference/9.4.0/shellfoundry-intro.md new file mode 100644 index 0000000..5ff5dab --- /dev/null +++ b/_reference/9.4.0/shellfoundry-intro.md @@ -0,0 +1,395 @@ +--- +layout: page +title: Shellfoundry +category: ref +order: 5 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + + +Shellfoundry is a command-line utility that allows you to easily create, package and install Shells in CloudShell. It allows creating basic Shells or Shells based on a common 1st or 2nd generation Shell template. + +**Notes:** + +* In order to use shellfoundry on an offline computer, you will need to first download the shellfoundry templates locally and configure shellfoundry to run in offline mode. For details, see [Downloading shellfoundry templates](#downloading-shellfoundry-templates). +* Shellfoundary cannot work if there's a proxy server present between the shellfoundry machine and the remote Quali Server machine. + + +In this article: + +* [Usage](#usage) +* [Version History](#version-history) + + +### Usage + + +#### Installation +This command installs Shellfoundry on your computer. For more information, see [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html). + +**Syntax:** + +{% highlight bash %} +pip install shellfoundry +{% endhighlight %} + +To install a specific shellfoundry version, run: `pip install shellfoundry==` + +For the version history, click [here](#version-history). + + +#### Upgrade + +If you already have shellfoundry installed on your computer, run this command to upgrade it to the latest version. For more information, see [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html). + +**Note:** To prevent backwards compatibility issues, old versions of shellFoundry are disabled with every new shellfoundry release. + +**Syntax:** + +{% highlight bash %}python -m pip install shellfoundry ––upgrade{% endhighlight %} + + +#### Configuring CloudShell settings + +(Required) This command sets the CloudShell Portal settings and user access credentials in Shellfoundry. Note that in offline mode, the `shellfoundry list` command lists the shell templates residing locally in the folder defined in the shellfoundry config's template_location attribute. For more information, see [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html). + +**Syntax:** + +Run this command from the Shell’s root folder. + +{% highlight bash %}shellfoundry config {% endhighlight %} + +**Example - Setting the username:** + +{% highlight bash %}shellfoundry config username admin{% endhighlight %} + +The following keys are available: +* **username**: CloudShell username. For example: “admin”. +* **domain**: CloudShell domain. Note that for 2nd Generation Shells, the domain must be “Global”. +* **github_login**: GitHub username. To be used to download shellfoundry templates via `shellfoundry get_templates`. +* **github_password**: GitHub user password. To be used to download shellfoundry templates via `shellfoundry get_templates`. +* **defaultview**: Set the default view. Possible values are: **gen**, **gen2**, **all** and **layer1**. Default is **gen2**. +* **online_mode**: Shellfoundry computer's mode (online or offline). Online mode (`True`) is the default. in online mode, shellfoundry templates on GitHub are used, while for offline mode, you will need to copy the shellfoundry templates to your local machine. For offline mode, use `template_location` to define the local templates folder. +* **author**: The author to be specified on the shell (in the shell's metadata). +* **template_location**: (Required if `online_mode` is set to `False`) File system path to the folder containing the offline shell templates. Alternatively, you can specify the template location using "local:" when running 'shellfoundry new' in command-line. +* **password**: CloudShell password (encrypted). +* **host**: The hostname or IP address of the CloudShell Portal machine. +* **port**: The port to be used for Quali API. Default is “9000”. + + +#### Creating a Shell + +The `new` command creates a shell from a shell template. + +**Syntax:** + +Run this command in the Shell’s root folder. + +{% highlight bash %}shellfoundry new {% endhighlight %} + +**Options:** + +* **-****-template**: Creates a shell from a specific shellfoundry template, featuring the template's settings, attributes and driver commands. For details, see [Modeling Shells with TOSCA]({{site.baseurl}}/shells/{{pageVersion}}/modeling-the-shell.html). +
**Note:** If you don't specify a template, shellfoundry will create the shell from the **gen2/resource** template. Use this if you want to create a Shell to customize or experiment on. For more information, see [The Shell Project Guide]({{site.baseurl}}/shells/{{pageVersion}}/the-shell-project.html). +* **-****-version**: Creates a shell based on a specific shell version. If you don't specify the version, shellfoundry will create the shell using the latest shell version that is supported by your CloudShell installation. +* **-****-python**: Determines the python version of the shell. Options are "2" or "3" (Default is 2). + +**Examples:** + +Generic resource shell based on python 3: + +{% highlight bash %} +shellfoundry new my-basic-shell --python 3 +{% endhighlight %} + +Networking switch shell version 5.0.2: + +{% highlight bash %} +shellfoundry new my-switch-shell --template gen2/networking/switch --version 5.0.2 +{% endhighlight %} + + +#### Creating a Shell from a local template + +This section explains how to create Shells using a Shell template that isn’t online for some reason. For example, you want to use a Shell template you are currently developing. + +**Syntax:** + +Run this command from the directory that will contain the new Shell: + +{% highlight bash %}shellfoundry new --template local: {% endhighlight %} + +The path can be a URL to the Shell template's zip package on GitHub or the filesystem path (prefixed by `local:./`) to the extracted zip folder: + +![Shell Commands]({{site.baseurl}}/assets/download_shell_zip.png) + +**Example:** +{% highlight bash %}shellfoundry new my-service-ext --template local:C:\Temp\shell-pdu-standard-master {% endhighlight %} + +     The new shell is added to the path from which you ran the `shellfounfry new` command. + + +#### Deleting a shell from Cloudshell + +Starting with CloudShell 9.1, it is possible to delete shells that are installed on CloudShell. Note that you cannot delete shells that have resources. + +**Syntax:** + +{% highlight bash %}shellfoundry delete ""{% endhighlight %} + +Where the shell's name is the name of the shell, as displayed in CloudShell Portal's **Manage - Shells** page. + +**Example:** + +{% highlight bash %}shellfoundry delete "Juniper JunOS Switch Shell 2G"{% endhighlight %} + + +#### Downloading shellfoundry templates + +This command downloads all shellfoundry templates from GitHub, which you can use to create shells in offline mode. Note that shellfoundry uses GitHub API to fetch the templates, so you will need to set a GitHub user (via `shellfoundry config`) to grant shellfoundry unrestricted access to GitHub API. For details, see this GitHub Developer article. + +**Syntax:** + +Run this command from the directory that will contain the shell templates: + +{% highlight bash %}shellfoundry get_templates {% endhighlight %} + +Optionally, add `--output_dir=""` to set a different containing folder. + +**Example:** + +{% highlight bash %}shellfoundry get_templates 9.1 --output_dir="c:\users\steven.g\shell templates"{% endhighlight %} + +Shellfoundry downloads the latest template versions that are compatible with the latest patch of the specified CloudShell version. + + +#### Listing available Shell templates + +This command lists the 1st and 2nd generation Shell templates you can use for your new Shell. For more information, see [Modeling Shells with TOSCA]({{site.baseurl}}/shells/{{pageVersion}}/modeling-the-shell.html). + +Note that in offline mode, the command lists the shell templates residing locally in the folder defined in the shellfoundry config's `template_location` attribute. + +**Syntax:** + +{% highlight bash %}shellfoundry list{% endhighlight %} + +To view templates of a specific type, add the appropriate flag to the command: `--gen1`, `--gen2`, `--layer1`, `--all` (lists all available templates). + + +#### Listing versions of a Shell template + +To display a list of the versions for a given template, run the following in command-line: + +{% highlight bash %}shellfoundry show {% endhighlight %} + +The versions are displayed in descending order from latest to earliest. + + +#### Packaging a Shell + +This command packs the shell's source code, data model and configuration into a ZIP package, which can be uploaded into CloudShell. For additional information, see [Deploying to Production]({{site.baseurl}}/shells/{{pageVersion}}/deploying-to-production.html). + +**Syntax:** + +Run this command from the Shell’s root folder. + +{% highlight bash %}shellfoundry pack{% endhighlight %} + +A ZIP package is created in the Shell’s *dist* directory with the name "nutshell.zip". + +**Note:** The `pack` command requires the presence of a *shell.yml* file, which is created by default in Shells created using Shellfoundry. However, if your shell was created elsewhere, make sure to add a *shell.yml* file with the following structure: + + {% highlight bash %} + ###shell.yml + + shell: + + name: nutshell +{% endhighlight %} + + +#### Packaging a shell's dependencies + +This command creates a zip file in the shell project's `dist` folder that includes the shell's dependencies. This is especially useful for offline mode and during shell development. + +Run this command from the Shell’s root folder. + +**Syntax:** + +{% highlight bash %} +shellfoundry dist +{% endhighlight %} + +By default, the command downloads the dependencies from public PyPi. However, if you are developing the shell and have some custom dependencies that are not in public PyPi, add the `--enable_cs_repo` flag to also include the shell's dependencies from your local PyPi repository folder. + + +#### Packaging and importing a Shell into CloudShell + +This command creates a distributable zip file for the Shell and imports it into CloudShell using the CloudShell Portal and user defined by the `shellfoundry configure` command. For more information, see [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html). + +**Syntax:** + +Run this command from the Shell’s root folder. + +{% highlight bash %}shellfoundry install{% endhighlight %} + + +#### Generating the Shell’s data model file + +The shell’s data model (*data_model.py* file) consists of the standard specifications and the extended data model, which is defined in the *shell-definition.yaml* file. The shell's data model is mainly used to work with resource attributes and implement the Auto-discovery process. For additional information, see [Managing the Shell’s Data Model]({{site.baseurl}}/shells/{{pageVersion}}/generating-shell-data-model.html). + +After importing the _data_model_, PyCharm (and some other IDEs) will recognize the docstring code-hint annotations and will enable autocomplete as you can see below: + +![Directory Structure]({{site.baseurl}}/assets/auto_complete_demo.png) + +**Syntax:** + +Run this command from the Shell’s root folder. + +{% highlight bash %} shellfoundry generate {% endhighlight %} + + +#### Customizing a 2nd Gen Shell + +This command downloads the source code of the Shell you wish to customize to your local machine and updates the Shell’s Author with the author specified in Shellfoundry. Note that extending official Shells (Shells that were released by Quali) will remove their official tag. For more information, see [Customizing a 2nd Gen Shell]({{site.baseurl}}/shells/{{pageVersion}}/customizing-shells.html). + + +**Syntax:** + +Run this command from the directory that will contain the new Shell: + +{% highlight bash %} +shellfoundy extend +{% endhighlight %} + +The path can be a URL to the source code of the shell's desired release on [Quali Community's Integrations](https://community.quali.com/integrations) page or the filesystem path (prefixed by `local:./`) to the extracted source code folder: + +![Shell Commands]({{site.baseurl}}/assets/download_shell_source_code.png) + + +**Examples:** + +     Extending a Shell residing on GitHub: +     {% highlight bash %} +shellfoundry extend https://github.com/QualiSystems/Cisco-NXOS-Switch-Shell-2G/archive/2.0.0.zip +{% endhighlight %} + +     Extending a local Shell: +     {% highlight bash %} +shellfoundry extend local:C:\temp\Cisco-NXOS-Switch-Shell-2G-2.0.0 +{% endhighlight %} + +Before extending a local Shell, make sure the Shell's destination folder is different from the original Shell's root folder. + +### Version History + +**1.2.12 (2019-10-16)** +* Added `--python` attribute to `new` command that allows setting the python version of the shell + +**1.2.10 (2019-04-22)** +* Added attribute to `dist` command that allows using locally installed shell dependencies when packaging the dependencies zip +* Added support for pip 19.1 + +**1.2.8 (2019-03-06)** +* Fixed issue in `generate` command after renaming root folder +* Added `get_templates` command for downloading shellfoundry templates for offline mode +* Added `delete` command for deleting shells installed on CloudShell (supported in CloudShell 9.1 GA and up) +* Implemented the capability to generate shell documentation based on the template +* Set strict python version + +**1.2.4 (2018-09-26)** +* Removed unnecessary *cloudshell-automation-api* dependency from requirements +* Set static version for package 'click' in *requirements.txt* file. click==6.7 + +**1.2.2 (2018-08-16)** +* Fixed bug related to template verifications and standards compatibilities + +**1.2.1 (2018-08-13)** +* Added dynamical determination of minimal CloudShell version from templates + +**1.2.0 (2018-07-26)** +* Extended `new` command behaviour for offline mode +* Added validation to check if the template and standard versions are compatible + +**1.1.9 (2018-05-03)** +* Added offline mode functionality +* Fixed typo in `pack` command behavior +* Added new online template for Cloud Provider +* Shellfoundry now packs deployment options if any exist +* Added limitation installing a gen2 shell (regular/service) into a non-Global domain + +**1.1.5 (2018-03-01)** +* Added new online template for Traffic Generator Controller Service +* Added new 2nd Gen online template for Traffic Generator Chassis + +**1.1.2 (2018-01-09)** +* Enhanced `extend` command logic +* Added new online templates +* Added specific error message to Layer 1 Shell `pack` and `install` commands + +**1.1.0 (2017-10-30)** +* Added `author` field to shellfoundry configuration +* Added `extend` command behavior +* Added verification when upgrading an official shell to unofficial +* Fixed some inconsistencies relating to the `extend` and `new` commands, specifically around the shell name + +**1.0.3 (2017-06-28)** +* `list` command aborts if there is a new major version on pypi +* Old Shellfoundry versions are NOT supported anymore. Therefore, in order to upgrade to the newest version, please run this command: +{% highlight bash %} pip install shellfoundry -U {% endhighlight %} +* `new` command aborts if there is a new major version on pypi +* `new` command now conforms to CloudShell naming rules +* `list` command now shows templates that are installable on your cloudshell +* `new` command now creates the latest version of the template that matches the standards installed on your cloudshell +* When running `new` or `list` commands, a notification is displayed if a new Shellfoundry version is available + +**0.2.7 (2017-05-16)** +* Shellfoundry now packs *categories.xml*, if it exists + +**0.2.6 (2017-03-14)** +* Minor bug fixes + +**0.2.5 (2017-03-13)** +* **gen2/resource** is the now the default template for the `new` command instead of **gen1/resource** +* `list` command filtering parameters have changed (legacy => **gen1**, TOSCA => **gen2**) +* Added another filtering parameter: **--layer1** +* Minimum CloudShell version column appears in the `list` command's output table +* **gen2** is now the default view for list command +* `config` now echoes all default configurations if they have not been overridden by the user +* `config` command now encrypts the password +* `show` command was added to display all available versions of a template +* A new option was added to the `new` command called **--version**. It enables template versioning on Shellfoundry. +* Fixed a bug with the `config` command, which caused Shellfoundry to crash if a config file was missing +* `list` command is now able to filter results based on shell type (**--tosca**, **--legacy**, **--all**) +* `config` command was added to allow setting configuration globally for all Shells in addition to local configuration +* Pack Shell icon if specified in the *shell-definition.yml* file under `metadatatemplate_icon` for TOSCA based shells +* Update reference to *cloudshell-rest-api 7.2.0.7* to use PUT method in update shell +* TOSCA support was added to the `pack` and `install` commands +* `generate` command was added to generate the Shell driver's data model in Python + +**0.0.32 (2016-08-10)** +* Dependency for git was removed +* Local Shell templates are supported +* Proxy support was added for access to github + +**0.0.31 (2016-08-04)** +* git prerequisite were removed. Shellfoundry now works even if git is not preinstalled + +**0.0.28 (2016-07-07)** +* Bug pertaining to the installation of packages in CloudShell was fixed + +**0.0.26 (2016-06-23)** +* Images copied to the *DataModel* folder (Issue #21) + +**0.0.17 (2016-05-25)** +* Fixed anj error message that is displayed when `install` command fails in logging in into CloudShell + +**0.0.1 (2016-05-15)** +* First release on PyPI + + diff --git a/_reference/9.4.0/short-dev-videos.md b/_reference/9.4.0/short-dev-videos.md new file mode 100644 index 0000000..850e812 --- /dev/null +++ b/_reference/9.4.0/short-dev-videos.md @@ -0,0 +1,83 @@ +--- +layout: page +title: Short Development Videos +category: ref +order: 14 +comments: true +version: + - 9.4.0 +--- + +Use this article to watch the different instructional videos and tutorials embedded throughout the dev guide. + +In this article: + +* [What are shells?](#ShellsIntro) +* [Modeling shells](#ShellsModeling) +* [Creating or extending a shell](#ShellsCreateOrExtend) +* [Adding shell commands](#ShellsCommands) +* [Adding attributes](#ShellsAttributes) +* [Debugging shells](#ShellsDebugging) + + + +### What are shells? + +This short video introduces the concept of shells and their use. + + + + + +### Modeling shells + +This video discusses the structure and logic of CloudShell shells. + + + + + +### Creating or extending a shell + +This video will assist you in deciding whether to create a new shell or extend an existing one. + + + + + +### Adding shell commands + +#### Introduction to shell commands + +This video introduces shell commands and their basic structure. + + + +#### Adding basic commands + +This tutorial demonstrates how to add basic commands that print information, use command inputs, and use the resource 'context' object. + + + +#### Using API in shell commands + +This tutorial discusses the use of CloudShell Automation API in shell commands and demonstrates API methods that apply to shells. + + + + + +### Adding attributes + +This tutorial demonstrates how to add attributes to the shell, including custom attributes like root, sub-model and discovery attributes, and global attributes that already exist in CloudShell. + + + + + +### Debugging shells + +This tutorial demonstrates how to debug your shell using the mock open source python package. + + + diff --git a/_reference/9.4.0/working-with-sandbox-metadata.md b/_reference/9.4.0/working-with-sandbox-metadata.md new file mode 100644 index 0000000..7aad593 --- /dev/null +++ b/_reference/9.4.0/working-with-sandbox-metadata.md @@ -0,0 +1,134 @@ +--- +layout: page +title: Custom Sandbox Metadata +category: ref +order: 19 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +Starting with CloudShell 9.2, it is possible to store information for later use in the sandbox using the API. This feature is related to this idea. + +This capability can be used to store sandbox-specific information that is required for your orchestration processes and shells, like user details, unique IDs returned from 3rd party systems, or data created during the setup process that is needed for teardown. + +The data is stored as key-value pairs in a container that is attached to the sandbox and can only be inserted/retrieved via CloudShell Automation API. Note that we don't impose any limitations on the data and its format, and the data isn’t stored encrypted. Also, for completed sandboxes, the data is kept but cannot be modified. + +The following API methods are provided: + ++ *SetSandboxData*: sets the key-value pairs in the custom data container. +* *GetSandboxData*: retrieves custom data from the sandbox +* *ClearSandboxData*: clears the custom data container + +Note that only sysadmins and domain admins can get/set/clear the sandbox data. + +So let's start by setting two key-value pairs: Key1 and Key2: + +{% highlight python %} +import cloudshell.api.cloudshell_api as api + +session = api.CloudShellAPISession('localhost', 'admin', 'admin', 'Global') + +res_id = 'f45905f5-1b67-431b-a813-6a4873966245' + +data1 = api.SandboxDataKeyValue('Key1', 'Value1') +data2 = api.SandboxDataKeyValue('Key2', 'Value2') + +all_data = [data1, data2] + +session.SetSandboxData(res_id, all_data) +{% endhighlight %} + +Now that we have some custom data on the sandbox, let's learn how to retrieve it: + +{% highlight python %} +import cloudshell.api.cloudshell_api as api + +session = api.CloudShellAPISession('localhost', 'admin', 'admin', 'Global') + +res_id = 'f45905f5-1b67-431b-a813-6a4873966245' + +data = session.GetSandboxData(res_id) + +for keyValue in data.SandboxDataKeyValues: + print keyValue.Key + " :: " + keyValue.Value +{% endhighlight %} + +And finally, let's learn how to remove the data we stored on the container: + +{% highlight python %} +import cloudshell.api.cloudshell_api as api + +session = api.CloudShellAPISession('localhost', 'admin', 'admin', 'Global') + +res_id = 'de035c9c-75b0-4eca-b797-698b0f26675f' + +session.ClearSandboxData(res_id) +{% endhighlight %} + +## Using custom sandbox data in JSON format + +JSON is a very common data format so we figured we'd include some code samples. + +**Storing custom sandbox data in JSON format** + +{% highlight python %} +import cloudshell.api.cloudshell_api as api +import json + +session = api.CloudShellAPISession('localhost', 'admin', 'admin', 'Global') + +res_id = 'f45905f5-1b67-431b-a813-6a4873966245' + +with open("C:\Json Examples\jsonExample1.json", "r") as myfile: + data_file = myfile.read() + + +with open("C:\Json Examples\jsonExample2.json", "r") as myfile: + data_file2 = myfile.read() + +big_data = api.SandboxDataKeyValue('jsonExample1', data_file) +big_data2 = api.SandboxDataKeyValue('jsonExample2', data_file2) + +all_data = [big_data, big_data2] + +session.SetSandboxData(res_id, all_data) +{% endhighlight %} + +**Retrieving custom sandbox data in JSON format** + +{% highlight python %} +import cloudshell.api.cloudshell_api as api +import json + +# method to Get the sandbox data by key +def GetSandboxDataValueByKey(sandboxData, key): + value = [attr.Value for attr in sandboxData.SandboxDataKeyValues if attr.Key == key] + if value.__len__() >= 1: + return value[0] + +# creating a cloudshell session +session = api.CloudShellAPISession('localhost', 'admin', 'admin', 'Global') + +sandbox_id = 'f45905f5-1b67-431b-a813-6a4873966245' + +# using the session to get all the sandbox data from a specific sandbobx +sandbox_data = session.GetSandboxData(sandbox_id) +key = 'jsonExample1' + +# getting a specific value from the sandbox data using a key +wanted_value = GetSandboxDataValueByKey(sandbox_data, key) + +# formatting the value as json +wanted_value_as_json = json.loads(wanted_value) + +# manipulating the json - getting to specific info inside the json +specific_inner_value = wanted_value_as_json['widget']['image'] + +# printing the value as string +print json.dumps(specific_inner_value) +{% endhighlight %} + diff --git a/_shells/9.4.0/1st-gen-shells.md b/_shells/9.4.0/1st-gen-shells.md new file mode 100644 index 0000000..6755b6b --- /dev/null +++ b/_shells/9.4.0/1st-gen-shells.md @@ -0,0 +1,34 @@ +--- +layout: page +title: 1st Gen Shells +category: tut +order: 17 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +1st Gen Shells are imported as CloudShell packages that contain the data model and driver for the intended sandbox element. 1st Gen Shells allow extensive control of the family and model, and therefore are not standardized. While they allow maximal flexibility, when using them, some Shell management capabilities may not be available. + +However, unlike 2nd Gen Shells, 1st Gen Shells carry several limitations: + +* Shell integrity isn’t enforced by CloudShell +* No way to uninstall a shell +* Versions are not managed +* Data model is shared, which may cause conflicts between shells + +Today you can still find 1st Gen Shells in our community but all new official shells are released as 2nd Gen Shells only. "2nd Gen" is the “code name” for shell entity, a new feature introduced in 8.0. For more information about 2nd Gen Shells, see the CloudShell online help. + +Note that the same 2nd Gen Shell standards apply to 1st Gen (but are not enforced by CloudShell). There are 1st Gen Shell templates available for Shell development in [Shellfoundry]({{site.baseurl}}/reference/{{pageVersion}}/shellfoundry-intro.html). However, it is recommended to develop only 2nd Gen Shells using the official Shell templates and guidelines. + +The following articles relate to 1st Gen Shells: + +* [Converting 1st Gen Shells into 2nd Gen]({{site.baseurl}}/reference/{{pageVersion}}/migrating_1st_gen_shells.html) + +* [Categories in 1st Gen Service Shells]({{site.baseurl}}/reference/{{pageVersion}}/associating-service-categories.html) + + + diff --git a/_shells/9.4.0/common-driver-recipes.md b/_shells/9.4.0/common-driver-recipes.md new file mode 100644 index 0000000..2f14269 --- /dev/null +++ b/_shells/9.4.0/common-driver-recipes.md @@ -0,0 +1,97 @@ +--- +layout: page +title: Common Driver Recipes +category: tut +comments: true +order: 14 +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this section we'll provide a few handy examples of common driver operations. +The intention is to grow this into a good source to copy paste common code from. +All of the examples will be available in the +[DevGuide Examples](https://github.com/QualiSystems/devguide_examples) repository +under the _common_driver_recipes_ folder. + +#### Decrypting a password attribute + +The password for logging into the device/app will be passed as an encrypted value to the driver +as a part of the context object. In order to be able to use it to log in you'll most likely +need to decrypt it. To do that, you can use the CloudShellAPI function **_DecryptPassword_**. + +See the code below for an example: + +{% github_sample_ref /QualiSystems/devguide_examples/blob/driver_deep_dive/adding_examples/common_driver_recipes/src/driver.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/driver_deep_dive/adding_examples/common_driver_recipes/src/driver.py 28 38 %} +{% endhighlight %} + +Note that in Python 3 shells, you no longer need to decrypt passwords as this is done out of the box by the [cloudshell-shell-networking-standard](https://pypi.org/project/cloudshell-shell-networking-standard/) python package. + +#### Updating the resource live status + +The resource live status can be used to indicate the current state of the resource on the diagram. +CloudShell comes with a pre-built collection of statuses you can use, and this collection of icons can also +be extended if needed. + +The following code will update the resource live status from offline to online: + +{% github_sample_ref /QualiSystems/devguide_examples/blob/driver_deep_dive/adding_examples/common_driver_recipes/src/driver.py %} +{% highlight python %} +{% github_sample /QualiSystems/devguide_examples/blob/driver_deep_dive/adding_examples/common_driver_recipes/src/driver.py 40 55 %} +{% endhighlight %} + +The full list of statuses can be found in the _ServerUniversalSettings.xml_ file which you can find on the Quali Server +machine under the %programdata%\\QualiSystems\\Settings\\Global directory: + +{% highlight xml %} + + C:\ProgramData\QualiSystems\Portal\Content\Images\online.png + C:\ProgramData\QualiSystems\Portal\Content\Images\online.png + C:\ProgramData\QualiSystems\Portal\Content\Images\offline.png + C:\ProgramData\QualiSystems\Portal\Content\Images\error.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress0.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress10.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress20.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress30.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress40.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress50.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress60.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress70.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress80.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress90.png + C:\ProgramData\QualiSystems\Portal\Content\Images\progress100.png + + + {% endhighlight %} + +#### Sending a message to the sandbox console + +Another way to update the sandbox regarding an operation progress is to use the _WriteMessageToReservationOutput_ +function to display a message in the Sandbox console pane. +We can easily modify the previous code to do that instead: + +{% github_sample_ref QualiSystems/devguide_examples/blob/master/common_driver_recipes/src/driver.py %} +{% highlight python %} +{% github_sample QualiSystems/devguide_examples/blob/master/common_driver_recipes/src/driver.py 57 70 %} +{% endhighlight %} + +#### Sending commands to a networking device +When adding a new command that requires communication with a networking device, such as a switch or router, you need to open a session to the device. An easy way to do that is by leveraging Quali’s shell framework (*cloudshell-cli* package). The framework can provide a session from a session pool via Telnet, SSH or TCP, based on the configuration saved in the **CLI Connection Type** attribute on the root resource. + +See the code below for an example: + +{% github_sample_ref QualiSystemsLab/Extended-Cisco-NXOS-Shell/blob/master/Ext_Cisco_NXOS_Shell_Package/Resource%20Drivers%20-%20Python/Generic%20Cisco%20NXOS%20Driver%20Version3%20Extended/cisco_nxos_resource_driver.py %} +{% highlight python %} + +{% github_sample QualiSystemsLab/Extended-Cisco-NXOS-Shell/blob/master/Ext_Cisco_NXOS_Shell_Package/Resource%20Drivers%20-%20Python/Generic%20Cisco%20NXOS%20Driver%20Version3%20Extended/cisco_nxos_resource_driver.py 120 133 %} +{% endhighlight %} + +#### Mapping connections using App sub-resources +CloudShell 8.3 allows developers to map connections between sub-resources residing on deployed Apps. This applies to scenarios where you want to map the port connections between virtual devices residing in App VMs. For example, to map the connection between port 1 residing on a virtual switch and port 2 residing on another virtual switch. For details, see [Mapping Connections using App Sub-resources]({{site.baseurl}}/reference/{{pageVersion}}/mapping-sub-resource-connections.html). + + diff --git a/_shells/9.4.0/customizing-driver-commands.md b/_shells/9.4.0/customizing-driver-commands.md new file mode 100644 index 0000000..10ad1d8 --- /dev/null +++ b/_shells/9.4.0/customizing-driver-commands.md @@ -0,0 +1,359 @@ +--- +layout: page +title: Commands Visibility and Usability +category: tut +date: "2016-04-30 13:02:32 +0300" +order: 7 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + + +The Shell driver commands are accessible to CloudShell users via the portal as well as to orchestration scripts running on the sandbox. +In this section we'll explore the different ways in which these commands can be customized in their appearance and behavior. The following customization options will be reviewed: + +* [Changing the shell's python version](#changing-the-shells-python-version) +* [Changing the function display name and description](#customize_names) +* [Specifying display name and descriptions for each parameter](#customize_parameter_names) +* [Optional parameters and default values](#customizing_optional_parameters) +* [Restricting input to a specific set of possible values](#customizing_lookup_values) +* [Adding categories](#customizing_categories) +* [Orchestration only commands (hidden commands)](#customizing_orchestration_only_commands) +* [Defining a connected command (which runs on another resource)](#connected_commands) + +If you haven't done some already it is recommended to go through the [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html) tutorial before continuing to get a better understanding of the overall process of creating and using a shell. We also assume you've gone through the steps described in the [Setting Up the Development IDE]({{site.baseurl}}/introduction/{{pageVersion}}/setting-up-the-development-ide.html) section of this guide. + + + +### Setting up + +We'll start by creating a new shell that we'll use in this example. In the [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html) tutorial we used the ShellFoundry CLI tool to generate a working shell to get started quickly. In this section we'll also be using the ShellFoundry to create an example shell, only this time we'll use a different project template. The _resource-clean_ template is similar to the default template we've used previously only it doesn't contain any example driver functions. This will make it easier for us to really go through the experience of starting from a clean slate. + +From the command line, navigate to a folder under which you'll want the new shell to be created and type in the following: +{% highlight bash %} +shellfoundry new customization-example +cd customization_example +{% endhighlight %} + +A new project with an empty driver is created. + +Let's start by creating a function we can experiment on. Open the driver file. The file is named _driver.py_ and is located in the _src_ folder of the shell project. +Add the following function to the shell driver class and save it: + +{% highlight python %} +def user_facing_function(self, context, some_parameter, some_other_parameter): + """ + :type context: cloudshell.shell.core.driver_context.ResourceCommandContext + :type some_parameter: str + :type some_other_parameter: str + """ + return "Thank you for calling this function." +{% endhighlight %} + +We've now created a shell with a single command. Similar to the flow covered in the [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html) tutorial, the next steps would be to create a resource instance of this shell in CloudShell and add it to a sandbox so we can experiment with it. You should refer to the [Getting Started]({{site.baseurl}}/shells/{{pageVersion}}/getting-started.html) tutorial for any question on how to accomplish the three steps below: + +1. Install the shell by running the following from the shell directory: _shellfoundry install_ +2. Create a resource instance from the CloudShell inventory. +3. Create a new sandbox and drag the resource instance into it. Open the Commands pane. + +At this point the Commands pane should look something like this: + +![Shell Commands]({{ site.baseurl}}/assets/commands_no_customization.png) + +As you can see, we didn't need to specify any additional details to get our Python functions to be visible as shell commands in the CloudShell sandbox, however the commands appear if a very 'code' like manner so our next step would be making them a bit more presentable. + +The information on how to display the driver functions in CloudShell is stored in the _drivermetadata.xml_ file located in the driver _src_ directory. + +### Changing the shell's python version + +To change the shell's default python version, open the *drivermetadata.xml* file and in the `Driver` line, add the `PythonVersion` property with the new version (2 or 3). For example: + +{% highlight xml %} + +{% endhighlight %} + + + +### Changing the function display name and description + +For the first round of customizations, we'll work on the user facing function and improve the naming of the command and its parameters as they appear to the user in CloudShell. Open the _drivermetadata.xml_ file and add the following text to it: + +{% highlight xml %} + + + + + + +{% endhighlight %} + +Each _Command_ element in the xml above customizes a single python function specified by the _Name_ attribute. The _DisplayName_ and _Description_ attributes determines the name of the command as it appears to the user and the description appearing next to it, respectively. And the _Visibility_ attribute determines which user types (*AdminOnly*/*Everyone*) can see the command in the blueprint or sandbox. + +Reinstall the Shell in CloudShell by running the following command-line and check the **Resource Commands** panel again. + +{% highlight bash %} shellfoundry install {% endhighlight %} + +The shell commands should now look like this: + +![Shell Commands]({{ site.baseurl}}/assets/commands_name_customization.png) + +Please note that changing the display name of a function will affect only how its visually rendered in the UI. +Code trying to execute this command using the CloudShell API will need to still refer to it by the command _name_. + + + +### Specify display name and descriptions for each parameter + +You may have noticed that the parameter names still look pretty raw and code like. We might want to customize that as well. We can do that by adding additional nested elements describing the parameters under each command. Update the _drivermetadata.xml_ file as follows: + +{% highlight xml %} + + + + + + + + + + + +{% endhighlight %} + +In the _drivermetadata.xml_ XML, the _Parameters_ element contains a list of _Parameter_ elements, each is responsible for a specific function parameter. The _DisplayName_ and _Description_ attributes will determine the name of the command as it appears to the user and the description appearing next to it, respectively. + +After installing the shell again, the parameters for the command will now appear in a more readable format: + +![Shell Commands]({{ site.baseurl}}/assets/commands_param_customization.png) + + + +### Optional parameters and default values + +For each of the command parameters, we may want to specify whether that parameter is mandatory for the user to supply, and whether there should be a default value in case the user didn't enter any value. + +In the following example, we will make the first parameter mandatory by setting the _Mandatory_ attribute to True. Users will be required to enter a value for parameters before being able to run the command. The second parameter is optional but now has a default value which we will set using the DefaultValue attribute: + +{% highlight xml %} + + + + + + + + + + + +{% endhighlight %} + +Re-installing the shell will update the commands pane accordingly: + +![Shell Commands]({{ site.baseurl}}/assets/commands_mandatory_customization.png) + + + +### Restricting input to a specific set of possible values + +For certain parameters, you might want the user to select between a pre-determined set of values, rather than leave the inputs as a free text entry. To set that up, you need to add that information to the parameter XML: + +{% highlight xml %} + + + + + + + + + + + +{% endhighlight %} + +In the xml above, we've specified that the second parameter must be selected out of a specific set of possible values. We did that by adding the _Type_ attribute to the parameter element and seting it as _Lookup_. To define the possible values, we've added the _AllowedValues_ attribute, which sets the possible values for this parameter, represented as a comma separated list. In this case, the possible values are _Yes_ and _No_. We've also changed the default value to conform to the possible options. + +After re-installing the shell, the Commands pane now reflects the parameter new value restriction: + +![Shell Commands]({{ site.baseurl}}/assets/commands_lookup_customization.png) + + + +### Adding categories + +As you add more commands to the resource, it can be useful to visually group them into categories and improve the usability of the commands panel. +These categories will appear as folders which the user can navigate. To demonstrate this capability, let's first add additional commands to the shell. Open the _driver.py_ file located in the _src_ directory and add the following two functions: + +{% highlight python %} +def backup(self, context): + """ + :type context: cloudshell.shell.core.driver_context.ResourceCommandContext + """ + return "Backing up..." + +def restore(self, context): + """ + :type context: cloudshell.shell.core.driver_context.ResourceCommandContext + """ + return "Restoring..." +{% endhighlight %} + +Grouping together commands under a category is pretty strait forward: Simply add a _Category_ element and nest the relevant commands under it. In this case, we've added a category called _Operate_ for the previous operation we've implemented and a category named _Management_ for the new save and restore functions: + +{% highlight xml %} + + + + + + + + + + + + + + + + + + +{% endhighlight %} + +After re-installing the shell we get the following command panel layout: + +![Shell Commands]({{ site.baseurl}}/assets/commands_category_customization.png) + + + +### Orchestration only commands (hidden commands) + +Sometimes, you may wish to create commands that are intended to be used as a part of an orchestration flow, for example to be called during setup, but want these commands to be inaccessible to users [hidden] from the UI. For example, a command that is called during a sandbox’s Setup process. +To support this use case, CloudShell supports a special category, the _Hidden Commands_ category, which allows you to omit commands from the web portal UI while allowing them to be invoked via the API. + +To demonstrate this capability, let's first add a new function to our driver class in the _driver.py_ file: + +{% highlight python %} +def api_only_function(self, context): + """ + :type context: cloudshell.shell.core.driver_context.ResourceCommandContext + """ + return "You didn't see this on the web interface..." +{% endhighlight %} + +Next, let's add the special category to the _drivermetadata.xml_ file and nest that command under it: + +{% highlight xml %} + + + + + + + + + + + + + + + + + + + + + +{% endhighlight %} + +After re-installing the shell you'll see the new function doesn't appear in the Commands pane: +![Shell Commands]({{ site.baseurl}}/assets/commands_hidden_customization.png) + +Note that the *Visibility* attribute we discussed earlier in this article does not apply to commands in this category and will be ignored. + +However if you query the list of commands on the shell via the API, you'll be able to see it as well as invoke it: +![Shell Commands]({{ site.baseurl}}/assets/commands_hidden_list_from_api.png) + + +### Defining a connected command (which runs on another resource) + +In some scenarios, a command that runs on a specific resource logically requires the use of a different resource. These types of commands are called connected commands. They are executed on a resource in CloudShell Portal but in reality run on the connected resource (for example, a power switch) that performs the action. + +There are two types of connected commands - power commands and generic connected commands. +* Power commands power on/off a resource and actually run on a power switch +* Generic resource commands may be used to modify a resource's settings and run on the server that performs the modifications + +Note that in CloudShell, the resource containing the connected command and the target resource must be directly connected to each other. If you have a switch resource in between, the connected command will not show on the target resource. + +To create a connected command, you need to use a shell based on a template that supports connected commands, currently, only **PDU** and **Generic Resource With Connected Commands**. In addition, in that shell’s driver, you will need to define which commands are connected. This is done in the driver’s *drivermetadata.xml*, as follows: + +For power commands, the command must include the `Tags=”power”` flag and the driver must include all three power commands (Power On, Power Off and Power Cycle): + +{% highlight xml %} + + + + + + + + + + + + + + + + + +{% endhighlight %} + +And for generic connected commands, you must use a shell that is based on a supporting standard (**Generic Resource With Connected Commands**) and include the `Tags=remote_connectivity"` flag on the command: + +{% highlight xml %} + + + + + + + + + + + + +{% endhighlight %} + +To enable intellisense, on the *driver.py*, include the `ResourceRemoteCommandContext` context in the command's definition: + +{% highlight python %} +def PowerOn(self, context, ports): + """ + :type context: cloudshell.shell.core.driver_context.ResourceRemoteCommandContext + """ + Pass +{% endhighlight %} + +For reference, see the PDU shell template or PDU shell template. + +### Summary + +In this section we reviewed different ways in which its possible to customize the appearance and behavior of commands and command parameters from the user's perspective. If you have feedback or additional suggestions for features and improvements be sure to post them on our idea box. We're always looking for new ideas! + diff --git a/_shells/9.4.0/customizing-shells.md b/_shells/9.4.0/customizing-shells.md new file mode 100644 index 0000000..6fdcd88 --- /dev/null +++ b/_shells/9.4.0/customizing-shells.md @@ -0,0 +1,421 @@ +--- +layout: page +title: Customizing Shells +category: tut +order: 13 +comments: true +version: + - 9.4.0 +--- + +{% assign pageUrlSplited = page.url | split: "/" %} +{% assign pageVersion = pageUrlSplited[2] %} + +In this section, we’ll learn how to create and modify a shell’s commands and attributes. + +This article addresses two flows: + +* Modifying an existing shell +* Creating a new shell with modifications to the standard + +_At the end of this article, you can find an end-to-end example of how to extend an existing shell with attributes and commands. To see the example, click [here](#end-to-end-example)._ + +Modifying an existing shell is done using the `shellfoundry extend` command. This command downloads the source code of the shell you wish to modify to your local machine and updates the shell’s Author. Note that extending official shells (shells that were released by Quali) will remove their official tag. Keep in mind that modifying a shell that is being used in CloudShell may affect any inventory resources that are based on a previous version of the shell. In the second flow, since we're creating a new shell from the appropriate shell standard, we will use the `shellfoundry new` command and modify the shell's settings. + +The common use cases for customizing a shell are: + +* Adding new commands +* Modifying existing commands +* Adding new attributes +* Modifying existing attributes +* Publishing attributes in a service shell +* Associating categories to a service shell + +*Please watch this video if you're not sure whether to create a new shell or customize an existing one:* + + + +## Customizing a shell’s commands + +When customizing an official shell you can add new commands, and also modify or hide existing ones. + +* **To add a new command:** Add the command in the shell’s driver.py file, and expose the command in the shell’s *drivermetadata.xml* file. + +The command’s logic should be placed either in the driver itself or in a separate python package. + +Modifications to a command can include adding some logic either before or after the command or changing the command logic itself. In order to do that, **copy the command code** from the original Quali python package to the shell driver or to the custom python package you created (*the command logic resides either in the vendor package or vendor-os package - for example in “cloudshell-cisco” or “cloudshell-cisco-ios”*). + +When modifying an existing command, you can add optional input parameters. Just make sure that the implementation is backwards compatible. Note that adding mandatory inputs or removing one of the original inputs is not supported. In these cases, it is recommended to create an additional command with a different name, instead of modifying an existing one. + +For example, in this customized [Cisco NXOS shell](https://github.com/QualiSystemsLab/Extended-Cisco-NXOS-Shell), we modified the commands that configure VLANs on multiple ports and port channels. + +It is also possible to hide or remove a command. Hiding a command is done by placing it in an “administrative” [category]({{site.baseurl}}/shells/{{pageVersion}}/customizing-driver-commands.html) in the drivermetadata.xml. Note that removing a command might affect how the shell is used since CloudShell and/or some orchestration scripts might depend on the existing driver’s commands. + +When adding or modifying a command, you can leverage Quali’s shell framework to ease the development process. For details, see [Quali's Shell Framework]({{site.baseurl}}/reference/{{pageVersion}}/quali-shell-framework.html). + +See some common command extension examples in [Common Driver Recipes]({{site.baseurl}}/shells/{{pageVersion}}/common-driver-recipes.html). + +Please check out the following instructional videos on how to develop basic driver commands: + + + + + + + + +## Customizing a shell’s attributes + +You can add and modify attributes associated with the shell's root model or with a specific sub-model within the shell. + +Modification applies to attributes that are defined in the shell’s standard. To find the attributes defined in the shell’s standard, see the documentation page of your shell’s standard. For such attributes, you can modify the description, default values, possible values and rules. + +**Note:** You cannot modify attribute properties **type** and **name**, and any attributes that are associated with the shell’s family as this will affect other shells that use this family. The family attributes are listed in the shell's standard. + + + +### Deployment-specific vs. shell-specific attributes + +CloudShell provides two ways to customize attributes, which differ depending on the attribute's usage: + +* **Customizing an existing shell**: Use this option when the attributes are related to a specific device but are not relevant to other shells. This is done by manually editing the shell’s *shell-definition.yaml* file. +* **Associating custom attributes with a shell that is installed in CloudShell**: Use this option when the additional attributes are deployment-related and relevant to multiple resources of different shells. For example, the Execution Server Selector attribute. *Starting with CloudShell version 8.3, this option applies both to the root model of the shell and to the shell’s sub-resource models, such as blades and ports.* + +The second option of associating custom attributes with an already installed shell is done either via CloudShell Portal or by calling the [SetCustomShellAttribute]({{site.baseurl}}/shells/{{pageVersion}}/deploying-to-production.html#SetCustomShellAttributeUsingAPI) API method. For additional information on this method, see [Deploying to Production]({{site.baseurl}}/shells/{{pageVersion}}/deploying-to-production.html). + +Deleting a 2nd Gen shell’s default attributes (those that come with the standard) is not supported. It is also not possible to customize a 2nd Gen shell’s data model (families and models) and its structure, which is as defined in the shell standard the original shell is based on. + +### Adding or modifying attributes in a shell’s root or sub-model + +This section explains how to add attributes to the shell's root model and to specific models within the shell. To learn how to expose attributes that are required for the discovery of the resource (in the **Inventory** dashboard's **Resource** discovery dialog box), see [Auto-discovery for Inventory Shells]({{site.baseurl}}/shells/{{pageVersion}}/implementing-discovery-for-inventory-shells.html). + +**To add/modify a shell's attributes:** + +1) Open command-line. + +2) To customize a shell that resides on your local machine, make sure command-line is pointing to a different path from the original shell’s root folder. + +3) Run the appropriate command in command-line: + +     **To modify a shell:** +     {% highlight bash %}shellfoundry extend {% endhighlight %} + +     The path can be a URL to the source code of the shell's desired release on [Quali Community's Integrations](https://community.quali.com/integrations) page or the filesystem path (prefixed by `local:`) to the extracted source code folder (like "local:C:\temp\Cisco-NXOS-Switch-Shell-2G-2.0.1"). + +     **To create a new shell based on a specific shell standard:** +     {% highlight bash %}shellfoundry new --template