Unable to invoke function on cluster

[MichaelKreutz]

Hi,

I followed the documentation https://github.com/hydro-project/cluster/blob/master/docs/getting-started-aws.md to setup a cloudburst cluster in AWS. The creation of the cluster is working as described. Then I spin up an additional EC2 instance in order to start a cloudburst client from there and try to invoke a function following https://github.com/hydro-project/cloudburst/blob/master/docs/function-execution.md :

>>> from cloudburst.client.client import CloudburstConnection
>>> cloudburst = CloudburstConnection(AWS_FUNCTION_ELB, MY_IP, local=local)
>>> incr = lambda _, a: a + 1
>>> cloud_incr = cloudburst.register(incr, 'incr')
>>> cloud_incr(1).get()

The first 4 commands are working without error but the invocation of the function (command 5) does not terminate. Unfortunately I do not have much ideas how to debug it. Can you help me out or give me some guidance what I could check?

Best regards and many thanks in advance,

Michael

[vsreekanti]

Hi @MichaelKreutz -- thanks for the question. Where are you running this from? Calling .get() tries to fetch the result from the KVS, which is asynchronous. To support this, ports 6460 and 6760 need to be open. If you can't open those ports, you can try running your script above from inside the k8s cluster, which should also work. Let me know if this doesn't fix things.

[MichaelKreutz]

Thanks for your super fast help.

I am starting an EC2 instance in the same VPC by using AWS Console.

I am a bit lost. Do you want me to open the ports by doing a port-forwarding in the functions-load-balancer in https://github.com/hydro-project/cluster/blob/master/hydro/cluster/yaml/services/function.yml, by adding

  - port: 6460
    targetPort: 6460
    name: anna1
  - port: 6760
    targetPort: 6760
    name: anna2

?

Besides I connected to the pod named function-nodes-{...} and tried to run my scripts from there by connecting via cloudburst = CloudburstConnection('DNS_OF_FUNCTIONS_SERVICE_ELB', '$PUBLIC_IP_OF_CLIENT', local=False) with the same outcome: the .get() call just does not terminate..

[vsreekanti]

When you say in the same VPC, do you mean that it's in the same VPC as the Kubernetes cluster? If so, then the easiest thing to do would be the switch PUBLIC_IP_OF_CLIENT to PRIVATE_IP_OF_CLIENT.

If you are in a separate VPC from the k8s cluster, the easiest thing to would be to go to the security group of the client machine and open ports 6460 and 6760 to traffic from anywhere.

Does this make sense?

[MichaelKreutz]

Again thanks a lot! You are right, with the same VPC I am referring to the VPC of the k8s cluster.

I tried as well to adapt my security group of the instance running in a different VPC (than the VPC of the k8s cluster :-D). It does not work if I open ports 6460 and 6760 only. However, if I open the TCP port-range 6000 - 7000, then it works.

As for running the code from an instance within the same VPC as the k8s cluster runs in, I can use the private IP of the instance (the public one works as well). In addition I have to open some ports, the same port-range as above works..

Is Anna not using static ports for the return channel or do I have to open other ports than 6460 and 6760?

[vsreekanti]

Glad you were able to get it working! The ports usually use a range to avoid conflicts if multiple threads are running on the same machine (e.g., thread 0 would use 6460, thread 1 would use 6461, etc.), but that shouldn't be a problem for your client machine. I'll go back and check to see if there were other ports that are used that I missed in my original comment.

In the meantime, let me know if there are any other issues you're running into in terms of setting up and running the system!