Litcoin pipeline integration including helm chart for k8s deployment

[hyi]

@EvanDietzMorris I am creating this PR for your perusal to get your inputs sooner rather than later - no need to get this PR merged until the graph building hanging issue specific to k8s deployment gets resolved.

[hyi]

@EvanDietzMorris I have addressed the status inconsistency in the get-task-status API endpoint by improving the interaction between the subprocess run and celery worker, and also addressed the hanging issue due to insufficient memory for the worker. So I think this PR is ready to be merged as the first pass implementation dependent upon your code review and approval.

[hyi]

@EvanDietzMorris This PR is now ready for review. It includes all the work needed for the LitCoin pipeline integration, specifically, predicate mapping now integrated in the loadLitCoin parser, adding an environmental variable to indicate loading LitCoin input data from a shared directory or a shared PVC path rather than the default stars server URL, and the helm chart for Sterling k8s deployment. I tested the whole pipeline in Sterling with a small subset of litcoin input data and everything works as expected with the neo4j graph created. However, when I test the whole LitCoin input data, the parser failed in the middle due to predicate mapping getting blocked by security rules (403 response) for some inputs. I have created a help ticket trying to get this fixed/unblocked, but otherwise, it seems the whole pipeline works, so I am asking you to do a code review when you get a chance. The pipeline delivery deadline is 5/27 including readme update to have step-by-step installation instructions for people to follow to get the whole pipeline deployed. So it'd be great if this PR can be merged by 5/27 at the latest. Thanks!

[EvanDietzMorris]

I haven't tested the helm charts or celery part yet but this all looks good and shouldn't break anything else, so I'm ok with merging it quickly.

My only concern is that users/developers might get confused about the predicate mapping parts, and think they refer to something more general, when it's really just for litcoin. Are you ok with renaming some of the terminology to make it clear that it's litcoin specific? ie PRED_MAPPING_URL to LITCOIN_PRED_MAPPING_URL.

[hyi]

@EvanDietzMorris Good point! I have just done renaming as you suggested to avoid potential confusion. Please take a look when you get a chance and let me know if you have any other concerns that need to be addressed. Thanks

[EvanDietzMorris]

I moved LITCOIN_PRED_MAPPING_URL to the optional section of the set_up_test_env.sh script. FYI we are also in the process of redesigning where the example env vars are set. #314 would move towards relying on .env files more, with a script to help populate them instead of setting them in local environments. Your opinion on that would be welcome if you want to look.

Other than that, I think we should double check whether we're handling the openAI credentials correctly, especially if we're handing this off to other people to deploy. We might want to move them over to actual secrets in the helm charts for production deployments. If you think we should change that, it's up to you whether to do it on this PR or start a new one.

[hyi]

@EvanDietzMorris Agreed relying on .env files more is a good move. Yeah, thanks for bringing up this issue with OpenAI credentials. I'll bring this up tomorrow at the scrum meeting to get input on this. If we want to keep this OpenAI dependency, I'll create a new PR to have its credentials set up as secrets in helm chart. The new PR will be most likely created after the 5/27 deadline. So I'll merge this PR for now and create new PR to address this if needed.