When working with multiple files for training, using MLOpsDatabricks, the train script will fail on importing dependencies

[riserrad]

Describe the bug
When working with multiple files for training - and referencing them from train.py for example - using MLOpsDatabricks, the train script will fail on importing dependencies

To Reproduce
Instead of using a single to train the model, have a main file that loads some other dependencies (stored on the folder structure) and submit the Pipeline Step to AML.

Expected behavior
Wether the file is on the same folder level as the train script or the file is available within the structure under sources_directory, it should be able to use/import the dependency.

Screenshots
Instead, we get an error saying it could not load the dependency:

ModuleNotFoundError                       Traceback (most recent call last)
<command--1> in <module>()
     13 
     14 with open(filename, "rb") as f:
---> 15   exec(f.read())
     16 

<string> in <module>()

ModuleNotFoundError: No module named 'documentprocessor'

[eedorenko]

The workaround (ugly but works) would be to add to the list of paths Python uses to search modules (sys.path.append) a folder where Databricks stores the source scripts. This folder is passed as a parameter--AZUREML_SCRIPT_DIRECTORY_NAME to the Python script. So, the solution looks like this:

import argparse
import sys
parser = argparse.ArgumentParser()
parser.add_argument(
    "--AZUREML_SCRIPT_DIRECTORY_NAME",
    type=str,
    help="folder",
)
args, unknown = parser.parse_known_args()
folder = args.AZUREML_SCRIPT_DIRECTORY_NAME
sys.path.append("/dbfs/" + folder)  # NOQA: E402
from test_db_util import test_print  # NOQA: E402

Where test_db_util is an imported module coming in the same folder.

[gitsathish]

My "Train model using AML with Remote Compute" step failed.
When I look into the databricks logs, I see this -

ImportError Traceback (most recent call last)
in ()
13
14 with open(filename, "rb") as f:
---> 15 exec(f.read())
16

in ()

ImportError: No module named 'torch'

[gitsathish]

Not sure if this was the right approach to fix this, but installed torch and torchvision in the databricks cluster... not I get a different error - show below ---

TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'Fri Oct 18 21:03:47 2019 py4j imported
Fri Oct 18 21:03:47 2019 Python shell started with PID 3010 and guid 3f38634867c44d8f9b7fba363ddf328e
Fri Oct 18 21:03:47 2019 Python shell executor start

TypeError Traceback (most recent call last)
in ()
13
14 with open(filename, "rb") as f:
---> 15 exec(f.read())
16

in ()

TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

[FlorianPydde]

@eedorenko THANK YOU !! I have been working for 3 days on this issue !!! This is brilliant !