Exception: Could not find CSV dataset at "xyz.csv" after bootstrapping.

[gdprtcl]

Hi Team,

I tried to bootstrap the MLOps project to work with another Model. I put the CSV file inside the "data" folder and changed accordingly in customer_churn_build_train_pipeline.py

But I'm getting the Exception that it couldn't find the CSV file.
I have specified file_name = 'xyz.csv' and commented out the rest.

[gdprtcl]

When I simply place the CSV file in the root of the project, it works. But, "data/xyz.csv" doesn't.

[eedorenko]

Hi @gdprtcl, in order to work with your data you need to create and configure a dataset as it is described here https://github.com/microsoft/MLOpsPython/blob/master/docs/custom_model.md#configure-training-data

[gdprtcl]

The CSV file gets picked up and the dataset is configured (checked in AML Workspace) when I place my file outside in the root of my Project structure, not when inside the "data" folder.
@eedorenko

[eedorenko]

1. Why should it be picked up from "data" folder? You are saying you specified "file_name = 'xyz.csv'" which means it's in the root folder.
2. Originally CSV files in data folder are not used for training, they are used for the example showing how to test data quality. That's it.
3. If there is no already configured dataset, it automatically generates diabetes.csv file, saves the file to the AML blob storage and configures a dataset for it.
4. In order to work with your data (xyz.csv) you need to create and configure a dataset as it is described here https://github.com/microsoft/MLOpsPython/blob/master/docs/custom_model.md#configure-training-data

[gdprtcl]

@eedorenko thanks for the inputs. I understand those.
The intent was to make it work when it's present in the folder. Obviously, we could provide just the name and it gets picked up from the root folder.
I had already tried with both file_name = 'xyz.csv' & file_name = 'data/xyz.csv'. Both didn't work.

By this time, I made it work with a small code tweak for file_name in _build_train_pipeline.py

    if (dataset_name not in aml_workspace.datasets):
    # This call creates an example CSV from sklearn sample data. If you
    # have already bootstrapped your project, you can comment this line
    # out and use your own CSV.
    #create_sample_data_csv()

    # Use a CSV to read in the data set.
    file_name = 'xyz.csv'

    file_name = os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, "data", file_name))
    print(file_name)
    if (not os.path.exists(file_name)):
        raise Exception("Could not find CSV dataset at \"%s\"....)

[eedorenko]

Great that you made it working for you! However, if you are using your own data, this condition if (dataset_name not in aml_workspace.datasets): should return false for you, you are not supposed to use/modify code below it as you are not supposed to have data files in your code. Perhaps, we need to make comments on that in the code more clear and delete the data folder as it brings confusion.

[gdprtcl]

@eedorenko Yes, exactly. That will definitely be better.