Feature/scikit #1
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Load and clean movie data from CSV, removing rows with missing crucial information and outliers. - Perform feature engineering by creating dummy variables for movie genres. - Split the dataset into training and testing sets. - Encode categorical text features using OrdinalEncoder. - Optimize RandomForestClassifier parameters using GridSearchCV with F1-score as the scoring metric. - Evaluate model performance with a custom probability threshold for classifying movies as Good or Bad. - Include an example prediction for a specific movie scenario.
- Updated data loading path to accommodate project structure changes. - Simplified data cleaning process by removing unnecessary print statements. - Enhanced feature engineering by directly creating genre dummy variables. - Implemented stability testing with multiple iterations to assess model performance. - Replaced GridSearchCV with fixed hyperparameters for RandomForestClassifier. - Added detailed accuracy and F1-Score reporting for each iteration. - Included stability assessment based on accuracy standard deviation. - Streamlined example prediction section to utilize the latest trained model.
devZenta
requested review from
Degalax,
GARATONCODE,
Lockxii,
eather55 and
francoisdotdev
devZenta
added
documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request adds a new script,
preprocessing.py, that implements a complete preprocessing and modeling pipeline for predicting whether a movie is "good" (IMDB score ≥ 7) using a Random Forest classifier. The script covers data loading, cleaning, feature engineering, model training with stability testing, and provides an example prediction.Key changes include:
Data Loading and Preprocessing:
data/processed/imdb_bdd.csvfile and performs data cleaning by dropping rows with missing or invalid values in key columns.genrescolumn and selects relevant numeric and categorical features for modeling.Model Training and Evaluation:
RandomForestClassifierusing selected features and evaluates its stability over 5 different random splits, reporting accuracy and F1-score for each run.OrdinalEncoderto handle categorical variables, ensuring unknown categories are encoded safely.Reporting and Example Prediction: