remove duplicate words from the dataset #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

iamdoron wants to merge 1 commit into karpathy:master from iamdoron:patch-1

iamdoron commented Dec 7, 2022

hi

thanks for your videos, just finished to watch the first part

when I tried to intersect between the test & train datasets I noticed some names repeat in the dataset

len(words) - len(list(set(words))) # 2539

it might create a bias in the test results and an additional small bias during training


          remove duplicate words

98a037c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet