New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add missing values and categorical features when generating datasets #28952
Comments
@lcrmorin This would be great for testing! I would also suggest adding infinities as possible values, bcs they also break stuff quite often. Also, if randomly generated, making sure to always include at least one NaN and inf value |
@lcrmorin I suggest adding a noise function or something similar which can generate structured randomness so as to make some sense in data and not pseudo-randomness. Perhaps something like Perlin Noise? |
Regarding the missing values I recall the following issues/PRs: #6284 / #7084. It seems that the consensus was to have something similar to the I almost a similar discussion for categorical features but I could not find. For sure, it would be handy to have those two parameters even though we could limit the complexity (e.g. only have a single missingness pattern) |
Regarding the categorical features, we have the following related issue: #12433 |
I am working on this |
Describe the workflow you want to enable
I am often using random datasets (typically with make_classification). However I often find myself having to add more realistic features to the dataset:
It would be nice to have both of those in datasets generation.
Describe your proposed solution
Introduce parameters to allow for generation of missing data (proportion of missingness, type of missingness - at random, not at random).
Introduce parameters to allow for generation of categorical features (number of features, type of repartition in categories - even - uneven - pareto.
Describe alternatives you've considered, if relevant
I usually handle this by hand.
Additional context
Could be used to illustrate imputing techniques, encoding techniques.
The text was updated successfully, but these errors were encountered: