Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add AudioFolder packaged loader #4530

Merged
merged 81 commits into from Aug 22, 2022
Merged

Add AudioFolder packaged loader #4530

merged 81 commits into from Aug 22, 2022

Commits on Jun 20, 2022

  1. add audiofolder loader (almost identical to imagefolder except for in… 

    …ferring labels is not default)
    @polinaeterna
    polinaeterna committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    cc87b0e View commit details
    Browse the repository at this point in the history

  2. check codestyle

    @polinaeterna
    polinaeterna committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    0adcd56 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2022

  1. add tests

    @polinaeterna
    polinaeterna committed Jun 21, 2022
    Configuration menu
    Copy the full SHA
    e46eecb View commit details
    Browse the repository at this point in the history

  2. remove unused imports

    @polinaeterna
    polinaeterna committed Jun 21, 2022
    Configuration menu
    Copy the full SHA
    7cc4ab9 View commit details
    Browse the repository at this point in the history

  3. add dummy data

    @polinaeterna
    polinaeterna committed Jun 21, 2022
    Configuration menu
    Copy the full SHA
    a648530 View commit details
    Browse the repository at this point in the history

  4. add instruction on how to obtain list of audio extensions

    @polinaeterna
    polinaeterna committed Jun 21, 2022
    Configuration menu
    Copy the full SHA
    5cbbad1 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2022

  1. fix comment

    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    53e9ce3 View commit details
    Browse the repository at this point in the history

  2. add audiofolder dummy files in tests

    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    60760ea View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    e4bb688 View commit details
    Browse the repository at this point in the history

  4. check if two separate files fix test error (i guess not but just in c… 

    …ase)
    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    d0b2592 View commit details
    Browse the repository at this point in the history

  5. remove unused imports

    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    15ca3cf View commit details
    Browse the repository at this point in the history

  6. Revert "check if two separate files fix test error (i guess not but j… 

    …ust in case)"

This reverts commit d0b2592.
    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    420dd2b View commit details
    Browse the repository at this point in the history

  7. add uppercased formats, modify test for zip archive (check that array… 

    …s are different)
    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    68b7f5a View commit details
    Browse the repository at this point in the history

  8. Merge branch 'huggingface:master' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    7c75e81 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2022

  1. add contributors

    @polinaeterna
    polinaeterna committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    eecc449 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'add-audio-folder-new' of github.com:polinaeterna/datase… 

    …ts into add-audio-folder-new
    @polinaeterna
    polinaeterna committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    a081364 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'huggingface:master' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    93c6afa View commit details
    Browse the repository at this point in the history

  4. Merge branch 'add-audio-folder-new' of github.com:polinaeterna/datase… 

    …ts into add-audio-folder-new
    @polinaeterna
    polinaeterna committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    f5d9841 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2022

  1. Merge branch 'huggingface:master' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jun 29, 2022
    Configuration menu
    Copy the full SHA
    91afd92 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2022

  1. Merge branch 'master' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jul 4, 2022
    Configuration menu
    Copy the full SHA
    cce1ebf View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2022

  1. add a generic loader

    @polinaeterna
    polinaeterna committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    3c6c56a View commit details
    Browse the repository at this point in the history

  2. change name of get_patterns

    @polinaeterna
    polinaeterna committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    6840eab View commit details
    Browse the repository at this point in the history

  3. update autofolder, align imagefolder and audiofolder with it

    @polinaeterna
    polinaeterna committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    ffa6d14 View commit details
    Browse the repository at this point in the history

  4. align audiofolder

    @polinaeterna
    polinaeterna committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    aa2f246 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2022

  1. move autofolder

    @polinaeterna
    polinaeterna committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    d27266c View commit details
    Browse the repository at this point in the history

  2. fix bug with incorrect itaration over archives (incorrect copypaste -_-)

    @polinaeterna
    polinaeterna committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    24a65fd View commit details
    Browse the repository at this point in the history

  3. get back comment

    @polinaeterna
    polinaeterna committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    f9ee90d View commit details
    Browse the repository at this point in the history

  4. patch autofolder for streaming manually

    @polinaeterna
    polinaeterna committed Jul 6, 2022
    Configuration menu
    Copy the full SHA
    c905c1b View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2022

  1. Merge branch 'huggingface:main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    f0ddbef View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2022

  1. check fro AutoFolder class specifically in patching, not its string n… 

    …ame; improve comment
    @polinaeterna
    polinaeterna committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    6c7a1f9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'huggingface:main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    9f9551c View commit details
    Browse the repository at this point in the history

  3. Merge branch 'add-audio-folder-new' of github.com:polinaeterna/datase… 

    …ts into add-audio-folder-new
    @polinaeterna
    polinaeterna committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    96189c2 View commit details
    Browse the repository at this point in the history

  4. pass missing use_auth_token for AutoFolder patching

    @polinaeterna
    polinaeterna committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    66d3877 View commit details
    Browse the repository at this point in the history

  5. fix docstrings

    @polinaeterna
    polinaeterna committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    ba9d059 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    1346488 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2022

  1. align autofolder with the latest imagefolder implementation

    @polinaeterna
    polinaeterna committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    a0b4093 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Aug 2, 2022
    Configuration menu
    Copy the full SHA
    86fbb99 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2022

  1. update tests

    @polinaeterna
    polinaeterna committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    d1e4a64 View commit details
    Browse the repository at this point in the history

  2. add test for duplicate label col

    @polinaeterna
    polinaeterna committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    b9eace0 View commit details
    Browse the repository at this point in the history

  3. copy test for dir names

    @polinaeterna
    polinaeterna committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    6a841df View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2022

  1. add tests for autofolder (+copied from imagefolder)

    @polinaeterna
    polinaeterna committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    eabece2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'add-audio-folder-new' of github.com:polinaeterna/datase… 

    …ts into add-audio-folder-new
    @polinaeterna
    polinaeterna committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    d250bfd View commit details
    Browse the repository at this point in the history

  3. Merge branch 'huggingface:main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    edd4803 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'add-audio-folder-new' of github.com:polinaeterna/datase… 

    …ts into add-audio-folder-new
    @polinaeterna
    polinaeterna committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    aab4746 View commit details
    Browse the repository at this point in the history

  5. add missed audio_file fixture

    @polinaeterna
    polinaeterna committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    997a01b View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2022

  1. add __name__ to audio/image features to avoid documentation building … 

    …errors
    @polinaeterna
    polinaeterna committed Aug 6, 2022
    Configuration menu
    Copy the full SHA
    56d35aa View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2022

  1. fix CI on windows

    @polinaeterna
    polinaeterna committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    42627ba View commit details
    Browse the repository at this point in the history

  2. check for __name__ attr too when creating base_feature_name

    @polinaeterna
    polinaeterna committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    76c319f View commit details
    Browse the repository at this point in the history

  3. add documentation

    @polinaeterna
    polinaeterna committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    dce047e View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2022

  1. make base_feature a private attr to be excluded from docs

    @polinaeterna
    polinaeterna committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    74474fd View commit details
    Browse the repository at this point in the history

  2. fix docs

    @polinaeterna
    polinaeterna committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    91c130b View commit details
    Browse the repository at this point in the history

  3. fix comment (rename base_feature)

    @polinaeterna
    polinaeterna committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    0c33f73 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2022

  1. Merge branch 'huggingface:main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    7a8e384 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2022

  1. fix typos (from code review) 

    Co-authored-by: Mario Šaško <mario@huggingface.co>
    @polinaeterna @mariosasko
    polinaeterna and mariosasko committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    75ac1f4 View commit details
    Browse the repository at this point in the history

  2. fix typo (from code review)

    @polinaeterna
    polinaeterna committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    3ab6136 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2022

  1. remove boilerplate, make base feature builder's class arg instead of … 

    …a config's one
    @polinaeterna
    polinaeterna committed Aug 13, 2022
    Configuration menu
    Copy the full SHA
    0b60893 View commit details
    Browse the repository at this point in the history

  2. patch relative imports from parent folder too

    @polinaeterna
    polinaeterna committed Aug 13, 2022
    Configuration menu
    Copy the full SHA
    bc1fb3d View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2022

  1. Merge remote-tracking branch 'upstream/main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Aug 15, 2022
    Configuration menu
    Copy the full SHA
    b4c8a2d View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2022

  1. Merge remote-tracking branch 'upstream/main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    676e6f3 View commit details
    Browse the repository at this point in the history

  2. remove self.config.label_name, use hardcoded 'label'

    @polinaeterna
    polinaeterna committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    724782e View commit details
    Browse the repository at this point in the history

  3. patch parents that inherit from DatasetBuilder, revert get_imports

    @polinaeterna
    polinaeterna committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    bfecab4 View commit details
    Browse the repository at this point in the history

  4. rename autofolder -> folder_builder

    @polinaeterna
    polinaeterna committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    90dc043 View commit details
    Browse the repository at this point in the history

  5. remove autofolder dir

    @polinaeterna
    polinaeterna committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    292a8c5 View commit details
    Browse the repository at this point in the history

  6. remove axtending for streaming from tests, it should work without man… 

    …ual extending
    @polinaeterna
    polinaeterna committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    3e32181 View commit details
    Browse the repository at this point in the history

  7. make base column name an abstract attr of FolderBuilder instead of co… 

    …nfig's parameter
    @polinaeterna
    polinaeterna committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    fe80766 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2022

  1. Update src/datasets/streaming.py 

    Co-authored-by: Mario Šaško <mario@huggingface.co>
    @polinaeterna @mariosasko
    polinaeterna and mariosasko committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    f74922c View commit details
    Browse the repository at this point in the history

  2. rename FolderBuilder -> FolderBasedBuilder

    @polinaeterna
    polinaeterna committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    227ce04 View commit details
    Browse the repository at this point in the history

  3. set drop_labels to None by default for AudioFolder

    @polinaeterna
    polinaeterna committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    034b88c View commit details
    Browse the repository at this point in the history

  4. remove dataclass decorator from audio/image folder configs as they do… 

    …n't introduce any new params

to be more consistent with usual datasets script as it doesn't do anything here anyway
    @polinaeterna
    polinaeterna committed Aug 17, 2022
    Configuration menu
    Copy the full SHA
    54c6cf2 View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2022

  1. remove ABC from FolderBasedBuilder as it does nothing

    @polinaeterna
    polinaeterna committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    7f6719b View commit details
    Browse the repository at this point in the history

  2. update documentation

    @polinaeterna
    polinaeterna committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    748576b View commit details
    Browse the repository at this point in the history

  3. fix docs

    @polinaeterna
    polinaeterna committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    615a839 View commit details
    Browse the repository at this point in the history

  4. SORRY another small fix in docs

    @polinaeterna
    polinaeterna committed Aug 18, 2022
    Configuration menu
    Copy the full SHA
    02f8f57 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2022

  1. get back abc and dataclasses just because of the magical thinking ¯\_… 

    …(ツ)_/¯
    @polinaeterna
    polinaeterna committed Aug 19, 2022
    Configuration menu
    Copy the full SHA
    fc41118 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2022

  1. Revert "get back abc and dataclasses just because of the magical thin… 

    …king ¯\_(ツ)_/¯"

This reverts commit fc41118.
    @polinaeterna
    polinaeterna committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    9ee04ed View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    accb8cd View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/main' into add-audio-folder-new

    @polinaeterna
    polinaeterna committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    adccfd8 View commit details
    Browse the repository at this point in the history

  4. check if builder extending for streaming is not in datasets.builder m… 

    …odule

Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>
    @polinaeterna @lhoestq
    polinaeterna and lhoestq committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    6a79a5f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'add-audio-folder-new' of github.com:polinaeterna/datase… 

    …ts into add-audio-folder-new
    @polinaeterna
    polinaeterna committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    189e98b View commit details
    Browse the repository at this point in the history

  6. fix linters

    @polinaeterna
    polinaeterna committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    89e298c View commit details
    Browse the repository at this point in the history

  7. add comment to the patching thing

    @polinaeterna
    polinaeterna committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    fbef2b0 View commit details
    Browse the repository at this point in the history