Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

migrate from csv files to sqlite databases for downstream use in queries #120

Open
wants to merge 42 commits into
base: staging-county-data
Choose a base branch
from
Open

migrate from csv files to sqlite databases for downstream use in queries #120

wants to merge 42 commits into from

Commits on Mar 6, 2024

  1. created sql db

    @rfl-urbaniak
    rfl-urbaniak committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    268ee04 View commit details
    Browse the repository at this point in the history

  2. started sql migration

    @rfl-urbaniak
    rfl-urbaniak committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    7df6f9e View commit details
    Browse the repository at this point in the history

  3. conversion of csvs to db

    @rfl-urbaniak
    rfl-urbaniak committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    c84a537 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. small speed test

    @rfl-urbaniak
    rfl-urbaniak committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    50e93ba View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2024

  1. data cleaning scripts migrated to a subfolder

    @rfl-urbaniak
    rfl-urbaniak committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    02dd6ce View commit details
    Browse the repository at this point in the history

  2. fixed pytest version at 7.4.3

    @rfl-urbaniak
    rfl-urbaniak committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    d8cced8 View commit details
    Browse the repository at this point in the history

  3. data export from csv to db with test

    @rfl-urbaniak
    rfl-urbaniak committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    c127307 View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2024

  1. fix indentation dg

    @rfl-urbaniak
    rfl-urbaniak committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    9c8d80c View commit details
    Browse the repository at this point in the history

  2. WIP

    @rfl-urbaniak
    rfl-urbaniak committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    d2cb7d2 View commit details
    Browse the repository at this point in the history

  3. DataGrabberDB with tests

    @rfl-urbaniak
    rfl-urbaniak committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    c2b731e View commit details
    Browse the repository at this point in the history

  4. refactored DataGrabberCSV

    @rfl-urbaniak
    rfl-urbaniak committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    e5f8a4b View commit details
    Browse the repository at this point in the history

  5. passed DataGrabberDB downstream

    @rfl-urbaniak
    rfl-urbaniak committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    5f942cf View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. performance tests

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    b46dc5a View commit details
    Browse the repository at this point in the history

  2. removed vscode settings

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    648974e View commit details
    Browse the repository at this point in the history

  3. lint with new mypy and pyro, ignore Adam mypy complaint

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    7c43efa View commit details
    Browse the repository at this point in the history

  4. added staging-* to workflow

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    79f74d7 View commit details
    Browse the repository at this point in the history

  5. force the most recent version of isort in setup

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    1312ff6 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'staging-county-data' of https://github.com/BasisResearc… 

    …h/cities into ru-sql
    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    2c5a8b1 View commit details
    Browse the repository at this point in the history

  7. typo in isort import

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    e5919f1 View commit details
    Browse the repository at this point in the history

  8. isort modeling_utils.py

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    b3664ce View commit details
    Browse the repository at this point in the history

  9. switch to --apply within isort in clean.sh

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    7f47359 View commit details
    Browse the repository at this point in the history

  10. removed --apply as redundant

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    41cda1f View commit details
    Browse the repository at this point in the history

  11. upgrade black

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    839a4ab View commit details
    Browse the repository at this point in the history

  12. add black profile to scripts

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    88bc3ef View commit details
    Browse the repository at this point in the history

  13. removed black from nbqa

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    99ddeb0 View commit details
    Browse the repository at this point in the history

  14. dealing with linter versions

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    0a37867 View commit details
    Browse the repository at this point in the history

  15. revising workflow

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    475c732 View commit details
    Browse the repository at this point in the history

  16. db pipeline to workflow

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    a9bc78d View commit details
    Browse the repository at this point in the history

  17. suspend black to avoid linting version issues

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    d5062a1 View commit details
    Browse the repository at this point in the history

  18. Merge branch 'staging-county-data' of https://github.com/BasisResearc… 

    …h/cities into ru-sql
    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    4764f3d View commit details
    Browse the repository at this point in the history

  19. decouple db pipeline from data grabber

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    a0f97a5 View commit details
    Browse the repository at this point in the history

  20. lint

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    c13a431 View commit details
    Browse the repository at this point in the history

  21. runner isort recommendations by hand

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    1885390 View commit details
    Browse the repository at this point in the history

  22. suspend isort switch to black

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    8e23fd3 View commit details
    Browse the repository at this point in the history

  23. switch to dev install (as torch is required to test inference now)

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    f7fb0ec View commit details
    Browse the repository at this point in the history

  24. suspend notebook tests

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    cfde8bc View commit details
    Browse the repository at this point in the history

  25. typo in test yaml

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    5ca57e1 View commit details
    Browse the repository at this point in the history

  26. remove parallel testing as different tests are collected at different… 

    … gws
    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    be24f85 View commit details
    Browse the repository at this point in the history

  27. fixing test.yml

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    71bc2b9 View commit details
    Browse the repository at this point in the history

  28. fixed pyro version to 1.8.5

    @rfl-urbaniak
    rfl-urbaniak committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    a0c0526 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. removed redundant code from test_inference

    @rfl-urbaniak
    rfl-urbaniak committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    9fe9130 View commit details
    Browse the repository at this point in the history

  2. format lint

    @rfl-urbaniak
    rfl-urbaniak committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    74cb94d View commit details
    Browse the repository at this point in the history