Skip to content

Latest commit

 

History

History
193 lines (115 loc) · 7.58 KB

CONTRIBUTING.md

File metadata and controls

193 lines (115 loc) · 7.58 KB
Raw

Contributing Guidelines

Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community.

Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution.

Reporting Bugs/Feature Requests

We welcome you to use the GitHub issue tracker to report bugs or suggest features.

When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:

  • A reproducible test case or series of steps
  • The version of our code being used
  • Any modifications you've made relevant to the bug
  • Anything unusual about your environment or deployment

Contributing via Pull Requests

Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:

  1. You are working against the latest source on the master branch.
  2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
  3. You open an issue to discuss any significant work - we would hate for your time to be wasted.

To send us a pull request, please:

  1. Fork the repository.
  2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
  3. Ensure local tests pass.
  4. Commit to your fork using clear commit messages.
  5. Send us a pull request, answering any default questions in the pull request interface.
  6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.

GitHub provides additional document on forking a repository and creating a pull request.

Finding contributions to work on

Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opensource-codeofconduct@amazon.com with any additional questions or comments.

Security issue notifications

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page. Please do not create a public github issue.

Licensing

See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.

We may ask you to sign a Contributor License Agreement (CLA) for larger changes.

Environments

We have hundreds of test functions that runs against several AWS Services. You don't need to test everything to open a Pull Request. You can choose from three different environments to test your fixes/changes, based on what makes sense for your case.

  • Mocked test environment

    • Based on moto.
    • Does not require real AWS resources
    • Fastest approach
    • Basically Limited only for Amazon S3 tests

  • Data Lake test environment

    • Requires some AWS services.
    • Amazon S3, Amazon Athena, AWS Glue Catalog, AWS KMS
    • Enable real tests on typical Data Lake cases

  • Full test environment

    • Requires a bunch of real AWS services.
    • Amazon S3, Amazon Athena, AWS Glue Catalog, AWS KMS, Amazon Redshift, Aurora PostgreSQL, Aurora MySQL, Amazon Quicksight, etc
    • Enable real tests on all use cases.

Step-by-step

Mocked test environment

  • Pick up a Linux or MacOS.
  • Install Python 3.6, 3.7 or 3.8
  • Fork the AWS Data Wrangler repository and clone that into your development environment
  • Go to the project's directory create a Python's virtual environment for the project

python -m venv .venv && source .venv/bin/activate

  • Then run the command bellow to install all dependencies:

./requirements.sh

  • Run the validation script:

./validation.sh

  • To run a specific test function:

pytest tests/test_moto::test_get_bucket_region_succeed

  • To run all mocked test functions (Using 8 parallel processes):

pytest -n 8 tests/test_moto

Data Lake test environment

DISCLAIMER: Make sure you know what you are doing. These steps will charge some services on your AWS account and require a minimum security skill to keep your environment safe.

  • Pick up a Linux or MacOS.
  • Install Python 3.6, 3.7 or 3.8
  • Fork the AWS Data Wrangler repository and clone that into your development environment
  • Go to the project's directory create a Python's virtual environment for the project

python -m venv .venv && source .venv/bin/activate

  • Then run the command bellow to install all dependencies:

./requirements.sh

  • Go to the cloudformation directory

cd cloudformation

  • Deploy the Cloudformation template base.yaml

./deploy-base.sh

  • Return to the project root directory

cd ..

  • Run the validation script:

./validation.sh

  • To run a specific test function:

pytest tests/test_s3_athena::test_to_parquet_modes

  • To run all data lake test functions (Using 8 parallel processes):

pytest -n 8 tests/test_s3_athena

Full test environment

DISCLAIMER: Make sure you know what you are doing. These steps will charge some services on your AWS account and require a minimum security skill to keep your environment safe.

  • Pick up a Linux or MacOS.
  • Install Python 3.6, 3.7 and 3.8
  • Fork the AWS Data Wrangler repository and clone that into your development environment
  • Go to the project's directory create a Python's virtual environment for the project

python -m venv .venv && source .venv/bin/activate

  • Then run the command bellow to install all dependencies:

./requirements.sh

  • Go to the cloudformation directory

cd cloudformation

  • Deploy the Cloudformation templates base.yaml and databases.yaml

./deploy-base.sh ./deploy-databases.sh

  • Go to the EC2 -> SecurityGroups console, open the aws-data-wrangler-* security group and configure to accept your IP from any TCP port.

P.S Make sure that your security group will not be open to the World! Configure your security group to only give access for your IP.

  • Return to the project root directory

cd ..

  • [OPTIONAL] If you intend to run all test, you also need to make sure that you have Amazon QuickSight activated and your AWS user must be register on that.

  • Run the validation script:

./validation.sh

  • To run a specific test function:

pytest tests/test_s3_athena::test_to_parquet_modes

  • To run all data lake test functions for all python versions (Only if Amazon QuickSight is activated):

./test.sh