The cfa_azure python module is intended to ease the challenge of working with Azure via multiple Azure python modules which require the correct steps and many lines of code to execute. cfa_azure simplifies many repeated workflows when interacting with Azure, Blob Storage, Batch, and more. For example, creating a pool in Azure may take different credentials and several clients to complete, but with cfa_azure, creating a pool is reduced to a single function with only a few parameters.
The cfa_azure module is composed of three submodules: batch, clients, and helpers. The module batch contains most of the main functions for interacting with Azure and, specifically, Azure Batch. The module clients contains what we call the AzureClient, which combines the multiple Azure Clients needed to interact with Azure and consolidates to a single client. The module helpers contains more fine-grained functions which are used within the batch and clients modules or independently for more control when working with Azure.
Functions:
- create_pool: creates input and output containers and sets up Azure pool based on config.
- upload_files_to_container: uploads files from specified folders to the specified Azure Blob container.
- run_job: runs and monitors a job in Azure Batch based on user input Docker command.
- package_and_upload_dockerfile: packages container specified in Dockerfile and uploads to Azure Container Registry based on info in config file.
Classes:
- AzureClient: a client object used for interacting with Azure. It initializes based on a supplied configuration file and creates various Azure clients under the hood. It can be used to upload containers, upload files, run jobs, and more.
Functions:
- read_config: reads in a configuration toml file to a python object
- create_container: creates an Azure Blob container
- get_autoscale_formula: finds and reads autoscale_formula.txt from working directory or subdirectory
- get_sp_secret: retrieves the user's service principal secret
- get_sp_credential: retrieves the service principal credential
- get_blob_service_client: creates a Blob Service Client for interacting with Azure Blob
- get_batch_mgmt_client: creates a Batch Management Client for interacting with Azure Batch
- create_blob_containers: uses create_container() to create input and output containers in Azure Blob
- get_batch_pool_json: creates a dict based on config for working in Azure
- create_batch_pool: creates a Azure Batch Pool based on info stored in the config file
- list_containers: lists the containers in Azure Blob
- upload_files: uploads all files in specified folder to the specified container
- get_batch_service_client: creates a Batch Service Client objectfor interacting with Batch jobs
- add_job: creates a job in Azure Batch
- add_task_to_job: adds a task to the specified job based on user-input Docker command
- monitor_tasks: monitors the tasks running in a job
- list_files_in_container: lists out all files stored in the specified Azure container
- df_to_yaml: converts a pandas dataframe to yaml file
- yaml_to_df: converts a yaml file to pandas dataframe
- edit_yaml_r0: takes in a yaml file and produces replicate yaml files with the r0 changed based on the start, stop, and step provided
This repository constitutes a work of the United States Government and is not subject to domestic copyright protection under 17 USC § 105. This repository is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication. All contributions to this repository will be released under the CC0 dedication. By submitting a pull request you are agreeing to comply with this waiver of copyright interest.
The repository utilizes code licensed under the terms of the Apache Software License and therefore is licensed under ASL v2 or later.
This source code in this repository is free: you can redistribute it and/or modify it under the terms of the Apache Software License version 2, or (at your option) any later version.
This source code in this repository is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Apache Software License for more details.
You should have received a copy of the Apache Software License along with this program. If not, see http://www.apache.org/licenses/LICENSE-2.0.html
The source code forked from other open source projects will inherit its license.
This repository contains only non-sensitive, publicly available data and information. All material and community participation is covered by the Disclaimer and Code of Conduct. For more information about CDC's privacy policy, please visit http://www.cdc.gov/other/privacy.html.
Anyone is encouraged to contribute to the repository by forking or creating a new branch and submitting a pull request. (If you are new to GitHub, you might start with a basic tutorial.) By contributing to this project, you grant a world-wide, royalty-free, perpetual, irrevocable, non-exclusive, transferable license to all users under the terms of the Apache Software License v2 or later.
All comments, messages, pull requests, and other submissions received through CDC including this GitHub page may be subject to applicable federal law, including but not limited to the Federal Records Act, and may be archived. Learn more at http://www.cdc.gov/other/privacy.html.
Help make this package/repo more robust and stable by creating issues as you see fit. Please use the following issues template as an outline for your issue: issue template
This repository is not a source of government records, but is a copy to increase collaboration and collaborative potential. All government records will be published through the CDC web site.
Please refer to CDC's Template Repository for more information about contributing to this repository, public domain notices and disclaimers, and code of conduct.