Add Support for Azure Cost Exports

[4upz]

Context

Overview

See #1175 for full context and spike #1203 for background information , but this task covers implementing Azure Cost Exports as a source for fetching usage data from Azure. It is recommended to only implement this task after #1222 has been implemented.

Out of Scope:

• Adding support (outside of CLI exports creation) to chunk data into smaller files

Needs to be considered:

• Whether to limit calculations for a date range to exports files that exist, or create new exports for date ranges not already saved
  • Will involve some additional date parsing and cross-checking of existing exports file (may be worth creating a different task)

Related issues/Dependencies:

• Support Azure Through Exports #1175
• Migrate Azure Consumption to Cost Details API #1222
• SPIKE: Azure Cost Export Support #1203

Acceptance Criteria

• Implement method for using account type information to decide whether to fetch usage data via legacy approach (Usage Details List) or Exports approach
• Document the process of setting up exports through the Azure Portal, including permissions and configurations.
• Create CLI command and script for creating one-time or recurring exports
• Create configs to specify desired source for Azure usage data
• Implement function for listing and fetching Exports from storage blobs
• Use Cost Details csv converter to parse file contents to JSON for calculation

Tech Tasks

• Use @azure/arm-costmanagement package for programmatic setup of exports in the CLI.
• Integrate the @azure/storage-blobs package to fetch cost export data from the configured storage container. and storage account
• Develop CLI commands (i.e. yarn cli:create-azure-exports for automation of export setup
• Implement any necessary logic to handle data retrieval and estimation requests based on the chosen configuration approach.

Tech Notes

• Document the specific steps, configurations, and limitations encountered during the implementation of cost data exports.
• Highlight any considerations for account types and necessary permissions

Review Tasks

• Test the cost data exports thoroughly to ensure they provide the desired scalability and performance improvements.
• Update the project documentation to include detailed instructions on setting up cost data exports through both the Azure Portal and the CLI util.

[MrKriss]

Hi, I'm very interested in this feature! I've had a go at deploying CCF for Azure where I work, but unfortunately ran into some issues around permissions for the Azure Usage API. So instead I've been looking into how to export cost data directly, to be used as a rough proxy for emissions in the meantime.

In building a Python CLI to do the exporting, I tried both the Cost Details API and the Azure Exports API, so thought I'd share my experiences. The biggest limitation I found with the Cost Details API when I tried it last month was the rate limit, which was a max of 10 calls an hour, and with each call pulling back a maximum of 1 months data for a particular scope. This didn't fit with me wanting to loop over a list of subscriptions and pull back 12 months of data for each, so I switched to the Export API instead.

Things learned using the Export API:

• Accessing historical exports via custom exports is a bit more hassle upfront as you need to create the custom export for a particular scope and period (up to 3 months), run it, then delete it once its complete (if you want to keep your list of export definitions tidy).
• At current at least, you can use custom exports to fetch data older than 12 months ago (I ended up going back around 32 months to Jan 2021)
• Using the Export API allows you to define the columns you want to keep, which is useful in vastly reducing the size of data. By default there are around 54 columns in the export data, of which 16 seem to be the most useful.
• The exports are fast to run, taking only a minute or two for my case, and depend on the size of data you are requesting to export i.e the number of resources in the scope plus columns requested.
• Creating Exports through the Azure portal is currently using an older or different version of the Export API, as some of the options differ (e.g. timeframes of WeekToDate vs Last7Days which are not the same thing!), and so if you edit a created export via the portal, it will save it in an older format, so always best to create/update programmatically.

[camcash17]

Hi @MrKriss, thanks so much for providing this analysis. It is very appreciated! This is very helpful as we look to add support for Azure Cost Exports.

Unfortunately at the moment, we have very limited bandwidth on our dev team to implement this feature. Would you be interested in submitting a PR eventually, with our support?

[MrKriss]

Hi @camcash17, no worries happy to pass on my experiences with this.

I'm certainly open to helping out this project, but unfortunately have little in the way of JavaScript experience or skills to bring. Python on the other hand I'm very familiar with. I'll reach out in the Google group to see if there is anything I could perhaps more easily help out with.