How to handle assets with over 25,000 partitions

[ammerzon]

The Dagster documentation advises the following:

We recommend limiting the number of partitions for each asset to 25,000 or fewer. Assets with partition counts exceeding this limit will likely have slower load times in the UI.

Consider this use case where each partition corresponds to an ISBN. Given over 25,000 ISBNs, this would exceed the recommended partition limit.

@asset(
    partitions_def=books_partitions # different ISBNs: 978-3-16-148410-0, etc.
)
def converted_books() -> Output[ConvertedBook]:
    ...

In exploring potential solutions, I encountered several challenges:

• Data Retention Policy: Removing old partitions isn't feasible as all ISBNs need to be permanently accessible.
• Partition Splitting: This could be a viable alternative, although it would significantly complicate the code and the logic.

Does anyone have an idea how this can be handled instead?

[garethbrickman]

Relates to #20802

[jamiedemaria]

hi @ammerzon just wanted to let you know that improving partitions, with a specific eye on improving performance of large (25,000+) partitions, is being actively worked on.

In terms of advice, splitting the partition is likely the most viable option, but i do get that that's annoyingly complicated in a lot of cases. I'll reiterate what the linked discussion says that the performance issues are primarily with UI load time, and mostly things like synced/unsynced status and reporting the materialization status of each partition. If you're able to, I'd recommend writing your asset with the large number of partitions first and if you experience performance issues, then consider restructuring the asset.

[ammerzon]

Hi @jamiedemaria, thank you for your response! It's encouraging to hear that efforts are underway to enhance the performance of large partitions. I'm curious to know if this improvement might also lead to the addition of pagination and search functionality on the asset details page in the future. This feature would greatly simplify finding the right partition for us.

For now, we'll continue with our single asset approach and consider splitting it by year or group identifier in the ISBN if any issues arise.

[marcilj]

Hey @jamiedemaria, could you list the different option that we have currently if we want to configure an asset with more than 25,000 partitions?

You've listed the option of reducing/grouping partitions together. This is possible but bring a lot of problems.

From my experience, grouping the partitions makes backfill process easy, but complicated everything after that.

Let's take a scenario where you have hourly partition and you are now changing them to daily to reduce the number of partitions. (Let's take 5 year worth of hourly data so it works with the number of partitions (43 800 hourly partitions vs 1825 daily partitions)

Since the partitions are grouped together, you can't easily configure your runs to consume the next hour you need to wait for the entire day to be finished, which can slow down drastically the process. If you decide to manually fix this and create your own hourly schedule and change the IO_Managers to insert only, but that makes your daily partition complete when the first hour job ran... which makes it hard to understand what's really finished and what's not.

I've also try turning off partition completly and create a state asset that my asset consume as it input. When you do that you are not limited to having a specific number of partition and you still can keep state of what runned and what not, but you lose all the benefits of having dagster. You lose the partition view, the jobs are all jobs, and not job for specific partitions, you need to implement your own backfill logic... it makes it hard.

If you have any other ways of doing this that you could share it would be very helpfull.