Scale with (tens of) thousands of queries?

[danthegoodman1]

How does a single node scale with thousands, or even tens of thousands of queries on a Kafka data source?

Is this something where I need to consider really beefing up the CPU as well? Or am I more worried about memory?

[bobbyiliev]

Hi there Dan,

A materialized view costs Materialize both CPU and RAM to maintain. Materialize must maintain the results of the query, but often it must also maintain an additional intermediate state.

To learn more about the costs of materialization, check out our architecture overview.

Are you referring to creating thousands of materialized views or running queries on a single view? In case you are referring to running thousands of queries on an already pre-created materialized view, the cost should be minimal, as the results would already have been pre-computed and incrementally kept up to date.

Also, the new Materialize Platform would be a production-ready, managed solution that would be able to handle large levels of workloads by scaling out. For more information you can check out this blog post here:

https://materialize.com/materialize-unbundled/

Let me know if you have any other questions!

Best,

Bobby

[bobbyiliev]

Thank you for the clarification Dan!

Indeed, as you mentioned besides adding more resources like CPU and RAM, this would also depend on various factors that can be tweaked, like the number of new messages coming in per second, the size of each individual message and etc. Also total amount of data that you have and want to store in Materialize. For that I could suggest taking a look at temporal filters:

https://materialize.com/docs/guides/temporal-filters/

You can also build views on top of views, just select from view A in the create statement of view B, and so on, that way you can take advantage of the work that Materialize has done for the previous views.

Hope that this helps! And let me know if you have any questions!

[danthegoodman1]

Thanks, what resource range might I be looking at? For example would 4 cores handle this, or should I not consider anything less than 16 cores and 64gb ram, etc?

[bobbyiliev]

Usually what we suggest is to start with a little bit more resources so starting with 16 CPUs and 64GB RAM is also an option, then you can scale down in case you notice that the resources are not utilized.

But depending on your data size and the aggregations you are planning to have 4 CPUs with 8GB of RAM is quite ok for a start too as listed in the docs.

We had a similar discussion about this recently on our community Slack here, here is one of the suggestions noted there:

I'd encourage you to start big when experimenting, and only size down once you know what views you really need. there tends to be bloat as you’re figuring things out, and then a healthy amount of trimming once you have exactly what you want (in terms of number of materialized views) and this corresponds to the amount of resource load that gets imposed. start with the biggest instance you can afford!

[danthegoodman1]

Awesome, thanks!

[bobbyiliev]

No problem! Happy to help!