How does the CSS handle requests and can the server run in multi-threading mode?

[SiemenVdH]

Recently I generated 40,000 accounts on my local CSS for test purposes. For every account I created one pod linked to one webID of the account. The server took a lot of time to generate all this accounts and pods and it did not feel efficient to me. My end goal is querying all of these accounts for resources in their pods, and because this first step took such a long time (there were a lot of internal server errors so I often needed to restart the process and look for wrongly generated accounts and pods manually in my file explorer) I decided to limit the further steps (generating tokens, creating resources in the pods, granting accesses) to only 1,000 accounts.

I checked my .internal folder after the 40,000th pod was generated and it contained around the 400,000 files, so that would mean the server stores 10 files per pod if every pod is linked to one account. Is the reason for the slow handling of the http requests the fact that the server needs to run over and search all these files with every request? Or is this not how it works? How does it work and will the sever react faster if it only needs to read data compared to generate new data (accounts, webids, pods, tokens, resources)?

Is it possible to run the server in multi-thread mode? I found an out dated version of the CS server where u could enable this by adding -w in the run command. Is this still possible in the most recent version of the server? If not would it be interesting to use multi-threading in the client that sends requests to the server or does this not matter?

For the generation of the accounts, pods and tokens I used the json API of the CSS.

Thanks in advance and please correct me if I misinterpret something or if I don't understand something correctly. I hope my problem is clear, if not don't hesitate to ask.

[joachimvh]

It's hard to say what exactly would be the issue, but chances are it is the file system. The files you see are part of the indexes that are used in the backend so the server can quickly find which pods are associated with which account and so on. All these files should be independent of each other as these are for all different accounts and pods, but perhaps once you have 40,000 files in the same folder, the file system starts having issues. With those numbers you probably want an actual database in the backend instead of a simulated one on the file system, but currently there is no implementation of that in CSS so that would have to be newly made.

will the sever react faster if it only needs to read data compared to generate new data (accounts, webids, pods, tokens, resources)?

I would expect so.

Is it possible to run the server in multi-thread mode?

This is still available: https://communitysolidserver.github.io/CommunitySolidServer/latest/usage/starting-server/#configuring-the-server. But if the issue is the file system, I'm not sure how much this will help. But definitely something you can try out.

[SiemenVdH]

I understand, but doesn't this mean that scalability becomes a problem if solid is ever going to be used as a standard for companies and private individuals in Flanders? Because I know the flemish government is heavily investing on solid to be used by its citizens. That the CSS is only meant for testing and developing purposes is clear to me but I can imagine that the more solid will be used, the more people are going to want to setup large test environments with the CSS. How does this go about then? Or will ESS be used for this scenario, how does ESS differ from CSS in the case of storage? Do they already use a database in their backend?

And thanks for the multi-threading tip, I must have overlooked the multi-threading option...sorry

Thanks anyway for the response!

[joachimvh]

The Solid spec does not mandate that anything has to be stored on the file system. That is just an implementation detail. The default CSS configurations do it like this because this makes it easiest for users to get started, as solutions that require database would require the user to then have a running database on their system. I don't know the exact details of ESS but I know they use a robust database solution in the backend for example. The fact is also that CSS can be extended to use other solutions if someone wants to use. It currently already has 2-3 backend solutions between in-memory, file-based, and SPARQL endpoint (although the latter can't be used for account data).

[SiemenVdH]

Ok, that seems clear to me. Thanks again for the informative response. I hope a backend database solution will soon be implemented so that large test environment setups can be done easily and fast :)

By the way, when I want to run the server in multi-threading mode using the -w parameter, it only works when I pass the standard value 1. When I want to pass -1,0, or 2 and above as value it creates an internal server error:

InternalServerError: [WebSocketMap] is not threadsafe and should not be run in multithreaded setups!

What could be the problem for this? I run the server with the standard configuration as follows:

community-solid-server -c @css:config/file.json -f .data