Server crash on startup - empty .internal files

[BelgianNoise]

Hey all,

We have an instance of the css (v6.0.0) running inside a Kubernetes cluster. From time to time, when the server restarts (pod in k8s), it will crash because it is trying to read files from the /.internal/setup folder which are empty for some reason. (The files containing the version number and the base url)
This errors the server because it tries to parse the content as json, which an empty string clearly is not.

(see attached logs below from when I empty these files and run locally on my machine)

So I guess my question is: Why are these files empty ? Does the server ever write to these files again and could that be the reason of the files being empty ?

Since I am not able to reproduce this (on the deployed version) just like that and it doesn't always happen, I am not sure where to even look. Maybe this is a known issue that was fixed in a later version 🤷🏻 .
Until now, we have gone in and deleted the files manually afterwhich the server boots correctly.

Thanks in advance for the help :)

2023-12-21T08:45:28.978Z [Components.js] info: Initiating component discovery from /Users/arthur/Documents/useid/apps/useid-pods
2023-12-21T08:45:29.034Z [Components.js] info: Discovered 1 component packages within 61 packages
2023-12-21T08:45:29.035Z [Components.js] info: Initiating component loading
2023-12-21T08:45:31.546Z [Components.js] info: Registered 459 components
2023-12-21T08:45:31.546Z [Components.js] info: Loaded configs
2023-12-21T08:45:33.977Z [AppRunner] {Primary} error: Could not start the server: Unexpected end of JSON input
Could not start the server
Cause: Unexpected end of JSON input
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at JsonResourceStorage.get (/Users/arthur/Documents/useid/node_modules/.pnpm/@solid+community-server@6.0.0/node_modules/@solid/community-server/dist/storage/keyvalue/JsonResourceStorage.js:34:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async BaseUrlVerifier.handle (/Users/arthur/Documents/useid/node_modules/.pnpm/@solid+community-server@6.0.0/node_modules/@solid/community-server/dist/init/BaseUrlVerifier.js:19:31)
    at async SequenceHandler.handle (/Users/arthur/Documents/useid/node_modules/.pnpm/@solid+community-server@6.0.0/node_modules/@solid/community-server/dist/util/handlers/SequenceHandler.js:27:26)
    at async SequenceHandler.handle (/Users/arthur/Documents/useid/node_modules/.pnpm/@solid+community-server@6.0.0/node_modules/@solid/community-server/dist/util/handlers/SequenceHandler.js:27:26)
    at async App.start (/Users/arthur/Documents/useid/node_modules/.pnpm/@solid+community-server@6.0.0/node_modules/@solid/community-server/dist/init/App.js:17:9)
    at async AppRunner.runCli (/Users/arthur/Documents/useid/node_modules/.pnpm/@solid+community-server@6.0.0/node_modules/@solid/community-server/dist/init/AppRunner.js:128:13)

[BelgianNoise]

I did a little more debugging and it seems to just be the json file containing the base url. The server seems to handle the empty version file just fine.

[joachimvh]

Why are these files empty ? Does the server ever write to these files again and could that be the reason of the files being empty ?

I have no idea unfortunately. If something goes wrong writing those files the server should just throw an error which should stop the server. The base URL resource is is only read and written in this file: https://github.com/CommunitySolidServer/CommunitySolidServer/blob/main/src/init/BaseUrlVerifier.ts, so I would expect the set call there to fail if it couldn't write.

Solution might just be to update the server so it ignores/deletes JSON files it can't interpret, although that has the risk of losing data.

[BelgianNoise]

Just to make sure, what exactly do you mean by "update" ?

1. Is this a change that is implemented already in a newer version ?
2. add a custom Verifier or reader (not sure what it's called here :) )
3. change config to use an already existing reader that ignores uninterpretable json

[joachimvh]

Ah, I meant more that this is a feature that could be added to the server to work around this problem. Currently there is nothing like that and your only solution is indeed to delete the empty file.

I would like to know why it occurs though 😅

[BelgianNoise]

You and me both :)
Sadly it's a real hassle to debug this on our Kubernetes cluster and since I can not reproduce this in any way, I am thinking it's something to do with our NFS and/or the mount of it.

We plan on moving our deployment anyways, so lets hope it's just an "us"-problem.