Periodic 503 Error

[nickchampion]

Hi,

I have some code that runs on AWS Lambda, triggered every 5 minutes, most the time this code runs successfully, but a few times a day it fails with the error below

"message": "An exception occurred while contacting https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true: request to https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true failed, reason: write EPIPE . \nFetchError: request to https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true failed, reason: write EPIPE\n    at ClientRequest.<anonymous> (/node_modules/.pnpm/node-fetch@2.7.0/node_modules/node-fetch/lib/index.js:1501:11)\n    at ClientRequest.emit (node:events:530:35)\n    at ClientRequest.emit (node:domain:488:12)\n    at TLSSocket.socketErrorListener (node:_http_client:495:9)\n    at TLSSocket.emit (node:events:518:28)\n    at TLSSocket.emit (node:domain:488:12)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\nThe server at https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true responded with status code: 503: request to https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true failed, reason: write EPIPE",
    "stack": [
        "RavenException: An exception occurred while contacting https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true: request to https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true failed, reason: write EPIPE . ",
        "FetchError: request to https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true failed, reason: write EPIPE",
        "    at ClientRequest.<anonymous> (/node_modules/.pnpm/node-fetch@2.7.0/node_modules/node-fetch/lib/index.js:1501:11)",
        "    at ClientRequest.emit (node:events:530:35)",
        "    at ClientRequest.emit (node:domain:488:12)",
        "    at TLSSocket.socketErrorListener (node:_http_client:495:9)",
        "    at TLSSocket.emit (node:events:518:28)",
        "    at TLSSocket.emit (node:domain:488:12)",
        "    at emitErrorNT (node:internal/streams/destroy:169:8)",
        "    at emitErrorCloseNT (node:internal/streams/destroy:128:3)",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)",
        "The server at https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true responded with status code: 503: request to https://db.ravendb.cloud/databases/Database/queries?queryHash=b9cf10b3c7d985140b7af07270c87de9&addTimeSeriesNames=true failed, reason: write EPIPE",
        "    at getError (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Exceptions/index.ts:40:19)",
        "    at Function.get (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Exceptions/index.ts:192:16)",
        "    at Function._readExceptionFromServer (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Http/RequestExecutor.ts:1915:36)",
        "    at _RequestExecutor._handleServerDown (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Http/RequestExecutor.ts:1574:43)",
        "    at _RequestExecutor._sendRequestToServer (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Http/RequestExecutor.ts:1146:29)",
        "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "    at _RequestExecutor._executeOnSpecificNode (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Http/RequestExecutor.ts:991:35)",
        "    at _DocumentQuery._executeActualQuery (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Documents/Session/AbstractDocumentQuery.ts:2163:9)",
        "    at _DocumentQuery.iterator (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Documents/Session/AbstractDocumentQuery.ts:2176:9)",
        "    at _DocumentQuery.all (/node_modules/.pnpm/ravendb@5.4.2/node_modules/ravendb/src/Documents/Session/AbstractDocumentQuery.ts:2182:25)",
        "    at recalculateLoadsStatus (/node_modules/.pnpm/@hectare+platform.modules.logistics.events@1.1.155_@aws-sdk+client-cloudwatch-logs@3.477.0_vitest@1.3.1/node_modules/@hectare/platform.modules.logistics.events/handlers/loads/recalculate-loads-status.ts:7:17)",
        "    at execute (/node_modules/.pnpm/@hectare+platform.components.context@1.1.140_@aws-sdk+client-cloudwatch-logs@3.477.0_vitest@1.3.1/node_modules/@hectare/platform.components.context/middy.ts:80:5)",
        "    at instance (/node_modules/.pnpm/@hectare+platform.components.context@1.1.140_@aws-sdk+client-cloudwatch-logs@3.477.0_vitest@1.3.1/node_modules/@hectare/platform.components.context/middy.ts:37:12)",
        "    at middy (/node_modules/.pnpm/@hectare+platform.components.context@1.1.140_@aws-sdk+client-cloudwatch-logs@3.477.0_vitest@1.3.1/node_modules/@hectare/platform.components.context/middy.ts:27:3)",
        "    at tryExecuteHandler (/node_modules/.pnpm/@hectare+platform.components.context@1.1.140_@aws-sdk+client-cloudwatch-logs@3.477.0_vitest@1.3.1/node_modules/@hectare/platform.components.context/handlers/event.ts:118:7)",
        "    at _EventsHandler.handle (/node_modules/.pnpm/@hectare+platform.components.context@1.1.140_@aws-sdk+client-cloudwatch-logs@3.477.0_vitest@1.3.1/node_modules/@hectare/platform.components.context/handlers/event.ts:126:9)"
    ]

Looks like a service unavailable type error, the RavenDB node is our test environment with a single node cluster.

I have a bunch of other Lambdas that are periodically failing due to a similar error, I'm not noticing any unavailability of the database other that from these Lambdas.

Any idea as to the cause?

Thanks

[ayende]

Is it possible that you are not running anything else in that time frame?
What I suspect may be happening is that you have a TCP connection to the server, but after a few minutes, it dies because of lack of activity.
Then you try to write to it, then get the EPIPE error, because it was disconnected.

Because you are running in a single node, there is no high availability to cover for that.

See also this: stripe/stripe-node#1040 (comment)

[nickchampion]

Thanks for the info @ayende that makes logical sense. I am creating the document store outside the lambda handler code and there is a reasonable probability that there is no other activity on the database between Lambda invocations.

I presume you are talking about a TCP connection the store has open to the server and this is whats timing out, we're creating a new session for each lambda invocation.

Is there anything I can do to check the store connection status or reset it?

[ayende]

The problem is that this happens behind the scenes of the entire system, due to the Lambda runtime nature.
You can try running this with a shorter TTL or disabling keep-alive - but that would have worse perf (need to re-connect each time).

Is that just for dev? If so, just ensure the lambda settings make the instance last longer. Can also try doing a setTimeout() that would ping the server once a second to keep the connection running.

Note that for production system, RavenDB's client will manage that for you.