Blank error message when trying to insert record into cloud datastore from cloud run

[mdobson]

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS: Google Cloud Run -
• Node.js version: 10.17.0
• npm version: 6.11.3
• @google-cloud/datastore version: 4.5.2

Steps to reproduce

1. Hit microservice with http request in cloud run
2. Application receives request (either in cold start or when already running)
3. Begin cloud datastore insert of record
4. Error with no message occurs and application returns 500 error within application server

We are seeing weird intermittent behavior in cloud run using the cloud datastore library. Periodically when trying to insert a record to cloud datastore we'll see a connection error with no message and the following stack trace. We are having a hard time trying to diagnose and remedy this issue aside from a hard crash a retry of the particular process. I couldn't really find other issues that had this specific error in cloud datastore so I posted it here.

Error: <no message>
at Http2CallStream.call.on (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js line 96 col 44)
at Http2CallStream.emit (events.js line 203 col 14)
at process.nextTick (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js line 71 col 21)
this.emit('status', status);
at process._tickCallback (internal/process/next_tick.js line 61 col 10)

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[bcoe]

👋 thanks for the bug report @mdobson.

[mdobson]

Happy to help!

[vishald123]

Hi @mdobson, I deployed the following code to Google cloud run which inserts data to the cloud Datastore. I couldn't reproduce the error please let me know if I'm missing anything.

const express = require('express');
const app = express();

app.get('/', (req, res) => {
  async function addTask() {
    const { Datastore } = require('@google-cloud/datastore');
    const projectId = "[YOUR_PROJECT_ID]";

    const datastore = new Datastore({ projectId: projectId });

    const taskKey = datastore.key({ namespace: "Apple1", path: ["iPhone2", "2007"] });

    const data = {
	name: 'iPhone3Gs',
	created: "2009"
    };

    const entity = {
	key: taskKey,
	data: data
    };

    datastore.save(entity)
      .then((response) => {
        res.send(`This is the ${JSON.stringify(response[0])}`);
      })
      .catch((error) => {
	res.send(`error`, error)
      });
  }
  addTask();
});

const port = 8080;
app.listen(port, () => {
	console.log(`Server started on ${port}`);
});

[mdobson]

Hi @vishald123 We see the behavior when there are many inserts performed at once in cloud run.

[kevinohara80]

This issue keeps coming back and biting us. It’s very sporadic this making it hard to reproduce. Has there been any progress on this? Any other issues in other GCP libs that are similar?

[mdobson]

Hi Google APIs Team,

We're still seeing this issue, but with some slight changes. The error will have a similar stack trace, but now send back an error message of Error: 13 INTERNAL: at Object.callErrorFromStatus. This will randomly disrupt inserts and other operations to datastore so we were wondering if there were any recommendations on how to remedy the issue, or if there is a potential fix in the works.

Stack trace:

(/workspace/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:30:26) at Http2CallStream.call.on (/workspace/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:33) at Http2CallStream.emit (events.js:203:15) at Http2CallStream.EventEmitter.emit (domain.js:466:23) at process.nextTick (/workspace/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22) at process._tickCallback (internal/process/next_tick.js:61:11)

Thanks,

-Matt

[ChrisWestcottUK]

We are also seeing this issue using @google-cloud/datastore version: 5.1.0 on Cloud Run:

"Error: 13 INTERNAL: 
    at Object.callErrorFromStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
        -> /app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call.ts:79:24
    at Http2CallStream.<anonymous> (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:96:33)
        -> /app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/client.ts:155:18
    at Http2CallStream.emit (events.js:327:22)
    at /app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:100:22
        -> /app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:186:14
    at processTicksAndRejections (internal/process/task_queues.js:79:11)"

[kevinohara80]

@ChrisWestcottUK Yeah, it's a nasty one. I hope they are looking at it though this thread has gone pretty stale. Any chance you've found a way to reproduce it consistently? We haven't been able to.

[meredithslota]

Another issue related to Error: 13 INTERNAL was recently fixed — googleapis/nodejs-firestore#1023 for more context (we centralized discussion there). The fix was recently released; if this issue is still happening for you once all dependencies have been upgraded, can you please add additional repro steps?

[mdobson]

@meredithslota Understood! We ended up incorporating exponential backoffs to solve this, but I'll perform the upgrade and report back.

[meredithslota]

I'm going to go ahead and close this out, but if you are still seeing issues after the upgrade, please file a new issue and reference this one. Thank you!