[Firestore] Error: connect EADDRNOTAVAIL 2607:f8b0:400e:c01::5f:443 -> Thousand times over a day

[MorenoMdz]

• Operating System version: Ubuntu
• Firebase SDK version: ^11.9.0
• Firebase Product: Firestore
• Node.js version: 16.14.0
• NPM version: 8.3.1

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

We have a NestJS server running on Render.com on a GCP network.
All our Firestore calls seem to work fine, we can read and write data as expected, with very large access at some moments on a day, that is not a problem.
But while checking our DataDog logs, we notice thousands of errors that always have the same pattern:
connect EADDRNOTAVAIL 2607:f8b0:400e:c08::5f:443 - Local (:::0)
queryTxt ENODATA firestore.googleapis.com

But the requests itself work.

Relevant Code:

The Nest FB admin is setup like this:

@Module({})
export class FirestoreModule {
  static forRoot(options: FirestoreModuleOptions): DynamicModule {
    const optionsProvider = {
      provide: FirestoreOptionsProvider,
      useFactory: options.useFactory,
      inject: options.inject,
    }

    function initializeFirebaseAdminApp(configService) {
      if (!firebaseAdmin.apps.length) {
        // Initialize the firebase admin app
        // https:firebase.google.com/docs/reference/admin/node/admin.ServiceAccount
        firebaseAdmin.initializeApp({
          credential: firebaseAdmin.credential.cert({
            projectId: configService['projectId'],
            clientEmail: configService['clientEmail'],
            privateKey: JSON.parse(configService['privateKey']),
          }),
          databaseURL: configService['databaseURL'],
          storageBucket: configService['storageBucket'],
        })
        firebaseAdmin.firestore().settings({ ignoreUndefinedProperties: true })
      }
    }

    const dbProvider = {
      provide: FirestoreDatabaseProvider,
      useFactory: (configService: ConfigService) => {
        initializeFirebaseAdminApp(configService)
        return firebaseAdmin.firestore()
      },
      inject: [FirestoreOptionsProvider],
    }

    const collectionProviders = FirestoreCollectionProviders.map(
      (providerName) => ({
        provide: providerName,
        useFactory: (db) => db.collection(providerName),
        inject: [FirestoreDatabaseProvider],
      }),
    )

    return {
      global: true,
      module: FirestoreModule,
      imports: options.imports,
      providers: [
        optionsProvider,
        dbProvider,
        ...collectionProviders,
        ...collectionHelpers,
      ],
      exports: [dbProvider, ...collectionProviders, ...collectionHelpers],
    }
  }
}

We use Firestore extensively with:

• Transactions
• Batches
• Geo queries

But we cant align the errors with any specific request, it seems to happen a lot and everywhere. Just to confirm, the Firestore requests seem to work correctly even with these error Spans in DataDog, as the actual request works fine and returns/updates the data as needed.

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[ehsannas]

Hi @MorenoMdz , thanks for reporting. I am not familiar with the NestJS setup you're using. The error message seems to be related to your networking (e.g. are you using a VPN? network proxy? is the address and port available/reachable?).

[MorenoMdz]

Hi @MorenoMdz , thanks for reporting. I am not familiar with the NestJS setup you're using. The error message seems to be related to your networking (e.g. are you using a VPN? network proxy? is the address and port available/reachable?).

Hey, we are not using a VPN or proxy, our service runs on a Render.com Node service without any specific configurations. The weird part is that the requests do work normally, all the Firebase calls like Auth and Storage work fine and so do the Firestore calls, but we do see those errors happening in our DataDog logs.

Could it be related to the Firestore snapshot listeners? We have a listener open to capture any changes in our feature flag document we have in Firestore, I wonder if those could be affected by the Node express requests somehow?

[ehsannas]

It could be related to your listeners. That could explain why it occurs many times over the course of the day. Might be due to intermittent connectivity issues and the listener reacting to them. Listeners also try to reconnect if they experience non-fatal errors.

[MorenoMdz]

It could be related to your listeners. That could explain why it occurs many times over the course of the day. Might be due to intermittent connectivity issues and the listener reacting to them. Listeners also try to reconnect if they experience non-fatal errors.

Any ideas on how we can handle these? Our NestJs server will initialize the listener every time a new instance of the service starts and should reflect any changes to the specific document in the upcoming requests, now sure why it is causing the errors in the DataDog span list but not actually throwing any exceptions anywhere else.

For a bit more context, this is what one of the requests looks like

[MorenoMdz]

I think it is related to this, I disabled the onSnapshotListener and deployed to test and started seeing the GRCP RST errors like in this issue:

googleapis/nodejs-firestore#1023 (comment)

[ehsannas]

Thanks for digging further into it. The nodejs-firestore repo is the right place for filing and tracking progress for Firestore-specific Node.js issues. I'll close this issue and I'll ping the team to see if we can get a second look at googleapis/nodejs-firestore#1023