[firestore-emulator: datastore-mode] Operation failed: transactional kindless queries are unimplemented

[artoale]

[REQUIRED] Environment info

firebase-tools: Using the gcloud cli version 467.0.0, firestore emulator: 1.19.2

Platform: macOS (intel), python client library

[REQUIRED] Test case

import os

from google.cloud import datastore
from google.cloud.datastore.query import PropertyFilter


def main():
    PORT = 8081
    os.environ["DATASTORE_EMULATOR_HOST"] = "127.0.0.1:%s" % PORT
    os.environ["DATASTORE_PROJECT_ID"] = "test"

    def get_new_client():
        return datastore.Client(
            project="test",
            namespace=None,
            _http=None,
        )

    client = get_new_client()

    kind = "a_kind"
    key = client.key(kind)
    entity = datastore.Entity(key=key)
    entity.update({"foo": "bar"})

    client.put(entity)

    key = entity.key

    assert key.is_partial == False

    with client.transaction():
        qry = client.query(namespace=None, ancestor=key)
        qry.keys_only()
        qry.add_filter(filter=PropertyFilter("__key__", "=", key))
        # import ipdb; ipdb.sset_trace()
        fetched = list(qry.fetch(limit=1))

    print(fetched)
    # list(fetched)[0].key == key


if __name__ == "__main__":
    main()

[REQUIRED] Steps to reproduce

1. Start the emulator with gcloud emulators firestore start --host-port=127.0.0.1:10901 --database-mode=datastore-mode
2. Install the python deps: pip install google-cloud-datastore==2.19.0
3. Run the above python script

[REQUIRED] Expected behavior

Completes, printing the entity created.

[REQUIRED] Actual behavior

An exception is raised:

<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = ""
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"", grpc_status:2, created_time:"2024-03-07T12:23:33.354476+01:00"}"
>
Traceback (most recent call last):
  File "/Users/aartoni/dev/django-gcloud-connectors/firestore.py", line 49, in <module>
    main()
  File "/Users/aartoni/dev/django-gcloud-connectors/firestore.py", line 42, in main
    fetched = list(qry.fetch(limit=1))
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 208, in _items_iter
    for page in self._page_iter(increment=False):
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/page_iterator.py", line 244, in _page_iter
    page = self._next_page()
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/cloud/datastore/query.py", line 813, in _next_page
    response_pb = self.client._datastore_api.run_query(
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/cloud/datastore_v1/services/datastore/client.py", line 637, in run_query
    response = rpc(
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/aartoni/.pyenv/versions/3.9.17/envs/gcloudc39/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 79, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.Unknown: None 

Emulator logs

Upon receiving this request, the emulator breaks with

[firestore] WARNING: Operation failed: transactional kindless queries are unimplemented
[firestore] java.lang.NullPointerException: transactional kindless queries are unimplemented
[firestore]     at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:990)
[firestore]     at com.google.cloud.datastore.emulator.impl.firestore.FirestoreEmulatorHelper.transactionalQuery(FirestoreEmulatorHelper.java:482)
[firestore]     at com.google.cloud.datastore.emulator.impl.firestore.FirestoreEmulatorHelper.lambda$query$11(FirestoreEmulatorHelper.java:459)
[firestore]     at com.google.cloud.datastore.emulator.impl.transactions.EmulatorTransactionManager$ReadWriteTransaction.fold(EmulatorTransactionManager.java:439)
[firestore]     at com.google.cloud.datastore.emulator.impl.firestore.FirestoreEmulatorHelper.query(FirestoreEmulatorHelper.java:457)
[firestore]     at com.google.cloud.datastore.emulator.impl.firestore.CloudDatastoreV1.runQuery(CloudDatastoreV1.java:298)
[firestore]     at com.google.cloud.datastore.emulator.impl.firestore.CloudDatastoreV1Router.runQuery(CloudDatastoreV1Router.java:83)
[firestore]     at com.google.cloud.datastore.emulator.firestore.cloudv1.CloudDatastoreV1GrpcAdapter$1.lambda$runQuery$2(CloudDatastoreV1GrpcAdapter.java:74)
[firestore]     at com.google.cloud.datastore.emulator.firestore.cloudv1.CloudDatastoreV1GrpcAdapter.unary(CloudDatastoreV1GrpcAdapter.java:49)
[firestore]     at com.google.cloud.datastore.emulator.firestore.cloudv1.CloudDatastoreV1GrpcAdapter$1.runQuery(CloudDatastoreV1GrpcAdapter.java:74)
[firestore]     at com.google.datastore.v1.DatastoreGrpc$MethodHandlers.invoke(DatastoreGrpc.java:742)
[firestore]     at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
[firestore]     at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:351)
[firestore]     at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:861)
[firestore]     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[firestore]     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
[firestore]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[firestore]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[firestore]     at java.base/java.lang.Thread.run(Thread.java:833)
[firestore] 

Additional information

Removing the client.transaction() context manager results in the emulator not reporting any error, but the test case above prints an empty list, while running the example against an actual firestore in datastore mode instance returns the entity that was created just before.

[harshyyy21]

Hi @artoale, looking into the issue here. Will leave some updates once root cause is known!