Document that this project is in maintenance mode?

[snarfed]

From googleapis/google-cloud-python#10566 (comment) :

The current owners of the Datastore libraries team let me know that NDB is not currently being developed, but things may change in the future.

OK! Fair enough. Maybe consider documenting that in the README and on https://googleapis.dev/python/python-ndb/latest/ ? And also whether it's maintained, and just not actively adding features, or totally unmaintained?

(Thank you for all of your work up until this point, btw!)

[meredithslota]

Hi @snarfed — thanks for filing this issue. Agreed that projects should clearly communicate their level of support. We're doing our planning for the next couple of quarters now and will follow-up.

[jacobg]

I'm sad to read that NDB won't be actively updated moving forwards. NDB is a highly productive library with useful higher level abstractions, such as typed data models, conjunctive/disjunctive queries, etc. Whereas the Python Datastore library is a lower level library that requires a lot more app code for parallel queries, in-memory filtering/merging, type validation, etc. Both have their use cases.

Even if NDB is pretty mature and doesn't need major new features, we still need its dependencies to be actively bumped in order for it to keep up with the latest ecosystem.

[jacobg]

I briefly took a look at what it would take to bump dependencies, i.e., edit setup.py:

        "google-cloud-datastore >= 2.7.0, < 3.0.0",
        "pymemcache",
        "redis",
        "pytz"

It's not quite a trivial upgrade. The whole way the google-cloud-datastore generates code stubs changed:
https://github.com/googleapis/python-datastore/blob/main/UPGRADING.md

There's a lot of calls in the ndb code directly to proto and grpc functions that has totally changed. For me, I'd have to delve into how it all works, which I haven't dealt with before. There'd probably be value in making a community effort, so it's not just every individual doing this privately.

Has anyone attempted this?

[jacobg]

I went ahead, and bumped to google-cloud-datastore 2 on a forked repo here:
https://github.com/jacobg/python-ndb/tree/datastore-v2

Here are the commit notes:

[Release 2.0.0:]
Bump to google-cloud-datastore v2;
Drop Python 2.7 and 3.6 (match versions supported by GAE/GCF runtimes);
Default to Python 3.9

All unit and systems tests pass on my dev computer. I've also updated my own business applications to use it, and there are no known issues.

Here are a few extra thoughts:

1. It would be great if Google were to merge and take on maintaining this v2 branch. It could even be a non-main v2 branch on this repo, as long as they merge future commits into it so that it stays up to date. Or if anyone has suggestions to make this branch a community repo so that the responsibility of maintaining it does not fall upon me.
2. It would great to be able to enable CI on GitHub to run all tests for this fork. It seems that Google has their own internal build system, and they just clone the source code to GitHub without using GitHub for CI.
3. Cloud Datastore recently released native support for !=, IN, and NOT_IN query filters for Firestore in Datastore mode (legacy Datastore not supported). It would be great to take advantage of that NDB to avoid multiple queries where possible. -- https://cloud.google.com/datastore/docs/release-notes#June_17_2022

[snarfed]

@jacobg oh wow, awesome, thank you for doing this! I tried it out and ran the tests for a handful of my projects, ~2k total, and ran a couple of the apps locally and poked at them interactively, and everything passed and seems fine.

Re your extra thoughts, fully agreed with 1 and 3. For 2, I assume you mean the ndb unit tests? Looking at an example PR eg #778, doesn't look like they're run automatically in CI. Maybe try setting up and running them locally?

[jacobg]

@snarfed With regards to CI, what I means is we should configure Travis CI or something like that so that unit tests automatically run when a commit is made. And shows badges on the README. It does seem Google has some of its own build hooks in this repo, but not sure if we would use that for an independent repo, and it doesn't seem they run unit tests.

[meredithslota]

Hi @snarfed — thanks for filing this issue. Agreed that projects should clearly communicate their level of support. We're doing our planning for the next couple of quarters now and will follow-up.

Development efforts are ramping back up after a quiet period; thanks for your patience here (and thanks @rwhogg for your recent contributions!).

[meredithslota]

Dependency updates pending in #841

[zyzniewski]

How does this project compare to https://cloud.google.com/appengine/docs/standard/python3/reference/services/bundled/google/appengine/ext/ndb?

[rwhogg]

@zyzniewski the library documented on that page is the App Engine NDB library (source code). It's an older version of this project and we recommend you use this one, which is the Cloud NDB library.

The most important differences are:

1. The App Engine NDB library does not support Python 3
2. The App Engine NDB library only supports the App Engine environment

Whereas Cloud NDB supports Python 3 (some older releases also supported Python 2) and will work both within and outside the App Engine environment.

[zyzniewski]

@rwhogg I'm not referring to datastore-ndb-python but appengine-python-standard ("Google App Engine bundled services SDK for Python 3" https://github.com/GoogleCloudPlatform/appengine-python-standard) - which also provides ndb

[rwhogg]

Ah, sorry for the confusion. That is also a version of the App Engine NDB library that I mentioned above.

We recommend you use this repo's library rather than the one there.

[snarfed]

Hi all! I'd like to revisit this issue. Hope you don't mind!

Some work on this project is clearly happening, primarily by @rwhogg. Great, thank you! We really do appreciate it. It's not a lot, though. Details below.

I don't mean to be ungrateful. ndb is awesome, many of us have obviously used and benefited it for decades (!) now. Thank you! However, for those of us still actively building and maintaining our own projects on ndb, it's important for us to understand the maintainers' (ie Google's) level of ongoing commitment and investment here.

Will this project eventually be sunsetted? Will it continue on with minimal support? Or is there a path toward you all engaging more actively here? Any answer is ok! But if we want a datastore client library that's more actively maintained, do we need to start planning to eventually migrate to the official datastore client, or another library?

Thanks in advance.

--

Looking at recent PRs, the vast majority are administrative or chores: docs, tests, dependencies, other minor refactorings and cleanups. Which are good! But of the ~100 PRs over the last year, it looks like just five are actual bug fixes or otherwise noticeable code changes: #882, #877, #873, #852, #815.

That may be OK. Development on a mature, actively maintained project can be slow, and still reasonable. But looking at the ~25 issues filed over the last year, many of which are real bugs or problems, it looks like only five received any response at all from maintainers - #911, #897, #862, #836, #804 - and three of those were filed by maintainers themselves.

Previously: #816 etc

[snarfed]

(^ Moved to new issue, #916.)