RavenDB 5.4.112: Query for nullable types with Linq returns unexpected results

[bwmeyer]

When querying for a nullable field, it is possible to query for e.g
"[NullableValueField].HasValue == true" or "[NullableValueField] == null".
This works, as long as the field itself is present in the document.

But when the field is missing, the result of the query contains unexpected documents.
A filter like "IsLocked != null" will return documents, where the field does not exist.

RavenDB 5.4.112
.net 7.0.14 x64

Here the failing test:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json;
using Raven.Client;
using Raven.Client.Documents;
using Raven.Client.Documents.Operations;
using Raven.Client.Json.Serialization.NewtonsoftJson;
using Raven.TestDriver;
using Sparrow.Json;
using System;
using System.ComponentModel;
using System.Linq;

namespace RavenDBTest
{
    [TestClass]
    public class RavenDBTestDriver : RavenTestDriver
    {
        protected override void PreInitialize(IDocumentStore documentStore)
        {
            documentStore.Conventions.FindCollectionName = t => t.Name;
            documentStore.Conventions.FindIdentityProperty = m => m.Name == nameof(IDocument.Key);
            documentStore.Conventions.UseOptimisticConcurrency = true;
            documentStore.Conventions.DisableTopologyCache = true;
            documentStore.Conventions.MaxNumberOfRequestsPerSession = int.MaxValue;

            documentStore.OnBeforeQuery += (sender, args) => args.QueryCustomization.WaitForNonStaleResults();

            documentStore.OnBeforeStore += (sender, args) =>
            {
                DateTime timestampUtc = DateTime.UtcNow;

                args.DocumentMetadata[nameof(DocumentBase.TimestampUtc)] = timestampUtc;
                if (args.Entity is DocumentBase documentBase)
                {
                    documentBase.TimestampUtc = timestampUtc;
                }
            };

            Func<Type, BlittableJsonReaderObject, object> deserializeEntityFromBlittable = documentStore.Conventions.Serialization.DeserializeEntityFromBlittable;
            documentStore.Conventions.Serialization = new NewtonsoftJsonSerializationConventions
            {
                DeserializeEntityFromBlittable = (type, doc) =>
                {
                    object entity = deserializeEntityFromBlittable(type, doc);
                    if (entity is DocumentBase documentBase)
                    {
                        if (doc.TryGet(Constants.Documents.Metadata.Key, out BlittableJsonReaderObject metadata))
                        {
                            metadata.TryGet(nameof(DocumentBase.TimestampUtc), out DateTime timestampUtc);
                            documentBase.TimestampUtc = timestampUtc;
                        }
                    }
                    return entity;
                }
            };
        }

        [ClassInitialize]
        public static void ClassInitialize(TestContext testContext)
        {
            ConfigureServer(new TestServerOptions
            {
                FrameworkVersion = null,
                DataDirectory = "C:\\Temp\\RavenDBTestDir"
            });
        }

        [TestMethod]
        public void QueryHasValueTest()
        {
            using (var store = GetDocumentStore())
            {
                using (var session = store.OpenSession())
                {
                    session.Store(new TestDoc { Key = "Test 1 true", IsLocked = true });
                    session.Store(new TestDoc { Key = "Test 2 false", IsLocked = false });
                    session.Store(new TestDoc { Key = "Test 3 null", IsLocked = null });
                    session.Store(new TestDoc { Key = "Test 4 null", IsLocked = null });
                    session.Store(new TestDoc { Key = "Test 5 null", IsLocked = null });
                    session.SaveChanges();
                }

                //OpenBrowser(store.Urls[0]);

                using (var session = store.OpenSession())
                {
                    var query1 = session.Query<TestDoc>().Where(x => x.IsLocked == true);
                    var query1Result = query1.ToList();
                    Assert.AreEqual(1, query1Result.Count);

                    var query2 = session.Query<TestDoc>().Where(x => x.IsLocked == false);
                    var query2Result = query2.ToList();
                    Assert.AreEqual(1, query2Result.Count);

                    var query3 = session.Query<TestDoc>().Where(x => x.IsLocked != null);
                    var query3Result = query3.ToList();
                    Assert.AreEqual(2, query3Result.Count);

                    var query4 = session.Query<TestDoc>().Where(x => x.IsLocked == null);
                    var query4Result = query4.ToList();
                    Assert.AreEqual(3, query4Result.Count);

                    var query5 = session.Query<TestDoc>().Where(x => x.IsLocked.HasValue);
                    var query5Result = query5.ToList();
                    Assert.AreEqual(2, query5Result.Count);

                    var query6 = session.Query<TestDoc>().Where(x => !x.IsLocked.HasValue);
                    var query6Result = query6.ToList();
                    Assert.AreEqual(3, query6Result.Count);
                }

                using (var session = store.OpenSession())
                {
                    session.Advanced.DocumentStore.Operations.ForDatabase(session.Advanced.DocumentStore.Database).Send(new PatchByQueryOperation(
                        """
                        from TestDoc
                        update 
                        { 
                            delete this.IsLocked;
                        }
                        """)).WaitForCompletion(TimeSpan.FromSeconds(30));
                }

                using (var session = store.OpenSession())
                {
                    var query1 = session.Query<TestDoc>().Where(x => x.IsLocked == true);
                    var query1Result = query1.ToList();
                    Assert.AreEqual(0, query1Result.Count);

                    var query2 = session.Query<TestDoc>().Where(x => x.IsLocked == false);
                    var query2Result = query2.ToList();
                    Assert.AreEqual(0, query2Result.Count);

                    var query3 = session.Query<TestDoc>().Where(x => x.IsLocked != null);
                    var query3Result = query3.ToList();
                    Assert.AreEqual(0, query3Result.Count);

                    var query4 = session.Query<TestDoc>().Where(x => x.IsLocked == null);
                    var query4Result = query4.ToList();
                    Assert.AreEqual(5, query4Result.Count);

                    var query5 = session.Query<TestDoc>().Where(x => x.IsLocked.HasValue);
                    var query5Result = query5.ToList();
                    Assert.AreEqual(0, query5Result.Count);

                    var query6 = session.Query<TestDoc>().Where(x => !x.IsLocked.HasValue);
                    var query6Result = query6.ToList();
                    Assert.AreEqual(5, query6Result.Count);
                }
            }
        }

        public interface IDocument
        {
            string Key { get; }
        }

        public class DocumentBase : IDocument
        {
            [DesignerSerializationVisibility(DesignerSerializationVisibility.Hidden)]
            [ImmutableObject(true)]
            [ReadOnly(true)]
            public string Key { get; set; }

            [JsonIgnore]
            [Browsable(false)]
            [DesignerSerializationVisibility(DesignerSerializationVisibility.Hidden)]
            [ImmutableObject(true)]
            [ReadOnly(true)]
            public DateTime TimestampUtc { get; set; }
        }

        public class TestDoc : DocumentBase
        {
            public string Id { get; set; }
            public bool? IsLocked { get; set; }
        }
    }
}

[Lwiel]

IsLocked != null is effectively translated to true and not IsLocked == null. Because the right-hand side expression is true for each of your documents, all of them are returned.

Handling non-existing fields is explained with more details in our docs here and here.

Your 3rd and 4th query could look like this to achieve the desired result

var query3 = session.Advanced.DocumentQuery<TestDoc>().WhereExists("IsLocked");
var query3Result = query3.ToList();
Assert.Equal(0, query3Result.Count);

var query4 = session.Advanced.DocumentQuery<TestDoc>().Not.WhereExists("IsLocked");
var query4Result = query4.ToList();
Assert.Equal(5, query4Result.Count);

[ayende]

The idea here is that for existence (vs. nullability) you need to test differently.
In general, we assume that your fields exists unless you explicitly state otherwise. That is the same for nullables (sorry for misleading earlier). Since you are checking for a value, not the absence of one.

[bwmeyer]

For my understanding, the big benefit of a NoSql-database is that you don't need to enforce the schema of every document.
To extend a document by a new field, you need just to include the field in a new document.
All "old" documents don't need to be altered at all.

But if this will lead into unexpected/wrong query-results, there needs to be a schema-enforcement on database level.
Whenever a json document is persisted/patched/restored in the database, you have to validate for a missing field.
Whenever you load an older database or are deploying a new software release, you have to enforce adding missing fields on older documents.

Currently, I will just enforce updating fields for the whole database, when deploying a new software release.
So I can be sure my queries run as expected for that moment.
But e.g. if a revision of a document is restored, the query results are broken.

To be honest, I don't see a stable and easy solution to enforce a schema on the database level.
It would be much easier to just exclude results from the query results, where the field just not exists.

[ayende]

The problem in this case is that you are querying on negation, which is a relatively rare scenario. In the case where you are querying on a value (equivalence, range or prefix), the system will behave in the way you specified, yes.

For negation, this is more complex, since we need to tell explicitly whatever the field is set to null or not there at all.

The default behavior is to treat this as a does not equate to null which also include items where the value doesn't exist.
You can customize that (here is how:

ravendb/test/SlowTests/Issues/RavenDB_12943.cs

Line 34 in cf8f473

private class MyQueryMethodConverter : QueryMethodConverter

) if you really want, but in that case, I would actually recommend to use the explicit DocumentQuery API instead

[bwmeyer]

I don't think it's rarely to query for nullable fields, if they have a value or not.
Your hint about the QueryMethodConverter is promising, but I cannot find a documentation about it, and it looks that I need to implement the whole expression tree by my self.

If I could just convert every occurrence of Nullable<>.HasValue in a generic way, this would be great:
session.Query<TestDoc>().Where(x => x.IsLocked.HasValue);
should become something like this:
from TestDoc where (exists(IsLocked) and IsLocked != null)

Is there a way to accomplish this on a Linq-Query?

[arekpalinski]

Something that could be tried is the usage of OnBeforeQuery event:
https://ravendb.net/docs/article-page/5.4/Csharp/client-api/session/how-to/subscribe-to-events#onbeforequery

It's a bit tricky because I'd require explicit casting but it could be something like:

session.Advanced.OnBeforeQuery += (sender, args) =>
{
    var queryToBeExecuted = (DocumentQuery<Article>) args.QueryCustomization;
    queryToBeExecuted.AndAlso(wrapPreviousQueryClauses: true);
    queryToBeExecuted.WhereExists(nameof(Article.IsDeleted));
};