[BUG] LiteDB 5.0.10 - Maximum number of transactions reached

[byteAbit0101]

.NET Standard 2.0
LiteDB version 5.0.10

//database is a singleton instance
protected ILiteCollection<T> GetCollectionWithIndex()
{
   var collection = database.GetCollection<T>(collectionName);
   collection.EnsureIndex(x => x.InternalId);
   collection.EnsureIndex(x => x.Name);
   return collection;
}

public async Task<T> FindOneAsync(BsonExpression expression)
{
   var collection = database.GetCollectionWithIndex(); 
   return await resilientPolicy.ExecuteAsync(async () =>
   {
        var result = collection.FindOne(expression);
         await Task.CompletedTask;
        return result;
   });
}

TransactionMonitor.GetTransaction (System.Boolean create, System.Boolean queryOnly, System.Boolean& isNew)
Maximum number of transactions reached

[lbnascimento]

@byteAbit0101 LiteDB has a limit of 100 open transactions at a time. If youy program happens to create a lot of these FindOneAsync tasks at the same time, it may reach the limit.

[byteAbit0101]

There are not 100 transactions open at a time, except LiteDB is not closing them. Is there any possibility to trace how many are still open or which ones weren't closed?

The issue is once it's running into this exception the database file gets unusable until everything is disposed and resetup. I even managed that it's even with the dispose not working anymore and both files needed to be deleted and recreated which is critical due to potential data loss.

[alb3ric]

Hello,

I'm encountering the same error with 5.0.10 and .net 4.7 in a production database.
I'm going to dig deeper to see if I can get more info about the cause. I'm just doing insert on a multithread windows service (but not 100 at a time..., Like 5-6).

Regards,

[alb3ric]

Hello,
Me again. I analyzed the problem, it s happeingn after I did a rebuild of the database (reduced of 8Go).

Then each time I'm trying to insert in the filestorage I have this error (I'm not closing the database after the rebuild) :
-> LiteDB ENSURE: page type must be collection page

And then after 100 occurences of this error I get :
-> Maximum number of transactions reached

It's like the "LiteDB ENSURE: page type must be collection page" is not closing the collection

I think my problem is linked with #1958

[DamienDoumer]

Hello,

Please, did anyone find a solution to this problem ? I have this issue in production.

[dgodwin1175]

Hi,
I am also seeing this issue.

In LiteDB 4.x we had our own locking around the LiteDb database, and after upgrading to 5.0.16 we started seeing the application hanging trying to get the lock to the database.
After reading that LiteDb was now threadsafe, we removed the locking and are relying solely on the internal locking done by LiteDb.
This was working fine, occaisonally we would see a lock timeout, like this:

Caught exception: Collection 'redacted' lock timeout when entering in write mode after 00:01:00
at LiteDB.Engine.LockService.EnterLock(String collectionName)
at LiteDB.Engine.Snapshot..ctor(LockMode mode, String collectionName, HeaderPage header, UInt32 transactionID, TransactionPages transPages, LockService locker, WalIndexService walIndex, DiskReader reader, Boolean addIfNotExists)
at LiteDB.Engine.TransactionService.g__create|43_0(<>c__DisplayClass43_0& )
at LiteDB.Engine.TransactionService.CreateSnapshot(LockMode mode, String collection, Boolean addIfNotExists)
at LiteDB.Engine.LiteEngine.<>c__DisplayClass27_0.b__0(TransactionService transaction)
at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func2 fn) at LiteDB.Engine.LiteEngine.Update(String collection, IEnumerable1 docs)
at LiteDB.LiteCollection`1.Update(T entity)

The application would recover after this and continue working just fine, however, it seems the lock timeouts became more prevalent, and then we started seeing the max transactions reached exceptions:

Caught exception: Maximum number of transactions reached
at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
at LiteDB.Engine.QueryExecutor.ExecuteQuery()
at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
at LiteDB.LiteQueryable1.<ToDocuments>d__26.MoveNext() at System.Linq.Enumerable.Single[TSource](IEnumerable1 source)
at LiteDB.LiteQueryable1.Count() at LiteDB.LiteCollection1.Count()

After this, the application does not recover and database access is not possible at all.
A restart of the application is the only way to recover.

I suspect each time we get the lock timeout, a LiteDb transaction is hung and never released, once we get this error 100 times, we get the max transactions reached error.

@lbnascimento, @mbdavid, any ideas on this problem?

[PeterHagen]

I run into this exception also on LiteDB 5.0.18

LiteDB.LiteException: Maximum number of transactions reached
    at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
    at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
    at LiteDB.LiteQueryable`1.ToDocuments()+MoveNext()
    at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
    at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)

[dgodwin1175]

+1
Same here. As per #2418 , we have installed the latest version 5.0.18 and are getting this exception, after only a few accesses to the database:

Caught exception: Maximum number of transactions reached
tack trace following:
at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew)
at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan)
at LiteDB.Engine.QueryExecutor.ExecuteQuery()
at LiteDB.Engine.LiteEngine.Query(String collection, Query query)
at LiteDB.LiteQueryable1.<ToDocuments>d__26.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext()
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)

@mbdavid , any thoughts on this?

[PeterHagen]

On 5.0.18 and 5.0.19 I have the same. I'm going to try to revert to 5.0.17. My applications are not working anymore unfortunately

[edoust]

I have the same issue on v5.0.19, @PeterHagen did reverting to v5.0.17 solve the issue for now?

[b-zijlstra]

I have the same issue on v5.0.19, @PeterHagen did reverting to v5.0.17 solve the issue for now?

I had this issue on v5.0.18 and reverting to v5.0.17 fixed it.

[dgodwin1175]

Note that this issue is fixed by #2435 , but the fix is not included in any release yet.

We test this fix by rolling it into 5.0.19 and found that it resolves the "Maximum number of transactions reached" error, and also the DiskWriterQueue locking issue #2307 which has been preventing us from upgrading to v5.

So please push for a new release (5.0.20) which includes #2435 :)

[nsulikowski]

I have the same issue. Please release the fix!

[nikubesliu]

Downgrade to v5.0.17 didn't help.

[radektomis]

Downgrade to 5.0.17 helped in my case.

[mikebm]

Had the same issue here. After deleting my log file, it no longer had the issue. My previous run was attached to a debugger which I killed the process which left a 4096kb log file. Deleting it (which was fine in my test case) resolved the issue. Bizzare...

Edit: Scratch that. Ran into it again. Downgrading to v5.0.17 from v5.0.19 seems to have fixed it.

[DanielGoehler]

We had the same problem after upgrading from 5.0.16 to 5.0.19.

[MathieuCrossover]

I use liteDB on Xamarin forms, I have the same exception in IOS but not in Android, on which platform do you use LiteDb?

[rualb]

Same issue with version 5.0.19
Fixed by downgrade to version 5.0.17

[RvanSchr]

Same here (5.0.19), windows 11 Pro, .NET8.0. I was performing a lot of GetCollecton<T> followed by FindAll(). Downgraded to 5.0.17 and the error did not occur. Did not try other versions.

[petertiedemann]

With 5.0.17, I get "LiteDB ENSURE: page type must be collection page" from our test suite. With 5.0.19 Maximum number of transactions reached :( We have been stuck on v4 for awhile.

[jluniba]

Same issue here: "LiteDB ENSURE..." with 5.0.17 and "Maximum number of transactions reached.." with 5.0.19.

We are definitely stuck since v4 is also marked as vulnerable.

Any expected date for new release?

[bestouff]

Jellyfin (10.9 released this week) has the same problem too:

mai 13 13:41:38 awak jellyfin[2428612]: LiteDB.LiteException: Maximum number of transactions reached 
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.Engine.TransactionMonitor.GetTransaction(Boolean create, Boolean queryOnly, Boolean& isNew) 
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.Engine.QueryExecutor.ExecuteQuery(Boolean executionPlan) 
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.Engine.LiteEngine.Query(String collection, Query query) 
mai 13 13:41:38 awak jellyfin[2428612]:    at LiteDB.LiteQueryable`1.ToDocuments()+MoveNext() 
mai 13 13:41:38 awak jellyfin[2428612]:    at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext() 
mai 13 13:41:38 awak jellyfin[2428612]:    at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1] source, Boolean& found) 
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.Data.DbRepo.<>c__DisplayClass10_0.<SetUserInfoSync>b__0(UserItemDataDto dto) 
mai 13 13:41:38 awak jellyfin[2428612]:    at System.Collections.Generic.List`1.ForEach(Action`1 action) 
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.Data.DbRepo.SetUserInfoSync(List`1 dtos, List`1 itemRefs, String userId) 
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.EntryPoints.UserSyncNotification.SaveUserChanges(List`1 dtos, List`1 itemRefs, String userName, String userId) 
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.EntryPoints.UserSyncNotification.SendNotifications(IEnumerable`1 changes, List`1 itemRefs, CancellationToken cancellationToken) 
mai 13 13:41:38 awak jellyfin[2428612]:    at Jellyfin.Plugin.KodiSyncQueue.EntryPoints.UserSyncNotification.UpdateTimerCallback(Object state)