Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lock 'TRIGGER_ACCESS' attempt to return by: de9325af-3e1c-4ae9-a99b-24be994b75f4 -- but not owner! #1236

Closed
MartinDemberger opened this issue Jul 6, 2021 · 1 comment · Fixed by #1242
Closed

Lock 'TRIGGER_ACCESS' attempt to return by: de9325af-3e1c-4ae9-a99b-24be994b75f4 -- but not owner! #1236

MartinDemberger opened this issue Jul 6, 2021 · 1 comment · Fixed by #1242
Milestone
3.3.3

Comments

@MartinDemberger
Copy link
Contributor

Describe the bug

In my application I got following warning: Lock 'TRIGGER_ACCESS' attempt to return by: de9325af-3e1c-4ae9-a99b-24be994b75f4 -- but not owner!

The stacktrace in the other log message is (cleaned):

   at Quartz.Impl.AdoJobStore.DBSemaphore.ReleaseLock(Guid requestorId, String lockName, CancellationToken cancellationToken)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<ReleaseLock>d__146.MoveNext()
   at Quartz.Impl.AdoJobStore.JobStoreSupport.ReleaseLock(Guid requestorId, String lockName, Boolean doIt, CancellationToken cancellationToken)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<ExecuteInNonManagedTXLock>d__269`1.MoveNext()
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<RetryExecuteInNonManagedTXLock>d__265`1.MoveNext()
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<ExecuteInNonManagedTXLock>d__269`1.MoveNext()
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<>c__DisplayClass269_0`1.<<ExecuteInNonManagedTXLock>b__0>d.MoveNext()
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<>c__DisplayClass233_0.<<AcquireNextTriggers>b__1>d.MoveNext()
   at Quartz.Impl.AdoJobStore.StdAdoDelegate.<SelectInstancesFiredTriggerRecords>d__122.MoveNext()

(the full stacktrace is below)

Version used

3.3.2

To Reproduce

I can't reproduce it. It only happens sometimes.

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

It looks to me like the lock is obtained twice by calling ExecuteInNonManagedTXLock in AcquireNextTrigger and in RetryExecuteInNonManagedTXLock.
This shouldn't be a problem because the Semaphore has the ability to return false in ObtainLock which suppress the release.

But as I can see in DBSemaphore always true is returned.

I think DBSemaphore.OptainLock should be corrected like this:

public async Task<bool> ObtainLock(
    Guid requestorId,
    ConnectionAndTransactionHolder? conn,
    string lockName,
    CancellationToken cancellationToken = default)
{
    var isDebugEnabled = Log.IsDebugEnabled();
    if (isDebugEnabled)
    {
        Log.DebugFormat("Lock '{0}' is desired by: {1}", lockName, requestorId);
    }

    var key = new ThreadLockKey(requestorId, lockName);
    if (!IsLockOwner(key))
    {
        await ExecuteSQL(requestorId, conn!, lockName, expandedSQL, expandedInsertSQL, cancellationToken)
            .ConfigureAwait(false);

        if (isDebugEnabled)
        {
            Log.DebugFormat("Lock '{0}' given to: {1}", lockName, requestorId);
        }

        return locks.TryAdd(key, null);
    }
    else
    {
        if (isDebugEnabled)
        {
            Log.DebugFormat("Lock '{0}' Is already owned by: {1}", lockName, requestorId);
        }
        return false;
    }
}

Full stacktrace

stack-trace of wrongful returner:    at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at Quartz.Impl.AdoJobStore.DBSemaphore.ReleaseLock(Guid requestorId, String lockName, CancellationToken cancellationToken)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<ReleaseLock>d__146.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine](TStateMachine& stateMachine)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.ReleaseLock(Guid requestorId, String lockName, Boolean doIt, CancellationToken cancellationToken)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<ExecuteInNonManagedTXLock>d__269`1.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<RetryExecuteInNonManagedTXLock>d__265`1.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<ExecuteInNonManagedTXLock>d__269`1.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<>c__DisplayClass269_0`1.<<ExecuteInNonManagedTXLock>b__0>d.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.<>c__DisplayClass233_0.<<AcquireNextTriggers>b__1>d.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Quartz.Impl.AdoJobStore.StdAdoDelegate.<SelectInstancesFiredTriggerRecords>d__122.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.Finish(Boolean bUserDelegateExecuted)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(Task task, Boolean taskWasPreviouslyQueued)
   at System.Threading.Tasks.TaskScheduler.TryRunInline(Task task, Boolean taskWasPreviouslyQueued)
   at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(Task task, Boolean needsProtection)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Threading.Tasks.UnwrapPromise`1.TrySetFromTask(Task task, Boolean lookForOce)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessInnerTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.Invoke(Task completingTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.Finish(Boolean bUserDelegateExecuted)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
@MartinDemberger MartinDemberger mentioned this issue Jul 9, 2021
Merged
@lahma
Copy link
Member

lahma commented Jul 9, 2021

Thank you for the report and also fixing the issue! I had this on my backlog but didn't find the time yet so this is much appreciated 👍🏻

@lahma lahma added this to the 3.3.3 milestone Jul 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
3.3.3
Development

Successfully merging a pull request may close this issue.

fixes #1236 double release lock MartinDemberger/quartznet
2 participants
@lahma @MartinDemberger