Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataFrame IndexOufRange exception on attemp to call Apply method #7122

Open
asmirnov82 opened this issue Apr 8, 2024 · 0 comments
Open

DataFrame IndexOufRange exception on attemp to call Apply method #7122

asmirnov82 opened this issue Apr 8, 2024 · 0 comments
Labels
untriaged New issue has not been triaged

Comments

@asmirnov82
Copy link
Contributor

Test to reproduce:

public void TestApplyMethod()
{
    PrimitiveDataFrameColumn<byte> column = new PrimitiveDataFrameColumn<byte>("Byte1", int.MaxValue / 2 - 1);
    PrimitiveDataFrameColumn<double> newColumn = column.Apply<double>(x => (double?)x);
}

Root cause is in Apply method:

public void Apply<TResult>(Func<T?, TResult?> func, PrimitiveColumnContainer<TResult> resultContainer)
    where TResult : unmanaged
{
    for (int b = 0; b < Buffers.Count; b++)
    {
        var sourceBuffer = Buffers[b];
        var sourceNullBitMap = NullBitMapBuffers[b].ReadOnlySpan;

        Span<TResult> mutableResultBuffer = resultContainer.Buffers.GetOrCreateMutable(b).Span;
        Span<byte> mutableResultNullBitMapBuffers = resultContainer.NullBitMapBuffers.GetOrCreateMutable(b).Span;

        for (int i = 0; i < sourceBuffer.Length; i++)
        {
            bool isValid = BitUtility.IsValid(sourceNullBitMap, i);
            TResult? value = func(isValid ? sourceBuffer[i] : null);
            mutableResultBuffer[i] = value.GetValueOrDefault();
            resultContainer.SetValidityBit(mutableResultNullBitMapBuffers, i, value != null);
        }
    }
}

mutableResultBuffer has TResult type of underlying data, so it's max length is 2Gb / sizeof(TResult)
sourceBuffer has T type of underlying data, so it's max length is 2 Gb/ sizeof(T)

when sizeof(TResult) > sizeof(T) and source buffer is large enough - it resulted in IndexOutOfRange exception

@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged label Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
untriaged New issue has not been triaged
Projects
None yet
Development

No branches or pull requests

1 participant