Skip to content
This repository has been archived by the owner on Aug 13, 2019. It is now read-only.

Attempt to reduce memory footprint of compaction #627

Closed
wants to merge 49 commits into from
Closed

Attempt to reduce memory footprint of compaction #627

wants to merge 49 commits into from

Commits on Jun 11, 2019

  1. Returning series ref in ChunkSeriesSet.At() 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 11, 2019
    Configuration menu
    Copy the full SHA
    9811a14 View commit details
    Browse the repository at this point in the history

  2. newCompactionMerger takes []ChunkSeriesSet. Calculating (old series i… 

    …d -> new series id) map.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 11, 2019
    Configuration menu
    Copy the full SHA
    12a10a0 View commit details
    Browse the repository at this point in the history

  3. User (old series id -> new series id) map to write new postings 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 11, 2019
    Configuration menu
    Copy the full SHA
    b7839c4 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2019

  1. Get rid of postingMap 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 12, 2019
    Configuration menu
    Copy the full SHA
    5d3bf62 View commit details
    Browse the repository at this point in the history

  2. NumSeries() for BlockReader. Pre-allocate memory for seriesMap maps. 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 12, 2019
    Configuration menu
    Copy the full SHA
    76b67fd View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2019

  1. Set a higher initial value for postingBuf 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    64e4487 View commit details
    Browse the repository at this point in the history

  2. Efficiently iterate over all sorted label-values 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    3a4a6e6 View commit details
    Browse the repository at this point in the history

  3. Hint number of postings writes to the index Writer to pre-allocate me… 

    …mory.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    0a0dbfe View commit details
    Browse the repository at this point in the history

  4. Pre-allocate slice for label values in populateBlock 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 14, 2019
    Configuration menu
    Copy the full SHA
    c1c39ed View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2019

  1. Re-use bigEndianPostings in populateBlock 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    3593059 View commit details
    Browse the repository at this point in the history

  2. Reuse string buffer when swapping stringTuples 

    This reduces the allocs of WriteLabelIndex significantly.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    e78ec69 View commit details
    Browse the repository at this point in the history

  3. Reuse byte slice for writing chunk meta hash 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    f344fa2 View commit details
    Browse the repository at this point in the history

  4. Fix race in writeHash 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    9fab207 View commit details
    Browse the repository at this point in the history

  5. Fix lint errors 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    0ff98c0 View commit details
    Browse the repository at this point in the history

  6. Reuse slice in ReadOffsetTable. Reduces allocs for OpenBlock. 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    407fe55 View commit details
    Browse the repository at this point in the history

  7. Reuse ListPostings in populateBlock. 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    39b97e5 View commit details
    Browse the repository at this point in the history

  8. Check for error before wrapping in blockIndexReader.Series 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    4ff0ed8 View commit details
    Browse the repository at this point in the history

  9. Check for error before wrapping in blockIndexReader.Postings 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    387705d View commit details
    Browse the repository at this point in the history

  10. Use already allocated byte buffer in writeHash 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    cff36d4 View commit details
    Browse the repository at this point in the history

  11. WriteChunks takes a byte buffer to reuse 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 18, 2019
    Configuration menu
    Copy the full SHA
    f9dfe07 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2019

  1. Breakdown generic writeOffsetTable into writeLabelIndexesOffsetTable … 

    …and writePostingsOffsetTable

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 19, 2019
    Configuration menu
    Copy the full SHA
    e812fba View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2019

  1. Fix review comments 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 20, 2019
    Configuration menu
    Copy the full SHA
    b421a4b View commit details
    Browse the repository at this point in the history

  2. Reset() for Postings interface 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 20, 2019
    Configuration menu
    Copy the full SHA
    53ccf99 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/master' into compact-opt 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 20, 2019
    Configuration menu
    Copy the full SHA
    eb79899 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2019

  1. Fix review comments 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 22, 2019
    Configuration menu
    Copy the full SHA
    c7af8aa View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2019

  1. Remove Reset(...) from Postings interface 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    b2b2647 View commit details
    Browse the repository at this point in the history

  2. More code comments and fix review comments 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jun 28, 2019
    Configuration menu
    Copy the full SHA
    b007998 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2019

  1. Revert changes to writeOffsetTable 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 2, 2019
    Configuration menu
    Copy the full SHA
    0068b3c View commit details
    Browse the repository at this point in the history

  2. Revert checking error before wrapping 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 2, 2019
    Configuration menu
    Copy the full SHA
    606fcf1 View commit details
    Browse the repository at this point in the history

  3. Revert re-use of 'keys' in ReadOffsetTable 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 2, 2019
    Configuration menu
    Copy the full SHA
    0898907 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2019

  1. Revert changes made to stringTuples 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 3, 2019
    Configuration menu
    Copy the full SHA
    caa715e View commit details
    Browse the repository at this point in the history

  2. Revert passing byte buffer to WriteChunks and writeHash 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 3, 2019
    Configuration menu
    Copy the full SHA
    1d54273 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2019

  1. Merge remote-tracking branch 'upstream/master' into compact-opt 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 5, 2019
    Configuration menu
    Copy the full SHA
    dbec7eb View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2019

  1. Merge remote-tracking branch 'upstream/master' into compact-opt

    @codesome
    codesome committed Jul 10, 2019
    Configuration menu
    Copy the full SHA
    85bdf35 View commit details
    Browse the repository at this point in the history

  2. hashEntry -> postingsHashEntry after merging upstream 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 10, 2019
    Configuration menu
    Copy the full SHA
    89d459e View commit details
    Browse the repository at this point in the history

  3. writePostings perf improvements 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 10, 2019
    Configuration menu
    Copy the full SHA
    f8ddc03 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2019

  1. Fix review comments 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 12, 2019
    Configuration menu
    Copy the full SHA
    1582ebc View commit details
    Browse the repository at this point in the history

  2. Add RemappedPostings 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 12, 2019
    Configuration menu
    Copy the full SHA
    5993889 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'upstream/master' into compact-opt

    @codesome
    codesome committed Jul 12, 2019
    Configuration menu
    Copy the full SHA
    4293e37 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2019

  1. Fix review comments 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 15, 2019
    Configuration menu
    Copy the full SHA
    e0693dc View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2019

  1. Reset ref in compactionMerger when there are 0 chunks 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 16, 2019
    Configuration menu
    Copy the full SHA
    048790b View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2019

  1. compactionMerger doesn't return series with 0 chunks 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    605f5b7 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into compact-opt 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    40ad33c View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2019

  1. Fix out-of-order series bug 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 24, 2019
    Configuration menu
    Copy the full SHA
    b72273f View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into compact-opt

    @codesome
    codesome committed Jul 24, 2019
    Configuration menu
    Copy the full SHA
    ffe544e View commit details
    Browse the repository at this point in the history

  3. Bug fix in compactionMerger.Next() 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 24, 2019
    Configuration menu
    Copy the full SHA
    570d5e1 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2019

  1. Don't put deleted series in seriesMap 

    This fixes series out-of-order bug when compacting from on-disk block,
because postings used to contain invalid series references as even
the deleted series were present in seriesMap with bogus series references.

Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 25, 2019
    Configuration menu
    Copy the full SHA
    4ed0cac View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2019

  1. Better allocation strategy for remapped postings 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Jul 28, 2019
    Configuration menu
    Copy the full SHA
    c252f96 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2019

  1. Faster way to merge remapped postings 

    Signed-off-by: Ganesh Vernekar <cs15btech11018@iith.ac.in>
    @codesome
    codesome committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    a75fa32 View commit details
    Browse the repository at this point in the history