Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add encrypted diskqueue #31808

Closed
wants to merge 7 commits into from
Closed

Add encrypted diskqueue #31808

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4763029
Add encrypted diskqueue
leehinman May 11, 2022
f2e2393
Add docs for on-disk structures
leehinman Jun 3, 2022
b2b50ba
make encryption/compression it's own schema version
leehinman Jun 6, 2022
979ad39
writer_loop use bytes.Buffer to reduce Write syscalls
leehinman Jun 6, 2022
3a69909
add additional checks to encryption test
leehinman Jun 7, 2022
5b6dc95
update docs for v3
leehinman Jun 7, 2022
49df4cf
switch to stream compression with flush
leehinman Jun 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
76 changes: 38 additions & 38 deletions NOTICE.txt
View file
Open in desktop
Expand Up @@ -13745,6 +13745,44 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.


--------------------------------------------------------------------------------
Dependency : github.com/pierrec/lz4
Version: v2.6.0+incompatible
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/pierrec/lz4@v2.6.0+incompatible/LICENSE:

Copyright (c) 2015, Pierre Curto
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

* Neither the name of xxHash nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.



--------------------------------------------------------------------------------
Dependency : github.com/pierrre/gotestcover
Version: v0.0.0-20160517101806-924dca7d15f0
Expand Down Expand Up @@ -34505,44 +34543,6 @@ Contents of probable licence file $GOMODCACHE/github.com/oxtoacart/bpool@v0.0.0-
limitations under the License.


--------------------------------------------------------------------------------
Dependency : github.com/pierrec/lz4
Version: v2.6.0+incompatible
Licence type (autodetected): BSD-3-Clause
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/pierrec/lz4@v2.6.0+incompatible/LICENSE:

Copyright (c) 2015, Pierre Curto
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

* Neither the name of xxHash nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.



--------------------------------------------------------------------------------
Dependency : github.com/power-devops/perfstat
Version: v0.0.0-20210106213030-5aafc221ea8c
Expand Down
4 changes: 3 additions & 1 deletion go.mod
View file
Open in desktop
Expand Up @@ -163,6 +163,7 @@ require (
github.com/elastic/elastic-agent-autodiscover v0.1.1
github.com/elastic/elastic-agent-libs v0.2.5
github.com/elastic/elastic-agent-system-metrics v0.3.1
github.com/pierrec/lz4 v2.6.0+incompatible
github.com/shirou/gopsutil/v3 v3.21.12
go.elastic.co/apm/module/apmelasticsearch/v2 v2.0.0
go.elastic.co/apm/module/apmhttp/v2 v2.0.0
Expand Down Expand Up @@ -262,7 +263,7 @@ require (
github.com/morikuni/aec v1.0.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/pierrec/lz4 v2.6.0+incompatible // indirect
github.com/pierrec/lz4/v4 v4.1.15-0.20220523073431-885e900f46f2 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/prometheus/client_golang v1.11.0 // indirect
github.com/samuel/go-parser v0.0.0-20130731160455-ca8abbf65d0e // indirect
Expand Down Expand Up @@ -316,6 +317,7 @@ replace (
github.com/google/gopacket => github.com/elastic/gopacket v1.1.20-0.20211202005954-d412fca7f83a
github.com/insomniacslk/dhcp => github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3 // indirect
github.com/tonistiigi/fifo => github.com/containerd/fifo v0.0.0-20190816180239-bda0ff6ed73c
github.com/pierrec/lz4/v4 v4.1.15-0.20220523073431-885e900f46f2 => /Users/leehinman/src/pierrec_lz4
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rename is necessary because the Flush function needs a small patch, it currently prevents further writing to the stream. The patch allows you to write to the stream after a flush. So before merging it would be good to see if upstream will take the fix or if we need to maintain a fork.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The upstream project looks active enough, hopefully they'll just take the PR. I'm curious what the change here looks like as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pierrec/lz4#188

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already merged! Nice, that was fast.

)

// Exclude this version because the version has an invalid checksum.
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Expand Up @@ -1401,6 +1401,8 @@ github.com/pierrec/lz4 v1.0.2-0.20190131084431-473cd7ce01a1/go.mod h1:3/3N9NVKO0
github.com/pierrec/lz4 v2.0.5+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
github.com/pierrec/lz4 v2.6.0+incompatible h1:Ix9yFKn1nSPBLFl/yZknTp8TU5G4Ps0JDmguYK6iH1A=
github.com/pierrec/lz4 v2.6.0+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
github.com/pierrec/lz4/v4 v4.1.15-0.20220523073431-885e900f46f2 h1:rfcsVKBrjqnG3+q599T+JFw+wG94uL/ehfyQtUdav6U=
github.com/pierrec/lz4/v4 v4.1.15-0.20220523073431-885e900f46f2/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrre/gotestcover v0.0.0-20160517101806-924dca7d15f0 h1:i5VIxp6QB8oWZ8IkK8zrDgeT6ORGIUeiN+61iETwJbI=
github.com/pierrre/gotestcover v0.0.0-20160517101806-924dca7d15f0/go.mod h1:4xpMLz7RBWyB+ElzHu8Llua96TRCB3YwX+l5EP1wmHk=
github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4/go.mod h1:4OwLy04Bl9Ef3GJJCoec+30X3LQs/0/m4HFRt/2LUSA=
Expand Down
26 changes: 11 additions & 15 deletions libbeat/publisher/queue/diskqueue/acks_test.go
View file
Open in desktop
Expand Up @@ -66,7 +66,7 @@ func TestAddFrames(t *testing.T) {
rf(0, 0, true, 100),
},
frameID(1),
&queuePosition{0, segmentHeaderSize + 100, 1},
&queuePosition{0, segmentHeaderSizeV1 + 100, 1},
nil,
},
{
Expand All @@ -85,7 +85,7 @@ func TestAddFrames(t *testing.T) {
rf(0, 1, false, 75),
},
frameID(2),
&queuePosition{0, segmentHeaderSize + 175, 2},
&queuePosition{0, segmentHeaderSizeV1 + 175, 2},
nil,
},
{
Expand All @@ -94,7 +94,7 @@ func TestAddFrames(t *testing.T) {
rf(0, 2, false, 100),
},
frameID(5),
&queuePosition{1, segmentHeaderSize + 150, 2},
&queuePosition{1, segmentHeaderSizeV1 + 150, 2},
// This time we crossed a boundary so we should get an ACK for segment
// 0 on the notification channel.
segmentIDRef(0),
Expand Down Expand Up @@ -122,7 +122,7 @@ func TestAddFrames(t *testing.T) {
rf(0, 0, true, 100),
},
frameID(2),
&queuePosition{0, segmentHeaderSize + 150, 2},
&queuePosition{0, segmentHeaderSizeV1 + 150, 2},
nil,
},
{
Expand All @@ -131,7 +131,7 @@ func TestAddFrames(t *testing.T) {
rf(0, 2, false, 75),
},
frameID(3),
&queuePosition{0, segmentHeaderSize + 225, 3},
&queuePosition{0, segmentHeaderSizeV1 + 225, 3},
nil,
},
{
Expand All @@ -140,7 +140,7 @@ func TestAddFrames(t *testing.T) {
rf(1, 3, false, 100),
},
frameID(7),
&queuePosition{2, segmentHeaderSize + 100, 1},
&queuePosition{2, segmentHeaderSizeV1 + 100, 1},
segmentIDRef(1),
},
{
Expand All @@ -149,7 +149,7 @@ func TestAddFrames(t *testing.T) {
rf(2, 7, false, 100),
},
frameID(9),
&queuePosition{2, segmentHeaderSize + 300, 3},
&queuePosition{2, segmentHeaderSizeV1 + 300, 3},
nil,
},
},
Expand All @@ -165,7 +165,7 @@ func TestAddFrames(t *testing.T) {
rf(0, 0, true, 100),
},
frameID(4),
&queuePosition{3, segmentHeaderSize + 100, 1},
&queuePosition{3, segmentHeaderSizeV1 + 100, 1},
// We advanced from segment 0 to segment 3, so we expect
// segmentID 2 on the ACK channel.
segmentIDRef(2),
Expand All @@ -182,7 +182,7 @@ func TestAddFrames(t *testing.T) {
rf(10, 35, true, 100),
},
frameID(36),
&queuePosition{10, segmentHeaderSize + 100, 1},
&queuePosition{10, segmentHeaderSizeV1 + 100, 1},
// We advanced to segment 10, so we expect segmentID 9 on
// the ACK channel.
segmentIDRef(9),
Expand Down Expand Up @@ -218,7 +218,7 @@ func TestAddFrames(t *testing.T) {
[]*readFrame{
{
segment: &queueSegment{
schemaVersion: uint32Ref(0),
schemaVersion: 0,
},
bytesOnDisk: 100,
},
Expand Down Expand Up @@ -299,15 +299,11 @@ func (dqa *diskQueueACKs) assertACKedSegment(
}
}

func uint32Ref(v uint32) *uint32 {
return &v
}

// rf assembles a readFrame with the given parameters and a spoofed
// queue segment, whose firstFrameID field is set to match the given frame
// if "first" is true.
func rf(seg segmentID, frame frameID, first bool, size uint64) *readFrame {
s := &queueSegment{id: seg}
s := &queueSegment{id: seg, schemaVersion: 1}
if first {
s.firstFrameID = frame
}
Expand Down