Compress WAL #5592

csmarchbanks · 2019-05-23T03:28:36Z

This is a PR in order to run benchmarks for compressing the WAL (see prometheus-junkyard/tsdb#609).

@krasi-georgiev would you be willing to start a benchmark run when you get a chance?

Currently WALCompression is defaulted to true for the benchmarks, I will switch it to false before marking this ready to merge.

krasi-georgiev · 2019-05-23T08:03:51Z

/benchmark

prombot · 2019-05-23T08:03:54Z

@krasi-georgiev: Welcome to Prometheus Benchmarking Tool.

The two prometheus versions that will be compared are pr-5592 and master

The logs can be viewed at the links provided in the GitHub check blocks at the end of this conversation

After successfull deployment, the benchmarking metrics can be viewed at :

prometheus-meta - label {namespace="prombench-5592"}
grafana - template-variable "pr-number" : 5592

The Prometheus servers being benchmarked can be viewed at :

PR - prombench.prometheus.io/5592/prometheus-pr
master - prombench.prometheus.io/5592/prometheus-release

To stop the benchmark process comment /benchmark cancel .

In response to this:

/benchmark

If you have questions or suggestions related to my behavior, please file an issue against the prometheus/prombench repository.

krasi-georgiev · 2019-05-23T08:23:02Z

Will run for a day and will also check the WAL size savings.

csmarchbanks · 2019-05-23T14:24:57Z

Looks like compression ratios in the 1.8 - 2.8 range (prom query)

Looks good to me so far, WAL fsync latencies are also down significantly. Is it possible to restart each of the Prometheus instances? It would be interesting to see how reading the WAL at startup differs.

krasi-georgiev · 2019-05-24T00:21:00Z

restarted.

not sure if there is a metric for the time it took to read the logs so let me know and I can probably figure out the timings from the logs.

csmarchbanks · 2019-05-24T02:30:31Z

Thanks! I can get an estimate based on when samples start getting ingested again:
master: 00:10:10 -> 00:24:55 = 14 minutes, 45 seconds
pr: 00:18:10 -> 00:33:00 = 14 minutes, 50 seconds

Almost no difference, so seems like decoding speed is not something to be worried about either.

krasi-georgiev · 2019-05-24T09:01:48Z

amazing difference in the WAL size

no compression

7.9G    /mnt/disks/ssd0/wal/checkpoint.001516
63G     /mnt/disks/ssd0/wal/

compressed

828M    /mnt/disks/ssd0/wal/checkpoint.000798
30G     /mnt/disks/ssd0/wal/

krasi-georgiev · 2019-05-24T09:02:35Z

do we need more data or can I shut it down?

csmarchbanks · 2019-05-24T12:19:15Z

That’s all the data I want, feel free to shut it down.

krasi-georgiev · 2019-05-24T21:13:25Z

/benchmark cancel

krasi-georgiev · 2019-05-29T23:42:57Z

/benchmark

prombot · 2019-05-29T23:42:59Z

@krasi-georgiev: Welcome to Prometheus Benchmarking Tool.

The two prometheus versions that will be compared are pr-5592 and master

The logs can be viewed at the links provided in the GitHub check blocks at the end of this conversation

After successfull deployment, the benchmarking metrics can be viewed at :

prometheus-meta - label {namespace="prombench-5592"}
grafana - template-variable "pr-number" : 5592

The Prometheus servers being benchmarked can be viewed at :

PR - prombench.prometheus.io/5592/prometheus-pr
master - prombench.prometheus.io/5592/prometheus-release

To stop the benchmark process comment /benchmark cancel .

In response to this:

/benchmark

If you have questions or suggestions related to my behavior, please file an issue against the prometheus/prombench repository.

krasi-georgiev · 2019-05-30T08:41:41Z

/benchmark cancel

bwplotka · 2019-07-03T13:04:21Z

@csmarchbanks TSDB https://github.com/prometheus/tsdb/releases/tag/v0.9.1 is released. (:

csmarchbanks · 2019-07-03T13:32:08Z

@brancz @bwplotka @krasi-georgiev This is ready for a review now.

Did a quick smoke test of remote write in both compressed and non-compressed modes and everything is looking good.

csmarchbanks · 2019-07-03T13:34:34Z

I broke it into two commits for ease of review - one for a minimal tsdb update, one for adding the WAL compression flag.

brancz · 2019-07-03T13:36:24Z

storage/remote/wal_watcher.go

@@ -78,6 +78,7 @@ var (
 		},
 		[]string{queue},
 	)
+	liveReaderMetrics = wal.NewLiveReaderMetrics(prometheus.DefaultRegisterer)


I'd love to see us refactor this eventually to not use global :) not a blocker here though

Would also love to see that, I think the struct has to be exported to enable that and then we would have to pass it down from somewhere pretty high. Seemed like a more invasive change than making it global for now.

Definitely, just evangelizing this, this pattern is pretty spread around the codebase right now, so definitely something to solve here :)

I can do this as part of prometheus-junkyard/tsdb#606

csmarchbanks · 2019-07-03T13:38:10Z

storage/remote/wal_watcher_test.go

@@ -505,5 +505,5 @@ func TestCheckpointSeriesReset(t *testing.T) {
 	// If you modify the checkpoint and truncate segment #'s run the test to see how
 	// many series records you end up with and change the last Equals check accordingly
 	// or modify the Equals to Assert(len(wt.seriesLabels) < seriesCount*10)
-	testutil.Equals(t, 14, wt.checkNumLabels())
+	testutil.Equals(t, 13, wt.checkNumLabels())


This change is due to the WAL compression causing a different number of segments.

krasi-georgiev

TSDB changes LGTM

krasi-georgiev · 2019-07-03T13:40:19Z

storage/remote/wal_watcher_test.go

@@ -171,7 +171,7 @@ func TestReadToEndNoCheckpoint(t *testing.T) {
 	err = os.Mkdir(wdir, 0777)
 	testutil.Ok(t, err)

-	w, err := wal.NewSize(nil, nil, wdir, 128*pageSize)
+	w, err := wal.NewSize(nil, nil, wdir, 128*pageSize, true)


is this intentional for making the wal compressed?

I wanted the tests to run using the compressed WAL since that is the future. Uncompressed is the default right now though so I could change them to false for now?

I would probably make sense to test both cases here. I do agree compressed should some day become the default, but for now uncompressed is the default and the default should be tested.

I don't mind either way.

(or for this PR I'd be ok with just testing the default and adding the compressed case as a follow up)

Adding the compressed case is pretty easy will be done in a couple minutes. If it looks too complicated I can quickly revert it.

Added t.Run statements for both compressed and uncompressed.

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

bwplotka

OK from my side.

In terms of test. Isn't wotth to test... both? uncompressed and compressed? (as @brancz mentioned)

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

csmarchbanks · 2019-07-03T14:12:13Z

In terms of test. Isn't wotth to test... both?

Yep, I agree. I updated all the tests to test both cases similar to the tsdb tests.

csmarchbanks force-pushed the compress-wal branch from 0e17c43 to 0945873 Compare May 23, 2019 04:22

prombot added the benchmark label May 23, 2019

prombot removed the benchmark label May 24, 2019

krasi-georgiev mentioned this pull request May 28, 2019

Prometheus restart very slowly when the size of data folder is huge prometheus-junkyard/tsdb#611

Closed

prombot added the benchmark label May 29, 2019

prombot removed the benchmark label May 30, 2019

siggy mentioned this pull request Jun 21, 2019

Evaluate Prometheus resource usage linkerd/linkerd2#2980

Open

This was referenced Jul 2, 2019

Cannot run more than one wal.LiveReader with metrics prometheus-junkyard/tsdb#646

Closed

Updated TSDB dependency to 0.9.0. #5733

Closed

csmarchbanks force-pushed the compress-wal branch from 0945873 to 9a3fab4 Compare July 3, 2019 13:26

csmarchbanks changed the title ~~WIP: Compress WAL~~ Compress WAL Jul 3, 2019

csmarchbanks marked this pull request as ready for review July 3, 2019 13:27

brancz approved these changes Jul 3, 2019

View reviewed changes

csmarchbanks commented Jul 3, 2019

View reviewed changes

krasi-georgiev reviewed Jul 3, 2019

View reviewed changes

Update to tsdb 0.9.1

475ca2e

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

bwplotka approved these changes Jul 3, 2019

View reviewed changes

Provide flag to compress the tsdb WAL

06f1ba7

Signed-off-by: Chris Marchbanks <csmarchbanks@gmail.com>

csmarchbanks force-pushed the compress-wal branch from 9a3fab4 to 06f1ba7 Compare July 3, 2019 14:03

brancz mentioned this pull request Jul 3, 2019

Cut v2.11.0-rc.0 #5729

Merged

brancz merged commit 0727318 into prometheus:master Jul 3, 2019

csmarchbanks deleted the compress-wal branch July 3, 2019 15:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compress WAL #5592

Compress WAL #5592

csmarchbanks commented May 23, 2019

krasi-georgiev commented May 23, 2019

prombot commented May 23, 2019

krasi-georgiev commented May 23, 2019

csmarchbanks commented May 23, 2019

krasi-georgiev commented May 24, 2019

csmarchbanks commented May 24, 2019

krasi-georgiev commented May 24, 2019

krasi-georgiev commented May 24, 2019

csmarchbanks commented May 24, 2019

krasi-georgiev commented May 24, 2019

krasi-georgiev commented May 29, 2019

prombot commented May 29, 2019

krasi-georgiev commented May 30, 2019

bwplotka commented Jul 3, 2019

csmarchbanks commented Jul 3, 2019

csmarchbanks commented Jul 3, 2019

brancz Jul 3, 2019

csmarchbanks Jul 3, 2019

brancz Jul 3, 2019

cstyan Jul 3, 2019

csmarchbanks Jul 3, 2019

krasi-georgiev left a comment

krasi-georgiev Jul 3, 2019

csmarchbanks Jul 3, 2019

brancz Jul 3, 2019

krasi-georgiev Jul 3, 2019

brancz Jul 3, 2019

csmarchbanks Jul 3, 2019

csmarchbanks Jul 3, 2019

bwplotka left a comment

csmarchbanks commented Jul 3, 2019

Compress WAL #5592

Compress WAL #5592

Conversation

csmarchbanks commented May 23, 2019

krasi-georgiev commented May 23, 2019

prombot commented May 23, 2019

krasi-georgiev commented May 23, 2019

csmarchbanks commented May 23, 2019

krasi-georgiev commented May 24, 2019

csmarchbanks commented May 24, 2019

krasi-georgiev commented May 24, 2019

krasi-georgiev commented May 24, 2019

csmarchbanks commented May 24, 2019

krasi-georgiev commented May 24, 2019

krasi-georgiev commented May 29, 2019

prombot commented May 29, 2019

krasi-georgiev commented May 30, 2019

bwplotka commented Jul 3, 2019

csmarchbanks commented Jul 3, 2019

csmarchbanks commented Jul 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krasi-georgiev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

csmarchbanks commented Jul 3, 2019