How to delete binary keys created by benchmark tool in ETCD

[shaoqin2]

I ran a load test using the benchmark tool and ended up with lots of binary formatted keys in my ETCD cluster. I tried many ways of passing the right ASCII sequence to etcdctl del to no avail. Anyone know how to delete those keys?

I ran benchmark with

./benchmark  put  --val-size 10000  --clients=100 --conns=100 --total=1000000 --sequential-keys --key-space-size=100000

bin/etcdctl get "" --prefix --keys-only --hex | head -n 10 
\x00\x00\x00\x00\x00\x00\x00\x00
\x02\x00\x00\x00\x00\x00\x00\x00
\x04\x00\x00\x00\x00\x00\x00\x00

These are the keys created by benchmark that I'm trying to delete

I've tried the following

import os
key = "\x02\x00\x00\x00\x00\x00\x00\x00"
os.system('bin/etcdctl del ' + repr(key))

bin/etcdctl del "\x00\x00\x00\x00\x00\x00\x00\x00"

bin/etcdctl del "\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00"

bin/etcdctl del $(bin/etcdctl get "" --prefix --keys-only --write-out="json" | jq -r ".kvs[0] | .key")

A little confused at this point, similar to what was asked in this issue #9545

[jmhbnz]

Hey @shaoqin2 - Thanks for your question. I managed to delete the binary keys as follows.

1 - Start etcd instance

./bin/etcd

2 - Run benchmark tool to create keys

Note: I'm only adding 10 keys to keep this example simple.

benchmark  put  --val-size 10000  --clients=100 --conns=100 --total=10 --sequential-keys --key-space-size=100000

3 - Confirm keys present

 james  ~  Documents  etcd  bin   main                                                                                                                                        07:43:55 
 ➜ ./etcdctl get "" --prefix --keys-only --hex
\x00\x00\x00\x00\x00\x00\x00\x00

\x02\x00\x00\x00\x00\x00\x00\x00

\x04\x00\x00\x00\x00\x00\x00\x00

\x06\x00\x00\x00\x00\x00\x00\x00

\x08\x00\x00\x00\x00\x00\x00\x00

\x0a\x00\x00\x00\x00\x00\x00\x00

\x0c\x00\x00\x00\x00\x00\x00\x00

\x0e\x00\x00\x00\x00\x00\x00\x00

\x10\x00\x00\x00\x00\x00\x00\x00

\x12\x00\x00\x00\x00\x00\x00\x00

4 - Create minimal deletion program

Here is where it gets interesting, I could not find a way to delete these binary keys using etcdctl. Nothing I tried worked, so I instead wrote a minimal deletion golang program while experimenting.

cat << EOF > delete.go
package main

import (
	"context"
	"encoding/hex"
	"flag"
	"log"
	"strings"
	"time"
	"fmt"

	"go.etcd.io/etcd/client/v3"
)

func main() {
	// Command-line flags
	etcdEndpoints := flag.String("endpoints", "localhost:2379", "Comma-separated list of etcd server endpoints")
	hexPrefix := flag.String("prefix", "", "Hexadecimal prefix of keys to delete")
	flag.Parse()

	if *hexPrefix == "" {
		log.Fatal("Hexadecimal prefix is required")
	}

	// Convert hex prefix to bytes
	prefixBytes, err := hex.DecodeString(*hexPrefix)
	if err != nil {
		log.Fatalf("Error decoding hex prefix: %v", err)
	}

	// Create etcd client
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   strings.Split(*etcdEndpoints, ","),
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		log.Fatalf("Error creating etcd client: %v", err)
	}
	defer cli.Close()

	// Create context with timeout
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	// Retrieve keys with the given prefix
	resp, err := cli.Get(ctx, string(prefixBytes), clientv3.WithPrefix())
	if err != nil {
		log.Fatalf("Error retrieving keys with prefix: %v", err)
	}

	// Delete keys with the given prefix
	for _, kv := range resp.Kvs {
		key := kv.Key
		fmt.Printf("Deleting key: %v\n", key)
		_, err := cli.Delete(ctx, string(key))
		if err != nil {
			log.Printf("Error deleting key %v: %v", key, err)
		}
	}

	fmt.Println("Deletion completed.")
}
EOF

5 - Delete the keys

 james  ~  Documents  etcd  bin   main                                                                                                                                        07:47:20 
 ➜ for key in $(./etcdctl get "" --prefix --keys-only --hex | sed 's/\\x//g' | xargs); do go run delete.go -endpoints=localhost:2379 -prefix=$key; done
Deleting key: [0 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [2 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [4 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [6 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [8 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [10 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [12 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [14 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [16 0 0 0 0 0 0 0]
Deletion completed.
Deleting key: [18 0 0 0 0 0 0 0]
Deletion completed.

6 - Confirm keys deleted

 james  ~  Documents  etcd  bin   main                                                                                                                                        07:48:15 
 ➜ ./etcdctl get "" --prefix --keys-only --hex

Warning

Obviously this is just my experimental tinkering, please do your own testing and validation before using any similar approach in a real environment.

[shaoqin2]

Oh wow thank you so much for such a detailed response! This totally works for us!

Requiring a full on client side go program to delete those keys might be a feature gap in etcd benchmark tool / etcdctl though.

[jmhbnz]

Agreed it should certainly not be this hard to clean up after tools/benchmark.