Slow insertion of LPM map into HASH of maps (~20ms)

[NHAS]

Howdy!

I've recently come across an issue where using Put to insert a new LPM into a hash of maps is very slow, unusually so as normally pushing other data into maps is quite quick (in the microsecond range).

I dont think the code is complicated and Im more than happy to hear Im just doing this wrong.

But for the following code why would this take so long?

struct bpf_map_def SEC("maps") policies_table = {
    .type = BPF_MAP_TYPE_HASH_OF_MAPS,
    .max_entries = MAX_MAP_ENTRIES,
    .key_size = MAX_USERID_LENGTH,
    .value_size = sizeof(__u32),
    .map_flags = 0,
};

func addInnerMapTo(key interface{}, spec *ebpf.MapSpec, table *ebpf.Map) (*ebpf.Map, error) {

	inner, err := ebpf.NewMap(spec)
	if err != nil {
		return nil, fmt.Errorf("%s creating new map: %s", table.String(), err)
	}

	err = table.Put(key, uint32(inner.FD()))
	if err != nil {
		return nil, fmt.Errorf("%s adding new map to table: %s", table.String(), err)
	}

	return inner, nil
}

[chenhengqi]

We've encounter similar situation in production, I think commit torvalds/linux@1ae80cf31938 is relevant.

[chenhengqi]

That depends on your use-case. The following is what we do now:

• Insert all inner maps at init stage, so no runtime penalty
• During bpf program run, we only insert/delete elements in inner map

[NHAS]

Yep that makes perfect sense to me. Im currently using the "create new map" as a way of clearing the map entirely, but Im sure I could just BatchDelete all keys or somehow clear the map which would work quite nicely.

Thanks!

[NHAS]

I might be missing something incredibly obvious here. But why is looking up in a hash of maps from userspace still equally as slow?

start := time.Now()
err = xdpObjects.PoliciesTable.Lookup(userid, &innerMapID)
log.Println("x hit fast path: ", x, "elapsed: ", time.Since(start))

Still takes 30ms per operation

[NHAS]

Some example timings:

2024/01/09 15:51:32 x hit fast path:  0 elapsed:  26.646803ms
2024/01/09 15:51:32 x hit fast path:  0 elapsed:  23.285096ms
2024/01/09 15:51:32 x hit fast path:  0 elapsed:  33.312167ms
2024/01/09 15:51:32 x hit fast path:  0 elapsed:  39.979199ms
2024/01/09 15:51:32 x hit fast path:  0 elapsed:  33.314726ms
2024/01/09 15:51:32 x hit fast path:  0 elapsed:  39.976751ms
2024/01/09 15:51:33 x hit fast path:  0 elapsed:  43.312766ms
2024/01/09 15:51:33 x hit fast path:  0 elapsed:  33.306626ms
2024/01/09 15:51:33 x hit fast path:  0 elapsed:  29.988249ms
2024/01/09 15:51:33 x hit fast path:  0 elapsed:  29.96062ms
2024/01/09 15:51:33 x hit fast path:  0 elapsed:  33.313157ms

[NHAS]

To answer my own question, https://github.com/torvalds/linux/blob/0dd3ee31125508cd67f7e7172247f05b7fd1753a/kernel/bpf/syscall.c#L267 exists on read for hash map of maps as well. Which is quite stink. But oh well.