Replies: 10 comments 19 replies
-
|
The previous examples assume a flat combination of all parameters, although there is likely a need to have a layered view of parameters as in multi-layer nesting. Example sketch: This is different than saying "all combiantions of (a,b,c) and (1,2,3)", as this simple example says nothing about the structure of traversal, ordering, expressions in the native workload, etc. |
Beta Was this translation helpful? Give feedback.
-
|
Example workload template for parameter expansion: With this set of template parameters, it would be possible for a user to generate 12 distinct scenarios, each with two commands to run an activity, both of which contain the specific combination of the 12 possible from the supplied values for dataset and k. This is just to demonstrate the basic scenario of applying combinations of parameters over a single layer image of rendered templates. The equivalent long-form rendering would look like this: |
Beta Was this translation helpful? Give feedback.
-
|
I think that it is important to be able to present the proper layers of the tests. The |
Beta Was this translation helpful? Give feedback.
-
|
If I follow your meaning, it would be "inner loop" because it needs to happen as an integral part of each ensemble of activities around a specific set of parameters. |
Beta Was this translation helpful? Give feedback.
-
|
I can see where this can be implemented. |
Beta Was this translation helpful? Give feedback.
-
|
Let's look at the following:
Would it make sense to use ITERATION instead of TEMPLATE? |
Beta Was this translation helpful? Give feedback.
-
|
I think this is the best plan. With this structure we shift the scenarios that we which to repeat under a ~macro key. With this approach we are working within the By doing this work while loading the yaml from source we are operating at the correct layer. We introduce two new concepts:
|
Beta Was this translation helpful? Give feedback.
-
|
This is an alternative that would remove the need to scope variable substitution: If We rewrite each TEMPLATE(dataset) and TEMPLATE(k) to use the form with a default value which varies with the iteration. |
Beta Was this translation helpful? Give feedback.
-
|
I'm -1 on overloading comments. This has been done in other systems and invariably leads to surprises, however rare. Often the least likely or expected surprises are the ones that hurt the most. I think a better trade-off here would be to avoid directly leveraging the template variable system, but allow it to be composed as a distinct and separate layer if/when needed. We can use a standard type of offset symbol for variables just like we do for the other constructs. At least in this way, we can avoid mixing the designs too much and constraining what we do in this layer with previous design decisions. It may eventually wholly replace the template variable mechanism. An example: This is a combined parameter example, overloading the typical foreach syntax to deal with tuples in a parameter space. There is no reason we couldn't provide this out of the box. We could allow an ordering option which automatically visits the outside edges of the parameter space first, thus providing a strategic ordering which establishes testing bounds first. This is useful for two main reasons. First, you nearly always want to map the boundaries and understand those results first in order to determine whether sub-dividing the parameter space makes sense. A simple way to do this is to apply a global order over all parameter tuples which establishes a precedence between tuples which have an interior parameter value and those which don't. In the example above, interior parameter values are taken simply as those which are not at the beginning or end of a list of a parameter's value, like "other_k". The TEMPLATE reference here is just showing that this would work if applied as it is now at the top level of YAML loading, before proper parsing occurs, but it is not essential to the ability to specify parameter coordinates. Some caveats and possibilities to consider:
|
Beta Was this translation helpful? Give feedback.
-
|
If we go down the macro template path you don't like then it would make sense to use Jinja2 which would work by converting a For the other path we will need to fully understand the object model to make sure that processing proceeds correctly. Looking forward to a discussion. |
Beta Was this translation helpful? Give feedback.
-
|
I think jinja2 is worth looking at for examples and ideas. Yet, I want to be very careful about adding "bolt-on" runtimes or external stack requirements into the mix where it isn't essential. There is a Java library for jinja(2?), though. For jinja2-specific needs, users can already do that without us needing to provide it as part of NB. There is a trade-off between including external commodity languages and having something more direct and built-in. I would generally opt for something which users aren't required to know about unless they needed it, but which could be added at such time. (by adding, I mean the syntax only, nothing else needed) This suggests going from what users would normally be doing directly in the YAML form to adding a few directive lines to virtualize parameters; no changes in indentation needed, etc. Case in point: We've used the approach of adding an extant [expression] language to NB to support advanced templates. We have Jsonnet support. You can use a
Based on this, I want to add some possible qualifications to the requirements:
|
Beta Was this translation helpful? Give feedback.
-
|
I think we've settled on this format for the yaml:
|
Beta Was this translation helpful? Give feedback.
-
|
Working on implementation in #1657 |
Beta Was this translation helpful? Give feedback.
-
|
How about this, which reads similarly to the original form? |
Beta Was this translation helpful? Give feedback.
-
|
I'm rethinking the need to have a ~ in front of the scenario name. Instead we can detect foreach from steps that have either ~k or (k). |
Beta Was this translation helpful? Give feedback.
-
|
If we have a scenario defined like: The |
Beta Was this translation helpful? Give feedback.
-
|
On the original issue:
I presumed we would be rendering the logical result into the yaml which we use then to match the use-specified scenario against. I wouldn't expec them, for example, to This does raise the question of specifying which scenarios they would want to run out of the full set. They would need to be able to specify specific scenarios, potentially in two ways: 1) by specifying the fully rendered scenario name which would be present after expansion, or 2) by referencing the base in some way and providing appropriate parameters. We also need to ensure that the scenario name is unique, so having it be templated over unique parameter tuples is something we probably can't avoid. It does seem, however, that we've missed one usage pattern which I mentioned above. Suppose a user does want to specify one of the scenarios in terms of the template and a set of parameters. That means we'd need a nominal form for the template itself so it can be invoked programattically. In other words, as a user I want to invoke "scenario_(k)_(dataset)" with k=50 and dataset=foo |
Beta Was this translation helpful? Give feedback.
-
|
We also need to support defaults for the parameters in the yaml or scenario definition, not requiring the user to provide them for the basic case. The easiest way to do this is probably to provide them if possible in the template or structure somewhere and let other values replace them if provided by the user. |
Beta Was this translation helpful? Give feedback.
-
|
The changes to use post processed scenario names from the workload.yaml turns this whole approach on its head.
Then with workload.xml containing This will create a single scenario with the name: "scenario_50_foo". |
Beta Was this translation helpful? Give feedback.
-
|
Each scenario step does the following in allowing key=values set on the step to supercede those set from the command line. // consume each of the parameters from the steps to produce a composited command
// order is primarily based on the step template, then on user-provided parameters
for (CmdArg cmdarg : parsedStep.values()) {
// allow user provided parameter values to override those in the template,
// if the assignment operator used in the template allows for it
if (usersCopy.containsKey(cmdarg.getName())) {
cmdarg = cmdarg.override(usersCopy.remove(cmdarg.getName()));
}
buildingCmd.put(cmdarg.getName(), cmdarg.toString());
}Is this the proper order of precedence? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure I see the issue. We're walking the args for the parsed step from the named scenario. In other words, the precedence looks to be what the user provides on the command line first, unless it is not allowed by the value locking syntax. |
Beta Was this translation helpful? Give feedback.
-
|
In looking at how we are actually using dataset in existing workloads it turns out that the dataset choice should automatically set other key value pairs. For example: textembedding-gecko:
dimensions: 768
trainsize: 100000
testsize: 10000
sfunction: cosine
glove-25-angular:
dimensions: 25
trainsize: 1183514
testsize: 10000
indexname: glove25
sfunction: cosineI would like to extend NB5 as follows:
Take a look at min_version: "5.17.3"
description: |
A workload which reads ann-benchmarks vector data from HDF5 file format.
scenarios:
pinecone_vectors:
rampup: run tags==block:vector_writes cycles=TEMPLATE(trainsize) threads=TEMPLATE(rampup_threads,100) driver=pinecone apiKey=TEMPLATE(apikey) projectName=f88a480 environment=eu-west4-gcp
search_and_index: run tags='block:vector_reads' threads=TEMPLATE(search_threads,100) driver=pinecone apiKey=TEMPLATE(apikey) projectName=f88a480 environment=eu-west4-gcp stride=100 striderate=13.5
bindings:
rw_key: ToString()
train_vector: HdfFileToFloatList("testdata/TEMPLATE(dataset).hdf5", "/train");
test_vector: HdfFileToFloatList("testdata/TEMPLATE(dataset).hdf5", "/test");
validation_set: HdfFileToIntArray("testdata/TEMPLATE(dataset).hdf5", "/neighbors");
random_vector: HashedFloatVectors(TEMPLATE(dimensions));
blocks:
vector_writes:
ops:
rampup-op:
upsert: TEMPLATE(index_name)
namespace: TEMPLATE(namespace)
upsert_vectors:
- id: "{rw_key}"
values: "{train_vector}"
vector_invalidate_writes:
ops:
rampup-op:
upsert: TEMPLATE(index_name)
namespace: TEMPLATE(namespace)
upsert_vectors:
- id: "{rw_key}"
values: "{random_vector}"
vector_reads:
ops:
read-op:
query: TEMPLATE(index_name)
namespace: TEMPLATE(namespace)
vector: "{test_vector}"
top_k: TEMPLATE(select_limit,100)
include_values: true
include_metadata: true
verifier-init: |
relevancy=scriptingmetrics.newRelevancyMeasures(_parsed_op,"group","relevancy");
for (int k in List.of(100)) {
relevancy.addFunction(io.nosqlbench.engine.extensions.computefunctions.RelevancyFunctions.recall("recall",k));
relevancy.addFunction(io.nosqlbench.engine.extensions.computefunctions.RelevancyFunctions.precision("precision",k));
relevancy.addFunction(io.nosqlbench.engine.extensions.computefunctions.RelevancyFunctions.F1("F1",k));
relevancy.addFunction(io.nosqlbench.engine.extensions.computefunctions.RelevancyFunctions.reciprocal_rank("RR",k));
relevancy.addFunction(io.nosqlbench.engine.extensions.computefunctions.RelevancyFunctions.average_precision("AP",k));
}
verifier: |
// driver-specific function
actual_indices=pinecone_utils.responseIdsToIntArray(result)
// driver-agnostic function
relevancy.accept({validation_set},actual_indices);
return true;The scenario block could change like this: (I'm skipping the foreach entries) scenarios:
pinecone_vectors:
rampup: run tags==block:vector_writes cycles={trainsize} threads=TEMPLATE(rampup_threads,100) driver=pinecone apiKey=TEMPLATE(apikey) projectName=f88a480 environment=eu-west4-gcp
search_and_index: run tags='block:vector_reads' threads=TEMPLATE(search_threads,100) driver=pinecone apiKey=TEMPLATE(apikey) projectName=f88a480 environment=eu-west4-gcp stride=100 striderate=13.5Cycles (trainsize) is a function of the chosen dataset. For bindings it's even more live data: bindings:
rw_key: ToString()
train_vector: HdfFileToFloatList("testdata/{dataset}.hdf5", "/train");
test_vector: HdfFileToFloatList("testdata/{dataset}.hdf5", "/test");
validation_set: HdfFileToIntArray("testdata/{dataset}.hdf5", "/neighbors");
random_vector: HashedFloatVectors({dimensions});Here's a block ops with changes: vector_writes:
ops:
rampup-op:
upsert: {index_name}
namespace: TEMPLATE(namespace)
upsert_vectors:
- id: "{rw_key}"
values: "{train_vector}"Should we somehow make these into bindings for their use in ops? |
Beta Was this translation helpful? Give feedback.
-
Goals
We need to be able to iterate over a set of parameter combinations so that we can test all meaningful cases around a given set of questions.
Example question: Which tuning parameters work best across our canonical datasets to achieve a minimum recall of 90% while maintaining high performance on a standard deployment?
Practical result:
Motivating Example
Suppose you want to understand relevancy measures over a set of workloads, K, and system loads. Here are some parameters that you might use:
This yields a search space of 240 distinct combinations, which is an incredible number of tests to run for a single battery. Ideally, we do not run all of these as full-length tests. It becomes feasible to run these at this grain of detail if each combination can be test quickly and with cohesion between the parameters and the results for each.
Given that large search spaces like this yield higher value information when you measure the edges of the space first, it should be default that the bounds of each parameter are tested first. This is simply using the parameters provided while ignoring any interior value. Thus, the search space becomes 2^P (P is the set of parameter names)
Possible Approaches
template variable enhancements
We presently support
TEMPLATE(name,default)style variables which can contain a single value. If we were allowed to provide values for these template variables as inputs, perhaps from a list form likename=[glove25,glove50,...]then it would be simple to build an iterable form of the workload over all combinations.structural templates
Introduce a in-yaml (and in-json) templating scheme based on examples from other projects to auto-render combinations from procedural template language definitions.
jsonnet
lean on Jsonnet more to render these blocks.
Beta Was this translation helpful? Give feedback.
All reactions