Task name namespacing

[ryanrichholt]

Namespaces are a honking good idea. But tasks have no namespace. If two pipelines are run on the same project, and both pipelines have a task called start, it's tracking data will be overwritten by the last pipeline ran. This has been bugging me for a long time, but think I have an idea for how to fix it:

Since the import system is handled entirely by Jinja, the template files would have to be pre-processed before loading and rendering. During this process, data could be added which the task loader can identify to prepend task names with pipeline/template info.

note in these examples I've excluded the template extensions in the syntax (see #83)

Example pipeline foo:

# template_a

- name: foo
  cmd:  echo hello world

- name: bar
  cmd: echo the answer is 42

{% include 'foo/template_b' with context %}

# template_b

- name: baz
  cmd: echo What do you get if you multiply six by nine?

After preprocessing (still not sure if this would have to happen on disk when installing, or if the DictLoader in Jinja could be used)

---
pipeline: foo
template: template_a
---
# template_a

- name: foo
  cmd:  echo hello world

- name: bar
  cmd: echo the answer is 42

{% include 'foo/template_b' with context %}

---
pipeline: foo
template: template_b
---
# template_b

- name: baz
  cmd: echo What do you get if you multiply six by nine?

Final document after rendering:

---
pipeline: foo
template: template_a
---
# template_a

- name: foo
  cmd:  echo hello world

- name: bar
  cmd: echo the answer is 42

---
pipeline: foo
template: template_b
---
# template_b

- name: baz
  cmd: echo What do you get if you multiply six by nine?

And the loader would just:

data = yaml.safe_load_all()

if data[0] != None:
    raise

# while get next two items
for td in tasks_data:
    td[name] = f'{pipeline/None}.{template}.{task_name/hash}
    task  = Task(**td)

[ryanrichholt]

Moved this back to the 1.7 release so that new features can be added soon. Still need to figure out how before/after directives would work with this feature. Ideas:

• before/after need to include namespaces as well. This would be a lot of repetitive code and is my least favorite option.

• before/after can only link to tasks within a pipeline input/output would still be universal and could be used to link tasks between pipelines.

• before/after apply to this pipeline only unless specifically namespaced: before: taskA vs before: phoenix.taskA Since task names are already restricted to alpha-numeric hyphen and underscore, this would be an easy determination to make. But, I would eventually like to relax the task name rules. and this might make that harder to do.

[ryanrichholt]

Additionally, the preprocessing idea would fail for macros or extends statements depending on the final order of the template. Because of the nature of the template processing, I think the namespaces could only be scoped to the root-template or pipeline.

[ryanrichholt]

New idea - when loading a template, if a pipeline context is provided, the task names are converted to <pipeline_name>/<task_name>. A bonus feature of this pattern is that it defaults logs to be stored in sub-directories.

When declaring dependencies, task names can include a pipeline or not. This would allow before/after directives to link between pipelines. Input/output directives would still be "global".