caddyfile: Pass blocks to import for snippets

[elee1766]

as the amount of caddyfiles and snippets in our configuration grows, we have found it might be useful to be able to declare something like an {outlet} which spits out a configured block in the import statement.

The benefit is that it allows us to create snippets which can serve as entrypoints that must contain a superset. this is makes the configs less prone to error, as the rule becomes to use the snippet, instead of to not forget the import.

for instance, imagine a config like this:

(default_config) {
  foo bar
  bar foo
  one two 
}

:3838 {
  route / {
    module {
      import (default_config)
      extended_one one
      extended_two 2
    }
    respond "hello"
  }
}

after:

(module_func) {
    module {
      foo bar
      bar foo
      one two 
      {outlet}
    }
}
:3838 {
  route / {
    import module_func {
      extended_one one
      extended_two 2
    }
    respond "hello"
  }
}

my snippet, module_func, is now able to act as a default configuration for module. downstream users can be told ONLY to use module_func. This will avoid the mistake of ever using module without import default_config.

maybe this is a feature you guys might also find interesting?

as import statements currently don't consume blocks at all, i dont think it would break anything. it would allow for some more contrived caddyfiles (is that a bad thing)

i've made this a PR as we were testing this out with our config, and those few lines in the code made it possible - but maybe im missing something and this is really dangerous and there's a good reason the feature doesn't exist.

we refactored parts of our config to use it and it does seem safer - doing X is much easier than not forgetting Y.

[francislavoie]

Interesting idea. I could see the value in that, since right now only simple arguments are supported.

I think this is still quite limited, and there's room for making it even more powerful. Also, I don't like the "outlet" name, doesn't feel like it fits well with Caddyfile conventions.

I only spend like two minutes thinking about it, but I could see something like this being possible; since we already have {args[*]}, we could introduce {block} and {block.*}. The former is basically exactly what you have. But {block.*} would allow for "named blocks" to be used, so you could have more than one "outlet". I'm taking inspiration from the "slots" pattern in React's JSX.

(snippet) {
	module_one {
		foo
		{block.one}
	}
	module_two {
		{block.two}
	}
}

:8080 {
	import snippet {
		one {
			foo bar
		}
		two {
			bar baz
		}
	}
}

Producing 👇:

:8080 {
	module_one {
		foo
		foo bar
	}
	module_two {
		bar baz
	}
}

The implementation may be tricky, means having to reach into the block if we encounter blocks.* to grab the segment marked by that name.

[elee1766]

Interesting idea. I could see the value in that, since right now only simple arguments are supported.

the limitation is as you point out - we are currently able to pass []string into snippets, so i was looking for a way to provide []Token. Being able to pass tokens is what opens up a lot of windows, since now snippets can become almost like an inline ast transformer.

I think this is still quite limited, and there's room for making it even more powerful. Also, I don't like the "outlet" name, doesn't feel like it fits well with Caddyfile conventions.

I only spend like two minutes thinking about it, but I could see something like this being possible; since we already have {args[*]}, we could introduce {block} and {block.*}. The former is basically exactly what you have. But {block.*} would allow for "named blocks" to be used, so you could have more than one "outlet". I'm taking inspiration from the "slots" pattern in React's JSX.

"blocks" makes more sense to me. (funny, outlet was taken from a react project as well https://reactrouter.com/en/main/components/outlet )

i think multiple named blocks makes sense. for instance, if args supplies a []string, then blocks would provide {string: []Token}, with an omitted name being available at {blocks.}.

The implementation may be tricky, means having to reach into the block if we encounter blocks.* to grab the segment marked by that name.

another option would be arrays, ala

(snippet) {
	{blocks[0]}
	{blocks[1]}
}
import snippet {
	{
		bar
	}
	{
		foo
	}
}

however, i'm not convinced that this makes implementation any easier.

[francislavoie]

"anonymous" blocks is not a convention we support, except for global options specifically as a special case. So I don't think numerically indexed blocks like that is the right approach.

Of course "naming things" is always hard (often tricky with named matchers, and even snippet names, what should they be called?) but it's definitely more flexible/scalable than numeric indexes, and less confusing to read because the name can carry some intent.

One thing to think about is we did deprecate {args.*} in favour of {args[*]} recently because [] lets us do fun things like [1:] and whatnot. So because of that, I feel like {blocks.*} might feel incongruous? I don't think {blocks[foo]} makes sense though, the [] syntax only really works with numbers as indexes.

So yeah my thoughts right now are {block} (singular) and {blocks.*} (plural) as the syntax.

[elee1766]

ah, i didn't know unnamed blocks were a no-no.

some more things i thought of:

1. should nested keys should exist? for instance -

import snippet {
    foo {
       bar {
          baz 
       }
    }
}

{blocks.foo.bar} -> [baz]
{blocks.foo} -> [bar {\n baz\n }\n]
{blocks.*} -> [foo {\n bar {\n baz\n }\n }\n]

imo, if . will have special meaning - it might be prudent to ensure that caddy will error if keys contain ., so that this feature can be implemented in the future, if not immediately.

2. why just blocks? i think we could also allow tokenizing the remaining args of the line.

import snippet {
    foo hello world
}

{blocks.foo} -> [hello world]
{blocks.*} -> [foo hello world]

3. it feels weird to have {block} and {blocks.*} when {blocks.foo} can be thought of as short fo {blocks.foo.*} so perhaps the block containing all tokens should be {blocks.*} and not {block}

import snippet {
    foo hello world
}
{blocks.*} -> [foo hello world]
{blocks} -> [foo hello world]

4. since the parsed blocks are just arrays of tokens, could [] syntax be used to index the tokens of the matched block? this also doesn't need to be implemented immediately, it seems pretty niche, but it might be prudent to forbid "[" and "]" in keys regardless

import snippet {
    foo hello world
}

{blocks.foo[1]} -> [world]

[francislavoie]

should nested keys should exist

I don't think so, that's out of scope. It's not import's responsibility to deal with nesting. We want to keep it simple.

{blocks.*}

I don't think we'd want {blocks.*} to literally be a thing. You'd either use {block} or {blocks.foo}. Trying to splat all blocks doesn't make sense IMO, just use {block} for that.

imo, if . will have special meaning - it might be prudent to ensure that caddy will error if keys contain ., so that this feature can be implemented in the future, if not immediately.

Not really a concern, . is conventional in placeholder keys. Just use \. to escape a dot. But either way, no nesting is allowed for this IMO so it doesn't matter.

why just blocks? i think we could also allow tokenizing the remaining args of the line.

It's not just blocks, you will still be able to use the args on the same line, backwards compatible. It doesn't make sense to have two sets of args.

[elee1766]

(snippet_block) {
    basic_auth {
      {block}
    }
}
(snippet) {
    basic_auth {
      {blocks.users}
      {blocks.admin}
    }
}

# pw: hiccup
:3838 {
  route / {
    import snippet {
      users {
        user $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
      }
      admin admin $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
    }
    respond "hello"
  }
  route /block {
    import snippet_block {
      block $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
    }
    respond "hello"
  }
}

this simple example works as intended! so it's a start

i'm parsing by just putting the tokens (for handling {block}) into a newly made dispenser. that's the only real hack and the rest seems to play nicely?

there's no logic for nested. i extract the key by doing strings.TrimPrefix(strings.TrimSuffix(token.Text, "}"), "{blocks.")

[elee1766]

https://github.com/caddyserver/caddy/tree/master/caddytest/integration/caddyfile_adapt

should tests for this be created here? if i create new .caddyfiletest in the directory, will it just work tm

[francislavoie]

Yep, files in there are automatically run.

[elee1766]

created some simple tests, block, blocks, nested snippet with name conflict. let me know if you have any other thoughts, i can try to make some more

[elee1766]

@francislavoie

i was wondering if i needed to do anything to move this along. if not, thats fine, just not sure if i should be doing anything other than merging master and making sure it continues to work and build

[mholt]

Apologies from my end; you're doing great work, and this is a good contribution; I am just behind. I have less than 2 pages left on my notifications backlog though so I am getting around to it!

Francis I'm sure will also get around to it eventually but between any of us maintainers, we will address this :)

[mholt]

This is still on my list just so you know! My list is much smaller now, and this is near the top. Now we're getting ready for the 2.8 release, and this might have to come after 2.8, but it'll be one of the first new features we circle back to in that case. 👍