Function Scaffolding

[lkingland]

I would appreciate any comments

Function Scaffolding

Proposed is a method by which we can deploy Functions with no dependencies
on the underlying Functions infrastructure.

Introduced is the term "Function Scaffolding", which refers to both the code and
stage of deployment process where a Function is wrapped in the code necessary
to run it as a process.

Background

Functions enable a developer to focus almost entirely on the logic of their
program by providing a simple interface with the underlying infrastructure which
will run their code. This interface is the function signature in their language
of choice.

To enable this simplicity, it is necessary to augment the developer's function
with the plumbing which exposes their function as a process, listening on
a port, etc.

This code, which exists to adapt the developer's Function to the operating
system's process scheduler within a cluster context, is its scaffolding.

It is imperative that this scaffolding not be part of a Function developer's
source code by default, since it is logically part of the platform which runs
Functions, rather than the Function itself.

This is a requirement for providing a function platform as a service because it
decouples the Function developer's source code from the method by which this
code is run. A Function's source code repository which contains none of this
scaffolding can remain unchanged even as the platform is updated.

As an analogy, just as we need not rebuild a virtual machine when our cloud provider
updates their hypervisor, nor do we need to rebuild our containers when our
container engine is updated, so too a Function developer should not need to
mutate their Function's source code when the Functions Runtime is updated. From
the perspective of the Function developer, a restart (likely automated) should
be sufficient.

Three examples of situations in which the Function as a Service platform should
be able to update without requiring a Function Developer's code to change:

• A CVE which affects the HTTP server used to translate a CloudEvent request
  into a Function invocation call has a patch released.
• A new endpoint for monitoring health and/or metrics needs to be added to
  improve platform visibility or health.
• Debugging needs to be enabled on the runtime binary for problem triage.

The proposal below outlines how to separate this platform code from
the Function's source repository, enabling the flexibility expected of a
platform, while importantly not losing the ability for a Function developer to
run their code directly, locally, outside of a container, or even opt in to
managing this scaffolding code themselves for more advanced cases.

Proposal

The following primary steps are executed in order to produce a Function,
running as a service in the currently connected Function Platform (cluster):

                        func deploy
   ┌─────────────────────────────────────────────────────────┐
   │ Scaffold → Build Container → Push → Deploy as a Service │

When a Function is deployed, scaffolding necessary to run the Function is first
written which imports the developer's Function as a dependency. This scaffolded
Function is now ready to be built into a container and deployed because it is
now a full Service.

This scaffolding code is transient, kept in the non-source-controlled .func
directory until the run or deploy task completes.

The func scaffold Subcommand

In addition to this scaffolding process being a default part of the deployment
process, the act of scaffolding is also exposed using a new command:
func scaffold.

This command allows a user to write the scaffolding code into their project,
effectively opting to take on the responsibility of updates and maintenance
themselves, but enabling them to accomplish more advanced tasks, such as
altering the code used for health and readiness endpoints, etc.

This subcommand is also useful in CI/CD contexts such as Tekton Pipelines
which do not support Functions natively, as they can run func scaffold in any
Function project retreived from source control, and immediately have a full
Service on which to operate.

Expanding the func run Subcommand

The implementation of func run would also use this same scaffoldng step,
because scaffolding is part of building the function container.

However, rather than performing the containerization task by default, it is
proposed that the default behavior of func run be to invoke a developer's
function locally on the host, just as if the function scaffolding code existed
in their project. From the developer's perspective, their host OS is now
capable of running Functions directly.

Containerizing and running the Funciton in a local container runtime would then
be accomplished using the command func run --container.

Implementation

In general, the suggested implementation of Function Scaffolding is to roughly
mirror the process of Function Templating. Each core supported language
includes its scaffolding code embedded in the library, with the ephemeral
.func runtime directory used for output.

Language scaffolding code would be kept in the ./scaffolding subdirectory
of the func repository and encoded as source code on generate in the same
manner as templates. This code is expected to be quite small, since all logic
is kept in dedicated functions runtime libraries for each supported language.

When a Function is built, the contents of the scaffolding is written into
.func/build/0/. This directory increments with any concurrent calls to build
in the same way run increments its runtime directory today, and the directory
is by default removed when the process completes. This scaffolding code
consists of everything necessary to run the Function as a Service except the
function code itself.

For example, in the case of Go, this would be main.go etc. This scaffolding
imports the user's Function code locally, as a library of the expected
method signature and residing at the relative path ./f
(myFunctionProject/.func/runs/0/f). This path would be a link
./f -> ../../../. Together, this creates a complete Service consisting of
the process boundary (main), Functions tooling (imported), and the Function
source code itself.

Included in the scaffolding is the command necessary to invoke the service
as run.sh, and for languages which require compilation, a build.sh.
run.sh then becomes the entrypoint for the final container.

If a developer chooses to func scaffold their project and take the source
code, by default it would be written to ./dist, and scaffolded:true written
to func.yaml. On subsequent builds, the scaffolding step would be skipped
and the commands ./dist/build.sh and ./dist/run.sh used instead.

Running Locally with func run

Similarly to deployment, func run scaffolds the project, but uses the local
path .func/runs/0. By default, the run script is executed as a subprocess
to run the scaffolded function (service). Running can optionally be from
within a containerized environment using func run --container which
continues on to create the container and invoke it rather than the code in
.func/runs/0.

Additional Configuration

• Additional build and run arguments can be defined in func.yaml, which are
  appended to the parameters used in the default build and run scripts OR
• The build and run scripts can be overridden entirely.
• The optional flag --keep can be added when running or deploying to cause the
  retention of the ephemeral .func artifacts for educational or debugging
  purposes.

Summary

By adding the "scaffolding" step as a distinct phase of a Function's lifecycle,
we allow our Functions to be free of platform dependencies. This enables us
to create a clear distinction between a Function and a Function Platform, and
is a prerequisite to providing services expected of a platform such as
independent update cycles, as well as more advanced Function features which
will depend on there being a distinct phase at which a Function is scaffolded.

I would appreciate any comments or questions

[lkingland]

Example

Below is an example walkthrough of the root of the proposal, using a highly simplified Go language runtime for illustration.

The func source code project would contain scaffolding and template files for projects:

.
├── scaffolding
│   └── go
│       └── http
│           ├── build.sh
│           ├── run.sh
│           ├── main.go
│           └── f
└── templates
    └── go
        └── http
            ├── go.mod
            └── handle.go

Creating a new Go function writes the template files as normal:

❯ mkdir f1 && cd $_
❯ func create -l go
❯ ls -a
  .func
  .gitignore
  func.yaml
  go.mod
  handle.go

Notice how the function written has no dependencies on a function language
runtime:

❯ cat handle.go
package function

import (
	"context"
	"fmt"
	"net/http"
)

func Handle(res http.ResponseWriter, req *http.Request) {
	fmt.Fprintf(res, req) // echo
}

We can now run this Funciton locally (outside of a container) just as though
it had a main and the whole process boundary:

❯ func run --keep
Function listening on http port 8080
^C
❯

To see how this happened we can take a look in the ephemeral directory
that was created for this run (not removed because we used --keep).

❯ ls -l .func/runs/0
lrwxr-xr-x  1     9 f -> ../../../
-rw-r--r--  1    94 go.mod
-rw-r--r--  1   129 main.go

The main file starts the functions language middleware. Note that here is where the dependency is inverted: the functions middleware "the platform"
has a dependency on the user's function, rather than the function having a
dependency on the middleware.

❯ cat .func/runs/0/main.go
package main

import (
	f "f"
	"net/http"
)

func main() {
      // TODO: import and invoke the Go functions language runtime
      // middleware.  A simple http.ListenAndServe is used here for illustration.
	http.HandleFunc("/", f.Handle)
	http.ListenAndServe(":8080", nil)
}

The go.mod file registers the symplink as containing the function's source code.

❯ cat .func/runs/0/go.mod
module s

replace f => ./f

go 1.19

require f v0.0.0-00010101000000-000000000000 // indirect

The build.sh and run.sh scripts run go build and ./s respectively (from within .func/runs/0 directory to avoid missing module errors).

This directory then can be the Service which is built into a container
using func run --contianer, or when running func deploy.

[lance]

I really like this proposal, and it's nice that it will work with interpreted languages such as Node.js and Python. As an aside, I do like having https://github.com/knative-sandbox/func-go primarily for visibility and potential continuity with other runtimes. Based on what you write here,

This code is expected to be quite small, since all logic
is kept in dedicated functions runtime libraries for each supported language.

using a more sophisticated scaffolding than in your example, the ./func/runs/0/main.go file itself would be quite simple, having primarily func main() and a dependency on the full scaffolding in a versioned library. Is that a correct reading?

We also should be considering how this will be surfaced in Language Packs. Do these now have two top level directories at the repository root containing scaffolding and functions separately? Essentially the same as the func binary?

.
├── scaffolding
│   └── go
│       └── http
│           ├── build.sh
│           ├── run.sh
│           ├── main.go
│           └── f
└── templates
    └── go
        └── http
            ├── go.mod
            └── handle.go

Or do they instead have scaffolding defined per-template or per-runtime within the current structure?

root
├── ruby
|    ├── cloudevent
|    │   ├── func.rb
|    |   ├── manifest.yaml
|    │   ├── Gemfile
|    │   ├── Rakefile
|    │   ├── README.md
|    │   └── scaffolding
│    │       ├─── build.sh
│    │       ├─── run.sh
│    │       └─── main.rb
|    ├── http
|    │   ├── func.rb
|    |   ├── manifest.yaml
|    │   ├── Gemfile
|    │   ├── Rakefile
|    │   ├── README.md
|    │   └── scaffolding
│    │       ├─── build.sh
│    │       ├─── run.sh
│    │       └─── main.rb
|    └── manifest.yaml
└── manifest.yaml

The latter option would make scaffolding a reserved directory name in the template file structure, of course. I think the first option is cleaner, however, it means we need to version our Language Pack templating in some way to account for backward compatibility. Having a plan for Language Packs is important, but I don't think the implementation of that plan should delay/defer moving ahead with this proposal.

/cc @knative/func-reviewers @knative/func-writers @knative/functions-wg-leads @knative/technical-oversight-committee

[lkingland]

using a more sophisticated scaffolding than in your example, the ./func/runs/0/main.go file itself would be quite simple, having primarily func main() and a dependency on the full scaffolding in a versioned library. Is that a correct reading?

Yes, that's exactly what I was thinking as well. Just a main file which glues together the user's function and the runtime libraries (func-go in this case)

in Language Packs. Do these now have two top level directories
...
Or do they instead have scaffolding defined per-template or per-runtime

It would be nice to keep them as similar to each other as possible, with the scaffolding directories being of course optional. However a deviation from the way it is structured in the func repository should be no problem if something else seems more logical. I like the example you give, for example. Perhaps that's a better way to structure it within the source repository as well.

[zroubalik]

I like the proposal too!

While reading the proposal I had the same concern regarding the structure as @lance .

[aslom]

I think it would be great if it were possible to allow multiple scaffolding and templates directories to be applied to source based on manifest.yaml or override with what developers want to happen by modifying func.yaml - they could add any behavior and rewrite code generated as needed?

[aslom]

In the scaffolding example where is the yaml file(s) of what is deployed to Kubernetes namespace? How can I customize it?

[lkingland]

In the scaffolding example where is the yaml file(s) of what is deployed to Kubernetes namespace? How can I customize it?

When a function is created using func create ..., a func.yaml is written to disk, which includes all configuration options for how the deployer does the eventual deployment. There is no .yaml file which is deployed; we use the knative serving API directly.

To customize what is included in the default func.yaml written out when a function is created using a custom template like you are creating, use a manifest.yaml in the template directory. It's a bit like a func.yaml template.

[lance]

Check out the manifest.yaml reference here: https://github.com/knative/func/blob/main/docs/language-packs/language-pack-contract.md#language-pack-manifests

[zroubalik]

When a Function is built, the contents of the scaffolding is written into
.func/build/0/. This directory increments with any concurrent calls to build

I am sorry, I don't follow why we need to increment the directory?

[lance]

I wondered about this too. It makes sense in the context of func run in case there are possibly two processes running at the same time. It's an edge case, but not impossible. Possibly overkill in this case?

[lkingland]

This is for cuoncurrency: in case one is deploying concurrently from multiple terminals. Just like run, which allows running multiple instances and assigns each a unique port.
The other option would of course be to detect there is a build/run in progress and just return an error to whichever was last to the party.

[zroubalik]

Included in the scaffolding is the command necessary to invoke the service
as run.sh, and for languages which require compilation, a build.sh.
run.sh then becomes the entrypoint for the final container.

I am curious, can't we somehow skip these scripts and run the functionality directly from func binary? This way, we woudn't have to maintain platfrom dependent versions of scripts. Or is it a bad idea? I haven't thought about it properly.

[lkingland]

This is because a run.sh and build.sh is language-independent, so the func tooling can remain agnostic as to the actual language nuances. For example, func can just invoke the scripts, and the individual language packs can decide exactly how to implement build and run. I would assume this is why Buildpacks use this methodology as well (along with a detection script)
Of course, once we get into the work of implementation, this may not be necessary; it is a suggestion of a first approach
"Everyone has a plan until they get punched in the face" ~Tyson

[aslom]

Quick question: how func repositories will work with scaffolding? Can I provide some part of scaffolding in tempalte repository that will override (merge) into existing scaffolding?

[lkingland]

Yes I expect that scaffolding can be optionally provided within the template directory to override the scaffolding which would by default be applied to a function.
Note that there would be no need to provide your own scaffolding unless you wanted to change how the function is wrapped and presented as a network service

[aslom]

For scaffolding can we also have an option to scaffold Kubernetes artifacts that are deployed? I think right now they are only in func code and not possible to modify or overwrite without changing source code?

[lkingland]

To reiterate what we briefly discussed on the call, Labels, Annotations and other metadata should be available separately without needing to use scaffolding. Scaffolding really affects the source code itself.

[aslom]

What mechanism could be used in future to affect knative func deployment?

Would it be an extension of scaffolding or something else?

[lkingland]

I would probably need an example of what sorts of additional effects you're looking for to be more helpful, but I can confirm this is unlikely to be part of scaffolding, since that works at the source-code level whereas the knative deployment process works on/after the container has been built.

[aslom]

I like the approach that buildpacks took for extensibility with buildpack groups [1] and I wonder if it can be extended to scaffolding, templates, etc. in knative func?

For use cases I think about it would be great to generate all source code scaffolding and deployment artifacts by applying set of builders/transformations that are encoded as Docker images similiar to buildpack groups. It would be easy to create ecosystem of knative func scaffolding builders/transformers and users could easily apply them to their knative function project.

[1] https://buildpacks.io/docs/concepts/components/buildpack-group/

[aslom]

Can user put .func scaffolding into git repo? What would happen? What id scaffolding was modified or outdated or our-of-sync (from other version of func)?

[lkingland]

Scaffolding is written into the .func directory specifically to avoid it going out-of-date. Since .func is not checked into source control, and scaffolding is generated every time a function is deployed, it always will have the latest scaffolding presuming the user is using the lastest func binary or client library.

[aslom]

Writing down points from WG meeting: there is contract that func expects (funcion signature etc) - we may have some way of checking it and giving developers warnings or validation. also some kind of version checking about what is required version of func to use scaffolding if it were modified?

[lkingland]

This system is now available for the Go runtime behind the feature-flag FUNC_ENABLE_HOST_BUILDER

[lkingland]

See the Epic-issue #1703 which enumerates the (larger) issues related to implementing this MVP of the Scaffolded-functions.

[lkingland]

Reopening in leau of having actual documentation just yet