CLI does not read config file if workspace is specified

[Goldziher]

Hi There,

I encountered a bug using the CLI to lint a project. If the buf.work.yaml file exists, the CLI ignores the buf.yaml file with the lint configuration.

Trying to force loading of the configuration with npx buf lint --config ./buf.yaml results in:

Failure: the --config flag is not compatible with workspaces

This indicates the culprit.

Expected behavior:

Even given a workspace, the lining configuration is loaded from a buf.yaml file.

[timostamm]

Hey @Goldziher, can you tell me a bit more about your directory structure?

Since you are using buf.work, I would expect a tree similar to this:

.
├── buf.work.yaml
├── paymentapis
│   ├── a.proto
│   └── buf.yaml
└── petapis
    ├── b.proto
    └── buf.yaml

buf lint will honor buf.work.yaml, and lint both paymentapis, and petapis. If you provide a --config flag, it is no longer clear what the CLI should do with the configuration files in the module directories.

If you have just a single module in your repository, I would recommend to remove buf.work, and run buf lint with the path to your module as an argument.

So for the following example:

.
└── paymentapis
    ├── a.proto
    └── buf.yaml

You would simply run buf lint paymentapis.

But I might be off on what you actually need. Happy to hear about details 🙂

[Goldziher]

So, what I had is

.
├── buf.work.yaml
├── buf.yaml
├── buf.gen.yaml
└──  proto
    └── x.proto

I use a monorepo with top level configs, and I generate all output into a gen namespace on the root.

The reason I had to use buf.work.yaml to begin with is that I did not want the output path to include /proto. That is, I wanted:

.
└──  gen
    └── x.go

rather than

.
└──  gen
     └──  proto
           └── x.go

[timostamm]

Ah, I think I see the issue. It is perfectly valid to move your proto files into a sub-directory like proto. We do it ourselves in the buf CLI here. But it is important to move buf.yaml into this directory as well. From the documentation:

The buf.yaml file defines a module, and is placed at the root of the Protobuf source files it defines.

I suppose you started with:

.
├── buf.gen.yaml
├── buf.yaml
└── proto
    └── x.proto

Now you run buf generate, and end up with generated code within a proto directory:

.
├── buf.gen.yaml
├── buf.yaml
├── gen
│   └── proto
│       └── x.pb.go
└── proto
    └── x.proto

It's understandable that you wouldn't want that. We definitely recommend to avoid it too!
The CLI still does it anyways because it took the root of your project as the root of your proto files, and finds the file proto/x.proto. The plugin for generating code receives the file proto/x.proto, and generates a file proto/x.pb.go.

The most simple solution is to move buf.yaml into the proto directory:

.
├── buf.gen.yaml
└── proto
    ├── buf.yaml
    └── x.proto

Now you run buf generate proto (telling the CLI to use the proto/ directory as the input), and you end up with a reasonable result:

.
├── buf.gen.yaml
├── gen
│   └── x.pb.go
└── proto
    ├── buf.yaml
    └── x.proto

If you don't want to append proto to all CLI commands, simply add a buf.work at the root of your repository with a single entry for the proto directory. We do that ourselves here.

Apologies for the bumpy start. We will likely improve how workspaces work soon, so that it's much less likely to run into such a situation.

[Goldziher]

Apologies for the bumpy start. We will likely improve how workspaces work soon, so that it's much less likely to run into such a situation.

Its all good, thanks a bunch.

So for my use case - I really hate to move the buf.yaml file into the proto folder - i want to have all configs top level.

Frankly, I dont think I need a buf.work.yaml file since I intend to have a single proto folder for the time being.

Some feature requests I would like to make (perhaps one or the other) - I would be happy to open issues if any of them are relevant:

1. allow configuration of the output path in the buf.yaml file globally. This can take the format of a template, e.g. my_output_folder/<lang>/<source_file_path>, and thus I could simply decide how the output path should look like, potentially deciding what I want is instead this: my_output_folder/<lang> or some other formula.
2. allow omitting the proto using buf.yaml without buf.work.yaml. I don't really need a buf.work.yaml file in this case.
3. allow specifying the output directories per plugin in buf.gen.yaml. By this i dont mean the base output folder, but rather the absolute output folder - this is currently predetermined to mirror the path of the proto file, but this does not give a lot of control.

[timostamm]

Heard you on point 1 and 2.

Point three has some less-obvious complications: Consider a plugin a generating foo/bar.pb.go, and a second plugin generating foo/bar.client.go that imports from foo/bar.pb.go. In Go, imports are absolute, but in JS and other languages, they are relative. How should the buf CLI allow you to modify the output in a reasonable but reliable way? It might be best to discuss based on a concrete example 🙂

[Goldziher]

Heard you on point 1 and 2.

Point three has some less-obvious complications: Consider a plugin a generating foo/bar.pb.go, and a second plugin generating foo/bar.client.go that imports from foo/bar.pb.go. In Go, imports are absolute, but in JS and other languages, they are relative. How should the buf CLI allow you to modify the output in a reasonable but reliable way? It might be best to discuss based on a concrete example 🙂

I would consider adding per source overrides. This also relates to the other issue I opened --> #2432

Basically, when using monorepos of multiple languages, you might want to control the outputs in a more granular way - apply certain plugins only to certain files, and perhaps output files to different places based on the source files.

[timostamm]

apply certain plugins only to certain files

You can do that today with the following flags:

      --path strings           Limit to specific files or directories, e.g. "proto/a/a.proto", "proto/a"
      --exclude-path strings   Exclude specific files or directories, e.g. "proto/a/a.proto", "proto/a"
      --type strings           The types (package, message, enum, extension, service, method) that should be included in this image. When specified, the resulting image will only include descriptors to describe the requested types. Flag usage overrides buf.gen.yaml

... in combination with multiple template files.

We are working on making this functionality available in configuration, so that you do not end up with many command line calls 🙂

output files to different places based on the source files

Again, this is best to reason about on a concrete example. If you have foo/bar.proto, where would you want to generate the output to?

[Goldziher]

apply certain plugins only to certain files

You can do that today with the following flags:

      --path strings           Limit to specific files or directories, e.g. "proto/a/a.proto", "proto/a"
      --exclude-path strings   Exclude specific files or directories, e.g. "proto/a/a.proto", "proto/a"
      --type strings           The types (package, message, enum, extension, service, method) that should be included in this image. When specified, the resulting image will only include descriptors to describe the requested types. Flag usage overrides buf.gen.yaml

... in combination with multiple template files.

We are working on making this functionality available in configuration, so that you do not end up with many command line calls 🙂

output files to different places based on the source files

Again, this is best to reason about on a concrete example. If you have foo/bar.proto, where would you want to generate the output to?

I understand. What i need is not to limit to file x, but to limit to a subset of plugins for that file.