Skip to content

Commit

Permalink
Add semantic conventions generation Make target (#2758)
Browse files Browse the repository at this point in the history
* No wrap RELEASING Semantic Convention Generation section

* Initial generator

* Update template render

* Add exception and schema templates

* Add semconv/internal http unification

* Add http template

* Add licenses header

* Embed the templates

* Update static version in schema tmpl

* Add semconv-generate target to Makefile

Use this target to generate versions of the semconv packages.

* Generate semconv packages

* Update RELEASING to use make semconv-generate

* Add comments to semconvkit

* Make SemVer a method instead of a field

* Remove semconv/v* from codecov

* Fix lint for semconvkit main.go

* Fix documentation of validateHTTPStatusCode

Co-authored-by: Chester Cheung <cheung.zhy.csu@gmail.com>
  • Loading branch information
MrAlias and hanyuancheung committed Apr 12, 2022
1 parent 33a4673 commit db7fd1b
Show file tree
Hide file tree
Showing 16 changed files with 809 additions and 4,609 deletions.
15 changes: 14 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ $(TOOLS)/semconvgen: PACKAGE=go.opentelemetry.io/build-tools/semconvgen
CROSSLINK = $(TOOLS)/crosslink
$(TOOLS)/crosslink: PACKAGE=go.opentelemetry.io/otel/$(TOOLS_MOD_DIR)/crosslink

SEMCONVKIT = $(TOOLS)/semconvkit
$(TOOLS)/semconvkit: PACKAGE=go.opentelemetry.io/otel/$(TOOLS_MOD_DIR)/semconvkit

DBOTCONF = $(TOOLS)/dbotconf
$(TOOLS)/dbotconf: PACKAGE=go.opentelemetry.io/build-tools/dbotconf

Expand All @@ -69,7 +72,7 @@ GOJQ = $(TOOLS)/gojq
$(TOOLS)/gojq: PACKAGE=github.com/itchyny/gojq/cmd/gojq

.PHONY: tools
tools: $(CROSSLINK) $(DBOTCONF) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(PORTO) $(GOJQ) $(SEMCONVGEN) $(MULTIMOD)
tools: $(CROSSLINK) $(DBOTCONF) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(PORTO) $(GOJQ) $(SEMCONVGEN) $(MULTIMOD) $(SEMCONVKIT)

# Build

Expand Down Expand Up @@ -126,6 +129,7 @@ test-coverage: | $(GOCOVMERGE)
(cd "$${dir}" && \
$(GO) list ./... \
| grep -v third_party \
| grep -v 'semconv/v.*' \
| xargs $(GO) test -coverpkg=./... -covermode=$(COVERAGE_MODE) -coverprofile="$(COVERAGE_PROFILE)" && \
$(GO) tool cover -html=coverage.out -o coverage.html); \
done; \
Expand Down Expand Up @@ -197,6 +201,15 @@ check-clean-work-tree:
exit 1; \
fi

SEMCONVPKG ?= "semconv/"
.PHONY: semconv-generate
semconv-generate: | $(SEMCONVGEN) $(SEMCONVKIT)
@[ "$(TAG)" ] || ( echo "TAG unset: missing opentelemetry specification tag"; exit 1 )
@[ "$(OTEL_SPEC_REPO)" ] || ( echo "OTEL_SPEC_REPO unset: missing path to opentelemetry specification repo"; exit 1 )
@$(SEMCONVGEN) -i "$(OTEL_SPEC_REPO)/semantic_conventions/trace" -t "$(SEMCONVPKG)/template.j2" -s "$(TAG)"
@$(SEMCONVGEN) -i "$(OTEL_SPEC_REPO)/semantic_conventions/resource" -t "$(SEMCONVPKG)/template.j2" -s "$(TAG)"
@$(SEMCONVKIT) -output "$(SEMCONVPKG)/$(TAG)" -tag "$(TAG)"

.PHONY: prerelease
prerelease: | $(MULTIMOD)
@[ "${MODSET}" ] || ( echo ">> env var MODSET is not set"; exit 1 )
Expand Down
40 changes: 15 additions & 25 deletions RELEASING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,23 @@

## Semantic Convention Generation

If a new version of the OpenTelemetry Specification has been released it will be necessary to generate a new
semantic convention package from the YAML definitions in the specification repository. There is a `semconvgen` utility
installed by `make tools` that can be used to generate the a package with the name matching the specification
version number under the `semconv` package. This will ideally be done soon after the specification release is
tagged. Make sure that the specification repo contains a checkout of the the latest tagged release so that the
generated files match the released semantic conventions.
New versions of the [OpenTelemetry specification] mean new versions of the `semconv` package need to be generated.
The `semconv-generate` make target is used for this.

There are currently two categories of semantic conventions that must be generated, `resource` and `trace`.
1. Checkout a local copy of the [OpenTelemetry specification] to the desired release tag.
2. Run the `make semconv-generate ...` target from this repository.

```
.tools/semconvgen -i /path/to/specification/repo/semantic_conventions/resource -t semconv/template.j2
.tools/semconvgen -i /path/to/specification/repo/semantic_conventions/trace -t semconv/template.j2
```

Using default values for all options other than `input` will result in using the `template.j2` template to
generate `resource.go` and `trace.go` in `/path/to/otelgo/repo/semconv/<version>`.
For example,

There are several ancillary files that are not generated and should be copied into the new package from the
prior package, with updates made as appropriate to canonical import path statements and constant values.
These files include:

* doc.go
* exception.go
* http(_test)?.go
* schema.go
```sh
export TAG="v1.7.0" # Change to the release version you are generating.
export OTEL_SPEC_REPO="/absolute/path/to/opentelemetry-specification"
cd "$OTEL_SPEC_REPO" && git checkout "tags/$TAG" && cd -
make semconv-generate # Uses the exported TAG and OTEL_SPEC_REPO.
```

Uses of the previous schema version in this repository should be updated to use the newly generated version.
No tooling for this exists at present, so use find/replace in your editor of choice or craft a `grep | sed`
pipeline if you like living on the edge.
This should create a new sub-package of [`semconv`](./semconv).
Ensure things look correct before submitting a pull request to include the addition.

## Pre-Release

Expand Down Expand Up @@ -130,3 +118,5 @@ Once verified be sure to [make a release for the `contrib` repository](https://g

Update [the documentation](./website_docs) for [the OpenTelemetry website](https://opentelemetry.io/docs/go/).
Importantly, bump any package versions referenced to be the latest one you just released and ensure all code examples still compile and are accurate.

[OpenTelemetry specification]: https://github.com/open-telemetry/opentelemetry-specification
84 changes: 84 additions & 0 deletions internal/tools/semconvkit/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package semconvkit is used to generate opentelemetry-go specific semantic
// convention code. It is expected to be used in with the semconvgen utility
// (go.opentelemetry.io/build-tools/semconvgen) to completely generate
// versioned sub-packages of go.opentelemetry.io/otel/semconv.
package main

import (
"embed"
"flag"
"log"
"os"
"path/filepath"
"strings"
"text/template"
)

var (
out = flag.String("output", "./", "output directory")
tag = flag.String("tag", "", "OpenTelemetry tagged version")

//go:embed templates/*.tmpl
rootFS embed.FS
)

// SemanticConventions are information about the semantic conventions being
// generated.
type SemanticConventions struct {
// TagVer is the tagged version (i.e. v1.7.0 and not 1.7.0).
TagVer string
}

func (sc SemanticConventions) SemVer() string {
return strings.TrimPrefix(*tag, "v")
}

// render renders all templates to the dest directory using the data.
func render(dest string, data *SemanticConventions) error {
tmpls, err := template.ParseFS(rootFS, "templates/*.tmpl")
if err != nil {
return err
}
for _, tmpl := range tmpls.Templates() {
target := filepath.Join(dest, strings.TrimSuffix(tmpl.Name(), ".tmpl"))
wr, err := os.Create(target)
if err != nil {
return err
}

err = tmpl.Execute(wr, data)
if err != nil {
return err
}
}

return nil
}

func main() {
flag.Parse()

if *tag == "" {
log.Fatalf("invalid tag: %q", *tag)
}

sc := &SemanticConventions{TagVer: *tag}

if err := render(*out, sc); err != nil {
log.Fatal(err)
}
}
20 changes: 20 additions & 0 deletions internal/tools/semconvkit/templates/doc.go.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package semconv implements OpenTelemetry semantic conventions.
//
// OpenTelemetry semantic conventions are agreed standardized naming
// patterns for OpenTelemetry things. This package represents the conventions
// as of the {{.TagVer}} version of the OpenTelemetry specification.
package semconv // import "go.opentelemetry.io/otel/semconv/{{.TagVer}}"
20 changes: 20 additions & 0 deletions internal/tools/semconvkit/templates/exception.go.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package semconv // import "go.opentelemetry.io/otel/semconv/{{.TagVer}}"

const (
// ExceptionEventName is the name of the Span event representing an exception.
ExceptionEventName = "exception"
)
114 changes: 114 additions & 0 deletions internal/tools/semconvkit/templates/http.go.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package semconv // import "go.opentelemetry.io/otel/semconv/{{.TagVer}}"

import (
"net/http"

"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/semconv/internal"
"go.opentelemetry.io/otel/trace"
)

// HTTP scheme attributes.
var (
HTTPSchemeHTTP = HTTPSchemeKey.String("http")
HTTPSchemeHTTPS = HTTPSchemeKey.String("https")
)

var sc = &internal.SemanticConventions{
EnduserIDKey: EnduserIDKey,
HTTPClientIPKey: HTTPClientIPKey,
HTTPFlavorKey: HTTPFlavorKey,
HTTPHostKey: HTTPHostKey,
HTTPMethodKey: HTTPMethodKey,
HTTPRequestContentLengthKey: HTTPRequestContentLengthKey,
HTTPRouteKey: HTTPRouteKey,
HTTPSchemeHTTP: HTTPSchemeHTTP,
HTTPSchemeHTTPS: HTTPSchemeHTTPS,
HTTPServerNameKey: HTTPServerNameKey,
HTTPStatusCodeKey: HTTPStatusCodeKey,
HTTPTargetKey: HTTPTargetKey,
HTTPURLKey: HTTPURLKey,
HTTPUserAgentKey: HTTPUserAgentKey,
NetHostIPKey: NetHostIPKey,
NetHostNameKey: NetHostNameKey,
NetHostPortKey: NetHostPortKey,
NetPeerIPKey: NetPeerIPKey,
NetPeerNameKey: NetPeerNameKey,
NetPeerPortKey: NetPeerPortKey,
NetTransportIP: NetTransportIP,
NetTransportOther: NetTransportOther,
NetTransportTCP: NetTransportTCP,
NetTransportUDP: NetTransportUDP,
NetTransportUnix: NetTransportUnix,
}

// NetAttributesFromHTTPRequest generates attributes of the net
// namespace as specified by the OpenTelemetry specification for a
// span. The network parameter is a string that net.Dial function
// from standard library can understand.
func NetAttributesFromHTTPRequest(network string, request *http.Request) []attribute.KeyValue {
return sc.NetAttributesFromHTTPRequest(network, request)
}

// EndUserAttributesFromHTTPRequest generates attributes of the
// enduser namespace as specified by the OpenTelemetry specification
// for a span.
func EndUserAttributesFromHTTPRequest(request *http.Request) []attribute.KeyValue {
return sc.EndUserAttributesFromHTTPRequest(request)
}

// HTTPClientAttributesFromHTTPRequest generates attributes of the
// http namespace as specified by the OpenTelemetry specification for
// a span on the client side.
func HTTPClientAttributesFromHTTPRequest(request *http.Request) []attribute.KeyValue {
return sc.HTTPClientAttributesFromHTTPRequest(request)
}

// HTTPServerMetricAttributesFromHTTPRequest generates low-cardinality attributes
// to be used with server-side HTTP metrics.
func HTTPServerMetricAttributesFromHTTPRequest(serverName string, request *http.Request) []attribute.KeyValue {
return sc.HTTPServerMetricAttributesFromHTTPRequest(serverName, request)
}

// HTTPServerAttributesFromHTTPRequest generates attributes of the
// http namespace as specified by the OpenTelemetry specification for
// a span on the server side. Currently, only basic authentication is
// supported.
func HTTPServerAttributesFromHTTPRequest(serverName, route string, request *http.Request) []attribute.KeyValue {
return sc.HTTPServerAttributesFromHTTPRequest(serverName, route, request)
}

// HTTPAttributesFromHTTPStatusCode generates attributes of the http
// namespace as specified by the OpenTelemetry specification for a
// span.
func HTTPAttributesFromHTTPStatusCode(code int) []attribute.KeyValue {
return sc.HTTPAttributesFromHTTPStatusCode(code)
}

// SpanStatusFromHTTPStatusCode generates a status code and a message
// as specified by the OpenTelemetry specification for a span.
func SpanStatusFromHTTPStatusCode(code int) (codes.Code, string) {
return internal.SpanStatusFromHTTPStatusCode(code)
}

// SpanStatusFromHTTPStatusCodeAndSpanKind generates a status code and a message
// as specified by the OpenTelemetry specification for a span.
// Exclude 4xx for SERVER to set the appropriate status.
func SpanStatusFromHTTPStatusCodeAndSpanKind(code int, spanKind trace.SpanKind) (codes.Code, string) {
return internal.SpanStatusFromHTTPStatusCodeAndSpanKind(code, spanKind)
}
20 changes: 20 additions & 0 deletions internal/tools/semconvkit/templates/schema.go.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package semconv // import "go.opentelemetry.io/otel/semconv/{{.TagVer}}"

// SchemaURL is the schema URL that matches the version of the semantic conventions
// that this package defines. Semconv packages starting from v1.4.0 must declare
// non-empty schema URL in the form https://opentelemetry.io/schemas/<version>
const SchemaURL = "https://opentelemetry.io/schemas/{{.SemVer}}"

0 comments on commit db7fd1b

Please sign in to comment.