Mixer Compiled In Adapter Walkthrough

From Istio 1.0 (July 2018) please prefer to implement an Out of Process / grpc adapter

• Out of process adapters decouple you from Mixer.
• You maintain adapter code in your own repository.
• Your adapter is still documented on istio.io

This document walks through step-by-step instructions to implement, test and plug a simple adapter into Mixer. For complete details on the adapter life cycle, please refer to the Compiled-In Adapter Developer's Guide.

Note: To complete this walkthrough, it is optional to read the adapter developer's guide. However, to create a real production quality adapter, it is highly recommended you read the guide to better understand adapter lifecycle and various interfaces and objects that Mixer uses to interact with adapters.

In this walkthrough you're going to create a simple adapter that:

• Supports the metric template which ships with Mixer.

• For every request, prints to a file the data it receives from Mixer at request time.

It should take approximately ~30 minutes to finish this task

• Before you start
• Step 1: Write basic adapter skeleton code
• Step 2: Write adapter configuration
• Step 3: Link adapter config with adapter code
• Step 4: Write business logic into your adapter
• Step 5: Plug adapter into the Mixer
• Step 6: Write sample operator config
• Step 7: Start Mixer and validate the adapter
• Step 8: Write test and validate your adapter
• Step 9: Cleanup
• Step 10: Next

Before you start

Download a local copy of the Mixer repo

git clone https://github.com/istio/istio

Install protoc (version 3.5.1 or higher) from https://github.com/google/protobuf/releases and add it to your PATH

Set the MIXER_REPO variable to the path where the mixer repository is on the local machine. Example export MIXER_REPO=$GOPATH/src/istio.io/istio/mixer Also, $ISTIO should point to $GOPATH/src/istio.io.

Successfully build the mixer server.

pushd $ISTIO/istio && make mixs

Step 1: Write basic adapter skeleton code

Create the mysampleadapter directory and navigate to it.

cd $MIXER_REPO/adapter && mkdir mysampleadapter && cd mysampleadapter

Create the file named mysampleadapter.go with the following content

It defines the adapter's builder and handler types along with the interfaces required to support the 'metric' template. This code so far does not add any functionality for printing details in a file. It is done in later steps.

package mysampleadapter

import (
  "context"

  "github.com/gogo/protobuf/types"
  "istio.io/istio/mixer/pkg/adapter"
  "istio.io/istio/mixer/template/metric"
)

type (
  builder struct {
  }
  handler struct {
  }
)

// ensure types implement the requisite interfaces
var _ metric.HandlerBuilder = &builder{}
var _ metric.Handler = &handler{}

///////////////// Configuration-time Methods ///////////////

// adapter.HandlerBuilder#Build
func (b *builder) Build(ctx context.Context, env adapter.Env) (adapter.Handler, error) {
  return &handler{}, nil
}

// adapter.HandlerBuilder#SetAdapterConfig
func (b *builder) SetAdapterConfig(cfg adapter.Config) {
}

// adapter.HandlerBuilder#Validate
func (b *builder) Validate() (ce *adapter.ConfigErrors) { return nil }

// metric.HandlerBuilder#SetMetricTypes
func (b *builder) SetMetricTypes(types map[string]*metric.Type) {
}

////////////////// Request-time Methods //////////////////////////
// metric.Handler#HandleMetric
func (h *handler) HandleMetric(ctx context.Context, insts []*metric.Instance) error {
  return nil
}

// adapter.Handler#Close
func (h *handler) Close() error { return nil }

////////////////// Bootstrap //////////////////////////
// GetInfo returns the adapter.Info specific to this adapter.
func GetInfo() adapter.Info {
  return adapter.Info{
     Name:        "mysampleadapter",
     Description: "Logs the metric calls into a file",
     SupportedTemplates: []string{
        metric.TemplateName,
     },
     NewBuilder:    func() adapter.HandlerBuilder { return &builder{} },
     DefaultConfig: &types.Empty{},
  }
}

Just to ensure everything is good, let's build the code

go build ./...

Now we have the basic skeleton of an adapter with empty implementation for interfaces for the 'metric' templates. Later steps adds the core code for this adapter.

Step 2: Write adapter configuration

Since this adapter just prints the data it receives from Mixer into a file, the adapter configuration will take the path of that file as a configuration field.

Create the config proto file under the 'config' dir

mkdir config

Create a new config.proto file inside the config directory with the following content:

syntax = "proto3";

package adapter.mysampleadapter.config;

import "google/protobuf/duration.proto";
import "gogoproto/gogo.proto";

option go_package="config";

message Params {
    // Path of the file to save the information about runtime requests.
    string file_path = 1;
}

Let's now generate the corresponding go file from the config.proto. To do this, add the following go generate comment to the adapter code. The bold text shows the new added text.

//go:generate $GOPATH/src/istio.io/istio/bin/mixer_codegen.sh -f mixer/adapter/mysampleadapter/config/config.proto
package mysampleadapter

import (
  "context"

  "github.com/gogo/protobuf/types"
  "istio.io/istio/mixer/pkg/adapter"
  "istio.io/istio/mixer/template/metric"
)
..
..

Just to ensure everything is good, let's generate the file and build the code

go generate ./...
go build ./...

If nothing happens on go generate, make sure protoc is installed and is in path.

Step 3: Link adapter config with adapter code.

Modify the adapter code (mysampleadapter.go) to use the adapter-specific configuration (defined in mysampleadapter/config/config.proto) to instantiate the file to write to. Also update the GetInfo function to allow operators to pass the adapter-specific config and for the adapter to validate the operator provided config. Copy the following code and the bold text shows the new added code.

//go:generate $GOPATH/src/istio.io/istio/bin/mixer_codegen.sh -f mixer/adapter/mysampleadapter/config/config.proto
package mysampleadapter

import (
    // "github.com/gogo/protobuf/types"
	"context"
	"os"
	"path/filepath"

	"istio.io/istio/mixer/adapter/mysampleadapter/config"
	"istio.io/istio/mixer/pkg/adapter"
    "istio.io/istio/mixer/template/metric"
)

type (
	builder struct {
		adpCfg *config.Params
	}
	handler struct {
		f *os.File
	}
)

// ensure types implement the requisite interfaces
var _ metric.HandlerBuilder = &builder{}
var _ metric.Handler = &handler{}

///////////////// Configuration-time Methods ///////////////

// adapter.HandlerBuilder#Build
func (b *builder) Build(ctx context.Context, env adapter.Env) (adapter.Handler, error) {
	file, err := os.Create(b.adpCfg.FilePath)
	return &handler{f: file}, err

}

// adapter.HandlerBuilder#SetAdapterConfig
func (b *builder) SetAdapterConfig(cfg adapter.Config) {
	b.adpCfg = cfg.(*config.Params)
}

// adapter.HandlerBuilder#Validate
func (b *builder) Validate() (ce *adapter.ConfigErrors) {
	// Check if the path is valid
	if _, err := filepath.Abs(b.adpCfg.FilePath); err != nil {
		ce = ce.Append("file_path", err)
	}
	return
}

// metric.HandlerBuilder#SetMetricTypes
func (b *builder) SetMetricTypes(types map[string]*metric.Type) {
}

////////////////// Request-time Methods //////////////////////////
// metric.Handler#HandleMetric
func (h *handler) HandleMetric(ctx context.Context, insts []*metric.Instance) error {
	return nil
}

// adapter.Handler#Close
func (h *handler) Close() error {
	return h.f.Close()
}

////////////////// Bootstrap //////////////////////////
// GetInfo returns the adapter.Info specific to this adapter.
func GetInfo() adapter.Info {
	return adapter.Info{
		Name:        "mysampleadapter",
		Description: "Logs the metric calls into a file",
		SupportedTemplates: []string{
			metric.TemplateName,
		},
		NewBuilder:    func() adapter.HandlerBuilder { return &builder{} },
		DefaultConfig: &config.Params{},
	}
}

Just to ensure everything is good, let's build the code

go build ./...

Step 4: Write business logic into your adapter.

Print Instance and associated Type information in the file configured via adapter config. This requires storing the metric type information at configuration-time and using it at request-time. To add this functionality, update the file mysampleadapter.go to look like the following. Note the bold text shows the newly added code.

//go:generate $GOPATH/src/istio.io/istio/bin/mixer_codegen.sh -f mixer/adapter/mysampleadapter/config/config.proto
package mysampleadapter

import (
	"context"

	// "github.com/gogo/protobuf/types"
	"fmt"
	"os"
	"path/filepath"
	config "istio.io/istio/mixer/adapter/mysampleadapter/config"
	"istio.io/istio/mixer/pkg/adapter"
	"istio.io/istio/mixer/template/metric"
)

type (
	builder struct {
		adpCfg      *config.Params
		metricTypes map[string]*metric.Type
	}
	handler struct {
		f           *os.File
		metricTypes map[string]*metric.Type
		env         adapter.Env
	}
)

// ensure types implement the requisite interfaces
var _ metric.HandlerBuilder = &builder{}
var _ metric.Handler = &handler{}

///////////////// Configuration-time Methods ///////////////

// adapter.HandlerBuilder#Build
func (b *builder) Build(ctx context.Context, env adapter.Env) (adapter.Handler, error) {
	var err error
	var file *os.File
	file, err = os.Create(b.adpCfg.FilePath)
	return &handler{f: file, metricTypes: b.metricTypes, env: env}, err

}

// adapter.HandlerBuilder#SetAdapterConfig
func (b *builder) SetAdapterConfig(cfg adapter.Config) {
	b.adpCfg = cfg.(*config.Params)
}

// adapter.HandlerBuilder#Validate
func (b *builder) Validate() (ce *adapter.ConfigErrors) {
	// Check if the path is valid
	if _, err := filepath.Abs(b.adpCfg.FilePath); err != nil {
		ce = ce.Append("file_path", err)
	}
	return
}

// metric.HandlerBuilder#SetMetricTypes
func (b *builder) SetMetricTypes(types map[string]*metric.Type) {
	b.metricTypes = types
}

////////////////// Request-time Methods //////////////////////////
// metric.Handler#HandleMetric
func (h *handler) HandleMetric(ctx context.Context, insts []*metric.Instance) error {
	for _, inst := range insts {
		if _, ok := h.metricTypes[inst.Name]; !ok {
			h.env.Logger().Errorf("Cannot find Type for instance %s", inst.Name)
			continue
		}
		h.f.WriteString(fmt.Sprintf(`HandleMetric invoke for :
		Instance Name  :'%s'
		Instance Value : %v,
		Type           : %v`, inst.Name, *inst, *h.metricTypes[inst.Name]))
	}
	return nil
}

// adapter.Handler#Close
func (h *handler) Close() error {
	return h.f.Close()
}

////////////////// Bootstrap //////////////////////////
// GetInfo returns the adapter.Info specific to this adapter.
func GetInfo() adapter.Info {
	return adapter.Info{
		Name:        "mysampleadapter",
		Description: "Logs the metric calls into a file",
		SupportedTemplates: []string{
			metric.TemplateName,
		},
		NewBuilder:    func() adapter.HandlerBuilder { return &builder{} },
		DefaultConfig: &config.Params{},
	}
}

Just to ensure everything is good, let's build the code

go build ./...

This concludes the implementation part of the adapter code. Next steps show how to plug an adapter into a build of Mixer and to verify your code's behavior.

Step 5: Plug adapter into the Mixer.

Update the $MIXER_REPO/adapter/inventory.yaml file to add the new 'mysampleadapter' into the Mixer's adapter inventory.

Add the lines in bold to the existing file. The yaml should look like the following

...
prometheus: "istio.io/istio/mixer/adapter/prometheus"
servicecontrol: "istio.io/istio/mixer/adapter/servicecontrol"
stackdriver: "istio.io/istio/mixer/adapter/stackdriver"
statsd: "istio.io/istio/mixer/adapter/statsd"
stdio: "istio.io/istio/mixer/adapter/stdio"
mysampleadapter: "istio.io/istio/mixer/adapter/mysampleadapter"
...

Now, regenerate the inventory code by running go generate inside the $MIXER_REPO/adapter directory

go generate $MIXER_REPO/adapter/doc.go

Now your adapter is plugged into Mixer and ready to receive data.

Step 6: Write sample operator config

To see if your adapter works, we will need a sample operator configuration. So, let's write a simple operator configuration that we will give to Mixer for it to dispatch data to your sample adapter. We will need instance, handler and rule configuration to be passed to the Mixers configuration server.

Create a sample operator config file with name mysampleadapter.yaml inside the $MIXER_REPO/adapter/mysampleadapter/testdata directory with the following content:

Add the following content to the file $MIXER_REPO/adapter/mysampleadapter/testdata/mysampleadapter.yaml.

# instance configuration for template 'metric'
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
 name: requestcount
 namespace: istio-system
spec:
 value: "1"
 dimensions:
   target: destination.service | "unknown"

---
# handler configuration for adapter 'metric'
apiVersion: "config.istio.io/v1alpha2"
kind: mysampleadapter
metadata:
 name: hndlrTest
 namespace: istio-system
spec:
 file_path: "out.txt"
---
# rule to dispatch to your handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
 name: mysamplerule
 namespace: istio-system
spec:
 match: "true"
 actions:
 - handler: hndlrTest.mysampleadapter
   instances:
   - requestcount.metric

Let's also copy the attribute vocabulary into our testdata directory.

cp $MIXER_REPO/testdata/config/attributes.yaml $MIXER_REPO/adapter/mysampleadapter/testdata

Step 7: Start Mixer and validate the adapter.

Start the mixer pointing it to the testdata operator configuration

pushd $ISTIO/istio && make mixs
// locate mixs binary, should be $GOPATH/out/linux_amd64/release/mixs on linux os and
// $GOPATH/out/darwin_amd64/release/mixs on mac os.
// Choose command below according to your os:
$GOPATH/out/linux_amd64/release/mixs server --configStoreURL=fs://$(pwd)/mixer/adapter/mysampleadapter/testdata

The terminal will have the following output and will be blocked waiting to serve requests

..
..
Mixer started with
MaxMessageSize: 1048576
MaxConcurrentStreams: 1024
APIWorkerPoolSize: 1024
AdapterWorkerPoolSize: 1024
ExpressionEvalCacheSize: 1024
APIPort: 9091
MonitoringPort: 9093
SingleThreaded: false
ConfigStore2URL: fs:///usr/local/google/home/guptasu/go/src/istio.io/istio/mixer/adapter/mysampleadapter/sampleoperatorconfig
ConfigDefaultNamespace: istio-system
ConfigIdentityAttribute: destination.service
ConfigIdentityAttributeDomain: svc.cluster.local
LoggingOptions: log.Options{OutputPaths:[]string{"stdout"}, ErrorOutputPaths:[]string{"stderr"}, RotateOutputPath:"", RotationMaxSize:104857600, RotationMaxAge:30, RotationMaxBackups:1000, JSONEncoding:false, IncludeCallerSourceLocation:false, stackTraceLevel:"none", outputLevel:"info"}
TracingOptions: tracing.Options{ZipkinURL:"", JaegerURL:"", LogTraceSpans:false}

2018-01-06T01:43:12.305995Z	info	template Kind: kubernetesenv, &InstanceParam{SourceUid:,SourceIp:,DestinationUid:,DestinationIp:,OriginUid:,OriginIp:,AttributeBindings:map[string]string{},}
...

Now let's call 'report' using mixer client. This step should cause the mixer server to call your sample adapter with instance objects constructed using the operator configuration.

Start a new terminal window and make sure $ISTIO is set to $GOPATH/src/istio.io. Example export ISTIO=$GOPATH/src/istio.io eg. ~/go/src/istio.io. Also, MIXER_REPO variable to the path where the mixer repository is on the local machine. Example export MIXER_REPO=$GOPATH/src/istio.io/istio/mixer

In the new window call the following

pushd $ISTIO/istio && make mixc && mixc report -s destination.service="svc.cluster.local" -t request.time="2017-02-01T10:12:14Z"

Inspect the out.text file that your adapter would have printed. If you have followed the above steps, then the out.txt should be in your directory $MIXER_REPO

tail $ISTIO/istio/out.txt

You should see something like:

HandleMetric invoke for
       Instance Name  : requestcount.metric.istio-system
       Instance Value : {requestcount.metric.istio-system 1 map[response_code:200 service:unknown source:unknown target:unknown version:unknown method:unknown] UNSPECIFIED map[]}
       Type           : {INT64 map[response_code:INT64 service:STRING source:STRING target:STRING version:STRING method:STRING] map[]}

You can even try passing other attributes to mixer server and inspect your out.txt file to see how the data passed to the adapter changes. For example

pushd $ISTIO/istio && make mixc && mixc report -s="destination.service=svc.cluster.local,destination.service=mySrvc" -i="response.code=400" --stringmap_attributes="destination.labels=app:dummyapp"

If you have reached this far, congratulate yourself !!. You have successfully created a Mixer adapter. You can close (cltr + c) on your terminal that was running mixer server to shut it down.

Step 8: Write test and validate your adapter.

The above steps 7 (start mixer server and validate ..) were mainly to test your adapter code. You can achieve the same thing by writing a simple test that uses the Mixer's 'testenv' package to start a inproc Mixer and make calls to it via mixer client. For complete reference details on how to test adapter code check out Test an adapter

Add a test file for your adapter code

touch $MIXER_REPO/adapter/mysampleadapter/mysampleadapter_test.go

Add the following content to that file

// Copyright 2018 Istio Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package mysampleadapter

import (
	"fmt"
	"testing"

	"io/ioutil"
	"istio.io/istio/mixer/pkg/adapter"
	adapter_integration "istio.io/istio/mixer/pkg/adapter/test"
	"os"
	"strings"
)

func TestReport(t *testing.T) {
	operatorCfgBytes, err := ioutil.ReadFile("testdata/mysampleadapter.yaml")
	if err != nil {
		t.Fatalf("could not read file: %v", err)
	}
	operatorCfg := string(operatorCfgBytes)

	defer func() {
		if removeErr := os.Remove("out.txt"); removeErr != nil {
			t.Logf("Could not remove temporary file %s: %v", "out.txt", removeErr)
		}
	}()

	adapter_integration.RunTest(
		t,
		func() adapter.Info {
			return GetInfo()
		},
		adapter_integration.Scenario{
			ParallelCalls: []adapter_integration.Call{
				{
					CallKind: adapter_integration.REPORT,
				},
			},

			GetState: func(ctx interface{}) (interface{}, error) {
				// validate if the content of "out.txt" is as expected
				bytes, err := ioutil.ReadFile("out.txt")
				if err != nil {
					return nil, err
				}
				s := string(bytes)
				wantStr := `
HandleMetric invoke for :
				Instance Name  :'requestcount.metric.istio-system'
				Instance Value : {requestcount.metric.istio-system 1 map[target:svc.cluster.local]  map[]},
				Type           : {INT64 map[target:STRING] map[]}
`
				if normalize(s) != normalize(wantStr) {
					return nil, fmt.Errorf("got adapters state as : '%s'; want '%s'", s, wantStr)
				}
				return nil, nil
			},

			GetConfig: func(ctx interface{}) ([]string, error) {
				return []string{
					operatorCfg,
				}, nil
			},

			Want: `
            		{
		 "AdapterState": null,
		 "Returns": [
		  {
		   "Check": {
		    "Status": {},
		    "ValidDuration": 0,
		    "ValidUseCount": 0
		   },
		   "Quota": null,
		   "Error": null
		  }
		 ]
		}`,
		},
	)
}

func normalize(s string) string {
	s = strings.TrimSpace(s)
	s = strings.Replace(s, "\t", "", -1)
	s = strings.Replace(s, "\n", "", -1)
	s = strings.Replace(s, " ", "", -1)
	return s
}

Now run the test

cd $MIXER_REPO/adapter/mysampleadapter && go build ./... && go test *.go

Step 9: Cleanup

Delete the adapter/mysampleadapter directory, undo the edits made inside the adapter/inventory.yaml and adapter/inventory.gen.go files.

Step 10: Next

Next step is to build your own adapter and integrate with Mixer. Refer to Developer's guide for necessary details.