From 98a0bad1071074f0c04c35ff209f0333c5d7274b Mon Sep 17 00:00:00 2001 From: Tomofumi Hayashi Date: Fri, 3 Feb 2023 19:00:27 +0900 Subject: [PATCH] `OCPBUGS-6917: Upstream sync 0c6df81` (#22) * Add pod-iptables option to store pod iptables This change introduces pod-iptables option to store iptables-rules in pod's network namespace. This helps administrator/engineer to troubleshooting. * Fix owners file * Update CI pipeline * Add label to Dockerfile * Update github action to simplify * Use GITHUB_TOKEN for push packages * Update slack URL in README * fix workflows * Fix some timing issue and change memory limit * Add namespace check between pod and multi-networkpolicy * Use TCP as default for Port.Protocol Add ginkgo test to the suite with only default values. Add `renderProtocol` function with fallback logic. Signed-off-by: Andrea Panattoni * Fix to work namespacveSelector policy, without labelSelector * Support for `NamespaceSelector` (#16) * Add test case for namespace selector The case is about having two namespaces with pods and net-attach-def and a multi networkpolicy that goes through namespace borders. Signed-off-by: Andrea Panattoni * Add test case with net-attach-def in other ns Signed-off-by: Andrea Panattoni * Improve logging in server.go (#19) * Add object information to update events This should make it clearer what k8s object the daemon is working on. Increase verbosity threshlod for invoke handlers logs. Signed-off-by: Andrea Panattoni * Improve error logging Signed-off-by: Andrea Panattoni * Add IPv6 support in TODO list * Set specific version for `revive` tool (#20) "go getting" github.com/mgechev/revive can lead to unreproducible builds, as it download the latest "dev" version. Stick to the latest (v1.2.1) version. Signed-off-by: Andrea Panattoni * Log filter rules (#23) * Log filter rules Logging iptables rules before applying them can be useful to debug complex scenarios. Setting verbosity level to 6 as they can be quite cumbersome. Signed-off-by: Andrea Panattoni * Clean up logging code Signed-off-by: Andrea Panattoni * Refine policy generation routine to support multiple policies This change refines policy rule generation to introduce conntrack and support multiple policies in a pod. Fix #17 and #18 * Fix capabilities (#25) fix #24 * Update github action to fit to latest golang * Remove docker from support runtime due to obsolated * Bump github.com/containernetworking/cni from 0.7.1 to 0.8.1 (#31) Bumps [github.com/containernetworking/cni](https://github.com/containernetworking/cni) from 0.7.1 to 0.8.1. - [Release notes](https://github.com/containernetworking/cni/releases) - [Commits](https://github.com/containernetworking/cni/compare/v0.7.1...v0.8.1) --- updated-dependencies: - dependency-name: github.com/containernetworking/cni dependency-type: direct:production ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump vendor packages. * Graceful shutdown for daemonset (#32) * Remove unused errCh `server.Run()` is not a blocking function and returns always `nil`. There is no need for a struct field channel. Signed-off-by: Andrea Panattoni * Allow stopping the server Add signal handler for SIGTERM and SIGINT to main.go. Add Stop() method to Options to forward os signals. Add a channel to stop `syncRunner` and clean iptables afterward. Signed-off-by: Andrea Panattoni Signed-off-by: Andrea Panattoni * Add sync-period option for fast sync * Remove deprecated parameters in deploy.yml * Add e2e test * e2e-test: Add script to update server image (#35) Add a script to redeploy the server in the kind cluster. It is useful to quickly test new changes without tearing down the cluster and bringing it up again. Signed-off-by: Andrea Panattoni Signed-off-by: Andrea Panattoni * Fix yaml syntax error in GH workflow (#36) Signed-off-by: Andrea Panattoni Signed-off-by: Andrea Panattoni * Add CodeQL workflow for GitHub code scanning (#38) Co-authored-by: LGTM Migrator * Add NOTICE file for Apache license 2.0 (#39) This change adds NOTICE file in repository as [1]. [1]: https://infra.apache.org/apply-license.html#new * IPv6 support in multi-networkpolicy-iptables (#40) * Support IPv6 networks (#27) Make Server generates rules for both IP family. Make iptableBuffer aware of the IP family it is managing, in order to skip wrong addresses. Add unit and e2e tests for IPv6 and dual stack networks. Remove IPv6 item from TODO Signed-off-by: Andrea Panattoni Signed-off-by: Andrea Panattoni * fix merge-conflict to rebase * Add e2e ipv6 ingress tests * IPv6 fix for NDP and DHCPv6 (#37) * Add Requirements section to README Signed-off-by: Andrea Panattoni * Allow ipv6 Neighbor Discovery Protocol NDP leverages icmpv6 packets to discover hosts IPv6 addresses. This kind of packet must be allowed between hosts, otherwise some policy-allowed traffic may get blocked. Adjust unit tests expected output strings. See https://www.rfc-editor.org/rfc/rfc2373 Signed-off-by: Andrea Panattoni * Allow DHCPv6 traffic Signed-off-by: Andrea Panattoni * Refine icmp/dhcpv6 code Signed-off-by: Andrea Panattoni Co-authored-by: Tomofumi Hayashi * Use string instead of byte in unit-test cases In real code, use bytes for performance, however, we don't care about performance for unit-test, hence change bytes to string for ease of troubleshooting. * Make INGRESS/EGRESS-COMMON configurable by command line option This change makes MULTI-{INGRESS,EGRESS}-COMMON chain configurable to provide a way to support various v4/v6 network. * Fix CodeQL warnings * Update docs/configurations.md Co-authored-by: Nikhil Simha * Update docs/configurations.md Co-authored-by: Nikhil Simha * Wait for sync between policy/iptables in e2e tests --------- Signed-off-by: Andrea Panattoni Co-authored-by: Andrea Panattoni Co-authored-by: Nikhil Simha * Fix github action * Avoid using cri-api `v1alpha2` (#43) As of v1.26.0 kubernetes removed support for api cri-api v1alpha2 https://github.com/kubernetes/kubernetes/pull/110618 Signed-off-by: Andrea Panattoni --------- Signed-off-by: Andrea Panattoni Signed-off-by: dependabot[bot] Co-authored-by: Doug Smith Co-authored-by: Andrea Panattoni Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: lgtm-com[bot] <43144390+lgtm-com[bot]@users.noreply.github.com> Co-authored-by: LGTM Migrator Co-authored-by: Nikhil Simha --- .github/workflows/build-images.yml | 17 +- .github/workflows/build.yml | 60 +- .github/workflows/codeql.yml | 41 + .github/workflows/kind-e2e.yml | 35 + .github/workflows/push-master.yml | 48 +- .github/workflows/push-release.yml | 64 +- .github/workflows/test.yml | 47 + .gitignore | 2 +- Dockerfile | 1 + NOTICE | 1 + README.md | 16 +- cmd/multi-networkpolicy-iptables/main.go | 4 + deploy.yml | 62 +- docs/configurations.md | 51 + e2e/Dockerfile | 5 + e2e/README.md | 30 + e2e/cni-install.yml | 64 + e2e/get_tools.sh | 13 + e2e/multi-network-policy-iptables-e2e.yml | 178 +++ e2e/setup_cluster.sh | 89 ++ e2e/tests/common.bash | 21 + e2e/tests/simple-v4-egress.bats | 99 ++ e2e/tests/simple-v4-egress.yml | 104 ++ e2e/tests/simple-v4-ingress.bats | 99 ++ e2e/tests/simple-v4-ingress.yml | 104 ++ e2e/tests/simple-v6-ingress.bats | 99 ++ e2e/tests/simple-v6-ingress.yml | 104 ++ e2e/tests/stacked.bats | 55 + e2e/tests/stacked.yml | 143 ++ e2e/update_image_on_cluster.sh | 13 + pkg/controllers/doc.go | 16 + pkg/controllers/pod.go | 2 +- pkg/server/doc.go | 16 + pkg/server/options.go | 127 +- pkg/server/policyrules.go | 228 +++- pkg/server/policyrules_test.go | 1163 +++++++++++++++-- pkg/server/server.go | 98 +- pkg/utils/doc.go | 16 + .../apis/runtime/{v1alpha2 => v1}/api.pb.go | 1077 ++++++++------- .../apis/runtime/{v1alpha2 => v1}/api.proto | 28 +- .../runtime/{v1alpha2 => v1}/constants.go | 4 +- vendor/modules.txt | 2 +- 42 files changed, 3588 insertions(+), 858 deletions(-) create mode 100644 .github/workflows/codeql.yml create mode 100644 .github/workflows/kind-e2e.yml create mode 100644 .github/workflows/test.yml create mode 100644 NOTICE create mode 100644 docs/configurations.md create mode 100644 e2e/Dockerfile create mode 100644 e2e/README.md create mode 100644 e2e/cni-install.yml create mode 100755 e2e/get_tools.sh create mode 100644 e2e/multi-network-policy-iptables-e2e.yml create mode 100755 e2e/setup_cluster.sh create mode 100644 e2e/tests/common.bash create mode 100755 e2e/tests/simple-v4-egress.bats create mode 100644 e2e/tests/simple-v4-egress.yml create mode 100755 e2e/tests/simple-v4-ingress.bats create mode 100644 e2e/tests/simple-v4-ingress.yml create mode 100755 e2e/tests/simple-v6-ingress.bats create mode 100644 e2e/tests/simple-v6-ingress.yml create mode 100755 e2e/tests/stacked.bats create mode 100644 e2e/tests/stacked.yml create mode 100755 e2e/update_image_on_cluster.sh create mode 100644 pkg/controllers/doc.go create mode 100644 pkg/server/doc.go create mode 100644 pkg/utils/doc.go rename vendor/k8s.io/cri-api/pkg/apis/runtime/{v1alpha2 => v1}/api.pb.go (94%) rename vendor/k8s.io/cri-api/pkg/apis/runtime/{v1alpha2 => v1}/api.proto (98%) rename vendor/k8s.io/cri-api/pkg/apis/runtime/{v1alpha2 => v1}/constants.go (97%) diff --git a/.github/workflows/build-images.yml b/.github/workflows/build-images.yml index 7c515b84..9500fa29 100644 --- a/.github/workflows/build-images.yml +++ b/.github/workflows/build-images.yml @@ -1,16 +1,19 @@ name: build-images on: [pull_request] jobs: - build: - name: Docker build + build-images: runs-on: ubuntu-latest env: - DOCKER_CLI_EXPERIMENTAL: enabled - GO111MODULE: on - REPOSITORY: nfvpe/multi-networkpolicy-iptables + REPOSITORY: ghcr.io/${{ github.repository }} steps: - name: Check out code into the Go module directory uses: actions/checkout@v2 - - name: Build latest-amd64 - run: docker build -t ${REPOSITORY}:latest-amd64 . + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v1 + + - name: Build container image + uses: docker/build-push-action@v2 + with: + push: false + tags: gchr.io/${{ github.repository }}:latest-amd64 diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 978301fb..230fd8d7 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -1,53 +1,27 @@ -name: build on: [push, pull_request] +name: build jobs: - build: - name: Build and test + strategy: + matrix: + go-version: [1.17.x, 1.18.x] + goarch: [amd64, arm64] runs-on: ubuntu-latest if: > (( github.event.pull_request.head.repo.owner.login != github.event.pull_request.base.repo.owner.login ) && github.event_name == 'pull_request' ) || (github.event_name == 'push' && github.event.commits != '[]' ) env: GO111MODULE: on - TARGET: amd64 steps: - - name: Set up Go 1.13 - uses: actions/setup-go@v1 - with: - go-version: 1.13 - id: go - - - name: Check out code into the Go module directory - uses: actions/checkout@v2 - - - name: install goberalls - run: go get github.com/mattn/goveralls - - - name: install golint - run: GOBIN=$(pwd)/bin go get golang.org/x/lint/golint - - - name: golint - run: ./bin/golint ./... | grep -v vendor | grep -v ALL_CAPS | xargs -r false - - - name: gofmt - run: go fmt ./... - - - name: go vet - run: go vet ./... - - - name: go test - run: go test ./... - - - name: Build - run: GOARCH="${TARGET}" go build ./cmd/multi-networkpolicy-iptables/ - -# TBD: -# - name: Go test -# run: sudo ./test.sh -# -# - name: goveralls -# uses: shogo82148/actions-goveralls@v1 -# with: -# path-to-profile: coverage.out - + - name: Install Go + uses: actions/setup-go@v2 + with: + go-version: ${{ matrix.go-version }} + + - name: Check out code into the Go module directory + uses: actions/checkout@v2 + + - name: Build + env: + GOARCH: ${{ matrix.goarch }} + run: go build -o multi-networkpolicy-iptables_${{ matrix.goarch }} ./cmd/multi-networkpolicy-iptables/ diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 00000000..abb3cb79 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,41 @@ +name: "CodeQL" + +on: + push: + branches: [ "master" ] + pull_request: + branches: [ "master" ] + schedule: + - cron: "58 6 * * 4" + +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + language: [ go ] + + steps: + - name: Checkout + uses: actions/checkout@v3 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v2 + with: + languages: ${{ matrix.language }} + queries: +security-and-quality + + - name: Autobuild + uses: github/codeql-action/autobuild@v2 + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v2 + with: + category: "/language:${{ matrix.language }}" diff --git a/.github/workflows/kind-e2e.yml b/.github/workflows/kind-e2e.yml new file mode 100644 index 00000000..34b40968 --- /dev/null +++ b/.github/workflows/kind-e2e.yml @@ -0,0 +1,35 @@ +name: e2e-kind +on: [push, pull_request] +jobs: + e2e-kind: + runs-on: ubuntu-latest + if: > + (( github.event.pull_request.head.repo.owner.login != github.event.pull_request.base.repo.owner.login ) && + github.event_name == 'pull_request' ) || (github.event_name == 'push' && github.event.commits != '[]' ) + steps: + - name: Install bats + run: sudo apt install bats + + - name: Check out code into the Go module directory + uses: actions/checkout@v2 + + - name: Setup registry + run: docker run -d --restart=always -p "5000:5000" --name "kind-registry" registry:2 + + - name: Get tools + working-directory: ./e2e + run: ./get_tools.sh + + - name: Setup cluster + working-directory: ./e2e + run: ./setup_cluster.sh + + - name: "Test: simple" + working-directory: ./e2e + run: | + export TERM=dumb + bats ./tests/simple-v4-ingress.bats + bats ./tests/simple-v4-egress.bats + bats ./tests/stacked.bats + # this should be validated once the v6 issue is fixed. + #bats ./v6work/simple-v6-ingress.bats diff --git a/.github/workflows/push-master.yml b/.github/workflows/push-master.yml index 563b4c53..e285daa9 100644 --- a/.github/workflows/push-master.yml +++ b/.github/workflows/push-master.yml @@ -3,43 +3,47 @@ on: push: branches: - master +env: + image-push-owner: 'k8snetworkplumbingwg' jobs: - build: - name: Docker build + push-master: runs-on: ubuntu-latest env: - DOCKER_CLI_EXPERIMENTAL: enabled GO111MODULE: on - TARGET: amd64 - REPOSITORY: nfvpe/multi-networkpolicy-iptables - REPOSITORY_USER: nfvperobot + REPOSITORY: ghcr.io/${{ github.repository }} steps: - name: Check out code into the Go module directory - uses: actions/checkout@v2 + uses: actions/checkout@v3 - - name: Build latest-amd64 - run: docker build -t ${REPOSITORY}:latest-amd64 . + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 - - name: Tag snapshot - run: | - docker tag ${REPOSITORY}:latest-amd64 ${REPOSITORY}:snapshot-amd64 - - - name: Login to registry - if: github.repository == 'k8snetworkplumbingwg/multi-networkpolicy-iptables' - run: docker login -u ${REPOSITORY_USER} -p ${{ secrets.REPOSITORY_PASS }} + - name: Login to GitHub Container Registry + if: github.repository_owner == 'k8snetworkplumbingwg' + uses: docker/login-action@v2 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} - - name: Push latest/snapshot images - if: github.repository == 'k8snetworkplumbingwg/multi-networkpolicy-iptables' - run: | - docker push ${REPOSITORY}:latest-amd64 - docker push ${REPOSITORY}:snapshot-amd64 + - name: Push to GitHub Container Registry + if: github.repository_owner == 'k8snetworkplumbingwg' + uses: docker/build-push-action@v3 + with: + push: true + platform: linux/arm64 + tags: | + ghcr.io/${{ github.repository }}:latest-amd64 + ghcr.io/${{ github.repository }}:snapshot-amd64 - name: Create manifest for multi-arch images - if: github.repository == 'k8snetworkplumbingwg/multi-networkpolicy-iptables' + if: github.repository_owner == 'k8snetworkplumbingwg' run: | + docker pull ${REPOSITORY}:snapshot-amd64 docker manifest create ${REPOSITORY}:snapshot ${REPOSITORY}:snapshot-amd64 docker manifest annotate ${REPOSITORY}:snapshot ${REPOSITORY}:snapshot-amd64 --arch amd64 docker manifest push ${REPOSITORY}:snapshot + docker pull ${REPOSITORY}:latest-amd64 docker manifest create ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 --arch amd64 docker manifest push ${REPOSITORY}:latest diff --git a/.github/workflows/push-release.yml b/.github/workflows/push-release.yml index f7748307..7fd0e458 100644 --- a/.github/workflows/push-release.yml +++ b/.github/workflows/push-release.yml @@ -1,51 +1,53 @@ -name: push-stable +name: push-release on: push: tags: - v* jobs: - build: + push-release: name: Docker build runs-on: ubuntu-latest env: - DOCKER_CLI_EXPERIMENTAL: enabled GO111MODULE: on - TARGET: amd64 - REPOSITORY: nfvpe/multi-networkpolicy-iptables - REPOSITORY_USER: nfvperobot + REPOSITORY: ghcr.io/${{ github.repository }} steps: - name: Check out code into the Go module directory uses: actions/checkout@v2 - - name: Build latest-amd64 - run: docker build -t ${REPOSITORY}:latest-amd64 . + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v1 - - name: Tag stable - run: | - docker tag ${REPOSITORY}:latest-amd64 ${REPOSITORY}:stable-amd64 - docker tag ${REPOSITORY}:latest-amd64 ${REPOSITORY}:${GITHUB_REF##*/}-amd64 + - name: Login to GitHub Container Registry + if: github.repository_owner == 'k8snetworkplumbingwg' + uses: docker/login-action@v1 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} - - name: Login to registry - if: github.repository == 'k8snetworkplumbingwg/multi-networkpolicy-iptables' - run: docker login -u ${REPOSITORY_USER} -p ${{ secrets. REPOSITORY_PASS }} + - name: Docker meta + id: docker_meta + uses: crazy-max/ghaction-docker-meta@v1 + with: + images: ${{ env.REPOSITORY }} + tag-latest: false - - name: Push latest/snapshot images - if: github.repository == 'k8snetworkplumbingwg/multi-networkpolicy-iptables' - run: | - docker push ${REPOSITORY}:latest-amd64 - docker push ${REPOSITORY}:stable-amd64 - docker push ${REPOSITORY}:${GITHUB_REF##*/}-amd64 + - name: Push to GitHub Container Registry + if: github.repository_owner == 'k8snetworkplumbingwg' + uses: docker/build-push-action@v2 + with: + push: true + tags: | + ghcr.io/${{ github.repository }}:stable-amd64 + ${{ steps.docker_meta.outputs.tags }}-amd64 - name: Create manifest for multi-arch images - if: github.repository == 'k8snetworkplumbingwg/multi-networkpolicy-iptables' + if: github.repository_owner == 'k8snetworkplumbingwg' run: | - docker manifest create ${REPOSITORY}:stable ${REPOSITORY}:stable-amd64 - docker manifest annotate ${REPOSITORY}:stable ${REPOSITORY}:stable-amd64 --arch amd64 - docker manifest push ${REPOSITORY}:stable - docker manifest create ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 - docker manifest annotate ${REPOSITORY}:latest ${REPOSITORY}:latest-amd64 --arch amd64 - docker manifest push ${REPOSITORY}:latest - docker manifest create ${REPOSITORY}:${GITHUB_REF##*/} ${REPOSITORY}:${GITHUB_REF##*/}-amd64 - docker manifest annotate ${REPOSITORY}:${GITHUB_REF##*/} ${REPOSITORY}:${GITHUB_REF##*/}-amd64 --arch amd64 - docker manifest push ${REPOSITORY}:${GITHUB_REF##*/} + docker manifest create ${{ env.REPOSITORY }}:stable ${{ env.REPOSITORY }}:stable-amd64 + docker manifest annotate ${{ env.REPOSITORY }}:stable ${{ env.REPOSITORY }}:stable-amd64 --arch amd64 + docker manifest push ${{ env.REPOSITORY }}:stable + docker manifest create ${{ steps.docker_meta.outputs.tags }} ${{ steps.docker_meta.outputs.tags }}-amd64 + docker manifest annotate ${{ steps.docker_meta.outputs.tags }} ${{ steps.docker_meta.outputs.tags }}-amd64 --arch amd64 + docker manifest push ${{ steps.docker_meta.outputs.tags }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 00000000..f00ca874 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,47 @@ +on: [push, pull_request] +name: test +jobs: + test: + strategy: + matrix: + go-version: [1.18.x] + os: [ubuntu-latest] #, macos-latest, windows-latest] + runs-on: ${{ matrix.os }} + steps: + - name: Install Go + uses: actions/setup-go@v2 + with: + go-version: ${{ matrix.go-version }} + + - name: Checkout code + uses: actions/checkout@v2 + + - name: Run Revive Action by pulling pre-built image + uses: docker://morphy/revive-action:v2 + with: + exclude: "./vendor/..." + + - name: Run go fmt + run: diff -u <(echo -n) <(gofmt -d -s ./cmd/ ./pkg/) + + - name: Run go vet + run: go vet ./... + + - name: Test + run: sudo go test -v -coverprofile=profile.cov ./... + + - name: Send coverage + uses: shogo82148/actions-goveralls@v1 + with: + path-to-profile: profile.cov + flag-name: Go-${{ matrix.go }} + parallel: true + + # notifies that all test jobs are finished. + finish: + needs: test + runs-on: ubuntu-latest + steps: + - uses: shogo82148/actions-goveralls@v1 + with: + parallel-finished: true diff --git a/.gitignore b/.gitignore index 75a539e9..5102b9a3 100644 --- a/.gitignore +++ b/.gitignore @@ -3,7 +3,7 @@ bin/ e2e/bin/ # binary at the top -./multi-networkpolicy-iptables +/multi-networkpolicy-iptables # GOPATH created by the build script gopath/ diff --git a/Dockerfile b/Dockerfile index 4975f7e0..fea05eba 100644 --- a/Dockerfile +++ b/Dockerfile @@ -8,6 +8,7 @@ RUN cd /usr/src/multi-networkpolicy-iptables && \ go build ./cmd/multi-networkpolicy-iptables/ FROM centos:centos7 +LABEL org.opencontainers.image.source https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables RUN yum install -y iptables-utils COPY --from=build /usr/src/multi-networkpolicy-iptables/multi-networkpolicy-iptables /usr/bin WORKDIR /usr/bin diff --git a/NOTICE b/NOTICE new file mode 100644 index 00000000..dd3fc395 --- /dev/null +++ b/NOTICE @@ -0,0 +1 @@ +Copyright 2020 Kubernetes Network Plumbing Working Group diff --git a/README.md b/README.md index 19b97ba0..1b1b49fb 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ # multi-networkpolicy-iptables +[![build](https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables/actions/workflows/build.yml/badge.svg)](https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables/actions/workflows/build.yml)[![test](https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables/actions/workflows/test.yml/badge.svg)](https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables/actions/workflows/test.yml) [multi-networkpolicy](https://github.com/k8snetworkplumbingwg/multi-networkpolicy) implementation with iptables @@ -34,6 +35,19 @@ serviceaccount/multi-networkpolicy created daemonset.apps/multi-networkpolicy-ds-amd64 created ``` +## Requirements + +This project leverages `iptables` and `ip6tables` commands to do its work. Hence, `ip_tables` and `ip6_tables` kernel modules +need to be loaded on the container host: + +``` +# modprobe ip_tables ip6_tables +``` + +## Configurations + +See [Configurations](docs/configurations.md). + ## Demo (TBD) @@ -50,4 +64,4 @@ MultiNetworkPolicy creates DaemonSet and it runs `multi-networkpolicy-iptables` ## Contact Us -For any questions about multi-networkpolicy-iptables, feel free to ask a question in #k8s-npwg-discussion in the [Intel-Corp Slack](https://intel-corp.herokuapp.com/), or open up a GitHub issue. +For any questions about Multus CNI, feel free to ask a question in #general in the [NPWG Slack](https://npwg-team.slack.com/), or open up a GitHub issue. Request an invite to NPWG slack [here](https://intel-corp.herokuapp.com/). diff --git a/cmd/multi-networkpolicy-iptables/main.go b/cmd/multi-networkpolicy-iptables/main.go index 84feb19d..dd00cd9b 100644 --- a/cmd/multi-networkpolicy-iptables/main.go +++ b/cmd/multi-networkpolicy-iptables/main.go @@ -14,6 +14,10 @@ See the License for the specific language governing permissions and limitations under the License. */ +// This is a Kubernetes controller to generate iptables rules for +// multi-networkpolicy. +// It reads multiNetworkpolicy object and generates iptables rules into +// container network namespaces. package main import ( diff --git a/deploy.yml b/deploy.yml index 05a2985c..6502d087 100644 --- a/deploy.yml +++ b/deploy.yml @@ -54,6 +54,42 @@ metadata: name: multi-networkpolicy namespace: kube-system --- +kind: ConfigMap +apiVersion: v1 +metadata: + name: multi-networkpolicy-custom-v4-rules + namespace: kube-system + labels: + tier: node + app: multi-networkpolicy +data: + custom-v4-rules.txt: | + # accept redirect + -p icmp --icmp-type redirect -j ACCEPT + # accept fragmentation-needed (for MTU discovery) + -p icmp --icmp-type fragmentation-needed -j ACCEPT +--- +kind: ConfigMap +apiVersion: v1 +metadata: + name: multi-networkpolicy-custom-v6-rules + namespace: kube-system + labels: + tier: node + app: multi-networkpolicy +data: + custom-v6-rules.txt: | + # accept NDP + -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT + -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT + # accept RA/RS + -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT + -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT + # accept redirect + -p icmpv6 --icmpv6-type redirect -j ACCEPT + # accept packet-too-big (for MTU discovery) + -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT +--- apiVersion: apps/v1 kind: DaemonSet metadata: @@ -85,18 +121,26 @@ spec: serviceAccountName: multi-networkpolicy containers: - name: multi-networkpolicy - # crio support requires multus:latest for now. support 3.3 or later. - image: docker.io/nfvpe/multi-networkpolicy-iptables:snapshot-amd64 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:snapshot-amd64 imagePullPolicy: Always command: ["/usr/bin/multi-networkpolicy-iptables"] args: - "--host-prefix=/host" - # uncomment this if runtime is docker - #- "--container-runtime=docker" # change this if runtime is different that crio default - "--container-runtime-endpoint=/run/crio/crio.sock" # uncomment this if you want to store iptables rules - "--pod-iptables=/var/lib/multi-networkpolicy/iptables" + # uncomment this if you need to accept link-local address traffic + #- "--allow-ipv6-src-prefix=fe80::/10" + #- "--allow-ipv6-dst-prefix=fe80::/10" + # uncomment this if you need to add custom iptables rules defined above configmap + #- "--custom-v4-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-rules.txt" + #- "--custom-v4-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-rules.txt" + #- "--custom-v6-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-rules.txt" + #- "--custom-v6-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-rules.txt" + # uncomment if you want to accept ICMP/ICMPv6 traffic + #- "--accept-icmp" + #- "--accept-icmpv6" resources: requests: cpu: "100m" @@ -113,6 +157,9 @@ spec: mountPath: /host - name: var-lib-multinetworkpolicy mountPath: /var/lib/multi-networkpolicy + - name: multi-networkpolicy-custom-rules + mountPath: /etc/multi-networkpolicy/rules + readOnly: true volumes: - name: host hostPath: @@ -120,3 +167,10 @@ spec: - name: var-lib-multinetworkpolicy hostPath: path: /var/lib/multi-networkpolicy + - name: multi-networkpolicy-custom-rules + projected: + sources: + - configMap: + name: multi-networkpolicy-custom-v4-rules + - configMap: + name: multi-networkpolicy-custom-v6-rules diff --git a/docs/configurations.md b/docs/configurations.md new file mode 100644 index 00000000..f12170d9 --- /dev/null +++ b/docs/configurations.md @@ -0,0 +1,51 @@ +## Multi-networkpolicy-iptables Configurations + + +### Command Line Options + +Most command line options have description in help, so please execute with `--help` to see the option. + +``` +$ ./multi-networkpolicy-iptables --help +``` + +### Advanced Options + +#### Add exceptional IPv6 prefix address to accept + +Some IPv6 networks may require accepting traffic from/to specific address prefixes for the network, such as multicast address (all routers multicast address, link-local address and so on). You can configure `--allow-ipv6-src-prefix` and `--allow-ipv6-dst-prefix` to specify which prefix should be accepted (even though network policy does not have it). Both options accept comma separated IPv6 prefix list. + +``` +--allow-ipv6-src-prefix=fe80::/10 +--allow-ipv6-dst-prefix=fe80::/10,ff00::/8 +``` + +#### Add custom iptables/ip6tables rules + +Some IPv4/v6 networks may require accepting some specific traffic (e.g. DHCP). You can add custom iptable rules in ingress/egress for IPv4/v6 network to accept such traffics, by + +- `--custom-v4-ingress-rule-file` +- `--custom-v4-egress-rule-file` +- `--custom-v6-ingress-rule-file` +- `--custom-v6-egress-rule-file` + +Each option takes file path for iptable rules. This file can contain + +- iptable rules (no `-A` option) +- comment (begins with '#') + +Here is the example to accept DHCPv6 packets using the options. +``` +$ cat testv6IngressRules.txt +# comment: this accepts DHCPv6 packets from link-local address +-m udp -p udp --dport 546 -d fe80::/64 -j ACCEPT + +$ cat testv6EgressRules.txt +# comment: this rules accepts DHCPv6 packet to dhcp relay agents/servers +-m udp -p udp --dport 547 -d ff02::1:2 -j ACCEPT + +$ ./multi-networkpolicy-iptables \ + (snip, some options here) \ + --custom-v6-ingress-rule-file testv6IngressRules.txt \ + --custom-v6-egress-rule-file testv6EgressRules.txt +``` diff --git a/e2e/Dockerfile b/e2e/Dockerfile new file mode 100644 index 00000000..7628ec48 --- /dev/null +++ b/e2e/Dockerfile @@ -0,0 +1,5 @@ +FROM docker.io/fedora:36 + +LABEL org.opencontainers.image.source https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables +LABEL org.opencontainers.image.base.name ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test +RUN dnf install -y nginx iptables net-tools iputils iproute tcpdump wireshark-cli nmap-ncat diff --git a/e2e/README.md b/e2e/README.md new file mode 100644 index 00000000..889f1352 --- /dev/null +++ b/e2e/README.md @@ -0,0 +1,30 @@ +## e2e test with kind + + +### How to test e2e + +This requires [Bats](https://github.com/bats-core/bats-core) for test runner. Please install bats (e.g. dnf, apt and so on). + +``` +$ git clone https://github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables +$ cd multi-networkpolicy-iptables/e2e +$ ./get_tools.sh +$ ./setup_cluster.sh +$ ./tests/simple-v4-ingress.bats +``` + +### How to teardown cluster + +``` +$ kind delete cluster +$ docker kill kind-registry +$ docker rm kind-registry +``` + +### How to deploy server image with new changes + +After making changes to the code, it is possible to update the server Daemonset image with the script: + +``` +./update_image_on_cluster.sh +``` diff --git a/e2e/cni-install.yml b/e2e/cni-install.yml new file mode 100644 index 00000000..1d57fd4d --- /dev/null +++ b/e2e/cni-install.yml @@ -0,0 +1,64 @@ +--- +kind: ConfigMap +apiVersion: v1 +metadata: + name: cni-install-sh + namespace: kube-system +data: + install_cni.sh: | + cd /tmp + wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz + cd /host/opt/cni/bin + tar xvfzp /tmp/cni-plugins-linux-amd64-v1.1.1.tgz + sleep infinite +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: install-cni-plugins + namespace: kube-system + labels: + name: cni-plugins +spec: + selector: + matchLabels: + name: cni-plugins + template: + metadata: + labels: + name: cni-plugins + spec: + hostNetwork: true + nodeSelector: + kubernetes.io/arch: amd64 + tolerations: + - operator: Exists + effect: NoSchedule + containers: + - name: install-cni-plugins + image: alpine + command: ["/bin/sh", "/scripts/install_cni.sh"] + resources: + requests: + cpu: "100m" + memory: "50Mi" + limits: + cpu: "100m" + memory: "50Mi" + securityContext: + privileged: true + volumeMounts: + - name: cni-bin + mountPath: /host/opt/cni/bin + - name: scripts + mountPath: /scripts + volumes: + - name: cni-bin + hostPath: + path: /opt/cni/bin + - name: scripts + configMap: + name: cni-install-sh + items: + - key: install_cni.sh + path: install_cni.sh diff --git a/e2e/get_tools.sh b/e2e/get_tools.sh new file mode 100755 index 00000000..d6571745 --- /dev/null +++ b/e2e/get_tools.sh @@ -0,0 +1,13 @@ +#!/bin/sh +set -o errexit + +if [ ! -d bin ]; then + mkdir bin +fi + +curl -Lo ./bin/kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.15.0/kind-$(uname)-amd64" +chmod +x ./bin/kind +curl -Lo ./bin/kubectl https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl +chmod +x ./bin/kubectl +curl -Lo ./bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 +chmod +x ./bin/jq diff --git a/e2e/multi-network-policy-iptables-e2e.yml b/e2e/multi-network-policy-iptables-e2e.yml new file mode 100644 index 00000000..dc8e4387 --- /dev/null +++ b/e2e/multi-network-policy-iptables-e2e.yml @@ -0,0 +1,178 @@ +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: multi-networkpolicy +rules: + - apiGroups: ["k8s.cni.cncf.io"] + resources: + - '*' + verbs: + - '*' + - apiGroups: + - "" + resources: + - pods + - namespaces + verbs: + - list + - watch + - get + # Watch for changes to Kubernetes NetworkPolicies. + - apiGroups: ["networking.k8s.io"] + resources: + - networkpolicies + verbs: + - watch + - list + - apiGroups: + - "" + - events.k8s.io + resources: + - events + verbs: + - create + - patch + - update +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: multi-networkpolicy +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: multi-networkpolicy +subjects: +- kind: ServiceAccount + name: multi-networkpolicy + namespace: kube-system +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: multi-networkpolicy + namespace: kube-system +--- +kind: ConfigMap +apiVersion: v1 +metadata: + name: multi-networkpolicy-custom-v4-rules + namespace: kube-system + labels: + tier: node + app: multi-networkpolicy +data: + custom-v4-rules.txt: | + # accept redirect + -p icmp --icmp-type redirect -j ACCEPT + # accept fragmentation-needed (for MTU discovery) + -p icmp --icmp-type fragmentation-needed -j ACCEPT +--- +kind: ConfigMap +apiVersion: v1 +metadata: + name: multi-networkpolicy-custom-v6-rules + namespace: kube-system + labels: + tier: node + app: multi-networkpolicy +data: + custom-v6-rules.txt: | + # accept NDP + -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT + -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT + # accept RA/RS + -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT + -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT + # accept redirect + -p icmpv6 --icmpv6-type redirect -j ACCEPT + # accept packet-too-big (for MTU discovery) + -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: multi-networkpolicy-ds-amd64 + namespace: kube-system + labels: + tier: node + app: multi-networkpolicy + name: multi-networkpolicy +spec: + selector: + matchLabels: + name: multi-networkpolicy + updateStrategy: + type: RollingUpdate + template: + metadata: + labels: + tier: node + app: multi-networkpolicy + name: multi-networkpolicy + spec: + hostNetwork: true + nodeSelector: + kubernetes.io/arch: amd64 + tolerations: + - operator: Exists + effect: NoSchedule + serviceAccountName: multi-networkpolicy + containers: + - name: multi-networkpolicy + image: localhost:5000/multus-networkpolicy-iptables:e2e + imagePullPolicy: Always + command: ["/usr/bin/multi-networkpolicy-iptables"] + args: + - "--host-prefix=/host" + # change this if runtime is different that crio default + - "--container-runtime-endpoint=/run/containerd/containerd.sock" + # uncomment this if you want to store iptables rules + - "--pod-iptables=/var/lib/multi-networkpolicy/iptables" + # (e2e test only) enshorten sync period to fast sync + - "--sync-period=1" + # uncomment this if you need to accept link-local address traffic + #- "--allow-ipv6-src-prefix=fe80::/10" + #- "--allow-ipv6-dst-prefix=fe80::/10" + # uncomment this if you need to add custom iptables rules defined above configmap + - "--custom-v4-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-rules.txt" + - "--custom-v4-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-rules.txt" + - "--custom-v6-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-rules.txt" + - "--custom-v6-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-rules.txt" + # uncomment if you want to accept ICMP/ICMPv6 traffic + #- "--accept-icmp" + #- "--accept-icmpv6" + resources: + requests: + cpu: "100m" + memory: "80Mi" + limits: + cpu: "100m" + memory: "150Mi" + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN", "NET_ADMIN"] + volumeMounts: + - name: host + mountPath: /host + - name: var-lib-multinetworkpolicy + mountPath: /var/lib/multi-networkpolicy + - name: multi-networkpolicy-custom-rules + mountPath: /etc/multi-networkpolicy/rules + readOnly: true + volumes: + - name: host + hostPath: + path: / + - name: var-lib-multinetworkpolicy + hostPath: + path: /var/lib/multi-networkpolicy + - name: multi-networkpolicy-custom-rules + projected: + sources: + - configMap: + name: multi-networkpolicy-custom-v4-rules + - configMap: + name: multi-networkpolicy-custom-v6-rules diff --git a/e2e/setup_cluster.sh b/e2e/setup_cluster.sh new file mode 100755 index 00000000..961bcd60 --- /dev/null +++ b/e2e/setup_cluster.sh @@ -0,0 +1,89 @@ +#!/bin/sh +set -o errexit + +export PATH=./bin:${PATH} + +# define the OCI binary to be used. Acceptable values are `docker`, `podman`. +# Defaults to `docker`. +OCI_BIN="${OCI_BIN:-docker}" + +kind_network='kind' +reg_name='kind-registry' +reg_port='5000' +running="$($OCI_BIN inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" +if [ "${running}" != 'true' ]; then + $OCI_BIN run -d --restart=always -p "${reg_port}:5000" --name "${reg_name}" registry:2 +fi + +$OCI_BIN build -t localhost:5000/multus-networkpolicy-iptables:e2e -f ../Dockerfile .. +$OCI_BIN push localhost:5000/multus-networkpolicy-iptables:e2e + +reg_host="${reg_name}" +echo "Registry Host: ${reg_host}" + +# deploy cluster with kind +cat < server" { + # nc should succeed from client-a to server by no policy definition for the direction + run kubectl -n test-simple-v4-egress exec pod-client-a -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-simple-v4-egress check client-b -> server" { + # nc should succeed from client-b to server by no policy definition for the direction + run kubectl -n test-simple-v4-egress exec pod-client-b -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-simple-v4-egress check server -> client-a" { + # nc should succeed from server to client-a by policy definition + run kubectl -n test-simple-v4-egress exec pod-server -- sh -c "echo x | nc -w 1 ${client_a_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-simple-v4-egress check server -> client-b" { + # nc should NOT succeed from server to client-b by policy definition + run kubectl -n test-simple-v4-egress exec pod-server -- sh -c "echo x | nc -w 1 ${client_b_net1} 5555" + [ "$status" -eq "1" ] +} + +@test "disable multi-networkpolicy and check iptables rules" { + # disable multi-networkpolicy pods by adding invalid nodeSelector + kubectl -n kube-system patch daemonsets multi-networkpolicy-ds-amd64 -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}' + # check multi-networkpolicy pod is deleted + kubectl -n kube-system wait --for=delete -l app=multi-networkpolicy pod --timeout=${kubewait_timeout} + + # check iptable rules in pod-server + run kubectl -n test-simple-v4-egress exec pod-server -it -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + + # enable multi-networkpolicy again + kubectl -n kube-system patch daemonsets multi-networkpolicy-ds-amd64 --type json -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]' + sleep 3 + kubectl -n kube-system wait --for=condition=ready -l app=multi-networkpolicy pod --timeout=${kubewait_timeout} +} + +@test "cleanup environments" { + # remove test manifests + kubectl delete -f simple-v4-egress.yml + run kubectl -n test-simple-v4-egress wait --for=delete -l app=test-simple-v4-egress pod --timeout=${kubewait_timeout} + [ "$status" -eq "0" ] + + sleep 3 + # check that no iptables files in pod-iptables + pod_name=$(kubectl -n kube-system get pod -o wide | grep 'kind-worker' | grep multi-net | cut -f 1 -d ' ') + run kubectl -n kube-system exec ${pod_name} -- \ + sh -c "find /var/lib/multi-networkpolicy/iptables/ -name '*.iptables' | wc -l" + [ "$output" = "0" ] +} diff --git a/e2e/tests/simple-v4-egress.yml b/e2e/tests/simple-v4-egress.yml new file mode 100644 index 00000000..bb18cd67 --- /dev/null +++ b/e2e/tests/simple-v4-egress.yml @@ -0,0 +1,104 @@ +--- +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + namespace: default + name: macvlan1-simple +spec: + config: '{ + "cniVersion": "0.3.1", + "name": "macvlan1-simple", + "plugins": [ + { + "type": "macvlan", + "mode": "bridge", + "ipam":{ + "type":"host-local", + "subnet":"2.2.6.0/24", + "rangeStart":"2.2.6.8", + "rangeEnd":"2.2.6.67" + } + }] + }' +--- +# namespace for MultiNetworkPolicy +apiVersion: v1 +kind: Namespace +metadata: + name: test-simple-v4-egress +--- +# Pods +apiVersion: v1 +kind: Pod +metadata: + name: pod-server + namespace: test-simple-v4-egress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v4-egress + name: pod-server +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-a + namespace: test-simple-v4-egress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v4-egress + name: pod-client-a +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-b + namespace: test-simple-v4-egress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v4-egress + name: pod-client-b +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +# MultiNetworkPolicies +# this policy accepts egress trafic from pod-client-a to pod-server +apiVersion: k8s.cni.cncf.io/v1beta1 +kind: MultiNetworkPolicy +metadata: + name: test-multinetwork-policy-simple-1 + namespace: test-simple-v4-egress + annotations: + k8s.v1.cni.cncf.io/policy-for: default/macvlan1-simple +spec: + podSelector: + matchLabels: + name: pod-server + policyTypes: + - Egress + egress: + - to: + - podSelector: + matchLabels: + name: pod-client-a diff --git a/e2e/tests/simple-v4-ingress.bats b/e2e/tests/simple-v4-ingress.bats new file mode 100755 index 00000000..1aee85fd --- /dev/null +++ b/e2e/tests/simple-v4-ingress.bats @@ -0,0 +1,99 @@ +#!/usr/bin/env bats + +# Note: +# These test cases, simple, will create simple (one policy for ingress) and test the +# traffic policying by ncat (nc) command. In addition, these cases also verifies that +# simple iptables generation check by iptables-save and pod-iptable in multi-networkpolicy pod. + +setup() { + cd $BATS_TEST_DIRNAME + load "common" + server_net1=$(get_net1_ip "test-simple-v4-ingress" "pod-server") + client_a_net1=$(get_net1_ip "test-simple-v4-ingress" "pod-client-a") + client_b_net1=$(get_net1_ip "test-simple-v4-ingress" "pod-client-b") +} + +@test "setup simple test environments" { + # create test manifests + kubectl create -f simple-v4-ingress.yml + + # verify all pods are available + run kubectl -n test-simple-v4-ingress wait --for=condition=ready -l app=test-simple-v4-ingress pod --timeout=${kubewait_timeout} + [ "$status" -eq "0" ] +} + +@test "check generated iptables rules" { + # wait for sync + sleep 3 + # check pod-server has multi-networkpolicy iptables rules for ingress + run kubectl -n test-simple-v4-ingress exec pod-server -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "0" ] + # check pod-client-a has NO multi-networkpolicy iptables rules for ingress + run kubectl -n test-simple-v4-ingress exec pod-client-a -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + # check pod-client-b has NO multi-networkpolicy iptables rules for ingress + run kubectl -n test-simple-v4-ingress exec pod-client-b -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + + # wait for sync + sleep 3 + # check that iptables files in pod-iptables + pod_name=$(kubectl -n kube-system get pod -o wide | grep 'kind-worker' | grep multi-net | cut -f 1 -d ' ') + run kubectl -n kube-system exec ${pod_name} -- \ + sh -c "find /var/lib/multi-networkpolicy/iptables/ -name '*.iptables' | wc -l" + [ "$output" = "6" ] +} + +@test "test-simple-v4-ingress check client-a -> server" { + # nc should succeed from client-a to server by policy + run kubectl -n test-simple-v4-ingress exec pod-client-a -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-simple-v4-ingress check client-b -> server" { + # nc should NOT succeed from client-b to server by policy + run kubectl -n test-simple-v4-ingress exec pod-client-b -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "1" ] +} + +@test "test-simple-v4-ingress check server -> client-a" { + # nc should succeed from server to client-a by no policy definition for direction (egress for pod-server) + run kubectl -n test-simple-v4-ingress exec pod-server -- sh -c "echo x | nc -w 1 ${client_a_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-simple-v4-ingress check server -> client-b" { + # nc should succeed from server to client-b by no policy definition for direction (egress for pod-server) + run kubectl -n test-simple-v4-ingress exec pod-server -- sh -c "echo x | nc -w 1 ${client_b_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "disable multi-networkpolicy and check iptables rules" { + # disable multi-networkpolicy pods by adding invalid nodeSelector + kubectl -n kube-system patch daemonsets multi-networkpolicy-ds-amd64 -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}' + # check multi-networkpolicy pod is deleted + kubectl -n kube-system wait --for=delete -l app=multi-networkpolicy pod --timeout=${kubewait_timeout} + + # check iptable rules in pod-server + run kubectl -n test-simple-v4-ingress exec pod-server -it -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + + # enable multi-networkpolicy again + kubectl -n kube-system patch daemonsets multi-networkpolicy-ds-amd64 --type json -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]' + sleep 3 + kubectl -n kube-system wait --for=condition=ready -l app=multi-networkpolicy pod --timeout=${kubewait_timeout} +} + +@test "cleanup environments" { + # remove test manifests + kubectl delete -f simple-v4-ingress.yml + run kubectl -n test-simple-v4-ingress wait --for=delete -l app=test-simple-v4-ingress pod --timeout=${kubewait_timeout} + [ "$status" -eq "0" ] + + sleep 3 + # check that no iptables files in pod-iptables + pod_name=$(kubectl -n kube-system get pod -o wide | grep 'kind-worker' | grep multi-net | cut -f 1 -d ' ') + run kubectl -n kube-system exec ${pod_name} -- \ + sh -c "find /var/lib/multi-networkpolicy/iptables/ -name '*.iptables' | wc -l" + [ "$output" = "0" ] +} diff --git a/e2e/tests/simple-v4-ingress.yml b/e2e/tests/simple-v4-ingress.yml new file mode 100644 index 00000000..d2218b76 --- /dev/null +++ b/e2e/tests/simple-v4-ingress.yml @@ -0,0 +1,104 @@ +--- +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + namespace: default + name: macvlan1-simple +spec: + config: '{ + "cniVersion": "0.3.1", + "name": "macvlan1-simple", + "plugins": [ + { + "type": "macvlan", + "mode": "bridge", + "ipam":{ + "type":"host-local", + "subnet":"2.2.6.0/24", + "rangeStart":"2.2.6.8", + "rangeEnd":"2.2.6.67" + } + }] + }' +--- +# namespace for MultiNetworkPolicy +apiVersion: v1 +kind: Namespace +metadata: + name: test-simple-v4-ingress +--- +# Pods +apiVersion: v1 +kind: Pod +metadata: + name: pod-server + namespace: test-simple-v4-ingress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v4-ingress + name: pod-server +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-a + namespace: test-simple-v4-ingress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v4-ingress + name: pod-client-a +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-b + namespace: test-simple-v4-ingress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v4-ingress + name: pod-client-b +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +# MultiNetworkPolicies +# this policy accepts ingress trafic from pod-client-a to pod-server +apiVersion: k8s.cni.cncf.io/v1beta1 +kind: MultiNetworkPolicy +metadata: + name: test-multinetwork-policy-simple-1 + namespace: test-simple-v4-ingress + annotations: + k8s.v1.cni.cncf.io/policy-for: default/macvlan1-simple +spec: + podSelector: + matchLabels: + name: pod-server + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + name: pod-client-a diff --git a/e2e/tests/simple-v6-ingress.bats b/e2e/tests/simple-v6-ingress.bats new file mode 100755 index 00000000..5b8efaab --- /dev/null +++ b/e2e/tests/simple-v6-ingress.bats @@ -0,0 +1,99 @@ +#!/usr/bin/env bats + +# Note: +# These test cases, simple, will create simple (one policy for ingress) and test the +# traffic policying by ncat (nc) command. In addition, these cases also verifies that +# simple ip6tables generation check by ip6tables-save and pod-iptable in multi-networkpolicy pod. + + +setup() { + cd $BATS_TEST_DIRNAME + load "common" + server_net1=$(get_net1_ip6 "test-simple-v6-ingress" "pod-server") + client_a_net1=$(get_net1_ip6 "test-simple-v6-ingress" "pod-client-a") + client_b_net1=$(get_net1_ip6 "test-simple-v6-ingress" "pod-client-b") +} + +@test "setup simple test environments" { + # create test manifests + kubectl create -f simple-v6-ingress.yml + + # verify all pods are available + run kubectl -n test-simple-v6-ingress wait --for=condition=ready -l app=test-simple-v6-ingress pod --timeout=${kubewait_timeout} + [ "$status" -eq "0" ] +} + +@test "check generated ip6tables rules" { + # wait for sync + sleep 3 + + # check pod-server has multi-networkpolicy ip6tables rules for ingress + run kubectl -n test-simple-v6-ingress exec pod-server -- sh -c "ip6tables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "0" ] + # check pod-client-a has NO multi-networkpolicy ip6tables rules for ingress + run kubectl -n test-simple-v6-ingress exec pod-client-a -- sh -c "ip6tables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + # check pod-client-b has NO multi-networkpolicy ip6tables rules for ingress + run kubectl -n test-simple-v6-ingress exec pod-client-b -- sh -c "ip6tables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + + # check that ip6tables files in pod-iptables + pod_name=$(kubectl -n kube-system get pod -o wide | grep 'kind-worker' | grep multi-net | cut -f 1 -d ' ') + run kubectl -n kube-system exec ${pod_name} -- \ + sh -c "find /var/lib/multi-networkpolicy/iptables/ -name '*.ip6tables' | wc -l" + [ "$output" = "6" ] +} + +@test "test-simple-v6-ingress check client-a -> server" { + # nc should succeed from client-a to server by policy + run kubectl -n test-simple-v6-ingress exec pod-client-a -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-simple-v6-ingress check client-b -> server" { + # nc should NOT succeed from client-b to server by policy + run kubectl -n test-simple-v6-ingress exec pod-client-b -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "1" ] +} + +@test "test-simple-v6-ingress check server -> client-a" { + # nc should succeed from server to client-a by no policy definition for direction (egress for pod-server) + run kubectl -n test-simple-v6-ingress exec pod-server -- sh -c "echo x | nc -w 1 ${client_a_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-simple-v6-ingress check server -> client-b" { + # nc should succeed from server to client-b by no policy definition for direction (egress for pod-server) + run kubectl -n test-simple-v6-ingress exec pod-server -- sh -c "echo x | nc -w 1 ${client_b_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "disable multi-networkpolicy and check ip6tables rules" { + # disable multi-networkpolicy pods by adding invalid nodeSelector + kubectl -n kube-system patch daemonsets multi-networkpolicy-ds-amd64 -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}' + # check multi-networkpolicy pod is deleted + kubectl -n kube-system wait --for=delete -l app=multi-networkpolicy pod --timeout=${kubewait_timeout} + + # check ip6table rules in pod-server + run kubectl -n test-simple-v6-ingress exec pod-server -it -- sh -c "ip6tables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + + # enable multi-networkpolicy again + kubectl -n kube-system patch daemonsets multi-networkpolicy-ds-amd64 --type json -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]' + sleep 3 + kubectl -n kube-system wait --for=condition=ready -l app=multi-networkpolicy pod --timeout=${kubewait_timeout} +} + +@test "cleanup environments" { + # remove test manifests + kubectl delete -f simple-v6-ingress.yml + run kubectl -n test-simple-v6-ingress wait --for=delete -l app=test-simple-v6-ingress pod --timeout=${kubewait_timeout} + [ "$status" -eq "0" ] + + sleep 3 + # check that no ip6tables files in pod-iptables + pod_name=$(kubectl -n kube-system get pod -o wide | grep 'kind-worker' | grep multi-net | cut -f 1 -d ' ') + run kubectl -n kube-system exec ${pod_name} -- \ + sh -c "find /var/lib/multi-networkpolicy/iptables/ -name '*.ip6tables' | wc -l" + [ "$output" = "0" ] +} diff --git a/e2e/tests/simple-v6-ingress.yml b/e2e/tests/simple-v6-ingress.yml new file mode 100644 index 00000000..69bc0f48 --- /dev/null +++ b/e2e/tests/simple-v6-ingress.yml @@ -0,0 +1,104 @@ +--- +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + namespace: default + name: macvlan1-simple +spec: + config: '{ + "cniVersion": "0.3.1", + "name": "macvlan1-simple", + "plugins": [ + { + "type": "macvlan", + "mode": "bridge", + "ipam":{ + "type":"host-local", + "subnet": "2001::/64", + "rangeStart": "2001::8", + "rangeEnd": "2001::67" + } + }] + }' +--- +# namespace for MultiNetworkPolicy +apiVersion: v1 +kind: Namespace +metadata: + name: test-simple-v6-ingress +--- +# Pods +apiVersion: v1 +kind: Pod +metadata: + name: pod-server + namespace: test-simple-v6-ingress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v6-ingress + name: pod-server +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-a + namespace: test-simple-v6-ingress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v6-ingress + name: pod-client-a +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-b + namespace: test-simple-v6-ingress + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-simple + labels: + app: test-simple-v6-ingress + name: pod-client-b +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-klp", "5555"] + securityContext: + privileged: true +--- +# MultiNetworkPolicies +# this policy accepts ingress trafic from pod-client-a to pod-server +apiVersion: k8s.cni.cncf.io/v1beta1 +kind: MultiNetworkPolicy +metadata: + name: test-multinetwork-policy-simple-1 + namespace: test-simple-v6-ingress + annotations: + k8s.v1.cni.cncf.io/policy-for: default/macvlan1-simple +spec: + podSelector: + matchLabels: + name: pod-server + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + name: pod-client-a diff --git a/e2e/tests/stacked.bats b/e2e/tests/stacked.bats new file mode 100755 index 00000000..483b31f0 --- /dev/null +++ b/e2e/tests/stacked.bats @@ -0,0 +1,55 @@ +#!/usr/bin/env bats + +# Note: +# These test cases, stacked, will create stacked policy rules in one multi-networkpolicy and test the +# traffic policying by ncat (nc) command. + +setup() { + cd $BATS_TEST_DIRNAME + load "common" + + server_net1=$(get_net1_ip "test-stacked" "pod-server") + client_a_net1=$(get_net1_ip "test-stacked" "pod-client-a") + client_b_net1=$(get_net1_ip "test-stacked" "pod-client-b") + client_c_net1=$(get_net1_ip "test-stacked" "pod-client-c") +} + +@test "setup stacked test environments" { + kubectl create -f stacked.yml + run kubectl -n test-stacked wait --for=condition=ready -l app=test-stacked pod --timeout=${kubewait_timeout} + [ "$status" -eq "0" ] +} + +@test "check generated iptables rules" { + # wait for sync + sleep 3 + run kubectl -n test-stacked exec pod-server -it -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "0" ] + run kubectl -n test-stacked exec pod-client-a -it -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + run kubectl -n test-stacked exec pod-client-b -it -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] + run kubectl -n test-stacked exec pod-client-c -it -- sh -c "iptables-save | grep MULTI-0-INGRESS" + [ "$status" -eq "1" ] +} + +@test "test-stacked check client-a" { + run kubectl -n test-stacked exec pod-client-a -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-stacked check client-b" { + run kubectl -n test-stacked exec pod-client-b -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "0" ] +} + +@test "test-stacked check client-c" { + run kubectl -n test-stacked exec pod-client-c -- sh -c "echo x | nc -w 1 ${server_net1} 5555" + [ "$status" -eq "1" ] +} + +@test "cleanup environments" { + kubectl delete -f stacked.yml + run kubectl -n test-stacked wait --for=delete -l app=test-stacked pod --timeout=${kubewait_timeout} + [ "$status" -eq "0" ] +} diff --git a/e2e/tests/stacked.yml b/e2e/tests/stacked.yml new file mode 100644 index 00000000..e64c4766 --- /dev/null +++ b/e2e/tests/stacked.yml @@ -0,0 +1,143 @@ +--- +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + namespace: default + name: macvlan1-stacked +spec: + config: '{ + "cniVersion": "0.3.1", + "name": "macvlan1-stacked", + "plugins": [ + { + "type": "macvlan", + "mode": "bridge", + "ipam":{ + "type":"host-local", + "subnet":"2.2.5.0/24", + "rangeStart":"2.2.5.8", + "rangeEnd":"2.2.5.67" + } + }] + }' +--- +apiVersion: v1 +kind: Namespace +metadata: + name: test-stacked +--- +# Pods +apiVersion: v1 +kind: Pod +metadata: + name: pod-server + namespace: test-stacked + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-stacked + labels: + app: test-stacked + name: pod-server +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-kl", "0.0.0.0", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-a + namespace: test-stacked + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-stacked + labels: + app: test-stacked + name: pod-client-a +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-kl", "0.0.0.0", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-b + namespace: test-stacked + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-stacked + labels: + app: test-stacked + name: pod-client-b +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-kl", "0.0.0.0", "5555"] + securityContext: + privileged: true +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-client-c + namespace: test-stacked + annotations: + k8s.v1.cni.cncf.io/networks: default/macvlan1-stacked + labels: + app: test-stacked + app: pod-d +spec: + containers: + - name: macvlan-worker1 + image: ghcr.io/k8snetworkplumbingwg/multi-networkpolicy-iptables:e2e-test + command: ["nc", "-kl", "0.0.0.0", "5555"] + securityContext: + privileged: true +--- +# MultiNetworkPolicies +# this policy accepts ingress trafic from pod-client-a to pod-server +# next policy accepts ingress trafic from pod-client-b to pod-server +# as a result, these policies accepts ingress traffic from pod-client-a +# or from pod-client-b, to pod-server. +apiVersion: k8s.cni.cncf.io/v1beta1 +kind: MultiNetworkPolicy +metadata: + name: testnetwork-policy-stacked-1 + namespace: test-stacked + annotations: + k8s.v1.cni.cncf.io/policy-for: default/macvlan1-stacked +spec: + podSelector: + matchLabels: + name: pod-server + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + name: pod-client-a +--- +apiVersion: k8s.cni.cncf.io/v1beta1 +kind: MultiNetworkPolicy +metadata: + name: testnetwork-policy-stacked-2 + namespace: test-stacked + annotations: + k8s.v1.cni.cncf.io/policy-for: default/macvlan1-stacked +spec: + podSelector: + matchLabels: + name: pod-server + policyTypes: + - Ingress + ingress: + - from: + - podSelector: + matchLabels: + name: pod-client-b diff --git a/e2e/update_image_on_cluster.sh b/e2e/update_image_on_cluster.sh new file mode 100755 index 00000000..823ff8c3 --- /dev/null +++ b/e2e/update_image_on_cluster.sh @@ -0,0 +1,13 @@ +#!/bin/sh +set -o errexit + +E2E="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )" +export PATH=${PATH}:${E2E}/bin +OCI_BIN="${OCI_BIN:-docker}" +IMAGE="localhost:5000/multus-networkpolicy-iptables:e2e" + +$OCI_BIN build -t ${IMAGE} ${E2E}/.. +$OCI_BIN push ${IMAGE} +new_image_with_digest=`${OCI_BIN} inspect --format='{{index .RepoDigests 0}}' ${IMAGE}` + +kubectl set image -n kube-system ds/multi-networkpolicy-ds-amd64 multi-networkpolicy=${new_image_with_digest} diff --git a/pkg/controllers/doc.go b/pkg/controllers/doc.go new file mode 100644 index 00000000..7ce4aa01 --- /dev/null +++ b/pkg/controllers/doc.go @@ -0,0 +1,16 @@ +// Copyright (c) 2021 Multus Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package controllers is the package that contains controller functions. +package controllers diff --git a/pkg/controllers/pod.go b/pkg/controllers/pod.go index 52c28882..af088709 100644 --- a/pkg/controllers/pod.go +++ b/pkg/controllers/pod.go @@ -36,7 +36,7 @@ import ( utilruntime "k8s.io/apimachinery/pkg/util/runtime" coreinformers "k8s.io/client-go/informers/core/v1" "k8s.io/client-go/tools/cache" - pb "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" + pb "k8s.io/cri-api/pkg/apis/runtime/v1" "k8s.io/klog" k8sutils "k8s.io/kubernetes/pkg/kubelet/util" ) diff --git a/pkg/server/doc.go b/pkg/server/doc.go new file mode 100644 index 00000000..56364f28 --- /dev/null +++ b/pkg/server/doc.go @@ -0,0 +1,16 @@ +// Copyright (c) 2021 Multus Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package server is the package that contains server functions. +package server diff --git a/pkg/server/options.go b/pkg/server/options.go index b2d32c98..810627f6 100644 --- a/pkg/server/options.go +++ b/pkg/server/options.go @@ -17,7 +17,11 @@ limitations under the License. package server import ( + "bufio" "flag" + "net" + "os" + "strings" "github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables/pkg/controllers" "github.com/spf13/pflag" @@ -31,14 +35,30 @@ type Options struct { // kubeconfig is the path to a KubeConfig file. Kubeconfig string // master is used to override the kubeconfig's URL to the apiserver - master string - hostnameOverride string - hostPrefix string - containerRuntime controllers.RuntimeKind - containerRuntimeEndpoint string - networkPlugins []string - podIptables string - + master string + hostnameOverride string + hostPrefix string + containerRuntime controllers.RuntimeKind + containerRuntimeEndpoint string + networkPlugins []string + podIptables string + syncPeriod int + acceptICMPv6 bool + acceptICMP bool + allowIPv6SrcPrefixText string + allowIPv6DstPrefixText string + customIPv4IngressRuleFile string + customIPv4EgressRuleFile string + customIPv6IngressRuleFile string + customIPv6EgressRuleFile string + + // updated by command line parsing + allowIPv6SrcPrefix []string + allowIPv6DstPrefix []string + customIPv4IngressRule []string + customIPv4EgressRule []string + customIPv6IngressRule []string + customIPv6EgressRule []string // stopCh is used to stop the command stopCh chan struct{} } @@ -55,9 +75,94 @@ func (o *Options) AddFlags(fs *pflag.FlagSet) { fs.StringVar(&o.hostPrefix, "host-prefix", o.hostPrefix, "If non-empty, will use this string as prefix for host filesystem.") fs.StringSliceVar(&o.networkPlugins, "network-plugins", []string{"macvlan"}, "List of network plugins to be be considered for network policies.") fs.StringVar(&o.podIptables, "pod-iptables", o.podIptables, "If non-empty, will use this path to store pod's iptables for troubleshooting helper.") + fs.IntVar(&o.syncPeriod, "sync-period", defaultSyncPeriod, "sync period for multi-networkpolicy syncRunner") + fs.BoolVar(&o.acceptICMP, "accept-icmp", false, "accept all ICMP traffic") + fs.BoolVar(&o.acceptICMPv6, "accept-icmpv6", false, "accept all ICMPv6 traffic") + fs.StringVar(&o.allowIPv6SrcPrefixText, "allow-ipv6-src-prefix", "", "Accept source IPv6 prefix list, comma separated (e.g. \"fe80::/10\")") + fs.StringVar(&o.allowIPv6DstPrefixText, "allow-ipv6-dst-prefix", "", "Accept destination IPv6 prefix list, comma separated (e.g. \"fe80:/10,ff00::/8\")") + fs.StringVar(&o.customIPv4IngressRuleFile, "custom-v4-ingress-rule-file", "", "custom rule file for IPv4 ingress") + fs.StringVar(&o.customIPv4EgressRuleFile, "custom-v4-egress-rule-file", "", "custom rule file for IPv4 egress") + fs.StringVar(&o.customIPv6IngressRuleFile, "custom-v6-ingress-rule-file", "", "custom rule file for IPv6 ingress") + fs.StringVar(&o.customIPv6EgressRuleFile, "custom-v6-egress-rule-file", "", "custom rule file for IPv6 egress") fs.AddGoFlagSet(flag.CommandLine) } +func parseCustomRuleFile(filename string, rules *[]string) error { + if filename != "" { + *rules = []string{} + fp, err := os.Open(filename) + if err != nil { + return err + } + defer fp.Close() + + scanner := bufio.NewScanner(fp) + for scanner.Scan() { + rule := scanner.Text() + if strings.HasPrefix(rule, "#") { // skip rule begin with '#' + continue + } + *rules = append(*rules, rule) + } + + if err := scanner.Err(); err != nil { + return err + } + } + return nil +} + +func parseIPPrefixText(prefixText string, prefixList *[]string) error { + if prefixText != "" { + *prefixList = []string{} + for _, addrRaw := range strings.Split(prefixText, ",") { + addr := strings.TrimSpace(addrRaw) + _, _, err := net.ParseCIDR(addr) + if err != nil { + return err + } + *prefixList = append(*prefixList, addr) + } + } + return nil +} + +// Validate checks several options and fill processed value +func (o *Options) Validate() error { + + // Validate IPv6 source prefix list + if err := parseIPPrefixText(o.allowIPv6SrcPrefixText, &o.allowIPv6SrcPrefix); err != nil { + return err + } + + // Validate IPv6 destination prefix list + if err := parseIPPrefixText(o.allowIPv6DstPrefixText, &o.allowIPv6DstPrefix); err != nil { + return err + } + + // Validate v4 ingress rules + if err := parseCustomRuleFile(o.customIPv4IngressRuleFile, &o.customIPv4IngressRule); err != nil { + return err + } + + // Validate v4 engress rules + if err := parseCustomRuleFile(o.customIPv4EgressRuleFile, &o.customIPv4EgressRule); err != nil { + return err + } + + // Validate v6 ingress rules + if err := parseCustomRuleFile(o.customIPv6IngressRuleFile, &o.customIPv6IngressRule); err != nil { + return err + } + + // Validate v6 engress rules + if err := parseCustomRuleFile(o.customIPv6EgressRuleFile, &o.customIPv6EgressRule); err != nil { + return err + } + + return nil +} + // Run invokes server func (o *Options) Run() error { server, err := NewServer(o) @@ -72,6 +177,12 @@ func (o *Options) Run() error { klog.Infof("hostname: %v", hostname) klog.Infof("container-runtime: %v", o.containerRuntime) + // validate option and update it (check v6prefix list) + err = o.Validate() + if err != nil { + return err + } + server.Run(hostname, o.stopCh) return nil diff --git a/pkg/server/policyrules.go b/pkg/server/policyrules.go index e621641d..19372544 100644 --- a/pkg/server/policyrules.go +++ b/pkg/server/policyrules.go @@ -47,6 +47,7 @@ type iptableBuffer struct { currentFilter map[utiliptables.Chain][]byte currentChain map[utiliptables.Chain]bool activeChain map[utiliptables.Chain]bool + policyCommon *bytes.Buffer policyIndex *bytes.Buffer ingressPorts *bytes.Buffer ingressFrom *bytes.Buffer @@ -54,11 +55,13 @@ type iptableBuffer struct { egressTo *bytes.Buffer filterChains *bytes.Buffer filterRules *bytes.Buffer + isIPv6 bool } func newIptableBuffer() *iptableBuffer { buf := &iptableBuffer{ currentFilter: make(map[utiliptables.Chain][]byte), + policyCommon: bytes.NewBuffer(nil), policyIndex: bytes.NewBuffer(nil), ingressPorts: bytes.NewBuffer(nil), ingressFrom: bytes.NewBuffer(nil), @@ -73,11 +76,13 @@ func newIptableBuffer() *iptableBuffer { } func (ipt *iptableBuffer) Init(iptables utiliptables.Interface) { + ipt.isIPv6 = iptables.IsIPv6() + tmpbuf := bytes.NewBuffer(nil) tmpbuf.Reset() err := iptables.SaveInto(utiliptables.TableFilter, tmpbuf) if err != nil { - klog.Error("failed to get iptable filter") + klog.Errorf("failed to get iptable filter: %v", err) return } ipt.currentFilter = utiliptables.GetChainLines(utiliptables.TableFilter, tmpbuf.Bytes()) @@ -93,7 +98,7 @@ func (ipt *iptableBuffer) Init(iptables utiliptables.Interface) { // Make sure we keep stats for the top-level chains, if they existed // (which most should have because we created them above). - for _, chainName := range []utiliptables.Chain{ingressChain, egressChain} { + for _, chainName := range []utiliptables.Chain{ingressChain, ingressCommonChain, egressChain, egressCommonChain} { ipt.activeChain[chainName] = true if chain, ok := ipt.currentFilter[chainName]; ok { writeBytesLine(ipt.filterChains, chain) @@ -105,6 +110,7 @@ func (ipt *iptableBuffer) Init(iptables utiliptables.Interface) { // Reset clears iptableBuffer func (ipt *iptableBuffer) Reset() { + ipt.policyCommon.Reset() ipt.policyIndex.Reset() ipt.ingressPorts.Reset() ipt.ingressFrom.Reset() @@ -123,6 +129,7 @@ func (ipt *iptableBuffer) FinalizeRules() { writeLine(ipt.policyIndex, "-X", string(chainName)) } ipt.filterRules.Write(ipt.filterChains.Bytes()) + ipt.filterRules.Write(ipt.policyCommon.Bytes()) ipt.filterRules.Write(ipt.policyIndex.Bytes()) ipt.filterRules.Write(ipt.ingressPorts.Bytes()) ipt.filterRules.Write(ipt.ingressFrom.Bytes()) @@ -160,14 +167,58 @@ func (ipt *iptableBuffer) CreateFilterChain(chainName string) { } } +func (ipt *iptableBuffer) renderIngressCommon(s *Server) { + // Add jump from MULTI-INGRESS + writeLine(ipt.policyIndex, "-A", ingressChain, "-j", ingressCommonChain) + + if ipt.isIPv6 { + if s.Options.acceptICMPv6 { + // Allow incoming ICMPv6 traffic + writeLine(ipt.policyCommon, "-A", ingressCommonChain, "-p icmpv6 -j ACCEPT") + } + + // add source prefix whitelist + if len(s.Options.allowIPv6SrcPrefix) != 0 { + for _, addr := range s.Options.allowIPv6SrcPrefix { + writeLine(ipt.policyCommon, "-A", ingressCommonChain, + "-s", strings.TrimSpace(addr), "-j ACCEPT") + } + } + + // add destination prefix whitelist + if len(s.Options.allowIPv6DstPrefix) != 0 { + for _, addr := range s.Options.allowIPv6DstPrefix { + writeLine(ipt.policyCommon, "-A", ingressCommonChain, + "-d", strings.TrimSpace(addr), "-j ACCEPT") + } + } + + // add custom rules + if s.Options.customIPv6IngressRule != nil { + for _, rule := range s.Options.customIPv6IngressRule { + writeLine(ipt.policyCommon, "-A", ingressCommonChain, rule) + } + } + } else { // IPv4 + if s.Options.acceptICMP { + // Allow incoming ICMPv6 traffic to let Neighbor Discovery Protocol work (RFC4861) + writeLine(ipt.policyCommon, "-A", ingressCommonChain, "-p icmp -j ACCEPT") + } + + // add custom rules + if s.Options.customIPv4IngressRule != nil { + for _, rule := range s.Options.customIPv4IngressRule { + writeLine(ipt.policyCommon, "-A", ingressCommonChain, rule) + } + } + } + writeLine(ipt.policyCommon, "-A", ingressCommonChain, "-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT") +} + func (ipt *iptableBuffer) renderIngress(s *Server, podInfo *controllers.PodInfo, idx int, policy *multiv1beta1.MultiNetworkPolicy, policyNetworks []string) { chainName := fmt.Sprintf("MULTI-%d-INGRESS", idx) ipt.CreateFilterChain(chainName) - ingresses := policy.Spec.Ingress - if idx == 0 { - writeLine(ipt.policyIndex, "-A", ingressChain, "-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT") - } for _, podIntf := range podInfo.Interfaces { if podIntf.CheckPolicyNetwork(policyNetworks) { comment := fmt.Sprintf("\"policy:%s net-attach-def:%s\"", policy.Name, podIntf.NetattachName) @@ -179,7 +230,7 @@ func (ipt *iptableBuffer) renderIngress(s *Server, podInfo *controllers.PodInfo, } } - for n, ingress := range ingresses { + for n, ingress := range policy.Spec.Ingress { writeLine(ipt.policyIndex, "-A", chainName, "-j", "MARK", "--set-xmark 0x0/0x30000") ipt.renderIngressPorts(s, podInfo, idx, n, ingress.Ports, policyNetworks) @@ -277,16 +328,20 @@ func (ipt *iptableBuffer) renderIngressFrom(s *Server, podInfo *controllers.PodI continue } for _, ip := range sPodIntf.IPs { - writeLine(ipt.ingressFrom, "-A", chainName, - "-i", podIntf.InterfaceName, "-s", ip, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") - validPeers++ + if ipt.isIPFamilyCompatible(ip) { + writeLine(ipt.ingressFrom, "-A", chainName, + "-i", podIntf.InterfaceName, "-s", ip, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + validPeers++ + } } // ingress should accept reverse path for _, ip := range podIntf.IPs { - writeLine(ipt.ingressFrom, "-A", chainName, - "-i", podIntf.InterfaceName, "-s", ip, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") + if ipt.isIPFamilyCompatible(ip) { + writeLine(ipt.ingressFrom, "-A", chainName, + "-i", podIntf.InterfaceName, "-s", ip, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + } } } } @@ -297,28 +352,34 @@ func (ipt *iptableBuffer) renderIngressFrom(s *Server, podInfo *controllers.PodI if !podIntf.CheckPolicyNetwork(policyNetworks) { continue } - writeLine(ipt.ingressFrom, "-A", chainName, - "-i", podIntf.InterfaceName, "-s", except, "-j", "DROP") - validPeers++ + if ipt.isIPFamilyCompatible(except) { + writeLine(ipt.ingressFrom, "-A", chainName, + "-i", podIntf.InterfaceName, "-s", except, "-j", "DROP") + validPeers++ + } } } for _, podIntf := range podInfo.Interfaces { if !podIntf.CheckPolicyNetwork(policyNetworks) { continue } - writeLine(ipt.ingressFrom, "-A", chainName, - "-i", podIntf.InterfaceName, "-s", peer.IPBlock.CIDR, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") - validPeers++ + if ipt.isIPFamilyCompatible(peer.IPBlock.CIDR) { + writeLine(ipt.ingressFrom, "-A", chainName, + "-i", podIntf.InterfaceName, "-s", peer.IPBlock.CIDR, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + validPeers++ + } } for _, podIntf := range podInfo.Interfaces { if !podIntf.CheckPolicyNetwork(policyNetworks) { continue } for _, ip := range podIntf.IPs { - writeLine(ipt.ingressFrom, "-A", chainName, - "-i", podIntf.InterfaceName, "-s", ip, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") + if ipt.isIPFamilyCompatible(ip) { + writeLine(ipt.ingressFrom, "-A", chainName, + "-i", podIntf.InterfaceName, "-s", ip, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + } } } } else { @@ -335,14 +396,58 @@ func (ipt *iptableBuffer) renderIngressFrom(s *Server, podInfo *controllers.PodI return } +func (ipt *iptableBuffer) renderEgressCommon(s *Server) { + // Add jump from MULTI-EGRESS + writeLine(ipt.policyIndex, "-A", egressChain, "-j", egressCommonChain) + if ipt.isIPv6 { + if s.Options.acceptICMPv6 { + // Allow outgoing ICMPv6 traffic + writeLine(ipt.policyCommon, "-A", egressCommonChain, "-p icmpv6 -j ACCEPT") + } + + // add source prefix whitelist + if s.Options.allowIPv6SrcPrefix != nil { + for _, addr := range s.Options.allowIPv6SrcPrefix { + writeLine(ipt.policyCommon, "-A", egressCommonChain, + "-s", strings.TrimSpace(addr), "-j ACCEPT") + } + } + + // add destination prefix whitelist + if s.Options.allowIPv6DstPrefix != nil { + for _, addr := range s.Options.allowIPv6DstPrefix { + writeLine(ipt.policyCommon, "-A", egressCommonChain, + "-d", strings.TrimSpace(addr), "-j ACCEPT") + } + } + + // add custom rules + if s.Options.customIPv6EgressRule != nil { + for _, rule := range s.Options.customIPv6EgressRule { + writeLine(ipt.policyCommon, "-A", egressCommonChain, rule) + } + } + } else { // IPv4 + if s.Options.acceptICMP { + // Allow outgoing ICMP traffic + writeLine(ipt.policyCommon, "-A", egressCommonChain, "-p icmp -j ACCEPT") + } + + // add custom rules + if s.Options.customIPv4EgressRule != nil { + for _, rule := range s.Options.customIPv4EgressRule { + writeLine(ipt.policyCommon, "-A", egressCommonChain, rule) + } + } + } + + writeLine(ipt.policyCommon, "-A", egressCommonChain, "-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT") +} + func (ipt *iptableBuffer) renderEgress(s *Server, podInfo *controllers.PodInfo, idx int, policy *multiv1beta1.MultiNetworkPolicy, policyNetworks []string) { chainName := fmt.Sprintf("MULTI-%d-EGRESS", idx) ipt.CreateFilterChain(chainName) - egresses := policy.Spec.Egress - if idx == 0 { - writeLine(ipt.policyIndex, "-A", egressChain, "-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT") - } for _, podIntf := range podInfo.Interfaces { if podIntf.CheckPolicyNetwork(policyNetworks) { comment := fmt.Sprintf("\"policy:%s net-attach-def:%s\"", policy.Name, podIntf.NetattachName) @@ -353,7 +458,8 @@ func (ipt *iptableBuffer) renderEgress(s *Server, podInfo *controllers.PodInfo, "-m", "mark", "--mark", "0x30000/0x30000", "-j", "RETURN") } } - for n, egress := range egresses { + + for n, egress := range policy.Spec.Egress { writeLine(ipt.policyIndex, "-A", chainName, "-j", "MARK", "--set-xmark 0x0/0x30000") ipt.renderEgressPorts(s, podInfo, idx, n, egress.Ports, policyNetworks) ipt.renderEgressTo(s, podInfo, idx, n, egress.To, policyNetworks) @@ -451,16 +557,20 @@ func (ipt *iptableBuffer) renderEgressTo(s *Server, podInfo *controllers.PodInfo continue } for _, ip := range sPodIntf.IPs { - writeLine(ipt.egressTo, "-A", chainName, - "-o", podIntf.InterfaceName, "-d", ip, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") - validPeers++ + if ipt.isIPFamilyCompatible(ip) { + writeLine(ipt.egressTo, "-A", chainName, + "-o", podIntf.InterfaceName, "-d", ip, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + validPeers++ + } } // egress should accept reverse path for _, ip := range podIntf.IPs { - writeLine(ipt.egressTo, "-A", chainName, - "-o", podIntf.InterfaceName, "-d", ip, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") + if ipt.isIPFamilyCompatible(ip) { + writeLine(ipt.egressTo, "-A", chainName, + "-o", podIntf.InterfaceName, "-d", ip, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + } } } } @@ -471,19 +581,23 @@ func (ipt *iptableBuffer) renderEgressTo(s *Server, podInfo *controllers.PodInfo if !multi.CheckPolicyNetwork(policyNetworks) { continue } - writeLine(ipt.egressTo, "-A", chainName, - "-o", multi.InterfaceName, "-d", except, "-j", "DROP") - validPeers++ + if ipt.isIPFamilyCompatible(except) { + writeLine(ipt.egressTo, "-A", chainName, + "-o", multi.InterfaceName, "-d", except, "-j", "DROP") + validPeers++ + } } } for _, podIntf := range podInfo.Interfaces { if !podIntf.CheckPolicyNetwork(policyNetworks) { continue } - writeLine(ipt.egressTo, "-A", chainName, - "-o", podIntf.InterfaceName, "-d", peer.IPBlock.CIDR, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") - validPeers++ + if ipt.isIPFamilyCompatible(peer.IPBlock.CIDR) { + writeLine(ipt.egressTo, "-A", chainName, + "-o", podIntf.InterfaceName, "-d", peer.IPBlock.CIDR, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + validPeers++ + } } // egress should accept reverse path for _, podIntf := range podInfo.Interfaces { @@ -491,9 +605,11 @@ func (ipt *iptableBuffer) renderEgressTo(s *Server, podInfo *controllers.PodInfo continue } for _, ip := range podIntf.IPs { - writeLine(ipt.egressTo, "-A", chainName, - "-o", podIntf.InterfaceName, "-d", ip, - "-j", "MARK", "--set-xmark", "0x20000/0x20000") + if ipt.isIPFamilyCompatible(ip) { + writeLine(ipt.egressTo, "-A", chainName, + "-o", podIntf.InterfaceName, "-d", ip, + "-j", "MARK", "--set-xmark", "0x20000/0x20000") + } } } } else { @@ -510,6 +626,18 @@ func (ipt *iptableBuffer) renderEgressTo(s *Server, podInfo *controllers.PodInfo return } +func (ipt *iptableBuffer) isIPFamilyCompatible(ip string) bool { + if ipt.isIPv6 && isAddressIPv6(ip) { + return true + } + + if !ipt.isIPv6 && isAddressIPv4(ip) { + return true + } + + return false +} + // Join all words with spaces, terminate with newline and write to buf. func writeLine(buf *bytes.Buffer, words ...string) { // We avoid strings.Join for performance reasons. @@ -536,3 +664,11 @@ func renderProtocol(proto *v1.Protocol) string { return strings.ToLower(string(p)) } + +func isAddressIPv6(ip string) bool { + return strings.Contains(ip, ":") +} + +func isAddressIPv4(ip string) bool { + return strings.Contains(ip, ".") +} diff --git a/pkg/server/policyrules_test.go b/pkg/server/policyrules_test.go index c5324120..2382025e 100644 --- a/pkg/server/policyrules_test.go +++ b/pkg/server/policyrules_test.go @@ -19,6 +19,9 @@ package server import ( "bytes" "fmt" + "io/ioutil" + "os" + "path/filepath" "time" "github.com/k8snetworkplumbingwg/multi-networkpolicy-iptables/pkg/controllers" @@ -86,7 +89,8 @@ func NewFakeServer(hostname string) *Server { ConfigSyncPeriod: 15 * time.Minute, NodeRef: nodeRef, ip4Tables: fakeiptables.NewFake(), - //ip6Tables: fakeiptables.NewIPv6Fake(), + ip6Tables: fakeiptables.NewIPv6Fake(), + Options: &Options{}, hostPrefix: hostPrefix, policyChanges: policyChanges, @@ -99,8 +103,8 @@ func NewFakeServer(hostname string) *Server { podLister: informerFactory.Core().V1().Pods().Lister(), } podConfig.RegisterEventHandler(server) - go podConfig.Run(wait.NeverStop) informerFactory.Start(wait.NeverStop) + go podConfig.Run(wait.NeverStop) return server } @@ -204,7 +208,7 @@ func NewCNIConfigList(cniName, cniType string) string { cniConfigTemp := ` { "name": "%s", - "plugins": [ + "plugins": [ { "type": "%s" }] @@ -213,6 +217,19 @@ func NewCNIConfigList(cniName, cniType string) string { } var _ = Describe("policyrules testing", func() { + var tmpDir string + + BeforeEach(func() { + var err error + tmpDir, err = ioutil.TempDir("", "multi-networkpolicy-iptables") + Expect(err).NotTo(HaveOccurred()) + }) + + AfterEach(func() { + err := os.RemoveAll(tmpDir) + Expect(err).NotTo(HaveOccurred()) + }) + It("Initialization", func() { ipt := fakeiptables.NewFake() Expect(ipt).NotTo(BeNil()) @@ -221,33 +238,726 @@ var _ = Describe("policyrules testing", func() { // verify buf initialized at init buf.Init(ipt) - filterChains := []byte("*filter\n:MULTI-INGRESS - [0:0]\n:MULTI-EGRESS - [0:0]\n") - Expect(buf.filterChains.Bytes()).To(Equal(filterChains)) - emptyBytes := []byte("") - Expect(buf.policyIndex.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressFrom.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressTo.Bytes()).To(Equal(emptyBytes)) + filterChains := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +` + Expect(buf.filterChains.String()).To(Equal(filterChains)) + Expect(buf.policyIndex.String()).To(Equal("")) + Expect(buf.ingressPorts.String()).To(Equal("")) + Expect(buf.ingressFrom.String()).To(Equal("")) + Expect(buf.egressPorts.String()).To(Equal("")) + Expect(buf.egressTo.String()).To(Equal("")) // finalize buf and verify rules buffer buf.FinalizeRules() - filterRules := []byte("*filter\n:MULTI-INGRESS - [0:0]\n:MULTI-EGRESS - [0:0]\nCOMMIT\n") - Expect(buf.filterRules.Bytes()).To(Equal(filterRules)) + filterRules := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +COMMIT +` + Expect(buf.filterRules.String()).To(Equal(filterRules)) // sync and verify iptable Expect(buf.SyncRules(ipt)).To(BeNil()) iptableRules := bytes.NewBuffer(nil) ipt.SaveInto(utiliptables.TableFilter, iptableRules) - Expect(iptableRules.Bytes()).To(Equal(filterRules)) + Expect(iptableRules.String()).To(Equal(filterRules)) // reset and verify empty buf.Reset() - Expect(buf.policyIndex.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressFrom.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressTo.Bytes()).To(Equal(emptyBytes)) + Expect(buf.policyIndex.String()).To(Equal("")) + Expect(buf.ingressPorts.String()).To(Equal("")) + Expect(buf.ingressFrom.String()).To(Equal("")) + Expect(buf.egressPorts.String()).To(Equal("")) + Expect(buf.egressTo.String()).To(Equal("")) + }) + + It("ingress common - default", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderIngressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderIngressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("ingress common - icmp", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.acceptICMP = true + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderIngressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -p icmp -j ACCEPT +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderIngressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("ingress common - icmpv6", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.acceptICMPv6 = true + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderIngressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderIngressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -p icmpv6 -j ACCEPT +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("ingress common - allow src v6 prefix", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.allowIPv6SrcPrefixText = "11::/8 , 22::/64" + err := s.Options.Validate() + Expect(err).NotTo(HaveOccurred()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderIngressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderIngressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -s 11::/8 -j ACCEPT +-A MULTI-INGRESS-COMMON -s 22::/64 -j ACCEPT +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("ingress common - allow dst v6 prefix", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.allowIPv6DstPrefixText = "11::/8 , 22::/64" + Expect(s.Options.Validate()).To(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderIngressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderIngressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -d 11::/8 -j ACCEPT +-A MULTI-INGRESS-COMMON -d 22::/64 -j ACCEPT +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("ingress common - custom v4 rules", func() { + tmpRuleFile := filepath.Join(tmpDir, "testInputRules.txt") + ioutil.WriteFile(tmpRuleFile, []byte( + `# comment: this accepts DHCP packet +-m udp -p udp --sport bootps --dport bootpc -j ACCEPT +`), 0600) + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + // configure rule file and parse it + s.Options.customIPv4IngressRuleFile = tmpRuleFile + Expect(s.Options.Validate()).To(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderIngressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m udp -p udp --sport bootps --dport bootpc -j ACCEPT +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderIngressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("ingress common - custom v6 rules", func() { + tmpRuleFile := filepath.Join(tmpDir, "testInputRules.txt") + ioutil.WriteFile(tmpRuleFile, []byte( + `# comment: this accepts DHCPv6 packets from link-local address +-m udp -p udp --dport 546 -d fe80::/64 -j ACCEPT +`), 0600) + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + // configure rule file and parse it + s.Options.customIPv6IngressRuleFile = tmpRuleFile + Expect(s.Options.Validate()).To(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderIngressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderIngressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-INGRESS-COMMON -m udp -p udp --dport 546 -d fe80::/64 -j ACCEPT +-A MULTI-INGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-INGRESS -j MULTI-INGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("egress common - default", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderEgressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderEgressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("egress common - icmp", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.acceptICMP = true + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderEgressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -p icmp -j ACCEPT +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderEgressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("egress common - icmpv6", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.acceptICMPv6 = true + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderEgressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderEgressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -p icmpv6 -j ACCEPT +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("egress common - allow src v6 prefix", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.allowIPv6SrcPrefixText = "11::/8 , 22::/64" + Expect(s.Options.Validate()).To(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderEgressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderEgressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -s 11::/8 -j ACCEPT +-A MULTI-EGRESS-COMMON -s 22::/64 -j ACCEPT +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("egress common - allow dest v6 prefix", func() { + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + s.Options.allowIPv6DstPrefixText = "11::/8 , 22::/64" + Expect(s.Options.Validate()).To(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderEgressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderEgressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -d 11::/8 -j ACCEPT +-A MULTI-EGRESS-COMMON -d 22::/64 -j ACCEPT +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("egress common - custom v4 rules", func() { + tmpRuleFile := filepath.Join(tmpDir, "testInputRules.txt") + ioutil.WriteFile(tmpRuleFile, []byte( + `# comment: this rules accepts DHCP packets +-m udp -p udp --sport bootc --dport bootps -j ACCEPT +`), 0600) + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + // configure rule file and parse it + s.Options.customIPv4EgressRuleFile = tmpRuleFile + Expect(s.Options.Validate()).To(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderEgressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m udp -p udp --sport bootc --dport bootps -j ACCEPT +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderEgressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) + }) + + It("egress common - custom v6 rules", func() { + tmpRuleFile := filepath.Join(tmpDir, "testInputRules.txt") + ioutil.WriteFile(tmpRuleFile, []byte( + `# comment: this rules accepts DHCPv6 packet to dhcp relay agents/servers +-m udp -p udp --dport 547 -d ff02::1:2 -j ACCEPT +`), 0600) + buf4 := newIptableBuffer() + buf6 := newIptableBuffer() + Expect(buf4).NotTo(BeNil()) + Expect(buf6).NotTo(BeNil()) + + // verify buf initialized at init + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + // configure rule file and parse it + s.Options.customIPv6EgressRuleFile = tmpRuleFile + Expect(s.Options.Validate()).To(BeNil()) + + buf4.Init(s.ip4Tables) + buf6.Init(s.ip6Tables) + + // check IPv4 case + buf4.renderEgressCommon(s) + buf4.FinalizeRules() + finalizedRules4 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf4.filterRules.String()).To(Equal(finalizedRules4)) + + // check IPv6 case + buf6.renderEgressCommon(s) + buf6.FinalizeRules() + finalizedRules6 := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +-A MULTI-EGRESS-COMMON -m udp -p udp --dport 547 -d ff02::1:2 -j ACCEPT +-A MULTI-EGRESS-COMMON -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A MULTI-EGRESS -j MULTI-EGRESS-COMMON +COMMIT +` + Expect(buf6.filterRules.String()).To(Equal(finalizedRules6)) }) It("ingress rules ipblock", func() { @@ -303,25 +1013,31 @@ var _ = Describe("policyrules testing", func() { nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) buf.renderIngress(s, podInfo1, 0, ingressPolicies1, []string{"testns1/net-attach1"}) - portRules := []byte("-A MULTI-0-INGRESS-0-PORTS -i net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000\n") - Expect(buf.ingressPorts.Bytes()).To(Equal(portRules)) + portRules := `-A MULTI-0-INGRESS-0-PORTS -i net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000 +` + Expect(buf.ingressPorts.String()).To(Equal(portRules)) - fromRules := []byte("-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.254 -j DROP\n-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1/24 -j MARK --set-xmark 0x20000/0x20000\n-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000\n") - Expect(buf.ingressFrom.Bytes()).To(Equal(fromRules)) + fromRules := + `-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.254 -j DROP +-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1/24 -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 +` + Expect(buf.ingressFrom.String()).To(Equal(fromRules)) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-INGRESS - [0:0] :MULTI-0-INGRESS-0-PORTS - [0:0] :MULTI-0-INGRESS-0-FROM - [0:0] --A MULTI-INGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-INGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:testns1/net-attach1" -i net1 -j MULTI-0-INGRESS -A MULTI-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN -A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 @@ -332,8 +1048,8 @@ var _ = Describe("policyrules testing", func() { -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1/24 -j MARK --set-xmark 0x20000/0x20000 -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) It("ingress rules podselector/matchlabels", func() { @@ -392,7 +1108,7 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) pod2 := NewFakePodWithNetAnnotation( "testns1", @@ -406,21 +1122,26 @@ COMMIT buf.renderIngress(s, podInfo1, 0, ingressPolicies1, []string{"testns1/net-attach1"}) - portRules := []byte("-A MULTI-0-INGRESS-0-PORTS -i net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000\n") - Expect(buf.ingressPorts.Bytes()).To(Equal(portRules)) + portRules := `-A MULTI-0-INGRESS-0-PORTS -i net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000 +` + Expect(buf.ingressPorts.String()).To(Equal(portRules)) - fromRules := []byte("-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000\n-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000\n") - Expect(buf.ingressFrom.Bytes()).To(Equal(fromRules)) + fromRules := + `-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 +` + Expect(buf.ingressFrom.String()).To(Equal(fromRules)) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-INGRESS - [0:0] :MULTI-0-INGRESS-0-PORTS - [0:0] :MULTI-0-INGRESS-0-FROM - [0:0] --A MULTI-INGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-INGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:testns1/net-attach1" -i net1 -j MULTI-0-INGRESS -A MULTI-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN -A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 @@ -430,8 +1151,8 @@ COMMIT -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000 -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) It("ingress rules namespace selector", func() { @@ -487,7 +1208,7 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) pod2 := NewFakePodWithNetAnnotation( "testns2", @@ -499,14 +1220,15 @@ COMMIT buf.renderIngress(s, podInfo1, 0, ingressPolicies1, []string{"testns1/net-attach1", "testns2/net-attach1"}) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-INGRESS - [0:0] :MULTI-0-INGRESS-0-PORTS - [0:0] :MULTI-0-INGRESS-0-FROM - [0:0] --A MULTI-INGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-INGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:testns1/net-attach1" -i net1 -j MULTI-0-INGRESS -A MULTI-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN -A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 @@ -516,7 +1238,7 @@ COMMIT -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000 -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) +` Expect(buf.filterRules.String()).To(Equal(string(finalizedRules))) }) @@ -571,7 +1293,7 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) pod2 := NewFakePodWithNetAnnotation( "testns2", @@ -583,14 +1305,15 @@ COMMIT buf.renderIngress(s, podInfo1, 0, ingressPolicies1, []string{"default/net-attach1"}) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-INGRESS - [0:0] :MULTI-0-INGRESS-0-PORTS - [0:0] :MULTI-0-INGRESS-0-FROM - [0:0] --A MULTI-INGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-INGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:default/net-attach1" -i net1 -j MULTI-0-INGRESS -A MULTI-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN -A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 @@ -600,7 +1323,7 @@ COMMIT -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000 -A MULTI-0-INGRESS-0-FROM -i net1 -s 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) +` Expect(buf.filterRules.String()).To(Equal(string(finalizedRules))) }) @@ -657,25 +1380,31 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) buf.renderEgress(s, podInfo1, 0, egressPolicies1, []string{"testns1/net-attach1"}) - portRules := []byte("-A MULTI-0-EGRESS-0-PORTS -o net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000\n") - Expect(buf.egressPorts.Bytes()).To(Equal(portRules)) + portRules := `-A MULTI-0-EGRESS-0-PORTS -o net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000 +` + Expect(buf.egressPorts.String()).To(Equal(portRules)) - toRules := []byte("-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.254 -j DROP\n-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1/24 -j MARK --set-xmark 0x20000/0x20000\n-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000\n") - Expect(buf.egressTo.Bytes()).To(Equal(toRules)) + toRules := + `-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.254 -j DROP +-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1/24 -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 +` + Expect(buf.egressTo.String()).To(Equal(toRules)) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-EGRESS - [0:0] :MULTI-0-EGRESS-0-PORTS - [0:0] :MULTI-0-EGRESS-0-TO - [0:0] --A MULTI-EGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-EGRESS -m comment --comment "policy:EgressPolicies1 net-attach-def:testns1/net-attach1" -o net1 -j MULTI-0-EGRESS -A MULTI-EGRESS -m mark --mark 0x30000/0x30000 -j RETURN -A MULTI-0-EGRESS -j MARK --set-xmark 0x0/0x30000 @@ -686,8 +1415,8 @@ COMMIT -A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1/24 -j MARK --set-xmark 0x20000/0x20000 -A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) It("egress rules podselector/matchlabels", func() { @@ -746,7 +1475,7 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) pod2 := NewFakePodWithNetAnnotation( "testns1", @@ -760,21 +1489,26 @@ COMMIT buf.renderEgress(s, podInfo1, 0, egressPolicies1, []string{"testns1/net-attach1"}) - portRules := []byte("-A MULTI-0-EGRESS-0-PORTS -o net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000\n") - Expect(buf.egressPorts.Bytes()).To(Equal(portRules)) + portRules := `-A MULTI-0-EGRESS-0-PORTS -o net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000 +` + Expect(buf.egressPorts.String()).To(Equal(portRules)) - toRules := []byte("-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000\n-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000\n") - Expect(buf.egressTo.Bytes()).To(Equal(toRules)) + toRules := + `-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 +` + Expect(buf.egressTo.String()).To(Equal(toRules)) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-EGRESS - [0:0] :MULTI-0-EGRESS-0-PORTS - [0:0] :MULTI-0-EGRESS-0-TO - [0:0] --A MULTI-EGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-EGRESS -m comment --comment "policy:EgressPolicies1 net-attach-def:testns1/net-attach1" -o net1 -j MULTI-0-EGRESS -A MULTI-EGRESS -m mark --mark 0x30000/0x30000 -j RETURN -A MULTI-0-EGRESS -j MARK --set-xmark 0x0/0x30000 @@ -784,8 +1518,8 @@ COMMIT -A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.2 -j MARK --set-xmark 0x20000/0x20000 -A MULTI-0-EGRESS-0-TO -o net1 -d 10.1.1.1 -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) It("default values", func() { @@ -843,16 +1577,221 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) buf.renderIngress(s, podInfo1, 0, policies1, []string{"testns1/net-attach1"}) buf.renderEgress(s, podInfo1, 0, policies1, []string{"testns1/net-attach1"}) - portRules := []byte("-A MULTI-0-INGRESS-0-PORTS -i net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000\n") - Expect(buf.ingressPorts.Bytes()).To(Equal(portRules)) + portRules := `-A MULTI-0-INGRESS-0-PORTS -i net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000 +` + Expect(buf.ingressPorts.String()).To(Equal(portRules)) - portRules = []byte("-A MULTI-0-EGRESS-0-PORTS -o net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000\n") - Expect(buf.egressPorts.Bytes()).To(Equal(portRules)) + portRules = `-A MULTI-0-EGRESS-0-PORTS -o net1 -m tcp -p tcp --dport 8888 -j MARK --set-xmark 0x10000/0x10000 +` + Expect(buf.egressPorts.String()).To(Equal(portRules)) + }) + + Context("IPv6", func() { + It("shoud avoid using IPv4 addresses on ip6tables", func() { + + policy1 := &multiv1beta1.MultiNetworkPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ingressPolicies1", + Namespace: "testns1", + }, + Spec: multiv1beta1.MultiNetworkPolicySpec{ + Ingress: []multiv1beta1.MultiNetworkPolicyIngressRule{{ + From: []multiv1beta1.MultiNetworkPolicyPeer{{ + PodSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "foobar": "enabled", + }, + }, + }}, + }}, + Egress: []multiv1beta1.MultiNetworkPolicyEgressRule{{ + To: []multiv1beta1.MultiNetworkPolicyPeer{{ + PodSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "foobar": "enabled", + }, + }, + }}, + }}, + }, + } + + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + AddNamespace(s, "testns1") + + Expect( + s.netdefChanges.Update(nil, NewNetDef("testns1", "net-attach1", NewCNIConfig("testCNI", "multi"))), + ).To(BeTrue()) + + pod1 := NewFakePodWithNetAnnotation( + "testns1", + "testpod1", + "net-attach1", + NewFakeNetworkStatus("testns1", "net-attach1", "192.168.1.1", "10.1.1.1"), + nil) + AddPod(s, pod1) + podInfo1, err := s.podMap.GetPodInfo(pod1) + Expect(err).NotTo(HaveOccurred()) + + pod2 := NewFakePodWithNetAnnotation( + "testns1", + "testpod2", + "net-attach1", + NewFakeNetworkStatus("testns1", "net-attach1", "192.168.1.2", "10.1.1.2"), + map[string]string{ + "foobar": "enabled", + }) + AddPod(s, pod2) + + ipt := fakeiptables.NewIPv6Fake() + buf := newIptableBuffer() + buf.Init(ipt) + + buf.renderIngress(s, podInfo1, 0, policy1, []string{"testns1/net-attach1"}) + buf.renderEgress(s, podInfo1, 0, policy1, []string{"testns1/net-attach1"}) + + buf.FinalizeRules() + + expectedRules := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +:MULTI-0-INGRESS - [0:0] +:MULTI-0-INGRESS-0-PORTS - [0:0] +:MULTI-0-INGRESS-0-FROM - [0:0] +:MULTI-0-EGRESS - [0:0] +:MULTI-0-EGRESS-0-PORTS - [0:0] +:MULTI-0-EGRESS-0-TO - [0:0] +-A MULTI-INGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:testns1/net-attach1" -i net1 -j MULTI-0-INGRESS +-A MULTI-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN +-A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 +-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-PORTS +-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-FROM +-A MULTI-EGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:testns1/net-attach1" -o net1 -j MULTI-0-EGRESS +-A MULTI-EGRESS -m mark --mark 0x30000/0x30000 -j RETURN +-A MULTI-0-EGRESS -j MARK --set-xmark 0x0/0x30000 +-A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-PORTS +-A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-TO +-A MULTI-0-INGRESS-0-PORTS -m comment --comment "no ingress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 +-A MULTI-0-INGRESS-0-FROM -m comment --comment "no ingress from, skipped" -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-EGRESS-0-PORTS -m comment --comment "no egress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 +-A MULTI-0-EGRESS-0-TO -m comment --comment "no egress to, skipped" -j MARK --set-xmark 0x20000/0x20000 +COMMIT +` + + Expect(buf.filterRules.String()).To(Equal(expectedRules), buf.filterRules.String()) + }) + + It("shoud manage dual stack networks", func() { + + policy1 := &multiv1beta1.MultiNetworkPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ingressPolicies1", + Namespace: "testns1", + }, + Spec: multiv1beta1.MultiNetworkPolicySpec{ + Ingress: []multiv1beta1.MultiNetworkPolicyIngressRule{{ + From: []multiv1beta1.MultiNetworkPolicyPeer{{ + PodSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "foobar": "enabled", + }, + }, + }}, + }}, + Egress: []multiv1beta1.MultiNetworkPolicyEgressRule{{ + To: []multiv1beta1.MultiNetworkPolicyPeer{{ + PodSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "foobar": "enabled", + }, + }, + }}, + }}, + }, + } + + s := NewFakeServer("samplehost") + Expect(s).NotTo(BeNil()) + + AddNamespace(s, "testns1") + + Expect( + s.netdefChanges.Update(nil, NewNetDef("testns1", "net-attach1", NewCNIConfig("testCNI", "multi"))), + ).To(BeTrue()) + + pod1 := NewFakePodWithNetAnnotation( + "testns1", + "testpod1", + "net-attach1", + NewFakeNetworkStatus("testns1", "net-attach1", "192.168.1.1", "10.1.1.1\",\"2001:db8:a::11"), + nil) + AddPod(s, pod1) + podInfo1, err := s.podMap.GetPodInfo(pod1) + Expect(err).NotTo(HaveOccurred()) + + pod2 := NewFakePodWithNetAnnotation( + "testns1", + "testpod2", + "net-attach1", + NewFakeNetworkStatus("testns1", "net-attach1", "192.168.1.2", "10.1.1.2\",\"2001:db8:a::12"), + map[string]string{ + "foobar": "enabled", + }) + AddPod(s, pod2) + _, err = s.podMap.GetPodInfo(pod2) + Expect(err).NotTo(HaveOccurred()) + + ipt := fakeiptables.NewIPv6Fake() + buf := newIptableBuffer() + buf.Init(ipt) + + buf.renderIngress(s, podInfo1, 0, policy1, []string{"testns1/net-attach1"}) + buf.renderEgress(s, podInfo1, 0, policy1, []string{"testns1/net-attach1"}) + + buf.FinalizeRules() + + expectedRules := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +:MULTI-0-INGRESS - [0:0] +:MULTI-0-INGRESS-0-PORTS - [0:0] +:MULTI-0-INGRESS-0-FROM - [0:0] +:MULTI-0-EGRESS - [0:0] +:MULTI-0-EGRESS-0-PORTS - [0:0] +:MULTI-0-EGRESS-0-TO - [0:0] +-A MULTI-INGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:testns1/net-attach1" -i net1 -j MULTI-0-INGRESS +-A MULTI-INGRESS -m mark --mark 0x30000/0x30000 -j RETURN +-A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 +-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-PORTS +-A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-FROM +-A MULTI-EGRESS -m comment --comment "policy:ingressPolicies1 net-attach-def:testns1/net-attach1" -o net1 -j MULTI-0-EGRESS +-A MULTI-EGRESS -m mark --mark 0x30000/0x30000 -j RETURN +-A MULTI-0-EGRESS -j MARK --set-xmark 0x0/0x30000 +-A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-PORTS +-A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-TO +-A MULTI-0-INGRESS-0-PORTS -m comment --comment "no ingress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 +-A MULTI-0-INGRESS-0-FROM -i net1 -s 2001:db8:a::12 -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-INGRESS-0-FROM -i net1 -s 2001:db8:a::11 -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-EGRESS-0-PORTS -m comment --comment "no egress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 +-A MULTI-0-EGRESS-0-TO -o net1 -d 2001:db8:a::12 -j MARK --set-xmark 0x20000/0x20000 +-A MULTI-0-EGRESS-0-TO -o net1 -d 2001:db8:a::11 -j MARK --set-xmark 0x20000/0x20000 +COMMIT +` + Expect(buf.filterRules.String()).To(Equal(expectedRules)) + }) }) }) @@ -865,33 +1804,45 @@ var _ = Describe("policyrules testing - invalid case", func() { // verify buf initialized at init buf.Init(ipt) - filterChains := []byte("*filter\n:MULTI-INGRESS - [0:0]\n:MULTI-EGRESS - [0:0]\n") - Expect(buf.filterChains.Bytes()).To(Equal(filterChains)) - emptyBytes := []byte("") - Expect(buf.policyIndex.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressFrom.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressTo.Bytes()).To(Equal(emptyBytes)) + filterChains := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +` + Expect(buf.filterChains.String()).To(Equal(filterChains)) + Expect(buf.policyIndex.String()).To(Equal("")) + Expect(buf.ingressPorts.String()).To(Equal("")) + Expect(buf.ingressFrom.String()).To(Equal("")) + Expect(buf.egressPorts.String()).To(Equal("")) + Expect(buf.egressTo.String()).To(Equal("")) // finalize buf and verify rules buffer buf.FinalizeRules() - filterRules := []byte("*filter\n:MULTI-INGRESS - [0:0]\n:MULTI-EGRESS - [0:0]\nCOMMIT\n") - Expect(buf.filterRules.Bytes()).To(Equal(filterRules)) + filterRules := + `*filter +:MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] +:MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] +COMMIT +` + Expect(buf.filterRules.String()).To(Equal(filterRules)) // sync and verify iptable Expect(buf.SyncRules(ipt)).To(BeNil()) iptableRules := bytes.NewBuffer(nil) ipt.SaveInto(utiliptables.TableFilter, iptableRules) - Expect(iptableRules.Bytes()).To(Equal(filterRules)) + Expect(iptableRules.String()).To(Equal(filterRules)) // reset and verify empty buf.Reset() - Expect(buf.policyIndex.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.ingressFrom.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressPorts.Bytes()).To(Equal(emptyBytes)) - Expect(buf.egressTo.Bytes()).To(Equal(emptyBytes)) + Expect(buf.policyIndex.String()).To(Equal("")) + Expect(buf.ingressPorts.String()).To(Equal("")) + Expect(buf.ingressFrom.String()).To(Equal("")) + Expect(buf.egressPorts.String()).To(Equal("")) + Expect(buf.egressTo.String()).To(Equal("")) }) It("ingress rules ipblock", func() { @@ -947,27 +1898,28 @@ var _ = Describe("policyrules testing - invalid case", func() { nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) buf.renderIngress(s, podInfo1, 0, ingressPolicies1, []string{}) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-INGRESS - [0:0] :MULTI-0-INGRESS-0-PORTS - [0:0] :MULTI-0-INGRESS-0-FROM - [0:0] --A MULTI-INGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 -A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-PORTS -A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-FROM -A MULTI-0-INGRESS-0-PORTS -m comment --comment "no ingress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 -A MULTI-0-INGRESS-0-FROM -m comment --comment "no ingress from, skipped" -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) It("ingress rules podselector/matchlabels", func() { @@ -1026,7 +1978,7 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) pod2 := NewFakePodWithNetAnnotation( "testns1", @@ -1041,22 +1993,23 @@ COMMIT buf.renderIngress(s, podInfo1, 0, ingressPolicies1, []string{}) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-INGRESS - [0:0] :MULTI-0-INGRESS-0-PORTS - [0:0] :MULTI-0-INGRESS-0-FROM - [0:0] --A MULTI-INGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-0-INGRESS -j MARK --set-xmark 0x0/0x30000 -A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-PORTS -A MULTI-0-INGRESS -j MULTI-0-INGRESS-0-FROM -A MULTI-0-INGRESS-0-PORTS -m comment --comment "no ingress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 -A MULTI-0-INGRESS-0-FROM -m comment --comment "no ingress from, skipped" -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) It("egress rules ipblock", func() { @@ -1112,27 +2065,28 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) buf.renderEgress(s, podInfo1, 0, egressPolicies1, []string{}) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-EGRESS - [0:0] :MULTI-0-EGRESS-0-PORTS - [0:0] :MULTI-0-EGRESS-0-TO - [0:0] --A MULTI-EGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-0-EGRESS -j MARK --set-xmark 0x0/0x30000 -A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-PORTS -A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-TO -A MULTI-0-EGRESS-0-PORTS -m comment --comment "no egress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 -A MULTI-0-EGRESS-0-TO -m comment --comment "no egress to, skipped" -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) It("egress rules podselector/matchlabels", func() { @@ -1191,7 +2145,7 @@ COMMIT nil) AddPod(s, pod1) podInfo1, err := s.podMap.GetPodInfo(pod1) - Expect(err).To(BeNil()) + Expect(err).NotTo(HaveOccurred()) pod2 := NewFakePodWithNetAnnotation( "testns1", @@ -1206,22 +2160,23 @@ COMMIT buf.renderEgress(s, podInfo1, 0, egressPolicies1, []string{"testns2/net-attach1"}) buf.FinalizeRules() - finalizedRules := []byte( + finalizedRules := `*filter :MULTI-INGRESS - [0:0] +:MULTI-INGRESS-COMMON - [0:0] :MULTI-EGRESS - [0:0] +:MULTI-EGRESS-COMMON - [0:0] :MULTI-0-EGRESS - [0:0] :MULTI-0-EGRESS-0-PORTS - [0:0] :MULTI-0-EGRESS-0-TO - [0:0] --A MULTI-EGRESS -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A MULTI-0-EGRESS -j MARK --set-xmark 0x0/0x30000 -A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-PORTS -A MULTI-0-EGRESS -j MULTI-0-EGRESS-0-TO -A MULTI-0-EGRESS-0-PORTS -m comment --comment "no egress ports, skipped" -j MARK --set-xmark 0x10000/0x10000 -A MULTI-0-EGRESS-0-TO -m comment --comment "no egress to, skipped" -j MARK --set-xmark 0x20000/0x20000 COMMIT -`) - Expect(buf.filterRules.Bytes()).To(Equal(finalizedRules)) +` + Expect(buf.filterRules.String()).To(Equal(finalizedRules)) }) }) diff --git a/pkg/server/server.go b/pkg/server/server.go index 9981bf50..2cea5f19 100644 --- a/pkg/server/server.go +++ b/pkg/server/server.go @@ -59,6 +59,8 @@ import ( "k8s.io/utils/exec" ) +const defaultSyncPeriod = 30 + // Server structure defines data for server type Server struct { podChanges *controllers.PodChangeTracker @@ -239,7 +241,7 @@ func NewServer(o *Options) (*Server, error) { Namespace: "", } - syncPeriod := 30 * time.Second + syncPeriod := time.Duration(o.syncPeriod) * time.Second minSyncPeriod := 0 * time.Second burstSyncs := 2 @@ -291,7 +293,9 @@ func NewServer(o *Options) (*Server, error) { // Sync ... func (s *Server) Sync() { klog.V(4).Infof("Sync Done!") - s.syncRunner.Run() + if s.syncRunner != nil { + s.syncRunner.Run() + } } // AllSynced ... @@ -479,8 +483,7 @@ func (s *Server) syncMultiPolicy() { klog.V(8).Infof("pod: %s/%s %s", p.Namespace, p.Name, netnsPath) _ = netns.Do(func(_ ns.NetNS) error { - err := s.generatePolicyRules(p, podInfo) - s.backupIptablesRules(p, "current") + err := s.generatePolicyRulesForPod(p, podInfo) return err }) } else { @@ -489,7 +492,7 @@ func (s *Server) syncMultiPolicy() { } } -func (s *Server) backupIptablesRules(pod *v1.Pod, suffix string) error { +func (s *Server) backupIptablesRules(pod *v1.Pod, suffix string, iptables utiliptables.Interface) error { // skip it if no podiptables option if s.Options.podIptables == "" { return nil @@ -504,46 +507,80 @@ func (s *Server) backupIptablesRules(pod *v1.Pod, suffix string) error { return err } } - file, err := os.Create(fmt.Sprintf("%s/%s.iptables", podIptables, suffix)) + fileExt := "iptables" + if iptables.IsIPv6() { + fileExt = "ip6tables" + } + file, err := os.Create(fmt.Sprintf("%s/%s.%s", podIptables, suffix, fileExt)) + if err != nil { + klog.Errorf("cannot create pod file %s/%s.%s: %v", podIptables, suffix, fileExt, err) + return err + } defer file.Close() var buffer bytes.Buffer // store iptable result to file //XXX: need error handling? (see kube-proxy) - err = s.ip4Tables.SaveInto(utiliptables.TableMangle, &buffer) - err = s.ip4Tables.SaveInto(utiliptables.TableFilter, &buffer) - err = s.ip4Tables.SaveInto(utiliptables.TableNAT, &buffer) + _ = iptables.SaveInto(utiliptables.TableMangle, &buffer) + _ = iptables.SaveInto(utiliptables.TableFilter, &buffer) + _ = iptables.SaveInto(utiliptables.TableNAT, &buffer) _, err = buffer.WriteTo(file) return err } const ( - ingressChain = "MULTI-INGRESS" - egressChain = "MULTI-EGRESS" + ingressChain = "MULTI-INGRESS" + egressChain = "MULTI-EGRESS" + ingressCommonChain = "MULTI-INGRESS-COMMON" + egressCommonChain = "MULTI-EGRESS-COMMON" ) -func (s *Server) generatePolicyRules(pod *v1.Pod, podInfo *controllers.PodInfo) error { +func (s *Server) generatePolicyRulesForPod(pod *v1.Pod, podInfo *controllers.PodInfo) error { + err := s.generatePolicyRulesForPodAndFamily(pod, podInfo, s.ip4Tables) + if err != nil { + return fmt.Errorf("can't generate iptables for pod [%s]: %w", podNamespacedName(pod), err) + } + + err = s.generatePolicyRulesForPodAndFamily(pod, podInfo, s.ip6Tables) + if err != nil { + return fmt.Errorf("can't generate ip6tables for pod [%s]: %w", podNamespacedName(pod), err) + } + + return nil +} + +func (s *Server) generatePolicyRulesForPodAndFamily(pod *v1.Pod, podInfo *controllers.PodInfo, iptables utiliptables.Interface) error { klog.V(8).Infof("Generate rules for Pod: %v/%v\n", podInfo.Namespace, podInfo.Name) - // -t filter -N MULTI-POLICY-INGRESS # ensure chain - s.ip4Tables.EnsureChain(utiliptables.TableFilter, ingressChain) - // -t filter -N MULTI-POLICY-EGRESS # ensure chain - s.ip4Tables.EnsureChain(utiliptables.TableFilter, egressChain) + // -t filter -N MULTI-INGRESS # ensure chain + iptables.EnsureChain(utiliptables.TableFilter, ingressChain) + // -t filter -N MULTI-EGRESS # ensure chain + iptables.EnsureChain(utiliptables.TableFilter, egressChain) + // -t filter -N MULTI-INGRESS-COMMON # ensure chain + iptables.EnsureChain(utiliptables.TableFilter, ingressCommonChain) + // -t filter -N MULTI-EGRESS-COMMON # ensure chain + iptables.EnsureChain(utiliptables.TableFilter, egressCommonChain) for _, multiIF := range podInfo.Interfaces { - // -A INPUT -j MULTI-POLICY-INGRESS # ensure rules - s.ip4Tables.EnsureRule( + // -A INPUT -j MULTI-INGRESS # ensure rules + iptables.EnsureRule( utiliptables.Prepend, utiliptables.TableFilter, "INPUT", "-i", multiIF.InterfaceName, "-j", ingressChain) - // -A OUTPUT -j MULTI-POLICY-EGRESS # ensure rules - s.ip4Tables.EnsureRule( + // -A OUTPUT -j MULTI-EGRESS # ensure rules + iptables.EnsureRule( utiliptables.Prepend, utiliptables.TableFilter, "OUTPUT", "-o", multiIF.InterfaceName, "-j", egressChain) // -A PREROUTING -i net1 -j RETURN # ensure rules - s.ip4Tables.EnsureRule( + iptables.EnsureRule( utiliptables.Prepend, utiliptables.TableNAT, "PREROUTING", "-i", multiIF.InterfaceName, "-j", "RETURN") } + // -A MULTI-INGRESS -j MULTI-INGRESS-COMMON # ensure rules + iptables.EnsureRule( + utiliptables.Prepend, utiliptables.TableFilter, ingressChain, "-j", ingressCommonChain) + // -A MULTI-EGRESS -j MULTI-EGRESS-COMMON # ensure rules + iptables.EnsureRule( + utiliptables.Prepend, utiliptables.TableFilter, egressChain, "-j", egressCommonChain) iptableBuffer := newIptableBuffer() - iptableBuffer.Init(s.ip4Tables) + iptableBuffer.Init(iptables) iptableBuffer.Reset() idx := 0 @@ -596,10 +633,12 @@ func (s *Server) generatePolicyRules(pod *v1.Pod, podInfo *controllers.PodInfo) if podInfo.CheckPolicyNetwork(policyNetworks) { if ingressEnable { + iptableBuffer.renderIngressCommon(s) iptableBuffer.renderIngress(s, podInfo, idx, policy, policyNetworks) ingressRendered++ } if egressEnable { + iptableBuffer.renderEgressCommon(s) iptableBuffer.renderEgress(s, podInfo, idx, policy, policyNetworks) egressRendered++ } @@ -614,22 +653,29 @@ func (s *Server) generatePolicyRules(pod *v1.Pod, podInfo *controllers.PodInfo) } if !iptableBuffer.IsUsed() { - iptableBuffer.Init(s.ip4Tables) + iptableBuffer.Init(iptables) } iptableBuffer.FinalizeRules() /* store generated iptables rules if podIptables is enabled */ if s.Options.podIptables != "" { - filePath := fmt.Sprintf("%s/%s/networkpolicy.iptables", s.Options.podIptables, pod.UID) - iptableBuffer.SaveRules(filePath) + if iptables.IsIPv6() { + filePath := fmt.Sprintf("%s/%s/networkpolicy.ip6tables", s.Options.podIptables, pod.UID) + iptableBuffer.SaveRules(filePath) + } else { + filePath := fmt.Sprintf("%s/%s/networkpolicy.iptables", s.Options.podIptables, pod.UID) + iptableBuffer.SaveRules(filePath) + } } - if err := iptableBuffer.SyncRules(s.ip4Tables); err != nil { + if err := iptableBuffer.SyncRules(iptables); err != nil { klog.Errorf("sync rules failed for pod [%s]: %v", podNamespacedName(pod), err) return err } + s.backupIptablesRules(pod, "current", iptables) + return nil } diff --git a/pkg/utils/doc.go b/pkg/utils/doc.go new file mode 100644 index 00000000..fbf689e2 --- /dev/null +++ b/pkg/utils/doc.go @@ -0,0 +1,16 @@ +// Copyright (c) 2021 Multus Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package utils is the package that contains utility functions. +package utils diff --git a/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go b/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.pb.go similarity index 94% rename from vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go rename to vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.pb.go index 800310a6..9504bf0d 100644 --- a/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.pb.go +++ b/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.pb.go @@ -17,7 +17,7 @@ limitations under the License. // Code generated by protoc-gen-gogo. DO NOT EDIT. // source: api.proto -package v1alpha2 +package v1 import ( context "context" @@ -433,7 +433,7 @@ func (m *DNSConfig) GetOptions() []string { // PortMapping specifies the port mapping configurations of a sandbox. type PortMapping struct { // Protocol of the port mapping. - Protocol Protocol `protobuf:"varint,1,opt,name=protocol,proto3,enum=runtime.v1alpha2.Protocol" json:"protocol,omitempty"` + Protocol Protocol `protobuf:"varint,1,opt,name=protocol,proto3,enum=runtime.v1.Protocol" json:"protocol,omitempty"` // Port number within the container. Default: 0 (not specified). ContainerPort int32 `protobuf:"varint,2,opt,name=container_port,json=containerPort,proto3" json:"container_port,omitempty"` // Port number on the host. Default: 0 (not specified). @@ -517,7 +517,7 @@ type Mount struct { // If set, the mount needs SELinux relabeling. SelinuxRelabel bool `protobuf:"varint,4,opt,name=selinux_relabel,json=selinuxRelabel,proto3" json:"selinux_relabel,omitempty"` // Requested propagation mode. - Propagation MountPropagation `protobuf:"varint,5,opt,name=propagation,proto3,enum=runtime.v1alpha2.MountPropagation" json:"propagation,omitempty"` + Propagation MountPropagation `protobuf:"varint,5,opt,name=propagation,proto3,enum=runtime.v1.MountPropagation" json:"propagation,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_sizecache int32 `json:"-"` } @@ -594,16 +594,16 @@ type NamespaceOption struct { // Network namespace for this container/sandbox. // Note: There is currently no way to set CONTAINER scoped network in the Kubernetes API. // Namespaces currently set by the kubelet: POD, NODE - Network NamespaceMode `protobuf:"varint,1,opt,name=network,proto3,enum=runtime.v1alpha2.NamespaceMode" json:"network,omitempty"` + Network NamespaceMode `protobuf:"varint,1,opt,name=network,proto3,enum=runtime.v1.NamespaceMode" json:"network,omitempty"` // PID namespace for this container/sandbox. // Note: The CRI default is POD, but the v1.PodSpec default is CONTAINER. // The kubelet's runtime manager will set this to CONTAINER explicitly for v1 pods. // Namespaces currently set by the kubelet: POD, CONTAINER, NODE, TARGET - Pid NamespaceMode `protobuf:"varint,2,opt,name=pid,proto3,enum=runtime.v1alpha2.NamespaceMode" json:"pid,omitempty"` + Pid NamespaceMode `protobuf:"varint,2,opt,name=pid,proto3,enum=runtime.v1.NamespaceMode" json:"pid,omitempty"` // IPC namespace for this container/sandbox. // Note: There is currently no way to set CONTAINER scoped IPC in the Kubernetes API. // Namespaces currently set by the kubelet: POD, NODE - Ipc NamespaceMode `protobuf:"varint,3,opt,name=ipc,proto3,enum=runtime.v1alpha2.NamespaceMode" json:"ipc,omitempty"` + Ipc NamespaceMode `protobuf:"varint,3,opt,name=ipc,proto3,enum=runtime.v1.NamespaceMode" json:"ipc,omitempty"` // Target Container ID for NamespaceMode of TARGET. This container must have been // previously created in the same pod. It is not possible to specify different targets // for each namespace. @@ -867,7 +867,7 @@ func (m *LinuxSandboxSecurityContext) GetSeccompProfilePath() string { // A security profile which can be used for sandboxes and containers. type SecurityProfile struct { // Indicator which `ProfileType` should be applied. - ProfileType SecurityProfile_ProfileType `protobuf:"varint,1,opt,name=profile_type,json=profileType,proto3,enum=runtime.v1alpha2.SecurityProfile_ProfileType" json:"profile_type,omitempty"` + ProfileType SecurityProfile_ProfileType `protobuf:"varint,1,opt,name=profile_type,json=profileType,proto3,enum=runtime.v1.SecurityProfile_ProfileType" json:"profile_type,omitempty"` // Indicates that a pre-defined profile on the node should be used. // Must only be set if `ProfileType` is `Localhost`. // For seccomp, it must be an absolute path to the seccomp profile. @@ -1089,11 +1089,6 @@ type PodSandboxConfig struct { // E.g., // PodSandboxConfig.LogDirectory = `/var/log/pods//` // ContainerConfig.LogPath = `containerName/Instance#.log` - // - // WARNING: Log management and how kubelet should interface with the - // container logs are under active discussion in - // https://issues.k8s.io/24677. There *may* be future change of direction - // for logging as the discussion carries on. LogDirectory string `protobuf:"bytes,3,opt,name=log_directory,json=logDirectory,proto3" json:"log_directory,omitempty"` // DNS config for the sandbox. DnsConfig *DNSConfig `protobuf:"bytes,4,opt,name=dns_config,json=dnsConfig,proto3" json:"dns_config,omitempty"` @@ -1228,7 +1223,7 @@ type RunPodSandboxRequest struct { // If the runtime handler is unknown, this request should be rejected. An // empty string should select the default handler, equivalent to the // behavior before this feature was added. - // See https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class + // See https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class/README.md RuntimeHandler string `protobuf:"bytes,2,opt,name=runtime_handler,json=runtimeHandler,proto3" json:"runtime_handler,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_sizecache int32 `json:"-"` @@ -1751,7 +1746,7 @@ type PodSandboxStatus struct { // Metadata of the sandbox. Metadata *PodSandboxMetadata `protobuf:"bytes,2,opt,name=metadata,proto3" json:"metadata,omitempty"` // State of the sandbox. - State PodSandboxState `protobuf:"varint,3,opt,name=state,proto3,enum=runtime.v1alpha2.PodSandboxState" json:"state,omitempty"` + State PodSandboxState `protobuf:"varint,3,opt,name=state,proto3,enum=runtime.v1.PodSandboxState" json:"state,omitempty"` // Creation timestamp of the sandbox in nanoseconds. Must be > 0. CreatedAt int64 `protobuf:"varint,4,opt,name=created_at,json=createdAt,proto3" json:"created_at,omitempty"` // Network contains network status if network is handled by the runtime. @@ -1927,7 +1922,7 @@ func (m *PodSandboxStatusResponse) GetInfo() map[string]string { // PodSandboxStateValue is the wrapper of PodSandboxState. type PodSandboxStateValue struct { // State of the sandbox. - State PodSandboxState `protobuf:"varint,1,opt,name=state,proto3,enum=runtime.v1alpha2.PodSandboxState" json:"state,omitempty"` + State PodSandboxState `protobuf:"varint,1,opt,name=state,proto3,enum=runtime.v1.PodSandboxState" json:"state,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_sizecache int32 `json:"-"` } @@ -2092,7 +2087,7 @@ type PodSandbox struct { // Metadata of the PodSandbox. Metadata *PodSandboxMetadata `protobuf:"bytes,2,opt,name=metadata,proto3" json:"metadata,omitempty"` // State of the PodSandbox. - State PodSandboxState `protobuf:"varint,3,opt,name=state,proto3,enum=runtime.v1alpha2.PodSandboxState" json:"state,omitempty"` + State PodSandboxState `protobuf:"varint,3,opt,name=state,proto3,enum=runtime.v1.PodSandboxState" json:"state,omitempty"` // Creation timestamps of the PodSandbox in nanoseconds. Must be > 0. CreatedAt int64 `protobuf:"varint,4,opt,name=created_at,json=createdAt,proto3" json:"created_at,omitempty"` // Labels of the PodSandbox. @@ -2348,8 +2343,6 @@ func (m *KeyValue) GetValue() string { // LinuxContainerResources specifies Linux specific configuration for // resources. -// TODO: Consider using Resources from opencontainers/runtime-spec/specs-go -// directly. type LinuxContainerResources struct { // CPU CFS (Completely Fair Scheduler) period. Default: 0 (not specified). CpuPeriod int64 `protobuf:"varint,1,opt,name=cpu_period,json=cpuPeriod,proto3" json:"cpu_period,omitempty"` @@ -2951,7 +2944,7 @@ type WindowsSandboxSecurityContext struct { RunAsUsername string `protobuf:"bytes,1,opt,name=run_as_username,json=runAsUsername,proto3" json:"run_as_username,omitempty"` // The contents of the GMSA credential spec to use to run this container. CredentialSpec string `protobuf:"bytes,2,opt,name=credential_spec,json=credentialSpec,proto3" json:"credential_spec,omitempty"` - // Indicates whether the container be asked to run as a HostProcess container. + // Indicates whether the container requested to run as a HostProcess container. HostProcess bool `protobuf:"varint,3,opt,name=host_process,json=hostProcess,proto3" json:"host_process,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_sizecache int32 `json:"-"` @@ -3438,8 +3431,6 @@ type ContainerConfig struct { LogPath string `protobuf:"bytes,11,opt,name=log_path,json=logPath,proto3" json:"log_path,omitempty"` // Variables for interactive containers, these have very specialized // use-cases (e.g. debugging). - // TODO: Determine if we need to continue supporting these fields that are - // part of Kubernetes's Container Spec. Stdin bool `protobuf:"varint,12,opt,name=stdin,proto3" json:"stdin,omitempty"` StdinOnce bool `protobuf:"varint,13,opt,name=stdin_once,json=stdinOnce,proto3" json:"stdin_once,omitempty"` Tty bool `protobuf:"varint,14,opt,name=tty,proto3" json:"tty,omitempty"` @@ -3970,7 +3961,7 @@ var xxx_messageInfo_RemoveContainerResponse proto.InternalMessageInfo // ContainerStateValue is the wrapper of ContainerState. type ContainerStateValue struct { // State of the container. - State ContainerState `protobuf:"varint,1,opt,name=state,proto3,enum=runtime.v1alpha2.ContainerState" json:"state,omitempty"` + State ContainerState `protobuf:"varint,1,opt,name=state,proto3,enum=runtime.v1.ContainerState" json:"state,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_sizecache int32 `json:"-"` } @@ -4152,7 +4143,7 @@ type Container struct { // image ID. ImageRef string `protobuf:"bytes,5,opt,name=image_ref,json=imageRef,proto3" json:"image_ref,omitempty"` // State of the container. - State ContainerState `protobuf:"varint,6,opt,name=state,proto3,enum=runtime.v1alpha2.ContainerState" json:"state,omitempty"` + State ContainerState `protobuf:"varint,6,opt,name=state,proto3,enum=runtime.v1.ContainerState" json:"state,omitempty"` // Creation time of the container in nanoseconds. CreatedAt int64 `protobuf:"varint,7,opt,name=created_at,json=createdAt,proto3" json:"created_at,omitempty"` // Key-value pairs that may be used to scope and select individual resources. @@ -4369,7 +4360,7 @@ type ContainerStatus struct { // Metadata of the container. Metadata *ContainerMetadata `protobuf:"bytes,2,opt,name=metadata,proto3" json:"metadata,omitempty"` // Status of the container. - State ContainerState `protobuf:"varint,3,opt,name=state,proto3,enum=runtime.v1alpha2.ContainerState" json:"state,omitempty"` + State ContainerState `protobuf:"varint,3,opt,name=state,proto3,enum=runtime.v1.ContainerState" json:"state,omitempty"` // Creation time of the container in nanoseconds. CreatedAt int64 `protobuf:"varint,4,opt,name=created_at,json=createdAt,proto3" json:"created_at,omitempty"` // Start time of the container in nanoseconds. Default: 0 (not specified). @@ -7120,472 +7111,470 @@ func (m *ReopenContainerLogResponse) XXX_DiscardUnknown() { var xxx_messageInfo_ReopenContainerLogResponse proto.InternalMessageInfo func init() { - proto.RegisterEnum("runtime.v1alpha2.Protocol", Protocol_name, Protocol_value) - proto.RegisterEnum("runtime.v1alpha2.MountPropagation", MountPropagation_name, MountPropagation_value) - proto.RegisterEnum("runtime.v1alpha2.NamespaceMode", NamespaceMode_name, NamespaceMode_value) - proto.RegisterEnum("runtime.v1alpha2.PodSandboxState", PodSandboxState_name, PodSandboxState_value) - proto.RegisterEnum("runtime.v1alpha2.ContainerState", ContainerState_name, ContainerState_value) - proto.RegisterEnum("runtime.v1alpha2.SecurityProfile_ProfileType", SecurityProfile_ProfileType_name, SecurityProfile_ProfileType_value) - proto.RegisterType((*VersionRequest)(nil), "runtime.v1alpha2.VersionRequest") - proto.RegisterType((*VersionResponse)(nil), "runtime.v1alpha2.VersionResponse") - proto.RegisterType((*DNSConfig)(nil), "runtime.v1alpha2.DNSConfig") - proto.RegisterType((*PortMapping)(nil), "runtime.v1alpha2.PortMapping") - proto.RegisterType((*Mount)(nil), "runtime.v1alpha2.Mount") - proto.RegisterType((*NamespaceOption)(nil), "runtime.v1alpha2.NamespaceOption") - proto.RegisterType((*Int64Value)(nil), "runtime.v1alpha2.Int64Value") - proto.RegisterType((*LinuxSandboxSecurityContext)(nil), "runtime.v1alpha2.LinuxSandboxSecurityContext") - proto.RegisterType((*SecurityProfile)(nil), "runtime.v1alpha2.SecurityProfile") - proto.RegisterType((*LinuxPodSandboxConfig)(nil), "runtime.v1alpha2.LinuxPodSandboxConfig") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.LinuxPodSandboxConfig.SysctlsEntry") - proto.RegisterType((*PodSandboxMetadata)(nil), "runtime.v1alpha2.PodSandboxMetadata") - proto.RegisterType((*PodSandboxConfig)(nil), "runtime.v1alpha2.PodSandboxConfig") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandboxConfig.AnnotationsEntry") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandboxConfig.LabelsEntry") - proto.RegisterType((*RunPodSandboxRequest)(nil), "runtime.v1alpha2.RunPodSandboxRequest") - proto.RegisterType((*RunPodSandboxResponse)(nil), "runtime.v1alpha2.RunPodSandboxResponse") - proto.RegisterType((*StopPodSandboxRequest)(nil), "runtime.v1alpha2.StopPodSandboxRequest") - proto.RegisterType((*StopPodSandboxResponse)(nil), "runtime.v1alpha2.StopPodSandboxResponse") - proto.RegisterType((*RemovePodSandboxRequest)(nil), "runtime.v1alpha2.RemovePodSandboxRequest") - proto.RegisterType((*RemovePodSandboxResponse)(nil), "runtime.v1alpha2.RemovePodSandboxResponse") - proto.RegisterType((*PodSandboxStatusRequest)(nil), "runtime.v1alpha2.PodSandboxStatusRequest") - proto.RegisterType((*PodIP)(nil), "runtime.v1alpha2.PodIP") - proto.RegisterType((*PodSandboxNetworkStatus)(nil), "runtime.v1alpha2.PodSandboxNetworkStatus") - proto.RegisterType((*Namespace)(nil), "runtime.v1alpha2.Namespace") - proto.RegisterType((*LinuxPodSandboxStatus)(nil), "runtime.v1alpha2.LinuxPodSandboxStatus") - proto.RegisterType((*PodSandboxStatus)(nil), "runtime.v1alpha2.PodSandboxStatus") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandboxStatus.AnnotationsEntry") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandboxStatus.LabelsEntry") - proto.RegisterType((*PodSandboxStatusResponse)(nil), "runtime.v1alpha2.PodSandboxStatusResponse") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandboxStatusResponse.InfoEntry") - proto.RegisterType((*PodSandboxStateValue)(nil), "runtime.v1alpha2.PodSandboxStateValue") - proto.RegisterType((*PodSandboxFilter)(nil), "runtime.v1alpha2.PodSandboxFilter") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandboxFilter.LabelSelectorEntry") - proto.RegisterType((*ListPodSandboxRequest)(nil), "runtime.v1alpha2.ListPodSandboxRequest") - proto.RegisterType((*PodSandbox)(nil), "runtime.v1alpha2.PodSandbox") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandbox.AnnotationsEntry") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.PodSandbox.LabelsEntry") - proto.RegisterType((*ListPodSandboxResponse)(nil), "runtime.v1alpha2.ListPodSandboxResponse") - proto.RegisterType((*ImageSpec)(nil), "runtime.v1alpha2.ImageSpec") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ImageSpec.AnnotationsEntry") - proto.RegisterType((*KeyValue)(nil), "runtime.v1alpha2.KeyValue") - proto.RegisterType((*LinuxContainerResources)(nil), "runtime.v1alpha2.LinuxContainerResources") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.LinuxContainerResources.UnifiedEntry") - proto.RegisterType((*HugepageLimit)(nil), "runtime.v1alpha2.HugepageLimit") - proto.RegisterType((*SELinuxOption)(nil), "runtime.v1alpha2.SELinuxOption") - proto.RegisterType((*Capability)(nil), "runtime.v1alpha2.Capability") - proto.RegisterType((*LinuxContainerSecurityContext)(nil), "runtime.v1alpha2.LinuxContainerSecurityContext") - proto.RegisterType((*LinuxContainerConfig)(nil), "runtime.v1alpha2.LinuxContainerConfig") - proto.RegisterType((*WindowsSandboxSecurityContext)(nil), "runtime.v1alpha2.WindowsSandboxSecurityContext") - proto.RegisterType((*WindowsPodSandboxConfig)(nil), "runtime.v1alpha2.WindowsPodSandboxConfig") - proto.RegisterType((*WindowsContainerSecurityContext)(nil), "runtime.v1alpha2.WindowsContainerSecurityContext") - proto.RegisterType((*WindowsContainerConfig)(nil), "runtime.v1alpha2.WindowsContainerConfig") - proto.RegisterType((*WindowsContainerResources)(nil), "runtime.v1alpha2.WindowsContainerResources") - proto.RegisterType((*ContainerMetadata)(nil), "runtime.v1alpha2.ContainerMetadata") - proto.RegisterType((*Device)(nil), "runtime.v1alpha2.Device") - proto.RegisterType((*ContainerConfig)(nil), "runtime.v1alpha2.ContainerConfig") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerConfig.AnnotationsEntry") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerConfig.LabelsEntry") - proto.RegisterType((*CreateContainerRequest)(nil), "runtime.v1alpha2.CreateContainerRequest") - proto.RegisterType((*CreateContainerResponse)(nil), "runtime.v1alpha2.CreateContainerResponse") - proto.RegisterType((*StartContainerRequest)(nil), "runtime.v1alpha2.StartContainerRequest") - proto.RegisterType((*StartContainerResponse)(nil), "runtime.v1alpha2.StartContainerResponse") - proto.RegisterType((*StopContainerRequest)(nil), "runtime.v1alpha2.StopContainerRequest") - proto.RegisterType((*StopContainerResponse)(nil), "runtime.v1alpha2.StopContainerResponse") - proto.RegisterType((*RemoveContainerRequest)(nil), "runtime.v1alpha2.RemoveContainerRequest") - proto.RegisterType((*RemoveContainerResponse)(nil), "runtime.v1alpha2.RemoveContainerResponse") - proto.RegisterType((*ContainerStateValue)(nil), "runtime.v1alpha2.ContainerStateValue") - proto.RegisterType((*ContainerFilter)(nil), "runtime.v1alpha2.ContainerFilter") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerFilter.LabelSelectorEntry") - proto.RegisterType((*ListContainersRequest)(nil), "runtime.v1alpha2.ListContainersRequest") - proto.RegisterType((*Container)(nil), "runtime.v1alpha2.Container") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.Container.AnnotationsEntry") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.Container.LabelsEntry") - proto.RegisterType((*ListContainersResponse)(nil), "runtime.v1alpha2.ListContainersResponse") - proto.RegisterType((*ContainerStatusRequest)(nil), "runtime.v1alpha2.ContainerStatusRequest") - proto.RegisterType((*ContainerStatus)(nil), "runtime.v1alpha2.ContainerStatus") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerStatus.AnnotationsEntry") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerStatus.LabelsEntry") - proto.RegisterType((*ContainerStatusResponse)(nil), "runtime.v1alpha2.ContainerStatusResponse") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerStatusResponse.InfoEntry") - proto.RegisterType((*UpdateContainerResourcesRequest)(nil), "runtime.v1alpha2.UpdateContainerResourcesRequest") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.UpdateContainerResourcesRequest.AnnotationsEntry") - proto.RegisterType((*UpdateContainerResourcesResponse)(nil), "runtime.v1alpha2.UpdateContainerResourcesResponse") - proto.RegisterType((*ExecSyncRequest)(nil), "runtime.v1alpha2.ExecSyncRequest") - proto.RegisterType((*ExecSyncResponse)(nil), "runtime.v1alpha2.ExecSyncResponse") - proto.RegisterType((*ExecRequest)(nil), "runtime.v1alpha2.ExecRequest") - proto.RegisterType((*ExecResponse)(nil), "runtime.v1alpha2.ExecResponse") - proto.RegisterType((*AttachRequest)(nil), "runtime.v1alpha2.AttachRequest") - proto.RegisterType((*AttachResponse)(nil), "runtime.v1alpha2.AttachResponse") - proto.RegisterType((*PortForwardRequest)(nil), "runtime.v1alpha2.PortForwardRequest") - proto.RegisterType((*PortForwardResponse)(nil), "runtime.v1alpha2.PortForwardResponse") - proto.RegisterType((*ImageFilter)(nil), "runtime.v1alpha2.ImageFilter") - proto.RegisterType((*ListImagesRequest)(nil), "runtime.v1alpha2.ListImagesRequest") - proto.RegisterType((*Image)(nil), "runtime.v1alpha2.Image") - proto.RegisterType((*ListImagesResponse)(nil), "runtime.v1alpha2.ListImagesResponse") - proto.RegisterType((*ImageStatusRequest)(nil), "runtime.v1alpha2.ImageStatusRequest") - proto.RegisterType((*ImageStatusResponse)(nil), "runtime.v1alpha2.ImageStatusResponse") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ImageStatusResponse.InfoEntry") - proto.RegisterType((*AuthConfig)(nil), "runtime.v1alpha2.AuthConfig") - proto.RegisterType((*PullImageRequest)(nil), "runtime.v1alpha2.PullImageRequest") - proto.RegisterType((*PullImageResponse)(nil), "runtime.v1alpha2.PullImageResponse") - proto.RegisterType((*RemoveImageRequest)(nil), "runtime.v1alpha2.RemoveImageRequest") - proto.RegisterType((*RemoveImageResponse)(nil), "runtime.v1alpha2.RemoveImageResponse") - proto.RegisterType((*NetworkConfig)(nil), "runtime.v1alpha2.NetworkConfig") - proto.RegisterType((*RuntimeConfig)(nil), "runtime.v1alpha2.RuntimeConfig") - proto.RegisterType((*UpdateRuntimeConfigRequest)(nil), "runtime.v1alpha2.UpdateRuntimeConfigRequest") - proto.RegisterType((*UpdateRuntimeConfigResponse)(nil), "runtime.v1alpha2.UpdateRuntimeConfigResponse") - proto.RegisterType((*RuntimeCondition)(nil), "runtime.v1alpha2.RuntimeCondition") - proto.RegisterType((*RuntimeStatus)(nil), "runtime.v1alpha2.RuntimeStatus") - proto.RegisterType((*StatusRequest)(nil), "runtime.v1alpha2.StatusRequest") - proto.RegisterType((*StatusResponse)(nil), "runtime.v1alpha2.StatusResponse") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.StatusResponse.InfoEntry") - proto.RegisterType((*ImageFsInfoRequest)(nil), "runtime.v1alpha2.ImageFsInfoRequest") - proto.RegisterType((*UInt64Value)(nil), "runtime.v1alpha2.UInt64Value") - proto.RegisterType((*FilesystemIdentifier)(nil), "runtime.v1alpha2.FilesystemIdentifier") - proto.RegisterType((*FilesystemUsage)(nil), "runtime.v1alpha2.FilesystemUsage") - proto.RegisterType((*ImageFsInfoResponse)(nil), "runtime.v1alpha2.ImageFsInfoResponse") - proto.RegisterType((*ContainerStatsRequest)(nil), "runtime.v1alpha2.ContainerStatsRequest") - proto.RegisterType((*ContainerStatsResponse)(nil), "runtime.v1alpha2.ContainerStatsResponse") - proto.RegisterType((*ListContainerStatsRequest)(nil), "runtime.v1alpha2.ListContainerStatsRequest") - proto.RegisterType((*ContainerStatsFilter)(nil), "runtime.v1alpha2.ContainerStatsFilter") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerStatsFilter.LabelSelectorEntry") - proto.RegisterType((*ListContainerStatsResponse)(nil), "runtime.v1alpha2.ListContainerStatsResponse") - proto.RegisterType((*ContainerAttributes)(nil), "runtime.v1alpha2.ContainerAttributes") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerAttributes.AnnotationsEntry") - proto.RegisterMapType((map[string]string)(nil), "runtime.v1alpha2.ContainerAttributes.LabelsEntry") - proto.RegisterType((*ContainerStats)(nil), "runtime.v1alpha2.ContainerStats") - proto.RegisterType((*CpuUsage)(nil), "runtime.v1alpha2.CpuUsage") - proto.RegisterType((*MemoryUsage)(nil), "runtime.v1alpha2.MemoryUsage") - proto.RegisterType((*ReopenContainerLogRequest)(nil), "runtime.v1alpha2.ReopenContainerLogRequest") - proto.RegisterType((*ReopenContainerLogResponse)(nil), "runtime.v1alpha2.ReopenContainerLogResponse") + proto.RegisterEnum("runtime.v1.Protocol", Protocol_name, Protocol_value) + proto.RegisterEnum("runtime.v1.MountPropagation", MountPropagation_name, MountPropagation_value) + proto.RegisterEnum("runtime.v1.NamespaceMode", NamespaceMode_name, NamespaceMode_value) + proto.RegisterEnum("runtime.v1.PodSandboxState", PodSandboxState_name, PodSandboxState_value) + proto.RegisterEnum("runtime.v1.ContainerState", ContainerState_name, ContainerState_value) + proto.RegisterEnum("runtime.v1.SecurityProfile_ProfileType", SecurityProfile_ProfileType_name, SecurityProfile_ProfileType_value) + proto.RegisterType((*VersionRequest)(nil), "runtime.v1.VersionRequest") + proto.RegisterType((*VersionResponse)(nil), "runtime.v1.VersionResponse") + proto.RegisterType((*DNSConfig)(nil), "runtime.v1.DNSConfig") + proto.RegisterType((*PortMapping)(nil), "runtime.v1.PortMapping") + proto.RegisterType((*Mount)(nil), "runtime.v1.Mount") + proto.RegisterType((*NamespaceOption)(nil), "runtime.v1.NamespaceOption") + proto.RegisterType((*Int64Value)(nil), "runtime.v1.Int64Value") + proto.RegisterType((*LinuxSandboxSecurityContext)(nil), "runtime.v1.LinuxSandboxSecurityContext") + proto.RegisterType((*SecurityProfile)(nil), "runtime.v1.SecurityProfile") + proto.RegisterType((*LinuxPodSandboxConfig)(nil), "runtime.v1.LinuxPodSandboxConfig") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.LinuxPodSandboxConfig.SysctlsEntry") + proto.RegisterType((*PodSandboxMetadata)(nil), "runtime.v1.PodSandboxMetadata") + proto.RegisterType((*PodSandboxConfig)(nil), "runtime.v1.PodSandboxConfig") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandboxConfig.AnnotationsEntry") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandboxConfig.LabelsEntry") + proto.RegisterType((*RunPodSandboxRequest)(nil), "runtime.v1.RunPodSandboxRequest") + proto.RegisterType((*RunPodSandboxResponse)(nil), "runtime.v1.RunPodSandboxResponse") + proto.RegisterType((*StopPodSandboxRequest)(nil), "runtime.v1.StopPodSandboxRequest") + proto.RegisterType((*StopPodSandboxResponse)(nil), "runtime.v1.StopPodSandboxResponse") + proto.RegisterType((*RemovePodSandboxRequest)(nil), "runtime.v1.RemovePodSandboxRequest") + proto.RegisterType((*RemovePodSandboxResponse)(nil), "runtime.v1.RemovePodSandboxResponse") + proto.RegisterType((*PodSandboxStatusRequest)(nil), "runtime.v1.PodSandboxStatusRequest") + proto.RegisterType((*PodIP)(nil), "runtime.v1.PodIP") + proto.RegisterType((*PodSandboxNetworkStatus)(nil), "runtime.v1.PodSandboxNetworkStatus") + proto.RegisterType((*Namespace)(nil), "runtime.v1.Namespace") + proto.RegisterType((*LinuxPodSandboxStatus)(nil), "runtime.v1.LinuxPodSandboxStatus") + proto.RegisterType((*PodSandboxStatus)(nil), "runtime.v1.PodSandboxStatus") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandboxStatus.AnnotationsEntry") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandboxStatus.LabelsEntry") + proto.RegisterType((*PodSandboxStatusResponse)(nil), "runtime.v1.PodSandboxStatusResponse") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandboxStatusResponse.InfoEntry") + proto.RegisterType((*PodSandboxStateValue)(nil), "runtime.v1.PodSandboxStateValue") + proto.RegisterType((*PodSandboxFilter)(nil), "runtime.v1.PodSandboxFilter") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandboxFilter.LabelSelectorEntry") + proto.RegisterType((*ListPodSandboxRequest)(nil), "runtime.v1.ListPodSandboxRequest") + proto.RegisterType((*PodSandbox)(nil), "runtime.v1.PodSandbox") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandbox.AnnotationsEntry") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.PodSandbox.LabelsEntry") + proto.RegisterType((*ListPodSandboxResponse)(nil), "runtime.v1.ListPodSandboxResponse") + proto.RegisterType((*ImageSpec)(nil), "runtime.v1.ImageSpec") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ImageSpec.AnnotationsEntry") + proto.RegisterType((*KeyValue)(nil), "runtime.v1.KeyValue") + proto.RegisterType((*LinuxContainerResources)(nil), "runtime.v1.LinuxContainerResources") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.LinuxContainerResources.UnifiedEntry") + proto.RegisterType((*HugepageLimit)(nil), "runtime.v1.HugepageLimit") + proto.RegisterType((*SELinuxOption)(nil), "runtime.v1.SELinuxOption") + proto.RegisterType((*Capability)(nil), "runtime.v1.Capability") + proto.RegisterType((*LinuxContainerSecurityContext)(nil), "runtime.v1.LinuxContainerSecurityContext") + proto.RegisterType((*LinuxContainerConfig)(nil), "runtime.v1.LinuxContainerConfig") + proto.RegisterType((*WindowsSandboxSecurityContext)(nil), "runtime.v1.WindowsSandboxSecurityContext") + proto.RegisterType((*WindowsPodSandboxConfig)(nil), "runtime.v1.WindowsPodSandboxConfig") + proto.RegisterType((*WindowsContainerSecurityContext)(nil), "runtime.v1.WindowsContainerSecurityContext") + proto.RegisterType((*WindowsContainerConfig)(nil), "runtime.v1.WindowsContainerConfig") + proto.RegisterType((*WindowsContainerResources)(nil), "runtime.v1.WindowsContainerResources") + proto.RegisterType((*ContainerMetadata)(nil), "runtime.v1.ContainerMetadata") + proto.RegisterType((*Device)(nil), "runtime.v1.Device") + proto.RegisterType((*ContainerConfig)(nil), "runtime.v1.ContainerConfig") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerConfig.AnnotationsEntry") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerConfig.LabelsEntry") + proto.RegisterType((*CreateContainerRequest)(nil), "runtime.v1.CreateContainerRequest") + proto.RegisterType((*CreateContainerResponse)(nil), "runtime.v1.CreateContainerResponse") + proto.RegisterType((*StartContainerRequest)(nil), "runtime.v1.StartContainerRequest") + proto.RegisterType((*StartContainerResponse)(nil), "runtime.v1.StartContainerResponse") + proto.RegisterType((*StopContainerRequest)(nil), "runtime.v1.StopContainerRequest") + proto.RegisterType((*StopContainerResponse)(nil), "runtime.v1.StopContainerResponse") + proto.RegisterType((*RemoveContainerRequest)(nil), "runtime.v1.RemoveContainerRequest") + proto.RegisterType((*RemoveContainerResponse)(nil), "runtime.v1.RemoveContainerResponse") + proto.RegisterType((*ContainerStateValue)(nil), "runtime.v1.ContainerStateValue") + proto.RegisterType((*ContainerFilter)(nil), "runtime.v1.ContainerFilter") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerFilter.LabelSelectorEntry") + proto.RegisterType((*ListContainersRequest)(nil), "runtime.v1.ListContainersRequest") + proto.RegisterType((*Container)(nil), "runtime.v1.Container") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.Container.AnnotationsEntry") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.Container.LabelsEntry") + proto.RegisterType((*ListContainersResponse)(nil), "runtime.v1.ListContainersResponse") + proto.RegisterType((*ContainerStatusRequest)(nil), "runtime.v1.ContainerStatusRequest") + proto.RegisterType((*ContainerStatus)(nil), "runtime.v1.ContainerStatus") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerStatus.AnnotationsEntry") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerStatus.LabelsEntry") + proto.RegisterType((*ContainerStatusResponse)(nil), "runtime.v1.ContainerStatusResponse") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerStatusResponse.InfoEntry") + proto.RegisterType((*UpdateContainerResourcesRequest)(nil), "runtime.v1.UpdateContainerResourcesRequest") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.UpdateContainerResourcesRequest.AnnotationsEntry") + proto.RegisterType((*UpdateContainerResourcesResponse)(nil), "runtime.v1.UpdateContainerResourcesResponse") + proto.RegisterType((*ExecSyncRequest)(nil), "runtime.v1.ExecSyncRequest") + proto.RegisterType((*ExecSyncResponse)(nil), "runtime.v1.ExecSyncResponse") + proto.RegisterType((*ExecRequest)(nil), "runtime.v1.ExecRequest") + proto.RegisterType((*ExecResponse)(nil), "runtime.v1.ExecResponse") + proto.RegisterType((*AttachRequest)(nil), "runtime.v1.AttachRequest") + proto.RegisterType((*AttachResponse)(nil), "runtime.v1.AttachResponse") + proto.RegisterType((*PortForwardRequest)(nil), "runtime.v1.PortForwardRequest") + proto.RegisterType((*PortForwardResponse)(nil), "runtime.v1.PortForwardResponse") + proto.RegisterType((*ImageFilter)(nil), "runtime.v1.ImageFilter") + proto.RegisterType((*ListImagesRequest)(nil), "runtime.v1.ListImagesRequest") + proto.RegisterType((*Image)(nil), "runtime.v1.Image") + proto.RegisterType((*ListImagesResponse)(nil), "runtime.v1.ListImagesResponse") + proto.RegisterType((*ImageStatusRequest)(nil), "runtime.v1.ImageStatusRequest") + proto.RegisterType((*ImageStatusResponse)(nil), "runtime.v1.ImageStatusResponse") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ImageStatusResponse.InfoEntry") + proto.RegisterType((*AuthConfig)(nil), "runtime.v1.AuthConfig") + proto.RegisterType((*PullImageRequest)(nil), "runtime.v1.PullImageRequest") + proto.RegisterType((*PullImageResponse)(nil), "runtime.v1.PullImageResponse") + proto.RegisterType((*RemoveImageRequest)(nil), "runtime.v1.RemoveImageRequest") + proto.RegisterType((*RemoveImageResponse)(nil), "runtime.v1.RemoveImageResponse") + proto.RegisterType((*NetworkConfig)(nil), "runtime.v1.NetworkConfig") + proto.RegisterType((*RuntimeConfig)(nil), "runtime.v1.RuntimeConfig") + proto.RegisterType((*UpdateRuntimeConfigRequest)(nil), "runtime.v1.UpdateRuntimeConfigRequest") + proto.RegisterType((*UpdateRuntimeConfigResponse)(nil), "runtime.v1.UpdateRuntimeConfigResponse") + proto.RegisterType((*RuntimeCondition)(nil), "runtime.v1.RuntimeCondition") + proto.RegisterType((*RuntimeStatus)(nil), "runtime.v1.RuntimeStatus") + proto.RegisterType((*StatusRequest)(nil), "runtime.v1.StatusRequest") + proto.RegisterType((*StatusResponse)(nil), "runtime.v1.StatusResponse") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.StatusResponse.InfoEntry") + proto.RegisterType((*ImageFsInfoRequest)(nil), "runtime.v1.ImageFsInfoRequest") + proto.RegisterType((*UInt64Value)(nil), "runtime.v1.UInt64Value") + proto.RegisterType((*FilesystemIdentifier)(nil), "runtime.v1.FilesystemIdentifier") + proto.RegisterType((*FilesystemUsage)(nil), "runtime.v1.FilesystemUsage") + proto.RegisterType((*ImageFsInfoResponse)(nil), "runtime.v1.ImageFsInfoResponse") + proto.RegisterType((*ContainerStatsRequest)(nil), "runtime.v1.ContainerStatsRequest") + proto.RegisterType((*ContainerStatsResponse)(nil), "runtime.v1.ContainerStatsResponse") + proto.RegisterType((*ListContainerStatsRequest)(nil), "runtime.v1.ListContainerStatsRequest") + proto.RegisterType((*ContainerStatsFilter)(nil), "runtime.v1.ContainerStatsFilter") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerStatsFilter.LabelSelectorEntry") + proto.RegisterType((*ListContainerStatsResponse)(nil), "runtime.v1.ListContainerStatsResponse") + proto.RegisterType((*ContainerAttributes)(nil), "runtime.v1.ContainerAttributes") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerAttributes.AnnotationsEntry") + proto.RegisterMapType((map[string]string)(nil), "runtime.v1.ContainerAttributes.LabelsEntry") + proto.RegisterType((*ContainerStats)(nil), "runtime.v1.ContainerStats") + proto.RegisterType((*CpuUsage)(nil), "runtime.v1.CpuUsage") + proto.RegisterType((*MemoryUsage)(nil), "runtime.v1.MemoryUsage") + proto.RegisterType((*ReopenContainerLogRequest)(nil), "runtime.v1.ReopenContainerLogRequest") + proto.RegisterType((*ReopenContainerLogResponse)(nil), "runtime.v1.ReopenContainerLogResponse") } func init() { proto.RegisterFile("api.proto", fileDescriptor_00212fb1f9d3bf1c) } var fileDescriptor_00212fb1f9d3bf1c = []byte{ - // 5147 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xd4, 0x7c, 0x4d, 0x6c, 0x1b, 0x59, - 0x72, 0xb0, 0x9a, 0xa4, 0x24, 0xb2, 0x28, 0x52, 0xd4, 0xb3, 0x6c, 0xd1, 0xf4, 0xd8, 0x63, 0xb5, - 0xc7, 0xbf, 0x33, 0x96, 0xd7, 0x9a, 0x59, 0xcf, 0x67, 0x7b, 0xc6, 0x63, 0x59, 0x92, 0x6d, 0x7e, - 0x6b, 0x53, 0x4c, 0x53, 0x9a, 0x9f, 0x9d, 0x01, 0x7a, 0x5b, 0xec, 0x27, 0xaa, 0xd7, 0x64, 0x77, - 0x4f, 0x77, 0xd3, 0xb2, 0x36, 0x40, 0xb0, 0xc0, 0x02, 0x7b, 0xc8, 0x29, 0x40, 0x90, 0x4b, 0x6e, - 0xd9, 0x1c, 0x72, 0xc8, 0x29, 0x08, 0x72, 0xca, 0x69, 0x83, 0x1c, 0x16, 0x01, 0x82, 0xe4, 0xb4, - 0x48, 0x90, 0x4b, 0x66, 0x82, 0x00, 0x8b, 0x00, 0x01, 0x82, 0x9c, 0x73, 0x08, 0xde, 0x5f, 0xff, - 0x37, 0x7f, 0x6c, 0xcf, 0xce, 0xe4, 0xc4, 0x7e, 0xd5, 0x55, 0xf5, 0xea, 0xd5, 0xab, 0x57, 0xaf, - 0x5e, 0xd5, 0x6b, 0x42, 0x49, 0xb3, 0x8d, 0x35, 0xdb, 0xb1, 0x3c, 0x0b, 0xd5, 0x9c, 0xa1, 0xe9, - 0x19, 0x03, 0xbc, 0xf6, 0xfc, 0xa6, 0xd6, 0xb7, 0x0f, 0xb5, 0xf5, 0xc6, 0xf5, 0x9e, 0xe1, 0x1d, - 0x0e, 0xf7, 0xd7, 0xba, 0xd6, 0xe0, 0x46, 0xcf, 0xea, 0x59, 0x37, 0x28, 0xe2, 0xfe, 0xf0, 0x80, - 0xb6, 0x68, 0x83, 0x3e, 0x31, 0x06, 0xf2, 0x35, 0xa8, 0x7e, 0x8c, 0x1d, 0xd7, 0xb0, 0x4c, 0x05, - 0x7f, 0x39, 0xc4, 0xae, 0x87, 0xea, 0x30, 0xff, 0x9c, 0x41, 0xea, 0xd2, 0x79, 0xe9, 0x4a, 0x49, - 0x11, 0x4d, 0xf9, 0xcf, 0x24, 0x58, 0xf4, 0x91, 0x5d, 0xdb, 0x32, 0x5d, 0x9c, 0x8d, 0x8d, 0x56, - 0x61, 0x81, 0x0b, 0xa7, 0x9a, 0xda, 0x00, 0xd7, 0x73, 0xf4, 0x75, 0x99, 0xc3, 0x5a, 0xda, 0x00, - 0xa3, 0xcb, 0xb0, 0x28, 0x50, 0x04, 0x93, 0x3c, 0xc5, 0xaa, 0x72, 0x30, 0xef, 0x0d, 0xad, 0xc1, - 0x09, 0x81, 0xa8, 0xd9, 0x86, 0x8f, 0x5c, 0xa0, 0xc8, 0x4b, 0xfc, 0xd5, 0x86, 0x6d, 0x70, 0x7c, - 0xf9, 0x73, 0x28, 0x6d, 0xb5, 0x3a, 0x9b, 0x96, 0x79, 0x60, 0xf4, 0x88, 0x88, 0x2e, 0x76, 0x08, - 0x4d, 0x5d, 0x3a, 0x9f, 0x27, 0x22, 0xf2, 0x26, 0x6a, 0x40, 0xd1, 0xc5, 0x9a, 0xd3, 0x3d, 0xc4, - 0x6e, 0x3d, 0x47, 0x5f, 0xf9, 0x6d, 0x42, 0x65, 0xd9, 0x9e, 0x61, 0x99, 0x6e, 0x3d, 0xcf, 0xa8, - 0x78, 0x53, 0xfe, 0x85, 0x04, 0xe5, 0xb6, 0xe5, 0x78, 0x4f, 0x35, 0xdb, 0x36, 0xcc, 0x1e, 0xba, - 0x05, 0x45, 0xaa, 0xcb, 0xae, 0xd5, 0xa7, 0x3a, 0xa8, 0xae, 0x37, 0xd6, 0xe2, 0xd3, 0xb2, 0xd6, - 0xe6, 0x18, 0x8a, 0x8f, 0x8b, 0x2e, 0x42, 0xb5, 0x6b, 0x99, 0x9e, 0x66, 0x98, 0xd8, 0x51, 0x6d, - 0xcb, 0xf1, 0xa8, 0x8a, 0x66, 0x95, 0x8a, 0x0f, 0x25, 0xbd, 0xa0, 0x33, 0x50, 0x3a, 0xb4, 0x5c, - 0x8f, 0x61, 0xe4, 0x29, 0x46, 0x91, 0x00, 0xe8, 0xcb, 0x15, 0x98, 0xa7, 0x2f, 0x0d, 0x9b, 0x2b, - 0x63, 0x8e, 0x34, 0x9b, 0xb6, 0xfc, 0x6b, 0x09, 0x66, 0x9f, 0x5a, 0x43, 0xd3, 0x8b, 0x75, 0xa3, - 0x79, 0x87, 0x7c, 0xa2, 0x42, 0xdd, 0x68, 0xde, 0x61, 0xd0, 0x0d, 0xc1, 0x60, 0x73, 0xc5, 0xba, - 0x21, 0x2f, 0x1b, 0x50, 0x74, 0xb0, 0xa6, 0x5b, 0x66, 0xff, 0x98, 0x8a, 0x50, 0x54, 0xfc, 0x36, - 0x99, 0x44, 0x17, 0xf7, 0x0d, 0x73, 0xf8, 0x42, 0x75, 0x70, 0x5f, 0xdb, 0xc7, 0x7d, 0x2a, 0x4a, - 0x51, 0xa9, 0x72, 0xb0, 0xc2, 0xa0, 0x68, 0x0b, 0xca, 0xb6, 0x63, 0xd9, 0x5a, 0x4f, 0x23, 0x7a, - 0xac, 0xcf, 0x52, 0x55, 0xc9, 0x49, 0x55, 0x51, 0xb1, 0xdb, 0x01, 0xa6, 0x12, 0x26, 0x93, 0xff, - 0x41, 0x82, 0x45, 0x62, 0x3c, 0xae, 0xad, 0x75, 0xf1, 0x0e, 0x9d, 0x12, 0x74, 0x1b, 0xe6, 0x4d, - 0xec, 0x1d, 0x59, 0xce, 0x33, 0x3e, 0x01, 0x6f, 0x26, 0xb9, 0xfa, 0x34, 0x4f, 0x2d, 0x1d, 0x2b, - 0x02, 0x1f, 0xdd, 0x84, 0xbc, 0x6d, 0xe8, 0x74, 0xc0, 0x13, 0x90, 0x11, 0x5c, 0x42, 0x62, 0xd8, - 0x5d, 0xaa, 0x87, 0x49, 0x48, 0x0c, 0xbb, 0x4b, 0x94, 0xeb, 0x69, 0x4e, 0x0f, 0x7b, 0xaa, 0xa1, - 0xf3, 0x89, 0x2a, 0x32, 0x40, 0x53, 0x97, 0x65, 0x80, 0xa6, 0xe9, 0xdd, 0x7a, 0xef, 0x63, 0xad, - 0x3f, 0xc4, 0x68, 0x19, 0x66, 0x9f, 0x93, 0x07, 0x3a, 0x92, 0xbc, 0xc2, 0x1a, 0xf2, 0x57, 0x05, - 0x38, 0xf3, 0x84, 0x28, 0xb3, 0xa3, 0x99, 0xfa, 0xbe, 0xf5, 0xa2, 0x83, 0xbb, 0x43, 0xc7, 0xf0, - 0x8e, 0x37, 0x2d, 0xd3, 0xc3, 0x2f, 0x3c, 0xd4, 0x82, 0x25, 0x53, 0x74, 0xab, 0x0a, 0xbb, 0x25, - 0x1c, 0xca, 0xeb, 0xab, 0x23, 0x24, 0x64, 0xfa, 0x53, 0x6a, 0x66, 0x14, 0xe0, 0xa2, 0xc7, 0xc1, - 0xa4, 0x0a, 0x6e, 0x39, 0xca, 0x2d, 0x65, 0xbc, 0x9d, 0x6d, 0x2a, 0x19, 0xe7, 0x25, 0x66, 0x5d, - 0x70, 0xfa, 0x00, 0xc8, 0x92, 0x57, 0x35, 0x57, 0x1d, 0xba, 0xd8, 0xa1, 0x5a, 0x2b, 0xaf, 0xbf, - 0x91, 0xe4, 0x12, 0xa8, 0x40, 0x29, 0x39, 0x43, 0x73, 0xc3, 0xdd, 0x73, 0xb1, 0x83, 0xee, 0x51, - 0x27, 0x42, 0xa8, 0x7b, 0x8e, 0x35, 0xb4, 0xeb, 0xc5, 0x09, 0xc8, 0x81, 0x92, 0x3f, 0x22, 0xf8, - 0xd4, 0xc3, 0x70, 0x43, 0x55, 0x1d, 0xcb, 0xf2, 0x0e, 0x5c, 0x61, 0x9c, 0x02, 0xac, 0x50, 0x28, - 0xba, 0x01, 0x27, 0xdc, 0xa1, 0x6d, 0xf7, 0xf1, 0x00, 0x9b, 0x9e, 0xd6, 0x67, 0xdd, 0xb9, 0xf5, - 0xd9, 0xf3, 0xf9, 0x2b, 0x79, 0x05, 0x85, 0x5f, 0x51, 0xc6, 0x2e, 0x3a, 0x07, 0x60, 0x3b, 0xc6, - 0x73, 0xa3, 0x8f, 0x7b, 0x58, 0xaf, 0xcf, 0x51, 0xa6, 0x21, 0x08, 0xba, 0x4b, 0xbc, 0x4e, 0xb7, - 0x6b, 0x0d, 0xec, 0x7a, 0x29, 0x6b, 0x1e, 0xc4, 0x2c, 0xb6, 0x1d, 0xeb, 0xc0, 0xe8, 0x63, 0x45, - 0x50, 0xa0, 0x0f, 0xa1, 0xa8, 0xd9, 0xb6, 0xe6, 0x0c, 0x2c, 0xa7, 0x0e, 0x93, 0x52, 0xfb, 0x24, - 0xe8, 0x3d, 0x58, 0xe6, 0x9c, 0x54, 0x9b, 0xbd, 0x64, 0xcb, 0x7a, 0x9e, 0x58, 0xde, 0x83, 0x5c, - 0x5d, 0x52, 0x10, 0x7f, 0xcf, 0x69, 0xc9, 0x22, 0x97, 0xff, 0x4e, 0x82, 0xc5, 0x18, 0x4f, 0xd4, - 0x86, 0x05, 0xc1, 0xc1, 0x3b, 0xb6, 0x31, 0x5f, 0x5e, 0xd7, 0xc7, 0x0a, 0xb3, 0xc6, 0x7f, 0x77, - 0x8f, 0x6d, 0x4c, 0xd7, 0xaf, 0x68, 0xa0, 0x0b, 0x50, 0xe9, 0x5b, 0x5d, 0xad, 0x4f, 0x9d, 0x8d, - 0x83, 0x0f, 0xb8, 0xaf, 0x59, 0xf0, 0x81, 0x0a, 0x3e, 0x90, 0xef, 0x43, 0x39, 0xc4, 0x00, 0x21, - 0xa8, 0x2a, 0xac, 0xc3, 0x2d, 0x7c, 0xa0, 0x0d, 0xfb, 0x5e, 0x6d, 0x06, 0x55, 0x01, 0xf6, 0xcc, - 0x2e, 0xf1, 0xf0, 0x26, 0xd6, 0x6b, 0x12, 0xaa, 0x40, 0xe9, 0x89, 0x60, 0x51, 0xcb, 0xc9, 0xbf, - 0xc8, 0xc1, 0x49, 0x6a, 0x96, 0x6d, 0x4b, 0xe7, 0x6b, 0x86, 0x6f, 0x07, 0x17, 0xa0, 0xd2, 0xa5, - 0xb3, 0xab, 0xda, 0x9a, 0x83, 0x4d, 0x8f, 0xbb, 0xc3, 0x05, 0x06, 0x6c, 0x53, 0x18, 0xfa, 0x14, - 0x6a, 0x2e, 0x1f, 0x91, 0xda, 0x65, 0x6b, 0x8c, 0x2f, 0x80, 0x94, 0xb1, 0x8f, 0x58, 0x98, 0xca, - 0xa2, 0x9b, 0x58, 0xa9, 0xf3, 0xee, 0xb1, 0xdb, 0xf5, 0xfa, 0x6c, 0x5f, 0x29, 0xaf, 0xbf, 0x97, - 0xc1, 0x30, 0x2e, 0xf8, 0x5a, 0x87, 0x91, 0x6d, 0x9b, 0x9e, 0x73, 0xac, 0x08, 0x26, 0x8d, 0x3b, - 0xb0, 0x10, 0x7e, 0x81, 0x6a, 0x90, 0x7f, 0x86, 0x8f, 0xf9, 0xa0, 0xc8, 0x63, 0xe0, 0x51, 0x98, - 0xa6, 0x59, 0xe3, 0x4e, 0xee, 0xff, 0x49, 0xb2, 0x03, 0x28, 0xe8, 0xe5, 0x29, 0xf6, 0x34, 0x5d, - 0xf3, 0x34, 0x84, 0xa0, 0x40, 0x37, 0x6c, 0xc6, 0x82, 0x3e, 0x13, 0xae, 0x43, 0xee, 0x26, 0x4b, - 0x0a, 0x79, 0x44, 0x6f, 0x40, 0xc9, 0xf7, 0x1a, 0x7c, 0xd7, 0x0e, 0x00, 0x64, 0xf7, 0xd4, 0x3c, - 0x0f, 0x0f, 0x6c, 0x8f, 0xae, 0xb7, 0x8a, 0x22, 0x9a, 0xf2, 0x5f, 0xce, 0x42, 0x2d, 0x31, 0x27, - 0xf7, 0xa1, 0x38, 0xe0, 0xdd, 0x73, 0xaf, 0xf5, 0x56, 0xca, 0x16, 0x9a, 0x10, 0x55, 0xf1, 0xa9, - 0xc8, 0x0e, 0x45, 0x66, 0x3e, 0x14, 0x69, 0xf8, 0x6d, 0x66, 0x72, 0x3d, 0x55, 0x37, 0x1c, 0xdc, - 0xf5, 0x2c, 0xe7, 0x98, 0x8b, 0xbb, 0xd0, 0xb7, 0x7a, 0x5b, 0x02, 0x86, 0xee, 0x00, 0xe8, 0xa6, - 0xab, 0x52, 0x8b, 0xea, 0x51, 0xa1, 0xcb, 0xeb, 0x67, 0x92, 0x42, 0xf8, 0x61, 0x85, 0x52, 0xd2, - 0x4d, 0x97, 0x8b, 0xff, 0x00, 0x2a, 0x64, 0x77, 0x56, 0x07, 0x2c, 0x22, 0x60, 0x6e, 0xa3, 0xbc, - 0x7e, 0x36, 0x6d, 0x0c, 0x7e, 0xdc, 0xa0, 0x2c, 0xd8, 0x41, 0xc3, 0x45, 0x0f, 0x61, 0x8e, 0x6e, - 0x93, 0x6e, 0x7d, 0x8e, 0x12, 0xaf, 0x8d, 0x52, 0x00, 0xb7, 0x88, 0x27, 0x94, 0x80, 0x19, 0x04, - 0xa7, 0x46, 0x7b, 0x50, 0xd6, 0x4c, 0xd3, 0xf2, 0x34, 0xe6, 0xb5, 0xe7, 0x29, 0xb3, 0x77, 0x27, - 0x60, 0xb6, 0x11, 0x50, 0x31, 0x8e, 0x61, 0x3e, 0xe8, 0x43, 0x98, 0xa5, 0x6e, 0x9d, 0x7b, 0xe0, - 0xcb, 0x13, 0x1a, 0xad, 0xc2, 0xa8, 0xd0, 0x26, 0xcc, 0x1f, 0x19, 0xa6, 0x6e, 0x1d, 0xb9, 0xdc, - 0x1b, 0x5e, 0x4d, 0x32, 0xf8, 0x84, 0x21, 0x24, 0x58, 0x08, 0xca, 0xc6, 0x6d, 0x28, 0x87, 0x46, - 0x3c, 0x8d, 0xa5, 0x37, 0xee, 0x41, 0x2d, 0x3e, 0xbe, 0xa9, 0x56, 0xca, 0xef, 0xc2, 0xb2, 0x32, - 0x34, 0x03, 0xd1, 0x44, 0xb0, 0x7c, 0x07, 0xe6, 0xb8, 0xc5, 0x30, 0xb3, 0x95, 0xc7, 0x2b, 0x5a, - 0xe1, 0x14, 0xe1, 0xe8, 0xf7, 0x50, 0x33, 0xf5, 0x3e, 0x76, 0x78, 0xbf, 0x22, 0xfa, 0x7d, 0xcc, - 0xa0, 0xf2, 0x87, 0x70, 0x32, 0xd6, 0x39, 0x0f, 0xbe, 0xdf, 0x82, 0xaa, 0x6d, 0xe9, 0xaa, 0xcb, - 0xc0, 0x24, 0xb6, 0xe0, 0xbe, 0xcc, 0xf6, 0x71, 0x9b, 0x3a, 0x21, 0xef, 0x78, 0x96, 0x9d, 0x14, - 0x7e, 0x32, 0xf2, 0x3a, 0x9c, 0x8a, 0x93, 0xb3, 0xee, 0xe5, 0x8f, 0x60, 0x45, 0xc1, 0x03, 0xeb, - 0x39, 0x7e, 0x59, 0xd6, 0x0d, 0xa8, 0x27, 0x19, 0x70, 0xe6, 0x9f, 0xc1, 0x4a, 0x00, 0xed, 0x78, - 0x9a, 0x37, 0x74, 0xa7, 0x62, 0xce, 0x4f, 0x26, 0xfb, 0x96, 0xcb, 0xa6, 0xb3, 0xa8, 0x88, 0xa6, - 0xbc, 0x02, 0xb3, 0x6d, 0x4b, 0x6f, 0xb6, 0x51, 0x15, 0x72, 0x86, 0xcd, 0x89, 0x73, 0x86, 0x2d, - 0x1b, 0xe1, 0x3e, 0x5b, 0x2c, 0x42, 0x64, 0x5d, 0xc7, 0x51, 0xd1, 0x3d, 0xa8, 0x6a, 0xba, 0x6e, - 0x10, 0x73, 0xd2, 0xfa, 0xaa, 0x61, 0xb3, 0x03, 0x44, 0x79, 0x7d, 0x25, 0xd5, 0x00, 0x9a, 0x6d, - 0xa5, 0x12, 0xa0, 0x37, 0x6d, 0x57, 0x7e, 0x0c, 0x25, 0x3f, 0x0a, 0x23, 0xb1, 0x42, 0x34, 0xca, - 0x9a, 0x20, 0x66, 0xf3, 0x8f, 0x23, 0xbb, 0x89, 0x8d, 0x8e, 0x8b, 0x7c, 0x17, 0xc0, 0x77, 0xc8, - 0x22, 0x18, 0x3c, 0x33, 0x82, 0xb1, 0x12, 0x42, 0x97, 0x7f, 0x16, 0x71, 0xd3, 0x21, 0x25, 0xe8, - 0xbe, 0x12, 0xf4, 0x88, 0xdb, 0xce, 0xbd, 0x94, 0xdb, 0x7e, 0x1f, 0x66, 0x5d, 0x4f, 0xf3, 0x30, - 0x8f, 0xa6, 0x57, 0x47, 0x91, 0x13, 0x21, 0xb0, 0xc2, 0xf0, 0xd1, 0x59, 0x80, 0xae, 0x83, 0x35, - 0x0f, 0xeb, 0xaa, 0xc6, 0xf6, 0x98, 0xbc, 0x52, 0xe2, 0x90, 0x0d, 0x8f, 0xf8, 0x1b, 0x71, 0x22, - 0x98, 0xcd, 0xf2, 0x37, 0x19, 0x53, 0x1d, 0x9c, 0x0d, 0x7c, 0x9f, 0x37, 0x37, 0xa1, 0xcf, 0xe3, - 0x0c, 0xb8, 0xcf, 0x0b, 0x3c, 0xfa, 0xfc, 0x78, 0x8f, 0xce, 0x48, 0x27, 0xf1, 0xe8, 0xc5, 0xf1, - 0x1e, 0x9d, 0x33, 0x1b, 0xed, 0xd1, 0x53, 0xdc, 0x4f, 0x29, 0xcd, 0xfd, 0x7c, 0x9b, 0x6e, 0xf7, - 0x9f, 0x25, 0xa8, 0x27, 0xbd, 0x00, 0xf7, 0x7e, 0x77, 0x60, 0xce, 0xa5, 0x90, 0x49, 0x7c, 0x2f, - 0xa7, 0xe5, 0x14, 0xe8, 0x31, 0x14, 0x0c, 0xf3, 0xc0, 0xe2, 0x8b, 0xf6, 0xbd, 0x09, 0x28, 0x79, - 0xaf, 0x6b, 0x4d, 0xf3, 0xc0, 0x62, 0xda, 0xa4, 0x1c, 0x1a, 0xef, 0x43, 0xc9, 0x07, 0x4d, 0x35, - 0xb6, 0x1d, 0x58, 0x8e, 0xd9, 0x36, 0x3b, 0x00, 0xfa, 0x4b, 0x42, 0x9a, 0x6e, 0x49, 0xc8, 0x3f, - 0xcd, 0x85, 0x97, 0xec, 0x43, 0xa3, 0xef, 0x61, 0x27, 0xb1, 0x64, 0x3f, 0x10, 0xdc, 0xd9, 0x7a, - 0xbd, 0x34, 0x96, 0x3b, 0x3b, 0x53, 0xf1, 0x55, 0xf7, 0x05, 0x54, 0xa9, 0x51, 0xaa, 0x2e, 0xee, - 0xd3, 0xb8, 0x89, 0xc7, 0xb0, 0xdf, 0x1f, 0xc5, 0x86, 0x49, 0xc2, 0x4c, 0xbb, 0xc3, 0xe9, 0x98, - 0x06, 0x2b, 0xfd, 0x30, 0xac, 0x71, 0x1f, 0x50, 0x12, 0x69, 0x2a, 0x9d, 0x76, 0x88, 0x2f, 0x74, - 0xbd, 0xd4, 0x7d, 0xfa, 0x80, 0x8a, 0x31, 0x89, 0xad, 0x30, 0x81, 0x15, 0x4e, 0x21, 0xff, 0x67, - 0x1e, 0x20, 0x78, 0xf9, 0x7f, 0xc8, 0x09, 0xde, 0xf7, 0x1d, 0x10, 0x8b, 0x47, 0xaf, 0x8c, 0x62, - 0x9c, 0xea, 0x7a, 0x76, 0xa2, 0xae, 0x87, 0x45, 0xa6, 0xd7, 0x47, 0xb2, 0x99, 0xda, 0xe9, 0xcc, - 0x7f, 0xd7, 0x9c, 0xce, 0x13, 0x38, 0x15, 0x37, 0x22, 0xee, 0x71, 0xd6, 0x61, 0xd6, 0xf0, 0xf0, - 0x80, 0xe5, 0x11, 0x53, 0xd3, 0x10, 0x21, 0x22, 0x86, 0x2a, 0xff, 0x85, 0x04, 0xa5, 0xe6, 0x40, - 0xeb, 0xe1, 0x8e, 0x8d, 0xbb, 0xa4, 0x57, 0x83, 0x34, 0xb8, 0x24, 0xac, 0x81, 0x5a, 0x51, 0x35, - 0x33, 0xa7, 0xf4, 0x4e, 0x4a, 0x92, 0x43, 0xf0, 0x19, 0xad, 0xe5, 0x57, 0xd6, 0xc0, 0x3a, 0x14, - 0x7f, 0x80, 0x8f, 0x99, 0x3b, 0x9a, 0x90, 0x4e, 0xfe, 0x93, 0x02, 0xac, 0xd0, 0xed, 0x70, 0x53, - 0xa4, 0x15, 0x15, 0xec, 0x5a, 0x43, 0xa7, 0x8b, 0x5d, 0x6a, 0xa7, 0xf6, 0x50, 0xb5, 0xb1, 0x63, - 0x58, 0x3a, 0x4f, 0x6c, 0x95, 0xba, 0xf6, 0xb0, 0x4d, 0x01, 0xe8, 0x0c, 0x90, 0x86, 0xfa, 0xe5, - 0xd0, 0xe2, 0x4b, 0x28, 0xaf, 0x14, 0xbb, 0xf6, 0xf0, 0x77, 0x48, 0x5b, 0xd0, 0xba, 0x87, 0x9a, - 0x83, 0x5d, 0xba, 0x42, 0x18, 0x6d, 0x87, 0x02, 0xd0, 0x4d, 0x38, 0x39, 0xc0, 0x03, 0xcb, 0x39, - 0x56, 0xfb, 0xc6, 0xc0, 0xf0, 0x54, 0xc3, 0x54, 0xf7, 0x8f, 0x3d, 0xec, 0xf2, 0xd5, 0x80, 0xd8, - 0xcb, 0x27, 0xe4, 0x5d, 0xd3, 0x7c, 0x40, 0xde, 0x20, 0x19, 0x2a, 0x96, 0x35, 0x50, 0xdd, 0xae, - 0xe5, 0x60, 0x55, 0xd3, 0x7f, 0x4c, 0x23, 0x84, 0xbc, 0x52, 0xb6, 0xac, 0x41, 0x87, 0xc0, 0x36, - 0xf4, 0x1f, 0xa3, 0x37, 0xa1, 0xdc, 0xb5, 0x87, 0x2e, 0xf6, 0x54, 0xf2, 0x43, 0x03, 0x80, 0x92, - 0x02, 0x0c, 0xb4, 0x69, 0x0f, 0xdd, 0x10, 0xc2, 0x80, 0x18, 0xc4, 0x7c, 0x18, 0xe1, 0x29, 0x1e, - 0xd0, 0x0c, 0xda, 0xe1, 0xb0, 0x87, 0x6d, 0xad, 0x87, 0x99, 0x68, 0x62, 0xe7, 0x4e, 0xc9, 0xa0, - 0x3d, 0xe6, 0x88, 0x54, 0x4c, 0xa5, 0x7a, 0x18, 0x6e, 0xba, 0xa8, 0x0d, 0xf3, 0x43, 0xd3, 0x38, - 0x30, 0xb0, 0x5e, 0x2f, 0x51, 0x0e, 0xb7, 0x32, 0x02, 0x91, 0xa4, 0xe6, 0xd7, 0xf6, 0x18, 0x21, - 0xcf, 0x19, 0x70, 0x36, 0xe8, 0x0e, 0x34, 0xb8, 0xd2, 0xdc, 0x23, 0xcd, 0x8e, 0x6b, 0x0e, 0xa8, - 0x3a, 0x4e, 0x31, 0x8c, 0xce, 0x91, 0x66, 0x87, 0xb5, 0xd7, 0xb8, 0x03, 0x0b, 0x61, 0xa6, 0x53, - 0xd9, 0xd5, 0x03, 0xa8, 0x44, 0x86, 0x4a, 0x66, 0x9e, 0x2a, 0xc8, 0x35, 0x7e, 0x22, 0x96, 0x44, - 0x91, 0x00, 0x3a, 0xc6, 0x4f, 0x68, 0x26, 0x94, 0x4a, 0x46, 0xf9, 0x14, 0x14, 0xd6, 0x90, 0x35, - 0xa8, 0x44, 0x12, 0x8e, 0x08, 0x41, 0x81, 0x66, 0x16, 0x79, 0xba, 0x82, 0x3c, 0x13, 0x98, 0x63, - 0xf5, 0x85, 0x04, 0xf4, 0x99, 0xc0, 0x68, 0x0a, 0x8b, 0x1d, 0xfe, 0xe9, 0x33, 0xed, 0x02, 0x3f, - 0xe7, 0x19, 0xeb, 0x92, 0xc2, 0x1a, 0xb2, 0x0e, 0xb0, 0xa9, 0xd9, 0xda, 0xbe, 0xd1, 0x37, 0xbc, - 0x63, 0x74, 0x15, 0x6a, 0x9a, 0xae, 0xab, 0x5d, 0x01, 0x31, 0xb0, 0xa8, 0x23, 0x2c, 0x6a, 0xba, - 0xbe, 0x19, 0x02, 0xa3, 0xb7, 0x61, 0x49, 0x77, 0x2c, 0x3b, 0x8a, 0xcb, 0x0a, 0x0b, 0x35, 0xf2, - 0x22, 0x8c, 0x2c, 0xff, 0x66, 0x0e, 0xce, 0x46, 0xa7, 0x2d, 0x9e, 0xd4, 0xbd, 0x0f, 0x0b, 0xb1, - 0x5e, 0x33, 0x92, 0x9f, 0x81, 0xb4, 0x4a, 0x84, 0x22, 0x96, 0xa4, 0xcc, 0x25, 0x92, 0x94, 0xa9, - 0x69, 0xe3, 0xfc, 0x6b, 0x4d, 0x1b, 0x17, 0x5e, 0x4b, 0xda, 0x78, 0xf6, 0xd5, 0xd2, 0xc6, 0x0b, - 0x53, 0xa6, 0x8d, 0x2f, 0xd1, 0x6d, 0x4a, 0xf4, 0x4e, 0x93, 0x4a, 0xcc, 0x05, 0x54, 0xfc, 0x3e, - 0x4c, 0x51, 0xc0, 0x8a, 0xa5, 0x97, 0xe7, 0xa7, 0x49, 0x2f, 0x17, 0x33, 0xd3, 0xcb, 0xe7, 0x61, - 0xc1, 0xb4, 0x54, 0x13, 0x1f, 0xa9, 0x64, 0xba, 0xdc, 0x7a, 0x99, 0xcd, 0x9d, 0x69, 0xb5, 0xf0, - 0x51, 0x9b, 0x40, 0xd0, 0x2a, 0x2c, 0x0c, 0x34, 0xf7, 0x19, 0xd6, 0x69, 0x6e, 0xd7, 0xad, 0x57, - 0xa8, 0x9d, 0x95, 0x19, 0xac, 0x4d, 0x40, 0xe8, 0x22, 0xf8, 0x72, 0x70, 0xa4, 0x2a, 0x45, 0xaa, - 0x08, 0x28, 0x43, 0x0b, 0xa5, 0xaa, 0x17, 0x5f, 0x29, 0x55, 0x5d, 0x9b, 0x3e, 0x55, 0x7d, 0x1d, - 0x6a, 0xe2, 0x59, 0xe4, 0xaa, 0xd9, 0x31, 0x84, 0xa6, 0xa9, 0x17, 0xc5, 0x3b, 0x91, 0x8f, 0xce, - 0xca, 0x6c, 0xc3, 0xc8, 0xcc, 0xf6, 0x5f, 0x4b, 0xb0, 0x1c, 0x5d, 0x6a, 0x3c, 0x71, 0xf7, 0x08, - 0x4a, 0x8e, 0xf0, 0x95, 0x7c, 0x79, 0x5d, 0x9d, 0xd8, 0xb9, 0x2a, 0x01, 0x2d, 0xfa, 0x61, 0x66, - 0xbe, 0xf8, 0xc6, 0x38, 0x7e, 0xe3, 0x32, 0xc6, 0xf2, 0x1f, 0x4a, 0x70, 0x96, 0xe7, 0xc6, 0x32, - 0xaa, 0x3f, 0x29, 0xe6, 0x2a, 0x65, 0x98, 0x6b, 0xd7, 0xc1, 0x3a, 0x36, 0x3d, 0x43, 0xeb, 0xab, - 0xae, 0x8d, 0xbb, 0x22, 0xe3, 0x14, 0x80, 0x69, 0x98, 0xb2, 0x0a, 0x0b, 0xac, 0x18, 0xe8, 0x58, - 0x5d, 0xec, 0xba, 0xbc, 0xe6, 0x57, 0xa6, 0xf5, 0x40, 0x06, 0x92, 0x87, 0xb0, 0x92, 0x91, 0xb0, - 0x4b, 0x55, 0x86, 0x94, 0xa5, 0x8c, 0x91, 0x23, 0x4b, 0x2a, 0xe3, 0x8f, 0x24, 0x78, 0x93, 0x93, - 0x64, 0xfa, 0xcd, 0x6f, 0x43, 0x1d, 0xbf, 0x94, 0xe0, 0x54, 0x5c, 0x2e, 0xae, 0x8e, 0x66, 0xd2, - 0xc8, 0xde, 0xce, 0xd4, 0xc3, 0x68, 0x33, 0xfb, 0x22, 0xd3, 0xcc, 0x6e, 0x8e, 0xe7, 0x38, 0x56, - 0xb7, 0x7f, 0x2e, 0xc1, 0xe9, 0x4c, 0x31, 0x62, 0x81, 0x98, 0x14, 0x0f, 0xc4, 0x78, 0x10, 0xd7, - 0xb5, 0x86, 0xa6, 0x17, 0x0a, 0xe2, 0x36, 0x69, 0x0d, 0x9a, 0x45, 0x4b, 0xea, 0x40, 0x7b, 0x61, - 0x0c, 0x86, 0x03, 0x1e, 0xc5, 0x11, 0x76, 0x4f, 0x19, 0xe4, 0x25, 0xc2, 0x38, 0x79, 0x03, 0x96, - 0x7c, 0x29, 0x47, 0xd6, 0x2e, 0x42, 0xb5, 0x88, 0x5c, 0xb4, 0x16, 0x61, 0xc2, 0xdc, 0x16, 0x7e, - 0x6e, 0x74, 0xf1, 0x6b, 0x29, 0x92, 0x9f, 0x87, 0xb2, 0x8d, 0x9d, 0x81, 0xe1, 0xba, 0xfe, 0x36, - 0x5a, 0x52, 0xc2, 0x20, 0xf9, 0xdf, 0xe7, 0x60, 0x31, 0x6e, 0x1d, 0x1f, 0x25, 0x4a, 0x1f, 0x17, - 0x52, 0x36, 0xf8, 0xf8, 0x40, 0x43, 0xa7, 0xc7, 0x9b, 0xe2, 0x48, 0x91, 0xcb, 0xca, 0xf0, 0xf9, - 0xc7, 0x06, 0x71, 0xde, 0xa8, 0xc3, 0x7c, 0xd7, 0x1a, 0x0c, 0x34, 0x53, 0x17, 0x77, 0x1b, 0x78, - 0x93, 0xe8, 0x4f, 0x73, 0x7a, 0x44, 0xed, 0x04, 0x4c, 0x9f, 0xc9, 0xe4, 0x1d, 0x59, 0xce, 0x33, - 0xc3, 0xa4, 0x25, 0x14, 0xba, 0x15, 0x97, 0x14, 0xe0, 0xa0, 0x2d, 0xc3, 0x41, 0x6b, 0x50, 0xc0, - 0xe6, 0x73, 0x71, 0x3c, 0x4c, 0xb9, 0xfc, 0x20, 0x0e, 0x13, 0x0a, 0xc5, 0x43, 0x37, 0x60, 0x6e, - 0x40, 0xcc, 0x42, 0x24, 0xc6, 0x56, 0x32, 0xee, 0x00, 0x28, 0x1c, 0x0d, 0xad, 0xc3, 0xbc, 0x4e, - 0xe7, 0x49, 0xc4, 0xd0, 0xf5, 0x94, 0xc2, 0x0c, 0x45, 0x50, 0x04, 0x22, 0xda, 0xf6, 0x0f, 0xbf, - 0xa5, 0xac, 0x53, 0x6b, 0x6c, 0x2a, 0x52, 0x4f, 0xc0, 0xbb, 0xd1, 0xa3, 0x19, 0x50, 0x5e, 0xeb, - 0xe3, 0x79, 0x8d, 0x3e, 0x06, 0x9f, 0x86, 0x62, 0xdf, 0xea, 0x31, 0x33, 0x2a, 0xb3, 0x6b, 0x33, - 0x7d, 0xab, 0x47, 0xad, 0x68, 0x19, 0x66, 0x5d, 0x4f, 0x37, 0x4c, 0x1a, 0xb3, 0x14, 0x15, 0xd6, - 0x20, 0x8b, 0x8f, 0x3e, 0xa8, 0x96, 0xd9, 0xc5, 0xf5, 0x0a, 0x7d, 0x55, 0xa2, 0x90, 0x1d, 0xb3, - 0x4b, 0x0f, 0x69, 0x9e, 0x77, 0x5c, 0xaf, 0x52, 0x38, 0x79, 0x44, 0x1f, 0x88, 0xdc, 0xe5, 0x62, - 0x56, 0x9e, 0x27, 0x6d, 0x43, 0x14, 0xa9, 0xcb, 0x07, 0x41, 0xb9, 0x86, 0xed, 0xe9, 0x57, 0xc6, - 0xbb, 0x97, 0xef, 0x50, 0xb5, 0xe6, 0x6f, 0x25, 0x38, 0xb5, 0x49, 0xd3, 0x20, 0x21, 0x3f, 0x36, - 0x4d, 0xed, 0xe0, 0xb6, 0x5f, 0xd6, 0xc9, 0xcc, 0xc7, 0xc7, 0xc7, 0x2d, 0xaa, 0x3a, 0x4d, 0xa8, - 0x0a, 0xe6, 0x9c, 0x45, 0x7e, 0xe2, 0xca, 0x50, 0xc5, 0x0d, 0x37, 0xe5, 0x0f, 0x60, 0x25, 0x31, - 0x0a, 0x9e, 0x89, 0x58, 0x85, 0x85, 0xc0, 0x5f, 0xf9, 0x83, 0x28, 0xfb, 0xb0, 0xa6, 0x2e, 0xdf, - 0x81, 0x93, 0x1d, 0x4f, 0x73, 0xbc, 0x84, 0x0a, 0x26, 0xa0, 0xa5, 0x35, 0x9f, 0x28, 0x2d, 0x2f, - 0xcb, 0x74, 0x60, 0xb9, 0xe3, 0x59, 0xf6, 0x4b, 0x30, 0x25, 0x5e, 0x87, 0x8c, 0xdf, 0x1a, 0x8a, - 0xfd, 0x41, 0x34, 0xe5, 0x15, 0x56, 0xa1, 0x4a, 0xf6, 0x76, 0x17, 0x4e, 0xb1, 0x02, 0xd1, 0xcb, - 0x0c, 0xe2, 0xb4, 0x28, 0x4f, 0x25, 0xf9, 0x3e, 0x85, 0x13, 0xc1, 0xb6, 0x18, 0xa4, 0x5e, 0x6f, - 0x45, 0x53, 0xaf, 0xe7, 0x47, 0xcc, 0x7a, 0x24, 0xf3, 0xfa, 0xa7, 0xb9, 0x90, 0x5f, 0xcf, 0x48, - 0xbc, 0xde, 0x8d, 0x26, 0x5e, 0x2f, 0x8e, 0xe3, 0x1d, 0xc9, 0xbb, 0x26, 0xad, 0x36, 0x9f, 0x62, - 0xb5, 0x9f, 0x27, 0xb2, 0xb3, 0x85, 0xac, 0xf4, 0x76, 0x4c, 0xda, 0xdf, 0x4a, 0x72, 0x56, 0x61, - 0xc9, 0x59, 0xbf, 0x6b, 0xbf, 0x9e, 0x77, 0x3b, 0x96, 0x9c, 0x5d, 0x1d, 0x2b, 0xaf, 0x9f, 0x9b, - 0xfd, 0xab, 0x02, 0x94, 0xfc, 0x77, 0x09, 0x9d, 0x27, 0xd5, 0x96, 0x4b, 0x51, 0x5b, 0x78, 0x07, - 0xce, 0xbf, 0xd2, 0x0e, 0x5c, 0x98, 0x78, 0x07, 0x3e, 0x03, 0x25, 0xfa, 0x40, 0x6f, 0xc0, 0xb0, - 0x1d, 0xb5, 0x48, 0x01, 0x0a, 0x3e, 0x08, 0xcc, 0x70, 0x6e, 0x2a, 0x33, 0x8c, 0xa5, 0x83, 0xe7, - 0xe3, 0xe9, 0xe0, 0x8f, 0xfc, 0x1d, 0x91, 0x6d, 0xa2, 0x97, 0x47, 0xf0, 0x4d, 0xdd, 0x0b, 0x63, - 0x69, 0xca, 0x52, 0x56, 0x9a, 0x32, 0xe0, 0x32, 0x3a, 0x4d, 0xf9, 0x2d, 0xee, 0x10, 0x7b, 0x2c, - 0xc7, 0x1b, 0xb6, 0x45, 0xee, 0x59, 0xef, 0x02, 0xf8, 0x4e, 0x44, 0x24, 0x7a, 0xcf, 0x8c, 0x18, - 0xa3, 0x12, 0x42, 0x27, 0x6c, 0x23, 0x53, 0x13, 0xd4, 0xac, 0x27, 0xf3, 0x8f, 0x19, 0x05, 0xeb, - 0xff, 0x99, 0x0d, 0xf9, 0x97, 0x8c, 0x5a, 0xec, 0x47, 0x89, 0x32, 0xc4, 0x94, 0x56, 0x7c, 0x2b, - 0x5a, 0x85, 0x78, 0x49, 0xab, 0x4b, 0x14, 0x21, 0x68, 0xe4, 0xa2, 0x39, 0xfc, 0x35, 0x4b, 0xb5, - 0x96, 0x38, 0x64, 0x83, 0x9e, 0x0c, 0x0e, 0x0c, 0xd3, 0x70, 0x0f, 0xd9, 0xfb, 0x39, 0x76, 0x32, - 0x10, 0xa0, 0x0d, 0x9a, 0x22, 0xc4, 0x2f, 0x0c, 0x4f, 0xed, 0x5a, 0x3a, 0xa6, 0x36, 0x3d, 0xab, - 0x14, 0x09, 0x60, 0xd3, 0xd2, 0x71, 0xb0, 0xf2, 0x8a, 0x2f, 0xb7, 0xf2, 0x4a, 0xb1, 0x95, 0x77, - 0x0a, 0xe6, 0x1c, 0xac, 0xb9, 0x96, 0xc9, 0x12, 0x0a, 0x0a, 0x6f, 0x91, 0xa9, 0x19, 0x60, 0xd7, - 0x25, 0x3d, 0xf1, 0x70, 0x8d, 0x37, 0x43, 0x61, 0xe6, 0xc2, 0xd8, 0x30, 0x73, 0x44, 0x8d, 0x37, - 0x16, 0x66, 0x56, 0xc6, 0x86, 0x99, 0x13, 0x95, 0x78, 0x83, 0x40, 0xbb, 0x3a, 0x59, 0xa0, 0x1d, - 0x8e, 0x4b, 0x17, 0x23, 0x71, 0xe9, 0xb7, 0xb9, 0x58, 0x7f, 0x2d, 0xc1, 0x4a, 0x62, 0x59, 0xf1, - 0xe5, 0x7a, 0x3b, 0x56, 0x04, 0x5e, 0x1d, 0xab, 0x33, 0xbf, 0x06, 0xfc, 0x28, 0x52, 0x03, 0x7e, - 0x77, 0x3c, 0xe1, 0x6b, 0x2f, 0x01, 0xff, 0x77, 0x0e, 0xde, 0xdc, 0xb3, 0xf5, 0x58, 0x84, 0xc7, - 0x8f, 0xfd, 0x93, 0x3b, 0x8e, 0x8f, 0x44, 0xac, 0x9f, 0x9b, 0x36, 0x83, 0xc5, 0xc3, 0xfd, 0xed, - 0x20, 0xdc, 0xcf, 0x4f, 0x9f, 0x9f, 0x10, 0xb4, 0x48, 0x8f, 0x1a, 0x31, 0x0b, 0x3e, 0x1e, 0x24, - 0x59, 0x8d, 0x19, 0xf2, 0x37, 0x5c, 0xdc, 0x92, 0xe1, 0x7c, 0xb6, 0x00, 0x3c, 0x3e, 0xfc, 0x11, - 0x2c, 0x6e, 0xbf, 0xc0, 0xdd, 0xce, 0xb1, 0xd9, 0x9d, 0x62, 0x1e, 0x6a, 0x90, 0xef, 0x0e, 0x74, - 0x9e, 0xf0, 0x27, 0x8f, 0xe1, 0x90, 0x37, 0x1f, 0x0d, 0x79, 0x55, 0xa8, 0x05, 0x3d, 0x70, 0x5b, - 0x3e, 0x45, 0x6c, 0x59, 0x27, 0xc8, 0x84, 0xf9, 0x82, 0xc2, 0x5b, 0x1c, 0x8e, 0x1d, 0x76, 0x3f, - 0x8c, 0xc1, 0xb1, 0xe3, 0x44, 0x5d, 0x63, 0x3e, 0xea, 0x1a, 0xe5, 0x3f, 0x96, 0xa0, 0x4c, 0x7a, - 0x78, 0x25, 0xf9, 0xf9, 0xb9, 0x32, 0x1f, 0x9c, 0x2b, 0xfd, 0xe3, 0x69, 0x21, 0x7c, 0x3c, 0x0d, - 0x24, 0x9f, 0xa5, 0xe0, 0xa4, 0xe4, 0x73, 0x3e, 0x1c, 0x3b, 0x8e, 0x7c, 0x1e, 0x16, 0x98, 0x6c, - 0x7c, 0xe4, 0x35, 0xc8, 0x0f, 0x9d, 0xbe, 0x98, 0xbf, 0xa1, 0xd3, 0x97, 0x7f, 0x5f, 0x82, 0xca, - 0x86, 0xe7, 0x69, 0xdd, 0xc3, 0x29, 0x06, 0xe0, 0x0b, 0x97, 0x0b, 0x0b, 0x97, 0x1c, 0x44, 0x20, - 0x6e, 0x21, 0x43, 0xdc, 0xd9, 0x88, 0xb8, 0x32, 0x54, 0x85, 0x2c, 0x99, 0x02, 0xb7, 0x00, 0xb5, - 0x2d, 0xc7, 0x7b, 0x68, 0x39, 0x47, 0x9a, 0xa3, 0x4f, 0x77, 0xdc, 0x44, 0x50, 0xe0, 0xdf, 0x7f, - 0xe4, 0xaf, 0xcc, 0x2a, 0xf4, 0x59, 0xbe, 0x0c, 0x27, 0x22, 0xfc, 0x32, 0x3b, 0xbe, 0x0f, 0x65, - 0xba, 0xc9, 0xf1, 0x73, 0xc7, 0xcd, 0x70, 0x85, 0x79, 0xa2, 0x2d, 0x51, 0xfe, 0xff, 0xb0, 0x44, - 0x82, 0x21, 0x0a, 0xf7, 0xfd, 0xce, 0xf7, 0x63, 0x41, 0xf9, 0xd9, 0x0c, 0x46, 0xb1, 0x80, 0xfc, - 0x37, 0x12, 0xcc, 0x52, 0x78, 0x22, 0x40, 0x39, 0x03, 0x25, 0x07, 0xdb, 0x96, 0xea, 0x69, 0x3d, - 0xff, 0x6b, 0x1b, 0x02, 0xd8, 0xd5, 0x7a, 0xb4, 0x98, 0x41, 0x5f, 0xea, 0x46, 0x0f, 0xbb, 0x9e, - 0xf8, 0xe4, 0xa6, 0x4c, 0x60, 0x5b, 0x0c, 0x44, 0x94, 0x44, 0xcb, 0x84, 0x05, 0x5a, 0x0d, 0xa4, - 0xcf, 0x68, 0x8d, 0x5d, 0x4b, 0x9e, 0xa4, 0x3a, 0x44, 0x2f, 0x2d, 0x37, 0xa0, 0x18, 0x2b, 0xe8, - 0xf8, 0x6d, 0x74, 0x03, 0x0a, 0x34, 0x05, 0x3c, 0x3f, 0x5e, 0x6f, 0x14, 0x51, 0xde, 0x06, 0x14, - 0x56, 0x1b, 0x9f, 0xa0, 0x1b, 0x30, 0x47, 0xb5, 0x2a, 0x62, 0xc7, 0x95, 0x0c, 0x46, 0x0a, 0x47, - 0x93, 0x35, 0x40, 0x8c, 0x73, 0x24, 0x5e, 0x9c, 0x7e, 0x1a, 0x47, 0xc4, 0x8f, 0x7f, 0x23, 0xc1, - 0x89, 0x48, 0x1f, 0x5c, 0xd6, 0xeb, 0xd1, 0x4e, 0x32, 0x45, 0xe5, 0x1d, 0x6c, 0x46, 0x36, 0xcc, - 0x1b, 0x59, 0x22, 0x7d, 0x43, 0x9b, 0xe5, 0xdf, 0x4b, 0x00, 0x1b, 0x43, 0xef, 0x90, 0xe7, 0x4d, - 0xc3, 0x53, 0x29, 0xc5, 0xa6, 0xb2, 0x01, 0x45, 0x5b, 0x73, 0xdd, 0x23, 0xcb, 0x11, 0x27, 0x3e, - 0xbf, 0x4d, 0x33, 0x9c, 0x43, 0xef, 0x50, 0x94, 0x81, 0xc9, 0x33, 0xba, 0x08, 0x55, 0xf6, 0x49, - 0x98, 0xaa, 0xe9, 0xba, 0x83, 0x5d, 0x97, 0xd7, 0x83, 0x2b, 0x0c, 0xba, 0xc1, 0x80, 0x04, 0xcd, - 0xa0, 0x65, 0x01, 0xef, 0x58, 0xf5, 0xac, 0x67, 0xd8, 0xe4, 0x27, 0xb7, 0x8a, 0x80, 0xee, 0x12, - 0x20, 0xab, 0xba, 0xf5, 0x0c, 0xd7, 0x73, 0x04, 0x9a, 0xa8, 0x1d, 0x72, 0x28, 0x45, 0x23, 0x93, - 0x52, 0x6b, 0x0f, 0xfb, 0x7d, 0xa6, 0xe2, 0x97, 0x9f, 0xf6, 0xef, 0xf1, 0x01, 0xe5, 0xb2, 0x16, - 0x41, 0xa0, 0x34, 0x3e, 0xdc, 0xd7, 0x98, 0xa2, 0xfa, 0x1e, 0x2c, 0x85, 0xc6, 0xc0, 0xcd, 0x2a, - 0x12, 0x62, 0x4b, 0xd1, 0x10, 0x5b, 0x7e, 0x04, 0x88, 0x65, 0x65, 0x5e, 0x71, 0xdc, 0xf2, 0x49, - 0x38, 0x11, 0x61, 0xc4, 0xb7, 0xee, 0x6b, 0x50, 0xe1, 0xd7, 0x39, 0xb9, 0xa1, 0x9c, 0x86, 0x22, - 0x71, 0xc1, 0x5d, 0x43, 0x17, 0x77, 0x04, 0xe6, 0x6d, 0x4b, 0xdf, 0x34, 0x74, 0x47, 0xfe, 0x04, - 0x2a, 0xfc, 0xbb, 0x12, 0x8e, 0xfb, 0x10, 0xaa, 0xfc, 0xf2, 0xa7, 0x1a, 0xb9, 0xd6, 0x9d, 0xf6, - 0x95, 0x57, 0xb8, 0x13, 0xa5, 0x62, 0x86, 0x9b, 0xb2, 0x0e, 0x0d, 0x16, 0x63, 0x44, 0xd8, 0x8b, - 0xc1, 0x3e, 0x04, 0x71, 0xdb, 0x69, 0x6c, 0x2f, 0x51, 0xfa, 0x8a, 0x13, 0x6e, 0xca, 0x67, 0xe1, - 0x4c, 0x6a, 0x2f, 0x5c, 0x13, 0x36, 0xd4, 0x82, 0x17, 0xec, 0xee, 0xb1, 0x7f, 0x09, 0x42, 0x0a, - 0x5d, 0x82, 0x38, 0xe5, 0x87, 0xd0, 0x39, 0xb1, 0xeb, 0xd1, 0xf8, 0x38, 0x38, 0x0c, 0xe5, 0xb3, - 0x0e, 0x43, 0x85, 0xc8, 0x61, 0x48, 0xee, 0xf8, 0xfa, 0xe4, 0x87, 0xd4, 0x07, 0xf4, 0x30, 0xcd, - 0xfa, 0x16, 0x0e, 0x51, 0x1e, 0x35, 0x4a, 0x86, 0xaa, 0x84, 0xa8, 0xe4, 0xab, 0x50, 0x89, 0xba, - 0xc6, 0x90, 0x9f, 0x93, 0x12, 0x7e, 0xae, 0x1a, 0x73, 0x71, 0xef, 0xc7, 0xce, 0x07, 0xd9, 0x3a, - 0x8e, 0x9d, 0x0e, 0xee, 0x45, 0x9c, 0xdd, 0xb5, 0x94, 0x9a, 0xf6, 0x37, 0xe4, 0xe7, 0x96, 0xf9, - 0x7e, 0xf0, 0xd0, 0x25, 0xf4, 0x7c, 0xd0, 0xf2, 0x05, 0x28, 0xef, 0x65, 0x7d, 0x25, 0x58, 0x10, - 0x77, 0xb0, 0x6e, 0xc1, 0xf2, 0x43, 0xa3, 0x8f, 0xdd, 0x63, 0xd7, 0xc3, 0x83, 0x26, 0x75, 0x4a, - 0x07, 0x06, 0x76, 0xd0, 0x39, 0x00, 0x7a, 0xc0, 0xb3, 0x2d, 0xc3, 0xff, 0xde, 0x29, 0x04, 0x91, - 0xff, 0x43, 0x82, 0xc5, 0x80, 0x70, 0x8f, 0x1e, 0x6c, 0xdf, 0x80, 0x12, 0x19, 0xaf, 0xeb, 0x69, - 0x03, 0x5b, 0x54, 0xfb, 0x7c, 0x00, 0xba, 0x0b, 0xb3, 0x07, 0xae, 0x48, 0xa8, 0xa5, 0x96, 0x17, - 0xd2, 0x04, 0x51, 0x0a, 0x07, 0x6e, 0x53, 0x47, 0x1f, 0x00, 0x0c, 0x5d, 0xac, 0xf3, 0x0a, 0x5f, - 0x3e, 0x2b, 0xbc, 0xd8, 0x0b, 0xdf, 0xed, 0x20, 0x04, 0xec, 0xfa, 0xd6, 0x3d, 0x28, 0x1b, 0xa6, - 0xa5, 0x63, 0x5a, 0xdd, 0xd5, 0x79, 0xce, 0x6d, 0x0c, 0x39, 0x30, 0x8a, 0x3d, 0x17, 0xeb, 0x32, - 0xe6, 0x7b, 0xa1, 0xd0, 0x2f, 0x37, 0x94, 0x16, 0x2c, 0x31, 0xa7, 0x75, 0xe0, 0x0b, 0x2e, 0x2c, - 0x76, 0x75, 0xd4, 0xe8, 0xa8, 0xb6, 0x94, 0x9a, 0xc1, 0x63, 0x21, 0x41, 0x2a, 0xdf, 0x81, 0x93, - 0x91, 0xf3, 0xe3, 0x14, 0x07, 0x3a, 0xb9, 0x1d, 0x4b, 0x23, 0x05, 0xe6, 0xcc, 0x93, 0x34, 0xc2, - 0x9a, 0xc7, 0x25, 0x69, 0x5c, 0x96, 0xa4, 0x71, 0xe5, 0xcf, 0xe1, 0x74, 0x24, 0xdf, 0x15, 0x91, - 0xe8, 0x5e, 0x2c, 0xd4, 0xbb, 0x34, 0x8e, 0x6b, 0x2c, 0xe6, 0xfb, 0x2f, 0x09, 0x96, 0xd3, 0x10, - 0x5e, 0x32, 0x1f, 0xfb, 0xa3, 0x8c, 0x4b, 0xc6, 0xb7, 0x27, 0x13, 0xeb, 0xb7, 0x92, 0xcb, 0xde, - 0x85, 0x46, 0x9a, 0x3e, 0x93, 0xb3, 0x94, 0x9f, 0x66, 0x96, 0x7e, 0x9e, 0x0f, 0xd5, 0x25, 0x36, - 0x3c, 0xcf, 0x31, 0xf6, 0x87, 0xc4, 0xe4, 0x5f, 0x7b, 0xae, 0xaf, 0xe9, 0x67, 0xad, 0x98, 0x6a, - 0x6f, 0x8e, 0x20, 0x0f, 0xe4, 0x48, 0xcd, 0x5c, 0x7d, 0x9a, 0x76, 0xe8, 0xbf, 0x35, 0x19, 0xbf, - 0xef, 0x6c, 0x7a, 0xf8, 0xe7, 0x39, 0xa8, 0x46, 0xa7, 0x08, 0x6d, 0x03, 0x68, 0xbe, 0xe4, 0x7c, - 0xa1, 0x5c, 0x9c, 0x68, 0x98, 0x4a, 0x88, 0x10, 0xbd, 0x03, 0xf9, 0xae, 0x3d, 0xe4, 0xb3, 0x96, - 0x52, 0x2a, 0xdf, 0xb4, 0x87, 0xcc, 0xa3, 0x10, 0x34, 0x72, 0x08, 0x63, 0x37, 0x1f, 0xb2, 0xbd, - 0xe4, 0x53, 0xfa, 0x9e, 0xd1, 0x70, 0x64, 0xf4, 0x18, 0xaa, 0x47, 0x8e, 0xe1, 0x69, 0xfb, 0x7d, - 0xac, 0xf6, 0xb5, 0x63, 0xec, 0x70, 0x2f, 0x39, 0x81, 0x23, 0xab, 0x08, 0xc2, 0x27, 0x84, 0x4e, - 0xfe, 0x3d, 0x28, 0x0a, 0x89, 0xc6, 0xec, 0x08, 0xbb, 0xb0, 0x32, 0x24, 0x68, 0x2a, 0xbd, 0x56, - 0x6b, 0x6a, 0xa6, 0xa5, 0xba, 0x98, 0x6c, 0xe3, 0xe2, 0x9b, 0xa6, 0x31, 0x2e, 0x7a, 0x99, 0x52, - 0x6f, 0x5a, 0x0e, 0x6e, 0x69, 0xa6, 0xd5, 0x61, 0xa4, 0xf2, 0x73, 0x28, 0x87, 0x06, 0x38, 0x46, - 0x84, 0x26, 0x2c, 0x89, 0x8b, 0x0a, 0x2e, 0xf6, 0xf8, 0xf6, 0x32, 0x51, 0xe7, 0x8b, 0x9c, 0xae, - 0x83, 0x3d, 0x76, 0xb9, 0xe4, 0x1e, 0x9c, 0x56, 0xb0, 0x65, 0x63, 0xd3, 0x9f, 0xcf, 0x27, 0x56, - 0x6f, 0x0a, 0x0f, 0xfe, 0x06, 0x34, 0xd2, 0xe8, 0x99, 0x7f, 0xb8, 0x76, 0x09, 0x8a, 0xe2, 0xff, - 0x20, 0xd0, 0x3c, 0xe4, 0x77, 0x37, 0xdb, 0xb5, 0x19, 0xf2, 0xb0, 0xb7, 0xd5, 0xae, 0x49, 0xa8, - 0x08, 0x85, 0xce, 0xe6, 0x6e, 0xbb, 0x96, 0xbb, 0x36, 0x80, 0x5a, 0xfc, 0xcf, 0x10, 0xd0, 0x0a, - 0x9c, 0x68, 0x2b, 0x3b, 0xed, 0x8d, 0x47, 0x1b, 0xbb, 0xcd, 0x9d, 0x96, 0xda, 0x56, 0x9a, 0x1f, - 0x6f, 0xec, 0x6e, 0xd7, 0x66, 0xd0, 0x2a, 0x9c, 0x0d, 0xbf, 0x78, 0xbc, 0xd3, 0xd9, 0x55, 0x77, - 0x77, 0xd4, 0xcd, 0x9d, 0xd6, 0xee, 0x46, 0xb3, 0xb5, 0xad, 0xd4, 0x24, 0x74, 0x16, 0x4e, 0x87, - 0x51, 0x1e, 0x34, 0xb7, 0x9a, 0xca, 0xf6, 0x26, 0x79, 0xde, 0x78, 0x52, 0xcb, 0x5d, 0xfb, 0x10, - 0x2a, 0x91, 0xff, 0x2e, 0x20, 0x22, 0xb5, 0x77, 0xb6, 0x6a, 0x33, 0xa8, 0x02, 0xa5, 0x30, 0x9f, - 0x22, 0x14, 0x5a, 0x3b, 0x5b, 0xdb, 0xb5, 0x1c, 0x02, 0x98, 0xdb, 0xdd, 0x50, 0x1e, 0x6d, 0xef, - 0xd6, 0xf2, 0xd7, 0xee, 0xc0, 0x62, 0xec, 0x3b, 0x05, 0xb4, 0x04, 0x95, 0xce, 0x46, 0x6b, 0xeb, - 0xc1, 0xce, 0xa7, 0xaa, 0xb2, 0xbd, 0xb1, 0xf5, 0x59, 0x6d, 0x06, 0x2d, 0x43, 0x4d, 0x80, 0x5a, - 0x3b, 0xbb, 0x0c, 0x2a, 0x5d, 0x7b, 0x16, 0x5b, 0x6f, 0x18, 0x9d, 0x84, 0x25, 0xbf, 0x4b, 0x75, - 0x53, 0xd9, 0xde, 0xd8, 0xdd, 0x26, 0x92, 0x44, 0xc0, 0xca, 0x5e, 0xab, 0xd5, 0x6c, 0x3d, 0xaa, - 0x49, 0x84, 0x6b, 0x00, 0xde, 0xfe, 0xb4, 0x49, 0x90, 0x73, 0x51, 0xe4, 0xbd, 0xd6, 0x0f, 0x5a, - 0x3b, 0x9f, 0xb4, 0x6a, 0xf9, 0xf5, 0x5f, 0x2e, 0xf9, 0xdf, 0x93, 0x77, 0xb0, 0x43, 0xef, 0xff, - 0xb4, 0x61, 0x5e, 0xfc, 0xd7, 0x48, 0x8a, 0xb7, 0x8e, 0xfe, 0x43, 0x4a, 0x63, 0x75, 0x04, 0x06, - 0x8f, 0xbd, 0x67, 0xd0, 0x3e, 0x8d, 0x85, 0x43, 0xdf, 0x8d, 0x5c, 0x4a, 0x8d, 0x3c, 0x13, 0x9f, - 0xaa, 0x34, 0x2e, 0x8f, 0xc5, 0xf3, 0xfb, 0xc0, 0x24, 0xdc, 0x0d, 0x7f, 0x9a, 0x89, 0x2e, 0xa7, - 0xc5, 0xa9, 0x29, 0xdf, 0x7e, 0x36, 0xae, 0x8c, 0x47, 0xf4, 0xbb, 0x79, 0x06, 0xb5, 0xf8, 0x67, - 0x9a, 0x28, 0x25, 0xc9, 0x9c, 0xf1, 0x2d, 0x68, 0xe3, 0xda, 0x24, 0xa8, 0xe1, 0xce, 0x12, 0xdf, - 0x1d, 0x5e, 0x9d, 0xe4, 0xfb, 0xac, 0xcc, 0xce, 0xb2, 0x3e, 0xe5, 0x62, 0x0a, 0x8c, 0x7e, 0xea, - 0x81, 0x52, 0x3f, 0xf2, 0x4b, 0xf9, 0xa2, 0x28, 0x4d, 0x81, 0xe9, 0x5f, 0x8d, 0xc8, 0x33, 0xe8, - 0x10, 0x16, 0x63, 0x17, 0x39, 0x50, 0x0a, 0x79, 0xfa, 0x8d, 0x95, 0xc6, 0xd5, 0x09, 0x30, 0xa3, - 0x16, 0x11, 0xbe, 0xb8, 0x91, 0x6e, 0x11, 0x29, 0xd7, 0x42, 0xd2, 0x2d, 0x22, 0xf5, 0x0e, 0x08, - 0x35, 0xee, 0xc8, 0x85, 0x8d, 0x34, 0xe3, 0x4e, 0xbb, 0x26, 0xd2, 0xb8, 0x3c, 0x16, 0x2f, 0xac, - 0xb4, 0xd8, 0xf5, 0x8d, 0x34, 0xa5, 0xa5, 0x5f, 0x0f, 0x69, 0x5c, 0x9d, 0x00, 0x33, 0x6e, 0x05, - 0x41, 0x31, 0x38, 0xcb, 0x0a, 0x12, 0x57, 0x17, 0xb2, 0xac, 0x20, 0x59, 0x57, 0xe6, 0x56, 0x10, - 0x2b, 0xe2, 0x5e, 0x99, 0xa0, 0xe8, 0x94, 0x6d, 0x05, 0xe9, 0xe5, 0x29, 0x79, 0x06, 0xfd, 0x4c, - 0x82, 0x7a, 0x56, 0x8d, 0x03, 0xdd, 0x9c, 0xba, 0x20, 0xd3, 0x58, 0x9f, 0x86, 0xc4, 0x97, 0xe2, - 0x4b, 0x40, 0xc9, 0x3d, 0x10, 0xbd, 0x9d, 0x36, 0x33, 0x19, 0x3b, 0x6d, 0xe3, 0x9d, 0xc9, 0x90, - 0xfd, 0x2e, 0x3b, 0x50, 0x14, 0x55, 0x15, 0x94, 0xe2, 0xa5, 0x63, 0x35, 0x9d, 0x86, 0x3c, 0x0a, - 0xc5, 0x67, 0xfa, 0x08, 0x0a, 0x04, 0x8a, 0xce, 0xa6, 0x63, 0x0b, 0x66, 0xe7, 0xb2, 0x5e, 0xfb, - 0x8c, 0x9e, 0xc2, 0x1c, 0x2b, 0x23, 0xa0, 0x94, 0x2c, 0x44, 0xa4, 0xd8, 0xd1, 0x38, 0x9f, 0x8d, - 0xe0, 0xb3, 0xfb, 0x82, 0xfd, 0x0d, 0x15, 0xaf, 0x10, 0xa0, 0xb7, 0xd2, 0xff, 0x6d, 0x22, 0x5a, - 0x90, 0x68, 0x5c, 0x1c, 0x83, 0x15, 0x5e, 0x14, 0xb1, 0x08, 0xf8, 0xf2, 0xd8, 0x63, 0x4c, 0xf6, - 0xa2, 0x48, 0x3f, 0x28, 0x31, 0x23, 0x49, 0x1e, 0xa4, 0xd2, 0x8c, 0x24, 0xf3, 0xf8, 0x9a, 0x66, - 0x24, 0xd9, 0x67, 0x33, 0x79, 0x06, 0x79, 0x70, 0x22, 0x25, 0x6d, 0x86, 0xde, 0xc9, 0x32, 0xf2, - 0xb4, 0x1c, 0x5e, 0xe3, 0xfa, 0x84, 0xd8, 0xe1, 0xc9, 0xe7, 0x8b, 0xfe, 0xcd, 0xec, 0x5c, 0x52, - 0xe6, 0xe4, 0xc7, 0x97, 0xf8, 0xfa, 0xbf, 0xe4, 0x61, 0x81, 0xa5, 0x44, 0x79, 0x04, 0xf3, 0x19, - 0x40, 0x50, 0x8d, 0x40, 0x17, 0xd2, 0x75, 0x12, 0x29, 0xf1, 0x34, 0xde, 0x1a, 0x8d, 0x14, 0x36, - 0xb4, 0x50, 0x66, 0x3f, 0xcd, 0xd0, 0x92, 0x05, 0x8c, 0x34, 0x43, 0x4b, 0x29, 0x0f, 0xc8, 0x33, - 0xe8, 0x63, 0x28, 0xf9, 0x29, 0x64, 0x94, 0x96, 0x82, 0x8e, 0xe5, 0xc8, 0x1b, 0x17, 0x46, 0xe2, - 0x84, 0xa5, 0x0e, 0xe5, 0x87, 0xd3, 0xa4, 0x4e, 0xe6, 0xa1, 0xd3, 0xa4, 0x4e, 0x4b, 0x32, 0x07, - 0x3a, 0x61, 0x59, 0xa4, 0x4c, 0x9d, 0x44, 0x92, 0x78, 0x99, 0x3a, 0x89, 0xa6, 0xa2, 0xe4, 0x99, - 0x07, 0x97, 0x7e, 0xf5, 0xd5, 0x39, 0xe9, 0x9f, 0xbe, 0x3a, 0x37, 0xf3, 0xd3, 0xaf, 0xcf, 0x49, - 0xbf, 0xfa, 0xfa, 0x9c, 0xf4, 0x8f, 0x5f, 0x9f, 0x93, 0xfe, 0xf5, 0xeb, 0x73, 0xd2, 0x1f, 0xfc, - 0xdb, 0xb9, 0x99, 0x1f, 0x16, 0x05, 0xf5, 0xfe, 0x1c, 0xfd, 0x33, 0xb9, 0x77, 0xff, 0x37, 0x00, - 0x00, 0xff, 0xff, 0xf5, 0x05, 0x0e, 0x59, 0x12, 0x50, 0x00, 0x00, + // 5113 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xcc, 0x7c, 0xcf, 0x73, 0x1b, 0xc9, + 0x75, 0x3f, 0x07, 0x00, 0x49, 0xe0, 0x81, 0x00, 0xc1, 0x16, 0x45, 0x42, 0xa0, 0x44, 0x91, 0xa3, + 0x5d, 0xfd, 0xdc, 0xe5, 0x6a, 0xa5, 0x5d, 0xad, 0x56, 0xde, 0x5f, 0x10, 0xc9, 0x95, 0xb0, 0x96, + 0x40, 0x78, 0x40, 0xca, 0x5e, 0xfb, 0x5b, 0x9e, 0xef, 0x08, 0xd3, 0x04, 0xc7, 0x02, 0x66, 0xc6, + 0x33, 0x03, 0x51, 0xf4, 0x29, 0xd7, 0xe4, 0x94, 0xaa, 0x94, 0xe3, 0xaa, 0x54, 0xaa, 0x72, 0xcc, + 0xc1, 0x07, 0xe7, 0x92, 0x94, 0x2f, 0xce, 0x29, 0x07, 0x57, 0xaa, 0x5c, 0xe5, 0x4b, 0xaa, 0xf6, + 0x90, 0xaa, 0xd8, 0x9b, 0x5b, 0x0e, 0x39, 0xf9, 0x0f, 0x48, 0xf5, 0xaf, 0xc1, 0xf4, 0xcc, 0x60, + 0x48, 0x6a, 0xb5, 0xde, 0x3d, 0x11, 0xfd, 0xfa, 0xf5, 0xeb, 0xd7, 0xdd, 0xaf, 0x5f, 0xbf, 0xfe, + 0xbc, 0x1e, 0x42, 0xc9, 0x70, 0xad, 0x0d, 0xd7, 0x73, 0x02, 0x07, 0x81, 0x37, 0xb2, 0x03, 0x6b, + 0x88, 0x37, 0x9e, 0xbf, 0xdd, 0x78, 0xb3, 0x6f, 0x05, 0x07, 0xa3, 0xa7, 0x1b, 0x3d, 0x67, 0xf8, + 0x56, 0xdf, 0xe9, 0x3b, 0x6f, 0x51, 0x96, 0xa7, 0xa3, 0x7d, 0x5a, 0xa2, 0x05, 0xfa, 0x8b, 0x35, + 0x55, 0xaf, 0x43, 0xf5, 0x09, 0xf6, 0x7c, 0xcb, 0xb1, 0x35, 0xfc, 0xd3, 0x11, 0xf6, 0x03, 0x54, + 0x87, 0xd9, 0xe7, 0x8c, 0x52, 0x57, 0xd6, 0x94, 0xab, 0x25, 0x4d, 0x14, 0xd5, 0x7f, 0x54, 0x60, + 0x3e, 0x64, 0xf6, 0x5d, 0xc7, 0xf6, 0xf1, 0x64, 0x6e, 0xb4, 0x0e, 0x73, 0x5c, 0x2d, 0xdd, 0x36, + 0x86, 0xb8, 0x9e, 0xa3, 0xd5, 0x65, 0x4e, 0x6b, 0x1b, 0x43, 0x8c, 0xae, 0xc0, 0xbc, 0x60, 0x11, + 0x42, 0xf2, 0x94, 0xab, 0xca, 0xc9, 0xbc, 0x37, 0xb4, 0x01, 0x67, 0x04, 0xa3, 0xe1, 0x5a, 0x21, + 0x73, 0x81, 0x32, 0x2f, 0xf0, 0xaa, 0xa6, 0x6b, 0x71, 0x7e, 0xf5, 0x47, 0x50, 0xda, 0x6a, 0x77, + 0x37, 0x1d, 0x7b, 0xdf, 0xea, 0x13, 0x15, 0x7d, 0xec, 0x91, 0x36, 0x75, 0x65, 0x2d, 0x4f, 0x54, + 0xe4, 0x45, 0xd4, 0x80, 0xa2, 0x8f, 0x0d, 0xaf, 0x77, 0x80, 0xfd, 0x7a, 0x8e, 0x56, 0x85, 0x65, + 0xd2, 0xca, 0x71, 0x03, 0xcb, 0xb1, 0xfd, 0x7a, 0x9e, 0xb5, 0xe2, 0x45, 0xf5, 0xef, 0x15, 0x28, + 0x77, 0x1c, 0x2f, 0x78, 0x6c, 0xb8, 0xae, 0x65, 0xf7, 0xd1, 0x4d, 0x28, 0xd2, 0xb9, 0xec, 0x39, + 0x03, 0x3a, 0x07, 0xd5, 0x5b, 0x8b, 0x1b, 0xe3, 0x05, 0xd9, 0xe8, 0xf0, 0x3a, 0x2d, 0xe4, 0x42, + 0xaf, 0x43, 0xb5, 0xe7, 0xd8, 0x81, 0x61, 0xd9, 0xd8, 0xd3, 0x5d, 0xc7, 0x0b, 0xe8, 0xe4, 0x4c, + 0x6b, 0x95, 0x90, 0x4a, 0xe4, 0xa3, 0x15, 0x28, 0x1d, 0x38, 0x7e, 0xc0, 0x38, 0xf2, 0x94, 0xa3, + 0x48, 0x08, 0xb4, 0x72, 0x19, 0x66, 0x69, 0xa5, 0xe5, 0xf2, 0x69, 0x98, 0x21, 0xc5, 0x96, 0xab, + 0xfe, 0x5e, 0x81, 0xe9, 0xc7, 0xce, 0xc8, 0x0e, 0x62, 0xdd, 0x18, 0xc1, 0x01, 0x5f, 0xa2, 0x48, + 0x37, 0x46, 0x70, 0x30, 0xee, 0x86, 0x70, 0xb0, 0x55, 0x62, 0xdd, 0x90, 0xca, 0x06, 0x14, 0x3d, + 0x6c, 0x98, 0x8e, 0x3d, 0x38, 0xa2, 0x2a, 0x14, 0xb5, 0xb0, 0x4c, 0x96, 0xcf, 0xc7, 0x03, 0xcb, + 0x1e, 0xbd, 0xd0, 0x3d, 0x3c, 0x30, 0x9e, 0xe2, 0x01, 0x55, 0xa5, 0xa8, 0x55, 0x39, 0x59, 0x63, + 0x54, 0xf4, 0x11, 0x94, 0x5d, 0xcf, 0x71, 0x8d, 0xbe, 0x41, 0x66, 0xb0, 0x3e, 0x4d, 0x27, 0xe9, + 0x7c, 0x74, 0x92, 0xa8, 0xc2, 0x9d, 0x31, 0x8f, 0x16, 0x6d, 0xa0, 0xfe, 0x46, 0x81, 0x79, 0x62, + 0x30, 0xbe, 0x6b, 0xf4, 0xf0, 0x0e, 0x5d, 0x06, 0x74, 0x1b, 0x66, 0x6d, 0x1c, 0x1c, 0x3a, 0xde, + 0x33, 0x3e, 0xe9, 0xe7, 0xa2, 0xf2, 0x42, 0xee, 0xc7, 0x8e, 0x89, 0x35, 0xc1, 0x89, 0x6e, 0x40, + 0xde, 0xb5, 0x4c, 0x3a, 0xc8, 0xcc, 0x06, 0x84, 0x8b, 0x30, 0x5b, 0x6e, 0x8f, 0x8e, 0x3a, 0x9b, + 0xd9, 0x72, 0x7b, 0x64, 0x12, 0x03, 0xc3, 0xeb, 0xe3, 0x40, 0xb7, 0x4c, 0xbe, 0x20, 0x45, 0x46, + 0x68, 0x99, 0xaa, 0x0a, 0xd0, 0xb2, 0x83, 0x3b, 0xef, 0x3c, 0x31, 0x06, 0x23, 0x8c, 0x16, 0x61, + 0xfa, 0x39, 0xf9, 0x41, 0xf5, 0xce, 0x6b, 0xac, 0xa0, 0xfe, 0xa6, 0x00, 0x2b, 0x8f, 0xc8, 0xa4, + 0x75, 0x0d, 0xdb, 0x7c, 0xea, 0xbc, 0xe8, 0xe2, 0xde, 0xc8, 0xb3, 0x82, 0xa3, 0x4d, 0xc7, 0x0e, + 0xf0, 0x8b, 0x00, 0x3d, 0x84, 0x05, 0x5b, 0x74, 0xab, 0x0b, 0xcb, 0x24, 0x12, 0xca, 0xb7, 0x56, + 0x52, 0x75, 0x63, 0xf3, 0xa4, 0xd5, 0x6c, 0x99, 0xe0, 0xa3, 0xfb, 0xe3, 0x65, 0x13, 0x72, 0x72, + 0x54, 0x8e, 0x34, 0xc6, 0xee, 0x36, 0xd5, 0x86, 0x4b, 0x11, 0x2b, 0x2a, 0x64, 0xdc, 0x01, 0xb2, + 0x91, 0x75, 0xc3, 0xd7, 0x47, 0x3e, 0xf6, 0xe8, 0x1c, 0x95, 0x6f, 0x2d, 0x45, 0xdb, 0x8f, 0x07, + 0xac, 0x95, 0xbc, 0x91, 0xdd, 0xf4, 0xf7, 0x7c, 0xec, 0xa1, 0xbb, 0xd4, 0x29, 0x90, 0x76, 0x7d, + 0xcf, 0x19, 0xb9, 0xf5, 0x62, 0x66, 0x43, 0xa0, 0x0d, 0x1f, 0x10, 0x4e, 0xea, 0x2b, 0xb8, 0xe1, + 0xe9, 0x9e, 0xe3, 0x04, 0xfb, 0xbe, 0x30, 0x36, 0x41, 0xd6, 0x28, 0x15, 0xbd, 0x05, 0x67, 0xfc, + 0x91, 0xeb, 0x0e, 0xf0, 0x10, 0xdb, 0x81, 0x31, 0x60, 0x1d, 0xf9, 0xf5, 0xe9, 0xb5, 0xfc, 0xd5, + 0xbc, 0x86, 0xa2, 0x55, 0x54, 0xb0, 0x8f, 0x56, 0x01, 0x5c, 0xcf, 0x7a, 0x6e, 0x0d, 0x70, 0x1f, + 0x9b, 0xf5, 0x19, 0x2a, 0x34, 0x42, 0x41, 0xef, 0x12, 0xff, 0xd1, 0xeb, 0x39, 0x43, 0xb7, 0x5e, + 0x4a, 0xce, 0xb7, 0x58, 0xa7, 0x8e, 0xe7, 0xec, 0x5b, 0x03, 0xac, 0x09, 0x5e, 0xf4, 0x1e, 0x14, + 0x0d, 0xd7, 0x35, 0xbc, 0xa1, 0xe3, 0xd5, 0xe1, 0xf8, 0x76, 0x21, 0x33, 0x7a, 0x07, 0x16, 0xb9, + 0x0c, 0xdd, 0x65, 0x95, 0x6c, 0x6b, 0xce, 0x12, 0xab, 0xba, 0x9f, 0xab, 0x2b, 0x1a, 0xe2, 0xf5, + 0xbc, 0x2d, 0xd9, 0xa8, 0xea, 0xbf, 0x29, 0x30, 0x1f, 0x93, 0x89, 0x3e, 0x83, 0x39, 0x21, 0x21, + 0x38, 0x72, 0x31, 0xdf, 0x28, 0x57, 0x32, 0xd4, 0xd8, 0xe0, 0x7f, 0x77, 0x8f, 0x5c, 0x4c, 0xf7, + 0xa0, 0x28, 0xa0, 0x4b, 0x50, 0x19, 0x38, 0x3d, 0x63, 0x40, 0x5d, 0x85, 0x87, 0xf7, 0xb9, 0xa7, + 0x98, 0x0b, 0x89, 0x1a, 0xde, 0x57, 0x3f, 0x81, 0x72, 0x44, 0x00, 0x42, 0x50, 0xd5, 0x58, 0x57, + 0x5b, 0x78, 0xdf, 0x18, 0x0d, 0x82, 0xda, 0x14, 0xaa, 0x02, 0xec, 0xd9, 0x3d, 0xe2, 0x99, 0x6d, + 0x6c, 0xd6, 0x14, 0x54, 0x81, 0xd2, 0x23, 0x21, 0xa2, 0x96, 0x53, 0xff, 0x36, 0x07, 0x67, 0xa9, + 0xe1, 0x75, 0x1c, 0x93, 0xef, 0x04, 0xee, 0xc6, 0x2f, 0x41, 0xa5, 0x47, 0xd7, 0x52, 0x77, 0x0d, + 0x0f, 0xdb, 0x01, 0x77, 0x66, 0x73, 0x8c, 0xd8, 0xa1, 0x34, 0xa4, 0x41, 0xcd, 0xe7, 0x23, 0xd2, + 0x7b, 0x6c, 0xe7, 0x70, 0xe3, 0x96, 0x46, 0x9d, 0xb1, 0xd1, 0xb4, 0x79, 0x3f, 0xb1, 0xf3, 0x66, + 0xfd, 0x23, 0xbf, 0x17, 0x0c, 0xd8, 0x49, 0x50, 0xbe, 0xb5, 0x91, 0x10, 0x15, 0x57, 0x76, 0xa3, + 0xcb, 0x1a, 0x6c, 0xdb, 0x81, 0x77, 0xa4, 0x89, 0xe6, 0x8d, 0x7b, 0x30, 0x17, 0xad, 0x40, 0x35, + 0xc8, 0x3f, 0xc3, 0x47, 0x7c, 0x20, 0xe4, 0xe7, 0xd8, 0x37, 0xb0, 0xd9, 0x65, 0x85, 0x7b, 0xb9, + 0xbb, 0x8a, 0xea, 0x01, 0x1a, 0xf7, 0xf2, 0x18, 0x07, 0x86, 0x69, 0x04, 0x06, 0x42, 0x50, 0xa0, + 0x87, 0x2b, 0x13, 0x41, 0x7f, 0x13, 0xa9, 0x23, 0xee, 0xe4, 0x4a, 0x1a, 0xf9, 0x89, 0xce, 0x43, + 0x29, 0xf4, 0x02, 0xfc, 0x84, 0x1d, 0x13, 0xc8, 0x49, 0x67, 0x04, 0x01, 0x1e, 0xba, 0x01, 0xdd, + 0x51, 0x15, 0x4d, 0x14, 0xd5, 0xbf, 0x9c, 0x86, 0x5a, 0x62, 0x1d, 0xee, 0x41, 0x71, 0xc8, 0xbb, + 0xe7, 0xfe, 0x67, 0x55, 0x3a, 0xee, 0x12, 0x4a, 0x6a, 0x21, 0x3f, 0x39, 0x4d, 0xc8, 0x3a, 0x47, + 0xe2, 0x81, 0xb0, 0xcc, 0x0c, 0xac, 0xaf, 0x9b, 0x96, 0x87, 0x7b, 0x81, 0xe3, 0x1d, 0x71, 0x45, + 0xe7, 0x06, 0x4e, 0x7f, 0x4b, 0xd0, 0xd0, 0x3b, 0x00, 0xa6, 0xed, 0xeb, 0xd4, 0x7e, 0xfa, 0x54, + 0xdd, 0xf2, 0xad, 0xb3, 0xd1, 0xee, 0xc3, 0x63, 0x5f, 0x2b, 0x99, 0xb6, 0xcf, 0x55, 0xfe, 0x00, + 0x2a, 0xe4, 0x0c, 0xd5, 0x87, 0xec, 0xc4, 0x66, 0xce, 0xa0, 0x7c, 0x6b, 0x59, 0xd6, 0x3b, 0x3c, + 0xd1, 0xb5, 0x39, 0x77, 0x5c, 0xf0, 0xd1, 0x27, 0x30, 0x43, 0x8f, 0x31, 0xbf, 0x3e, 0x43, 0x9b, + 0x5d, 0x4d, 0x1f, 0x2e, 0x5f, 0xf9, 0x47, 0x94, 0x95, 0x2d, 0x3c, 0x6f, 0x87, 0x76, 0xa0, 0x6c, + 0xd8, 0xb6, 0x13, 0x18, 0xcc, 0xdb, 0xce, 0x52, 0x31, 0x6f, 0x66, 0x8a, 0x69, 0x8e, 0xf9, 0x99, + 0xac, 0xa8, 0x04, 0xf4, 0x1e, 0x4c, 0x53, 0x77, 0xcc, 0xfd, 0xe7, 0xfa, 0xb1, 0x06, 0xa9, 0x31, + 0x7e, 0xf4, 0x21, 0xcc, 0x1e, 0x5a, 0xb6, 0xe9, 0x1c, 0xfa, 0xdc, 0x97, 0x5d, 0x8a, 0x36, 0xfd, + 0x3e, 0xab, 0x4a, 0x34, 0x16, 0x6d, 0x1a, 0xef, 0x43, 0x39, 0x32, 0xbe, 0xd3, 0xd8, 0x6f, 0xe3, + 0x23, 0xa8, 0xc5, 0xc7, 0x74, 0x2a, 0xfb, 0x1f, 0xc1, 0xa2, 0x36, 0xb2, 0xc7, 0xaa, 0x89, 0x70, + 0xf5, 0x1d, 0x98, 0xe1, 0xd6, 0xc0, 0x8c, 0xf1, 0x7c, 0xd6, 0xb4, 0x6a, 0x9c, 0x37, 0x1a, 0x79, + 0x1e, 0x18, 0xb6, 0x39, 0xc0, 0x1e, 0xef, 0x51, 0x44, 0x9e, 0x0f, 0x19, 0x55, 0xfd, 0x10, 0xce, + 0xc6, 0xba, 0xe5, 0x81, 0xef, 0x6b, 0x50, 0x75, 0x1d, 0x53, 0xf7, 0x19, 0x99, 0x9c, 0xfa, 0xdc, + 0x1f, 0xb9, 0x21, 0x6f, 0xcb, 0x24, 0xcd, 0xbb, 0x81, 0xe3, 0x26, 0xd5, 0x3e, 0x59, 0xf3, 0x3a, + 0x2c, 0xc5, 0x9b, 0xb3, 0xee, 0xd5, 0x8f, 0x61, 0x59, 0xc3, 0x43, 0xe7, 0x39, 0x7e, 0x59, 0xd1, + 0x0d, 0xa8, 0x27, 0x05, 0x70, 0xe1, 0x9f, 0xc3, 0xf2, 0x98, 0xda, 0x0d, 0x8c, 0x60, 0xe4, 0x9f, + 0x4a, 0x38, 0xbf, 0x15, 0x3c, 0x75, 0x7c, 0xb6, 0x90, 0x45, 0x4d, 0x14, 0xd5, 0x65, 0x98, 0xee, + 0x38, 0x66, 0xab, 0x83, 0xaa, 0x90, 0xb3, 0x5c, 0xde, 0x38, 0x67, 0xb9, 0x6a, 0x2f, 0xda, 0x67, + 0x9b, 0xc5, 0x6b, 0xac, 0xeb, 0x38, 0x2b, 0xba, 0x0b, 0x55, 0xc3, 0x34, 0x2d, 0x62, 0x48, 0xc6, + 0x40, 0xb7, 0x5c, 0x16, 0xbc, 0x97, 0x6f, 0x2d, 0xc4, 0x96, 0xbe, 0xd5, 0xd1, 0x2a, 0x63, 0xc6, + 0x96, 0xeb, 0xab, 0xf7, 0xa1, 0x14, 0xc6, 0x47, 0xe4, 0x5c, 0x97, 0xe3, 0x9f, 0xcc, 0x38, 0x2a, + 0x0c, 0xff, 0xdb, 0x89, 0x03, 0x8a, 0xab, 0xf9, 0x2e, 0x40, 0xe8, 0x54, 0x45, 0x68, 0x76, 0x36, + 0x55, 0xa4, 0x16, 0x61, 0x54, 0xff, 0xab, 0x10, 0x75, 0xb2, 0x91, 0x21, 0x9b, 0xe1, 0x90, 0x4d, + 0xc9, 0xe9, 0xe6, 0x4e, 0xe9, 0x74, 0xdf, 0x86, 0x69, 0x3f, 0x30, 0x02, 0xcc, 0x23, 0xd9, 0x95, + 0xf4, 0x86, 0xa4, 0x63, 0xac, 0x31, 0x4e, 0x74, 0x01, 0xa0, 0xe7, 0x61, 0x23, 0xc0, 0xa6, 0x6e, + 0xb0, 0x53, 0x21, 0xaf, 0x95, 0x38, 0xa5, 0x19, 0x10, 0x2f, 0x22, 0x62, 0xef, 0xe9, 0xa4, 0x17, + 0x99, 0xb0, 0x8c, 0xe3, 0x28, 0x3c, 0xf4, 0x5e, 0x33, 0xc7, 0x7a, 0x2f, 0xde, 0x94, 0x7b, 0xaf, + 0xb1, 0x27, 0x9e, 0xcd, 0xf2, 0xc4, 0xac, 0xd1, 0x49, 0x3c, 0x71, 0x31, 0xcb, 0x13, 0x73, 0x31, + 0xd9, 0x9e, 0x38, 0xc5, 0x91, 0x94, 0xd2, 0x1c, 0xc9, 0x37, 0xe9, 0x3a, 0x7f, 0xa7, 0x40, 0x3d, + 0xb9, 0x9f, 0xb9, 0x1f, 0x7b, 0x07, 0x66, 0x7c, 0x4a, 0xc9, 0xf6, 0x9f, 0xbc, 0x15, 0xe7, 0x45, + 0xf7, 0xa1, 0x60, 0xd9, 0xfb, 0x0e, 0xdf, 0x78, 0x1b, 0x99, 0x6d, 0x78, 0x4f, 0x1b, 0x2d, 0x7b, + 0xdf, 0x61, 0x33, 0x48, 0xdb, 0x36, 0xde, 0x83, 0x52, 0x48, 0x3a, 0xd5, 0x78, 0x5a, 0xb0, 0x18, + 0xb3, 0x5b, 0x76, 0xb1, 0x0a, 0x0d, 0x5d, 0x39, 0xa9, 0xa1, 0xab, 0x7f, 0x52, 0xa2, 0x9b, 0xef, + 0x53, 0x6b, 0x10, 0x60, 0x2f, 0xb1, 0xf9, 0xee, 0x08, 0xb9, 0x6c, 0xe7, 0xad, 0x65, 0xc8, 0x65, + 0xf7, 0x16, 0xbe, 0x8b, 0x9e, 0x40, 0x95, 0x9a, 0x9d, 0xee, 0xe3, 0x01, 0x8d, 0x5f, 0x78, 0xfc, + 0xf8, 0x56, 0xba, 0x00, 0xd6, 0x3b, 0x33, 0xdb, 0x2e, 0x6f, 0xc1, 0xe6, 0xab, 0x32, 0x88, 0xd2, + 0x1a, 0x9f, 0x00, 0x4a, 0x32, 0x9d, 0x6a, 0x06, 0x1f, 0x13, 0x1f, 0xe6, 0x07, 0xa9, 0xa7, 0xe9, + 0x3e, 0x55, 0x23, 0xdb, 0x1a, 0x98, 0xaa, 0x1a, 0xe7, 0x55, 0xff, 0x23, 0x0f, 0x30, 0xae, 0xfc, + 0x96, 0x3b, 0xaf, 0x7b, 0xa1, 0x13, 0x61, 0x51, 0xa0, 0x9a, 0x2e, 0x32, 0xd5, 0x7d, 0xb4, 0x64, + 0xf7, 0xc1, 0xe2, 0xc1, 0x2b, 0x13, 0x04, 0x9c, 0xda, 0x71, 0xcc, 0x7e, 0xdb, 0x1c, 0xc7, 0xa7, + 0xb0, 0x14, 0x37, 0x13, 0xee, 0x35, 0xde, 0x80, 0x69, 0x2b, 0xc0, 0x43, 0x86, 0xa8, 0xc5, 0x2e, + 0xf0, 0x11, 0x76, 0xc6, 0xa4, 0xfe, 0x52, 0x81, 0x52, 0x6b, 0x68, 0xf4, 0x71, 0xd7, 0xc5, 0x3d, + 0xd2, 0x9f, 0x45, 0x0a, 0x5c, 0x07, 0x56, 0x40, 0x0f, 0xe5, 0xa9, 0x65, 0x8e, 0xe5, 0xb2, 0x04, + 0x0c, 0x08, 0x09, 0xd9, 0x33, 0xfb, 0x95, 0x47, 0x7d, 0x0b, 0x8a, 0xdf, 0xc5, 0x47, 0xcc, 0xa5, + 0x9c, 0xb0, 0x9d, 0xfa, 0xf3, 0x02, 0x2c, 0xd3, 0xa3, 0x6b, 0x53, 0x40, 0x6b, 0x1a, 0xf6, 0x9d, + 0x91, 0xd7, 0xc3, 0x3e, 0xb5, 0x47, 0x77, 0xa4, 0xbb, 0xd8, 0xb3, 0x1c, 0x93, 0x83, 0x3e, 0xa5, + 0x9e, 0x3b, 0xea, 0x50, 0x02, 0x5a, 0x01, 0x52, 0xd0, 0x7f, 0x3a, 0x72, 0xf8, 0xf6, 0xc8, 0x6b, + 0xc5, 0x9e, 0x3b, 0xfa, 0x1e, 0x29, 0x8b, 0xb6, 0xfe, 0x81, 0xe1, 0x61, 0x9f, 0xee, 0x01, 0xd6, + 0xb6, 0x4b, 0x09, 0xe8, 0x6d, 0x38, 0x3b, 0xc4, 0x43, 0xc7, 0x3b, 0xd2, 0x07, 0xd6, 0xd0, 0x0a, + 0x74, 0xcb, 0xd6, 0x9f, 0x1e, 0x05, 0xd8, 0xe7, 0x56, 0x8f, 0x58, 0xe5, 0x23, 0x52, 0xd7, 0xb2, + 0xef, 0x93, 0x1a, 0xa4, 0x42, 0xc5, 0x71, 0x86, 0xba, 0xdf, 0x73, 0x3c, 0xac, 0x1b, 0xe6, 0x4f, + 0xe8, 0x09, 0x9e, 0xd7, 0xca, 0x8e, 0x33, 0xec, 0x12, 0x5a, 0xd3, 0xfc, 0x09, 0xba, 0x08, 0xe5, + 0x9e, 0x3b, 0xf2, 0x71, 0xa0, 0x93, 0x3f, 0xf4, 0x98, 0x2e, 0x69, 0xc0, 0x48, 0x9b, 0xee, 0xc8, + 0x8f, 0x30, 0x0c, 0x89, 0x11, 0xcc, 0x46, 0x19, 0x1e, 0xe3, 0x21, 0xc5, 0x98, 0x0e, 0x46, 0x7d, + 0xec, 0x1a, 0x7d, 0xcc, 0x54, 0x13, 0x67, 0xad, 0x84, 0x31, 0x3d, 0xe4, 0x2c, 0x54, 0x41, 0xad, + 0x7a, 0x10, 0x2d, 0xfa, 0xe8, 0x33, 0x98, 0x1d, 0xd9, 0xd6, 0xbe, 0x85, 0xcd, 0x7a, 0x89, 0xb6, + 0xbd, 0x99, 0x08, 0x14, 0x92, 0xb3, 0xbd, 0xb1, 0xc7, 0x9a, 0xf0, 0x9b, 0x37, 0x17, 0x80, 0xee, + 0x41, 0x83, 0x4f, 0x94, 0x7f, 0x68, 0xb8, 0xf1, 0xd9, 0x02, 0x3a, 0x05, 0x4b, 0x8c, 0xa3, 0x7b, + 0x68, 0xb8, 0xd1, 0x19, 0x23, 0xb7, 0xf6, 0xa8, 0xd0, 0x53, 0xd9, 0xd2, 0x7d, 0xa8, 0x48, 0x83, + 0x24, 0xab, 0x4d, 0x27, 0xc5, 0xb7, 0x7e, 0x26, 0x36, 0x40, 0x91, 0x10, 0xba, 0xd6, 0xcf, 0x28, + 0x32, 0x48, 0x35, 0xa3, 0x72, 0x0a, 0x1a, 0x2b, 0xa8, 0x06, 0x54, 0x24, 0x30, 0x8e, 0x5c, 0xfa, + 0x29, 0xea, 0xc6, 0x2f, 0xfd, 0xe4, 0x37, 0xa1, 0x79, 0xce, 0x40, 0x68, 0x40, 0x7f, 0x13, 0x1a, + 0x85, 0x7d, 0xd8, 0x45, 0x9a, 0xfe, 0xa6, 0x5d, 0xe0, 0xe7, 0x1c, 0xa9, 0x2d, 0x69, 0xac, 0xa0, + 0x9a, 0x00, 0x9b, 0x86, 0x6b, 0x3c, 0xb5, 0x06, 0x56, 0x70, 0x84, 0xae, 0x41, 0xcd, 0x30, 0x4d, + 0xbd, 0x27, 0x28, 0x16, 0x16, 0xc8, 0xf9, 0xbc, 0x61, 0x9a, 0x9b, 0x11, 0x32, 0xba, 0x01, 0x0b, + 0xa6, 0xe7, 0xb8, 0x32, 0x2f, 0x83, 0xd2, 0x6b, 0xa4, 0x22, 0xca, 0xac, 0xfe, 0xeb, 0x0c, 0x5c, + 0x90, 0x97, 0x2d, 0x0e, 0x72, 0xde, 0x83, 0xb9, 0x58, 0xaf, 0x09, 0x78, 0x70, 0xac, 0xa7, 0x26, + 0xf1, 0xc6, 0x60, 0xbc, 0x5c, 0x02, 0xc6, 0x4b, 0x05, 0x50, 0xf3, 0xaf, 0x08, 0x40, 0x2d, 0x7c, + 0x45, 0x00, 0x75, 0xfa, 0x65, 0x01, 0xd4, 0xb9, 0x13, 0x03, 0xa8, 0x97, 0xe9, 0x81, 0x23, 0x7a, + 0xa4, 0x10, 0x0c, 0xdb, 0xd8, 0x95, 0x50, 0xba, 0x2d, 0x92, 0x32, 0x31, 0xa0, 0x75, 0xf6, 0x34, + 0x40, 0x6b, 0x71, 0x22, 0xd0, 0xba, 0x06, 0x73, 0xb6, 0xa3, 0xdb, 0xf8, 0x50, 0x27, 0xcb, 0xe2, + 0xd7, 0xcb, 0x6c, 0x8d, 0x6c, 0xa7, 0x8d, 0x0f, 0x3b, 0x84, 0x82, 0xd6, 0x61, 0x6e, 0x68, 0xf8, + 0xcf, 0xb0, 0x49, 0x11, 0x4f, 0xbf, 0x5e, 0xa1, 0x96, 0x54, 0x66, 0xb4, 0x0e, 0x21, 0xa1, 0xd7, + 0x21, 0xd4, 0x83, 0x33, 0x55, 0x29, 0x53, 0x45, 0x50, 0x19, 0x5b, 0x04, 0xb4, 0x9d, 0x7f, 0x49, + 0xd0, 0xb6, 0x76, 0x1a, 0xd0, 0xf6, 0x4d, 0xa8, 0x89, 0xdf, 0x02, 0xb5, 0x65, 0x17, 0x01, 0x0a, + 0xd8, 0xce, 0x8b, 0x3a, 0x81, 0xcc, 0x4e, 0xc2, 0x78, 0x21, 0x13, 0xe3, 0xfd, 0x95, 0x02, 0x8b, + 0xf2, 0x06, 0xe2, 0x00, 0x57, 0x13, 0x4a, 0x9e, 0xf0, 0x80, 0x7c, 0xd3, 0x5c, 0x3a, 0x81, 0xb3, + 0xd4, 0xc6, 0xad, 0xd0, 0xee, 0x44, 0xe4, 0xf4, 0xda, 0x64, 0x49, 0xc7, 0x61, 0xa7, 0xea, 0xdf, + 0x28, 0x70, 0x81, 0xa3, 0x4a, 0x13, 0xf2, 0x1a, 0x29, 0x66, 0xa9, 0x4c, 0x30, 0xcb, 0x9e, 0x87, + 0x4d, 0x6c, 0x07, 0x96, 0x31, 0xd0, 0x7d, 0x17, 0xf7, 0x04, 0x62, 0x33, 0x26, 0xd3, 0xf0, 0x62, + 0x1d, 0xe6, 0x58, 0x3a, 0xcb, 0x73, 0x7a, 0xd8, 0xf7, 0x79, 0xd6, 0xaa, 0x4c, 0x33, 0x5a, 0x8c, + 0xa4, 0x3a, 0xb0, 0x3c, 0x01, 0xea, 0x4a, 0x9d, 0x06, 0x25, 0x39, 0x0d, 0x99, 0x63, 0x4a, 0x4e, + 0xc3, 0xcf, 0x15, 0xb8, 0xc8, 0x9b, 0x4c, 0xf4, 0x7d, 0xdf, 0xc4, 0x44, 0xfc, 0xb3, 0x02, 0x4b, + 0x71, 0xbd, 0xf8, 0x44, 0x6c, 0x26, 0x4d, 0xea, 0xf5, 0x94, 0x19, 0xc8, 0x36, 0xaa, 0x27, 0x13, + 0x8d, 0xea, 0x46, 0x96, 0xac, 0x63, 0xe7, 0xf3, 0x97, 0x0a, 0x9c, 0x9b, 0xa8, 0x40, 0x2c, 0x68, + 0x52, 0xe2, 0x41, 0x13, 0x0f, 0xb8, 0x7a, 0xce, 0xc8, 0x0e, 0x22, 0x01, 0xd7, 0x26, 0xcd, 0x99, + 0xb2, 0xc8, 0x46, 0x1f, 0x1a, 0x2f, 0xac, 0xe1, 0x68, 0xc8, 0x23, 0x2e, 0x22, 0xee, 0x31, 0xa3, + 0xbc, 0x44, 0xc8, 0xa5, 0x36, 0x61, 0x21, 0xd4, 0x32, 0x13, 0xb9, 0x8f, 0x20, 0xf1, 0x39, 0x19, + 0x89, 0xb7, 0x61, 0x66, 0x0b, 0x3f, 0xb7, 0x7a, 0xf8, 0x95, 0x24, 0x75, 0xd7, 0xa0, 0xec, 0x62, + 0x6f, 0x68, 0xf9, 0x7e, 0x78, 0x08, 0x96, 0xb4, 0x28, 0x49, 0xfd, 0xd5, 0x0c, 0xcc, 0xc7, 0x2d, + 0xe2, 0xfd, 0x04, 0xf0, 0x7f, 0x41, 0x3a, 0x98, 0xe3, 0x43, 0x8c, 0xdc, 0xe2, 0x6e, 0x88, 0x90, + 0x3f, 0x97, 0x44, 0xc5, 0xc2, 0xb0, 0x5e, 0xdc, 0x04, 0xea, 0x30, 0xdb, 0x73, 0x86, 0x43, 0xc3, + 0x36, 0x45, 0xe6, 0x9d, 0x17, 0xc9, 0x9c, 0x19, 0x5e, 0x9f, 0x4c, 0x35, 0x21, 0xd3, 0xdf, 0x64, + 0xc1, 0x0e, 0x1d, 0xef, 0x99, 0x65, 0xd3, 0xd4, 0x01, 0x3d, 0x48, 0x4b, 0x1a, 0x70, 0xd2, 0x96, + 0xe5, 0xa1, 0xab, 0x50, 0xc0, 0xf6, 0x73, 0x71, 0x59, 0x93, 0x52, 0xf3, 0x22, 0xcc, 0xd7, 0x28, + 0x07, 0xba, 0x06, 0x33, 0x43, 0x62, 0x04, 0x02, 0x5e, 0x5a, 0x48, 0x64, 0xa8, 0x35, 0xce, 0x80, + 0xde, 0x80, 0x59, 0x93, 0xae, 0x87, 0x88, 0x6b, 0x91, 0x94, 0x84, 0xa0, 0x55, 0x9a, 0x60, 0x41, + 0x1f, 0x87, 0x57, 0xce, 0x52, 0xf2, 0xc6, 0x18, 0x9b, 0xe6, 0xd4, 0x7b, 0x67, 0x5b, 0xbe, 0x1c, + 0x01, 0x95, 0xf2, 0x46, 0x96, 0x94, 0xec, 0xcb, 0xe7, 0x39, 0x28, 0x0e, 0x9c, 0x3e, 0x33, 0x8e, + 0x32, 0x7b, 0xb6, 0x31, 0x70, 0xfa, 0xd4, 0x36, 0x16, 0xc9, 0x85, 0xdb, 0xb4, 0x6c, 0x1a, 0x59, + 0x14, 0x35, 0x56, 0x20, 0x5b, 0x8a, 0xfe, 0xd0, 0x1d, 0xbb, 0x87, 0xeb, 0x15, 0x5a, 0x55, 0xa2, + 0x94, 0x1d, 0xbb, 0x47, 0xaf, 0x49, 0x41, 0x70, 0x54, 0xaf, 0x52, 0x3a, 0xf9, 0x89, 0xee, 0x08, + 0x8c, 0x6f, 0x3e, 0x89, 0x99, 0xa4, 0x1d, 0x61, 0x02, 0xe2, 0xfb, 0x60, 0x9c, 0xa0, 0x60, 0xe7, + 0xaf, 0x9a, 0xe5, 0x28, 0xbe, 0x45, 0xf9, 0x89, 0x5f, 0x2b, 0xb0, 0xb4, 0x49, 0x41, 0x86, 0x88, + 0x47, 0x3a, 0x0d, 0x66, 0x7e, 0x3b, 0x4c, 0x64, 0xa4, 0xa0, 0xd1, 0xf1, 0x11, 0x8b, 0x3c, 0xc6, + 0x26, 0x54, 0x85, 0x58, 0xde, 0x38, 0x7f, 0x82, 0x2c, 0x48, 0xc5, 0x8f, 0x16, 0xd5, 0x0f, 0x60, + 0x39, 0xa1, 0x39, 0xbf, 0xe7, 0xaf, 0xc3, 0xdc, 0xd8, 0xdb, 0x84, 0x8a, 0x97, 0x43, 0x5a, 0xcb, + 0x54, 0xef, 0xc1, 0xd9, 0x6e, 0x60, 0x78, 0x41, 0x62, 0xd8, 0x27, 0x68, 0x4b, 0xf3, 0x1b, 0x72, + 0x5b, 0x9e, 0x82, 0xe8, 0xc2, 0x62, 0x37, 0x70, 0xdc, 0x97, 0x10, 0x4a, 0xfc, 0x07, 0x19, 0xb9, + 0x33, 0x12, 0xde, 0x5d, 0x14, 0xd5, 0x65, 0x96, 0x8d, 0x49, 0xf6, 0xf6, 0x1d, 0x58, 0x62, 0xc9, + 0x90, 0x97, 0x19, 0xc4, 0x39, 0x91, 0x8a, 0x49, 0xca, 0x7d, 0x00, 0x67, 0xc6, 0x87, 0xda, 0x18, + 0xa8, 0xbc, 0x29, 0x03, 0x95, 0x8d, 0xd4, 0x95, 0x96, 0x70, 0xca, 0x5f, 0xe4, 0x22, 0xfe, 0x78, + 0x02, 0x4c, 0xf9, 0xae, 0x0c, 0x53, 0x5e, 0x9c, 0x2c, 0x55, 0x42, 0x29, 0x93, 0xd6, 0x99, 0x4f, + 0xb1, 0xce, 0xbd, 0x04, 0x96, 0x59, 0x48, 0x42, 0xbf, 0x31, 0x0d, 0xff, 0x2c, 0x50, 0xe6, 0x23, + 0x06, 0x65, 0x86, 0x5d, 0x87, 0x99, 0xaa, 0xdb, 0x31, 0x28, 0x73, 0x25, 0x43, 0xd3, 0x10, 0xc9, + 0xfc, 0x45, 0x01, 0x4a, 0x61, 0x5d, 0x62, 0x86, 0x93, 0x53, 0x95, 0x4b, 0x99, 0xaa, 0xe8, 0x39, + 0x99, 0x7f, 0xc9, 0x73, 0xb2, 0x70, 0x82, 0x73, 0x72, 0x05, 0x4a, 0xf4, 0x07, 0x7d, 0x8d, 0xc1, + 0xce, 0xbd, 0x22, 0x25, 0x68, 0x78, 0x7f, 0x6c, 0x62, 0x33, 0x27, 0x34, 0xb1, 0x18, 0x6c, 0x3a, + 0x1b, 0x87, 0x4d, 0xdf, 0x0f, 0xcf, 0x30, 0x76, 0xe0, 0xad, 0xa7, 0x4a, 0x4c, 0x3d, 0xbd, 0x62, + 0xd0, 0x5e, 0x29, 0x09, 0xed, 0x8d, 0xdb, 0x67, 0x43, 0x7b, 0xdf, 0xa0, 0x7f, 0xdf, 0x61, 0x58, + 0x68, 0xd4, 0xce, 0xb8, 0x8f, 0x7c, 0x17, 0x20, 0x74, 0x07, 0x02, 0x10, 0x3d, 0x9b, 0x3a, 0x3a, + 0x2d, 0xc2, 0xa8, 0xee, 0xc1, 0x92, 0xb4, 0x10, 0xe3, 0x1c, 0xeb, 0xc9, 0x7c, 0xdc, 0x84, 0x04, + 0xeb, 0x17, 0xd3, 0x11, 0x4f, 0x31, 0x21, 0x9b, 0xf8, 0x7e, 0x02, 0x90, 0x3f, 0xb1, 0x85, 0xde, + 0x94, 0xf1, 0xf8, 0x53, 0xdb, 0x55, 0x02, 0x8e, 0xa7, 0x91, 0x85, 0xe1, 0xf1, 0x6a, 0x06, 0x46, + 0x96, 0x38, 0xa5, 0x49, 0xe3, 0xf1, 0x7d, 0xcb, 0xb6, 0xfc, 0x03, 0x56, 0x3f, 0xc3, 0xe2, 0x71, + 0x41, 0x6a, 0x52, 0x40, 0x0d, 0xbf, 0xb0, 0x02, 0xbd, 0xe7, 0x98, 0x98, 0x5a, 0xed, 0xb4, 0x56, + 0x24, 0x84, 0x4d, 0xc7, 0xc4, 0xe3, 0xfd, 0x54, 0x3c, 0xed, 0x7e, 0x2a, 0xc5, 0xf6, 0xd3, 0x12, + 0xcc, 0x78, 0xd8, 0xf0, 0x1d, 0x9b, 0x5d, 0xd1, 0x35, 0x5e, 0x22, 0x0b, 0x31, 0xc4, 0xbe, 0x4f, + 0xfa, 0xe0, 0x81, 0x14, 0x2f, 0x46, 0x82, 0xbe, 0xb9, 0x8c, 0xa0, 0x2f, 0x23, 0x57, 0x19, 0x0b, + 0xfa, 0x2a, 0x19, 0x41, 0xdf, 0x89, 0x52, 0x95, 0xe3, 0xf0, 0xb6, 0x7a, 0x5c, 0x78, 0x1b, 0x8d, + 0x0f, 0xe7, 0xa5, 0xf8, 0xf0, 0x9b, 0xdc, 0x82, 0xff, 0xae, 0xc0, 0x72, 0x62, 0xcb, 0xf0, 0x4d, + 0x78, 0x3b, 0x96, 0xc6, 0x5c, 0xc9, 0x98, 0xa7, 0x30, 0x8b, 0xd9, 0x94, 0xb2, 0x98, 0x6f, 0x66, + 0x35, 0x79, 0xe5, 0x49, 0xcc, 0x3f, 0xe4, 0xe0, 0xe2, 0x9e, 0x6b, 0xc6, 0xa2, 0x2e, 0x7e, 0x85, + 0x3e, 0xb9, 0x23, 0x78, 0x5f, 0xc4, 0xd9, 0xb9, 0x93, 0xa3, 0x3e, 0x3c, 0xd4, 0xfe, 0x78, 0x1c, + 0x6a, 0xe7, 0x4f, 0x73, 0xbf, 0x17, 0xad, 0xd0, 0x8f, 0x65, 0x03, 0x65, 0x01, 0xc1, 0x07, 0x51, + 0x21, 0xc7, 0x0c, 0xf0, 0x6b, 0x4e, 0xe4, 0xa8, 0xb0, 0x36, 0x59, 0x01, 0x1e, 0xa1, 0xfd, 0x7f, + 0x98, 0xdf, 0x7e, 0x81, 0x7b, 0xdd, 0x23, 0xbb, 0x77, 0x8a, 0x59, 0xaf, 0x41, 0xbe, 0x37, 0x34, + 0x39, 0xd0, 0x4d, 0x7e, 0x46, 0x83, 0xce, 0xbc, 0x1c, 0x74, 0xea, 0x50, 0x1b, 0xf7, 0xc0, 0xad, + 0x75, 0x89, 0x58, 0xab, 0x49, 0x98, 0x89, 0xf0, 0x39, 0x8d, 0x97, 0x38, 0x1d, 0x7b, 0xec, 0x35, + 0x12, 0xa3, 0x63, 0xcf, 0x93, 0x9d, 0x5c, 0x5e, 0x76, 0x72, 0xea, 0xdf, 0x29, 0x50, 0x26, 0x3d, + 0x7c, 0x25, 0xfd, 0xf9, 0x0d, 0x2e, 0x3f, 0xbe, 0xc1, 0x85, 0x17, 0xc1, 0x42, 0xf4, 0x22, 0x38, + 0xd6, 0x7c, 0x9a, 0x92, 0x93, 0x9a, 0xcf, 0x84, 0x74, 0xec, 0x79, 0xea, 0x1a, 0xcc, 0x31, 0xdd, + 0xf8, 0xc8, 0x6b, 0x90, 0x1f, 0x79, 0x03, 0xb1, 0x7e, 0x23, 0x6f, 0xa0, 0xfe, 0x95, 0x02, 0x95, + 0x66, 0x10, 0x18, 0xbd, 0x83, 0x53, 0x0c, 0x20, 0x54, 0x2e, 0x17, 0x55, 0x2e, 0x39, 0x88, 0xb1, + 0xba, 0x85, 0x09, 0xea, 0x4e, 0x4b, 0xea, 0xaa, 0x50, 0x15, 0xba, 0x4c, 0x54, 0xb8, 0x0d, 0xa8, + 0xe3, 0x78, 0xc1, 0xa7, 0x8e, 0x77, 0x68, 0x78, 0xe6, 0xe9, 0x2e, 0x79, 0x08, 0x0a, 0xfc, 0xbd, + 0x7f, 0xfe, 0xea, 0xb4, 0x46, 0x7f, 0xab, 0x57, 0xe0, 0x8c, 0x24, 0x6f, 0x62, 0xc7, 0xf7, 0xa0, + 0x4c, 0x0f, 0x2d, 0x1e, 0xff, 0xdf, 0x88, 0xe6, 0x51, 0x8f, 0x39, 0xdc, 0xd4, 0x2d, 0x58, 0x20, + 0xe1, 0x0b, 0xa5, 0x87, 0xfe, 0xe5, 0xad, 0x58, 0x88, 0xbc, 0x9c, 0x10, 0x11, 0x0b, 0x8f, 0xff, + 0x53, 0x81, 0x69, 0x4a, 0x4f, 0x84, 0x14, 0x2b, 0x50, 0xf2, 0xb0, 0xeb, 0xe8, 0x81, 0xd1, 0x0f, + 0xbf, 0xa5, 0x20, 0x84, 0x5d, 0xa3, 0x4f, 0x61, 0x7d, 0x5a, 0x69, 0x5a, 0x7d, 0xec, 0x07, 0xe2, + 0x83, 0x8a, 0x32, 0xa1, 0x6d, 0x31, 0x12, 0x99, 0x18, 0x9a, 0x12, 0x2b, 0xd0, 0xcc, 0x17, 0xfd, + 0x8d, 0xae, 0xb2, 0x87, 0xac, 0xd9, 0xb9, 0x11, 0xfa, 0xc0, 0xb5, 0x01, 0xc5, 0x58, 0x52, 0x23, + 0x2c, 0xa3, 0x6b, 0x50, 0xa0, 0x20, 0xe9, 0x6c, 0xd6, 0x2c, 0x51, 0x16, 0xf5, 0x63, 0x40, 0xd1, + 0x49, 0xe2, 0x0b, 0x71, 0x0d, 0x66, 0xe8, 0x1c, 0x8a, 0xd8, 0x6e, 0x21, 0x21, 0x42, 0xe3, 0x0c, + 0xea, 0x8f, 0x00, 0x31, 0x99, 0x52, 0x3c, 0x77, 0x9a, 0x85, 0xca, 0x88, 0xec, 0xfe, 0x45, 0x81, + 0x33, 0x92, 0x74, 0xae, 0xdf, 0x15, 0x59, 0x7c, 0x8a, 0x7a, 0x5c, 0xf4, 0x87, 0xd2, 0x71, 0x77, + 0x2d, 0xa9, 0xc6, 0xd7, 0x74, 0xd4, 0xfd, 0x4e, 0x01, 0x68, 0x8e, 0x82, 0x03, 0x8e, 0x23, 0x46, + 0x17, 0x4b, 0x89, 0x2d, 0x56, 0x03, 0x8a, 0xae, 0xe1, 0xfb, 0x87, 0x8e, 0x27, 0xee, 0x56, 0x61, + 0x99, 0xa2, 0x7f, 0xa3, 0xe0, 0x40, 0xa4, 0x33, 0xc9, 0x6f, 0xf4, 0x3a, 0x54, 0xd9, 0xc7, 0x3c, + 0xba, 0x61, 0x9a, 0x1e, 0xf6, 0x7d, 0x9e, 0xd7, 0xac, 0x30, 0x6a, 0x93, 0x11, 0x09, 0x9b, 0x45, + 0xa1, 0xf1, 0xe0, 0x48, 0x0f, 0x9c, 0x67, 0xd8, 0xe6, 0xf7, 0xa5, 0x8a, 0xa0, 0xee, 0x12, 0x22, + 0xcb, 0x2d, 0xf5, 0x2d, 0x3f, 0xf0, 0x04, 0x9b, 0xc8, 0x90, 0x71, 0x2a, 0x65, 0x53, 0xff, 0x49, + 0x81, 0x5a, 0x67, 0x34, 0x18, 0xb0, 0xc9, 0x7d, 0x99, 0x45, 0xbe, 0xce, 0x87, 0x92, 0x4b, 0x9a, + 0xf6, 0x78, 0xa2, 0xf8, 0x10, 0x5f, 0x09, 0xc4, 0x73, 0x13, 0x16, 0x22, 0x1a, 0x73, 0xc3, 0x91, + 0x02, 0x5e, 0x45, 0x0e, 0x78, 0xd5, 0x26, 0x20, 0x86, 0x6a, 0xbc, 0xf4, 0x28, 0xd5, 0xb3, 0x70, + 0x46, 0x12, 0xc1, 0x8f, 0xdc, 0xeb, 0x50, 0xe1, 0x0f, 0x03, 0xb9, 0x41, 0x9c, 0x83, 0x22, 0x71, + 0x9d, 0x3d, 0xcb, 0x14, 0x39, 0xed, 0x59, 0xd7, 0x31, 0x37, 0x2d, 0xd3, 0x53, 0xbf, 0x07, 0x15, + 0xfe, 0x05, 0x01, 0xe7, 0xfd, 0x04, 0xaa, 0xfc, 0x19, 0xa1, 0x2e, 0x3d, 0xfb, 0x95, 0xbf, 0xcf, + 0x89, 0x8a, 0xd7, 0x2a, 0x76, 0xb4, 0xa8, 0xfe, 0x18, 0x1a, 0x2c, 0x2a, 0x90, 0x04, 0x8b, 0x01, + 0x7e, 0x02, 0xe2, 0xfd, 0x4d, 0x86, 0x7c, 0xb9, 0x65, 0xc5, 0x8b, 0x16, 0xd5, 0x0b, 0xb0, 0x92, + 0x2a, 0x9f, 0x8f, 0xde, 0x85, 0xda, 0xb8, 0x82, 0xbd, 0x4d, 0x0d, 0x13, 0xf5, 0x4a, 0x24, 0x51, + 0xbf, 0x14, 0x06, 0xb4, 0x39, 0x71, 0x42, 0xd1, 0x98, 0x75, 0x7c, 0x11, 0xc9, 0x4f, 0xba, 0x88, + 0x14, 0xa4, 0x8b, 0x88, 0xfa, 0x38, 0x9c, 0x43, 0x7e, 0x1d, 0xfc, 0x80, 0x5e, 0x58, 0x59, 0xdf, + 0xc2, 0xa9, 0x9d, 0x4f, 0x1f, 0x1f, 0x63, 0xd2, 0x22, 0xfc, 0xea, 0x35, 0xa8, 0xc8, 0xee, 0x2d, + 0xe2, 0xb1, 0x94, 0x84, 0xc7, 0xaa, 0xc6, 0x9c, 0xd5, 0xdb, 0xb1, 0x38, 0x3d, 0x6d, 0x5e, 0x63, + 0x51, 0xfa, 0x5d, 0xc9, 0x6d, 0xbd, 0x26, 0xe5, 0x63, 0xbf, 0x26, 0x8f, 0xb5, 0xc8, 0xfd, 0xf8, + 0xa7, 0x3e, 0x69, 0xcf, 0x07, 0xaa, 0x5e, 0x82, 0xf2, 0xde, 0xa4, 0xef, 0xb8, 0x0a, 0xe2, 0x25, + 0xd0, 0x1d, 0x58, 0xfc, 0xd4, 0x1a, 0x60, 0xff, 0xc8, 0x0f, 0xf0, 0xb0, 0x45, 0xdd, 0xcb, 0xbe, + 0x85, 0x3d, 0xb4, 0x0a, 0x40, 0x2f, 0x57, 0xae, 0x63, 0x85, 0xdf, 0xae, 0x44, 0x28, 0xea, 0x17, + 0x0a, 0xcc, 0x8f, 0x1b, 0xee, 0xd1, 0x2b, 0xe4, 0x79, 0x28, 0x91, 0x91, 0xfa, 0x81, 0x31, 0x74, + 0x45, 0x1e, 0x2b, 0x24, 0xa0, 0x77, 0x61, 0x7a, 0xdf, 0x17, 0x20, 0x54, 0x0c, 0x62, 0x4f, 0x53, + 0x41, 0x2b, 0xec, 0xfb, 0x2d, 0x13, 0xdd, 0x01, 0x18, 0xf9, 0xd8, 0xe4, 0x59, 0xab, 0x7c, 0xf2, + 0xe0, 0xdf, 0x8b, 0xbe, 0x3c, 0x20, 0xac, 0xec, 0xe1, 0xd0, 0x5d, 0x28, 0x5b, 0xb6, 0x63, 0x62, + 0x9a, 0x9f, 0x34, 0x39, 0x42, 0x35, 0xb1, 0x21, 0x30, 0xde, 0x3d, 0x1f, 0x9b, 0xaa, 0xce, 0xcf, + 0x2d, 0x31, 0x9b, 0xdc, 0x14, 0x1e, 0xc2, 0x02, 0x73, 0x3f, 0xfb, 0xa1, 0xb2, 0xc2, 0x1a, 0x57, + 0xd2, 0xc7, 0x42, 0x67, 0x45, 0xab, 0x59, 0x3c, 0x32, 0x11, 0x8d, 0xd4, 0x7b, 0x70, 0x56, 0xba, + 0xaf, 0x9d, 0xe2, 0x02, 0xa5, 0x7e, 0x16, 0x83, 0x61, 0xc6, 0xa6, 0xca, 0xa1, 0x0e, 0x61, 0xa9, + 0x93, 0xa1, 0x0e, 0x9f, 0x41, 0x1d, 0xbe, 0xba, 0x07, 0xe7, 0x24, 0x8c, 0x48, 0xd2, 0xe5, 0x6e, + 0x2c, 0xd8, 0x5a, 0x9b, 0x2c, 0x2f, 0x16, 0x75, 0xfd, 0x8f, 0x02, 0x8b, 0x69, 0x0c, 0x2f, 0x89, + 0x4f, 0xfe, 0x70, 0xc2, 0xb3, 0xd4, 0xdb, 0xc7, 0x29, 0xf4, 0x67, 0xc1, 0x73, 0xdb, 0xd0, 0x48, + 0x9b, 0xc3, 0xe4, 0x9a, 0xe4, 0x4f, 0xb6, 0x26, 0x7f, 0xca, 0x45, 0x30, 0xf8, 0x66, 0x10, 0x78, + 0xd6, 0xd3, 0x11, 0x31, 0xe7, 0x57, 0x88, 0x89, 0x6d, 0x86, 0x48, 0x0f, 0x9b, 0xc8, 0x1b, 0xa9, + 0x0d, 0xc7, 0x7d, 0xa7, 0xa2, 0x3d, 0x5a, 0xda, 0x65, 0xfa, 0xe6, 0x71, 0x92, 0xbe, 0xb5, 0x70, + 0xe9, 0xff, 0x2a, 0x50, 0x95, 0x17, 0x04, 0x7d, 0x0c, 0x60, 0x84, 0x9a, 0xf3, 0x4d, 0x70, 0xf1, + 0x98, 0x01, 0x6a, 0x91, 0x26, 0xe8, 0x32, 0xe4, 0x7b, 0xee, 0x88, 0xaf, 0x8e, 0x94, 0xc8, 0xdd, + 0x74, 0x47, 0xcc, 0x37, 0x10, 0x06, 0x72, 0xad, 0x61, 0x59, 0xf8, 0x34, 0xef, 0xf6, 0x98, 0xd6, + 0x30, 0x6e, 0xce, 0x86, 0xee, 0x43, 0xf5, 0xd0, 0xb3, 0x02, 0xe3, 0xe9, 0x00, 0xeb, 0x03, 0xe3, + 0x08, 0x7b, 0xdc, 0xbb, 0x65, 0xba, 0xa1, 0x8a, 0x68, 0xf2, 0x88, 0xb4, 0x50, 0x5f, 0x40, 0x51, + 0x68, 0x71, 0x8c, 0xdf, 0x6e, 0xc3, 0xf2, 0x88, 0xb0, 0xe9, 0xf4, 0x09, 0xa6, 0x6d, 0xd8, 0x8e, + 0xee, 0x63, 0x72, 0xc0, 0x8a, 0xef, 0x50, 0x26, 0x3a, 0xd5, 0x45, 0xda, 0x6e, 0xd3, 0xf1, 0x70, + 0xdb, 0xb0, 0x9d, 0x2e, 0x6b, 0xa4, 0xba, 0x50, 0x8e, 0x0c, 0xea, 0x98, 0xce, 0x37, 0x61, 0x41, + 0xa4, 0xcb, 0x7d, 0x1c, 0xf0, 0x43, 0xe0, 0x98, 0x6e, 0xe7, 0x79, 0x8b, 0x2e, 0x0e, 0xd8, 0x83, + 0x86, 0x8f, 0xe0, 0x9c, 0x86, 0x1d, 0x17, 0xdb, 0xe1, 0x8a, 0x3d, 0x72, 0xfa, 0xa7, 0xf0, 0xb9, + 0xe7, 0xa1, 0x91, 0xd6, 0x9e, 0xed, 0xf1, 0xeb, 0x97, 0xa1, 0x28, 0xbe, 0x99, 0x47, 0xb3, 0x90, + 0xdf, 0xdd, 0xec, 0xd4, 0xa6, 0xc8, 0x8f, 0xbd, 0xad, 0x4e, 0x4d, 0x41, 0x45, 0x28, 0x74, 0x37, + 0x77, 0x3b, 0xb5, 0xdc, 0xf5, 0x21, 0xd4, 0xe2, 0x9f, 0x8d, 0xa3, 0x65, 0x38, 0xd3, 0xd1, 0x76, + 0x3a, 0xcd, 0x07, 0xcd, 0xdd, 0xd6, 0x4e, 0x5b, 0xef, 0x68, 0xad, 0x27, 0xcd, 0xdd, 0xed, 0xda, + 0x14, 0x5a, 0x87, 0x0b, 0xd1, 0x8a, 0x87, 0x3b, 0xdd, 0x5d, 0x7d, 0x77, 0x47, 0xdf, 0xdc, 0x69, + 0xef, 0x36, 0x5b, 0xed, 0x6d, 0xad, 0xa6, 0xa0, 0x0b, 0x70, 0x2e, 0xca, 0x72, 0xbf, 0xb5, 0xd5, + 0xd2, 0xb6, 0x37, 0xc9, 0xef, 0xe6, 0xa3, 0x5a, 0xee, 0xfa, 0x87, 0x50, 0x91, 0xbe, 0xfb, 0x26, + 0x2a, 0x75, 0x76, 0xb6, 0x6a, 0x53, 0xa8, 0x02, 0xa5, 0xa8, 0x9c, 0x22, 0x14, 0xda, 0x3b, 0x5b, + 0xdb, 0xb5, 0x1c, 0x02, 0x98, 0xd9, 0x6d, 0x6a, 0x0f, 0xb6, 0x77, 0x6b, 0xf9, 0xeb, 0xf7, 0x60, + 0x3e, 0xf6, 0x5e, 0x1d, 0x2d, 0x40, 0xa5, 0xdb, 0x6c, 0x6f, 0xdd, 0xdf, 0xf9, 0x81, 0xae, 0x6d, + 0x37, 0xb7, 0x3e, 0xaf, 0x4d, 0xa1, 0x45, 0xa8, 0x09, 0x52, 0x7b, 0x67, 0x97, 0x51, 0x95, 0xeb, + 0xcf, 0x62, 0x7b, 0x09, 0xa3, 0xb3, 0xb0, 0x10, 0x76, 0xa9, 0x6f, 0x6a, 0xdb, 0xcd, 0xdd, 0x6d, + 0xa2, 0x89, 0x44, 0xd6, 0xf6, 0xda, 0xed, 0x56, 0xfb, 0x41, 0x4d, 0x21, 0x52, 0xc7, 0xe4, 0xed, + 0x1f, 0xb4, 0x08, 0x73, 0x4e, 0x66, 0xde, 0x6b, 0x7f, 0xb7, 0xbd, 0xf3, 0xfd, 0x76, 0x2d, 0x7f, + 0xeb, 0xd7, 0xf3, 0xe1, 0x57, 0xbb, 0x5d, 0xec, 0xd1, 0x37, 0x27, 0x5b, 0x30, 0x2b, 0xfe, 0x13, + 0x83, 0xe4, 0x71, 0xe5, 0xff, 0x1c, 0xd1, 0x58, 0x49, 0xad, 0xe3, 0x71, 0xef, 0x14, 0x7a, 0x42, + 0xe3, 0xd0, 0xc8, 0x77, 0x02, 0x6b, 0xb1, 0xd8, 0x2f, 0xf1, 0x39, 0x42, 0x63, 0x3d, 0x83, 0x23, + 0x94, 0xfb, 0x39, 0x09, 0x32, 0xa3, 0x1f, 0xc9, 0xa1, 0x75, 0x39, 0x46, 0x4c, 0xf9, 0xfe, 0xae, + 0xa1, 0x66, 0xb1, 0x84, 0xa2, 0x75, 0xa8, 0xc5, 0x3f, 0x92, 0x43, 0x12, 0xc4, 0x3a, 0xe1, 0x1b, + 0xbc, 0xc6, 0x6b, 0xd9, 0x4c, 0xd1, 0x0e, 0x12, 0xdf, 0x7e, 0x5d, 0xca, 0xfe, 0x9a, 0x26, 0xa5, + 0x83, 0x49, 0x9f, 0xdc, 0xb0, 0xc9, 0x91, 0x9f, 0xf0, 0xa3, 0xd8, 0xe7, 0x56, 0x29, 0x5f, 0x81, + 0xc8, 0x93, 0x93, 0xfe, 0x05, 0x80, 0x3a, 0x85, 0xfe, 0x1f, 0xcc, 0xc7, 0x9e, 0x0d, 0x20, 0xa9, + 0x61, 0xfa, 0x6b, 0x88, 0xc6, 0xa5, 0x4c, 0x1e, 0x79, 0x55, 0xa3, 0x4f, 0x03, 0xe2, 0xab, 0x9a, + 0xf2, 0xe4, 0x20, 0xbe, 0xaa, 0xa9, 0x2f, 0x0b, 0xa8, 0x21, 0x4a, 0xcf, 0x00, 0x64, 0x43, 0x4c, + 0x7b, 0x76, 0xd0, 0x58, 0xcf, 0xe0, 0x88, 0x4e, 0x48, 0xec, 0x21, 0x80, 0x3c, 0x21, 0xe9, 0x4f, + 0x0c, 0x1a, 0x97, 0x32, 0x79, 0xe2, 0x2b, 0x39, 0x4e, 0x40, 0x26, 0x57, 0x32, 0x91, 0x04, 0x4f, + 0xae, 0x64, 0x32, 0x7f, 0xc9, 0x57, 0x32, 0x96, 0x32, 0x54, 0x33, 0x93, 0x21, 0x69, 0x2b, 0x99, + 0x9e, 0x30, 0x51, 0xa7, 0xd0, 0x21, 0xd4, 0x27, 0xc1, 0xf0, 0xe8, 0xc6, 0x29, 0xb2, 0x05, 0x8d, + 0x37, 0x4e, 0xc6, 0x1c, 0x76, 0x8c, 0x01, 0x25, 0x8f, 0x19, 0xf4, 0xba, 0x3c, 0xdd, 0x13, 0x8e, + 0xb1, 0xc6, 0xe5, 0xe3, 0xd8, 0xc2, 0x6e, 0x1e, 0x40, 0x51, 0x00, 0xfc, 0x48, 0x72, 0x81, 0xb1, + 0xc4, 0x42, 0xe3, 0x7c, 0x7a, 0x65, 0x28, 0xe8, 0x3b, 0x50, 0x20, 0x54, 0xb4, 0x1c, 0xe7, 0x13, + 0x02, 0xea, 0xc9, 0x8a, 0xb0, 0x71, 0x13, 0x66, 0x18, 0x72, 0x8d, 0xa4, 0x2b, 0xb5, 0x84, 0xac, + 0x37, 0x1a, 0x69, 0x55, 0xa1, 0x88, 0x0e, 0xfb, 0xbf, 0x36, 0x1c, 0x88, 0x46, 0xab, 0xf1, 0xcf, + 0xe3, 0x65, 0xc4, 0xbb, 0x71, 0x71, 0x62, 0x7d, 0xd4, 0x66, 0x63, 0x41, 0xe0, 0x7a, 0x46, 0xc4, + 0x9e, 0x66, 0xb3, 0xe9, 0xf7, 0x00, 0xb6, 0xb8, 0xc9, 0x7b, 0x82, 0xbc, 0xb8, 0x13, 0xef, 0x62, + 0xf2, 0xe2, 0x4e, 0xbe, 0x6e, 0xa8, 0x53, 0xe8, 0x00, 0xce, 0xa4, 0xa0, 0x39, 0xe8, 0x72, 0xd2, + 0x14, 0xd3, 0xe0, 0xa4, 0xc6, 0x95, 0x63, 0xf9, 0xa2, 0x0b, 0xc8, 0xf7, 0xde, 0xb9, 0x34, 0x88, + 0x23, 0x65, 0x01, 0xe3, 0x3b, 0xed, 0xd6, 0x3f, 0xe4, 0x61, 0x8e, 0xa1, 0x70, 0xfc, 0xe0, 0x7e, + 0x0c, 0x30, 0x06, 0xb4, 0xd1, 0x85, 0xf8, 0xa8, 0xa5, 0x6c, 0x40, 0x63, 0x75, 0x52, 0x75, 0xd4, + 0x40, 0x22, 0x40, 0xb1, 0x6c, 0x20, 0x49, 0xdc, 0x5b, 0x36, 0x90, 0x14, 0x84, 0x59, 0x9d, 0x42, + 0x9f, 0x41, 0x29, 0xc4, 0x25, 0x91, 0x8c, 0x68, 0xc6, 0x00, 0xd6, 0xc6, 0x85, 0x09, 0xb5, 0x51, + 0xed, 0x22, 0x70, 0xa3, 0xac, 0x5d, 0x12, 0xca, 0x94, 0xb5, 0x4b, 0xc3, 0x29, 0xc7, 0xe3, 0x65, + 0xc0, 0x45, 0xca, 0x78, 0x25, 0x7c, 0x28, 0x65, 0xbc, 0x32, 0xe2, 0xa1, 0x4e, 0xdd, 0x5f, 0xfb, + 0xed, 0x1f, 0x57, 0x95, 0x2f, 0xfe, 0xb8, 0x3a, 0xf5, 0x17, 0x5f, 0xae, 0x2a, 0xbf, 0xfd, 0x72, + 0x55, 0xf9, 0xfd, 0x97, 0xab, 0xca, 0x1f, 0xbe, 0x5c, 0x55, 0xfe, 0xfa, 0xbf, 0x57, 0xa7, 0x7e, + 0x98, 0x7b, 0xfe, 0xf6, 0xd3, 0x19, 0xfa, 0x5f, 0xa2, 0x6e, 0xff, 0x5f, 0x00, 0x00, 0x00, 0xff, + 0xff, 0x5e, 0xaf, 0xb3, 0x38, 0xdf, 0x4b, 0x00, 0x00, } // Reference imports to suppress errors if they are not otherwise used. @@ -7632,7 +7621,8 @@ type RuntimeServiceClient interface { // StopContainer stops a running container with a grace period (i.e., timeout). // This call is idempotent, and must not return an error if the container has // already been stopped. - // TODO: what must the runtime do after the grace period is reached? + // The runtime must forcibly kill the container after the grace period is + // reached. StopContainer(ctx context.Context, in *StopContainerRequest, opts ...grpc.CallOption) (*StopContainerResponse, error) // RemoveContainer removes the container. If the container is running, the // container must be forcibly removed. @@ -7681,7 +7671,7 @@ func NewRuntimeServiceClient(cc *grpc.ClientConn) RuntimeServiceClient { func (c *runtimeServiceClient) Version(ctx context.Context, in *VersionRequest, opts ...grpc.CallOption) (*VersionResponse, error) { out := new(VersionResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/Version", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/Version", in, out, opts...) if err != nil { return nil, err } @@ -7690,7 +7680,7 @@ func (c *runtimeServiceClient) Version(ctx context.Context, in *VersionRequest, func (c *runtimeServiceClient) RunPodSandbox(ctx context.Context, in *RunPodSandboxRequest, opts ...grpc.CallOption) (*RunPodSandboxResponse, error) { out := new(RunPodSandboxResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/RunPodSandbox", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/RunPodSandbox", in, out, opts...) if err != nil { return nil, err } @@ -7699,7 +7689,7 @@ func (c *runtimeServiceClient) RunPodSandbox(ctx context.Context, in *RunPodSand func (c *runtimeServiceClient) StopPodSandbox(ctx context.Context, in *StopPodSandboxRequest, opts ...grpc.CallOption) (*StopPodSandboxResponse, error) { out := new(StopPodSandboxResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/StopPodSandbox", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/StopPodSandbox", in, out, opts...) if err != nil { return nil, err } @@ -7708,7 +7698,7 @@ func (c *runtimeServiceClient) StopPodSandbox(ctx context.Context, in *StopPodSa func (c *runtimeServiceClient) RemovePodSandbox(ctx context.Context, in *RemovePodSandboxRequest, opts ...grpc.CallOption) (*RemovePodSandboxResponse, error) { out := new(RemovePodSandboxResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/RemovePodSandbox", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/RemovePodSandbox", in, out, opts...) if err != nil { return nil, err } @@ -7717,7 +7707,7 @@ func (c *runtimeServiceClient) RemovePodSandbox(ctx context.Context, in *RemoveP func (c *runtimeServiceClient) PodSandboxStatus(ctx context.Context, in *PodSandboxStatusRequest, opts ...grpc.CallOption) (*PodSandboxStatusResponse, error) { out := new(PodSandboxStatusResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/PodSandboxStatus", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/PodSandboxStatus", in, out, opts...) if err != nil { return nil, err } @@ -7726,7 +7716,7 @@ func (c *runtimeServiceClient) PodSandboxStatus(ctx context.Context, in *PodSand func (c *runtimeServiceClient) ListPodSandbox(ctx context.Context, in *ListPodSandboxRequest, opts ...grpc.CallOption) (*ListPodSandboxResponse, error) { out := new(ListPodSandboxResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/ListPodSandbox", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/ListPodSandbox", in, out, opts...) if err != nil { return nil, err } @@ -7735,7 +7725,7 @@ func (c *runtimeServiceClient) ListPodSandbox(ctx context.Context, in *ListPodSa func (c *runtimeServiceClient) CreateContainer(ctx context.Context, in *CreateContainerRequest, opts ...grpc.CallOption) (*CreateContainerResponse, error) { out := new(CreateContainerResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/CreateContainer", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/CreateContainer", in, out, opts...) if err != nil { return nil, err } @@ -7744,7 +7734,7 @@ func (c *runtimeServiceClient) CreateContainer(ctx context.Context, in *CreateCo func (c *runtimeServiceClient) StartContainer(ctx context.Context, in *StartContainerRequest, opts ...grpc.CallOption) (*StartContainerResponse, error) { out := new(StartContainerResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/StartContainer", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/StartContainer", in, out, opts...) if err != nil { return nil, err } @@ -7753,7 +7743,7 @@ func (c *runtimeServiceClient) StartContainer(ctx context.Context, in *StartCont func (c *runtimeServiceClient) StopContainer(ctx context.Context, in *StopContainerRequest, opts ...grpc.CallOption) (*StopContainerResponse, error) { out := new(StopContainerResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/StopContainer", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/StopContainer", in, out, opts...) if err != nil { return nil, err } @@ -7762,7 +7752,7 @@ func (c *runtimeServiceClient) StopContainer(ctx context.Context, in *StopContai func (c *runtimeServiceClient) RemoveContainer(ctx context.Context, in *RemoveContainerRequest, opts ...grpc.CallOption) (*RemoveContainerResponse, error) { out := new(RemoveContainerResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/RemoveContainer", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/RemoveContainer", in, out, opts...) if err != nil { return nil, err } @@ -7771,7 +7761,7 @@ func (c *runtimeServiceClient) RemoveContainer(ctx context.Context, in *RemoveCo func (c *runtimeServiceClient) ListContainers(ctx context.Context, in *ListContainersRequest, opts ...grpc.CallOption) (*ListContainersResponse, error) { out := new(ListContainersResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/ListContainers", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/ListContainers", in, out, opts...) if err != nil { return nil, err } @@ -7780,7 +7770,7 @@ func (c *runtimeServiceClient) ListContainers(ctx context.Context, in *ListConta func (c *runtimeServiceClient) ContainerStatus(ctx context.Context, in *ContainerStatusRequest, opts ...grpc.CallOption) (*ContainerStatusResponse, error) { out := new(ContainerStatusResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/ContainerStatus", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/ContainerStatus", in, out, opts...) if err != nil { return nil, err } @@ -7789,7 +7779,7 @@ func (c *runtimeServiceClient) ContainerStatus(ctx context.Context, in *Containe func (c *runtimeServiceClient) UpdateContainerResources(ctx context.Context, in *UpdateContainerResourcesRequest, opts ...grpc.CallOption) (*UpdateContainerResourcesResponse, error) { out := new(UpdateContainerResourcesResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/UpdateContainerResources", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/UpdateContainerResources", in, out, opts...) if err != nil { return nil, err } @@ -7798,7 +7788,7 @@ func (c *runtimeServiceClient) UpdateContainerResources(ctx context.Context, in func (c *runtimeServiceClient) ReopenContainerLog(ctx context.Context, in *ReopenContainerLogRequest, opts ...grpc.CallOption) (*ReopenContainerLogResponse, error) { out := new(ReopenContainerLogResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/ReopenContainerLog", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/ReopenContainerLog", in, out, opts...) if err != nil { return nil, err } @@ -7807,7 +7797,7 @@ func (c *runtimeServiceClient) ReopenContainerLog(ctx context.Context, in *Reope func (c *runtimeServiceClient) ExecSync(ctx context.Context, in *ExecSyncRequest, opts ...grpc.CallOption) (*ExecSyncResponse, error) { out := new(ExecSyncResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/ExecSync", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/ExecSync", in, out, opts...) if err != nil { return nil, err } @@ -7816,7 +7806,7 @@ func (c *runtimeServiceClient) ExecSync(ctx context.Context, in *ExecSyncRequest func (c *runtimeServiceClient) Exec(ctx context.Context, in *ExecRequest, opts ...grpc.CallOption) (*ExecResponse, error) { out := new(ExecResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/Exec", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/Exec", in, out, opts...) if err != nil { return nil, err } @@ -7825,7 +7815,7 @@ func (c *runtimeServiceClient) Exec(ctx context.Context, in *ExecRequest, opts . func (c *runtimeServiceClient) Attach(ctx context.Context, in *AttachRequest, opts ...grpc.CallOption) (*AttachResponse, error) { out := new(AttachResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/Attach", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/Attach", in, out, opts...) if err != nil { return nil, err } @@ -7834,7 +7824,7 @@ func (c *runtimeServiceClient) Attach(ctx context.Context, in *AttachRequest, op func (c *runtimeServiceClient) PortForward(ctx context.Context, in *PortForwardRequest, opts ...grpc.CallOption) (*PortForwardResponse, error) { out := new(PortForwardResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/PortForward", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/PortForward", in, out, opts...) if err != nil { return nil, err } @@ -7843,7 +7833,7 @@ func (c *runtimeServiceClient) PortForward(ctx context.Context, in *PortForwardR func (c *runtimeServiceClient) ContainerStats(ctx context.Context, in *ContainerStatsRequest, opts ...grpc.CallOption) (*ContainerStatsResponse, error) { out := new(ContainerStatsResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/ContainerStats", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/ContainerStats", in, out, opts...) if err != nil { return nil, err } @@ -7852,7 +7842,7 @@ func (c *runtimeServiceClient) ContainerStats(ctx context.Context, in *Container func (c *runtimeServiceClient) ListContainerStats(ctx context.Context, in *ListContainerStatsRequest, opts ...grpc.CallOption) (*ListContainerStatsResponse, error) { out := new(ListContainerStatsResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/ListContainerStats", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/ListContainerStats", in, out, opts...) if err != nil { return nil, err } @@ -7861,7 +7851,7 @@ func (c *runtimeServiceClient) ListContainerStats(ctx context.Context, in *ListC func (c *runtimeServiceClient) UpdateRuntimeConfig(ctx context.Context, in *UpdateRuntimeConfigRequest, opts ...grpc.CallOption) (*UpdateRuntimeConfigResponse, error) { out := new(UpdateRuntimeConfigResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/UpdateRuntimeConfig", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/UpdateRuntimeConfig", in, out, opts...) if err != nil { return nil, err } @@ -7870,7 +7860,7 @@ func (c *runtimeServiceClient) UpdateRuntimeConfig(ctx context.Context, in *Upda func (c *runtimeServiceClient) Status(ctx context.Context, in *StatusRequest, opts ...grpc.CallOption) (*StatusResponse, error) { out := new(StatusResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.RuntimeService/Status", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.RuntimeService/Status", in, out, opts...) if err != nil { return nil, err } @@ -7911,7 +7901,8 @@ type RuntimeServiceServer interface { // StopContainer stops a running container with a grace period (i.e., timeout). // This call is idempotent, and must not return an error if the container has // already been stopped. - // TODO: what must the runtime do after the grace period is reached? + // The runtime must forcibly kill the container after the grace period is + // reached. StopContainer(context.Context, *StopContainerRequest) (*StopContainerResponse, error) // RemoveContainer removes the container. If the container is running, the // container must be forcibly removed. @@ -8035,7 +8026,7 @@ func _RuntimeService_Version_Handler(srv interface{}, ctx context.Context, dec f } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/Version", + FullMethod: "/runtime.v1.RuntimeService/Version", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).Version(ctx, req.(*VersionRequest)) @@ -8053,7 +8044,7 @@ func _RuntimeService_RunPodSandbox_Handler(srv interface{}, ctx context.Context, } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/RunPodSandbox", + FullMethod: "/runtime.v1.RuntimeService/RunPodSandbox", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).RunPodSandbox(ctx, req.(*RunPodSandboxRequest)) @@ -8071,7 +8062,7 @@ func _RuntimeService_StopPodSandbox_Handler(srv interface{}, ctx context.Context } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/StopPodSandbox", + FullMethod: "/runtime.v1.RuntimeService/StopPodSandbox", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).StopPodSandbox(ctx, req.(*StopPodSandboxRequest)) @@ -8089,7 +8080,7 @@ func _RuntimeService_RemovePodSandbox_Handler(srv interface{}, ctx context.Conte } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/RemovePodSandbox", + FullMethod: "/runtime.v1.RuntimeService/RemovePodSandbox", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).RemovePodSandbox(ctx, req.(*RemovePodSandboxRequest)) @@ -8107,7 +8098,7 @@ func _RuntimeService_PodSandboxStatus_Handler(srv interface{}, ctx context.Conte } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/PodSandboxStatus", + FullMethod: "/runtime.v1.RuntimeService/PodSandboxStatus", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).PodSandboxStatus(ctx, req.(*PodSandboxStatusRequest)) @@ -8125,7 +8116,7 @@ func _RuntimeService_ListPodSandbox_Handler(srv interface{}, ctx context.Context } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/ListPodSandbox", + FullMethod: "/runtime.v1.RuntimeService/ListPodSandbox", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).ListPodSandbox(ctx, req.(*ListPodSandboxRequest)) @@ -8143,7 +8134,7 @@ func _RuntimeService_CreateContainer_Handler(srv interface{}, ctx context.Contex } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/CreateContainer", + FullMethod: "/runtime.v1.RuntimeService/CreateContainer", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).CreateContainer(ctx, req.(*CreateContainerRequest)) @@ -8161,7 +8152,7 @@ func _RuntimeService_StartContainer_Handler(srv interface{}, ctx context.Context } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/StartContainer", + FullMethod: "/runtime.v1.RuntimeService/StartContainer", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).StartContainer(ctx, req.(*StartContainerRequest)) @@ -8179,7 +8170,7 @@ func _RuntimeService_StopContainer_Handler(srv interface{}, ctx context.Context, } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/StopContainer", + FullMethod: "/runtime.v1.RuntimeService/StopContainer", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).StopContainer(ctx, req.(*StopContainerRequest)) @@ -8197,7 +8188,7 @@ func _RuntimeService_RemoveContainer_Handler(srv interface{}, ctx context.Contex } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/RemoveContainer", + FullMethod: "/runtime.v1.RuntimeService/RemoveContainer", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).RemoveContainer(ctx, req.(*RemoveContainerRequest)) @@ -8215,7 +8206,7 @@ func _RuntimeService_ListContainers_Handler(srv interface{}, ctx context.Context } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/ListContainers", + FullMethod: "/runtime.v1.RuntimeService/ListContainers", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).ListContainers(ctx, req.(*ListContainersRequest)) @@ -8233,7 +8224,7 @@ func _RuntimeService_ContainerStatus_Handler(srv interface{}, ctx context.Contex } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/ContainerStatus", + FullMethod: "/runtime.v1.RuntimeService/ContainerStatus", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).ContainerStatus(ctx, req.(*ContainerStatusRequest)) @@ -8251,7 +8242,7 @@ func _RuntimeService_UpdateContainerResources_Handler(srv interface{}, ctx conte } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/UpdateContainerResources", + FullMethod: "/runtime.v1.RuntimeService/UpdateContainerResources", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).UpdateContainerResources(ctx, req.(*UpdateContainerResourcesRequest)) @@ -8269,7 +8260,7 @@ func _RuntimeService_ReopenContainerLog_Handler(srv interface{}, ctx context.Con } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/ReopenContainerLog", + FullMethod: "/runtime.v1.RuntimeService/ReopenContainerLog", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).ReopenContainerLog(ctx, req.(*ReopenContainerLogRequest)) @@ -8287,7 +8278,7 @@ func _RuntimeService_ExecSync_Handler(srv interface{}, ctx context.Context, dec } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/ExecSync", + FullMethod: "/runtime.v1.RuntimeService/ExecSync", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).ExecSync(ctx, req.(*ExecSyncRequest)) @@ -8305,7 +8296,7 @@ func _RuntimeService_Exec_Handler(srv interface{}, ctx context.Context, dec func } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/Exec", + FullMethod: "/runtime.v1.RuntimeService/Exec", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).Exec(ctx, req.(*ExecRequest)) @@ -8323,7 +8314,7 @@ func _RuntimeService_Attach_Handler(srv interface{}, ctx context.Context, dec fu } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/Attach", + FullMethod: "/runtime.v1.RuntimeService/Attach", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).Attach(ctx, req.(*AttachRequest)) @@ -8341,7 +8332,7 @@ func _RuntimeService_PortForward_Handler(srv interface{}, ctx context.Context, d } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/PortForward", + FullMethod: "/runtime.v1.RuntimeService/PortForward", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).PortForward(ctx, req.(*PortForwardRequest)) @@ -8359,7 +8350,7 @@ func _RuntimeService_ContainerStats_Handler(srv interface{}, ctx context.Context } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/ContainerStats", + FullMethod: "/runtime.v1.RuntimeService/ContainerStats", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).ContainerStats(ctx, req.(*ContainerStatsRequest)) @@ -8377,7 +8368,7 @@ func _RuntimeService_ListContainerStats_Handler(srv interface{}, ctx context.Con } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/ListContainerStats", + FullMethod: "/runtime.v1.RuntimeService/ListContainerStats", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).ListContainerStats(ctx, req.(*ListContainerStatsRequest)) @@ -8395,7 +8386,7 @@ func _RuntimeService_UpdateRuntimeConfig_Handler(srv interface{}, ctx context.Co } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/UpdateRuntimeConfig", + FullMethod: "/runtime.v1.RuntimeService/UpdateRuntimeConfig", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).UpdateRuntimeConfig(ctx, req.(*UpdateRuntimeConfigRequest)) @@ -8413,7 +8404,7 @@ func _RuntimeService_Status_Handler(srv interface{}, ctx context.Context, dec fu } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.RuntimeService/Status", + FullMethod: "/runtime.v1.RuntimeService/Status", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(RuntimeServiceServer).Status(ctx, req.(*StatusRequest)) @@ -8422,7 +8413,7 @@ func _RuntimeService_Status_Handler(srv interface{}, ctx context.Context, dec fu } var _RuntimeService_serviceDesc = grpc.ServiceDesc{ - ServiceName: "runtime.v1alpha2.RuntimeService", + ServiceName: "runtime.v1.RuntimeService", HandlerType: (*RuntimeServiceServer)(nil), Methods: []grpc.MethodDesc{ { @@ -8548,7 +8539,7 @@ func NewImageServiceClient(cc *grpc.ClientConn) ImageServiceClient { func (c *imageServiceClient) ListImages(ctx context.Context, in *ListImagesRequest, opts ...grpc.CallOption) (*ListImagesResponse, error) { out := new(ListImagesResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.ImageService/ListImages", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.ImageService/ListImages", in, out, opts...) if err != nil { return nil, err } @@ -8557,7 +8548,7 @@ func (c *imageServiceClient) ListImages(ctx context.Context, in *ListImagesReque func (c *imageServiceClient) ImageStatus(ctx context.Context, in *ImageStatusRequest, opts ...grpc.CallOption) (*ImageStatusResponse, error) { out := new(ImageStatusResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.ImageService/ImageStatus", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.ImageService/ImageStatus", in, out, opts...) if err != nil { return nil, err } @@ -8566,7 +8557,7 @@ func (c *imageServiceClient) ImageStatus(ctx context.Context, in *ImageStatusReq func (c *imageServiceClient) PullImage(ctx context.Context, in *PullImageRequest, opts ...grpc.CallOption) (*PullImageResponse, error) { out := new(PullImageResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.ImageService/PullImage", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.ImageService/PullImage", in, out, opts...) if err != nil { return nil, err } @@ -8575,7 +8566,7 @@ func (c *imageServiceClient) PullImage(ctx context.Context, in *PullImageRequest func (c *imageServiceClient) RemoveImage(ctx context.Context, in *RemoveImageRequest, opts ...grpc.CallOption) (*RemoveImageResponse, error) { out := new(RemoveImageResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.ImageService/RemoveImage", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.ImageService/RemoveImage", in, out, opts...) if err != nil { return nil, err } @@ -8584,7 +8575,7 @@ func (c *imageServiceClient) RemoveImage(ctx context.Context, in *RemoveImageReq func (c *imageServiceClient) ImageFsInfo(ctx context.Context, in *ImageFsInfoRequest, opts ...grpc.CallOption) (*ImageFsInfoResponse, error) { out := new(ImageFsInfoResponse) - err := c.cc.Invoke(ctx, "/runtime.v1alpha2.ImageService/ImageFsInfo", in, out, opts...) + err := c.cc.Invoke(ctx, "/runtime.v1.ImageService/ImageFsInfo", in, out, opts...) if err != nil { return nil, err } @@ -8643,7 +8634,7 @@ func _ImageService_ListImages_Handler(srv interface{}, ctx context.Context, dec } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.ImageService/ListImages", + FullMethod: "/runtime.v1.ImageService/ListImages", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(ImageServiceServer).ListImages(ctx, req.(*ListImagesRequest)) @@ -8661,7 +8652,7 @@ func _ImageService_ImageStatus_Handler(srv interface{}, ctx context.Context, dec } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.ImageService/ImageStatus", + FullMethod: "/runtime.v1.ImageService/ImageStatus", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(ImageServiceServer).ImageStatus(ctx, req.(*ImageStatusRequest)) @@ -8679,7 +8670,7 @@ func _ImageService_PullImage_Handler(srv interface{}, ctx context.Context, dec f } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.ImageService/PullImage", + FullMethod: "/runtime.v1.ImageService/PullImage", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(ImageServiceServer).PullImage(ctx, req.(*PullImageRequest)) @@ -8697,7 +8688,7 @@ func _ImageService_RemoveImage_Handler(srv interface{}, ctx context.Context, dec } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.ImageService/RemoveImage", + FullMethod: "/runtime.v1.ImageService/RemoveImage", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(ImageServiceServer).RemoveImage(ctx, req.(*RemoveImageRequest)) @@ -8715,7 +8706,7 @@ func _ImageService_ImageFsInfo_Handler(srv interface{}, ctx context.Context, dec } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: "/runtime.v1alpha2.ImageService/ImageFsInfo", + FullMethod: "/runtime.v1.ImageService/ImageFsInfo", } handler := func(ctx context.Context, req interface{}) (interface{}, error) { return srv.(ImageServiceServer).ImageFsInfo(ctx, req.(*ImageFsInfoRequest)) @@ -8724,7 +8715,7 @@ func _ImageService_ImageFsInfo_Handler(srv interface{}, ctx context.Context, dec } var _ImageService_serviceDesc = grpc.ServiceDesc{ - ServiceName: "runtime.v1alpha2.ImageService", + ServiceName: "runtime.v1.ImageService", HandlerType: (*ImageServiceServer)(nil), Methods: []grpc.MethodDesc{ { diff --git a/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto b/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto similarity index 98% rename from vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto rename to vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto index 1e579187..65560039 100644 --- a/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/api.proto +++ b/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto @@ -1,5 +1,5 @@ /* -Copyright 2018 The Kubernetes Authors. +Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -17,8 +17,8 @@ limitations under the License. // To regenerate api.pb.go run hack/update-generated-runtime.sh syntax = "proto3"; -package runtime.v1alpha2; -option go_package = "v1alpha2"; +package runtime.v1; +option go_package = "v1"; import "github.com/gogo/protobuf/gogoproto/gogo.proto"; @@ -66,7 +66,8 @@ service RuntimeService { // StopContainer stops a running container with a grace period (i.e., timeout). // This call is idempotent, and must not return an error if the container has // already been stopped. - // TODO: what must the runtime do after the grace period is reached? + // The runtime must forcibly kill the container after the grace period is + // reached. rpc StopContainer(StopContainerRequest) returns (StopContainerResponse) {} // RemoveContainer removes the container. If the container is running, the // container must be forcibly removed. @@ -361,11 +362,6 @@ message PodSandboxConfig { // E.g., // PodSandboxConfig.LogDirectory = `/var/log/pods//` // ContainerConfig.LogPath = `containerName/Instance#.log` - // - // WARNING: Log management and how kubelet should interface with the - // container logs are under active discussion in - // https://issues.k8s.io/24677. There *may* be future change of direction - // for logging as the discussion carries on. string log_directory = 3; // DNS config for the sandbox. DNSConfig dns_config = 4; @@ -403,7 +399,7 @@ message RunPodSandboxRequest { // If the runtime handler is unknown, this request should be rejected. An // empty string should select the default handler, equivalent to the // behavior before this feature was added. - // See https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class + // See https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class/README.md string runtime_handler = 2; } @@ -566,8 +562,6 @@ message KeyValue { // LinuxContainerResources specifies Linux specific configuration for // resources. -// TODO: Consider using Resources from opencontainers/runtime-spec/specs-go -// directly. message LinuxContainerResources { // CPU CFS (Completely Fair Scheduler) period. Default: 0 (not specified). int64 cpu_period = 1; @@ -713,7 +707,7 @@ message WindowsSandboxSecurityContext { // The contents of the GMSA credential spec to use to run this container. string credential_spec = 2; - // Indicates whether the container be asked to run as a HostProcess container. + // Indicates whether the container requested to run as a HostProcess container. bool host_process = 3; } @@ -839,8 +833,6 @@ message ContainerConfig { // Variables for interactive containers, these have very specialized // use-cases (e.g. debugging). - // TODO: Determine if we need to continue supporting these fields that are - // part of Kubernetes's Container Spec. bool stdin = 12; bool stdin_once = 13; bool tty = 14; @@ -1018,10 +1010,10 @@ message UpdateContainerResourcesRequest { string container_id = 1; // Resource configuration specific to Linux containers. LinuxContainerResources linux = 2; - // Resource configuration specific to Windows containers. + // Resource configuration specific to Windows containers. WindowsContainerResources windows = 3; - // Unstructured key-value map holding arbitrary additional information for - // container resources updating. This can be used for specifying experimental + // Unstructured key-value map holding arbitrary additional information for + // container resources updating. This can be used for specifying experimental // resources to update or other options to use when updating the container. map annotations = 4; } diff --git a/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/constants.go b/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/constants.go similarity index 97% rename from vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/constants.go rename to vendor/k8s.io/cri-api/pkg/apis/runtime/v1/constants.go index 0e141b7d..6f9ad59e 100644 --- a/vendor/k8s.io/cri-api/pkg/apis/runtime/v1alpha2/constants.go +++ b/vendor/k8s.io/cri-api/pkg/apis/runtime/v1/constants.go @@ -1,5 +1,5 @@ /* -Copyright 2016 The Kubernetes Authors. +Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package v1alpha2 +package v1 // This file contains all constants defined in CRI. diff --git a/vendor/modules.txt b/vendor/modules.txt index 043ea7a1..fad5a46c 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -626,7 +626,7 @@ k8s.io/client-go/util/workqueue # k8s.io/component-base v0.22.8 => k8s.io/component-base v0.22.8 k8s.io/component-base/featuregate # k8s.io/cri-api v0.0.0 => k8s.io/cri-api v0.22.8 -k8s.io/cri-api/pkg/apis/runtime/v1alpha2 +k8s.io/cri-api/pkg/apis/runtime/v1 # k8s.io/klog v1.0.0 k8s.io/klog # k8s.io/klog/v2 v2.9.0