diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md new file mode 100644 index 00000000..9b9061de --- /dev/null +++ b/docs/TROUBLESHOOTING.md @@ -0,0 +1,286 @@ +# Troubleshooting guide + +## Enable debug logs + +This section describes how to increase log verbosity to enable debug logs of various GoVPP components. + +### core + +Set the environment variable `DEBUG_GOVPP` to any non-empty value. + +Example: +``` +DEBUG_GOVPP=y ./bin/rpc-service +``` + +To verify that it works, check for the following line in the console output: +``` +DEBU[0000] govpp: debug level enabled +``` + +### msgid + +Assign value `msgid` to the variable `DEBUG_GOVPP`. + +NOTE: To make msgid debugging work, you must also enable core debugging (in this case, it is enabled automatically, because `msgid` is a non-emtpy value). + +Example: +``` +DEBUG_GOVPP=msgid ./bin/rpc-service +``` + +It should output lines similar to this: +``` +DEBU[0000] message "show_version" (show_version_51077d14) has ID: 1380 +``` + +### socketclient + +Assign value `socketclient` to the variable `DEBUG_GOVPP`. + +Example: +``` +DEBUG_GOVPP=socketclient ./bin/rpc-service +``` + +To verify that it works, check for the following line in the console output: +``` +DEBU[0000] govpp: debug level enabled for socketclient +``` + +### msgtable + +Assign value `msgtable` to the variable `DEBUG_GOVPP`. + +NOTE: To make msgtable debugging work, you must also enable socketclient debugging. + +NOTE: To assign multiple values to a variable, you can separate the values with a colon. + +Example: +``` +DEBUG_GOVPP=socketclient:msgtable ./bin/rpc-service +``` + +It should output lines similar to this: +``` +DEBU[0000] - 1380: "show_version_51077d14" logger=govpp/socketclient +``` + +### statsclient + +Set the environment variable `DEBUG_GOVPP_STATS` to any non-empty value. + +NOTE: statsclient debugging only works if package `go.fd.io/govpp/adapter/statsclient` is imported. + +Example: +``` +DEBUG_GOVPP_STATS=y ./bin/vpp-proxy server +``` + +To verify that it works, check for the following line in the console output: +``` +DEBU[0000] govpp/statsclient: enabled debug mode +``` + +### proxy + +Set the environment variable `DEBUG_GOVPP_PROXY` to any non-empty value. + +NOTE: proxy debugging only works if package `go.fd.io/govpp/proxy` is imported. + +Example: +``` +DEBUG_GOVPP_PROXY=y ./bin/vpp-proxy server +``` + +To verify that it works, check for the following line in the console output: +``` +DEBU[0000] govpp/proxy: debug mode enabled +``` + +### Binapi generator debug logs + +You can enable logs separately for vppapi parser and for binapi generator. The binapi generator has two verbosity levels. + +To enable parser logs, assign value `parser` to the variable `DEBUG_GOVPP`. + +To enable the first verbosity level for generator, pass the flag `-debug` when invoking the generator binary. + +To enable the second verbosity level for generator, assign value `binapigen` to the variable `DEBUG_GOVPP`. + +Examples: +```bash +# first verbosity level for generator +./bin/binapi-generator -debug +# maximum logs for all components +DEBUG_GOVPP=parser:binapigen ./bin/binapi-generator +``` + + + +## Unknown message error + +**Symptoms**: When you try to send/receive a message, you see `unknown message: ...` error. + +**Cause**: Incompatibility of messages between VPP binary API and GoVPP's generated Go bindings for binary API (binapi). + +**Solution 1**: Regenerate binapi. Use make target `generate-binapi` (for VPP installed via package manager) or `gen-binapi-local` (for locally built VPP) or `gen-binapi-docker` (independent of local VPP). For more information about the generator, see [the generator documentation](GENERATOR.md). After generating binapi, rebuild GoVPP. + +**Solution 2**: Use VPP version that is compatible with GoVPP's binapi. To see for which VPP version the binapi was generated, look at the beginning of any `.ba.go` file (not `_rpc.ba.go`). + + + +## Connection problems + +This section describes various problems when GoVPP tries to connect to VPP. + +### Socket file does not exist + +**Symptoms**: You see the following error: +``` +ERROR: connecting to VPP failed: VPP API socket file /run/vpp/api.sock does not exist +``` + +**Probable cause**: VPP is not running or it is configured to have the socket at non-default path. + +**Possible solutions**: + +- Check if the VPP is running. It may have exited unexpectedly. + +- Run the VPP with `sudo`. It may need root permissions to create the socket. + +- Configure the socket path in VPP configuration. Find the file `startup.conf` (default location: `/etc/vpp/startup.conf`) or create it if it does not exist and add there the following: + ``` + socksvr { + socket-name /path/to/api.sock + } + ``` + (change the path to the actual path, the default is `/run/vpp/api.sock`). Then run the VPP with the `-c` flag and specify the path to the configuration file. For example, with default locations: `/usr/bin/vpp -c /etc/vpp/startup.conf`. + + - If you built VPP from source, you can apply the configuration using `make` with `STARTUP_CONF` environment variable. For example: `STARTUP_CONF=/etc/vpp/startup.conf make run`. + + - If VPP outputs a line similar to this: `clib_socket_init: bind (fd 9, '/home/user/vpp/api.sock'): No such file or directory`, it likely means that the directory for the socket does not exist. If this happens, create the directory manually and restart VPP. + +- In GoVPP code that connects to the VPP, use the actual socket path configured by VPP: + ```go + govpp.Connect("/path/to/api.sock") + ``` + (change the path to the actual path). + + - In GoVPP examples, you can usually set the path with the flag `-socket` or similar (use `--help` to see the actual flag). For example: `./bin/rpc-service -sock /run/vpp/api.sock`. + +- If you are running VPP inside Docker container, mount the socket directory as a Docker volume so it can be used from the host. If you use configuration file, also mount the file. For example: `docker run -it -v /run/vpp:/run/vpp -v /home/user/vpp/startup.conf:/etc/vpp/startup.conf ligato/vpp-base:22.10` + +### Connection refused + +**Symptoms**: You see the following error: +``` +ERROR: connecting to VPP failed: dial unix /run/vpp/api.sock: connect: connection refused +``` + +**Probable cause**: VPP exited or its configuration changed and you are trying to connect to the now obsolete socket. + +**Possible solutions**: See the solutions in the previous subsection. + +### Permission denied + +**Symptoms**: You see the following error: +``` +ERROR: connecting to VPP failed: dial unix /run/vpp/api.sock: connect: permission denied +``` + +**Probable cause**: The socket is owned by the `root` group. + +**Possible solutions**: + +- Configure the VPP to create the socket as current user's group. Add the following to the file `startup.conf`: + ``` + unix { + gid + } + ``` + (replace the `` with actual group ID; you can see the ID with `id -g`). + + - If you are running VPP inside Docker container, run the container as current user's group (but keep the user as root), for example `docker run -it -v /run/vpp:/run/vpp -u 0:$(id -g) ligato/vpp-base:22.10`. In this case, do NOT use the `unix { gid ... }` in `startup.conf`. + +- Run GoVPP with `sudo`. + +- Manually change permissions or ownership of the socket file. + + + +## Message versioning, deprecated and in-progesss messages + +When a GoVPP's message is updated, the message itself is not changed. Instead, a new version of the message is added that has a `v2` (or `v3`, `v4`, ...) suffix to its name. The old version of the message is then marked as deprecated. + +### Deprecated messages + +If a message is marked as deprecated, it should no longer be used. The message works fine for now, but it may be removed in the future. When a message is deprecated, there is likely available a new version of the message or an another message with a similar behaviour which should be used instead. Also, if some message is not deprecated, but there already exists a new version of the message, the new version should be used, as the older version may soon become deprecated. + +For example, here is an excerpt from the file `binapi/vhost_user/vhost_user.ba.go` (as of 31-Oct-2022): +```go +// CreateVhostUserIf defines message 'create_vhost_user_if'. +// Deprecated: the message will be removed in the future versions +type CreateVhostUserIf struct { + // ... +} + +// ... + +// CreateVhostUserIfReply defines message 'create_vhost_user_if_reply'. +// Deprecated: the message will be removed in the future versions +type CreateVhostUserIfReply struct { + // ... +} + +// ... + +// CreateVhostUserIfV2 defines message 'create_vhost_user_if_v2'. +type CreateVhostUserIfV2 struct { + // ... +} + +// ... + +// CreateVhostUserIfV2Reply defines message 'create_vhost_user_if_v2_reply'. +type CreateVhostUserIfV2Reply struct { + // ... +} +``` +The messages `CreateVhostUserIf` and `CreateVhostUserIfReply` are deprecated and should not be used. Instead, there are new versions of the messages: `CreateVhostUserIfV2` and `CreateVhostUserIfV2Reply`, which should be used. + +### In-progress messages + +When a new message is added, usually it is initially experimental and is marked as in-progress. In this case, an older message should be preferrably used. Later, when the message will be considered stable, the in-progress label will be removed, the message will become recommended to use and the old message may become deprecated. + +For example, here is an excerpt from the file `binapi/ip/ip.ba.go` (as of 31-Oct-2022): +```go +// IPRouteAddDel defines message 'ip_route_add_del'. +type IPRouteAddDel struct { + // ... +} + +// ... + +// IPRouteAddDelReply defines message 'ip_route_add_del_reply'. +type IPRouteAddDelReply struct { + // ... +} + +// ... + +// IPRouteAddDelV2 defines message 'ip_route_add_del_v2'. +// InProgress: the message form may change in the future versions +type IPRouteAddDelV2 struct { + // ... +} + +// ... + +// IPRouteAddDelV2Reply defines message 'ip_route_add_del_v2_reply'. +// InProgress: the message form may change in the future versions +type IPRouteAddDelV2Reply struct { + // ... +} +``` +In this case, the messages `IPRouteAddDel` and `IPRouteAddDelReply` should be used; the messages `IPRouteAddDelV2` and `IPRouteAddDelV2Reply` are in-progress.