vpp-agent pending my order even the dependencies interface are all exist

[py-162157]

I send a l2xconnect order to vpp-agent by etcd, and the vpp-agent watcher successfully caught my order as below:

+======================================================================================================================+
| #9 - NBTransaction                                                                                                   |
+======================================================================================================================+
  * transaction arguments:
      - seqNum: 9
      - type: NB Transaction
      - values:
          - key: config/vpp/l2/v2/xconnect/tap0
            val: { receive_interface:"tap0"  transmit_interface:"memif0/0" }

o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2021-11-04 06:53:54.824 +0000 UTC -> 2021-11-04 06:53:54.824 +0000 UTC, dur: 0s):
      1. CREATE [NOOP IS-PENDING]:
          - key: config/vpp/l2/v2/xconnect/tap0
          - value: { receive_interface:"tap0"  transmit_interface:"memif0/0" } 
x----------------------------------------------------------------------------------------------------------------------x
| #9 - NBTransaction                                                                                        took 200µs |
x----------------------------------------------------------------------------------------------------------------------x

in ligato document on KV Scheduler, I find that the state [NOOP IS-PENDING] imply that the order's dependencies are not meet, namely the two interface which are going to be l2xconnectd are not both exist, but after check it by vppctl order, I found they are all exist and state up. Why it gotta happend? It's a bug or I miss something?

    _______    _        _   _____  ___ 
 __/ __/ _ \  (_)__    | | / / _ \/ _ \
 _/ _// // / / / _ \   | |/ / ___/ ___/
 /_/ /____(_)_/\___/   |___/_/  /_/    

vpp# show int addr
local0 (dn):
loop0 (dn):
memif0/0 (up):
tap0 (up):

By vppctl, I can successfully l2xconnect them by cmd set int l2 xconnect memif0/0 tap0. And this is the version informations

                                       __
  _  _____  ___ _______ ____ ____ ___ / /_  
 | |/ / _ \/ _ /___/ _ '/ _ '/ -_/ _ / __/  vpp-agent v3.3.0-alpha-30-g7a574ece0
 |___/ .__/ .__/   \_'_/\_' /\__/_//_\__/   Wed Aug 18 10:13:33 UTC 2021 (78 days ago)
    /_/  /_/           /___/                root@buildkitsandbox (go1.15.2 linux/amd64)

[chrismetz09]

Looks to me like you did things correctly. If possible, run the following to assist troubleshooting:
agentctl report
agentct config history
agentct vpp show version
KVS txn history API.

@ofabry, thoughts?

[py-162157]

Looks to me like you did things correctly. If possible, run the following to assist troubleshooting: agentctl report agentct config history agentct vpp show version KVS txn history API.

@ofabry, thoughts?

Hi, thanks for your attention, here is some output of agentctl report, agentctl config history and agentctl vpp show version:

# agentctl report
Retrieving agent version... Done.
Retrieving vpp version... Done.
Retrieving vpp cpu information... Done.
Retrieving vpp numa information... Done.
Retrieving vpp main-heap memory information... Done.
Retrieving vpp numa-heap memory information... Done.
Retrieving vpp phys memory information... Done.
Retrieving vpp api-segment memory information... Done.
Retrieving vpp stats-segment memory information... Done.
Retrieving agent status... Done.
Retrieving agent transaction history configuration... Done.
Retrieving agent NB configuration... Done.
Retrieving agent kvscheduler NB configuration... Done.
Retrieving agent kvscheduler SB configuration... Error... (this error will be part of the report zip file, see _failed-reports.txt):
dumping kvscheduler data failed due to:
Failed to get data for SB view and key prefix config/vpp/dns/v1/dnscache due to: Error response from daemon: [500] {
  "Error": "descriptor does not support Retrieve operation"
}

Skipping this subreport.

Retrieving agent kvscheduler cached configuration... Done.
Retrieving vpp startup config... Done.
Retrieving vpp running configuration... Done.
Retrieving vpp start time information(for event-log)... Done.
Retrieving vpp event-log information... Done.
Retrieving vpp log information... Done.
Retrieving vpp interface statistics... Done.
Retrieving vpp error statistics... Done.
Saving vpp api trace remotely to a file... Done.
Retrieving vpp api trace from saved remote file... Done.
Retrieving vpp SRv6 localsid information... Done.
Retrieving vpp SRv6 policies information... Done.
Retrieving vpp SRv6 steering information... Done.
1 subreport(s) failed.

If you want to ignore errors from subreports and create report from the successfully retrieved/processed information then add the --ignore-errors (-i) argument to the command (i.e. 'agentctl report -i')
ERROR: dumping kvscheduler data failed due to:
Failed to get data for SB view and key prefix config/vpp/dns/v1/dnscache due to: Error response from daemon: [500] {
  "Error": "descriptor does not support Retrieve operation"
}

# agentctl config history
  SEQ  TYPE            START  INPUT      OPERATIONS          RESULT  SUMMARY                
  0    status sync     4m     <none>     <none>                      <none>                 
  1    config replace  4m     18 values  <none>                      <none>                 
  2    config change   4m     1  values  CREATE:1            ok      CONFIGURED:1           
  3    status update   4m     1  values  CREATE:1            ok      OBTAINED:1             
  4    status update   4m     1  values  CREATE:1            ok      OBTAINED:1             
  5    config change   4m     1  values  <none>                      <none>                 
  6    status update   4m     1  values  CREATE:1            ok      OBTAINED:1             
  7    config change   4m     1  values  CREATE:1            ok      CONFIGURED:1           
  8    status update   4m     1  values  CREATE:1            ok      OBTAINED:1             
  9    config change   4m     1  values  <none>                      <none>                 
  10   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  11   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  12   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  13   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  14   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  15   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  16   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  17   config change   4m     1  values  CREATE:1            ok      PENDING:1              
  18   status update   3m     2  values  CREATE:1, DELETE:1  ok      OBTAINED:1, REMOVED:1  
  19   status update   3m     2  values  CREATE:1, DELETE:1  ok      OBTAINED:1, REMOVED:1  

Here is some error when run agentctl vpp show version, but I got the vpp version information in vpp-agent start log as follows:

time="2021-11-05 02:00:40.62692" level=info msg="VPP version: 20.09.0-64~g4f7b92f0a~b84" ClientID=0 PID=17 loc="govppmux/plugin_impl_govppmux.go(384)" logger=govpp

[ondrej-fabry]

Could you upload the data exported by agentctl report command?