Reconfigure VM vSphere -> Network adapter not displayed

[Berhin59]

Hello,

When trying to reconfigure a vSphere virtual machine on the vCenter I don't see the network adapters displayed...
However I see them correctly in the VM details
A refresh doesn't change anything either... does anyone have a solution please?

Thanks

Bug :

Network in vm detail page

[agrare]

Thanks @Berhin59 I'm wondering if the LAN is missing, can you run the following in a rails console (ssh to appliance, cd /var/www/miq/vmdb, rails c)

vm.hardware.guest_devices.order(:device_name => 'asc').each do
  lan = Lan.find_by(:id => guest_device.lan_id)
  pp "#{guest_device.device_name} lan: #{lan&.name}"
end; nil

[Berhin59]

Here the result.
I did something wrong?

[agrare]

Oh sorry, you have to query for the vm first. In the rails console first run vm = ManageIQ::Providers::Vmware::InfraManager::Vm.find_by(:name => "NAME_OF_YOUR_VM")

[Berhin59]

The result

[agrare]

Wow it is the weekend and my head really isn't in it today 😆

vm.hardware.guest_devices.order(:device_name => 'asc').each do |guest_device|
  lan = Lan.find_by(:id => guest_device.lan_id)
  pp "#{guest_device.device_name} lan: #{lan&.name}"
end; nil

I did test this and this is what the output looks like for me:

irb(main):001:0> vm = ManageIQ::Providers::Vmware::InfraManager::Vm.first
=> 
#<ManageIQ::Providers::Vmware::InfraManager::Vm:0x00007f45db02a9e0
irb(main):002:1* vm.hardware.guest_devices.order(:device_name => 'asc').each do |guest_device|
irb(main):003:1*   lan = Lan.find_by(:id => guest_device.lan_id)
irb(main):004:1*   pp "#{guest_device.device_name} lan: #{lan&.name}"
irb(main):005:0> end; nil
"ethernet-0 lan: DC0_DVPG0"

[Berhin59]

Ok apparently there is no LAN it's weird.
How do I get him to see it?

[agrare]

Okay that's what I thought, couple of questions: what version of ManageIQ is this, what version of vSphere is this, and any info about that VM's NIC (is it a "normal" Network, a DV PortGroup, etc...)

[Berhin59]

So it's the last version of ManageIQ (Oparin) and the bug exist on Jansa-2 too
vCenter version 7.0.3
All VM are attached to networks in a dvswitch

In compute - Infra - networking i can see the dvswitch and all port group associated

[agrare]

Okay thank you, that is a pretty standard configuration. Can you turn on debug logging (accessible from the Application Settings page) the manually run a full refresh from the provider Configuration dropdown. Once the refresh completes grab the evm.log (journalctl --since "1 hour ago" --identifier=evm > evm.log)

[Berhin59]

evm.log

After the refresh I see correctly the network adapt in the reconfiguration menu of the VM

But after a few moments they disappear this is so strange

[agrare]

Okay the log looks clean no warnings or errors, and you still don't see the lan after running a full refresh?

Going to need to take a look at that nic's backing in the VMware ManagedObjectBrowser and probably drop some debug logging in the parser.

[Berhin59]

Exactly, in fact after the refresh i see network adapter during few minutes and after they disappear spontaneously

[agrare]

Okay I'd like to confirm if this is a targeted refresh issue (getting cleared on subsequent updates) or a parser/refresh issue.

Can you

# systemctl stop evmserverd.service
# cd /var/www/miq/vmdb
# rails c
> ems = ManageIQ::Providers::Vmware::InfraManager.first
> ems.class::Inventory::Collector.new(ems).refresh
> vm = ems.vms.find_by(:name => "NAME_OF_YOUR_VM"
> pp vm.hardware.guest_devices.map { |gd| "#{gd.device_name} #{gd.lan_id}" }.join("\n")

[Berhin59]

I do this tomorrow and i come back to you

[agrare]

Sounds good thanks @Berhin59

[Berhin59]

Hello, here is the output of your commands :

[agrare]

Okay thanks so after a full refresh the lan_id is there. We're going to need to see what object_update is causing us to lose that lan_id. I'll work up a patch that will log when the lan_id gets nil'd out so we can see what update caused it.

[agrare]

Okay can you apply this patch to the VMware refresh collector:

diff --git a/app/models/manageiq/providers/vmware/infra_manager/inventory/collector.rb b/app/models/manageiq/providers/vmware/infra_manager/inventory/collector.rb
index 7970bcd5..7e3ecfdc 100644
--- a/app/models/manageiq/providers/vmware/infra_manager/inventory/collector.rb
+++ b/app/models/manageiq/providers/vmware/infra_manager/inventory/collector.rb
@@ -96,6 +96,9 @@ class ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector
 
     save_inventory(vim, persister)
 
+    vms = vc.vms.select { |vm| vm.hardware.nics.any? { |nic| nic.lan_id.nil? } }
+    vms.each { |vm| _log.error("***** VM [#{vm.name}] [#{vm.ems_ref}] has nics with nil lan_id: #{vm.hardware.nics.select { |nic| nic.lan_id.nil? }.inspect}") }
+
     self.last_full_refresh = Time.now.utc
     clear_cis_taggings
 
@@ -113,6 +116,9 @@ class ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector
       parse_updates(vim, parser, updated_objects)
       save_inventory(vim, persister)
 
+      vms = vc.vms.select { |vm| vm.hardware.nics.any? { |nic| nic.lan_id.nil? } }
+      vms.each { |vm| _log.error("***** VM [#{vm.name}] [#{vm.ems_ref}] has nics with nil lan_id: #{vm.hardware.nics.select { |nic| nic.lan_id.nil? }.inspect}") }
+

You can find the path to this provider with cd /var/www/miq/vmdb; bundle show manageiq-providers-vmware and it'll be app/models/manageiq/providers/vmware/infra_manager/inventory/collector.rb in that directory

[Berhin59]

Hello,
Hello,

I did not deploy the manageiq via git but I just added these lines of code in the corresponding file :

/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-vmware-1f52e4394b2e/app/models/manageiq/providers/vmware/infra_manager/inventory/collector.rb

What should I do now?

Thanks

[agrare]

Now systemctl restart evmserverd to pick up the code changes and watch the logs for that error

[Berhin59]

Here the log file :
evm.log

[agrare]

Hm that doesn't seem right let me send you the whole file to apply so we don't have to paste lines
collector.zip

[Berhin59]

I pasted the file in the directory
Here are the logs after the server reboot and the loss of the network cards in VM reconfiguration mode :
evm.zip

[agrare]

Perfect thanks @Berhin59

So after the first full refresh I see one VM with nil lan_id on three adapters:

Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: ERROR -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#full_refresh) ***** VM [ADV001FM01] [vm-2893] has nics with nil lan_id: [#<GuestDevice id: 246, device_name: "Network adapter 2", device_type: "ethernet", location: nil, filename: nil, hardware_id: 141, mode: nil, controller_type:     "ethernet", size: nil, free_space: nil, size_on_disk: nil, address: "00:50:56:9b:41:2e", switch_id: nil, lan_id: nil, model: "VirtualVmxnet3", iscsi_name: nil, iscsi_alias: nil, present: false, start_connected: false, auto_detect: nil, uid_ems: "00:50:56:9b:41:2e", chap_auth_enabled: nil, manufacturer: nil, field_replaceable_unit: nil, parent_device_id: nil, vlan_key: nil,     vlan_enabled: nil, peer_mac_address: nil, ems_ref: nil, speed: nil>, #<GuestDevice id: 247, device_name: "Network adapter 3", device_type: "ethernet", location: nil, filename: nil, hardware_id: 141, mode: nil, controller_type: "ethernet", size: nil, free_space: nil, size_on_disk: nil, address: "00:50:56:9b:3f:7a", switch_id: nil, lan_id: nil, model: "VirtualVmxnet3",           iscsi_name: nil, iscsi_alias: nil, present: false, start_connected: false, auto_detect: nil, uid_ems: "00:50:56:9b:3f:7a", chap_auth_enabled: nil, manufacturer: nil, field_replaceable_unit: nil, parent_device_id: nil, vlan_key: nil, vlan_enabled: nil, peer_mac_address: nil, ems_ref: nil, speed: nil>, #<GuestDevice id: 248, device_name: "Network adapter 4", device_type:         "ethernet", location: nil, filename: nil, hardware_id: 141, mode: nil, controller_type: "ethernet", size: nil, free_space: nil, size_on_disk: nil, address: "00:50:56:9b:24:5e", switch_id: nil, lan_id: nil, model: "VirtualVmxnet3", iscsi_name: nil, iscsi_alias: nil, present: false, start_connected: false, auto_detect: nil, uid_ems: "00:50:56:9b:24:5e", chap_auth_enabled: nil,   manufacturer: nil, field_replaceable_unit: nil, parent_device_id: nil, vlan_key: nil, vlan_enabled: nil, peer_mac_address: nil, ems_ref: nil, speed: nil>]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]:  INFO -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#vim_collector) EMS: [vPACKVC], id: [2] Refreshing initial inventory...Complete

This seems like a different issue which we can look into.

Then right after the full finishes we get into the first set of targeted refreshes:

Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]:  INFO -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#process_update_set) EMS: [vPACKVC], id: [2] Processing 27 updates...
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-5537] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-53] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-71] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-46803] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-46802] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-133163] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-10676] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-10586] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-10680] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-4011] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-10554] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-760] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [ClusterComputeResource:domain-c47] Kind: [modify] Props: [summary.effectiveCpu, summary.effectiveMemory]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [ResourcePool:resgroup-48] Kind: [modify] Props: [summary.config.cpuAllocation.expandableReservation, summary.config.cpuAllocation.limit, summary.config.cpuAllocation.reservation,     summary.config.cpuAllocation.shares.level, summary.config.cpuAllocation.shares.shares, ...]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-4031] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-3774] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-3995] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-6125] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-1152] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-10834] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-31] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-3996] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-1748] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-751] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-5488] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-10577] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#log_object_update) EMS: [vPACKVC], id: [2] Object: [VirtualMachine:vm-48157] Kind: [modify] Props: [summary.storage.committed, summary.storage.unshared]
Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]:  INFO -- evm: MIQ(ManageIQ::Providers::Vmware::InfraManager::Inventory::Collector#process_update_set) EMS: [vPACKVC], id: [2] Processing 27 updates...Complete

We aren't deleting any Lans:

Mar 21 16:40:02 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: Skipping InventoryCollection:<Lan>, strategy: local_db_find_missing_references processing because it will do no operation.

We are updating a number of GuestDevices:

Mar 21 16:40:03 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: ----- BEGIN ----- Saving collection InventoryCollection:<GuestDevice>, strategy: local_db_find_missing_references of size 31 to the database, for the manager: 'vPACKVC'...
Mar 21 16:40:03 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: Processing InventoryCollection:<GuestDevice>, strategy: local_db_find_missing_references of size 31...
Mar 21 16:40:03 vpackmgiq-oparin.p0330531102.vpack evm[80035]: DEBUG -- evm: Processing InventoryCollection:<GuestDevice>, strategy: local_db_find_missing_references, created=0, updated=29, deleted=0...Complete

Then we see the warning about nil lan_id for each of these targeted VMs. The updates for these VMs don't have anything to do with their network adapters though so this is likely a targeted scope issues with the dvportgroups.

I'm going to see if I can reproduce this in specs and get back to you.

[Berhin59]

Perfect, thanks for your feedback, I understand better with the explanations.
It's still strange that nobody has reported this problem yet...

I'm waiting to hear from you and I remain available if you want me to do other operations

[Berhin59]

Hello,
Do you have any good news for me?

[agrare]

Hey @Berhin59 so far I haven't been able to reproduce this, we actually have spec tests that cover and unrelated not causing changes to any other type of inventory.

I'm going to see if something about the storage summary updates are causing this, if I can't reproduce it I might need to get a database dump and a recording of the vsphere updates.

I'll get back to you shortly on this, thanks.

[Berhin59]

Hello, do you have some news about this problem ?
Thanks

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 3 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

[Berhin59]

Hello,

We still have concerns about this problem.
Even on the latest versions, it reproduces. Do you have any ideas for resolutions ?

Thanks

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 3 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

[paulrichter2001]

Hello,
is there any progress on this?
We experience the exact same problem.

Best regards

[agrare]

Hey @paulrichter2001 @Berhin59 sorry for the delay, I am working on a patch now that will dump all updates to a set of yaml files so that I can reproduce your issue locally. I'll post a patch up shortly I'd really appreciate if you could apply it locally and reproduce.

[agrare]

Hey @paulrichter2001 @Berhin59 I'm attaching two .tar.gz files depending on if you're using oparin or petrosian

oparin_debug_updates.tar.gz
petrosian_debug_updates.tar.gz

These are based on the https://github.com/agrare/manageiq-providers-vmware/tree/oparin_debug_updates and https://github.com/agrare/manageiq-providers-vmware/tree/petrosian_debug_updates branches

Before applying these changes please ensure that debug logging is enabled.

To apply these changes:

$ scp RELEASE_debug_updates.tar.gz root@APPLIANCE_IP:/tmp
$ ssh root@APPLIANCE_IP
# vmdb
# cd $(bundle show manageiq-providers-vmware)
# tar xfz /tmp/RELEASE_debug_updates.tar.gz
# systemctl restart evmserverd.service

Once this is restarted it should create a unique directory /tmp/ManageIQProvidersVmwareInfraManagerInventoryCollector-* where all of the updates will be saved as yml files. Once you see the issue re-appear if you can tar up the temp directory, export all of the logs for the worker (https://www.manageiq.org/docs/reference/latest/troubleshooting/index.html)

[agrare]

najdorf_debug_updates.tar.gz