Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for 1G SFP copper in 40:10G SFP+ #2

Open
bluecmd opened this issue Oct 6, 2018 · 9 comments
Open

Support for 1G SFP copper in 40:10G SFP+ #2

bluecmd opened this issue Oct 6, 2018 · 9 comments

Comments

@bluecmd
Copy link

bluecmd commented Oct 6, 2018 

dhtech@wedge1:~$ sudo wedge_qsfp_util 15
Port 15
  ID: 0x0d
  Status: 0x00 0x00
  Interrupt Flags:
    LOS: 0x00
    Fault: 0x00
    Temp: 0x00
    Vcc: 0x00
    Rx Power: 0x00 0x00
    Tx Power: 0x00 0x00
    Tx Bias: 0x00 0x00
    Reserved Set 4: 0x00 0x00
    Reserved Set 5: 0x00 0x00
    Vendor Defined: 0x00 0x00 0x07
  Temperature: 0.000000 C
  Supply Voltage: 0.000000 V
  Channel Data:      RX Power        TX Power         TX Bias
    Channel 1:       0.000000mW      0.000000mW      0.000000mA
    Channel 2:       0.000000mW      0.000000mW      0.000000mA
    Channel 3:       0.000000mW      0.000000mW      0.000000mA
    Channel 4:       0.000000mW      0.000000mW      0.000000mA
    Power measurement is unsupported
    Reported RX Power is OMA
  Power set:  0x00      Extended ID:  0x9b      Ethernet Compliance:  0x00
  TX disable bits: 0x00
  Rate select is unsupported
  RX rate select bits: 0x00
  TX rate select bits: 0x00
  CDR support:  TX: supported   RX: unsupported
  CDR bits: 0x00
  Connector: 0x80
  Spec compliance: 0x00 0x00 0x00 0x000x00 0x00 0x00 0x00
  Encoding: 0x00
  Nominal Bit Rate: 0 MBps
  Ext rate select compliance: 0x00
  Length (SMF): 0 km
  Length (OM3): 0 m
  Length (OM2): 0 m
  Length (OM1): 0 m
  Length (Copper): 0 m
  Length (Copper dM): 12.7 m
  Length (Copper effective): 12.7 m
  Device Tech: 0x00
  Ext Module: 0x00
  Wavelength tolerance: 0x00 0x00
  Max case temp: 70C
  CC_BASE: 0x42
  Options: 0x00 0x00 0x00 0x00
  DOM Type: 0x00
  Enhanced Options: 0x00
  Reserved: 0xf2
  CC_EXT: 0xdd
  Vendor Specific:
    f2 00 07 e1 ca 3b 89 96  ce 6d 0a 26 7f ce 76 8b
    9f 31 aa 00 00 00 00 00  00 00 00 00 a7 68 73 1d
  Vendor: CISCO-DNI
  Vendor OUI: 00:00:00
  Vendor PN: CAZADERO-R
  Vendor Rev: 03
  Vendor SN: DTY212502JP
  Date Code: 20170703
@bluecmd
Copy link
Author

bluecmd commented Oct 6, 2018

With 10G SFP+ twinax connected to 40->10G converter:

Port 15
  ID: 0x0d
  Status: 0x00 0x00
  Interrupt Flags:
    LOS: 0x00
    Fault: 0x00
    Temp: 0x00
    Vcc: 0x00
    Rx Power: 0x00 0x00
    Tx Power: 0x00 0x00
    Tx Bias: 0x00 0x00
    Reserved Set 4: 0x00 0x00
    Reserved Set 5: 0x00 0x00
    Vendor Defined: 0x00 0x00 0x02
  Temperature: 0.000000 C
  Supply Voltage: 0.000000 V
  Channel Data:      RX Power        TX Power         TX Bias
    Channel 1:       0.000000mW      0.000000mW      0.000000mA
    Channel 2:       0.000000mW      0.000000mW      0.000000mA
    Channel 3:       0.000000mW      0.000000mW      0.000000mA
    Channel 4:       0.000000mW      0.000000mW      0.000000mA
    Power measurement is unsupported
    Reported RX Power is OMA
  Power set:  0x00      Extended ID:  0x9b      Ethernet Compliance:  0x00
  TX disable bits: 0x00
  Rate select is unsupported
  RX rate select bits: 0x00
  TX rate select bits: 0x00
  CDR support:  TX: supported   RX: unsupported
  CDR bits: 0x00
  Connector: 0x80
  Spec compliance: 0x00 0x00 0x00 0x000x00 0x00 0x00 0x00
  Encoding: 0x00
  Nominal Bit Rate: 0 MBps
  Ext rate select compliance: 0x00
  Length (SMF): 0 km
  Length (OM3): 0 m
  Length (OM2): 0 m
  Length (OM1): 0 m
  Length (Copper): 0 m
  Length (Copper dM): 12.7 m
  Length (Copper effective): 12.7 m
  Device Tech: 0x00
  Ext Module: 0x00
  Wavelength tolerance: 0x00 0x00
  Max case temp: 70C
  CC_BASE: 0x42
  Options: 0x00 0x00 0x00 0x00
  DOM Type: 0x00
  Enhanced Options: 0x00
  Reserved: 0xf2
  CC_EXT: 0xdd
  Vendor Specific:
    f2 00 07 e1 ca 3b 89 96  ce 6d 0a 26 7f ce 76 8b
    9f 31 aa 00 00 00 00 00  00 00 00 00 a7 68 73 1d
  Vendor: CISCO-DNI
  Vendor OUI: 00:00:00
  Vendor PN: CAZADERO-R
  Vendor Rev: 03
  Vendor SN: DTY212502JP
  Date Code: 20170703

Works when connected to Cisco 2960S.

@bluecmd
Copy link
Author

bluecmd commented Oct 6, 2018

Config with

     {
            "ingressVlan": 552,
            "logicalID": 64,
            "maxFrameSize": 1500,
            "minFrameSize": 64,
            "parserType": 0,
            "routable": false,
            "speed": 1000,
            "state": 2
        }

Wedge output:

dhtech@wedge1:/usr/local/bin$ sudo /usr/local/bin/wedge_agent -mode=wedge -mgmt_if=ma1 -config=/etc/wedge.json --alsologtostderr -v=1000 -vmodule=EventBase=0,AsyncSocket=0
E1006 20:07:25.431332  3095 WedgeProductInfo.cpp:136] json parse error on line 0: expected json value
E1006 20:07:25.431597  3095 WedgeProductInfo.cpp:67] json parse error on line 0: expected json value
I1006 20:07:25.434479  3095 FunctionScheduler.cpp:396] Starting FunctionScheduler with 0 functions.
I1006 20:07:25.435096  3098 FunctionScheduler.cpp:505] Now running 1
I1006 20:07:25.435415  3095 ThriftServer.cpp:264] libevent 2.0.21-stable method epoll
I1006 20:07:25.454430  3095 ThriftServer.cpp:395] Using 0 SSL handshake threads
DMA pool size: 16777216
PCI unit 0: Dev 0xb850, Rev 0x03, Chip BCM56850_A2, Driver BCM56850_A0

Initializing platform
Device Configuration - SUCCESS!
SOC unit 0 attached to PCI device BCM56850_A2
Boot flags: Cold boot
rc: unit 0 device BCM56850_A2
open /dev/linux-bcm-knet: : No such file or directory
rc: MMU initialized
rc: L2 Table shadowing enabled
rc: Port modes initialized
Common SDK init completed
I1006 20:07:39.323992  3116 BcmSwitch.cpp:562]  Performing cold boot
I1006 20:07:39.797052  3132 Cpp2Channel.cpp:160] Got a read error: N6apache6thrift9transport19TTransportExceptionE: AsyncSocketException: connect failed, type = Socket not open, errno = 111 (Connection refused): Connection refused
I1006 20:07:39.797111  3132 Cpp2Channel.cpp:198] Got an EOF on channel
I1006 20:07:39.842178  3133 Cpp2Channel.cpp:198] Got an EOF on channel
I1006 20:07:40.721724  3133 Cpp2Channel.cpp:198] Got an EOF on channel
E1006 20:07:46.623818  3130 SwSwitch.cpp:379] Unable to write switch state JSON to file
C1006 20:07:46.624393  3130 SwSwitch.cpp:954] error applying state change to hardware: N8facebook5fboss10FbossErrorE: Unsupported speed (1000) or transmitter technology (2) setting on port 57
*** Aborted at 1538856466 (unix time) try "date -d @1538856466" if you are using GNU date ***
PC: @     0x7f89c8942fff gsignal
*** SIGABRT (@0xc17) received by PID 3095 (TID 0x7f8967463700) from PID 3095; stack trace: ***
    @     0x7f89cf3820c0 (unknown)
    @     0x7f89c8942fff gsignal
    @     0x7f89c894442a abort
    @     0x55b4deabf33d folly::LogCategory::admitMessage()
    @     0x55b4dea8ece8 folly::LogStreamProcessor::logNow()
    @     0x55b4dea8ee0e folly::LogStreamVoidify<>::operator&()
    @     0x55b4ddfce02b facebook::fboss::SwSwitch::applyUpdate()
    @     0x55b4ddfcc278 facebook::fboss::SwSwitch::handlePendingUpdates()
    @     0x55b4ddfcb9dc facebook::fboss::SwSwitch::handlePendingUpdatesHelper()
    @     0x55b4ddfe22a8 _ZZN5folly9EventBase20runInEventBaseThreadIN8facebook5fboss8SwSwitchEEEbPFvPT_ES6_ENKUlvE_clEv
    @     0x55b4ddfef270 _ZN5folly6detail8function14FunctionTraitsIFvvEE9callSmallIZNS_9EventBase20runInEventBaseThreadIN8facebook5fboss8SwSwitchEEEbPFvPT_ESC_EUlvE_EEvRNS1_4DataE
    @     0x55b4dde44d55 folly::detail::function::FunctionTraits<>::operator()()
    @     0x55b4dea52192 folly::EventBase::FunctionRunner::messageAvailable()
    @     0x55b4dea54e7b folly::NotificationQueue<>::Consumer::consumeMessages()
    @     0x55b4dea53aa6 folly::NotificationQueue<>::Consumer::handlerReady()
    @     0x55b4dea5b4b5 folly::EventHandler::libeventCallback()
    @     0x7f89ce5e35a0 event_base_loop
    @     0x55b4dea4e5ce folly::EventBase::loopBody()
    @     0x55b4dea4e078 folly::EventBase::loop()
    @     0x55b4dea4f637 folly::EventBase::loopForever()
    @     0x55b4ddfd0f6c facebook::fboss::SwSwitch::threadLoop()
    @     0x55b4ddfd069e _ZZN8facebook5fboss8SwSwitch12startThreadsEvENKUlvE0_clEv
    @     0x55b4ddfda60e _ZNSt12_Bind_simpleIFZN8facebook5fboss8SwSwitch12startThreadsEvEUlvE0_vEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE
    @     0x55b4ddfda40f _ZNSt12_Bind_simpleIFZN8facebook5fboss8SwSwitch12startThreadsEvEUlvE0_vEEclEv
    @     0x55b4ddfda33e _ZNSt6thread11_State_implISt12_Bind_simpleIFZN8facebook5fboss8SwSwitch12startThreadsEvEUlvE0_vEEE6_M_runEv
    @     0x7f89c9283e6f (unknown)
    @     0x7f89cf378494 start_thread
    @     0x7f89c89f8acf clone
    @                0x0 (unknown)
Aborted

@bluecmd
Copy link
Author

bluecmd commented Oct 6, 2018

Port appears as 61 with module in the top-right slot. It reports as technology UNKNOWN.

V1006 21:05:15.855491  3869 BcmPort.cpp:361] Reprogramming BcmPort for port 61
V1006 21:05:15.855968  3869 BcmPort.cpp:425] Getting desired interface mode for port 61 (speed=1000, tech=0). RESULT=3
V1006 21:05:15.856180  3869 BcmPort.cpp:518] Finalizing BcmPort::setSpeed() by calling port_speed_set on port 61 ()
V1006 21:05:15.954299  3869 BcmPort.cpp:361] Reprogramming BcmPort for port 62
V1006 21:05:15.954775  3869 BcmPort.cpp:425] Getting desired interface mode for port 62 (speed=1000, tech=0). RESULT=3
V1006 21:05:15.954957  3869 BcmPort.cpp:518] Finalizing BcmPort::setSpeed() by calling port_speed_set on port 62 ()
I1006 21:05:16.045393  3866 SwSwitch.cpp:1129] Link state changed: 61->UP
V1006 21:05:16.045564  3866 SwSwitch.cpp:39] LinkState: Port 61 Up
V1006 21:05:16.049172  3869 BcmPort.cpp:361] Reprogramming BcmPort for port 63
V1006 21:05:16.055121  3869 BcmPort.cpp:425] Getting desired interface mode for port 63 (speed=1000, tech=0). RESULT=3
V1006 21:05:16.055312  3869 BcmPort.cpp:518] Finalizing BcmPort::setSpeed() by calling port_speed_set on port 63 ()
V1006 21:05:16.153724  3869 BcmPort.cpp:361] Reprogramming BcmPort for port 64
V1006 21:05:16.154205  3869 BcmPort.cpp:425] Getting desired interface mode for port 64 (speed=1000, tech=0). RESULT=3
V1006 21:05:16.154389  3869 BcmPort.cpp:518] Finalizing BcmPort::setSpeed() by calling port_speed_set on port 64 ()
V1006 21:05:16.254055  3869 SwSwitch.cpp:970] Update state took 14915398us
V1006 21:05:16.254055  3872 QsfpCache.cpp:95] Will sync port 1
V1006 21:05:16.254156  3872 QsfpCache.cpp:95] Will sync port 2
V1006 21:05:16.254210  3872 QsfpCache.cpp:95] Will sync port 3
V1006 21:05:16.254263  3872 QsfpCache.cpp:95] Will sync port 4
V1006 21:05:16.254316  3872 QsfpCache.cpp:95] Will sync port 5
V1006 21:05:16.254368  3872 QsfpCache.cpp:95] Will sync port 6

@bluecmd
Copy link
Author

bluecmd commented Oct 6, 2018

QSFP service says:

I1006 21:13:46.556105  3774 QsfpModule.cpp:540]  Unable to get transmitter technology for QSFP 12: Qsfp is either not present or the data is not read
V1006 21:13:46.556167  3774 QsfpModule.cpp:919] Customization not supported on 12
I1006 21:13:46.578995  3774 BaseWedgeI2CBus.cpp:26] successfully opened wedge CP2112 I2C bus
I1006 21:13:46.579020  3774 BaseWedgeI2CBus.cpp:93] selecting QSFP 14
I1006 21:13:46.585104  3774 QsfpModule.cpp:540]  Unable to get transmitter technology for QSFP 13: Qsfp is either not present or the data is not read
V1006 21:13:46.585172  3774 QsfpModule.cpp:919] Customization not supported on 13
I1006 21:13:46.605996  3774 BaseWedgeI2CBus.cpp:26] successfully opened wedge CP2112 I2C bus
I1006 21:13:46.606027  3774 BaseWedgeI2CBus.cpp:93] selecting QSFP 15
V1006 21:13:46.618068  3774 QsfpModule.cpp:1085] Port 14: Power control POWER_SET_BY_HW Ext ID 9b Ethernet compliance 0 Desired power control HIGH_POWER_OVERRIDE
I1006 21:13:46.640997  3774 BaseWedgeI2CBus.cpp:26] successfully opened wedge CP2112 I2C bus
I1006 21:13:46.641024  3774 BaseWedgeI2CBus.cpp:93] selecting QSFP 15
I1006 21:13:46.647038  3774 QsfpModule.cpp:1108] Port 14: QSFP set to power setting HIGH_POWER_OVERRIDE (5)
V1006 21:13:46.647104  3774 QsfpModule.cpp:848] Performing partial qsfp data cache refresh for transceiver 14
I1006 21:13:46.668000  3774 BaseWedgeI2CBus.cpp:26] successfully opened wedge CP2112 I2C bus
I1006 21:13:46.668079  3774 BaseWedgeI2CBus.cpp:93] selecting QSFP 15
I1006 21:13:46.695051  3774 QsfpModule.cpp:572] Detected QSFP 14, flatMem=0
I1006 21:13:46.717998  3774 BaseWedgeI2CBus.cpp:26] successfully opened wedge CP2112 I2C bus
I1006 21:13:46.718025  3774 BaseWedgeI2CBus.cpp:93] selecting QSFP 16
I1006 21:13:51.193442  3774 FunctionScheduler.cpp:505] Now running refreshTransceivers
I1006 21:13:51.214000  3774 BaseWedgeI2CBus.cpp:26] successfully opened wedge CP2112 I2C bus
I1006 21:13:51.214037  3774 BaseWedgeI2CBus.cpp:93] selecting QSFP 1
I1006 21:13:51.241034  3774 BaseWedgeI2CBus.cpp:26] successfully opened wedge CP2112 I2C bus

@bluecmd
Copy link
Author

bluecmd commented Oct 6, 2018

Changing to various transmitter technologies (GMII, SGMII) did not seem to change anything. Interface on Cisco side is down.

Next step would be to implement stats logging to see if any packets are registered as egressing. That would provide some data to where to look at.

@bluecmd
Copy link
Author

bluecmd commented Oct 7, 2018

The transmitter technology might be the wrong place to go about this - it seems like the speed thing is only half the story. See: facebook#61 (comment)

We might want to try using a newer OpenNSL, or trying to see if we can set OPENNSL_CONFIG_FILE to https://github.com/Broadcom-Switch/OpenNSL/blob/master/bin/wedge/config.wedge40 as according to http://broadcom-switch.github.io/OpenNSL/doc/html/OPENNSL_CUSTOMIZING_OPENNSL.html should work. But then again, we're talking about a config from 3.5 using with the 6.4 binaries so this might be hard.

@bluecmd
Copy link
Author

bluecmd commented Oct 7, 2018

OPENNSL_CONFIG_FILE does not show up in the 6.4 version, so we probably need to figure out how to make FBOSS run with 3.5. That's going to be interesting.

@bluecmd
Copy link
Author

bluecmd commented Oct 8, 2018

The documentation of the Trident II datasheet says:

128 Integrated SerDes (XFI, KR, SFI, XLAUI, HiGig)

None of those SerDes formats are the ones we want here. But it also mentions:

Standards-compliant 10GbE/40GbE switch
with support for up to 32 ports of 40GbE or up
to 104 logical ports of 1GbE

The Wedge OCP documentation says:

The Broadcom SerDes Warpcore is the versatile physical layer interface for the BCM56850, specifically
designed to support up to 42Gbps. Each Warpcore consists of four SerDes lanes. Each lane can operate
from 1.25Gbps to 10.9375Gbps. We can configure each Warpcore to operate up to a single 40Gbe link.
There are 32 SerDes Warpcores available in BCM 56850. That provides a total of 128 links. Half the ports
(64 ports) go to 16 QSFP front ports and remaining 64 are not used in the Wedge system.

@bluecmd
Copy link
Author

bluecmd commented Oct 8, 2018

I asked for advise in https://github.com/Broadcom-Switch/OpenNSL/issues/37

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bluecmd