改變 IB 卡為 ETH or IB 卡

一直記不起來所以要筆記

先去下載 NVIDIA 韌體工具 : NVIDIA Firmware Tools (MFT) 解開檔案之後安裝

[root@liho src]# tar zxvf mft-4.23.0-104-x86_64-rpm.tgz
mft-4.23.0-104-x86_64-rpm/
mft-4.23.0-104-x86_64-rpm/RPMS/
mft-4.23.0-104-x86_64-rpm/RPMS/mft-4.23.0-104.x86_64.rpm
mft-4.23.0-104-x86_64-rpm/RPMS/mft-oem-4.23.0-104.x86_64.rpm
mft-4.23.0-104-x86_64-rpm/RPMS/mft-pcap-4.23.0-104.x86_64.rpm
mft-4.23.0-104-x86_64-rpm/SRPMS/
mft-4.23.0-104-x86_64-rpm/SRPMS/kernel-mft-4.23.0-104.src.rpm
mft-4.23.0-104-x86_64-rpm/install.sh
mft-4.23.0-104-x86_64-rpm/uninstall.sh
mft-4.23.0-104-x86_64-rpm/old-mft-uninstall.sh
mft-4.23.0-104-x86_64-rpm/LICENSE.txt
[root@liho src]# cd mft-4.23.0-104-x86_64-rpm
[root@liho mft-4.23.0-104-x86_64-rpm]# ./install.sh
-I- Removing any old MFT file if exists...
-I- Building the MFT kernel binary RPM...
-I- Installing the MFT RPMs...
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:kernel-mft-4.23.0-4.18.0_425.3.1.################################# [100%]
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:mft-4.23.0-104                   ################################# [100%]
-I- In order to start mst, please run "mst start".

再執行 mst start 之前先確認好是否有安裝它依賴的套件

[root@liho ~]# dnf install gcc make kernel-devel python36 pciutils usbutils infiniband-diags -y

沒問題之後再執行 mst start

[root@liho ~]# mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success

確認沒問題後執行 mst status

[root@liho ~]# mst status
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4119_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:b0:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00
/dev/mst/mt4123_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:0e:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
                                   Chip revision is: 00

檢查 網卡 目前是什麼狀態

[root@liho ~]# mlxconfig -d /dev/mst/mt4123_pciconf0 query | grep LINK_TYPE_P1
         LINK_TYPE_P1                                ETH(2)

用下列指令改變目前的 ETH 為 IB

[root@liho ~]# mlxconfig -d /dev/mst/mt4123_pciconf0 set LINK_TYPE_P1=1

Device #1:
----------

Device type:    ConnectX6
Name:           MCX653105A-HDA_HPE_Ax
Description:    HPE InfiniBand HDR/Ethernet 200Gb 1-port MCX653105A-HDAT QSFP56 x16 Adapter
Device:         /dev/mst/mt4123_pciconf0

Configurations:                                      Next Boot       New
         LINK_TYPE_P1                                ETH(2)          IB(1)

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

反之，如果要改變 IB 卡為 ETH

[root@liho ~]# mlxconfig -d /dev/mst/mt4123_pciconf0 set LINK_TYPE_P1=2

Device #1:
----------

Device type:    ConnectX6
Name:           MCX653105A-HDA_HPE_Ax
Description:    HPE InfiniBand HDR/Ethernet 200Gb 1-port MCX653105A-HDAT QSFP56 x16 Adapter
Device:         /dev/mst/mt4123_pciconf0

Configurations:                                      Next Boot       New
         LINK_TYPE_P1                                IB(1)           ETH(2)

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

然後記得重新開機