Skip to content

Commit 1684096

Browse files
torvaldstorvalds
committed
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma
Pull rdma updates from Doug Ledford: "More exchaustive description of primary updates in this release: - Lots of driver fixes and misc fixes across the board. - I had to base on a net-next tree because the IPoIB Accelorator patches needed it. Unfortunately, it was known to Mellanox that there would need to be an IPoIB accelorator patch to the net tree (which left some functions turned off by an #ifdef construct to avoid warnings about defined but unused functions), then one to the RDMA tree, then a fixup that went back and re-enabled the functions in the net tree and enabled their use in the rdma tree Also, a sparse fix was sent to the net tree after I did my pull, and the fixup patch conflicts quite directly with that sparse fix, so I'm going to submit the fixup patch towards the end of the merge window by itself and based upon your master branch at the time. - Two separate rounds of hfi1 fixes, one that got dropped from last release because it came in just a day or two before the end of the merge window and then the one from this release cycle. Of note is that I now have a third series that just landed from Intel yesterday. It is not included in this pull request, but I may submit it by the end of the week. I'll talk to Intel about improving the timing of thier submissions for my workflow. - Changes to our idr usage in the RDMA subsystem that will tie into our cgroup management and also into the upcoming changes for the RDMA kernel<->userspace API. - Addition of support for a netdev to be tied to an RDMA device at the core level - Addition of the VNIC driver from Intel. While IPoIB provides IP over InfiniBand (and *only* IP, no lower layer protocol headers are allowed or supported), the VNIC driver presents a virtual Ethernet device with support for things like varying Ethertypes, VLANs, priorities and other features of Ethernet. The virtual devices are centrally managed by the OPA fabric manager, making this (for the time being) a strictly OPA specific feature. - Improvements to the On-Demand Paging support in the RDMA subsystem. - Addition of three significant OPA changes. While we added OPA support some time ago (via the hfi1 driver), the RDMA subsystem has so far glossed over the areas where OPA and InfiniBand differ. With this release we are starting to add support for the OPA extensions into the RDMA core in the following area: Extended port information for OPA is now supported, extended Address Handle attributes for OPA are now supported, and extended SA Queries to get OPA specific subnet information is now supported. Concise summary from the tag: - idr usage and locking changes - build fix for hns - ipoib debug path record file fix - hfi1 updates - core RDMA netdev addition - Intel VNIC driver addition - Enhanced accelerators for IPoIB addition - Debug cleanups in cxgb3/4 - Trivial cleanups from SF Markus Elfring - Misc rxe fixes from Mellanox - Misc ipoib fixes from Mellanox - Lots of mlx4/mlx5 changes from Mellanox - Misc fixes across the RDMA subsystem - ODP paging fixes and improvements - qedr updates - hfi1 updates - OPA port info patches - OPA AH patches - OPA SA Query patches" * tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma: (191 commits) infiniband: avoid dereferencing uninitialized dst on error path IB/SA: Add OPA addr header IB/mlx5: Add port_xmit_wait to counter registers read IB/ocrdma: fix out of bounds access to local buffer IB/mlx4: Fix incorrect order of formal and actual parameters IB/mlx4: Change flush logic so it adheres to the variable name mlx5: Fix mlx5_ib_map_mr_sg mr length IB/rxe: Don't clamp residual length to mtu IB/SA: Add support to query OPA path records IB/SA: Add OPA path record type IB/SA: Split struct sa_path_rec based on IB and ROCE specific fields IB/SA: Introduce path record specific types IB/SA: Rename ib_sa_path_rec to sa_path_rec IB/CM: Add braces when using sizeof IB/core: Define 'opa' rdma_ah_attr type IB/core: Define 'ib' and 'roce' rdma_ah_attr types IB/core: Use rdma_ah_attr accessor functions IB/core: Add accessor functions for rdma_ah_attr fields IB/PVRDMA: Rename ib_ah_attr related functions IB/mthca: Rename to_ib_ah_attr to to_rdma_ah_attr ...
2 parents 16a12fa + 24b43c9 commit 1684096

242 files changed

Lines changed: 14599 additions & 5505 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Documentation/infiniband/opa_vnic.txt‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
Intel Omni-Path (OPA) Virtual Network Interface Controller (VNIC) feature
2+
supports Ethernet functionality over Omni-Path fabric by encapsulating
3+
the Ethernet packets between HFI nodes.
4+
5+
Architecture
6+
=============
7+
The patterns of exchanges of Omni-Path encapsulated Ethernet packets
8+
involves one or more virtual Ethernet switches overlaid on the Omni-Path
9+
fabric topology. A subset of HFI nodes on the Omni-Path fabric are
10+
permitted to exchange encapsulated Ethernet packets across a particular
11+
virtual Ethernet switch. The virtual Ethernet switches are logical
12+
abstractions achieved by configuring the HFI nodes on the fabric for
13+
header generation and processing. In the simplest configuration all HFI
14+
nodes across the fabric exchange encapsulated Ethernet packets over a
15+
single virtual Ethernet switch. A virtual Ethernet switch, is effectively
16+
an independent Ethernet network. The configuration is performed by an
17+
Ethernet Manager (EM) which is part of the trusted Fabric Manager (FM)
18+
application. HFI nodes can have multiple VNICs each connected to a
19+
different virtual Ethernet switch. The below diagram presents a case
20+
of two virtual Ethernet switches with two HFI nodes.
21+
22+
+-------------------+
23+
| Subnet/ |
24+
| Ethernet |
25+
| Manager |
26+
+-------------------+
27+
/ /
28+
/ /
29+
/ /
30+
/ /
31+
+-----------------------------+ +------------------------------+
32+
| Virtual Ethernet Switch | | Virtual Ethernet Switch |
33+
| +---------+ +---------+ | | +---------+ +---------+ |
34+
| | VPORT | | VPORT | | | | VPORT | | VPORT | |
35+
+--+---------+----+---------+-+ +-+---------+----+---------+---+
36+
| \ / |
37+
| \ / |
38+
| \/ |
39+
| / \ |
40+
| / \ |
41+
+-----------+------------+ +-----------+------------+
42+
| VNIC | VNIC | | VNIC | VNIC |
43+
+-----------+------------+ +-----------+------------+
44+
| HFI | | HFI |
45+
+------------------------+ +------------------------+
46+
47+
48+
The Omni-Path encapsulated Ethernet packet format is as described below.
49+
50+
Bits Field
51+
------------------------------------
52+
Quad Word 0:
53+
0-19 SLID (lower 20 bits)
54+
20-30 Length (in Quad Words)
55+
31 BECN bit
56+
32-51 DLID (lower 20 bits)
57+
52-56 SC (Service Class)
58+
57-59 RC (Routing Control)
59+
60 FECN bit
60+
61-62 L2 (=10, 16B format)
61+
63 LT (=1, Link Transfer Head Flit)
62+
63+
Quad Word 1:
64+
0-7 L4 type (=0x78 ETHERNET)
65+
8-11 SLID[23:20]
66+
12-15 DLID[23:20]
67+
16-31 PKEY
68+
32-47 Entropy
69+
48-63 Reserved
70+
71+
Quad Word 2:
72+
0-15 Reserved
73+
16-31 L4 header
74+
32-63 Ethernet Packet
75+
76+
Quad Words 3 to N-1:
77+
0-63 Ethernet packet (pad extended)
78+
79+
Quad Word N (last):
80+
0-23 Ethernet packet (pad extended)
81+
24-55 ICRC
82+
56-61 Tail
83+
62-63 LT (=01, Link Transfer Tail Flit)
84+
85+
Ethernet packet is padded on the transmit side to ensure that the VNIC OPA
86+
packet is quad word aligned. The 'Tail' field contains the number of bytes
87+
padded. On the receive side the 'Tail' field is read and the padding is
88+
removed (along with ICRC, Tail and OPA header) before passing packet up
89+
the network stack.
90+
91+
The L4 header field contains the virtual Ethernet switch id the VNIC port
92+
belongs to. On the receive side, this field is used to de-multiplex the
93+
received VNIC packets to different VNIC ports.
94+
95+
Driver Design
96+
==============
97+
Intel OPA VNIC software design is presented in the below diagram.
98+
OPA VNIC functionality has a HW dependent component and a HW
99+
independent component.
100+
101+
The support has been added for IB device to allocate and free the RDMA
102+
netdev devices. The RDMA netdev supports interfacing with the network
103+
stack thus creating standard network interfaces. OPA_VNIC is an RDMA
104+
netdev device type.
105+
106+
The HW dependent VNIC functionality is part of the HFI1 driver. It
107+
implements the verbs to allocate and free the OPA_VNIC RDMA netdev.
108+
It involves HW resource allocation/management for VNIC functionality.
109+
It interfaces with the network stack and implements the required
110+
net_device_ops functions. It expects Omni-Path encapsulated Ethernet
111+
packets in the transmit path and provides HW access to them. It strips
112+
the Omni-Path header from the received packets before passing them up
113+
the network stack. It also implements the RDMA netdev control operations.
114+
115+
The OPA VNIC module implements the HW independent VNIC functionality.
116+
It consists of two parts. The VNIC Ethernet Management Agent (VEMA)
117+
registers itself with IB core as an IB client and interfaces with the
118+
IB MAD stack. It exchanges the management information with the Ethernet
119+
Manager (EM) and the VNIC netdev. The VNIC netdev part allocates and frees
120+
the OPA_VNIC RDMA netdev devices. It overrides the net_device_ops functions
121+
set by HW dependent VNIC driver where required to accommodate any control
122+
operation. It also handles the encapsulation of Ethernet packets with an
123+
Omni-Path header in the transmit path. For each VNIC interface, the
124+
information required for encapsulation is configured by the EM via VEMA MAD
125+
interface. It also passes any control information to the HW dependent driver
126+
by invoking the RDMA netdev control operations.
127+
128+
+-------------------+ +----------------------+
129+
| | | Linux |
130+
| IB MAD | | Network |
131+
| | | Stack |
132+
+-------------------+ +----------------------+
133+
| | |
134+
| | |
135+
+----------------------------+ |
136+
| | |
137+
| OPA VNIC Module | |
138+
| (OPA VNIC RDMA Netdev | |
139+
| & EMA functions) | |
140+
| | |
141+
+----------------------------+ |
142+
| |
143+
| |
144+
+------------------+ |
145+
| IB core | |
146+
+------------------+ |
147+
| |
148+
| |
149+
+--------------------------------------------+
150+
| |
151+
| HFI1 Driver with VNIC support |
152+
| |
153+
+--------------------------------------------+

‎MAINTAINERS‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5911,6 +5911,13 @@ F: drivers/block/cciss*
59115911
F: include/linux/cciss_ioctl.h
59125912
F: include/uapi/linux/cciss_ioctl.h
59135913

5914+
OPA-VNIC DRIVER
5915+
M: Dennis Dalessandro <dennis.dalessandro@intel.com>
5916+
M: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
5917+
L: linux-rdma@vger.kernel.org
5918+
S: Supported
5919+
F: drivers/infiniband/ulp/opa_vnic
5920+
59145921
HFI1 DRIVER
59155922
M: Mike Marciniszyn <mike.marciniszyn@intel.com>
59165923
M: Dennis Dalessandro <dennis.dalessandro@intel.com>
@@ -6519,6 +6526,7 @@ W: http://www.openfabrics.org/
65196526
Q: http://patchwork.kernel.org/project/linux-rdma/list/
65206527
T: git git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git
65216528
S: Supported
6529+
F: Documentation/devicetree/bindings/infiniband/
65226530
F: Documentation/infiniband/
65236531
F: drivers/infiniband/
65246532
F: include/uapi/linux/if_infiniband.h
@@ -11489,11 +11497,11 @@ S: Supported
1148911497
F: drivers/net/ethernet/emulex/benet/
1149011498

1149111499
EMULEX ONECONNECT ROCE DRIVER
11492-
M: Selvin Xavier <selvin.xavier@avagotech.com>
11493-
M: Devesh Sharma <devesh.sharma@avagotech.com>
11500+
M: Selvin Xavier <selvin.xavier@broadcom.com>
11501+
M: Devesh Sharma <devesh.sharma@broadcom.com>
1149411502
L: linux-rdma@vger.kernel.org
11495-
W: http://www.emulex.com
11496-
S: Supported
11503+
W: http://www.broadcom.com
11504+
S: Odd Fixes
1149711505
F: drivers/infiniband/hw/ocrdma/
1149811506
F: include/uapi/rdma/ocrdma-abi.h
1149911507

‎drivers/infiniband/Kconfig‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ source "drivers/infiniband/ulp/srpt/Kconfig"
8585
source "drivers/infiniband/ulp/iser/Kconfig"
8686
source "drivers/infiniband/ulp/isert/Kconfig"
8787

88+
source "drivers/infiniband/ulp/opa_vnic/Kconfig"
8889
source "drivers/infiniband/sw/rdmavt/Kconfig"
8990
source "drivers/infiniband/sw/rxe/Kconfig"
9091

‎drivers/infiniband/core/Makefile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,5 @@ ib_umad-y := user_mad.o
2929

3030
ib_ucm-y := ucm.o
3131

32-
ib_uverbs-y := uverbs_main.o uverbs_cmd.o uverbs_marshall.o
32+
ib_uverbs-y := uverbs_main.o uverbs_cmd.o uverbs_marshall.o \
33+
rdma_core.o uverbs_std_types.o

‎drivers/infiniband/core/addr.c‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -444,9 +444,9 @@ static int addr6_resolve(struct sockaddr_in6 *src_in,
444444
fl6.saddr = src_in->sin6_addr;
445445
fl6.flowi6_oif = addr->bound_dev_if;
446446

447-
dst = ip6_route_output(addr->net, NULL, &fl6);
448-
if ((ret = dst->error))
449-
goto put;
447+
ret = ipv6_stub->ipv6_dst_lookup(addr->net, NULL, &dst, &fl6);
448+
if (ret < 0)
449+
return ret;
450450

451451
rt = (struct rt6_info *)dst;
452452
if (ipv6_addr_any(&fl6.saddr)) {

‎drivers/infiniband/core/agent.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,13 +137,13 @@ void agent_send_response(const struct ib_mad_hdr *mad_hdr, const struct ib_grh *
137137
err2:
138138
ib_free_send_mad(send_buf);
139139
err1:
140-
ib_destroy_ah(ah);
140+
rdma_destroy_ah(ah);
141141
}
142142

143143
static void agent_send_handler(struct ib_mad_agent *mad_agent,
144144
struct ib_mad_send_wc *mad_send_wc)
145145
{
146-
ib_destroy_ah(mad_send_wc->send_buf->ah);
146+
rdma_destroy_ah(mad_send_wc->send_buf->ah);
147147
ib_free_send_mad(mad_send_wc->send_buf);
148148
}
149149

0 commit comments

Comments
 (0)