Re: [PLUG] Two node Infiniband connection hardware

On Thu, Sep 17, 2015 at 10:34 AM, K.S. Bhaskar <bhaskar@bhaskars.com> wrote:

Thanks for the comments, Keith and Brent. If we build this, I will certainly post our experience with it.

Regards
-- Bhaskar

On Tue, Sep 15, 2015 at 11:59 AM, Keith C. Perry <kperry@daotechnologies.com> wrote:
Just some initial findings...

There are loopback QSFP heads...
http://www.molex.com/molex/products/listview.jsp?channel=Products&fs=productseries:or(74763||106005)&sType=z

Which means there is a valid construction for a crossover cable. You just wire "half" of the connections to another connector instead of to the same connect, similar to an ethernet or T1 signaling crossover cable. For example, here is one of the loopback wirings for one of the molex connectors above.
http://www.molex.com/pdm_docs/sd/747630015_sd.pdf

This is an interesting idea so I'd like to know how you make out with it. I initially used to build backup networks this way but even at 100Mbs the performance wasn't consistent since not all NICs are created equal and without the switch to do proper bandwidth management, packet collisions fired the back off algorithms too much. You can play with that in software but even back then, small 100Mbs switches were a relative cheap and easy fix thus became because my protocol to dedicate a switch to the backup network (and these days that works beautifully for critical DRBD deployments).

IB is a different animal- it is built as a switching fabric so in my mind there should be mechanism present to allow without a dedicated switch. I seem to remember that that was a primary motivation and use for it.

IB cards appear to be cheaper than 10Gbs ethernet cards too.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Keith C. Perry, MS E.E.
Owner, DAO Technologies LLC
(O) +1.215.525.4165 x2033
(M) +1.215.432.5167
www.daotechnologies.com

From: "K.S. Bhaskar" <bhaskar@bhaskars.com>
To: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
Sent: Tuesday, September 15, 2015 10:47:19 AM
Subject: [PLUG] Two node Infiniband connection hardware

Is it possible to make a two node Infiniband connection with Infiniband cards in the two servers and a cable between them, or is a switch always required? There are hints (e.g., https://docs.oracle.com/cd/E19787-01/819-2993/bhcfghcc/index.html and http://www.bussink.ch/?p=1183 suggesting it is possible), but I can't find anything definitive. Vendors of course want one to purchase a switch.

Thank you very much.

Regards
-- Bhaskar

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug