Home / server / Questions / 383545
Accepted
ajdecon
Asked: 2012-04-27 12:32:37 +0800 CST

View Infiniband routing table generated by OpenSM?

  • 772

As I understand it, the subnet manager of an Infiniband network calculates the best routes between each pair of nodes on the network and provides these routes to the nodes when they want to communicate. Is there any way to get the subnet manager (and in particular, OpenSM) to print out these routing tables in a human-readable fashion?

At the moment this is purely for my own curiosity, but I can imagine cases where this would be useful for debugging.

routing hpc infiniband

2 Answers

  1. Best Answer

    In OpenFabrics, the infiniband-diags package comes with a tool called "ibroute" that should do exactly what you're asking for. There is a script called "dump_lfts.sh" in infiniband-diags that wraps around ibroute and will output the routing tables for all the switches in your fabric.

    There are other scripts out there to wrap around these for further debugging/routing analysis, but I'll leave that for another answer.

    Update:

    A colleague reminded me there is a mechanism in OpenSM to do this too. I haven't personally used it, but for completeness, here's the relevant portion from the OpenSM manpage:

    The ability to dump switch lid matrices (aka min hops tables) to file and later to load these is also supported.

    The usage is similar to unicast forwarding tables loading from a lfts file (introduced by 'file' routing engine), but new lid matrix file name should be specified by -M or --lid_matrix_file option. For example:

    opensm -R file -M ./opensm-lid-matrix.dump

    The dump file is named opensm-lid-matrix.dump and will be generated in standard opensm dump directory (/var/log by default) when OSM_LOG_ROUTING logging flag is set.

    When routing engine 'file' is activated, but the lfts file is not specified or not cannot be open default lid matrix algorithm will be used.

    There is also a switch forwarding tables dumper which generates a file compatible with dump_lfts.sh output. This file can be used as input for forwarding tables loading by 'file' routing engine. Both or one of options -U and -M can be specified together with -R file.

    • 3

  2. The command

    ibdiagnet --routing

    will create the file /var/tmp/ibdiagnet2/ibdiagnet2.fdbs, which contains the unicast routing tables of all switches and the file /var/tmp/ibdiagnet2/ibdiagnet2.mcfdbs, which contains the multicast routing tables of all switches. To facilitate the interpretation one should remove all lines ending in UNREACHABLE.

    • 1