BIER and In Network Computing

Introduction In Collective Communication, a communicator defines a specific group of processes/GPUs participating in a communication pattern, essentially acting as a communication channel or context where all members must act together (collectively) to perform operations like broadcasting, gathering, or reducing data, ensuring synchronized data movement among them. Each communicator can be mapped to an IP multicast group with a set of trees, each of which rooted at a participant, used for broadcasting in the forward direction and in-network computing/reduction in the reverse direction. The trees can be set up by the PIM protocol or other means. However, tree setup/maintenance has long been considered a burden and scaling/convergence limit, and BIER removes the trees while still providing efficient replication. On the other hand, the pre-established tree is important for INC. It allows a transit node to send its reduction result upstream once it has received all downstream reduction results instead of relying on a timer. A theoretical stateless reduction not using the tree relies on a timer at each hop (because it does not know if all downstream results have been received) introduces a fixed delay at every hop. Nonetheless, BIER can still be integrated with INC and provide additional INC-specific benefits, as described below. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Mode of Operation Essentially, the forward direction broadcasting does not need a pre-established explicit tree. BIER forwarding is used for that. The tree for INC purposes is established data-driven when the first packet traverses the network. We first assume that each GPU is an edge BFR and allocated with a BFR-ID.

Forward Direction Packets associated with INC carry the Communicator ID in the entropy field of the BIER header . The (BFIR-ID, Entropy) tuple from the packet uniquely identifies a (source, group) tree (INC-tree). The BIFT-ID field indicates that additional INC-related handling is needed, besides the regular BIER forwarding. When an incoming BIER packet matches the INC-relevant BIFT (INC-BIFT), the (BFIR-ID, Entropy) tuple is used to look up the corresponding INC-tree. If it is found, packets are forwarded accordingly, with necessary processing related to INC. Notice that this is different from the bit-by-bit lookup with regular BIER forwarding. If the tree is not found, it is created instantly on the node with the following information: upstream node and incoming bitstring downstream BFRs with their respective BFRs INC-specific information The downstream information is built from the bit-by-bit lookup for the regular BIER forwarding. The incoming bitstring is not expected to change in the life of the communicator barring routing changes. For simplicity, considerations for routing changes are deferred to .

Reduction Direction Packets are also encoded in BIER. The BFIR-ID and Entropy field are set according to the corresponding forward direction packets. In particular, the BFIR-ID identifies the destination and not the sender of the packet. The initial bitstring only identifies the sender itself. Another INC-BIFT ID is needed to indicate that this is for INC in the reverse direction. The corresponding INC-tree is identified from the (BFIR-ID, Entropy) tuple, and the INC processing kicks in. The INC state includes a bitstring that identifies all the BFERs from which the reduction packet has been received. It is the logical OR of the bitstring from all the corresponding reduction packets. The reduction result is sent to the upstream node when the bitstring matches the incoming bitstring saved in the tree state, or when a timer fires. The bitstring is encoded into the BIER header. The BFIR can determine which BFER has not sent back the reduction result based on the received bitstring - this is an additional benefit provided by BIER.

Routing Changes Naturally, routing changes happen because of link/node failure/recovery and congestion situations (when adaptive routing is used). When this happens, the INC tree state also changes. How that impacts the reduction will be discussed in a future revision, but it may not be much different from the existing tree-based multicast case.

Leaf Switches as Edge BFRs The above discussions assume that GPUs are edge BFRs. In a scaled setup, it may be that the edge BFRs are leaf switches, not GPUs. In this case, the leaf switch for the source of the reduction packets needs to identify the BFIR-ID.

Security Considerations To be added.

IANA Considerations No IANA action is needed.

Acknowledgments The authors thank Duncan Roweth and David Hewson for their collaboration on this topic.