Internals

While the following information is not crucial to using FM 2.1, some users may be interested in knowing some of the implementation-specific subtleties. Because the FM implementation will definitely change in future releases, programmers are strongly discouraged from writing programs that rely on this information.

The Myrinet and sockets implementations of FM 2.1 have radically different implementations. While both implementations conform to the FM interface, they do not necessarily behave similarly in a given situation and may break code that violates FM semantics but "seems to work" on the other implementation. The primary difference between the two implementations of FM stems from the different ways in which messages are streamed over the network. In the Myrinet version of FM, message streams are broken into small packets. A packet is injected into the network as soon as it becomes full or when FM_end_message() is called. Although the packets from one message stream arrive to their destination in order, packets from two or more message streams can arrive interleaved to a shared destination. To avoid copying and buffering, Myrinet FM processes packets (i.e. calls the associated handlers) in the order they arrive, even if this implies context switching from one handler to another in the middle of their execution.

In contrast, in the sockets version of FM, message streams are sent across the network along a dedicated socket and are therefore never interleaved with other message streams. This implies that once a handler is invoked, it runs to completion.

The implementation differences between Myrinet FM and Sockets FM have a number of ramifications on program execution:

1. In Myrinet FM, a handler may be called multiple times per message stream, even though it gives the illusion of running only once. When a handler tries to FM_receive() data that hasn't yet arrived, it saves its state and returns control to FM_extract(). When FM_extract() extracts the data that the handler was waiting for, the handler continues from where it left off, and the newly-arrived data is processed. Note that the only place a handler can yield control is in FM_receive(). In Sockets FM, handlers always run to completion.
2. In Myrinet FM, an FM_extract(i) call to extract up to i bytes from the network will extract at most a small constant number of bytes more than i. This is due to the fact that message streams are packetized, and packets are always extracted in their entirety. After each packet that FM_extract(i) processes, it checks to see if it has received a total of more than i bytes and, if so, returns control to the caller. In Sockets FM, an FM_extract(i) call can extract arbitrarily more than i bytes, bounded only by the largest message size used in a program. This is due to the fact that Sockets FM does not packetize message streams and extracts messages in their entirety. After each message that FM_extract(i) processes, it checks to see if it has received a total of more than i bytes and, if so, returns control to the caller.
3. In Myrinet FM, there are only two situations in which FM_begin_message() can fail (i.e. return NULL). The first is if another thread is already in the process of sending a message and, therefore, has the send queue locked. The second is if FM_begin_message() is called from within a handler and tries to begin a message for which insufficient network resources are immediately available. (This is used for deadlock avoidance.) Note that if a handler tries to send a particularly large message, there may never be sufficient resources to send it, and any attempt will return NULL. In Sockets FM, FM_begin_message() always fails when called from within a handler.
4. In Myrinet FM, the data specified by an FM_send_piece() may--but need not necessarily--be sent over the network before the FM_end_message(), depending on the length of the message stream and the proximity of the piece to the end. In Sockets FM, individual pieces of a message specified in FM_send_piece() are never sent until the FM_end_message() call is invoked. One implication of these semantics is that video (or other continuous-media) servers should send their data as multiple message streams, not as a single, "persistent" stream that is left open for the duration of the server's lifetime. The latter violates FM semantics and may never actually send the data.

Finally, both implementations of FM are thread-safe, but do not offer any additional performance benefit to multithreaded programs. That is, only one thread at a time may be sending or extracting data.

Go to the first, previous, next, last section, table of contents.