Michael Hucka, Andrew Finney, Herbert Sauro, Hamid Bolouri
{mhucka,afinney,hsauro,hbolouri}@cds.caltech.edu
Systems Biology Workbench Development Group
ERATO Kitano Systems Biology Project
Control and Dynamical Systems, MC 107-81
California Institute of Technology, Pasadena, CA 91125, USA
http://www.cds.caltech.edu/erato
Principal Investigators: John Doyle and Hiroaki Kitano
12 April 2001
Revised: 4 June 2001
The Systems Biology Workbench (SBW) is a broker-based architecture that provides infrastructure enabling software components to communicate with each other. A component in SBW (called an SBW module) can take many forms: it may be primarily computational and lack a GUI, or it may be a computational module having its own GUI, or it may consist solely of a GUI designed to control other tools. A critical requirement for SBW is supporting modules written in any major programming language.
For a variety of reasons that are outside the scope of the present report, the SBW project had at the time of this writing decided to avoid more complex technologies such as CORBA (Vinoski, 1997; OMG, 2001; Seetharaman, 1998) and ILU (Janssen et al., 1999), and settled instead on using a message-passing approach for communications between modules. We still needed to resolve two basic issues: the message format (how a structured piece of information is encoded prior to transmission; i.e., the content encoding) and the transport protocol (how a message is exchanged between agents, including issues of handshaking and network port numbers).
Our main criteria for choosing a suitable message-passing scheme were: performance, support for data types needed for SBW, simplicity, and portability. We examined a number of existing frameworks including XML-RPC (McLaughlin, 2000; Winer, 2001; UserLand Inc., 2001), SOAP (Box et al., 2000), MPI (Gropp et al., 1999), Java RMI (Steflik and Sridharan, 2000; Sun Microsystems, 2001), and a homegrown RPC scheme developed by Andrew Finney and Herbert Sauro in the first phase of the SBW project. We narrowed the possibilities to two candidates: our original ``homegrown-RPC'' implementation, and XML-RPC. In this report, we summarize the results of our comparison between these two alternatives.
It is first useful to clarify how a ``message-passing system'' differs from simply using sockets. Frankly, the distinctions are muddy. Nearly every network standard currently in widespread use already is a form of message passing: even a low-level protocol such as TCP/IP involves bundling data into messages that are sent from one agent to another using network sockets. The key difference is that this level of the protocol is hidden from a developer by an API that involves sockets and (perhaps) stream interfaces. So what we are doing is layering a higher-level message-passing scheme on top of the more basic socket functionality, with the details of the messages tailored specifically to the needs of the application we are developing.
The message-passing system implemented in the first phase of SBW development is a relatively simple scheme that encodes data values as bytes and sends them in a stream across a socket connection. The message encoding has a custom format, in which each data element in a message is prefixed by a byte that describes its data type. This ``homegrown-RPC'' implementation supports the following common and useful data types:
The operations in the basic API center around blocking and non-blocking remote procedure calls. The blocking version (call) invokes a specific method in a specific module, handing it arguments serialized into a message data stream. The call waits until the method on the remote module returns a value. The non-blocking version (send) is similar, except that it does not wait for a return value. The homegrown-RPC scheme also provides a means for returning exception codes to a caller.
We were reluctant to invent a custom message encoding format, given that a number of alternatives already existed. We examined a number of well-known candidates, including Sun RPC XDR (Sun Microsystems, 1987), DCE NDR (Open Software Foundation, 1993), and the Java Object Serialization Specification (Sun Microsystems, 1998). However, from the standpoint of our requirements, each of these alternatives suffered from deficiencies. For example, both XDR and NDR are untagged formats, which means that two parties communicating data must know ahead of time the exact structure being sent, and Java serialization has substantial (and for our purposes, unnecessary) baggage for dealing with object classes.
XML-RPC (http://www.xmlrpc.com) is a lightweight remote procedure
calling protocol that uses HTTP as the transport and XML as the message
encoding. The following is an example of an XML-RPC message body (minus
the HTTP headers); this example calls a function named
sumAndDifference on the object sample, giving it two
integer arguments as parameters:
XML-RPC supports a variety of data types, including heterogeneous lists and homogeneous arrays; other types can be transmitted by encoding them in base64 format. The types supported are:
An XML-RPC transaction is synchronous and involves returning a result code; this code may be a fault/exception.
We tested Java implementations of both the XML-RPC and homegrown-RPC systems. The performance tests consisted of timing how long it took each messaging implementation to exchange a certain number of messages. We tested five different scenarios:
We attempted to eliminate some confounding variables:
For the the XML-RPC portion of these tests, we started with an off-the-shelf solution available on the Internet (version 1.0beta4 of the Helma XML-RPC implementation, http://helma.at/hannes/xmlrpc). We substituted a different XML parser (MinML; Wilson, 2001), which we found to have higher performance on the test suite used here. (Appendix A lists the XML parsers that we examined for this report.) Initial tests showed that long arrays of double's took inordinately long to transfer, so we performed one optimization: we changed the XML-RPC implementation to encode arrays of double's as byte arrays. (Strictly speaking, this is not in the spirit of XML-RPC, but it is also not directly forbidden.) The test results here are based on this modified implementation.
The timing results are presented in Table 1. The table shows that in terms of absolute performance, XML-RPC message exchanges for short messages took 1.3-1.6 ms mean time, whereas the homegrown-RPC solution took 0.19-0.28 ms mean time. This represents a factor of 5 to 8 in difference. This difference is similar for the case of empty messages; there again, the XML-RPC approach took approximately ten times as long to exchange messages as the homegrown-RPC method. For long messages, exchange times were similar for both methods.
The results in the previous section imply that message exchanges using the XML-RPC approach take 5 to 10 times as long as the homegrown-RPC scheme. On the face of it, these results favor the homegrown-RPC scheme. However, to be fair, a number of additional factors should be considered:
Absolute message-exchange times are relatively meaningless unless placed in the context of an application's run times. In SBW, one of the situations that will demand high performance is optimization, in which an objective function involves repeatedly performing a simulation (e.g., using Jarnac; Sauro 2000; Sauro and Fell 1991) under the control of an optimizer (e.g., Gepasi; Mendes 2001,1997). The scenario is something like the following:
![\includegraphics[scale = 0.7]{optimization-sequence}](img2.gif)
We attempted to estimate how long a typical simulation might take by examining a few example Jarnac simulations and then making some rough estimates for other cases. The results are shown in Table 2.
The estimates in Table 2 show that the run time for reaching
steady-state in a model (the most likely situation used in an optimization
problem) is 1.5 ms at minimum. The actual run times for real cases
are almost guaranteed to be longer. Let us assume that actual ranges will
be closer to the mean value of the extremes: so, instead of 1.5-5 ms for a
small model and 15-50 ms for a large model, we take
ms (rounded down) as the average time to compute a result for a
realistic model in a system such as Jarnac.
This has implications for the message communications method. Let us
suppose that message round-trip times are on the order of 1 ms, as in the
XML-RPC tests. This implies that the total time spent on each
computational cycle will be 
ms. A message time of 1 ms is
equal to 1/31 of this total, or 3.2%. On the other hand, the
homegrown-RPC case takes approximately 0.1 ms, which implies that the total
time spent on each computational cycle will be
ms. A
message time of 0.1 ms, then, is equal to 0.1/30.1 of this total, or
0.33%.
Note, however, that although the difference in percentages of time spent in communications appears large (3.2% versus 0.33%), the difference in absolute times (31 ms versus 30.1 ms) is small: it is 2.9%.
Post-experimental analysis revealed that the homegrown-RPC test server code did not perform as many operations as the XML-RPC code. Here is an explanation of why.
The XML-RPC code used in these experiments was a full implementation that included code for invoking handlers for different message types. On the server side, each RPC message resulted in the server code parsing out the name of the method being invoked, looking up the handler for that method in a hash table, then invoking a specific method on the handler object. The handler object takes care of reading the arguments (either a string array or an array of double's) and returning a result.
Conversely, the code in the homegrown-RPC test server specifically looked
for one of two cases (either a string or an array), read that particular
data type directly, and returned the result immediately. The following is
the relevant code:
This implementation does not reflect the complete set of operations that a
production server would have to perform: a real-world server would need to
provide hooks for invoking different handler objects based on different RPC
method invocations. The XML-RPC test server therefore represents more of a
real-world implementation than the homegrown-RPC test server.
Compared to the homegrown-RPC version, the extra operations performed by the XML-RPC implementation undoubtedly added some overhead and slightly inflated the results in Table 1. The fixed overhead is likely to be small, but further testing would be necessary to determine exactly how much. Nevertheless, when reading the results in Table 1, it should be kept in mind that the two cases are not performing identical operations.
Can the implementations of the message-passing frameworks discussed here be improved? At the time of this writing, we investigated this question only for the XML-RPC implementation; thus, our discussion here is limited to that case, but we feel confident that improvements could also be made to the homegrown-RPC implementation.
As mentioned above, the test results already reflect one optimization on the XML-RPC implementation, namely to package arrays of double's as byte streams and thereby reduce the time required to parse the arrays by the XML parser. We also experimented with different XML parsers and chose the one that provided the fastest performance (see Appendix A).
The results in Table 1 show that empty messages take almost as long to exchange as messages containing actual data. This implies that most of the differences in run times between the XML-RPC and homegrown-RPC cases are due to fixed overhead costs in XML-RPC. Visual inspection of the XML-RPC implementation leads us to believe that a significant number of additional operations could in fact be streamlined. Brief profiling showed that it performs a substantial number of string operations; an obvious area to improve is reducing the use of the Java String data type. The XML-RPC implementation also uses threading; another possible area to investigate is to handle threading differently.
It is difficult to estimate the impact that such optimizations may have. Let us make some rough estimates and assume that further optimizations on the XML-RPC framework would improve performance by 10%-25%. Table 3 shows the run times that could be expected in that case.
Some additional issues are worth mentioning in the context of deciding on an approach for SBW.
XML-RPC has an established following among a number of software developers world-wide; it is even described and used in several chapters of a book on Java and XML (McLaughlin, 2000). Even Microsoft is using a more elaborate version of XML-RPC, SOAP, as the cornerstone of its .NET strategy. Although SBW will hide the message format and protocol behind libraries used by SBW developers, widespread support and the availability of books describing this message-passing approach lends it a certain cachet that a unique homegrown-RPC approach lacks.
The data types made available in the homegrown-RPC implementation were chosen (1) to support the types of data that we expected would be needed by systems biology simulators, and (2) to provide a reasonable balance between flexibility and conciseness. We knew that data types such as IEEE floating point would be important for simulation software. It is therefore important to note that XML-RPC does not specify the use of IEEE floating point numbers: XML-RPC's floating point type is double, whatever that may be in a given programming language, and when written out in XML, the double is expressed as a sequence of ASCII characters (e.g., "234.255"). As a consequence, there is no formalized way of expressing overflow and NaN. Thus, using XML-RPC as the basis of the SBW communications framework would require that we add mechanisms to support IEEE floating-point quantities (perhaps by manually encoding numbers using byte arrays). This would likely have no impact on performance, but it would call into question the logic of using XML-RPC when it does not support the basic data types needed by our application.
XML-RPC implementations are currently available for a wide variety of languages, including Python, Delphi, Java, C, C++, AppleScript, Guile, Tcl, and Perl. This suggests it would be easier to implement support for new languages in SBW if we used XML-RPC, because a starting point for the message-passing layer would likely be available. (But see the next subsection.)
Although many XML-RPC implementations are available, our informal survey of the available C/C++ and Delphi implementations shows that they were either immature or designed in an awkward-to-use manner. We believe that we would have to implement our own XML-RPC libraries for at least C/C++, and possibly other languages. This negates one of the primary motivations for considering XML-RPC, namely that we would be spared the effort of implementing a communications framework by using off-the-shelf software.
Because XML-RPC uses HTTP as its transport protocol, it is able to cross most network firewall without special requirements because most firewalls already allow HTTP traffic to pass through. Crossing a firewall using the homegrown-RPC method would require the firewall to allow network traffic on a nonstandard port. Most site administrators, especially in corporate network environments, are reluctant to introduces special-case holes in their network security systems.
Unfortunately, it turns out that for SBW, using XML-RPC would not make it easier to cross network firewalls. The problem is described in the following subsection. Thus, for our application, neither the homegrown-RPC nor the XML-RPC approaches provide an immediate solution.
The broker architecture of SBW requires bidirectional communications between modules. One module may call on another (via the broker as intermediary), and two modules may exchange information with each other. In all cases, any module can serve as the initiator. An important question, then, is whether a given communications framework cleanly supports this kind of full-duplex connectivity.
One of our realizations during the testing process was that XML-RPC does not support bidirectional connections. The root of the problem is the HTTP protocol, which is oriented towards client-server applications in which an agent initiates a connection to a server listening on a designated TCP/IP port. The implication of using XML-RPC for SBW is that if modules A and B needed to invoke operations on each other simultaneously, module A would have to initiate a connection to B and B would have to initiate a separate connection on module A. Although this would not be impossible, it has two important implications: (1) every module would need to be assigned a unique TCP/IP port on a given machine, so that it could listen for requests directed at it; and (2) the firewall transparency implied by using HTTP would be lost, because modules would have to use ports other than the standard HTTP port and communications could not all take place through a single connection through a firewall as in normal HTTP.
It appears the only work-around to this would be to violate the HTTP protocol and somehow tunnel a bidirectional data stream over a single HTTP connection. However, such an implementation would not adhere to the XML-RPC specification and would not be compatible with existing XML-RPC implementations.
The tests reported here show the homegrown-RPC system to be more efficient than the XML-RPC system tested. It is possible that the performance of the XML-RPC implementation can be improved; however, it will never be as fast as the homegrown-RPC scheme because XML-RPC involves a more verbose data encoding and a more costly parsing/writing approach. On the other hand, for the average run times expected for SBW modules, the time required to exchange a message using either scheme is relatively small, and the relative difference between the schemes is even smaller: roughly 3%.
But there are other considerations. Although XML-RPC has certain advantages as a more established approach that is documented and recognized on the Internet, XML-RPC as a standard lacks support for certain data types that are important for the Systems Biology Workbench. Further, although existing implementations are available for a variety of programming languages, the quality of some of the implementations has not impressed us. If we cannot count on being able to use existing implementations, we are not saved any work compared to implementing our own messaging/RPC scheme. From the standpoint of software developers who wish to use SBW, the existence of XML-RPC implementations may be a moot point: most developers will use our SBW libraries and these will hide the underlying message-passing system behind an API. Few developers will care about the details of the messaging scheme.
A more significant concern is that XML-RPC is not oriented towards bidirectional message exchanges (as discussed in Section 5.4.6). At best, this means that the theoretical firewall transparency of using XML-RPC would be lost in our application; but in fact, this particular issue raises the more fundamental question of whether XML-RPC has an architecture suitable for the needs of SBW.
Given that SOAP (Box et al., 2000) and XML-RPC are fundamentally quite similar, we believe most of the objections raised above to using XML-RPC would apply to SOAP as well.
For these reasons, we have decided to continue using our homegrown-RPC scheme as the basis for the message-passing infrastructure in SBW, rather than XML-RPC or its cousin SOAP. If it proves necessary, there is always the option of implementing multiple messaging protocols inside the SBW broker and libraries, and translating between them internally.
A number of free XML parsers written in Java were available at the time of this writing. We tested the following alternatives:
Informal experiments using the test suite described in this document revealed that MinML gave the fastest performance (i.e., the shortest message transmission times). We therefore used MinML in the tests discussed in Sections 3-5.
Shown below is the Java code for the test driver used in the timing tests presented in this report.
/*
** classname : TestBed.java
** Description : Test client
** Author(s) : Andrew Finney, Michael Hucka
** Organization: Caltech ERATO Kitano, California Institute of Technology
** Created : 2001-03-27
** Revision : $Id: messaging-comparison.html,v 1.1.1.1 2001/10/12 18:14:50 mhucka Exp $
** $Source: /cvs/sysbio/www/external/current/sbw/docs/messaging-comparison/html/messaging-comparison.html,v $
**
** Copyright (C) 2001 California Institute of Technology, and Japan
** Science and Technology Corporation.
*/
/**
* @author Andrew Finney, Michael Hucka
* @author $Author: mhucka $
* @version $version$
**/
public class TestBed
{
private static double[] getFilledArray(int x)
{
double[] result = new double[x];
int i = 0 ;
while (i != x)
{
double v = i ;
result[i] = v + v / 100 ;
i++ ;
}
return result ;
}
private static String getFilledString(int x)
{
String alphabet = "abcdefghijklmnopqrstuvwxyz";
StringBuffer buffer = new StringBuffer(x);
int i = 0 ;
while (i < x)
{
buffer.insert(i, alphabet);
i += alphabet.length() ;
}
buffer.setLength(x);
return buffer.toString();
}
public static void main(String[] args)
{
SimpleSend client = new SimpleSendClient();
int i = 0 ;
// Test sending empty messages.
System.out.println("Sending nothing.");
long emptyCount = 10000;
long time = System.currentTimeMillis();
while (i != emptyCount)
{
client.send();
i++ ;
}
long emptyTime = System.currentTimeMillis() - time ;
System.out.println(emptyCount + " sends of nothing took: "
+ emptyTime);
// Test sending short array of doubles.
System.out.println("Sending 1000 double[10]'s.");
long shortArrayCount = 1000;
double[] array = getFilledArray(10);
i = 0 ;
time = System.currentTimeMillis();
while (i != shortArrayCount)
{
client.send(array);
i++ ;
}
long shortArrayTime = System.currentTimeMillis() - time ;
System.out.println(shortArrayCount + " sends of double[10] took: "
+ shortArrayTime);
// Test sending large array of doubles.
System.out.println("Sending 100 double[1000]'s.");
long longArrayCount = 100;
array = getFilledArray(1000);
i = 0 ;
time = System.currentTimeMillis();
while (i != longArrayCount)
{
client.send(array);
i++ ;
}
long longArrayTime = System.currentTimeMillis() - time ;
System.out.println(longArrayCount + " sends of double[1000] took: "
+ longArrayTime);
// Test sending short strings.
System.out.println("Sending 1000 strings of 100 chars.");
long shortStringCount = 1000;
String string = getFilledString(100);
i = 0 ;
time = System.currentTimeMillis();
while (i != shortStringCount)
{
client.send(string);
i++ ;
}
long shortStringTime = System.currentTimeMillis() - time ;
System.out.println(shortStringCount + " sends of 100 char string took: "
+ shortStringTime);
// Test sending long strings.
System.out.println("Sending 100 strings of 10000 chars.");
long longStringCount = 100;
string = getFilledString(10000);
i = 0 ;
time = System.currentTimeMillis();
while (i != longStringCount)
{
client.send(string);
i++ ;
}
long longStringTime = System.currentTimeMillis() - time ;
System.out.println(longStringCount + " sends of 10000 char string took: "
+ longStringTime);
// Print stats.
System.out.println("Times: ");
System.out.println(" Total elapsed time = "
+ (emptyTime + longStringTime +
shortStringTime + longArrayTime + shortArrayTime));
System.out.println(" Mean time for empty = "
+ (double) emptyTime/emptyCount);
System.out.println(" Mean time for short array = "
+ (double) shortArrayTime/shortArrayCount);
System.out.println(" Mean time for long array = "
+ (double) longArrayTime/longArrayCount);
System.out.println(" Mean time for short string = "
+ (double) shortStringTime/shortStringCount);
System.out.println(" Mean time for long string = "
+ (double) longStringTime/longStringCount);
}
}
This document was generated using the LaTeX2HTML translator Version 99.2beta8 (1.46)
Copyright © 1993, 1994, 1995, 1996,
Nikos Drakos,
Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999,
Ross Moore,
Mathematics Department, Macquarie University, Sydney.
The command line arguments were:
latex2html -white -split +0 -show_section_numbers -image_type gif -no_navigation -local_icons -discard -mkdir -dir html messaging-comparison
The translation was initiated by on 2001-06-15