Benchmarking additive fabricators

From: Marshall Burns (Ennex  Corporation)
To: André Dolenc (Helsinki University of Technology), RP-ML
Cc: Richard Aubin (United Technologies), Ralph Wachter (Office of Naval Research)
Date: Sunday, August 14, 1994
Subject: Benchmarking additive fabricators
Dear Andre,

     Thank you for the package of information on your work. And thank you so much for creating and administering this marvelous thing called a mailing list! I have just spent the weekend reading the archived messages, 122 of them dating back to August of last year. The historians of 20th century technology will come back to your forum to understand what was going on at the grass roots level of autofab (or RP, pardon my terminology!) in its earliest stages.

     Now, a reply to your questions on benchmarking:
On 1994 07 26, you wrote to the RP-ML:
1. Is there a clear understanding on how to benchmark RP processes?
2. Is there a consensus on how to benchmark RP processes?
3. If not, isn't time we knew how to do this?
4. The test objects being used in benchmarks should be publicaly available. Are they easy to obtain? How does one obtain them?

     My responses would be:

     No. There is neither an understanding, nor a consensus. There is a short history which includes my work and that of Bruce Haas in 1991, followed by much more comprehensive results published separately by Vern Schmidt, Doug van Putte, and Geoff Lart in 1992. Analysis of Ed Gargiulo's "user part," while not a comparison of different machines, also addressed important issues in benchmarking.
     Now we have the exhaustive study conducted by Richard F. (Dick) Aubin, "A World Wide Assessment of Rapid Prototyping Technologies," sponsored by the U.S. government under the Intelligent Manufacturing Systems program. This report compares output from fabricators of nine commercial vendors in four different countries, as well as experimental processes at three university laboratories. Anyone who hasn't seen this beautiful analysis should ask Aubin for a copy of it (e-address below).
     In addition, a new study is underway by Amit Bagchi of Clemson University and Sean O'Reilly of Ford Motor. They have some preliminary results, and should have more to say in time.
     Both Dick Aubin's work and the Clemson/Ford benchmark were reported at the recent Austin Conference, so they will be in those proceedings, out in November.

     But getting back to your question, I don't think Aubin or Bagchi or O'Reilly would claim to have come anywhere near the last word in benchmarking these systems. If anything, the increasing complexity of the studies being done shows that we have a long way to go. Indeed, the complexity of the issues involved guarantee that there can never be a final, best benchmark. Look at automobiles. They have been around for 100 years, and every season Auto Trend and Car & Driver Magazine think of new ways to decide which ones are the best, and they seldom agree! Benchmarking is an attempt to analyse quantitatively what is at its core a subjective question: "Which one should I buy." As such it gives us important guidance, not a hard answer.

     One thing to note: All of the studies so far have focused on the process side of autofab. We still need 
     (a) benchmarking of different materials run in the same machine, measuring not only speed and accuracy, but also comparing physical properties, such as strength, shock resistance, creep, chemical stability, and a host more, and 
     (b) benchmarking of computational tools, such as StL file generators, slicing algorithms, etc.

Even these items only scratch the surface. For another list, look at the mail from Jake Tucker on 1994 07 22, where he listed 17 parameters that he uses to compare systems against each other. Each of these parameters and many more are candidate subjects for benchmarking.

     On your question about getting the test objects, I presume you mean the StL files for them. I believe that both of the current projects welcome requests for these files, and I hope they don't mind if I broadcast their e-addresses here:
     Richard Aubin, United Technologies, RFA@utrc.utc.com
     (I don't have Bagchi's or O'Reilly's e-address, try:)
     Elaine Persall, Clemson University, persall@eng.clemson.edu


     Boy, Andre, you seem to have got me into long-winded mode here. I hope the ML subscribers don't object to the length of my first submission, and I hope some of this may be helpful.