Research & Development
by Joyce Ann Guzik, NAR 37191
Editor’s note: Joyce has judged the R&D event at fifteen NARAMs. She is a staff scientist at Los Alamos National Laboratory and holds a Ph.D. degree in Astrophysics.
I thought it would be good to give you an idea of what the judges are looking for and, conversely, what the judges have often found lacking about R&D reports, and also give you some useful tips and advice.
First, it is important to read the R&D rules in the Model Rocket Sporting Code on this event. You must submit three (3) copies of your R&D report for judging, and starting in 2016 you are required to submit the report in electronic format two weeks before NARAM in order to give the judges time to review it before NARAM. The rules give a list of elements that must be included in a report in order to qualify for the event, such as stating how much it cost to do your project. You must also submit a separate 250-300 word written summary of the report. You must be prepared to give an oral presentation before the judges if your project is in contention for 1st through 4th place.
PROJECT IDEAS
According to the Sporting Code, an R&D project is supposed to either advance the state of the art of model rocketry, or use model rocketry as a research tool. Follow your curiosity. A good project has an idea that shows creativity and originality, is feasible to execute in a reasonable amount of time, and (sometimes fortuitously) leads to a result that is useful, or advances everyone’s understanding.
Is there something that has been mysterious, that you have always wanted to understand? Have you wanted to improve the performance of your rocket? Is there something that has always annoyed you about your rocket (misfires, the way your parachute tangles or won’t deploy, red barons, not being able to find it after launch, the difficulties of altitude tracking, lack of guidance, safety concerns, etc.) for which you might have a novel solution? Is there something that you could use model rocketry for (e.g., taking atmospheric science data, taking aerial photos for a particular use, developing a science curriculum to teach some aspect of physics or math or chemistry, etc.)? For examples of what R&D projects have been done in the past, you can search the bibliography that is posted at the bottom of this page. (Alternatively, click here for a list of R&D Reports available to NAR members).
START EARLY!
Don’t plan on doing your project a few days before NARAM! The winning R&D projects in the upper divisions often take months to over a year to complete, with multiple aspects/parts/approaches. Building your test rockets or equipments, testing, allowing time for failures and revising your approach, and getting a statistically significant amount of quantitative data from experiments and flight tests (see below) all take time.
Keep a logbook chronicling your progress, ideas, data, and observations. It will be useful when it comes time to write your report.
THE WRITTEN REPORT
Be sure that everything listed in Pink Book rule 63.5 is included in your report. Your entry could be disqualified if it does not contain all of the required elements in some form. Include an introduction about why you chose this project and what was your initial (and sometimes evolving) objective for the project.
A written summary of your report is also required. This is not simply a repeat of the introduction, or an advertisement for the rest of the project-it is a succinct summary of your objective, a brief statement of the means you used to meet it (e.g., calculations, flight tests, engineering tests, or other data gathering), and a statement of the conclusions and significant results (What did you learn? Did you meet the objective?). If a reader had time to read only your summary, what would you want him/her to know about your project?
Keep a bibliography and list the sources that you used for reference. Do go to the library (surprisingly, there is stuff there that isn’t on the web!), look at the web, and ask other experienced modelers. But make sure your writing is original-DO NOT cut and paste entire paragraphs from reference sources on the web or from books unless you have a purpose for them in the context of the report. Material taken from a source should appear within quotes in your report, and a reference to the original source should appear in your bibliography. Even if you paraphrase material, or put it in your own words, you need to reference the original source. Also, stick to the point of your project, and do not include extensive irrelevant information or descriptions. You will not earn extra credit for padding out your project with more pages.
Periodically during your preparations, you should try to step back and think carefully about your experimental setup and think about what else could account for or affect your results (was Rocket B heavier than rocket A by a little? Did Rocket B have slightly crooked fins and so not fly as straight or have more drag?). Include these thoughts in your report. Even if you don’t know how to correct for such systematic errors, just doing some thoughtful analysis and itemizing other possible causes for the results will earn you the esteem of the judges!
It would also be good after the conclusions in your report to include a retrospective on what you would do differently if you could start over, your thoughts on follow-up work that could be done to improve the project, or other ideas that this project inspired.
Please have several other people read your report before you enter it in the R&D event. Have them make suggestions for improvements. Find out what things they thought were confusing or missing. Also, be careful to eliminate all of the grammar and spelling mistakes (a good proofreader helps here too!).
RESULTS
One thing that the judges are especially looking for is that the contestant is not drawing incorrect or unsupported conclusions from their observations and experimental results.
This can arise for two reasons. First, the contestant may not have enough controls or checks in his/her experimental process. For example, if you are doing a lot of flight tests with several different models, you need to check for and record possible differences between the models. Do they have the same mass, surface finish, fin area, frontal area, etc.? Also, follow procedures that help to ‘randomize’ the effect of influences that you may not consider or can’t control. For example, if you are comparing the performance of A vs. B engines using what you think are two identical models, don’t fly all of the A engines in one model, and all of the B engines in the other model; divide the two types of engines evenly between the models. Similarly, if you are comparing two different types of models, don’t make all the flights of model #1 in the cool, clam morning, and then make all the flights of model #2 in the hot, windy afternoon; mix the flights together so that external changes affect both groups equally. The second reason people draw unsupported conclusions is that they don’t consider whether any differences in their results are statistically significant or not. For any quantitative data it is a must to have some statistical analysis. Most importantly, you need to have enough trials of an experiment to get statistically significant results. One to three trials is much too few to get significant results; ten to twenty would be much better. At a minimum, you should calculate the ‘mean’ (average) and the ‘standard deviation of the mean’ or all sets of experimental data you want to compare.
The short technical report Basic Statistics for R&D by NASA astronaut and rocketeer Dr. Jay Apt, available from NARTS (www.nar.org/narts), discusses in detail the calculation of mean and standard deviation of the mean. This is an estimate of how far your calculated mean is likely to differ from the true mean that would be found by doing a very large number of trials. There is a 68% chance that the true mean lies in the range of one standard deviation on either side of your sample mean; the chance is 95% that it’s in the range of two standard deviations.
If the standard deviation error ranges of two sets of experimental data overlap, then you cannot claim there is a statistically significant difference between the two sets. For example, as Dr. Apt points out, if you have a set of trials of rocket A with an average altitude 320 ± 20 meters, and a set of rocket B with 310 ± 20 meters, you cannot conclude that rocket A reaches higher altitudes than rocket B! Including more trials in a data set gives a narrower error range for the standard deviation and can result in you being able to determine a statistically significant difference that fewer trials will not reveal.
Figure 1. Rocket data example spreadsheet
Rather than duplicate Dr. Apt’s report by demonstrating how to calculate the mean and 
for your data, we show you how to do it in an Excel spreadsheet (see Figure 1). The bottom of the figure shows what formulas are in the cells of column B (the same formulas are copied across into column C). The “track lost” entry in cell C6 is there to show that the formulas ignore cells that are blank or contain non-numerical data.
The spreadsheet shows two sets of rocket altitude measurements and the mean (average) of each set. But is there a significant difference between the means? Using the values from row 10, we can see that the error ranges 191.0 ± 5.2 and 202.0 ± 4.7 do not overlap, so the data are significantly different (at least to ‘one sigma’-one standard deviation of the mean). The difference in the results would be more convincing if the data were different at the ‘two sigma’ level (95% confidence) but they are not, since doubling the error ranges causes them to overlap. By the way: if the we only had the first three flights in each set, the results are 191.7 ± 11.1 and 204.3 ± 8.0, which does not show a clear difference between the two sets since those ranges overlap.
The Student’s t-test formula in cell B13 is another statistical test that Excel can perform. The result shown tells us that there is only a 13% chance that the two data sets come from the same population, another indication that the data are significantly different.
ORAL PRESENTATION
In addition to the written report, the contestants that are in line to place must do an oral presentation in front of three NARAM R&D judges and a friendly audience of the interested NARAM attendees, and answer questions following their talk from the judges and audience. The presentation can significantly affect the judges’ opinions, but remember that you need to do well enough to impress the judges on the written report to earn the chance to present the oral in the first place!
Prepare your oral talk well: Have a Powerpoint digital presentation summarizing your report (proofread it!), and use large font sizes for the text so people in the back of the room can see them. Check with the event organizers to ensure that you know whether you are supposed to bring a laptop of your own or just a digital file to load onto their laptop. Have some props as well (samples of your rocket, apparatus, etc.) to show the judges and pass around. Practice in front of your family and friends two or three times before coming to NARAM and ask them for their honest constructive criticism regardless of how much it hurts your ego!
The R&D rules state that the oral presentation shall not exceed 15 minutes in length, but the constraints of time at NARAM usually lead to the judges to request talks with a length of 5 or 10 minutes, so you should be prepared for either length. Contestants that go on past the time limit may be cut off from necessity without finishing. One viewgraph per minute is a good rule of thumb. You could have an additional 5 minutes of material in reserve if it turns out there is time for a longer format. The judges and audience will not interrupt you with questions during your presentation, but there will typically be five minutes of questioning after you have finished your talk. Practice answering questions from your family and friends.
The judges may ask you to demonstrate some aspect of your project on the field on Friday, so be prepared for this if you think your project might warrant it.
Good luck! We are hoping to see many outstanding projects at NARAM.
Related Documents:
| Basic Statistics for R&D (Apt) | May 28, 2014, 9:31 pm | 190 KB | |
| Rocketry Technical Bibliography (Apr 2015) | April 12, 2015, 10:53 pm | 215 KB |