1.4: Should You Believe a Statistical Study?

Now we have looked at the basics of a statistical study, but how do you make sure that you conduct a good statistical study? You need to use the following guidelines.

Guidelines for Conducting a Statistical Study:

1. State the goal of your study precisely. Make sure you understand what you actually want to know before you collect any data. Determine exactly what you would like to learn about.
2. State the population you want to study and state the population parameter(s) of interest.
3. Choose a sampling method. A simple random sample is the best type of sample, though sometimes a stratified or cluster sample may be better depending on the question you are asking.
4. Collect the data for the sample and summarize these data by finding sample statistics.
5. Use the sample statistics to make inferences about the population parameters.
6. Draw conclusions: Determine what you learned and whether you achieved your goal.

The mistake that most people make when doing a statistical study is to collect the data, and then look at the data to see what questions can be answered. This is actually backwards. So, make sure you know what question you want to answer before you collect any data.

Even if you do not conduct your own study, you will be looking at studies that other people have conducted. Every day you hear and see statistics on the news, in newspapers and magazines, on the Internet and other places. Some of these statistics may be legitimate and beneficial, but some may be inaccurate and misleading. Here are some steps to follow when evaluating whether or not a statistical study is believable.

Steps for Determining whether a Statistical Study is Believable:

1. Are the population, goal of the study, and type of study clearly stated?

You should be able to answer the following questions when reading about a statistical study:

• Does the study have a clear goal? What is it?
• Is the population defined clearly? What is it?
• Is the type of study used clear and appropriate?

2. Is the source of the study identified? Are there any concerns with the source?

A study may not have been conducted fairly if those who funded the study are biased.

Example \(\PageIndex{1}\): Source of Study 1

Suppose a study is conducted to find out the percentage of United States college professors that belong to the Libertarian party. If this study was paid for by the Libertarian party, or another political party, then there may have been bias involved with conducting the study. Usually an independent group is a good source for conducting political studies.

Example \(\PageIndex{2}\): Source of Study 2

There was once a full-page ad in many of the newspapers around the U.S. that said that global warming was not happening. The ad gave some reasons why it was not happening based on studies conducted. At the bottom of the page, in small print, were the words that the study and ad were paid for by the oil and gas industry. So, the study may have been a good study, but since it was funded by the industry that would benefit from the results, then you should question the validity of the results.

3. Are there any confounding variables that could skew the results of the study?

Confounding variables are other possible causes that may produce the effect of interest besides the variable under study. In a scientific experiment researchers may be able to minimize the effect of confounding variables by comparing the results from a treatment group versus a control group.

Example \(\PageIndex{3}\): Confounding Variable

A study was done to show that microwave ovens were dangerous. The study involved plants, where one plant was given tap water and one plant was given water that was boiled in the microwave oven. The plant given the water that was boiled died. So the conclusion was that microwaving water caused damage to the water and thus caused the plant to die. However, it could easily have been the fact that boiling water was poured onto the plant that caused the plant to die.

4. Could there be any bias from the sampling method that was used?

Sometimes researchers will take a sample from the population and the results may be biased.

Selection Bias: This occurs when the sample chosen from the population is not representative of the population.

Participation Bias (or Nonresponse Bias): This occurs when the intended objects in the sample do not respond for many different reasons. Those who feel strongly about an issue will be more likely to participate.

Example \(\PageIndex{4}\): Bias

The 1936 Literary Digest Poll. The Literary Digest was a magazine that was founded in 1890. Starting with the 1916 U.S. presidential election, the magazine had predicted the winner of the each election. In 1936, the Literary Digest predicted that Alfred Landon would win the election in a landslide over Franklin Delano Roosevelt with fifty-seven percent of the popular vote. The process for predicting the winner was that the magazine sent out ten million mock ballots to its subscribers and names of people who had automobiles and telephones. Two million mock ballots were sent back. In reality, Roosevelt won the election with 62% of the popular vote. (“Case Study 1: The 1936 Literary Digest Poll,” n.d.)

A side note is that at the same time that the Literary Digest was publishing its prediction, a man by the name of George Gallup also conducted a poll to predict the winner of the election. Gallup only polled about fifty thousand voters using random sampling techniques, yet his prediction was that Roosevelt would win the election. His polling techniques were shown to be the more accurate method, and have been used to present-day.

Selection Bias: Because of the people whom the Literary Digest polled, they created something called a selection bias. The poll asked ten million people who owned cars, had telephones, and subscribed to the magazine. Today, you would probably think that this group of people would be representative of the entire U.S. However, in 1936 the country was in the midst of the Great Depression. So the people polled were mostly in the upper middle to upper class. They did not represent the entire country. It did not matter that the sample was very large. The most important part of a sample is that it is representative of the entire population. If the sample is not, then the results could be wrong, as demonstrated in this case. It is important to collect data so that it has the best chance of representing the entire population.

Nonresponse Bias: When looking at the number of ballots returned, two million appears to be a very large number. However, ten million ballots were sent out. So that means that only about one-fifth of all the ballots were actually returned. This is known as a nonresponse bias. The only people who probably took the time to fill out and return the ballot were those who felt strongly about the issue. So when you send out a survey, you have to pay attention to what percentage of surveys are actually returned. If at all possible, it is better to conduct the survey in person or through the telephone. Most credible polls conducted today, such as Gallup, are conducted either in person or over the telephone. Do be careful though, just because a polling group conducts the poll in person or on the telephone does not mean that it is necessarily credible.

5. Are there any problems with the setting of a survey?

The setting of the survey can create bias. So you want to make sure that the setting is as neutral as possible, so that someone does not answer based on where the survey is conducted or who is giving the survey.

Example \(\PageIndex{5}\): Setting Example

Suppose a survey is being conducted to learn more about illegal drug use among college students. If a uniformed police officer is conducting the survey, then the results will very likely be biased since the college students may feel uncomfortable telling the truth to the police officer.

6. Are there any problems with the wording of a survey?

How a question is worded can elicit a particular response. Also the order of the questions may affect a person’s answers. So make sure that the questions are worded in a way that would not lead a person to a particular answer.

Example \(\PageIndex{6}\): Wording Example

A question regarding the environment may ask “Do you think that global warming is the most important world environmental issue, or pollution of the oceans?” Alternatively, the question may be worded “Do you think that pollution of the oceans is the most important world environmental issue, or global warming?” The answers to these two questions will vary greatly simply because of how they are worded. The best way to handle a question like this is to present it in multiple choice format as follows:

What do you think is the most important world environmental issue?

a. Global warming

b. Pollution of the oceans

c. Other

7. Are the results presented fairly?

Be sure that any concluding statements accurately represent the data and statistics that were calculated from the data. Many times people make conclusions that are beyond the scope of the study, or are beyond the results of the data.

Example \(\PageIndex{7}\): Wrong Conclusion

Many studies have been done on cancer treatments using rats. The wrong conclusion is to say that because a treatment cured cancer in rats, then it will cure cancer in people. The fact that a treatment cured cancer in rats, means that there is a chance that it will cure cancer in people, but you would have to try it on people before making such claims. Rats and people have different physiology, so you cannot assume what works on one will work on the other.

8. Are there any misleading graphics?

Be sure that any graphics that are presented along with the results are not misleading. Some examples of misleading graphs are:

• The vertical axis does not start at zero. This means that any changes will look more dramatic than they really are.
• There is no title. This means that you do not know what the graph is actually portraying.
• There are missing labels or units. This means that you do not know what the variables are or what the units are.
• The wrong type of graph is used. Sometimes people use the wrong graph, like using a bar graph when a line graph would be more appropriate.

9. Final considerations

Ask yourself the following questions about the overall effectiveness of the statistical study.

• Do the conclusions of the study answer the initial goal of the study?
• Do the conclusions of the study follow from the data and statistics?
• Do the conclusions of the study indicate practical changes should be made?

Overall, you should follow these steps when analyzing the validity of any statistical study.