Chapter 6 Statistical Testing

Modified: 2006-03-23

Objectives for Chapter 6

After studying this chapter and working the problems, you should be able to:

• 1. Explain negative inference logic and how it applies to chi square problems of independence
• 2. Distinguish among the null hypothesis, alternative hypothesis, and the research hypothesis
• 3. Explain the logic of null hypothesis statistical testing (NHST)
• 4. For the statement, p < .05, name the event that has a probability of .05
• 5. Describe a sampling distribution and the part it plays in NHST
• 6. Distinguish between a one-tailed and a two-tailed statistical test
• 7. Distinguish between Type I errors and Type II errors
• 8. Interpret the outcome of studies analyzed with chi square, Pearson r, and t tests
• 9. Calculate and interpret the effect size indexes, f and d
• 10. Determine the effect size index for a Pearson r
• 11. Calculate and interpret a confidence interval about a mean difference
• 12. Define the power of a statistical test
• 13. Identify the four factors that determined the power of a statistical test
• 14. Identify the conventional cut-off probability values that signify statistical significance and adequate statistical power
• 15. Explain the concept of meta-analysis and its statistic,

Gender	Nonviolent	Violent	Total
Female	24	7	31
Male	87	65	152
Total	111	72	183

Table 6.1 Observed frequencies from study of gender and video games. Total frequencies are in the margins.

SAMPLING FLUCTUATION

sampling fluctuation --The chance differences between samples and the population the samples are from.

chi square test--A NHST test that is appropriate for category data.

A NEGATIVE INFERENCE LOGIC PROBLEM

• If a situation is either A or B and it is not A, what is it?
  • If either A or B and not A, then B is a way of thinking about negative inference.
  • This kind of thinking is called negative inference (AKA modus tollens)
    • Think of traffic problems or political choices each with two choices
    • Usual route is clogged, use other
    • You dislike one candidate, vote for the other

CHI SQUARE LOGIC

• Statistical Independence
  • Two variables that, as their own levels change, do not produce changes in the other variable.
• Probabilities
  • What is the probability of a sample occuring IF we assume statistical independence?
  • Notice that the two data arrays have the same row and column totals are the same. Those totals are also the same as the array above shown in Table 6.1.
    • The probability of Table 6.6 Array A occuring is .38

21	10	31
90	62	152
111	72	183

• The probability of Table 6.6 Array B occuring is .036

29	2	31
82	70	152
111	72	183

• If a low probabilty data array actually occurs, then we must reject the idea that the variables that produced the data were statiscally independent
• Can you see how this is a case of negative inference?

• Conclusions
  • research hypothesis--The researcher's expectation of what the data will show.
    • the research hypothesis here is that males are more likely to play violent video games
    • we will have to use negative inference to make such a statement

NULL HYPOTHESIS STATISTICAL TESTING (NHST) LOGIC

• Sampling Distribution
  • A theoretical distribution based on random sampling that shows probabilities of actual sample outcomes
  • Each statistical test has its own sampling distribution
    • chi-square distribution
    • we will use the values derived from the chi-square distribution in order to test our research hypothesis
    • the rejection region for a two-tailed test is shown below for the sampling distribution for the normal curve (See Figure 6.1)
    • look up the values for the sampling distribution critical values from the appropriate table
    • only use one-tailed tests when you are sure of directionality in the data

Rejection regions, two-tailed

Rejection regions, one-tailed (can also be at -1.64)

• Degrees of Freedom
  • Actually, there is not one sampling distribution but a family of sampling distributions
  • Which sampling distribution to use depends on the degrees of freedom (df)
  • When using a table, use the correct value for df in order to get the critical value

Sampling distribution for t-test with 1, 10, 20, and 30 df

Above figure from: http://www.itl.nist.gov/div898/handbook/eda/section3/eda3664.htm

• Type I and Type II errors
  • Can you be sure your negative inference decision is correct?
  • That is the issue addressed by Type I and Type II errors
    • Type I error: Null hypothesis is true AND is rejected, thus FALSE REJECTION, symbolized by a
    • Type I error: Null hypothesis is false AND is accepted, thus FALSE ACCEPTANCE, symbolized by b
  • Wikipedia on:
    • False Positives
    • False Negatives
  • Adjusting p value and Type I and Type II errors
    • Making the test more sensitive (p < .01) reduces probability of Type I error BUT increases possibility of Type II error

True situation in the population

		H0 true	H0 false
The decision based on	Reject H0	Type I error (False Rejection)	Correct decision
sample data	Retain H0	Correct decision	Type II error (False acceptance)

• Significant Results and Important Results
  • Statistical significance means the null hypothesis was rejected
  • The effect size index tells you how big the difference between means is
  • NOTE:
    • results can be statistically significant with effect sizes ranging from large to small
    • results can be statistically insignificant with effect sizes ranging from large to small

NHST (Null Hypothesis Statistical Testing) TESTS

• Chi Square Test of Independence
  • used when data are frequencies
  • df = (rows - 1) * (columns - 1) For 2 x 2 chi square, df = 1
  • chi square calculators:
    • http://www.georgetown.edu/faculty/ballc/webtools/web_chi.html
    • http://www.stat.sc.edu/~west/applets/chisqdemo.html (graphic calculator, use to find critical values too)
    • http://www.unc.edu/~preacher/chisq/chisq.htm (produces table with row and column totals, watch out for small windows!)
• Pearson Product-Moment Correlation Coefficient
  • Introduced in Chapter 4
  • Maximum values of r are -1.00 to +1.00
  • Measures relationship between two or more variables
  • Must obtain multiple values from each participant (2 is minimum)
  • df = N - 2
  • take care using r for NHST testing
  • Pearson product-moment calculator
    • http://faculty.vassar.edu/lowry/corr_stats.html (enter N, the number of paired samples first)
• t tests
  • use to test differences between two samples (independent samples or paired samples)
• Independent-samples t test
  • Assumptions:
    • samples come from populations that are normally distributed and have equal variances
    • t test still works well when assumptions are violated, t test is robust
• Paired t test (AKA--correlated t , related t, dependent t, within-subjects t, split plot t)
  • Use when scores are paired:
    • two scores from same person
    • matched scores (same family, same SES, similar pretest scores)
• t test calculators:
  • http://www.graphpad.com/quickcalcs/ttest1.cfm
  • http://calculators.stat.ucla.edu/twosamp/
• Effect size calculators for t test
  • http://web.uccs.edu/lbecker/Psy590/escalc3.htm (calculators using means/standard deviations and t values and df)

TESTS FOR RANKED DATA

• Use nonparametric tests for scaled data (e.g., 1st, 2nd, 3rd.....Nth)
• Nonparametric tests do not meet assumptions of parametric tests
• Some tests:
  • Mann-Whitney (analogous to independent-samples t)
  • Wilcoxson Matched-Pairs Signed-Ranks Test (analogous to paired-samples t)
  • Spearman correlation coefficient r_s (analogous to Pearson r)

CONFIDENCE INTERVALS

• Use to test the difference between two sample means
• Use 95% or 99% confidence intervals
• If 0 is not in the confidence interval, reject the null hypothesis
• If 0 is in the confidence interval, retain the null hypothesis

A STUDENT'S GUIDE TO ANALYZING DATA

After you finish gathering data for your experiment, the next task is to analyze it. For most researchers, this is the most exciting part of the project. Here is our suggested order for analyzing the data.

• Explore the data using frequency distributions and graphs
• Calculate descriptive statistics
• Calculate an effect size index
• Analyze the data with appropriate NHST techniques
• At every step analyze the experiment for uncontrolled extraneous variables

POWER

• Power = 1 - b
• The greater the power of a test, the more likely it is to detect that the null hypothesis is false (and thus to reject it)

META-ANALYSIS

• Use meta-analysis to test a large number of research studies with same IV and similar DVs

In The Know--Chi square was invented in 1900 by Karl Pearson (of Pearson product-moment correlation coefficient fame). It is used by researchers in almost every discipline that uses quantitative data. The importance of chi square was recognized when it was listed as one of the 20 greatest discoveries of the 20th century by the editors of a popular science magazine (Hacking, 1984).

In The Know--Ronald A. Fisher introduced the null hypothesis concept in 1925 in an influential book written for practical research workers. His goal was to give researchers guidelines and the .05 cutoff was proposed as a guideline. Unfortunately, later writers and researchers treated it as a rule. The exclusive reliance on NHST techniques for analyzing quantitative data is under attack these days. In the 1990s there was a move to ban its use. The American Psychological Association (APA) assembled a task force that recommended that NHST not be banned, but that researchers not rely exclusively on NHST. The task force mentioned exploratory data analysis and confidence intervals as examples of alternatives to NHST. The APA report can be viewed at http://www.apa.org/science/bsaweb-tfsi.html. Other accessible explanations of the controversy and its outcome are in Dillon (1999) and Spatz (2000).

In the Know--It is important to recognize that NHST is just an aid to decision making. The NHST technique results in one of two decisions: 1) reject H0, accept H1 and write a strong conclusion that eliminates sampling fluctuation as an explanation. 2) fail to reject H0, which leads to retaining both H0 and H1. In recent years a number of prominent researchers have questioned the value of the NHST technique. They argue that other approaches provide a more extensive analysis and will result in faster progress in our effort to understand behavior and cognitive processes. The issues raised are not simple and are beyond the scope of this introduction to research methods. References that help explain this controversy are Dillon (1999), Spatz (2000), and Nickerson (2000). Many of the researchers who raised the questions about NHST contributed to a book whose title captures the problem, What If There Were No Significance Tests? (Harlow, Mulaik, & Steiger, 1997).

GLOSSARY

alpha (a)--The probability that is the criterion for rejecting the null hypothesis.

alternative hypothesis--A hypothesis that two variables are related or that two population means are not equal.

chi square test--A NHST test that is appropriate for category data.

confidence interval--A range of scores that is expected with a specified degree of confidence to capture a parameter.

critical value--The number from a sampling distribution that determines whether H0 is rejected.

degrees of freedom --A concept used by mathematical statisticians to determine the sampling distribution that is appropriate for a set of data.

meta-analysis--A quantitative technique that summarizes the results of many studies of a single topic.

null hypothesis--Usually a hypothesis that there is no relationship or that population means are equal.

null hypothesis statistical testing --An inferential statistics technique that measures the uncertainty that surrounds samples.

one-tailed test--A statistical test to detect a difference in population means, either positive or negative but not both.

power--Power is the probability of correctly rejecting a false null hypothesis

power analysis--A statistical analysis that solves for one of the factors that is involved in rejecting a false H0 with a NHST test.

rejection region --The portion of a sampling distribution that includes sample data that is less probable than alpha (a).

research hypothesis--The researcher's expectation of what the data will show.

robust--A statistical test that produces reasonably accurate probabilities even when the assumptions the test is based on are not fulfilled.

sampling distribution --A theoretical distribution based on random sampling that shows probabilities of actual sample outcomes.

sampling fluctuation --The chance differences between samples and the population the samples are from.

statistical independence--Two variables that, as their own levels change, do not produce changes in the other variable.

statistically significant --Sample data with a probability less than .05.

t distribution--Sampling distribution used to determine probabilities for t tests.

two-tailed test--A statistical test to detect a difference in population means, regardless of direction.