An outlier in an independent samples design

Derrick, Ben

Abstract

There is a flaw with some of the most commonly performed statistical tests. A paradox of the one sample t-test is the contrariwise decrease in the p-value as the value of an outlier increases in the direction of the overall effect. Demonstration of this paradox is extended to the equal variances assumed and Welch's unrestricted to equal variances independent samples t-test. The phenomenon is explored using Monte-Carlo simulation, and compared with alternative two sample tests; the Mann-Whitney U test, and the Yuen-Welch t-test with 10% trimming per tail. Scenarios where the overall effect is concordant or discordant with the direction of the aberrant observation are considered.

Sample data is generated under normality, with the subsequent inclusion of an aberrant observation in one sample. The aberrant observation is systematically varied. The total sample sizes for each of the two samples within a factorial design are {10, 15, 20}. The variances within the factorial design are {1, 4}. For each parameter combination, the proportion of 10,000 iterations where the null hypothesis is rejected is calculated at the 5% significance level, two sided.

It is evidenced that the paradox for both forms of the independent samples t-test is exacerbated when the smaller sample size with the higher variance includes the aberrant observation, and as the imbalance between the sample sizes increases. Results also indicate that when the sample with the lower variance includes the aberrant observation, Welch's t-test and the Yuen-Welch t-test most closely retain Type I error robustness.

Recommendations on choice of test for independent samples designs are given, ending with discussion on how these results impact analyses for partially overlapping samples designs.

• Presentation Conference Type: Conference Paper (unpublished)
• Conference Name: Royal Statistical Society Conference
• Start Date: Sep 3, 2018
• End Date: Sep 7, 2018
• Acceptance Date: May 17, 2018
• Publicly Available Date: Jun 7, 2019
• Peer Reviewed: Not Peer Reviewed
• Keywords: outlier, extreme observation paradox, independent samples, partially overlapping samples
• Public URL: https://uwe-repository.worktribe.com/output/868039
• Additional Information: Title of Conference or Conference Proceedings : Royal Statistical Society Conference, 2018