As a scientist, significance is very important. Is using P300 more pleasant than using SSVEP? Does this analysis pipeline yield a higher detection accuracy than the other? Is for this mental task the brain activity different from general baseline activity?
Generally, we compute the chance p that the second set of samples is from the same population as the first. If p is small, we reject this hypothesis (Of course there is still a chance of p that we are wrong in this). As a convention, if p is smaller than 0.05 we say the difference is statistically significant. If p is smaller than 0.10, you can speak of a trend, which generally means that you should try to do more tests. Because with more samples it is easier to get a significant difference.
There is a nice article on measuringusability.com about this convention of needing a p < 0.05 for significance, which argues not to blindly follow this convention, but look at what your results really mean. Continue reading
BrainGaming 