Sign in
x
☰

Stats tests for “statistical significance”

Updated at 2020-02-12 18:40:41 UTC
Protobi toolbar showing hamburger menu icon (circled in red) and collapse/expand arrow controls for managing element visibility in the tree structure.

Protobi automatically runs stats tests to identify  “statistically significant” differences in the standard chart types:

  1. Baseline bar charts comparing the current scenario to the baseline scenario
  2. Crosstabs  compare each column to either the Overall column (default) or to each other column (pairwise option)

Current vs Baseline

When you press to query (e.g. "Excellent" in q2) Protobi shows results for just those respondents. The percentages and solid color bars reflect the current scenario, (e.g. only those in "Excellent" health). The thin black outlines reflect the baseline scenario (all respondents).

Two survey questions displayed side by side: Q.1 asking about happiness levels (Very happy 49.6%, Pretty happy 42.7%, Not too happy 6.0%) and Q.2 about health ratings showing 100% Excellent response, both with N=637.

Protobi shows blue triangles wherever the current scenario is significantly different from baseline. Here, the outward pointing triangle indicates that 49.6% is significantly higher than the baseline of 30.5%. Inward pointing triangles indicate the value for the current scenario is significantly lower than the value for the baseline scenario.

Question Q.1 about happiness levels with a blue tooltip displaying comparison values: current percentage of 49.6% versus baseline of 30.5% for the 'Very happy' response category.

Set current filters as baseline

If you want to make a strict comparison between non-overlapping groups, you can change the baseline scenario. Protobi allows you to set current filters as the base instead of all respondents (initial base).

Press the toolbar button "Set base" to make the baseline scenario equal to the current scenario (e.g. respondents in "Excellent" health). 

Protobi toolbar showing multiple action buttons with 'Set base' button highlighted by a red border, indicating the control for setting baseline comparisons. Other visible buttons include sample size (N=637), Clear, percentage toggle, Format, missing data options, Crosstab, Export, Save, and Scenarios. Side-by-side display of Q.1 happiness question (Very happy 49.6%, Pretty happy 42.7%) and Q.2 health rating question showing standard frequency distributions without baseline comparison indicators, both with blue bar charts and N=637.

Now shift+press on "Excellent" to select those respondents who are NOT in "Excellent" health.

Q.1 and Q.2 displayed side-by-side with blue triangular arrows next to frequency values, showing significance testing results comparing current filtered data (N=637) to baseline (N=1874) for happiness and health questions.

We can see above, we're now running a strong comparison between two distinct groups. The groups being those who are NOT in "Excellent" health (current scenario, solid bars) versus those who ARE in "Excellent" health (the baseline scenario, thin black outline).

Crosstabs

Crosstabs in Protobi use either Pairwise or Complement significance testing. Pairwise testing compares each column individually with each other column. Complement testing compares each column with the average of all other columns.

Descriptive statistical analysis (means, frequencies) and tests of differences (chi square, t-tests) within respondent types will be performed. Statistical significance will be set at p <0.05 by default, using 2-tailed tests.

Admins can change crosstab significance tests in Project properties. The default mode is Complement testing.

Project properties configuration dialog displaying significance testing settings, with 'P-value for sig tests' set to 0.05, 'Toolbar position' set to 'top', and 'Crosstab significance tests' dropdown menu showing 'Complement' selected with checkmark.

Complement

In Complement testing, each column is compared with the average of all other columns (excluding itself). Blue cells indicate the value is significantly higher in the specified column compared to all other columns, and grey indicates significantly lower. 

Note: If showOverall is set to false on a question, there is no Overall column to compare against. So it falls back to pairwise. If you'd like to compare each column to the Overall distribution, set showOverall to true.

Crosstab table displaying 'q2 by q1' comparing health ratings across happiness levels, showing percentage distributions across columns. The table displays Chi Square: 416.9 (p = 0.000) at the bottom, indicating a highly significant relationship between the two variables with N=2511.

Pairwise

Pairwise testing compares each column with each other individual column. This mode shows detailed superscripts like a traditional crosstab. 

Below we see that the percentage of respondents who rated their health as "Excellent" is much higher in column A (Very happy) than columns B, C or D. 

Crosstab table showing health by happiness with column letters (A, B, C, D) labeling each happiness category and blue letter codes (like 'BCD', 'C', 'AC', etc.) indicating which columns each cell's percentage significantly differs from, using complement testing approach. Chi Square: 416.9 (p = 0.000).

P-value

To change the P-value for significance tests open the Project properties dialog. The default value is 0.05.

Note: Protobi limits significance testing to N>=10 to avoid testing when the sample size is too small. 

Project properties configuration dialog with 'P-value for sig tests' dropdown menu expanded, displaying three significance level options: 0.10, 0.05 (currently selected with checkmark), and 0.01. The 'Crosstab significance tests' field below shows 'Complement' setting.