Lecture 01 (Continued): Statistics in the AI Era

What Changes, What Endures

Jihong Zhang, Ph.D.

Educational Statistics and Research Methods (ESRM) Program

University of Arkansas

2026-08-24

Presentation Outline

This is a working draft. The topics, examples, and relative emphasis will be revised later.

1. Part 1: AI capabilities and programming
   • What can AI do for you?
   • Do you still need to learn programming?
2. Part 2: Statistical judgment in the AI era
   • Do you still need to learn statistics?
   • Why does statistical thinking matter more than ever?
3. Part 3: Responsible practice and verification
   • How should you supervise AI-assisted statistical work?
   • Activity: Trust, but verify

Learning Objectives

By the end of this lecture, you will be able to:

1. Identify tasks for which AI can support statistical learning and practice.
2. Explain why programming literacy still matters when AI can generate code.
3. Explain why statistical knowledge is necessary for evaluating AI-generated analyses.
4. Use statistical thinking to question data, models, evidence, and conclusions.
5. Evaluate AI-generated statistical claims and analyses critically.

Part 1: AI Capabilities and Programming

Opening Question: What Can AI Do for You?

When AI can generate code, analyses, visualizations, and written interpretations, what does a statistician contribute?

• Think independently for two minutes.
• Submit your response: Opening-question Google Form.
• Discuss your answer with a partner.
• Share one idea with the class.

Statistics, Machine Learning, and AI

• Statistics focuses on learning from data while accounting for variability and uncertainty.
• Machine learning emphasizes algorithms that learn patterns and make predictions from data.
• Artificial intelligence is a broader collection of systems designed to perform tasks associated with human intelligence.

Discussion

Where do these areas overlap, and where do their goals differ?

What AI Can Help You Do

• Explain statistical concepts in different ways
• Generate, revise, and explain R code
• Help clean and reshape data
• Suggest statistical models and visualizations
• Interpret error messages
• Draft summaries, documentation, and reports
• Serve as an always-available learning assistant

AI + Data Analysis: What Is Changing?

AI can support the full data workflow

Collect \(\rightarrow\) Prepare \(\rightarrow\) Analyze \(\rightarrow\) Visualize \(\rightarrow\) Decide

• Automate repetitive preparation and reporting
• Detect patterns, anomalies, and emerging changes
• Support prediction and real-time analysis
• Enable natural-language questions, summaries, and visualizations

Analysts still must provide

• Meaningful questions and domain context
• Data-quality and measurement judgment
• Assumption checks and result validation
• Bias, privacy, and governance oversight
• Interpretation and accountability

Important

The central shift is from performing every routine task to directing, checking, and defending the analytical workflow.

Source: Databricks Staff (2026), How AI Is Transforming Data Analytics. Industry perspective used as context.

AI as an Assistant

• AI can accelerate routine work and help us explore possible solutions.
• AI can also produce incorrect code, fabricated information, and unsupported interpretations.
• A confident answer is not necessarily a correct answer.
• The analyst remains responsible for checking the work and defending the conclusion.

Important

AI lowers the cost of producing an analysis, but increases the value of judging an analysis.

Do You Still Need to Learn Programming?

The Short Answer: Yes, but the Goal Is Changing

• You may spend less time memorizing syntax.

• You still need enough programming knowledge to:

  • Read and understand generated code
  • Modify code for your own data and research question
  • Detect fabricated functions or packages
  • Debug errors and inspect intermediate results
  • Protect private or sensitive data
  • Make an analysis reproducible

Note

AI may help you write code, but you remain responsible for what the code does.

If You Do Not Understand the Code, How Can You Evaluate It?

Suppose we ask AI to estimate the mean pretest-to-posttest change and its standard error separately for two class sections.

score_data <- data.frame(
  section = rep(c("A", "B"), each = 4),
  pretest = c(70, 75, 82, 78, 68, 74, 79, 81),
  posttest = c(76, 79, 88, 83, 73, 80, 82, 87)
)

ai_summary <- score_data |>
  dplyr::mutate(change = posttest - pretest) |>
  dplyr::group_by(section) |>
  dplyr::summarise(
    mean_change = mean(change),
    se_change = sd(change) / sqrt(nrow(score_data)),
    .groups = "drop"
  )

ai_summary

# A tibble: 2 × 3
  section mean_change se_change
  <chr>         <dbl>     <dbl>
1 A              5.25     0.339
2 B              5        0.5  

Warning

The code runs and the output looks reasonable. Is the standard error calculated correctly for each section?

A Small Bug with a Meaningful Consequence

• group_by(section) creates groups of four students.
• nrow(score_data) still returns the total sample size of eight.
• The code therefore divides each group standard deviation by \(\sqrt{8}\) rather than \(\sqrt{4}\), underestimating both standard errors.

correct_summary <- score_data |>
  dplyr::mutate(change = posttest - pretest) |>
  dplyr::group_by(section) |>
  dplyr::summarise(
    mean_change = mean(change),
    se_change = sd(change) / sqrt(dplyr::n()),
    .groups = "drop"
  )

correct_summary

# A tibble: 2 × 3
  section mean_change se_change
  <chr>         <dbl>     <dbl>
1 A              5.25     0.479
2 B              5        0.707

Important

Programming knowledge helps you detect code that is syntactically valid but statistically wrong.

From Producing Syntax to Supervising a Workflow

• Before AI: Write, run, debug, and interpret the code.
• With AI: Specify, generate, inspect, test, challenge, revise, and interpret the code.
• Programming knowledge helps you distinguish a useful draft from a convincing mistake.

Warning

Faster code does not guarantee a better research question, better data, or a valid conclusion.

Part 2: Statistical Judgment in the AI Era

Do You Still Need to Learn Statistics?

The Short Answer: Yes—Perhaps More Than Before

AI can calculate statistics and generate plausible explanations, but it should not independently decide:

• Whether the research question is meaningful
• Whether variables measure the intended constructs
• Whether the data support the conclusion
• Whether model assumptions are reasonable
• Whether association is being mistaken for causation
• Whether uncertainty is adequately represented
• Whether the analysis is fair, ethical, and generalizable

Statistical Judgment Is Still Central

• What is the research question?
• How were the constructs measured?
• How were the data generated or collected?
• Which assumptions are reasonable?
• What evidence would change our conclusion?
• To which population or setting can we generalize?

Prediction Is Not Explanation

• A model may predict accurately without explaining why an outcome occurs.
• Association does not establish causation.
• A plausible explanation produced by AI is not empirical evidence.
• The purpose of the analysis should guide model choice and interpretation.

Uncertainty Does Not Disappear

• Sampling variability
• Measurement error
• Missing data
• Model uncertainty
• Distribution shift
• Human and algorithmic bias

Guiding Question

How should uncertainty be communicated when an AI system presents a single confident answer?

Why Statistical Thinking Matters More Than Ever

Six Questions for Statistical Thinking

1. Question: What exactly are we trying to learn?
2. Measurement: What do the variables actually represent?
3. Data generation: Where did the data come from?
4. Variation: How much uncertainty is present?
5. Evidence: What conclusions do the data support?
6. Validation: How do we know the result is trustworthy?

Task Execution and Statistical Judgment

Stage	AI can help with	Human judgment remains essential for
Research question	Brainstorming and refinement	Meaning, purpose, and context
Data preparation	Generating and explaining code	Data quality, measurement, and provenance
Modeling	Suggesting models and producing syntax	Appropriateness, assumptions, and design
Interpretation	Drafting explanations	Evidence, uncertainty, and causal limits
Communication	Editing and formatting	Accuracy, transparency, and accountability

A Different Prompt Can Change the Conclusion

Prompt A: Separate outcomes

Compare the intervention and control groups separately on reading, mathematics, and science scores.

• Likely analysis: three univariate ANOVAs
• Target: three outcome-specific mean differences
• Concern: multiplicity across tests

Prompt B: Joint outcome profile

Test whether the groups differ in their joint profile of correlated reading, mathematics, and science scores. Use MANOVA and plan justified follow-up analyses.

• Likely analysis: an omnibus MANOVA
• Target: the multivariate mean vector
• Uses covariance among the outcomes

Important

The data and outcomes are the same, but the hypotheses are not. Separate ANOVAs test marginal outcome differences; MANOVA tests the joint mean profile. Their inferential conclusions can differ.

Ask: Does the research question concern individual outcomes or the overall multivariate profile?

A Bridge to This Course

Throughout the semester, we will ask:

1. What multivariate structure is present in the data?
2. How can that structure be represented by a statistical model?
3. What assumptions make the representation meaningful?
4. How should we validate and interpret the result?

Part 3: Responsible Practice and Verification

Responsible AI-Assisted Statistical Practice

From Tool User to Critical Supervisor

The statistician’s role is changing—not disappearing. Working responsibly with AI requires us to direct the task, evaluate the output, and remain accountable for the result.

Common Failure Modes

• Fabricated references, data, or results
• Incorrect but convincing code
• Data leakage and overfitting
• Unexamined bias in data or models
• Disclosure of private or sensitive data
• Irreproducible analyses
• Confident interpretations that exceed the evidence

A Verification Loop

1. Specify the research question, data, and desired output.
2. Generate a draft with AI assistance when appropriate.
3. Inspect the code, assumptions, and intermediate results.
4. Verify important claims using documentation, diagnostics, and independent reasoning.
5. Revise the analysis and document human decisions.
6. Report AI assistance transparently when required.

Applying the Verification Loop

Activity: Trust, but Verify

Working in small groups, evaluate an AI-generated statistical response.

1. Identify the research question and proposed analysis.
2. List the assumptions the response makes.
3. Find claims that require verification.
4. Decide what information or diagnostics are missing.
5. Revise the response into a defensible analysis plan.

Tip

An example AI-generated analysis will be added after we select the activity dataset and statistical problem.

Closing Discussion

• Which statistical skills become more important in the AI era?
• Which routine tasks are likely to change?
• What should never be delegated without verification?
• How can AI support learning without replacing thinking?

Takeaways

1. AI can accelerate statistical learning, programming, analysis, and communication.
2. Programming is shifting from syntax memorization toward code literacy, inspection, and supervision.
3. Statistical knowledge is necessary for judging whether AI-generated work is meaningful and defensible.
4. Research design, measurement, uncertainty, validation, and interpretation remain central.
5. Productive AI use requires a deliberate cycle of generation, inspection, and verification.

Next Steps

• Refine the lecture subsections and their relative emphasis.
• Select a short case study or dataset for the audit activity.
• Decide whether to include a live AI demonstration.