Validity

Main ideas and lessons

• Validity is central to language testing and applied linguistics: It’s not mainly about how a test looks (e.g., “face validity”), but about whether the scores actually mean what we claim they mean.

• Validity asks: “How do we know the score supports the interpretation and decisions we make?” This is done through collecting and evaluating different kinds of evidence.

• The course reframes validity as “aspects of validity” (often called the “traditional Trinity”): approaches for building confidence in score meaning.

• Validity relies on an inferential chain:
  1. Define an abstract construct (what we claim to measure).
  2. Collect evidence relevant to that construct.
  3. Make an inference from evidence to the construct.
  4. Use the inference to support a decision/action (e.g., pass/fail, university admission, professional capability).
• Constructs are abstract and not directly observable (e.g., “love,” “fluency”). Therefore, validity depends on whether the collected data truly supports inferences about the construct.
• The lecture emphasizes warranted assertions: decisions should be supported strongly enough by evidence to justify reasonable inferences.
• A quoted/credited definition (by Messick) frames validity as an integrated evaluative judgment based on:
  • evidence,
  • theory,
  • adequacy/appropriateness of inferences,
  • and the consequences/decisions that follow.

---

Methodology / key instructional content (detailed)

A) “Traditional Trinity” / Aspects of validity: what counts as evidence?

1) Criterion-related validity evidence

Concurrent criterion evidence

• Meaning: The criterion is measured at the same time as the test.
• Example method (reading test for Applied Linguistics program skills):
  • Identify critical skills (e.g., skimming and scanning for key information).
  • Build two different tests covering those skills (different content/questions).
  • Administer both tests to the same group on the same day.
  • Use counterbalancing to avoid order effects:
    • Half take Test 1 first, half take Test 2 first.
  • Compare whether students get roughly the same scores and similar rankings across both tests.
• Interpretation: Similar results across tests support that the test scores have the intended meaning.

Predictive criterion evidence

• Meaning: The criterion is measured in the future to predict later performance.
• Example method (IELTS → university performance):
  • Use IELTS scores (possibly broken down by skills) as the original measurement.
  • Later, at university, collect criterion indicators such as:
    • content teachers’ evaluations of writing success in the domain, or
    • end-of-first-semester module scores.
  • Test the relationship by checking correlation between IELTS and university performance.
• Limitation noted: “Truncated sample”
  • People who score poorly on IELTS may not enter university, so the full range of ability isn’t observed.
  • A “perfect” predictive study would include everyone (including failures), but this rarely happens.
• Interpretation: Despite practical difficulty, predictive studies are essential for future-oriented decisions.

---

2) Content validity evidence

• Meaning: The test content is relevant to the domain where the score will be used.
• Example method (medical English test for hospital work):
  • Ensure reading tasks mirror the kinds of reading done by health professionals in hospitals.
  • Ensure speaking tasks reflect real communication needs:
    • communicating with patients,
    • communicating with colleagues, etc.
  • Ensure topics match the learners’ real professional work.
• Two main approaches described:
  • Expert review
    • Show proposed test content to hospital doctors/nurses (or other professional experts) and collect judgments on appropriateness.
  • Domain analysis
    • Collect authentic domain materials (texts) and mirror them in the test, or
    • Record real interactions (e.g., patient–nurse interactions) and identify language/discourse patterns to design tasks.

---

3) Construct validity evidence

• Meaning: Evidence that test scores can support inferences about an abstract construct.
• Core issue: Constructs are unobservable (e.g., “love,” “fluency”), so validity depends on whether we define:
  • the construct clearly enough to measure it, and
  • the right kinds of evidence.
• Illustration using “fluency”
  • We can’t point to “fluency” like a physical object.
  • We must specify what observable behaviors indicate fluent vs. non-fluent speech.
  • Example argument: pauses may reflect processing/planning, not lack of fluency—so pauses may not be appropriate evidence by themselves.
• Research logic described via hypotheses
  • Data → hypothesis → inference → decision
  • Null hypothesis (H0): no relationship between data and the construct
    • If true, we can’t make the inference, so we lack validity evidence.
  • Alternative hypothesis (H1): there is a relationship between data and the construct
    • Evidence then supports inferences about the construct.
• Construct theory and model-building (“normal logical net”)
  • Validity is tied to theory: constructs relate via assumptions (e.g., classroom participation depends on constructs like attitude, motivation, aptitude).
  • Defining constructs enables measuring them and testing relationships.
• Dispelling doubt + warranted assertions
  • Validity questions whether we truly know what we’re doing: whether evidence and interpretation match.
  • Warranted assertions are claims supported enough by evidence for sound decision-making.

---

B) Messick’s definition of validity (as explained)

• Validity = an integrated evaluative judgment
• The judgment integrates:
  • Degree to which there is empirical evidence supporting interpretations
    • Evidence includes:
      • criterion evidence,
      • evidence about construct measurability/relationships,
      • test-score results and supporting rationales.
  • Theoretical rationales that generate/justify the evidence
    • e.g., competing accounts of how accuracy and fluency relate (may be at odds, or may develop together after a threshold).
  • Adequacy and appropriateness of inferences
    • whether inferences are sound and suitable to the context.
  • Consequences / actions
    • decisions based on the scores and how those decisions affect lives (e.g., admissions, professional approval).

---

C) Major threats to validity (how inferences/decisions can be undermined)

1) Bias (threat category)

• Question: Could scores be biased, leading to unsound inferences about individuals?
• Commonly studied bias types mentioned:
  • Gender
    • Some items/tasks may systematically favor boys or girls.
    • Checked by analyzing performance across item types under assumptions of comparable ability distributions.
    • Example: a football-themed classroom quiz may engage boys more, skewing scores.
  • Nationality / L1 background variety effects
    • International tests must avoid penalizing acceptable varieties of English.
    • Example logic: grammar differences between local/national English varieties and “standard” varieties can cause unfair penalization.
    • World Englishes issue discussed (e.g., Malaysian English patterns affecting international test performance).
  • L1 background (language background)
    • Items might advantage speakers of certain first languages (e.g., Romance vs. Asian languages).
  • Disability
    • Use Universal Design so disability-related factors don’t unfairly lower scores.
    • Example: on computer reading tests, allow increased font size for visual impairment.

2) Test method effects (threat category)

• Question: Is the score influenced by test format rather than the intended construct?
• Examples described:
  • Item types
    • closed-response (e.g., multiple choice) vs open-response (writing)
    • Concern: reading scores shouldn’t depend on writing ability.
  • Test wiseness
    • Some examinees may score higher due to knowing how to respond (not because of language ability).
    • Example cue: in multiple choice, the correct option may be identifiable because it’s longer.
    • Test preparation may train these strategies, inflating scores.
  • Delivery features
    • Computer/UI design may affect performance:
      • color schemes,
      • navigation difficulty,
      • where arrows are placed,
      • readability of text.
    • The lecture distinguishes non-functional design features from delivery features that affect navigation/readability.
  • Administration
    • Physical/organizational conditions can affect scores unrelated to ability:
      • distractions,
      • seating changes,
      • incorrect equipment/air conditioning,
      • invigilator behavior and how it affects learner mood/conditions.
• Key consequence stated:
  • If these threats distort scores, they undermine construct validity because score differences reflect other factors, not the intended inferences.

---

Main speakers/sources featured (at end)

• Professor Bradsky (speaker in a referenced video/lecture; provides a traditional introduction to validity)
• Messick (author whose definition of validity is quoted: validity as an integrated evaluative judgment)
• Dewey (referenced for the concept of “warranted assertion”)