Die Pisa-Test Katastrophe. Wer prüft die Prüfer?

Main ideas / lessons conveyed

• The “PISA patent test catastrophe” (2025, natural sciences) is mentioned as context, but the speaker does not focus on scores. Instead, they argue that an under-discussed problem is the nature and content of the tasks.
• The speaker claims that in PISA-like natural science assessments, the exam/task content has changed over time:
  • Difficulty/competency demands stay relatively similar (at least within a defined framework).
  • Ideology levels increase over time, moving from mostly neutral contexts toward explicit value judgments embedded in tasks.
• They suggest that translation/coding issues and ideological framing may distort what is actually being tested (e.g., “hemisphere” confusion).
• The speaker proposes that modern task design may increasingly conflate natural-science methods with normative/political evaluation, which they criticize as a move away from a “value-free” scientific stance.
• They also emphasize that because AI can solve classic/mechanistic tasks quickly, modern assessments must evaluate what AI actually does—shifting the focus toward process/data credibility/analysis rather than simple computation.

---

Demonstration “game”: example task types and what they illustrate

The speaker uses example tasks to show where:

1) the technical physics/science content is, 2) the context/preamble is, and 3) whether the task implicitly or explicitly pushes values/ideology.

A) Energy / pumped-storage power plant (physics/engineering)

• Preamble (short text):
  • Energy supply is required for industrial development and improved supply for the population.
  • Alongside large power plants, pumped storage power plants help achieve this goal.
• Tasks:
  • Explain the economic importance of a pumped storage power plant.
  • Name the energy conversions during water circulation.
  • Calculation: Determine the power output (in kW) if:
    • Between 7:00 pm and 10:30 pm, 250,000 m³ of water flows evenly
    • The water moves from the reservoir 120 m higher
    • Efficiency is 0.86
    • (Task includes an added formula.)

B) District heating / thermodynamics (heat transfer)

• Preamble: District heating for many new apartments.
• Tasks:
  • If an apartment needs 1200 kilocalories per hour on a winter day:
    • Compute what volume of water must flow through radiators if water temperature drops from 80°C to 68°C.
  • Supplement: Why is water particularly suitable as a circulating medium?
    • Compare specific heat from reference tables with other liquids.

C) Uniformly accelerated motion / car test-drive (kinematics)

• Setup: Data from test drive of a passenger car (type unspecified in subtitle).
• Tasks:
  • Use given tabular data (time in seconds; speed in km/h to the power of −1 per subtitle) to determine what it implies about the unit/speed representation.
  • Use a table of gear intervals (start/end times for each gear; up to 4th gear).
  • Draw: the velocity–time diagram for the entire motion sequence.
  • Explain: physical quantity acceleration.
  • Additions: include phases like:
    • Car reaches top speed, maintains it for 10 seconds, then slows down.
  • Last task: Explain the relationship between braking distance and road conditions under different weather.
  • Conclusion: What should road users infer?

D) Daylight / Earth geometry (astronomy/concepts)

• Preamble (short reading text): shortest day in Melbourne on June 22, 2002; includes sunrise/sunset times and location latitude (~38° south).
• Tasks:
  1. Which statement explains why there is daylight and darkness on Earth?
     • Options include Earth rotation, Sun rotation, axial tilt, Earth’s orbit.
  2. Given a diagram with parallel rays from the “infinitely distant sun”:
     • Assuming the shortest day in Melbourne:
       • Draw Earth’s axis, label northern hemisphere, southern hemisphere, and equator.

E) Cloning (reading + conceptual questions, including scientific evaluation)

• Preamble: A short newspaper-style article about Dolly the sheep and cloning.
• Tasks:
  1. Which sheep is Dolly identical to? (Requires reading the text for named sheep.)
  2. A “very small piece” taken from the udder is described—choose what it is:
     • cell / gene / nucleus / chromosome.
  3. Evaluate reasons given for governments wanting to prohibit human cloning:
     • For each reason: decide whether it is scientifically sound (Yes/No), including:
       • “Cloned humans might be more susceptible to certain diseases than normal humans.”
       • “Humans should not assume the role of creator.”

F) Fast fashion / sustainable cotton (source reliability)

• Preamble: Concerns about environmental impact of clothing production/disposal; argues cotton vs synthetic fibers and need for sustainable production methods.
• Tasks:
  • Determine which source is most reliable.
  • Justify why the chosen source is more reliable than others.
• Example answer options (as described):
  • Reviewed by other experts.
  • Compares results of a scientific study over time.
  • Quote contains a definition.
  • Well-written / author expert at analyzing trends in data.

G) Offshore wind farm planning (trade-off + zoning decision)

• Preamble: A coastal town plans an offshore wind farm; some support it, others worry about environmental impacts.
• Diagram: seabed and zones.
• Task:
  • Choose which zone turbines should be built in for the best compromise between financial costs and environmental impact.

H) “Scientists and their work” / who should we believe? (media literacy)

• Preamble: Facebook article claims vaccines are dangerous.
• Tasks:
  • Choose which reasons make you doubt the article’s accuracy (check all that apply), including options such as:
    • Published online in a journal.
    • Not reviewed by scientists / not peer-reviewed.
    • Not conforming to scientific consensus.
    • Author is a nuclear physics specialist.
    • Author says she does not vaccinate her own children.
    • Contradicts the majority of scientific consensus.
  • Speaker notes the topic appears in German and English versions with observed discrepancies.

---

Claimed origins of the showcased tasks

• East German (GDR) exam questions for the 10th grade (about age ~15).
• Then PISA examples from different years:
  • One set referenced around 2003
  • And the most recent ones referenced as 2025
• Tasks are presented as chronologically sorted to compare changes over time.

---

Core argument: what changed over time

1) Difficulty / competency levels (speaker claims they stay bounded)

Using a competency framework, the speaker claims:

• Most tasks cluster around competency levels 3–4.
• They do not exceed 4 in their analysis.
• The distribution is fairly consistent even as topics/context change.

Speaker’s mapping of competency ideas (as described):

• Level 3: one-step calculations (or similar)
• Level 4: multi-step calculations / evaluate diagrams / explain models
• Higher levels (in the general framework):
  • planning experiments
  • justifying
  • up to a top “transfer to unknown context” around level 6

Interpretation offered:

• Even though contexts differ across decades, tasks remain relatively “mechanistic” in how students can succeed (memorize/compute rather than deeply understand).

2) Ideology increases (speaker claims it rises sharply)

The speaker introduces a codebook for “ideology” categories labeled 0 to 3:

• 0: no ideology; neutral technical context
• 1: societal/normative framing without direct value judgment
• 2: value judgments embedded in the task (e.g., environmental measures, vaccine skepticism treated as wrong/assumed)
• 3: explicitly political (example: defense-related political framing hidden behind physics)

Speaker’s claimed pattern:

• In GDR tasks, the “ideology” bar is small.
• In PISA over time, ideology bars increase drastically, moving into higher categories.

Important nuance the speaker claims to observe:

• In GDR tasks, ideology tends to be added as a separate pre-political statement, while the technical work remains “pure physics.”
• In later PISA-style tasks, ideology is more embedded in what counts as the correct answer (i.e., grading depends partly on the preferred evaluation direction like consensus or sustainability).

---

Methodology described by the speaker (detailed steps)

• Goal: Analyze PISA-like science tasks for:
  • competency level/difficulty (how students can solve them),
  • and ideological embedding (whether values influence what is “correct”).
• Step 1: Build a codebook
  • Create an ideology classification scheme with categories 0–3 (neutral → normative → value-judgment → explicitly political).
  • Define what belongs in each category (what counts as ideology vs neutral scientific framing).
• Step 2: Use AI-assisted coding instead of human-only coding
  • Previously:
    • the team devised codebooks and had students/self/code raters manually code tasks.
  • Now:
    • an AI is used to code tasks according to the codebook.
• Step 3: Competency framework alignment
  • Assign each task a competency level using a structured competency model:
    • from simple recall/name/remember,
    • to arithmetic,
    • to planning and justifying,
    • up to transfer to unknown contexts.
  • Speaker claims AI can look up competency frameworks when they exist.
• Step 4: Compare task distributions across time
  • Categorize tasks chronologically (GDR → PISA examples across years including 2025).
  • Compare:
    • orange bars (competency levels) → claimed “stable around 3–4”
    • blue bars (ideology levels) → claimed “rising over time”
• Step 5: Interpret results with attention to translation errors
  • Example claimed issue:
    • English “hemisphere” concept translated awkwardly in German, potentially harming comprehension and not testing the intended science concept.

---

Evaluation/assessment stance (speaker’s view of how to evaluate modern tasks)

Because AI can solve classic textbook/graph problems extremely fast, assessment must consider:

• whether AI outputs are credible,
• whether data collection/analysis is reliable,
• and whether the task is evaluating process/data-credibility competencies rather than only computation.

The speaker argues that embedding ideological judgments into correctness is a major problem:

• Natural science is largely “value-free” in principle.
• Mixing values/evaluation criteria into science questions pushes away from neutral scientific enlightenment and toward “substitute religion” / scientism.

Speaker’s claimed conclusions are two-part:

1. competency/difficulty shifts
2. ideology increases more problematically

---

Speakers / sources featured

Speakers

• Primary speaker/host (unnamed in subtitles): discusses PISA 2025 natural sciences, GDR vs PISA task comparisons, introduces the codebook and AI-assisted analysis, and critiques ideology in science tasks.

Sources mentioned (not as speakers)

• PISA studies (various years; examples from ~2003 and 2025 are referenced; plus “PISA 2000”).
• GDR (East German) exam questions for about 10th grade.
• Facebook article claiming vaccines are dangerous (used as an example text/source).
• Peer review / scientific consensus (described as criteria in the “who should we believe?” task).
• Dolly the sheep (referenced as a subject in a cloning newspaper-style article).
• Competency frameworks (as an external framework the speaker references).
• “Scientists and their work” / “Who should we believe?” (task wording appears in both German and English versions, as described by the speaker).