Tema 4 - Evaluación y retroalimentación en los albores de la IA - 1ra Parte

Main Ideas and Concepts (Topic 4: Evaluation and Feedback in the Dawn of AI — Part 1)

1) Unit purpose and four subtopics

The video introduces Unit Four, focused on evaluation and feedback using AI, organized into four topics:

• Automatic evaluation using the design of PROM for formative evaluation
• Plagiarism/originality detection and related issues
• Academic integrity
• Ethical integrity in the age of AI

---

2) Two types of assessment: summative vs. formative

The video distinguishes between two major assessment types:

Summative assessment

• Happens at the end of a learning process
• Used to determine whether learning occurred
• Focuses on results and final certification
• Example: AI can generate final exams or question sets, including final cases/projects where students apply what they learned

Formative assessment

• Happens during the course (e.g., throughout the semester)
• Focuses on progress and improvement
• Example: AI provides intelligent tutoring (more than a simple chatbot)
  • Often implemented within an LMS/platform
  • Supports predictive analysis
  • Provides immediate feedback so students can adjust while learning
  • Can also alert teachers when students are falling behind

Key lesson: AI’s strongest value is in formative assessment—because immediate feedback helps avoid discovering obstacles only after final evaluation.

---

3) Three types of automatic evaluation with AI

The speaker outlines three kinds of automatic evaluation:

A) Correction systems for closed questions

• Considered robust and fast
• Built on established approaches (e.g., question banks, question–answer associations)
• Important caveat: vulnerable to AI/chatbot-assisted cheating
  • Example scenario: a modern browser/platform may include built-in “agents”
  • A student can view the test and ask an assistant for answers without switching tabs
  • If OCR/text capture is needed to feed the assistant the questions, detection may change—but the risk remains

B) Automatic/text evaluation using NLP (Natural Language Processing)

• Analyzes student-written texts
• Can assess multiple dimensions, such as:
  • Coherence (how ideas connect)
  • Cohesion (linking mechanisms across sentences)
  • Lexical complexity
  • Argumentative structure
  • Use of sources
    • Whether sources are provided or searched externally
• Warning: NLP tools should be support, not a full replacement for professor judgment
  • Relying only on one tool may introduce biases and other risks

C) Learning analytics / predictive feedback

• Uses data about learner behavior on a platform, such as:
  • Connection time
  • Frequency of access
  • Response patterns
  • Sequences of activities
• Builds a detailed student profile, for example:
  • Time spent watching videos
  • Time spent reading
  • Time spent working on question generation
  • A “matrix” of activity to infer progress patterns
• Example risk pattern: all exams completed in one day
• Emphasis attributed to Siemens and Long (2011):
  • Learning analytics is not about the past; it’s about the future
  • Its value is in detecting weak signals early to prevent irreversible failures
  • Teachers should have time to intervene, advise, inspire, and guide

---

4) Free/open-source and other platforms for automatic evaluation

The video mentions multiple tools (some open-source, some paid), including:

• Moodle Quiz + Feedback (open source within Moodle)
• H5P + question set/quiz option (open source under MIT license)
• Open Assessment (open source; described as having a different learning curve)
• Quiz Quizis
  • Limited/free option and a premium paid version
• Google Forms + Google Sheets
  • May use JavaScript/automation logic to check correctness without costly AI tools
• PG (described as premium, with free options and paid options)

---

PROM for Formative Assessment (Instructions + Detailed Examples)

Method: use PROM to generate learning activities and evaluation instruments

The video connects PROM design directly to formative assessment:

• The human (teacher) gives the verdict, not the machine
• Tools may generate outputs, but professors review criteria, contributions, and conclusions
• Because biases can exist, oversight is necessary

---

Example PROM #1: Generate Socratic questions (critical thinking)

Prompt role example

• “You are a facilitator of critical thinking in a graduate business administration program.”

Input

• The student submits an analysis of a business case (text attached/pasted)

Instruction / output requirements

• Generate exactly five Socratic questions
• The questions must:
  • A. Identify the unquestioned assumptions in the student’s analysis
  • B. Explore alternative perspectives the student did not consider
  • C. Challenge the strength of their evidence
• Constraints:
  • Must be open-ended, not theoretical
  • Must be ordered from least to most cognitively complex
• Evaluation rule:
  • Do not answer the questions
  • Do not evaluate whether the analysis is correct or incorrect

Human oversight

• The teacher reviews the student’s analysis and the generated feedback items/questions

---

Example PROM #2: Generate an assessment rubric + feedback (nursing competencies)

Prompt role example

• “You are an expert evaluator of clinical nursing competencies at the university level.”

Input

• The student presents a nursing care plan

Instruction / output requirements

• Evaluate using a four-criteria rubric:

1. Patient assessment
   • Evaluate: complexity, completeness, (subtitle wording noted as unclear/“pardon”), and accuracy
2. Nursing diagnoses
   • Evaluate: relevance and prioritization
3. Interventions
   • Must be evidence-based and specific
4. Evaluation of results
   • Evaluate measurable indicators

Feedback format requirements (per criterion)

• For each criterion, provide feedback in two sentences:
  • Descriptive: what the student did
  • Prescriptive: what the student should do to improve

Ending element

• Include an overall assessment/strength of the work
• The teacher synthesizes and verifies alignment with rubric criteria and the student’s actual content

Human oversight emphasis

• The teacher reviews criteria and contributions; the machine assists but does not finalize the judgment

---

Additional Lessons / Good-Practice Considerations

1. Assessing soft skills via interaction

   • Students may claim work was done with a bot/assistant
   • If assessment is only the final document, authorship/process are hard to verify
   • Better approach:
     • Use interaction where students explain their work
     • Through the learning process, students should show structure and evolution of their thinking
     • This provides evidence they’re building understanding—not only submitting text

2. Teacher involvement reduces AI bias

   • Avoid letting AI grade alone
   • Teacher involvement helps prevent bias and false positives
   • Teachers can contextualize guidance:
     • Inspire and guide knowledge in a local context
     • Ensure evaluation happens within an ethical context (linked to an ethics module mentioned)

3. Practical requirement

   • This interactive and ethical approach requires:
     • Time
     • Patience
     • Sometimes mastery of digital tools
     • Possibly synchronous online sessions to evaluate learning

---

Speakers / Sources Featured

• Victor Cuchillac (main speaker/host)
• Siemens and Long (2011) (source referenced for learning analytics principles)