Architecture Design and Evaluation 5

Main ideas / lessons

• Purpose of architecture evaluation

  • Architecture evaluation is meant to assess how good an architecture design is in achieving the architecture drivers (requirements that guide design).
  • It helps detect design defects early in the software development lifecycle.
  • It supports identifying and managing risks, and it helps build agreement (consensus) among stakeholders before detailed design/implementation/test begins.

• Why evaluation matters

  • Early evaluation can reveal:
    • Trade-offs among quality attributes (e.g., improving one may harm another).
    • Ambiguities or contending goals in requirements.
    • Risks that can threaten project success.
  • Evaluation produces advantages such as:
    • Shared understanding among architects, evaluation teams, customers, project managers, and other stakeholders.
    • Clearer goals/requirements and fewer misunderstandings after release.

• Constraint on evaluation

  • Evaluation is often limited by time and budget, so not all scenarios can be checked in depth.
  • Therefore, evaluation should use prioritization to decide what to analyze first.

Methodology presented: ATAM-style Architecture Evaluation (qualitative)

The video targets the ATAM (Architecture Tradeoff Analysis Method) as a representative qualitative, scenario-based evaluation approach.

Inputs to the evaluation process (ATAM overview)

1. Requirements (scenario-based)

   • Quality attribute requirements must be expressed as scenarios (including conditions and measurable expectations).
   • These scenarios come from quality requirements tied to business drivers/architecture drivers.
   • The method also takes in the architecture as an artifact: e.g., selected components/decisions such as reference architectures, mapping tactics, and applied design techniques.

2. Architecture

   • The evaluation uses the current architecture design and captured decisions/approaches.
   • Architecture decisions are documented enough that the team can reason about how they satisfy scenarios.

Outputs produced by the evaluation

• Non-risk conclusions

  • Determinations that certain architecture decisions do not pose a risk for a scenario.
  • Must include the reasoning behind the conclusion.

• Risks

  • If architecture decisions negatively affect scenario fulfillment in critical ways, they become risks likely to cause project failure or inability to satisfy requirements.
  • Risks trigger the need for reconsideration/revision of requirements/architecture.

• Sensitivity points

  • A point where small changes (or differing conditions) can cause significant quality attribute impact.
  • Often tied to assumptions/conditions in design decisions.

• Trade-offs

  • An architecture decision that:
    • improves one quality attribute but
    • harms another quality attribute (or has negative side effects under certain conditions).

• Documentation of conclusions

  • For each scenario, evaluation results must be captured, including:
    • whether it is risk or non-risk
    • the rationale
    • relevant sensitivity and trade-off reasoning

Evaluation phases / steps (process overview)

• Overall structure
  • Typically conducted in about 3 days in the course’s framing.
  • Includes:
    1. Presentation phase
    2. Business drivers / requirement context sharing
    3. Architecture walkthrough / approach capture
    4. Quality attribute scenario organization + prioritization
    5. Analysis
    6. Testing / second iteration (where needed)
    7. Final reporting

Detailed bullet list: concrete step flow (as described)

• 1) Presentation phase (make common understanding)

  • The evaluation team presents the ATAM process to assemble stakeholders.
  • Output of the process presentation should be understood by:
    • evaluation team
    • architect / architecture team
    • customer representative
    • project manager
    • other stakeholders
  • The output of evaluation should include explicit handling of:
    • risks
    • non-risks
    • sensitivity points
    • trade-offs

• 2) Present the business drivers

  • Shared by people familiar with business context (e.g., project managers or customer reps).
  • Emphasis: the most important functional requirements from the business view.

• 3) Identify and list architecture approaches / map architecture

  • The architecture is analyzed to identify:
    • design techniques
    • structures and patterns
    • reference architecture usage
    • tactics / architectural approaches mapped to quality requirements
  • The team’s first job in this step is often not deep “evaluation,” but capturing what approaches exist in the current architecture.

• 4) Organize and prioritize quality attribute scenarios

  • Scenarios are structured into categories (using quality attribute categories such as performance, security, availability, modifiability, etc., as defined by the project).
  • Add dimensions for scenario attributes, including:
    • importance: high / medium / low (from business view)
    • difficulty: high / medium / low (technical realization difficulty)
  • Prioritize because time is limited:
    • evaluate most important + most difficult scenarios first
    • avoid the pitfall of “relative ranking that doesn’t match real business importance” (i.e., ensure prioritization reflects the business).

• 5) Scenario-by-scenario analysis

  • For each selected scenario, the evaluation team focuses on:
    • sensitivity points
    • trade-offs
    • whether negative effects are:
      • risk (critical) or
      • non-risk (acceptable/not critical)
  • If unclear, the team seeks input from:
    • architecture/technical people (solution/design assumptions)
    • business/customer people (business conditions and intent)

• 6) Define sensitivity points

  • Example given conceptually:
    • A decision (like caching) can help under one condition and hurt under another (e.g., data freshness / consistency).
  • The key idea: sensitivity depends on fit conditions and assumptions; evaluation should test “when does it work vs when does it fail.”

• 7) Identify trade-offs

  • Example given conceptually:
    • Encryption may improve security (positive) but degrade performance/latency/resource usage (negative).
  • Trade-offs must be explicitly documented as side effects and can easily be missed if not deliberately searched for.

• 8) Document conclusions properly

  • For each scenario, the evaluation record should include:
    • what the decision/approach is
    • how it affects the scenario
    • whether the outcome is risk or non-risk
    • the rationale and any conditions (including why negative effects are or aren’t critical)

• 9) Testing phase / second iteration (if needed)

  • After “analysis,” the course frames a “testing” phase as a second pass:
    • especially when scenarios/quality attribute drivers were not fully identified initially
    • or when missing scenarios are discovered by stakeholder brainstorming
  • The second iteration:
    • adds missing scenarios
    • re-runs analysis for them
    • updates risk/non-risk decisions, sensitivity points, and trade-offs

• 10) Final reporting

  • Summarize:
    • architecture decisions and approaches
    • scenarios identified
    • risks and non-risks
    • sensitivity points and trade-offs
    • rationale behind each conclusion

Quantitative vs qualitative evaluation (contrast)

• Quantitative evaluation

  • Uses numbers/metrics and objective judgments.
  • Requires clear criteria and sufficiently detailed architecture information (e.g., performance parameters to compute end-to-end latency).

• Qualitative evaluation

  • Uses scenario-based judgment with more subjective assessment.
  • Often depends on experience of the architect and evaluation team.
  • ATAM is presented as this qualitative, scenario-driven approach.

Named sources / speakers

Speakers

• No individual speaker is clearly identified in the subtitles (the text does not contain names).

Sources / named methods / organizations

• ATAM (Architecture Tradeoff Analysis Method)
• Software Engineering Institute (SEI) (at Carnegie Mellon University, referenced as “software engineering institute of universities” in the subtitles)
• Case study referenced: Boeing system architecture evaluation using ATAM
• Report referenced: “famous report” (the subtitle does not provide a specific title beyond the Boeing/ATAM mention)
• Earlier course references: “first class” / “previous classes” (no specific lecture titles named)