Week 2 - Video 7 - Working with an AI team

Working with an AI Team

The video explains how to collaborate with an AI team on a project, using automated inspection of coffee cups as an example. The key lesson is to define measurable project goals, provide suitable data, and agree on realistic performance expectations.

Defining project requirements

• Set clear acceptance criteria before development begins. For example, require the system to detect defective coffee cups with at least 95% accuracy.
• Specify how success will be measured. AI performance is statistical, so requirements should describe how often the system should be correct rather than demand perfection.
• Work with AI specialists to determine how much data is needed to evaluate the target level of performance.

How AI teams use data

AI teams typically separate labeled examples into distinct datasets:

• Training set: Examples the algorithm uses to learn the relationship between inputs and desired outputs. In this example, the inputs are cup images and the outputs are labels such as “good” or “defective.”
• Test set: A separate collection of labeled examples used to evaluate the trained system. The team compares predictions with the labels to calculate performance. For instance, correctly classifying two out of three images gives 66.7% accuracy.
• Development, validation, or “dev” set: Some teams use a second evaluation set for technical reasons. If the team requests one, providing it is reasonable.

Training sets are usually much larger than test sets, though the appropriate sizes depend on the task.

Why 100% accuracy may not be realistic

AI systems may make mistakes because:

• The task exceeds the capabilities of current machine-learning methods.
• There is not enough training data.
• Data is dirty or mislabeled.
• Some examples are ambiguous, or experts disagree about the correct label.

Possible responses include collecting more data, correcting mislabeled examples, and aligning experts on how ambiguous cases should be classified. However, even useful AI systems may not achieve perfect accuracy. The team should agree on a performance level that meets business needs while accounting for technical limitations.

If an organization lacks an AI team, engineers with relevant online training may be able to make a confident attempt at implementing a project.

Speakers and sources

• Speaker: One unnamed course instructor.
• Sources or other speakers: None featured in the subtitles.