Video summary

DSAI HDA AVS Tech Skills 3: The Art of Data Hunting Finding the Right Data for AI & Data Science

Main summary

Key takeaways

Educational

Main Ideas & Lessons (Data Hunting for AI / Data Science)

1) What “Data” is and why it matters

  • Data = raw facts/figures/information collected for later analysis.
  • Data becomes information after it is analyzed/processed.
  • Use cases depend on the problem. For example:
    • Stock prediction: trends alone may not be enough—sentiment can cause sudden drops.

Key reminder: before talking about “how to find data,” you must understand:

  • what data is, and
  • what it will be used for.

2) The “art” of data hunting (treat it like hunting)

Data hunting = finding the right data for the intended AI/data science project.

The hunter mindset:

  • Know what you need to look for.
  • Know where to get it and how to retrieve it properly.
  • Check whether the data is fact vs. hoax/assumption.
  • Understand the nature of the data (e.g., its certainty level).

3) “Correct data beats lots of data”

Core message: prioritize accuracy/precision over volume.

Too much can harm performance because it may include:

  • noise
  • duplicates
  • irrelevant/incorrect records

Ideally, you want both:

  • a large enough dataset, and
  • high-quality/accurate data.

Metaphors used:

  • Data as “garbage”: useless if not processed, valuable if properly utilized.
  • Data as “oil”: raw data has low value; processed data becomes information that supports decisions/prediction.

4) Essential skills for collecting and preparing data

The speaker emphasizes practical capabilities needed for data hunting:

  • Business understanding

    • Learn the business context behind the problem.
    • Example signals mentioned:
      • seasonality/session-based patterns in stocks or other domains
      • domain-specific variables (e.g., location, product type, weather context)

  • Problem solving

    • The biggest issue when obtaining data is often dirty data.
    • Use data profiling & assessment to find:
      • missing values
      • duplicates
      • outliers

  • Data processing / cleaning workflow

    • Handle missing values and duplicates.
    • Clean and transform data before analysis/modeling.

  • Querying data (SQL)

    • Know how to query the right tables and columns.
    • Data hunting includes knowing:
      • what to select
      • how to filter
      • where the data lives structurally

  • Using APIs

    • Retrieve internal/external data.
    • Example: weather data via an API; also internal data in a company.

  • Web scraping

    • Collect data from websites.
    • Mentioned tools/approaches:
      • Selenium
      • BeautifulSoup
    • Emphasis: scraping should be ethical/appropriate.

  • Ethical data awareness

    • Avoid datasets collected without transparency about purpose.
    • Ethical issues can lead to legal trouble.

Detailed Methodology / Step-by-Step Guidance (as Presented)

A) Data hunting procedure (conceptual steps)

  • Step 1: Understand the problem first
    • Identify business objectives and the use case.
  • Step 2: Determine what “features” you need
    • Decide which variables support the objective.
    • Examples:
      • House price prediction: historical price, location, number/layout of rooms, etc.
      • Sentiment analysis: customer review text and/or rating features
  • Step 3: Adjust data to the problem type
    • Consider what kind of ML task is needed (e.g., classification was referenced as something to be determined later).
  • Step 4: Choose appropriate data sources
    • Select sources that match the objective and domain.
  • Step 5: Validate quality
    • Check for:
      • missing values
      • duplicates
      • outliers
      • format/type consistency
      • freshness/timeliness
      • fitness for purpose
  • Step 6: Clean and prepare
    • Apply cleaning rules and transformations so the data becomes usable.
  • Step 7: Use data ethically and legally
    • Ensure permission, transparency, privacy, and accountability.

Data Sources Mentioned

Common sources

  • Public datasets / competitions
    • Kaggle
    • competition-provided datasets
  • Open datasets
    • mentioned examples: Google datasets, UCI (referred in the transcript with additional context around “UC Roboflow”)
  • Domain-specific dataset platforms
    • Roboflow (especially for computer vision/images)
  • APIs
    • example mentioned: Twitter
    • also weather data via API
  • Surveys
    • data from surveys conducted by researchers/organizers
  • Feedback
    • feedback data as another source
  • Databases
    • internal systems / database servers

Internal vs external

  • Internal data: from within an organization
    • examples mentioned: PostgreSQL, BigQuery
  • External data: outside sources (public/open data, web sources, APIs)

Data Quality Framework (What “Quality” Means)

After collecting data, assess whether it is:

  • Accurate or not
  • Complete or not
  • Valid or not
  • Consistent or not
  • Unique or not (duplicates)
  • Timely / up-to-date (not outdated)
  • Fitness for purpose (whether it suits the goal)

Data quality dimensions explicitly discussed

  • Completeness
    • Example: only 70/100 respondents filled an age field → incomplete/bias risk
  • Uniqueness
    • Example: one customer has two IDs → confusion
  • Validity
    • Example: phone number format/type correctness
  • Timeliness
    • up to date vs outdated
  • Accuracy
    • often requires a benchmark to verify
  • Consistency
    • unexpected spikes/changes may indicate errors or legitimate events
  • Fitness for purpose
    • appropriate for the intended task

Example issues from a case discussion

  • missing values
  • inconsistent data formats:
    • phone number formatting differences
    • date formats using different delimiters/orderings
    • “N/A” used as missing but inconsistently
  • duplicate IDs Conclusion: these issues require cleaning before use.

Strategies for Improving Data Quality (Governance & Operations)

Presented tactics include:

  • Data governance
  • Document and continue documenting
  • Data validation roles
    • data validation by the data engineering side
    • cleaning/preprocessing by:
      • the data analyst / data scientist, or
      • collaboration between teams
  • Automated pipelines
    • define structured cleaning/selection rules (e.g., “keep only 10 columns”)
    • impose constraints (e.g., age ≤ 100)
    • pipeline automation tool mentioned: Airflow
  • Feedback loops
    • monitor results and continuously improve data quality
  • Cleaning tools
    • use tools to clean and monitor data quality metrics

Ethical Data Requirements (Detailed Points)

Ethical data handling includes:

  • Permission
    • inform users / ask for permission when collecting data (especially customer data)
    • align usage with what the user consented to
  • Transparency
    • clearly explain:
      • what data is collected
      • what it will be used for
  • Limitations / scope control
    • do not change the purpose beyond the consented scope
  • Privacy / respect privacy
    • protect sensitive personal identifiers (e.g., ID numbers used for accounts/recruitment)
  • Accountability
    • the collecting organization must be responsible for security
    • prevent leakage/theft (data security is part of this)
  • Legal collection
    • collect and use data legally and responsibly
    • unauthorized use can cause legal consequences (even if the collector “didn’t know”)

Q&A Takeaways (Questions Addressed)

1) Data for MSMEs & financial inclusivity in hackathons

  • Competitions typically focus on analysis of MSME-related trends.
  • Data can come from:
    • organizer-provided datasets
    • publicly accessible government sources
    • separate sites listing MSMEs by region (example mentioned: Jakarta areas)
  • If doing your own survey:
    • define the scope (e.g., MSMEs in a specific area like Kelapa Gading)
    • sample size can be limited (example mentioned: “10 samples”)

2) Using AI for preprocessing data

  • Allowed, with a strict condition: you must validate AI outputs.
  • Warning: AI may apply inappropriate imputation (e.g., mean vs median/mode) if you don’t guide it or understand distributions.
  • The correct approach depends on data distribution and missing-value context.

3) When to use OLAP vs OLTP vs (DWH / “DWH, OLAP, OLTP”)

  • OLAP: analysis-oriented (dashboards, reporting; often business analysts/data analysts)
  • OLTP: transaction-oriented (real-time operational/transaction data; backend/data engineering)
  • DWH (data warehouse): supports analytical processing by consolidating data for querying/reporting
  • Final note: depends on the company/business domain and data needs.

4) Career trend note (automation + broader roles)

  • Automation is increasing (AI agents, easier app/tool creation).
  • However, fundamentals remain critical:
    • know how to analyze data and specify cleaning/imputation rules correctly
  • Roles may overlap more over time (data science ↔ data engineering ↔ ML engineering ↔ software/app development).

Speaker(s) / Sources Featured

Speaker(s)

  • Mr. Sardi Iransyah (founder of Ainoid; main presenter)

Other named people (from Q&A)

  • Christopher (question about MSME/financial inclusion competition data)
  • Fan (question prefaced in the transcript; answered by the speaker)
  • Adinda (identity on the question about using AI for preprocessing)
  • Muhammad Yusuf (participant who contributed a definition of data)

Organizations / tools mentioned

  • Ainoid (company founded by Mr. Sardi Iransyah)
  • IBM
  • Kaggle
  • Roboflow
  • PostgreSQL
  • BigQuery
  • Looker Studio
  • Twitter (example API)
  • Selenium and BeautifulSoup (web scraping tools)
  • Airflow (pipeline automation tool)
  • Gardner’s report (appears as “Gardner” in subtitles; used for a statistic about data quality losses)

Original video