CS 432 - Lecture 3, Data Understanding

Data Mining Process Overview & LLM Integration
📌 The lecture focuses on Data Understanding and Data Pre-processing, which follow Business/Domain Understanding in the seven-phase CRISP-DM process.
🤖 Large Language Models (LLMs) can semi-automate the data mining process by assisting with reasoning and interpretation in each phase, but human control remains crucial due to LLM limitations like hallucinations.
🔄 The relationship between Business Understanding and Data Understanding is iterative and interactive; data captures the domain, and domain knowledge helps interpret the data.

Data Understanding: Types and Characteristics
📊 Conceptually, data can be represented in four main logical formats: Tabular (relational, numeric, transactional), Graphs/Networks, Ordered Data (sequences, time series, text), and Spatial Data.
🔢 Attributes are primarily categorized as Numeric (interval or ratio scaled, the latter having an absolute zero) or Categorical (nominal or ordinal).
📉 Data quality assessment involves understanding Central Tendency (mean, median, mode) and Dispersion Tendency (spread, measured by standard deviation or range like $\text{Q}_3 - \text{Q}_1$).

Data Analysis and Relationship Quantification
🔗 To understand relationships between attributes:
* For two Categorical attributes, use the Chi-Square ($\chi^2$) statistic, where higher values indicate greater dependence.
* For two Numeric attributes, use the Pearson Correlation Coefficient (or covariance matrix); a value near 1 suggests duplication or strong linear relationship.
* For one Numeric and one Categorical attribute, measures like Entropy quantify homogeneity (lower entropy suggests higher dependence).
🖼️ Visualization aids understanding: Scatter plots are used for numeric-numeric comparisons, while Histograms and Bar Charts illustrate distributions and central tendency.

Data Pre-processing Activities
🛠️ Data Pre-processing aims to improve data quality and reshape/format data for the intended task, involving cleaning, integration, transformation, reduction, and discretization.
❗ Data quality assessment identifies issues like noise (random spikes) and missing values (blanks/NA/null), which can be addressed by discarding data or filling in missing entries cautiously to avoid introducing bias.
📉 Binning is a pre-processing technique that can be applied to numeric attributes to create discrete intervals, often used for data smoothing or discretization.

Key Points & Insights
➡️ Human in the Loop: Despite AI advancements, human oversight is mandatory in data mining due to LLM tendencies to hallucinate and make assumptions.
➡️ Domain Expertise: Deep domain understanding is essential for becoming a good data scientist; business understanding and data understanding must be iterated upon.
➡️ Dispersion Insight: Measuring dispersion tendency (spread) helps identify potential data errors or outliers by observing if min/max values fall outside the expected domain range.
➡️ Quiz Logistics: The first quiz will be held in person during the first 15 minutes of class on Monday and covers material up to the Wednesday lecture.

📸 Video summarized with SummaryTube.com on Feb 02, 2026, 07:28 UTC