Python Fundamentals
🐍 Start data science with Anaconda, which bundles Python, Jupyter Notebook, and essential libraries like Pandas and NumPy, for easy setup on Windows, Mac, or Linux.
💻 Utilize Jupyter Notebook for data cleaning, transformation, visualization, and analysis, leveraging its open-source, web-based interface.
📝 Master Jupyter Notebook's interface, including cells (code, markdown), modes (command, edit), and menu options (File, Edit, View, Insert, Cell, Kernel), for efficient workflow.
⌨️ Learn common shortcuts in Jupyter Notebook, like `F` for find/replace, `Y`/`M` for cell type changes, `A`/`B` for inserting cells, `X` for cutting, `V` for pasting, and `D + D` for deleting cells.

Python Data Structures & Control Flow
🏷️ Store data values using variables, assigning them with the `=` sign (e.g., `message_1 = "I'm learning Python"`).
➕ Concatenate strings using the `+` operator or f-strings (e.g., `f"I'm learning {message_1} and it's fun"`), embedding variables directly.
📝 Create lists using square brackets `[]` for ordered, mutable collections of items (e.g., `countries = ["USA", "India"]`), supporting mixed data types and duplicates.
🔍 Access list elements via zero-based indexing (e.g., `countries[0]` for the first element) and negative indexing (e.g., `countries[-1]` for the last element).
✂️ Perform slicing on lists (e.g., `countries[0:3]`) to access subsets of elements, where the start index is inclusive and the stop index is exclusive (for traditional Python slicing).
➕ Add list elements using `append()` for the end or `insert()` for specific positions, or `+` operator to join two lists.
❌ Remove list elements using `remove()` by value, `pop()` by index (and returns item), or `del` by index (doesn't return item).
📊 Sort lists using the `sort()` method, ascending by default, or descending with `reverse=True`.
🔄 Update list values by indexing and assignment (e.g., `numbers[0] = 1000`).
💾 Create copies of lists using slicing (`list[:]`) or the `copy()` method (`list.copy()`) to ensure independence from the original list.
📚 Define dictionaries using curly braces `{}` for unordered collections of key-value pairs (e.g., `my_data = {"name": "Frank", "age": 26}`).
🔑 Access dictionary keys, values, and items using `.keys()`, `.values()`, and `.items()` methods respectively.
➕ Add new key-value pairs (e.g., `my_data["height"] = 1.7`) or update existing ones using the `update()` method (e.g., `my_data.update({"height": 1.8})`).
❌ Remove dictionary items using `pop()` by key (returns value), `del` by key, or `clear()` to remove all items.
🚦 Control program flow with `if`/`elif`/`else` statements, executing code blocks based on conditions.
🔁 Iterate through iterable objects (like lists, dictionaries) using `for` loops, and use `enumerate()` to get both index and value.
🔧 Define reusable code blocks using functions (`def`), with parameters and a `return` value.
🔢 Utilize built-in Python functions like `len()` (length), `max()`/`min()` (max/min value), `type()` (object type), and `range()` (sequence of numbers).
📦 Access Python code in external files using modules and the `import` keyword (e.g., `import os`).

Pandas Core Concepts
📊 Understand Pandas DataFrames as the equivalent of Excel spreadsheets, with two axes (rows and columns/series) and an index.
📚 Create DataFrames from NumPy arrays, Python lists (nested lists for rows), or dictionaries (keys as columns, values as lists for data).
💾 Import data from CSV files (e.g., `pd.read_csv()`) or Excel files (e.g., `pd.read_excel()`) to easily create DataFrames.
👁️ Display DataFrames efficiently using `df.head()` (first 5 rows), `df.tail()` (last 5 rows), or by specifying the number of rows (e.g., `df.head(10)`).
⚙️ Adjust display options (e.g., `pd.set_option('display.max_rows', 1000)`) to view all rows in a DataFrame, similar to full view in Excel.
📝 Distinguish between attributes (values associated with an object, accessed with `.` e.g., `df.shape`), functions (standalone tasks, e.g., `len(df)`), and methods (functions within a class, accessed with `.` and `()` e.g., `df.head()`).

Data Manipulation with Pandas
🏷️ Access DataFrame columns using square bracket notation (`df['column_name']`) for a Series, or double square brackets (`df[['col1', 'col2']]`) for a DataFrame, allowing multiple column selection.
➕ Add new columns to a DataFrame:
* With a scalar value (e.g., `df['new_col'] = 70`) for a constant value across all rows.
* With a NumPy array (e.g., `df['new_col'] = np.random.randint(1, 100, 1000)`) for varied values, ensuring array length matches DataFrame rows.
* Using the `assign()` method (e.g., `df.assign(score1=series1, score2=series2)`) for adding multiple columns concisely; returns a copy unless assigned back.
* Using the `insert()` method (e.g., `df.insert(1, 'test_col', series1)`) to add a column at a specific position/index, modifying the DataFrame in place.
🧮 Perform column-wise operations like `sum()`, `count()`, `mean()`, `std()`, `max()`, `min()` directly on Series (e.g., `df['score'].sum()`), or get quick statistics for all numerical columns with `df.describe()`.
➕ Perform row-wise operations by combining column selections with arithmetic operators (e.g., `df['col1'] + df['col2']`).
📏 Count categorical values and their percentages using `df['column'].value_counts()` and `normalize=True` for relative frequencies.
🔄 Sort DataFrames using `df.sort_values(by='column_name')` ascending by default, or `ascending=False` for descending.
👯 Identify duplicate rows in one or more columns using `df.duplicated()` which returns a boolean Series.
🗑️ Drop duplicate rows using `df.drop_duplicates(subset=['col1'], keep='first'|'last'|False, inplace=True, ignore_index=True)`.
🔢 Get unique values in a Series using `df['column'].unique()` and count them with `df['column'].nunique()`.
📍 Select data by index label using `df.loc[]` (e.g., `df.loc['L. Messi', 'Height_cm']`), where both start and stop of a slice are inclusive.
🔢 Select data by integer position using `df.iloc[]` (e.g., `df.iloc[0, 3]`), where the start is inclusive but the stop is exclusive in slicing.
✍️ Set new values for single cells (e.g., `df.loc['L. Messi', 'Height_cm'] = 175`), entire columns (e.g., `df['Height_cm'] = 190`), or entire rows (e.g., `df.iloc[-1, :] = np.nan`) using `loc` or `iloc` with assignment.
📝 Create a conditional column from more than two choices using `np.select(conditions, values)` (e.g., `price_tiers` based on price ranges).
🔍 Filter DataFrames based on multiple conditions using logical operators (`&` for AND, `|` for OR) enclosed in parentheses (e.g., `df[(df['Company'] 'Apple') & (df['Price'] > 2000)]`).
❓ Filter DataFrames using the `query()` method for a more SQL-like syntax (e.g., `df.query('Age > 34 and Nationality "Italy"')`), supporting operations directly on column names and mathematical expressions.
⚙️ Apply functions to Series or DataFrames using the `apply()` method, supporting both built-in (e.g., `df['Age'].apply(np.sqrt)`) and custom (lambda) functions.
🚀 Use `lambda` functions for concise, anonymous functions, often with `apply()`, (e.g., `df['Height_cm'].apply(lambda x: x / 100)`) for element-wise operations.
💾 Create independent copies of DataFrames using `df.copy(deep=True)` to prevent unintended modifications to the original. A simple assignment (`new_df = old_df`) creates a shallow copy, linking the two.

Data Aggregation & Grouping
🧮 Aggregate DataFrame values using the `agg()` method (e.g., `df.agg(['sum', 'mean'])`), applying functions across columns.
📊 Apply different aggregations per column by passing a dictionary to `agg()`, where keys are column names and values are lists of functions (e.g., `{'Sales_in_thousands': ['sum', 'mean'], 'Price_in_thousands': ['sum', 'max']}`).
🔄 Aggregate over rows by setting `axis=1` in `agg()`, performing operations across specified columns.
分组 Group data into categories using the `groupby()` method (e.g., `df.groupby('Vehicle_type')`), often followed by an aggregation function (e.g., `.mean()`, `.count()`).
🔍 Access specific groups within a `groupby` object using `get_group()` (e.g., `groupby_obj.get_group('Ford')`).
🚫 Control null value handling during grouping with `drop_na=False` in `groupby()` to include `NaN` values as a separate group.
🎯 Combine `groupby()` with `agg()` using tuples for custom aggregation names (e.g., `df.groupby('Manufacturer').agg(Min_Engine_Size=('Engine_Size', 'min'))`).
🐍 Apply custom functions (lambda) to grouped data using `apply()` (e.g., `df.groupby('Manufacturer').apply(lambda x: x * 1000)`).
🚫 Filter groups based on an aggregate condition using the `filter()` method (e.g., `df.groupby('Manufacturer').filter(lambda x: x['Sales_in_thousands'].sum() > 52)`).

Data Merging & Joins
🔗 Combine DataFrames vertically (along rows) using `pd.concat([df1, df2], axis=0)` (default `axis=0`), summing rows with common columns. Use `ignore_index=True` to reset concatenated indexes.
↔️ Combine DataFrames horizontally (along columns) using `pd.concat([df1, df2], axis=1)`, summing columns with common indexes.
🤝 Perform an inner join (returns only matching values) using `df1.merge(df2, on='common_column', how='inner')`, or `pd.merge([df1, df2], on='common_column')` as `how='inner'` is default.
🌐 Perform a full (outer) join (returns all values, filling non-matches with `NaN`) using `df1.merge(df2, on='common_column', how='outer')`.
🚫 Perform an exclusive full join (returns only values unique to each DataFrame) by doing an `outer` merge with `indicator=True`, then `query('_merge "left_only" | _merge "right_only"')`.
⬅️ Perform a left join (returns all values from the left DataFrame and matching values from the right) using `df1.merge(df2, on='common_column', how='left')`.
🚫 Perform an exclusive left join (returns only values unique to the left DataFrame) by doing an `outer` merge with `indicator=True`, then `query('_merge == "left_only"')`.
➡️ Perform a right join (returns all values from the right DataFrame and matching values from the left) using `df1.merge(df2, on='common_column', how='right')`.
🚫 Perform an exclusive right join (returns only values unique to the right DataFrame) by doing an `outer` merge with `indicator=True`, then `query('_merge == "right_only"')`.

Data Cleaning & Preprocessing
🕵️ Identify missing data (NaN) using `df.isnull()` (returns boolean DataFrame) and count nulls per column with `df.isnull().sum()`. Convert counts to percentages using `.mean() * 100`.
🗑️ Deal with missing data:
* Drop columns with high percentages of nulls using `df.drop('column_name', axis=1, inplace=True)`.
* Drop rows containing nulls for specific columns using `df.dropna(subset=['column_name'], inplace=True)`.
* Filter out null rows using boolean indexing (e.g., `df[df['column'].notnull()]`).
* Fill null values using `df['column'].fillna(value, inplace=True)`:
* With `mode()` for categorical data (e.g., `df['Rating'].fillna(df['Rating'].mode()[0])`).
* With `mean()` or `median()` for numerical data.
* With an arbitrary number (e.g., `df['Duration'].fillna('0', inplace=True)`) to facilitate later operations.
* With `ffill` (forward fill) or `bfill` (backward fill) for sequential data (e.g., `df.fillna(method='ffill')`).
📝 Standardize inconsistent capitalization using string methods accessible via `.str` attribute: `lower()`, `upper()`, `title()` (e.g., `df['Title'].str.lower()`).
✂️ Remove blank spaces (leading/trailing) using `.str.strip()`, `.str.lstrip()` (left), or `.str.rstrip()` (right).
🔄 Replace strings using `.str.replace()` (for strings) or the more versatile `replace()` method (for various data types).
🔍 Use regular expressions with `.str.replace()` (`regex=True`) or `re.sub()` (with `apply()` and `lambda`) to replace patterns (e.g., removing punctuation).
🔑 Understand meta characters in regex (`\d` for digits, `\w` for word characters, `\s` for whitespace, `.` for any char except newline, `^` for start, `$` for end, `[]` for character sets).
🔢 Understand quantifiers in regex (`*` for zero or more, `+` for one or more, `?` for zero or one, `{n}` for exact n, `{n,}` for n or more, `{n,m}` for n to m).
🧩 Use parentheses `()` for capturing groups and square brackets `[]` for character sets in regex.

Data Visualization with Pandas & Plotly
📈 Create line plots (e.g., `df.plot(kind='line')`) to visualize trends over time, customizing labels and titles.
📊 Create bar plots (e.g., `df_year.plot(kind='bar')`) to compare categorical data, requiring specific data reshaping (transpose for countries as index).
🥧 Create pie charts (e.g., `df.plot(kind='pie', y='column_name', labels=df['label_column'])`) to show proportions, requiring labels and specific column data.
📦 Create box plots (e.g., `df.plot(kind='box')`) to visualize data distribution (min, Q1, median, Q3, max) and identify outliers.
📉 Create histograms (e.g., `df.plot(kind='hist', bins=10)`) to show frequency distribution of numerical data within ranges (bins).
✨ Make interactive visualizations using `cufflinks` and `plotly` (`df.iplot()`), offering zoom, pan, and hover functionalities for detailed data exploration.
💾 Export plots as PNG (e.g., `plt.savefig('my_plot.png')`) and DataFrames as Excel files (e.g., `df.to_excel('my_table.xlsx')`).

Machine Learning - Regression
🧠 Understand linear regression for modeling relationships between variables:
* Simple Linear Regression: One predictor, one target.
* Multiple Linear Regression: Multiple predictors, one target.
* Equation: `Y = B0 + B1*X1 + ... + Bn*Xn` (Y=dependent, X=independent, B0=intercept, B=coefficients).
🛠️ Implement linear regression with StatsModels:
* Import `statsmodels.api as sm`.
* Define dependent (Y) and independent (X) variables.
* Add a constant to X (e.g., `sm.add_constant(X)`), as StatsModels doesn't do this by default.
* Fit the model (e.g., `sm.OLS(Y, X).fit()`).
* Predict values (e.g., `lm.predict(X)`).
* Analyze model performance using `lm.summary()`, checking R-squared, coefficients, standard error.
🚀 Implement linear regression with Scikit-learn:
* Import `LinearRegression` from `sklearn.linear_model`.
* Define dependent (Y) and independent (X) variables.
* Initiate `LinearRegression` model.
* Fit the model (e.g., `lm.fit(X, Y)`), where Scikit-learn adds the constant by default.
* Predict values (e.g., `lm.predict(X)`).
* Access R-squared (`lm.score(X, Y)`), coefficients (`lm.coef_`), and intercept (`lm.intercept_`) individually.

Machine Learning - Classification (NLP Project)
📚 Prepare text data for machine learning:
* Under-sampling: Deleting samples from the majority class to balance data (e.g., using `df_majority.sample(n=len_minority)` or `imblearn.under_sampling.RandomUnderSampler`).
* Over-sampling: Duplicating samples from the minority class to balance data (e.g., `imblearn.over_sampling.RandomOverSampler`).
* Split data into train and test sets using `train_test_split` from `sklearn.model_selection`, setting `test_size` (e.g., `0.33`) and `random_state`.
* Separate train/test sets into independent (X) and dependent (Y) variables (e.g., `train_X` for reviews, `train_Y` for sentiment).
📝 Convert text into numerical vectors using Bag of Words (BoW):
* CountVectorizer (from `sklearn.feature_extraction.text`): Counts word frequencies, builds vocabulary, generates a document-term matrix (DTM).
* TF-IDF Vectorizer (from `sklearn.feature_extraction.text`): Computes term frequency-inverse document frequency, weighting word relevance (higher score for unique words).
* Apply `fit_transform()` on training data (`train_X`) and `transform()` on test data (`test_X`) for consistent vectorization.
🎯 Understand Supervised Learning for classification problems (predicting categories like positive/negative sentiment).
🤖 Implement Classification Algorithms with Scikit-learn:
* Support Vector Machine (SVM) (`sklearn.svm.SVC`): Finds a hyperplane to best separate classes; good for text classification.
* Decision Tree (`sklearn.tree.DecisionTreeClassifier`): Builds a tree-like model to make predictions based on rules.
* Naive Bayes (`sklearn.naive_bayes.GaussianNB`): Uses conditional probability, assuming feature independence (requires `.toarray()` for sparse matrix input).
* Logistic Regression (`sklearn.linear_model.LogisticRegression`): Predicts probability (0-1) for binary classification, using a sigmoid function.

Key Points & Insights
💡 Data Cleaning is Crucial: Before any analysis or modeling, proactively identify and handle missing data (using `isnull()`, `dropna()`, `fillna()` with `mode`/`mean`/`ffill`/`bfill`), inconsistent capitalization (`.str.upper()`/`.title()`), and blank spaces (`.str.strip()`).
📊 Choose the Right Visualization: Select plots (line, bar, pie, box, histogram, scatter) based on data type and the story you want to tell. Interactive plots (Plotly) offer deeper insights into data points and distributions, especially for outliers.
🧩 Understand Data Transformation: Convert raw text into numerical vectors using techniques like TF-IDF (`TfidfVectorizer`) to capture word relevance, which is essential for NLP tasks.
⚖️ Address Imbalanced Data: Be aware of imbalanced classes in your dataset (e.g., 9000 positive vs. 1000 negative reviews) and use techniques like under-sampling or over-sampling (`imblearn` library) to prevent bias in your machine learning models.
🧪 Follow ML Workflow: Always split data into train and test sets before model building to ensure an unbiased evaluation of performance.
📈 Evaluate Models Critically: Don't rely on a single metric. Use a Confusion Matrix to understand true positives/negatives and false positives/negatives. Calculate Accuracy for overall correct predictions and F1 Score (weighted average of precision and recall) for imbalanced datasets.
⚙️ Optimize Model Performance: Use GridSearchCV to systematically search for the best hyperparameters for your model, enhancing its predictive power.
🔄 Master Pandas Operations: Efficiently manipulate, aggregate, and merge data using Pandas' powerful methods like `groupby()`, `agg()`, `merge()`, `concat()`, `loc`, `iloc`, and `query()`.

📸 Video summarized with SummaryTube.com on Sep 01, 2025, 15:27 UTC