Python Data Analysis
Lộ trình phát triển toàn diện Python Data Analysis theo tiêu chuẩn quốc tế nilbuild/developer-roadmap
Hướng dẫn từng bước từ nền tảng đến chuyên sâu giúp bạn làm chủ Python Data Analysis. Tích hợp tài liệu lý thuyết, bài viết thực chiến, video tham khảo và bài tập lập trình trực tiếp trên IDE.
Nền Tảng & Khái Niệm Cốt Lõi
Giai đoạn 1 tập trung hoàn thiện 23 chủ đề then chốt.
Airflow
Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data pipelines. Pipelines are defined as DAGs (Directed Acyclic Graphs) in Python, where each node is a task and edges define dependencies. Airflow provides a web UI for monitoring runs, retrying failures, and tracking execution history. It is the standard orchestration tool for production Python data pipelines.
Altair
Altair is a declarative statistical visualization library for Python based on the Vega-Lite grammar. Charts are built by binding data columns to visual channels (x, y, color, size) and specifying the mark type. Altair produces interactive charts by default and generates JSON specifications that render in notebooks and web browsers.
APIs with requests
The `requests` library is the standard Python tool for making HTTP requests. It is used to call REST APIs that return JSON or XML data. A typical workflow involves calling `requests.get(url, params=params)`, checking the response status, and parsing the JSON body with `.json()` before loading it into a DataFrame.
args & kwargs
`*args` allows a function to accept any number of positional arguments as a tuple. `**kwargs` allows any number of keyword arguments as a dictionary. They make functions flexible when the number or names of arguments are not known in advance, and are widely used in Python libraries for passing options through layers of function calls.
Arithmetic
Arithmetic operators perform mathematical calculations: `+` (addition), `-` (subtraction), `*` (multiplication), `/` (division), `//` (floor division), `%` (modulo), and `**` (exponentiation). They are used constantly for computing derived columns, normalizing values, and performing aggregations.
Array Operations
NumPy supports a wide range of array operations: element-wise arithmetic, aggregation functions (`sum`, `mean`, `std`, `min`, `max`), reshaping, stacking, and splitting. These operations are vectorized, meaning they apply to the entire array at once without explicit loops, making them highly efficient.
Arrays & ndarray
The `ndarray` is NumPy's core data structure: a multi-dimensional, homogeneously typed array stored in contiguous memory. It supports element-wise operations, broadcasting, and vectorized computation far faster than Python lists. Understanding ndarray is fundamental to working efficiently with numerical data in Python.
BeautifulSoup
BeautifulSoup is a Python library for parsing HTML and XML documents. It provides methods for navigating the document tree, searching for elements by tag, class, or attribute, and extracting text and links. BeautifulSoup is used for web scraping when the target website does not provide an API.
Big Data Tools
Big data tools process datasets too large to fit in a single machine's memory using distributed or out-of-core computation. Dask and PySpark are the primary Python tools for scaling beyond what Pandas and NumPy can handle. They provide familiar DataFrame-like APIs while distributing computation across cores or clusters.
Booleans
Booleans (`bool`) have two values: `True` and `False`. They are the result of comparison and logical operations and are used to control flow and filter data.
Boxplot
A box plot displays the five-number summary of a variable: minimum, Q1, median, Q3, and maximum. The box covers the IQR, a line marks the median, and whiskers extend to the data range. Points beyond the whiskers are plotted individually as potential outliers. Box plots are effective for comparing distributions across groups.
Built-in Functions
Python's built-in functions are available without any imports and cover common operations: `len()`, `sum()`, `min()`, `max()`, `sorted()`, `enumerate()`, `zip()`, `map()`, `filter()`, and others. These functions simplify common tasks and are used constantly alongside data analysis libraries.
Casting Types
Casting types in Pandas converts a column from one data type to another using `.astype()`. Common conversions include converting string columns to numeric with `pd.to_numeric()`, converting to datetime with `pd.to_datetime()`, and converting integers to categories. Correct data types are required for accurate calculations and efficient memory use.
Categorical Plots
Seaborn's categorical plots visualize the relationship between a numeric variable and one or more categorical variables. `sns.boxplot()`, `sns.violinplot()`, `sns.barplot()`, `sns.stripplot()`, and `sns.countplot()` cover the main patterns. They are used to compare distributions or averages across groups.
Comparison
Comparison operators evaluate the relationship between two values and return a boolean. They include `==`, `!=`, `<`, `>`, `<=`, and `>=`. They form the building blocks of filtering conditions applied to DataFrames and arrays.
conda
conda is an open-source package and environment manager included with the Anaconda and Miniconda distributions. It manages both Python packages and non-Python dependencies, making it well suited for scientific computing. conda environments isolate project dependencies and can be exported to `environment.yml` for reproducibility.
Conditionals
Conditionals execute different code paths based on whether a condition is true. Python uses `if`, `elif`, and `else` for this. Python 3.10 introduced `match/case`, a structural pattern matching statement that cleanly handles multiple specific value checks as an alternative to long `elif` chains. They appear in custom functions applied to DataFrames, in filtering logic, and in branching pipeline code.
Correlation & Covariance
Correlation measures the strength and direction of the linear relationship between two variables, scaled between −1 and +1. Covariance measures the same relationship but is not normalized, making it harder to interpret across variables with different scales. `df.corr()` and `df.cov()` compute these matrices in Pandas, and heatmaps are used to visualize them.
Correlation Matrix
A correlation matrix shows the pairwise correlation coefficients between all numeric columns in a dataset. It is computed with `df.corr()` and typically visualized as a heatmap using Seaborn. It is a key EDA tool for identifying which features are strongly related, which helps with feature selection and multicollinearity detection.
Cross-tabulation
Cross-tabulation (crosstab) counts the frequency of combinations of values across two or more categorical variables. `pd.crosstab()` produces a table of frequencies or proportions. It is used to examine relationships between categorical variables, such as how customer segments differ across product categories.
CSV
CSV (Comma-Separated Values) is the most common format for tabular data exchange. `pd.read_csv()` loads a CSV file into a DataFrame and accepts dozens of parameters for handling separators, missing values, date parsing, and data types. CSV files are human-readable but lack type information, so columns often need type correction after loading.
Customizing Plots
Seaborn plots are customized through function parameters (palette, hue, size, style) and by accessing the underlying Matplotlib axes after creation. `sns.set_theme()` and `sns.set_style()` change the global appearance. Seaborn's theming system makes it easy to produce clean, publication-ready charts with minimal code.
Customizing Plots
Matplotlib allows extensive customization: titles with `set_title()`, axis labels with `set_xlabel()` / `set_ylabel()`, tick formatting, color palettes, line styles, font sizes, legends, and annotations. Customizing plots ensures they communicate clearly and meet the standards required for reports and presentations.
Kỹ Năng Trọng Tâm & Thực Hành
Giai đoạn 2 tập trung hoàn thiện 23 chủ đề then chốt.
Dash
Dash is a Python framework for building analytical web applications, developed by Plotly. It combines Plotly charts with reactive UI components and runs as a Flask web server. Dash provides more control and customization than Streamlit and is better suited for production-grade dashboards with complex interactivity.
Dashboards
Dashboards combine multiple visualizations and controls into a single interface for monitoring and exploring data. Python provides several tools for building data dashboards that can be shared as web applications without requiring frontend development skills. The main options are Streamlit, Dash, and connecting to BI tools like Power BI and Tableau.
Dask
Dask is a parallel computing library for Python that scales Pandas, NumPy, and Scikit-learn to larger-than-memory datasets. It breaks data into chunks and builds a task graph that is executed lazily. Dask DataFrames mirror the Pandas API, making it easy to adapt existing code for larger datasets without switching ecosystems.
Data Cleaning
Data cleaning identifies and resolves quality issues in raw data so it is accurate and consistent enough for analysis. In Python, cleaning is done primarily with Pandas and string processing tools. Common tasks include handling missing values, fixing data types, standardizing text, removing duplicates, and detecting outliers.
Data Pipelines
Data pipelines automate the sequence of steps that move and transform data from sources to destinations. They encapsulate the full workflow — ingestion, cleaning, transformation, and output — as code. Orchestration tools like Airflow schedule and monitor these pipelines in production, ensuring they run reliably and their failures are caught and handled.
Defining Functions
User-defined functions are created with the `def` keyword and encapsulate reusable logic. A function takes parameters, executes a body, and returns a value with `return`. Writing well-scoped functions makes analysis code modular, testable, and easier to apply across a dataset using Pandas' `apply()` method.
Dictionaries
Dictionaries store key-value pairs and provide fast lookup by key. They are used extensively in Python for mapping labels to values, building frequency counts, and configuring function arguments.
Distribution plots
Seaborn's distribution plots visualize the distribution of one or two variables. `sns.histplot()` and `sns.kdeplot()` show the shape of a single variable's distribution. `sns.displot()` combines both. `sns.pairplot()` shows pairwise distributions and relationships across all numeric columns in a DataFrame.
Dropping vs. Imputing
When handling missing values, dropping removes rows or columns with `dropna()`, while imputing fills them with a substitute value using `fillna()` or `SimpleImputer` from Scikit-learn. Dropping is appropriate when missing data is rare or random. Imputing is preferred when data is valuable or missing systematically, using the mean, median, mode, or a predicted value.
DuckDB
DuckDB is an in-process analytical database designed for fast SQL queries on large datasets stored as files or in memory. It can query CSV, Parquet, and Pandas DataFrames directly with SQL syntax. DuckDB is increasingly used in data analysis workflows as a fast alternative to loading data into a full database system.
Encoding Categories
Categorical encoding converts text category labels into numerical values that machine learning algorithms can process. Common approaches include label encoding (assigning each category an integer), one-hot encoding (creating binary columns for each category with `pd.get_dummies()`), and ordinal encoding for ordered categories.
Environment Setup
Setting up a proper Python environment for data analysis involves choosing a package manager, managing dependencies, and selecting a development environment. A well-configured environment ensures reproducibility and avoids package conflicts. The main tools are pip and conda for package management, and virtual environments for isolation.
Excel
Excel files (`.xlsx`, `.xls`) are loaded with `pd.read_excel()`, which supports selecting sheets, skipping rows, and reading specific columns. The `openpyxl` library is required for `.xlsx` files. Excel is common in business environments, and analysts frequently need to read and write it as part of reporting workflows.
Exploratory Data Analysis
Exploratory Data Analysis (EDA) is the process of examining a dataset to understand its structure, distributions, and relationships before formal modeling. It combines descriptive statistics and visualizations to surface patterns, anomalies, and hypotheses. EDA guides subsequent cleaning decisions and model choices.
Filtering & Querying
Filtering in Pandas selects rows that meet specified conditions. Boolean masks, the `.query()` method, and `.isin()` are common approaches. Multiple conditions can be combined with `&` (and) and `|` (or), and the `.query()` method allows SQL-like string syntax for readable filtering expressions.
Floats
Floats (`float`) represent real numbers with a decimal point. Most numerical data in analysis involves floats, including prices, measurements, and probabilities. Floating-point arithmetic has precision limitations that can cause small rounding errors, which are important to be aware of in financial and scientific calculations.
Forward / Backward Fill
Forward fill (`ffill`) propagates the last valid value forward to fill subsequent missing entries. Backward fill (`bfill`) does the reverse, filling from the next valid value. Both are commonly used for time series data where missing values represent periods where the previous or next observation is the best estimate.
Functions & Methods
Functions are reusable blocks of code that take inputs, perform operations, and return outputs. They encapsulate cleaning steps, transformations, and calculations that need to be applied consistently. Python supports built-in functions, user-defined functions, and anonymous lambda functions.
GeoPandas
GeoPandas is an open-source library that extends the capabilities of pandas by allowing spatial operations on geometric types. It simplifies working with geospatial data by enabling the use of familiar data structures like GeoSeries and GeoDataFrame, which store and manipulate vector-based geographic information. Through its integration with libraries like Shapely and PyGEOS, it allows you to perform complex geometric operations such as spatial joins, projections, and distance calculations using straightforward syntax.
Geospatial Analysis
Geospatial analysis is the process of gathering, manipulating, and mapping data that is tied to specific geographic locations on the Earth's surface. It involves using tools and libraries to perform spatial operations, such as calculating distances between coordinates, analyzing geographic patterns, or visualizing datasets on interactive maps. By integrating coordinate systems and geometric shapes into data workflows, this analysis allows for a deeper understanding of how physical location influences various trends and phenomena.
Google Colab
Google Colab is a cloud-hosted Jupyter notebook environment from Google. It requires no local setup and provides free access to GPUs and TPUs, making it popular for machine learning work. Colab notebooks are stored in Google Drive and can be shared like any other document.
Groupby & Aggregation
`groupby()` splits a DataFrame into groups based on one or more columns, applies a function to each group, and combines the results. Common aggregation functions include `sum()`, `mean()`, `count()`, `min()`, `max()`, and custom functions via `agg()`. This split-apply-combine pattern is one of the most powerful features of Pandas.
Heatmaps
`sns.heatmap()` visualizes matrix-style data using color intensity. It is most commonly used to display correlation matrices and pivot tables. Color maps, annotations, and masking options allow the heatmap to be customized for readability. Heatmaps are an effective way to show patterns across two categorical dimensions.
Kiến Trúc Nâng Cao & Tối Ưu
Giai đoạn 3 tập trung hoàn thiện 23 chủ đề then chốt.
Histogram
A histogram groups numeric values into bins and shows the count or frequency of each bin as a bar. It is the primary tool for visualizing the distribution of a single variable: its shape, center, spread, and whether it is skewed or has multiple peaks. `df['col'].hist()` and Matplotlib's `plt.hist()` are the standard ways to create one.
Array Operations
NumPy supports a wide range of array operations: element-wise arithmetic, aggregation functions (`sum`, `mean`, `std`, `min`, `max`), reshaping, stacking, and splitting. These operations are vectorized, meaning they apply to the entire array at once without explicit loops, making them highly efficient.
Indexing & Slicing
Pandas provides two primary indexing systems: `.loc[]` for label-based selection and `.iloc[]` for position-based selection. Both work on rows, columns, or both simultaneously. Boolean indexing with a condition (e.g., `df[df['age'] > 30]`) is the most common way to filter rows.
Integers
Integers (`int`) are whole numbers without a decimal point. They appear as counts, indices, IDs, and categorical encodings. Python integers have arbitrary precision, meaning they do not overflow like integers in lower-level languages.
Interactive Visualization
Interactive visualizations allow users to explore data by hovering, zooming, panning, and filtering directly in the chart. They are more engaging than static plots for dashboards and reports where the audience needs to examine specific data points. Python's main interactive visualization libraries are Plotly and Altair.
Introduction
Python is the dominant language for data analysis due to its readable syntax, rich ecosystem of libraries, and strong community support. Getting started requires understanding the core language features: operators, data types, control flow, and data structures. These fundamentals apply directly to every data manipulation and analysis task that follows.
IQR
The Interquartile Range (IQR) is the difference between the 75th percentile (Q3) and 25th percentile (Q1) of a dataset. Outliers are commonly defined as values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR. The IQR method is robust to extreme values and is the basis for the box plot's whiskers.
isnull, isna
`isnull()` and `isna()` are equivalent Pandas methods that return a boolean DataFrame or Series indicating which values are missing (NaN). They are the first step in assessing data completeness. Combined with `.sum()`, they give a count of missing values per column, and with boolean indexing they select rows with missing data.
JSON
JSON (JavaScript Object Notation) is a text format for structured data commonly returned by APIs. `pd.read_json()` converts JSON into a DataFrame, though nested structures often require normalization with `pd.json_normalize()`. JSON is flexible but can be irregular in structure, requiring careful handling of missing fields.
JupyterLab
JupyterLab is the modern, full-featured successor to the classic Jupyter Notebook interface. It supports notebooks where code, output, and narrative text coexist in a single document, while adding a tabbed layout, a file browser, a terminal, and support for multiple file types side by side. It is the standard environment for exploratory data analysis because results are visible immediately after each cell is run.
Lambda Functions
Lambda functions are anonymous, single-expression functions defined with the `lambda` keyword. They are used for short, throwaway operations, particularly as arguments to functions like `map()`, `filter()`, and Pandas' `apply()`. For example: `df['col'].apply(lambda x: x * 2)`.
Linear Algebra Basics
NumPy provides linear algebra operations, including matrix multiplication (`np.dot`, `@`), matrix inversion, determinants, and eigenvalues. These are used in statistics (covariance matrices), machine learning (feature transformations), and scientific computing. Understanding the basics of matrix operations is useful for reading ML algorithm implementations.
List Comprehensions
List comprehensions provide a concise syntax for creating lists by applying an expression to each item in an iterable, optionally filtering with a condition. For example: `[x**2 for x in range(10) if x % 2 == 0]`. They are faster and more readable than equivalent `for` loops for simple transformations.
Lists
Lists are ordered, mutable sequences that can hold elements of any type. They are one of the most used data structures in Python for storing collections of values. Typical uses include holding column names, storing results from loops, and passing multiple values to functions.
Logical
Logical operators combine boolean expressions. Python uses `and`, `or`, and `not` for this purpose. They are used heavily in data filtering conditions, such as selecting rows where multiple criteria are true simultaneously.
Loops
Loops execute a block of code repeatedly. Python provides `for` loops for iterating over sequences and `while` loops for condition-based repetition. They are used for batch processing files, iterating over grouped data, and automating repetitive tasks, though vectorized operations are preferred for performance.
Matplotlib
Matplotlib is Python's foundational plotting library. It provides a MATLAB-like interface for creating static, animated, and interactive visualizations. While more verbose than higher-level libraries, Matplotlib offers the most control over every aspect of a plot and is the basis for understanding how other Python visualization tools work.
Mean, Median, Mode
Mean, median, and mode are measures of central tendency that describe the typical value in a distribution. Pandas computes these with `mean()`, `median()`, and `mode()` on Series or DataFrame columns. Comparing them reveals distribution shape: in a symmetric distribution they are equal; in a skewed one they diverge.
Merging & Joining
Pandas provides `merge()` and `join()` for combining DataFrames based on common columns or indices. Merge supports inner, left, right, and outer joins, mirroring SQL JOIN behavior. `concat()` stacks DataFrames vertically or horizontally. These operations are used to combine data from multiple sources into a single analysis-ready table.
Null Values
Null values represent missing or undefined data within a dataset, signaling that a specific observation or entry is absent. In Python, this is typically represented by `None`, while libraries like pandas utilize `NaN` (Not a Number) to denote missing numeric information. Handling these values is a fundamental step in data cleaning, as they must be identified and addressed to ensure that statistical calculations and machine learning models perform accurately.
NumPy
NumPy is the foundational numerical computing library for Python. It provides the `ndarray`, a fast, multi-dimensional array, and a comprehensive library of mathematical functions that operate on arrays without Python loops. NumPy underpins Pandas, Scikit-learn, and most other scientific Python libraries.
OOP for Data Analysis
Object-oriented programming (OOP) organizes code around classes and objects rather than standalone functions and procedures. A class defines a blueprint with attributes (data) and methods (behavior), and objects are instances of that class. For data analysis work, OOP is useful when building reusable data processing components, custom dataset loaders, or analysis pipelines that need to maintain state across multiple steps. Most of the libraries used daily, including Pandas, NumPy, and Scikit-learn, are built around classes, so understanding OOP helps in reading documentation, subclassing existing components, and writing cleaner, more maintainable analysis code.
Operators
Operators are symbols that perform operations on values and variables. Python supports arithmetic, comparison, and logical operators, each serving a different purpose in data analysis code. Understanding how operators work and combine is necessary for writing correct filtering conditions, calculations, and control flow logic.
Hệ Sinh Thái & Triển Khai Thực Tế
Giai đoạn 4 tập trung hoàn thiện 23 chủ đề then chốt.
Pandas String Methods
Pandas exposes string methods through the `.str` accessor on Series, allowing vectorized text operations on entire columns. Methods include `.str.strip()`, `.str.lower()`, `.str.contains()`, `.str.replace()`, `.str.split()`, and `.str.extract()`. These methods avoid the need to loop over rows for string cleaning.
Pandas
Pandas is the primary data manipulation library for Python. It provides two core data structures: Series (one-dimensional) and DataFrame (two-dimensional tabular data). Pandas supports loading data from many formats, cleaning, transforming, grouping, merging, and exporting data, covering the full data preparation workflow.
Pandas
Pandas integrates with SQL through `pd.read_sql()`, which executes a SQL query against a database connection and returns the result as a DataFrame. This allows analysts to leverage SQL for initial data extraction and filtering while using Pandas for downstream manipulation and analysis.
Parquet
Parquet is a columnar file format optimized for analytical workloads. It stores data with type information, supports efficient compression, and reads much faster than CSV for large datasets. `pd.read_parquet()` requires the `pyarrow` or `fastparquet` library and is the preferred format for storing processed DataFrames on disk.
Parsing Dates
Date columns loaded from CSV are typically read as strings and must be converted to datetime objects for time-based operations. `pd.to_datetime()` parses date strings in many formats and accepts a `format` parameter for custom patterns. Once parsed, datetime columns enable operations like extracting year/month, computing differences, and resampling time series.
pip
pip is Python's default package installer. It installs packages from the Python Package Index (PyPI) using `pip install package-name`. Libraries like NumPy, Pandas, Matplotlib, and Scikit-learn are all installed this way. A `requirements.txt` file captures all dependencies for a project.
Plot Categories
Matplotlib supports a wide range of plot types: line plots (`plot`), bar charts (`bar`, `barh`), scatter plots (`scatter`), histograms (`hist`), box plots (`boxplot`), pie charts (`pie`), and more. Choosing the right plot type depends on the data structure and the relationship being communicated.
Plotly
Plotly is a Python library for creating interactive charts and dashboards. It produces web-based visualizations using JavaScript under the hood, with a Python API. Plotly Express provides a high-level interface for common chart types, while the `graph_objects` module offers full control. Plotly integrates with Dash for building full web dashboards.
Polars
Polars is a fast DataFrame library for Python written in Rust. It is designed as a high-performance alternative to Pandas, with a more consistent API and significantly better performance on large datasets. Polars uses lazy evaluation and query optimization to process data efficiently without loading everything into memory at once.
Power BI / Tableau
Power BI and Tableau are enterprise BI platforms for building interactive dashboards and reports. Python integrates with both: Power BI supports Python visuals and data transformation scripts, and Tableau supports Python through TabPy for custom calculations. Analysts who prepare data in Python can visualize and distribute it through these platforms for business audiences.
Printing Variables
Printing variables is done with Python's built-in `print()` function. During analysis, printing intermediate values helps verify that transformations are working as expected. F-strings (`f"value: {variable}"`) provide a clean way to format output with variable values embedded in strings.
PySpark
PySpark is the Python API for Apache Spark, the distributed data processing engine. It allows Python code to run Spark jobs on clusters, processing datasets at the scale of terabytes. PySpark provides DataFrame and SQL APIs similar to Pandas and integrates with MLlib for distributed machine learning. It is used when data volume exceeds what Dask or a single machine can handle.
Random Module
NumPy's `random` module generates pseudorandom numbers and samples. It provides functions for creating random arrays, sampling from distributions (normal, uniform, binomial), and setting a seed for reproducibility. Random number generation is used in simulation, bootstrapping, and initializing machine learning models.
re
The `re` module provides regular expression support for pattern matching and text manipulation. It is used for extracting structured data from unstructured text, validating formats, and performing complex find-and-replace operations. Key functions include `re.match()`, `re.search()`, `re.findall()`, and `re.sub()`.
Reading Data
Pandas provides functions to load data from many formats: `pd.read_csv()`, `pd.read_excel()`, `pd.read_json()`, `pd.read_parquet()`, `pd.read_sql()`, and others. Each function returns a DataFrame and accepts parameters for handling headers, delimiters, encoding, and data types. Reading data is always the first step in a Pandas workflow.
Reading Local Files
Reading local files loads data stored on disk into Python for analysis. Pandas supports the most common file formats used in data work. The right function to use depends on the file format, and parameters like delimiter, encoding, and header row often need to be specified.
Reading Web Data
Reading web data involves fetching data from URLs, REST APIs, and web pages directly into Python. This allows analysis workflows to incorporate live or frequently updated data without manual downloads. The main tools are the `requests` library for APIs and `BeautifulSoup` or `scrapy` for web scraping.
Regression Plots
Seaborn's regression plots visualize the relationship between two numeric variables with a fitted regression line. `sns.regplot()` plots data points and a linear regression fit with confidence interval. `sns.lmplot()` extends this to support faceting by a categorical variable, enabling comparison across groups.
Reshaping
Reshaping transforms the structure of a DataFrame without changing its data. `pivot()` and `pivot_table()` convert long-format data to wide format. `melt()` does the reverse, converting wide to long. `stack()` and `unstack()` move index levels to columns or vice versa. Reshaping is often needed to prepare data for specific visualizations or models.
Saving figures
Figures are saved to disk with `plt.savefig('filename.png', dpi=300, bbox_inches='tight')`. Supported formats include PNG, PDF, SVG, and JPEG. Saving high-resolution figures is important when embedding charts in reports or publications. The `bbox_inches='tight'` parameter prevents axis labels from being cut off.
Scatterplot
A scatter plot displays two numeric variables as points on an x-y axis to reveal their relationship. It is used during EDA to detect correlations, clusters, and outliers. A trend line or regression line can be added to show the direction and strength of the linear relationship between the variables.
Scikit-learn
Scikit-learn is the standard machine learning library for Python. It provides a consistent API for classification, regression, clustering, dimensionality reduction, model selection, and preprocessing. Models are trained with `.fit()`, used to predict with `.predict()`, and evaluated with a suite of metrics. Scikit-learn also provides tools for pipelines, cross-validation, and hyperparameter tuning.
SciPy
SciPy is a scientific computing library built on NumPy. It provides modules for statistics (`scipy.stats`), optimization (`scipy.optimize`), linear algebra, signal processing, and numerical integration. `scipy.stats` is used for hypothesis tests (t-tests, chi-square, ANOVA), probability distributions, and descriptive statistics beyond what NumPy provides.
Chuyên Gia & Mở Rộng Hệ Thống
Giai đoạn 5 tập trung hoàn thiện 21 chủ đề then chốt.
scrapy
Scrapy is a Python framework for large-scale web scraping. Unlike BeautifulSoup, which parses individual pages, Scrapy manages the full crawling workflow: following links, handling pagination, managing request queues, and exporting data. It is used when scraping requires collecting data from many pages across a site.
Seaborn
Seaborn is a Python visualization library built on Matplotlib that provides a higher-level interface for statistical graphics. It handles common plot types with less code and integrates tightly with Pandas DataFrames. Seaborn is particularly strong for visualizing statistical relationships, distributions, and grouped comparisons.
Series and DataFrame
A Series is a one-dimensional labeled array, analogous to a single column in a spreadsheet. A DataFrame is a two-dimensional table of Series that share an index, analogous to a spreadsheet or SQL table. These two structures are the foundation of all Pandas operations.
Sets
Sets are unordered collections of unique values. They support mathematical set operations like union, intersection, and difference. They are useful for finding unique values, checking membership, and comparing two groups of items.
SQL Fundamentals
SQL (Structured Query Language) is the standard language for querying relational databases. Data analysts use SQL to extract, filter, aggregate, and join data from databases before loading it into Python for further analysis. Python provides several libraries for running SQL queries directly from code.
SQLAlchemy
SQLAlchemy is a Python SQL toolkit and object-relational mapper (ORM) that provides a unified interface for connecting to many database backends including PostgreSQL, MySQL, SQLite, and others. It is used with Pandas via `pd.read_sql()` to load query results directly into DataFrames.
sqlite3
`sqlite3` is Python's built-in library for working with SQLite databases. SQLite is a lightweight, file-based relational database that requires no server setup. It is commonly used for local data storage, prototyping, and working with small to medium datasets entirely within Python.
Statistics & ML
Python has a rich ecosystem of libraries for statistical analysis and machine learning. SciPy extends NumPy with statistical tests, optimization, and signal processing. Scikit-learn provides a consistent API for building, evaluating, and deploying machine learning models. Together they cover the analytical needs of most data analysis work.
Streamlit
Streamlit is an open-source Python framework for building interactive data applications with minimal code. A Streamlit app is a Python script where each widget (slider, dropdown, text input) triggers a rerun of the script with the new value. It is popular for rapidly prototyping and sharing data analysis tools and ML demos.
Strings
Strings (`str`) represent text data. Column names, categorical values, labels, and file paths are all strings. Python provides extensive string methods for cleaning, parsing, and transforming text data, and the `re` module adds regex-based pattern matching.
strip, replace, split
`strip()` removes leading and trailing whitespace from a string. `replace()` substitutes one substring with another. `split()` divides a string into a list based on a delimiter. These three methods are among the most used for basic text cleaning, applied either to Python strings directly or through the Pandas `.str` accessor on a column.
Subplots and figures
Matplotlib's `Figure` is the top-level container, and `Axes` objects are the individual plots within it. `plt.subplots(rows, cols)` creates a grid of axes for displaying multiple plots side by side. Subplots are used to compare distributions across groups or to show multiple related variables in one figure.
Tuples
Tuples are ordered, immutable sequences. Unlike lists, their contents cannot be changed after creation. They are used to represent fixed collections of values, such as coordinate pairs, function return values, and dictionary keys where immutability is required.
Type Casting
Type casting converts a value from one data type to another using built-in functions like `int()`, `float()`, `str()`, and `bool()`. It is frequently needed when data is loaded with incorrect types, such as numbers stored as strings or booleans stored as integers.
uv
uv is a Python package and project manager written in Rust. It handles dependency installation, virtual environments, Python version management, and project builds in a single tool, replacing the need for separate tools like pip, pip-tools, virtualenv, and pyenv. It resolves and installs packages significantly faster than pip because of its Rust-based resolver and aggressive caching. Developers use it to create isolated environments, lock dependencies for reproducible builds, and run scripts with inline dependency declarations.
Variance & Std. Deviation
Variance measures the average squared deviation from the mean, and standard deviation is its square root in the original units. Both quantify how spread out data values are. Pandas computes them with `var()` and `std()`. A high standard deviation relative to the mean signals high variability in the data.
virtualenv / venv
`venv` is Python's built-in tool for creating isolated virtual environments. Each environment has its own Python interpreter and installed packages, preventing conflicts between projects. `virtualenv` is a third-party alternative with additional features. Virtual environments are a best practice for keeping project dependencies separate.
Visual Inspection
Visual inspection uses charts to identify outliers and anomalies that statistical thresholds might miss. Box plots show the IQR and flag points beyond the whiskers. Scatter plots reveal isolated points far from the main cluster. Histograms expose unusual spikes or gaps. Visual inspection is always a valuable complement to numerical outlier detection.
VS Code
Visual Studio Code is a lightweight, extensible code editor that supports Python development through extensions. With the Python and Jupyter extensions installed, VS Code supports notebooks, debugging, linting, and IntelliSense. It is a popular choice for analysts who want more IDE features than a browser-based notebook provides.
Working with Strings
Python provides a rich set of string methods for manipulating text: `strip()`, `split()`, `replace()`, `upper()`, `lower()`, `startswith()`, `endswith()`, and many more. These methods are applied directly to strings or through Pandas' `.str` accessor for vectorized text cleaning on entire columns.
Z-score
A Z-score measures how many standard deviations a value is from the mean. Values with a Z-score above 3 or below −3 are commonly flagged as outliers. Z-scores are calculated using `scipy.stats.zscore()` or manually and work best when the data is approximately normally distributed.