What Is Principal Component (PCA)

Principal Component Analysis 

PCA, or Principal Component Analysis, is a statistical technique used to identify patterns and relationships in large data sets. It works by reducing the dimensionality of the data by identifying and extracting the most important features or variables that contribute to the variation in the data.

In simpler terms, think of it as a way to simplify complex data into a smaller, more manageable set of information. This smaller set of information still captures the essence of the original data, but in a way that is easier to analyze and visualize.

For example

                        Imagine you have a dataset with many different variables, such as age, income, education level, and occupation. With PCA, you can identify which of these variables are most important for explaining the variation in the data, and then focus on analyzing those variables more closely. This can help you gain insights into patterns and relationships in the data that might not have been immediately apparent before.

PCA is used for several reasons, including:

Dimensionality reduction

                                                   PCA is a technique for reducing the number of variables in a dataset while still retaining the most important information. By reducing the dimensionality of the data, it becomes easier to analyze and visualize.

Feature extraction

                                      PCA can help identify the most important features or variables that contribute to the variation in the data. This can be useful for feature selection and identifying which variables are most important for modeling or prediction.

Data visualization

                                     PCA can be used to plot high-dimensional data in two or three dimensions, making it easier to visualize patterns and relationships in the data.

Noise reduction

                                  PCA can help remove noise or unwanted variability in the data by identifying and removing the dimensions that contain the least amount of information.

Speeding up machine learning algorithms

                                                                             PCA can be used to preprocess data before applying machine learning algorithms, which can lead to faster and more accurate predictions.

Overall, PCA is a powerful technique for exploring and analyzing complex data, and it has many practical applications in fields such as finance, marketing, biology, and engineering.   

1 - Standardization of the data

PCA (Principal Component Analysis) is a popular dimensionality reduction technique used in machine learning and data analysis. Before applying PCA, it is often necessary to standardize the data to ensure that the variables are on the same scale and have similar variances.

Standardization involves subtracting the mean of each variable from the data points and then dividing by the standard deviation. This process transforms the data so that it has a mean of zero and a standard deviation of one.

Standardization is important in PCA because the principal components are sensitive to the scale of the variables. If the variables are not standardized, those with larger variances will dominate the analysis and overshadow the contributions of the variables with smaller variances. This can lead to incorrect conclusions about the relationships between variables.

By standardizing the data, we ensure that each variable contributes equally to the analysis, and the principal components are based on the covariance matrix of the variables rather than the correlation matrix. This approach is preferred when the variables are measured on different scales or have different units of measurement.

Overall, standardization is a crucial step in PCA as it helps to ensure the accuracy and reliability of the results.

                                           Where:

                                                       Z is the standardized value

                                                       X is the original value

                                                       μ is the mean of the data set

                                                       σ is the standard deviation of the data set

                                    

2 - Covariance Matrix 

A covariance matrix is a mathematical concept used to describe the relationships between multiple variables. It's essentially a square matrix that contains the covariances of pairs of variables.

The covariance between two variables is a measure of how they vary together. If the two variables tend to increase or decrease together, they have a positive covariance. If one variable tends to increase while the other decreases, they have a negative covariance.

To compute a covariance matrix, you first need to calculate the means of each variable. Then, for each pair of variables, you calculate the covariance as the average of the product of their deviations from their respective means.

Once you have computed all the covariances, you arrange them in a matrix. The diagonal of the matrix contains the variances of each variable (which is the covariance of a variable with itself), while the off-diagonal elements contain the covariances between pairs of variables.

The resulting covariance matrix can provide insights into how variables relate to each other, which can be useful in statistical analysis and modeling.

3 - Calculation of Eigenvector & Eigenvalue 

Eigenvalues and eigenvectors play a crucial role in PCA. The eigenvalues represent the amount of variance explained by each principal component, and the eigenvectors represent the direction of the principal component.

Here's how to calculate eigenvectors and eigenvalues in PCA:

1 - Standardize the data: Subtract the mean of each variable from the respective values and divide by the standard deviation.

2 - Compute the covariance matrix: Calculate the covariance matrix of the standardized data.

3 - Compute the eigenvectors and eigenvalues: Calculate the eigenvectors and eigenvalues of the covariance matrix. The eigenvectors represent the principal components, and the corresponding eigenvalues represent the amount of variance explained by each principal component.

4 - Sort the eigenvectors and eigenvalues: Sort the eigenvectors and eigenvalues in descending order of the eigenvalues.

5 - Choose the principal components: Choose the top k eigenvectors corresponding to the k largest eigenvalues to represent the data in k dimensions.

6 - Calculate the transformed data: Multiply the standardized data by the selected eigenvectors to obtain the transformed data in k dimensions.

To summarize, the eigenvectors and eigenvalues in PCA can be calculated by standardizing the data, computing the covariance matrix, and then computing the eigenvectors and eigenvalues of the covariance matrix. The eigenvectors represent the direction of the principal components, and the eigenvalues represent the amount of variance explained by each principal component.

4 - Computing the Principle Components

Once we have computed the Eigenvectors and Eigenvalues all we have to do is order them in the descending order where the eigenvectors with the highest eigenvalues is the most significant and thus forms the first principal component  

5 - Reducing the dimension of the data

Reducing the dimension of data refers to the process of decreasing the number of features or variables in a dataset while retaining the essential information. Dimensionality reduction is often performed on large datasets with high dimensionality to simplify the analysis, reduce computational complexity, and improve the accuracy of machine learning models.