A. BAGGING

The term “bagging” comes from bootstrap + aggregating. The idea behind bagging is combining a bunch of individual weak models to learn the same task with the same goal in a parallel way. Each weak model is trained by a random subset of the data sampled with replacement (bootstrapping), and then the models’ predictions are aggregated.

This image above shows that the bagging example has three steps:

1. Bootstrap data(Sampling)
2. Aggregation or Model fit
3. Combination of different models with Result aggregation.

Part 1 – Bootstrapping: Bootstrapping is a sampling method, where a sample is chosen out of a set, using the replacement method. The learning algorithm is then run on the samples selected. Below example can be created for selecting random data point instances and combining them which is shown in the below figure.

Part 2 – Aggregation: Model predictions undergo aggregation to combine them for the final prediction to consider all the possible outcomes. 

The aggregation or Model training is explained in the below example. Here we have selected each boot instance and tried to make a classifier here we have used a decision tree classifier for each sample. Update the status of result of the classifier based on actual and predicted value.

Bagging works as follows:-

1. Consider there are n observations and m features in the training set. You need to select a random sample from the training dataset without replacement
2. A subset of m features is chosen randomly to create a model using sample observations
3. The feature offering the best split out of the lot is used to split the nodes
4. The tree is grown, so you have the best root nodes
5. The above steps are repeated n times. It aggregates the output of individual decision trees to give the best prediction.

Mathematically, Bagging is represented by the following formula,

The term on the left hand side is the bagged prediction, and terms on the right hand side are the individual learners.

Common Bagging algorithms:

1. Bagging meta-estimator
2. Random forest

To know more about implementation of bagging :

https://www.simplilearn.com/tutorials/machine-learning-tutorial/bagging-in-machine-learning

To build a bagging classifier from scratch click here: https://machinelearningmastery.com/implement-bagging-scratch-python/

References:

1] https://medium.com/ml-research-lab/bagging-ensemble-meta-algorithm-for-reducing-variance-c98fffa5489f