Ensemble Methods: Bagging vs Boosting

Ensemble Methods

The main causes of error in learning are due to noise, bias, and variance. Ensemble helps to minimize these factors. These methods are designed to improve the stability and the accuracy of Machine Learning algorithms.

The two main types of Ensemble methods are Bagging and Boosting.

In this blog, I will explain the difference between Bagging and Boosting ensemble methods.


The process for training an ensemble through bagging is as follows:

  1. Grab a sizable sample from your dataset, with replacement
  2. Train a classifier on this sample
  3. Repeat until all classifiers have been trained on their own sample from the dataset
  4. When making a prediction, have each classifier in the ensemble make a prediction
  5. Aggregate all predictions from all classifiers into a single prediction, using the method of your choice


The process for training an ensemble through boosting is as follows:

  1. The base algorithm reads the data and assigns equal weight to each sample observation.
  2. False predictions made by the base learner are identified. In the next iteration, these false predictions are assigned to the next base learner with a higher weightage on these incorrect predictions.
  3. Repeat step 2 until the algorithm can correctly classify the output.

Therefore, the main aim of Boosting is to focus more on miss-classified predictions.

Bagging vs. Boosting

When there is a low-performance issue, the bagging technique will not result in a better bias. However, the boosting technique generates a unified model with lower errors since it concentrates on the optimization of the advantages and reduction of shortcomings in a single model.

When the challenge in a single model is overfitting, the bagging method performs better than the boosting technique. Boosting faces the challenge of handling over-fitting since it comes with over-fitting in itself.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store