[Analysis] Overfitting and Underfitting – Bias | Variance | Cross-validation

  • Blog|FEMA & Banking|
  • 7 Min Read

Overfitting, underfitting, bias, variance and cross-validation explained from IIBF’s Emerging Technologies courseware, with ways to prevent overfitting and a banking example.

  • By Taxmann
  • |
  • Last Updated on 14 November, 2026

Overfitting and Underfitting

Overfitting and underfitting are the two main reasons machine learning algorithms perform poorly. An overfitted model learns the noise in its training data along with the real pattern, so its error rates on test data are high; it has low bias and high variance. An underfitted model is too simple to capture the pattern at all; it has high bias and low variance. The IIBF courseware on Emerging Technologies explains the bias-variance trade-off behind both, and techniques such as cross-validation and regularisation that guard against overfitting.

Table of Contents

  1. What are overfitting and underfitting?
  2. What causes overfitting?
  3. What causes underfitting?
  4. What are bias and variance in machine learning?
  5. What is the bias-variance trade-off?
  6. What is cross-validation?
  7. How can overfitting be prevented?
  8. Why does generalisation matter for a bank’s models?
  9. Quick answers

1. What are overfitting and underfitting?

Overfitting means a model has learned its training data too closely, noise included, and generalises poorly to new data. Underfitting means a model has not learned the problem from its training data well enough, so it can neither model the training data nor generalise to new data.

Both terms describe how well a model learns and generalises. Generalisation is how well a trained model can classify or forecast data it has never seen. A successful model should generalise well from its training data to any data in the problem domain, because that is what lets it make predictions about the future. They are defined in terms of supervised learning, in which models are trained on labelled data.

Point of comparison Overfitting Underfitting
What goes wrong The model learns noise and random fluctuations in the training data as if they were patterns The model fails to capture the underlying pattern in the data
Bias and variance Low bias, high variance High bias, low variance
Typical cause A model that is overtrained or too complex, or trained on a noisy dataset A model that is too simple, trained for too little time or with too few inputs
Result High error rates on test data and poor generalisation to new data Low performance even on the training data
How it is handled Resampling, a held-back dataset, cross-validation and regularisation Trying other machine learning algorithms

2. What causes overfitting?

Overfitting happens when a model learns so much detail and noise from its training data that its performance on new data suffers. The model picks up random fluctuations in the training data and learns them as concepts, but those concepts do not apply to new data, so the model generalises badly.

An overfitted model captures both the noise and the underlying pattern in the data, and it shows low bias and high variance. The usual causes are a model that has been overtrained or is too complex, and training on a noisy dataset. The result is high error rates on test data. The courseware calls overfitting the most prevalent issue in practice.

Taxmann.com | Research | Subscribe Now!

3. What causes underfitting?

Underfitting happens when a model is too simple to capture the relationship between the input and output variables. It can result from too little training time or too few inputs. A suitable model can be built from limited data, but it is harder when the relationship is non-linear.

Underfitted models show high bias and low variance. Models of this kind, like linear and logistic regression, are easy to use and understand. Underfitting gets less attention than overfitting because a good performance metric shows it up easily. The fix is to experiment with other machine learning algorithms.

4. What are bias and variance in machine learning?

Bias and variance are two kinds of error in a model’s predictions. Bias comes from wrong or overly simple assumptions in the learning algorithm. High bias leads to underfitting, and the model then struggles to predict accurately and to carry what it learned from the training set over to the test set.

Variance comes from using a learning algorithm that is too complex for the task. The algorithm becomes extremely sensitive to variation in the training data, carries too much of its noise into the model and overfits, so it is of little use on test data.

A third component, irreducible error, cannot be removed by this or any other model. It comes from factors outside anyone’s control, such as statistical noise in the observations, and it measures how unpredictable the noise in the data is.

5. What is the bias-variance trade-off?

The bias-variance trade-off is the balance a model has to strike between these two errors, and it is a central problem in supervised learning. The bias-variance decomposition breaks a learning algorithm’s expected generalisation error into three parts: bias, variance and irreducible error.

As a model becomes more complex and takes in more variables, its bias falls and its variance rises. Neither high bias nor high variance is a desirable characteristic. The ideal is a model with low bias and low variance, and a model with an ideal balance of the two will be neither overfit nor underfit. Building a good model therefore means balancing bias and variance to minimise the total error.

There is a practical limit. The actual bias and variance of a predictive modelling problem cannot be calculated, because the true mapping function is not known. Bias, variance, irreducible error and the trade-off are used instead as tools to select models, configure them and interpret results.

6. What is cross-validation?

Cross-validation is a statistical technique for evaluating a model’s performance and accuracy. When the amount of data is limited, it is used to prevent overfitting. A model can be tested on the same data used to train it, but that may not be the best way to judge it.

The chapter lists eight methods: leave-p-out, leave-one-out, holdout, repeated random sub-sampling, k-fold, stratified k-fold, time series and nested cross-validation.

K-fold cross-validation is used when only a small quantity of data is available, to get an unbiased estimate of the model’s performance. The data is divided into k equal-sized subsets, or folds. In each round, one fold is held out as validation data and the remaining k-1 folds are used for training. This is repeated k times, so that every fold serves once as the validation set, and the results are averaged at the end. When k equals the sample size, the method is known as leave-one-out.

7. How can overfitting be prevented?

Overfitting is reduced by testing the model on data it has not trained on and by limiting how complex the model can become. The techniques include:

  • Resampling techniques and a held-back dataset kept for verification.
  • Cross-validation, especially when data is limited.
  • Regularisation, which extends the training of a linear model so that it minimises both the sum of squared errors on the training data and the model’s complexity. Lasso regression (L1 regularisation) minimises the absolute sum of the coefficients, and ridge regression (L2 regularisation) minimises the squared absolute sum. Both help when input values have collinearity and ordinary least squares would overfit the training data.
  • Shrinkage, which fits a model with all the predictors but shrinks their coefficient values toward zero. Shrinkage, or regularisation, reduces variance and can also perform variable selection.
  • Pruning a regression tree, which checks the predictive usefulness of every node and replaces with leaves the nodes that do not improve prediction quality on new data.
  • Dimensionality reduction, which can ease overfitting where a model performs well on a small data set but does not generalise well to new data.

8. Why does generalisation matter for a bank’s models?

A model has to work on new examples that were not in its training set, and the training set cannot show whether it does. That is why a separate test set is used. It gives an unbiased evaluation of how well the algorithm has learned, at the cost of less data for training, a trade-off that has to be made. The testing set should be kept distinct from the training and validation sets, yet follow the same probability distribution as the training set.

The steps for building an AI-based credit appraisal system apply this directly. The bank splits its dataset into training and testing sets, trains the models and adjusts their hyperparameters on the first, and validates them on the second to check that they generalise well to new data. It then validates the whole system on historical data and simulated real-world scenarios, and keeps updating and retraining the models so that they stay accurate and relevant.

9. Quick answers

What is overfitting in machine learning?

An overfitted model captures the noise in its training data along with the underlying pattern. It has low bias and high variance, and it makes many errors on test data and new data.

What is underfitting?

Underfitting occurs when a model is too simple to capture the underlying pattern in the data, often because of too little training time or too few inputs. It has high bias and low variance.

What is the difference between overfitting and underfitting?

Overfitting is learning the training data too closely, noise included, so the model generalises poorly. Underfitting is failing to learn the problem from the training data well enough. The first has low bias and high variance, the second high bias and low variance.

What are bias and variance?

Bias is error from wrong or overly simple assumptions in the learning algorithm. Variance is error from an algorithm that is too complex for the task and too sensitive to variation in the training data.

What is the bias-variance trade-off?

As a model becomes more complex, its bias falls and its variance rises. The trade-off is finding the balance that minimises total error, since neither high bias nor high variance is desirable.

What is cross-validation?

Cross-validation is a statistical technique for evaluating a model’s performance and accuracy. When data is limited, it is used to prevent overfitting.

How does k-fold cross-validation work?

The data is divided into k equal-sized folds. Each fold in turn is held out as validation data while the rest are used for training, the process runs k times, and the results are averaged.

How can overfitting be prevented?

Techniques include resampling with a held-back dataset, cross-validation, regularisation such as lasso (L1) and ridge (L2) regression, shrinkage, pruning regression trees and dimensionality reduction.


Chapter 4 of Emerging Technologies (2026 Edition), the Indian Institute of Banking & Finance’s courseware for its Certificate Course on Emerging Technologies, published by Taxmann, takes the stages of machine learning from data collection to model testing, and explains bias, variance, the bias-variance trade-off, shrinkage, generalisation and cross-validation, along with regression models, supervised and unsupervised learning and machine learning use cases in banks. Its terminal questions ask candidates to explain overfitting and underfitting and the trade-off between bias and variance.

For the rules that banks work under, Taxmann.com | Research, FEMA & Banking brings together the Acts, Rules, circulars and notifications, case laws and articles on FEMA, banking and insurance. Taxmann.com | Store also lists IIBF’s courseware for its other certificate courses, including Information System for Banks and IT Security.

Related reading: RBI’s draft guidance on model risk management.

More in this series: emerging technologies in banking · types of machine learning · AI in banking · types of artificial intelligence · types of data analytics.

Based on the 2026 Edition of Emerging Technologies, IIBF’s courseware for the Certificate Course on Emerging Technologies.

Disclaimer: The content/information published on the website is only for general information of the user and shall not be construed as legal advice. While the Taxmann has exercised reasonable efforts to ensure the veracity of information/content published, Taxmann shall be under no liability in any manner whatsoever for incorrect information, if any.

Leave a Reply

Your email address will not be published. Required fields are marked *

Everything on Tax and Corporate Laws of India

To subscribe to our weekly newsletter please log in/register on Taxmann.com

Author: Taxmann

Taxmann Publications has a dedicated in-house Research & Editorial Team. This team consists of a team of Chartered Accountants, Company Secretaries, and Lawyers. This team works under the guidance and supervision of editor-in-chief Mr Rakesh Bhargava.

The Research and Editorial Team is responsible for developing reliable and accurate content for the readers. The team follows the six-sigma approach to achieve the benchmark of zero error in its publications and research platforms. The team ensures that the following publication guidelines are thoroughly followed while developing the content:

  • The statutory material is obtained only from the authorized and reliable sources
  • All the latest developments in the judicial and legislative fields are covered
  • Prepare the analytical write-ups on current, controversial, and important issues to help the readers to understand the concept and its implications
  • Every content published by Taxmann is complete, accurate and lucid
  • All evidence-based statements are supported with proper reference to Section, Circular No., Notification No. or citations
  • The golden rules of grammar, style and consistency are thoroughly followed
  • Font and size that's easy to read and remain consistent across all imprint and digital publications are applied