Sage-Tips

Is logistic regression robust to outliers?

Logistic regression methods have many applications in Health Sciences. The problem with maximum likelihood estimators is that they are not ‘robust’, i.e., their sensitivity to outliers could be arbitrarily large, and a minority of outliers could lead to a wrong logistic model.

Why is logistic regression sensitive to outliers?

Logistic Regression models are not much impacted due to the presence of outliers because the sigmoid function tapers the outliers. But the presence of extreme outliers may somehow affect the performance of the model and lowering the performance.

Why least square method is not used in logistic regression?

The structure of the logistic regression model is designed for binary outcomes. Least Square regression is not built for binary classification, as logistic regression performs a better job at classifying data points and has a better logarithmic loss function as opposed to least squares regression.

What is outliers in logistic regression?

In logistic regression, a set of observations whose values deviate from the expected range and produce extremely large residuals and may indicate a sample peculiarity is called outliers. These outliers can unduly influence the results of the analysis and lead to incorrect inferences.

Is logistic regression robust?

We consider logistic regression with arbitrary outliers in the covariate matrix. We propose a new robust logistic regression algorithm, called RoLR, that estimates the parameter through a simple linear programming procedure. We prove that RoLR is robust to a constant fraction of adversarial outliers.

How does logistic regression deal with outliers?

In logistic regression, a set of observations whose values deviate from the expected range and produce extremely large residuals and may indicate a sample peculiarity is called outliers. An observation with an extreme value on a predictor variable is called a point with high leverage.

Is logistic regression sensitive to outliers that is will the results of logistic regression change drastically due to the presence of outliers?

Logistic regression is affected by the outliers as we can see in the diagram above. SVM is not very robust to outliers. Presence of a few outliers can lead to very bad global misclassification. Algorithm is sensitive to outliers, since a single mislabeled example dramatically changes the class boundaries.

When should I use robust regression?

Robust regression is an alternative to least squares regression when data is contaminated with outliers or influential observations and it can also be used for the purpose of detecting influential observations.

How do outliers affect regression?

With respect to regression, outliers are influential only if they have a big effect on the regression equation. Sometimes, outliers do not have big effects. For example, when the data set is very large, a single outlier may not have a big effect on the regression equation.