DetectOutliers

DetectOutliers provides users with a list of the outliers detected in an array of numbers or strings.

This is useful in any case where you wish to identify unusual values in a set of values to extract them, delete them, or otherwise examine their properties.

DetectOutliers uses three common outlier detection algorithms to identify unusual values. Values identified by all three algorithms are returned in DetectOutliers. Our current simulations suggest this 'hybrid' method is more effective at detecting synthetic outliers than much more computationally advanced methods, and more efficient methods are added as they are published in peer-reviewed research.

In all cases, the mathematical difference in these algorithms lies in how the range of 'normal' values is calculated. The upper and lower bounds of this range are commonly called 'fences' in outlier detection literature.

The Interquartile Range (IQR) algorithm determines outlier values to be those that sit outside 1.5 times the range between the first and third quartiles of the data.

We can formulate this method as:

$$X_i=\{\begin{array}{}{x}^2& \text{if}{X}_i<\mathrm{med(X)}-1.5\mathrm{IQR}x+1& \text{if}{X}_n>\mathrm{med(X)}+1.5\mathrm{IQR}\end{array}$$

The IQR can be written as:

$$\mathrm{IQR}=Q_1-Q3$$
Where Q represents a quartile of the data, i.e. the medians of the upper and lower half of the data when sorted by value.

The standard deviation algorithm assumes a relatively normal distribution of scores, returning a positive result for any values outside 1.96 standard deviations above and below the mean of the data. As in all applications of this type of frequentist statistics, we use 1.96 as it represents 95% of the data (i.e. 2.5% of the distribution on each side will be considered outliers.)

p> We can formulate this method as:

$$X_i=\{\begin{array}{}{x}^2& \text{if}{X}_i<\mathrm{med(X)}-1.96\mathrm{SD}x+1& \text{if}{X}_n>\mathrm{med(X)}+1.96\mathrm{SD}\end{array}$$

Where the standard deviation of the data is taken as:

SD = ∑ i = 1 N ( x i - μ ) 2 N

Note that in the case of DetectOutliers we use the SD for a population rather than for a sample. This is appropriate for the case of the API, as the submitted data is the full population of points to be considered in the analysis, and we are not tring to extend our inference to other datasets.

Median Absolute Deviation (MAD) is a useful algorithm in the case where data are skewed, as it is based on medians rather than means (which can be easily affected by large values). It can be seen as the median of deviations from the median value of the data.

We can formulate this method as:

$$X_i=\{\begin{array}{}{x}^2& \text{if}{X}_i<\mathrm{med(X)}-\mathrm{MAD}x+1& \text{if}{X}_n>\mathrm{med(X)}+\mathrm{MAD}\end{array}$$

Where the MAD is defined as:

MAD = ∑ i = 1 n | x i − x | n

The set of elements in the data are considered outliers if all three algorithms detect them. We can write this in set notation as:

$$X_{(outliers)}={\mathrm{IQR}}_{(outliers)}\cap {\mathrm{SD}}_{(outliers)}\cap {\mathrm{MAD}}_{(outliers)}$$