Imputing outliers in tibble after identifying them
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
asked Nov 21 '18 at 22:38
NinaNina
53
add a comment |
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
asked Nov 21 '18 at 22:38
NinaNina
53
add a comment |
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
asked Nov 21 '18 at 22:38
NinaNina
53
I've written a function using this article (http://www.questionflow.org/2017/12/26/combined-outlier-detection-with-dplyr-and-ruler/) I found online which flags any outliers within my tbl based on either being outside of z-score threshold, or outside of 1.5*IQR. I want to enhance my function by adding an "impute" argument which can be set to mean, median, or mode, (or none, in which case the output is just a tbl of logicals) which will set the values that were flagged to the input argument for that column. How can I go about doing that? My code so far:
# Detecting outliers based on two outlier metrics (3 out of z-score or 1.5 out of IQR)
# Defining the functions to detect outliers
isnt_out_z <- function(x, thres = 3, na.rm = TRUE) {
abs(x - mean(x, na.rm = na.rm)) <= thres * sd(x, na.rm = na.rm)
}
isnt_out_IQR <- function(x, k = 1.5, na.rm = TRUE) {
quar <- quantile(x, probs = c(0.25, 0.75), na.rm = na.rm)
iqr <- diff(quar)
(quar[1] - k * iqr <= x) & (x <= quar[2] + k * iqr)
}
# Column-based non-outlier rows: row is not an outlier based on some column if it doesn't contain outlier (computed based on target column) on the intersection with that column
## Useable functions
find_Outliers <- function(data, method = 'z', impute = none) {
if (method == 'z') {
data %>%
transmute_if(is.numeric, isnt_out_z)
} else {
data %>%
transmute_if(is.numeric, isnt_out_IQR)
}
}
r dataframe outliers tbl
r dataframe outliers tbl
asked Nov 21 '18 at 22:38
NinaNina
53
asked Nov 21 '18 at 22:38
NinaNina
53
asked Nov 21 '18 at 22:38
NinaNina
53
asked Nov 21 '18 at 22:38
NinaNina
53
asked Nov 21 '18 at 22:38
NinaNina
53
53
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421412%2fimputing-outliers-in-tibble-after-identifying-them%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53421412%2fimputing-outliers-in-tibble-after-identifying-them%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
