数据清洗过程中常见的排序和去重操作
转载
数据操作中排序和去重是比较常见的数据操作，本专题对排序和去重做专门介绍，并且给出一种不常用却比较有启发意义的示例：多列无序去重
目 录
1 排序
1.1 sort 单列排序返回值
1.2 order 单列排序返回索引
1.3 rank 单列排序返回“秩”
1.4 arrage 多列排序
1.5、reorder 用在绘图中
2 去重
2.1 unique 单向量/多列完全重复去重
2.2 duplicated函数
3 多列无序去重
说明：多列无序重复比较值得学习
正 文
1 排序
1.1 sort 单列排序返回值
总结：sort是直接对向量排序，返回原数值
#sort相关语法
sort(x, decreasing = FALSE, ...)
## Default S3 method:
sort(x, decreasing = FALSE, na.last = NA, ...)
sort.int(x, partial = NULL, na.last = NA, decreasing = FALSE,
method = c("auto", "shell", "quick", "radix"), index.return = FALSE)
sort示例
> set.seed(416)
> x <- round(runif(10,1,20))
> x;sort(x)
[1] 9 13 7 13 20 16 4 1 6 17
[1] 1 4 6 7 9 13 13 16 17 20 #可以发现sort函数是对原始向量进行排序
#如果遇到矩阵，sort函数会将矩阵转换为向量
> set.seed(416)
> x <- round(runif(10,1,20))
> y <- matrix(x,nrow = 5)
> y;sort(y)
[,1] [,2]
[1,] 9 16
[2,] 13 4
[3,] 7 1
[4,] 13 6
[5,] 20 17
[1] 1 4 6 7 9 13 13 16 17 20 #sort(y)
1.2 order 单列排序返回索引
总结：order先对数值排序，然后返回排序后各数值的索引
#order相关语法
order(..., na.last = TRUE, decreasing = FALSE,
method = c("auto", "shell", "radix"))
order示例
> set.seed(416)
> x <- round(runif(10,1,20))
> x
[1] 9 13 7 13 20 16 4 1 6 17
> order(x)
[1] 8 7 9 3 1 2 4 6 10 5 #order返回x序列的索引值
> sort(x)
[1] 1 4 6 7 9 13 13 16 17 20
> x[order(x)]
[1] 1 4 6 7 9 13 13 16 17 20 #根据索引对x进行排序
#当遇到矩阵时，order将按列对原始矩阵进行排序，并且返回其索引向量
> set.seed(416)
> x <- round(runif(10,1,20))
> y <- matrix(x,nrow = 5)
> y
[,1] [,2]
[1,] 9 16
[2,] 13 4
[3,] 7 1
[4,] 13 6
[5,] 20 17
> order(y)
[1] 8 7 9 3 1 2 4 6 10 5 #str(order(y)) 返回int
> sort(y)
[1] 1 4 6 7 9 13 13 16 17 20
> y[order(y)]
[1] 1 4 6 7 9 13 13 16 17 20
1.3 rank 单列排序返回“秩”
总结：rank返回原数据各项排名（有并列的情况）
概念解释：秩是基于样本值的大小在全体样本中所占位次(秩)的统计量。
#rank语法
rank(x, na.last = TRUE,
ties.method = c("average", "first", "last", "random", "max", "min"))
rank示例
> set.seed(416)
> x <- round(runif(10,1,20))
> x
[1] 9 13 7 13 20 16 4 1 6 17
> rank(x) #rank返回x中每个元素的秩
[1] 5.0 6.5 4.0 6.5 10.0 8.0 2.0 1.0 3.0 9.0
1.4 arrage 多列排序
总结：arrange是dplyr包中的排序函数，可对数据框以列的形式进行因子排序
> library(dplyr) #加载dplyr
> arrange(mtcars, cyl, disp) #对mtcars数据框按照cyl和disp升序排序
mpg cyl disp hp drat wt qsec vs am gear carb
1 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
2 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
……
6 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
7 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
……
23 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
……
26 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
27 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
……
32 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
> arrange(mtcars, desc(disp)) #对mtcars数据框按照cyl升序和和disp降序排序
mpg cyl disp hp drat wt qsec vs am gear carb
1 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
2 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
3 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
……
12 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
13 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
14 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
15 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
……
27 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
28 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
……
32 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
1.5、reorder 用在绘图中
1.5.1 在graphics绘图系统中
require(graphics)
bymedian <- with(InsectSprays, reorder(spray, count, median))
boxplot(count ~ bymedian, data = InsectSprays,
xlab = "Type of spray", ylab = "Insect count",
main = "InsectSprays data", varwidth = TRUE,
col = "lightgray")
1.5.2 比如ggplot中绘条形图使x轴按y轴数值大小排序
说明：reorder函数具有对排序变量的因子化作用
> attach(mtcars)
> str(reorder(gear,disp))
Factor w/ 3 levels "4","5","3": 1 1 1 3 3 3 3 1 1 1 ...
- attr(*, "scores")= num [1:3(1d)] 326 123 202
..- attr(*, "dimnames")=List of 1
.. ..$ : chr [1:3] "3" "4" "5"
> str(factor(gear))
Factor w/ 3 levels "3","4","5": 2 2 2 1 1 1 1 2 2 2 ...
> detach(mtcars)
library(ggplot2)
data(mtcars)
head(mtcars)
ggplot(mtcars,aes(x=reorder(gear,disp), y= disp)) + geom_boxplot() + labs(title = "图1")
ggplot(mtcars,aes(x=factor(gear), y= disp)) + geom_boxplot() + labs(title = "图2")
2 去重
2.1 unique 单向量/多列完全重复去重
总结：unique中，R中默认的是fromLast=FALSE，即若样本点重复出现，则取首次出现的；否则去最后一次出现的。列名不变，去掉重复样本值之后的行名位置仍为原先的行名位置。
> df <- data.frame(x = c("A","B","C","D","E","B","C","B"), y = c("B","A","D","E","B","C","A","A"))
> df
x y
1 A B
2 B A
3 C D
4 D E
5 E B
6 B C
7 C A
8 B A
> unique(df)
x y
1 A B
2 B A
3 C D
4 D E
5 E B
6 B C
7 C A
> unique(df,fromLast = TRUE)
x y
1 A B
3 C D
4 D E
5 E B
6 B C
7 C A
8 B A
2.2 duplicated函数
总结：duplicated可对原数据框做单列或多列去重，并且返回波尔向量（索引）
> df <- data.frame(x = c("A","B","C","D","E","B","C","B"), y = c("B","A","D","E","B","C","A","A"))
> df
x y
1 A B
2 B A
3 C D
4 D E
5 E B
6 B C
7 C A
8 B A
> df_index <- duplicated(df$x) #构建一个布尔向量（索引）
> df_index
[1] FALSE FALSE FALSE FALSE FALSE TRUE TRUE TRUE
> df[!df_index,] #筛选
x y
1 A B
2 B A
3 C D
4 D E
5 E B
3 多列无序去重
总结：多列无序去重指，多列非按照独立列比较重复，而是指逐行比较每一行是否出现过此元素（不按照列顺序）.
例如：matrix(c("a","b"),nrow = 1) 和 matrix(c("b","a"),nrow = 1)也是重复
> data.frame(matrix(c("a","b"),nrow = 1))
X1 X2
1 a b
> data.frame(matrix(c("b","a"),nrow = 1))
X1 X2
1 b a多列无序去重示例
#生成测试集
> df <- data.frame(x = c("A","B","C","D","E","B","C","B"), y = c("B","C","D","E","B","C","A","A"),z = c(1:8))
#对数据集df[,c(1:2)]逐行操作排序，并将排序后结果合并
> df$merge <- apply(df[,c(1:2)],1,function(x) paste(sort(x),collapse=''))
#对逐行排序合并的结果进行去重，返回索引向量，然后（反向！）筛选
> df_du<-df[!duplicated(df$merge),]
> df
x y z merge
1 A B 1 AB
2 B C 2 BC
3 C D 3 CD
4 D E 4 DE
5 E B 5 BE
6 B C 6 BC
7 C A 7 AC
8 B A 8 AB
> df_du
x y z merge
1 A B 1 AB
2 B C 2 BC
3 C D 3 CD
4 D E 4 DE
5 E B 5 BE
7 C A 7 AC
