《R for Data Science》第七章 Exploratory Data Analysis 啃书知识点积累
参考书籍
- 《R for data science》
- 《R数据科学》
“There are no routine statistical questions, only questionable statistical routines.” — Sir David Cox
“Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.” — John Tukey
变动
- 连续变量等宽分箱并统计区间数
A variable is continuous if it can take any of an infinite set of ordered values.
Numbers and date-timesare two examples of continuous variables.
diamonds %>%
count(cut_width(carat, 0.5))
# # A tibble: 11 x 2
# `cut_width(carat, 0.5)` n
# <fct> <int>
# 1 [-0.25,0.25] 785
# 2 (0.25,0.75] 29498
# 3 (0.75,1.25] 15977
# 4 (1.25,1.75] 5313
# 5 (1.75,2.25] 2002
# 6 (2.25,2.75] 322
# 7 (2.75,3.25] 32
# 8 (3.25,3.75] 5
# 9 (3.75,4.25] 4
# 10 (4.25,4.75] 1
# 11 (4.75,5.25] 1
- 可视化
# 直方图geom_histogram
smaller <- diamonds %>%
filter(carat < 3)
ggplot(data = smaller, mapping = aes(x = carat)) +
# binwidth设置分箱宽度
geom_histogram(binwidth = 0.1)
# 频率折线图geom_freqpoly
ggplot(data = smaller, mapping = aes(x = carat, color = cut)) +
geom_freqpoly(binwidth = 0.1)
- 加强对异常值的观测
When you have a lot of data, outliers are sometimes difficult to see in a histogram.
- 分析变量分布
- coord_cartesian() 函数拓宽坐标轴刻度
ggplot(diamonds) +
geom_histogram(mapping = aes(x = y), binwidth = 0.5)
ggplot(diamonds) +
geom_histogram(mapping = aes(x = y), binwidth = 0.5) +
coord_cartesian(ylim = c(0, 50)) # 也可以调整xlim
# 会忽略溢出坐标轴范围的那些数据
缺失值
- 处理异常值
If you’ve encountered unusual values in your dataset, and simply want to move on to the rest of your analysis, you have two options.
Drop the entire row with the strange values
diamonds2 <- diamonds %>%
filter(between(y, 3, 20))
replacing the unusual values with missing values(better)
diamonds2 <- diamonds %>%
mutate(y = ifelse(y < 3 | y > 20, NA, y))
- ggplot2绘图中含有NA
Like R, ggplot2 subscribes to the philosophy that missing values should never silently go missing.
# 会自动忽略NA但有警告
ggplot(data = diamonds2, mapping = aes(x = x, y = y)) +
geom_point()
#> #> Warning: Removed 9 rows containing missing values
#> (geom_point).
# 用na.rm = TRUE可以忽略NA
ggplot(data = diamonds2, mapping = aes(x = x, y = y)) +
geom_point(na.rm = TRUE)
相关变动
Covariation is the tendency for the values of two or more variables to vary together in a related way
分类变量 vs 连续变量
- 密度
多组比较，如果组间数量差距较大，可以用密度对计数标化
To make the comparison easier we need to swap what is displayed on the y-axis. Instead of displaying count, we’ll display
density, which is the count standardised so that the area under each frequency polygon is one.
library(patchwork) # 拼图包
library(RColorBrewer) # 配色包
display.brewer.all()
Set1 <- brewer.pal(name = "Set1",n=9)[1:5]
p1 <- ggplot(data = diamonds, mapping = aes(x = price)) +
geom_freqpoly(mapping = aes(color = cut), binwidth = 500)+
scale_color_manual(values = Set1)
p2 <- ggplot(data = diamonds, mapping = aes(x = price, y = ..density..)) +
geom_freqpoly(mapping = aes(color = cut), binwidth = 500) +
scale_color_manual(values = Set1)
p1 + p2
- reorder() 重排序
p1 <- ggplot(data = mpg, mapping = aes(x = class, y = hwy)) +
geom_boxplot()
p2 <- ggplot(data = mpg) +
geom_boxplot(
mapping = aes(
# 按照中位数重新排序
x = reorder(class, hwy, FUN = median),
y = hwy
)
)
# 变量名较长或出于美观可颠倒坐标轴
p3 <- ggplot(data = mpg) +
geom_boxplot(
mapping = aes(
x = reorder(class, hwy, FUN = median),
y = hwy
)
) +
coord_flip()
p1 + p2 + p3
两个分类变量
# geom_count
p1 <- ggplot(data = diamonds) +
geom_count(mapping = aes(x = cut, y = color))
# geom_tile
p2 <- diamonds %>%
count(color, cut) %>%
ggplot(mapping = aes(x = color, y = cut)) +
geom_tile(mapping = aes(fill = n)) +
coord_flip()
p1 + p2
- 组间数量差距较大，不易分析相互分布
# color在cut中的分布
p1 <- diamonds %>%
count(cut, color) %>%
group_by(cut) %>%
mutate(prop = n / sum(n)) %>%
ggplot(mapping = aes(x = cut, y = color)) +
geom_tile(mapping = aes(fill = prop))
# cut在color中的分布
p2 <- diamonds %>%
count(color, cut) %>%
group_by(color) %>%
mutate(prop = n / sum(n)) %>%
ggplot(mapping = aes(x = color, y = cut)) +
geom_tile(mapping = aes(fill = prop))
p1 + p2
- 利用因子确定分类变量
flights %>%
filter(!is.na(dep_delay)) %>%
group_by(month, dest) %>%
summarize(delay_mean = mean(dep_delay)) %>%
group_by(dest) %>%
filter(n() == 12) %>% # 过滤出一年就有数据的目的地
ungroup() %>%
ggplot(aes(x = factor(month),
# 重新排序
y = reorder(dest, delay_mean))) +
geom_tile(aes(fill = delay_mean)) +
xlab('Month') +
ylab('Destination')
更多内容见：[R语言] 与ggplot2相关有趣的包
两个连续变量
- overplot 过绘制的解决
- 使用 alpha 图形属性添加透明度（数据量过大则不适用）
p1 <- ggplot(data = diamonds) +
geom_point(mapping = aes(x = carat, y = price))
p2 <- ggplot(data = diamonds) +
geom_point(mapping = aes(x = carat, y = price),
alpha = 1 / 100)
p1 + p2
- 分箱
(1) 二维分箱
geom_bin2d()创建长方形分箱
geom_hex()创建六边形分箱
p1 <- ggplot(diamonds) +
geom_bin2d(mapping = aes(x = carat, y = price))
library(hexbin)
p2 <- ggplot(diamonds) +
geom_hex(mapping = aes(x = carat, y = price))
p1 + p2
(2) 对一个连续变量分箱
By default, boxplots look roughly the same (apart from number of outliers) regardless of how many observations there are, so it’s difficult to tell that each boxplot summarises a different number of points.
p1 <- diamonds %>%
filter(carat <= 2.8) %>%
ggplot(aes(x = carat, y = price)) +
geom_boxplot(aes(group = cut_width(carat, 0.1)))
# varwidth让观测数和箱宽正相关
p2 <- diamonds %>%
filter(carat <= 2.8) %>%
ggplot(aes(x = carat, y = price)) +
geom_boxplot(aes(group = cut_width(carat, 0.1)),varwidth = T)
p1 / p2
用cut_number()近似反映分箱内的数据数
ggplot(data = smaller, mapping = aes(x = carat, y = price)) +
geom_boxplot(mapping = aes(group = cut_number(carat, 20)))
模型和模式
Patternsprovide one of the most useful tools for data scientists because they reveal covariation. If you think of variation as a phenomenon that creates uncertainty, covariation is a phenomenon that reduces it. If two variables covary, you can use the values of one variable to make better predictions about the values of the second. If the covariation is due to a causal relationship (a special case), then you can use the value of one variable to control the value of the second.
