《Pandas Cookbook》第01章 Pandas基础

96
SeanCheney
6.9 2018.09.24 15:53* 字数 493

相较于《利用Python进行数据分析》,本书最大的特点是所有操作都变成了分解动作,而且每步都有详细讲解。但是,书写的有点啰嗦,而Jupyter Notebook又有些错。我对两者做了整合和总结。

第一遍整理完,还有许多地方不足,还要再弄。


第01章 Pandas基础
第02章 DataFrame运算
第03章 数据分析入门
第04章 选取数据子集
第05章 布尔索引
第06章 索引对齐
第07章 分组聚合、过滤、转换
第08章 数据清理
第09章 合并Pandas对象
第10章 时间序列分析
第11章 用Matplotlib、Pandas、Seaborn进行可视化


作者Theodore Petrou,Dunder Data创始人

公司网址,http://www.dunderdata.com(dunder是蒸馏朗姆酒的残留液体,取这个名字是类比数据分析过程)
GitHub地址:https://github.com/tdpetrou
领英个人页面:https://www.linkedin.com/in/tedpetrou
推特:https://twitter.com/tedpetrou?lang=en
Medium博客:https://medium.com/@petrou.theodore


下载代码:https://github.com/PacktPublishing/Pandas-Cookbook
下载本书 pdf:链接
下载本书 mobi:链接


# 引入pandas和numpy的约定
in[1]: import pandas as pd
       import numpy as np

1. DataFrame的结构

# 设定最大列数和最大行数
 in[2]: pd.set_option('max_columns', 8, 'max_rows', 10)
# 用read_csv()方法读取csv文件
# head()方法可以查看前五行,head(n)可以查看前n行
 in[3]: movie = pd.read_csv('data/movie.csv')
       movie.head()
out[3]:

2. 访问DataFrame的组件

in[4]: # 提取列索引
       columns = movie.columns
       # 提取行索引
       index = movie.index
       # 提取数据
       data = movie.values
 in[5]: columns
out[5]: Index(['color', 'director_name', 'num_critic_for_reviews', 'duration',
       'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name',
       'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name',
       'movie_title', 'num_voted_users', 'cast_total_facebook_likes',
       'actor_3_name', 'facenumber_in_poster', 'plot_keywords',
       'movie_imdb_link', 'num_user_for_reviews', 'language', 'country',
       'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes',
       'imdb_score', 'aspect_ratio', 'movie_facebook_likes'],
      dtype='object')
 in[6]: index
out[6]: RangeIndex(start=0, stop=4916, step=1)
 in[7]: data
out[7]: array([['Color', 'James Cameron', 723.0, ..., 7.9, 1.78, 33000],
       ['Color', 'Gore Verbinski', 302.0, ..., 7.1, 2.35, 0],
       ['Color', 'Sam Mendes', 602.0, ..., 6.8, 2.35, 85000],
       ..., 
       ['Color', 'Benjamin Roberds', 13.0, ..., 6.3, nan, 16],
       ['Color', 'Daniel Hsia', 14.0, ..., 6.3, 2.35, 660],
       ['Color', 'Jon Gunn', 43.0, ..., 6.6, 1.85, 456]], dtype=object)
 in[8]: # index的类型
        type(index) # pandas.core.indexes.range.RangeIndex
out[8]: pandas.core.indexes.range.RangeIndex
 in[9]: # columns的类型
        type(columns) # pandas.core.indexes.base.Index
out[9]: pandas.core.indexes.base.Index
 in[10]: # data的类型
         type(data) # numpy.ndarray
out[10]: numpy.ndarray
 in[11]: # 判断是不是子类型
         issubclass(pd.RangeIndex, pd.Index) # True
out[11]: True

更多

 in[12]: # 访问index的值
         index.values
         # index的值是个列表,所以可以索引或切片
         index.values[0]
out[12]: array([   0,    1,    2, ..., 4913, 4914, 4915])
 in[13]: # 访问columns的值
         columns.values
out[13]: array(['color', 'director_name', 'num_critic_for_reviews', 'duration',
       'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name',
       'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name',
       'movie_title', 'num_voted_users', 'cast_total_facebook_likes',
       'actor_3_name', 'facenumber_in_poster', 'plot_keywords',
       'movie_imdb_link', 'num_user_for_reviews', 'language', 'country',
       'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes',
       'imdb_score', 'aspect_ratio', 'movie_facebook_likes'], dtype=object)

3. 理解数据类型

 in[14]: movie = pd.read_csv('data/movie.csv')
         # 各列的类型
 in[15]: movie.dtypes
out[15]: color                       object
         director_name               object
         num_critic_for_reviews     float64
         duration                   float64
         director_facebook_likes    float64
                                     ...   
         title_year                 float64
         actor_2_facebook_likes     float64
         imdb_score                 float64
         aspect_ratio               float64
         movie_facebook_likes         int64
         Length: 28, dtype: object
 in[16]: # 显示各类型的数量
         movie.get_dtype_counts()
out[16]: float64    13
         int64       3
         object     12
         dtype: int64

4. 选择一列数据,作为Series

 in[17]: movie = pd.read_csv('data/movie.csv')
 in[18]: # 选择director_name这列
         movie['director_name']
out[18]: 0           James Cameron
         1          Gore Verbinski
         2              Sam Mendes
         3       Christopher Nolan
         4             Doug Walker
                    ...        
         4911          Scott Smith
         4912                  NaN
         4913     Benjamin Roberds
         4914          Daniel Hsia
         4915             Jon Gunn
         Name: director_name, Length: 4916, dtype: object
 in[19]: # 也可以通过属性的方式选取
         movie.director_name
out[19]: 0           James Cameron
         1          Gore Verbinski
         2              Sam Mendes
         3       Christopher Nolan
         4             Doug Walker
                    ...        
         4911          Scott Smith
         4912                  NaN
         4913     Benjamin Roberds
         4914          Daniel Hsia
         4915             Jon Gunn
         Name: director_name, Length: 4916, dtype: object
# 查看类型
 in[20]: type(movie['director_name'])
out[20]: pandas.core.series.Series

更多

 in[21]: director = movie['director_name']
         # 查看选取的列的名字
         director.name
out[21]: 'director_name'
 in[22]: # 单列Series转换为DataFrame
         director.to_frame().head()
out[22]: 
    director_name
0   James Cameron
1   Gore Verbinski
2   Sam Mendes
3   Christopher Nolan
4   Doug Walker

5. 调用Series方法

准备

 in[23]: # 查看Series所有不重复的指令
         s_attr_methods = set(dir(pd.Series))
         # 该集合的大小
         len(s_attr_methods)
out[23]: 442
 in[24]: # 查看DataFrame所有不重复的指令
         df_attr_methods = set(dir(pd.DataFrame))
         len(df_attr_methods)
out[24]: 445
 in[25]: # 这两个集合中有多少共有的指令
         len(s_attr_methods & df_attr_methods)
out[25]: 376

原理

 in[26]: # 选取director和actor_1_fb_likes两列
         movie = pd.read_csv('data/movie.csv')
         director = movie['director_name']
         actor_1_fb_likes = movie['actor_1_facebook_likes']
# 查看头部
 in[27]: director.head()
out[27]: 0        James Cameron
         1       Gore Verbinski
         2           Sam Mendes
         3    Christopher Nolan
         4          Doug Walker
         Name: director_name, dtype: object
 in[28]: actor_1_fb_likes.head()
out[28]: 0     1000.0
         1    40000.0
         2    11000.0
         3    27000.0
         4      131.0
        Name: actor_1_facebook_likes, dtype: float64
 in[29]: # 分别计数
         pd.set_option('max_rows', 8)
         director.value_counts()
out[29]: Steven Spielberg    26
         Woody Allen         22
         Clint Eastwood      20
         Martin Scorsese     20
                              ..
         James Nunn           1
         Gerard Johnstone     1
         Ethan Maniquis       1
         Antony Hoffman       1
         Name: director_name, Length: 2397, dtype: int64
 in[30]: actor_1_fb_likes.value_counts()
out[30]: 1000.0     436
         11000.0    206
         2000.0     189
         3000.0     150
                  ... 
         216.0        1
         859.0        1
         225.0        1
         334.0        1
         Name: actor_1_facebook_likes, Length: 877, dtype: int64
 in[31]: director.size  
out[31]: 4916
 in[32]: director.shape 
out[33]: (4916,)
 in[33]: len(director)  
out[33]: 4916
 in[34]: # director有多少非空值
         director.count() 
out[34]: 4814 # 说明有102个缺失值
 in[35]: # actor_1_fb_likes有多少非空值
         actor_1_fb_likes.count()
out[35]: 4909
 in[36]: # actor_1_fb_likes的中位分位数
         actor_1_fb_likes.quantile()
out[36]: 982.0
 in[37]: # 求最小值、最大值、平均值、中位数、标准差、总和
         actor_1_fb_likes.min(), actor_1_fb_likes.max(), \
         actor_1_fb_likes.mean(), actor_1_fb_likes.median(), \
         actor_1_fb_likes.std(), actor_1_fb_likes.sum()
out[37]: (0.0, 640000.0, 6494.488490527602, 982.0, 15106.986883848309, 31881444.0)
 in[38]: # 打印描述信息
         actor_1_fb_likes.describe()
out[38]: count      4909.000000
         mean       6494.488491
         std       15106.986884
         min           0.000000
         25%         607.000000
         50%         982.000000
         75%       11000.000000
         max      640000.000000
         Name: actor_1_facebook_likes, dtype: float64
 in[39]: director.describe()
out[39]: count                 4814
         unique                2397
         top       Steven Spielberg
         freq                    26
         Name: director_name, dtype: object
 in[40]: actor_1_fb_likes.quantile(.2)
out[41]: 510.0
 in[41]: # 各个十分之一分位数
         actor_1_fb_likes.quantile([.1, .2, .3, .4, .5, .6, .7, .8, .9])
out[41]: 0.1      240.0
         0.2      510.0
         0.3      694.0
         0.4      854.0
               ...   
         0.6     1000.0
         0.7     8000.0
         0.8    13000.0
         0.9    18000.0
         Name: actor_1_facebook_likes, Length: 9, dtype: float64
# 非空值
 In[42]: director.isnull()
Out[42]: 0       False
         1       False
         2       False
         3       False
                 ...  
         4912     True
         4913    False
         4914    False
         4915    False
         Name: director_name, Length: 4916, dtype: bool
# 填充缺失值
 In[43]: actor_1_fb_likes_filled = actor_1_fb_likes.fillna(0)
         actor_1_fb_likes_filled.count()
Out[43]: 4916
# 删除缺失值
 In[44]: actor_1_fb_likes_dropped = actor_1_fb_likes.dropna()
         actor_1_fb_likes_dropped.size
Out[44]: 4909

更多

# value_counts(normalize=True) 可以返回频率
 In[45]: director.value_counts(normalize=True)
Out[45]: Steven Spielberg    0.005401
         Woody Allen         0.004570
         Clint Eastwood      0.004155
         Martin Scorsese     0.004155
                         ...   
         James Nunn          0.000208
         Gerard Johnstone    0.000208
         Ethan Maniquis      0.000208
         Antony Hoffman      0.000208
         Name: director_name, Length: 2397, dtype: float64
# 判断是否有缺失值
 In[46]: director.hasnans
Out[46]: True
# 判断是否是非缺失值
 In[47]: director.notnull()
Out[47]: 0        True
         1        True
         2        True
         3        True
                 ...  
         4912    False
         4913     True
         4914     True
         4915     True
         Name: director_name, Length: 4916, dtype: bool

6. 在Series上使用运算符

 In[48]: pd.options.display.max_rows = 6
 In[49]: 5 + 9    # 加法
Out[49]: 14
 In[50]: 4 ** 2   # 幂运算
Out[50]: 16
 In[51]: a = 10   # 赋值
 In[52]: 5 <= 9 # 小于等于
Out[52]: True
 In[53]: 'abcde' + 'fg' # 字符串拼接
Out[53]: 'abcdefg'
 In[54]: not (5 <= 9)  # 非运算符
Out[54]: False
 In[55]: 7 in [1, 2, 6]  # in运算符
Out[55]: False
 In[56]: set([1,2,3]) & set([2,3,4])  # 求交集
Out[56]: {2, 3}
# 不支持列表和整数间的运算
 In[57]: [1, 2, 3] - 3 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-57-7ca967348b32> in <module>()
----> 1 [1, 2, 3] - 3

TypeError: unsupported operand type(s) for -: 'list' and 'int'
 In[58]: a = set([1,2,3])     
         a[0]                 # 集合不支持索引

准备

# 选取imdb_score这列
 In[59]: movie = pd.read_csv('data/movie.csv')
         imdb_score = movie['imdb_score']
         imdb_score
Out[59]: 0       7.9
         1       7.1
         2       6.8
             ... 
         4913    6.3
         4914    6.3
         4915    6.6
         Name: imdb_score, Length: 4916, dtype: float64
# 每列值加1
 In[60]: imdb_score + 1
Out[60]: 0       8.9
         1       8.1
         2       7.8
             ... 
         4913    7.3
         4914    7.3
         4915    7.6
         Name: imdb_score, Length: 4916, dtype: float64
# 每列值乘以2.5
 In[61]: imdb_score * 2.5
Out[61]: 0       19.75
         1       17.75
         2       17.00
             ...  
         4913    15.75
         4914    15.75
         4915    16.50
         Name: imdb_score, Length: 4916, dtype: float64
# 每列值除以7的余数
 In[62]: imdb_score // 7
Out[62]: 0       1.0
         1       1.0
         2       0.0
              ... 
         4913    0.0
         4914    0.0
         4915    0.0
         Name: imdb_score, Length: 4916, dtype: float64
# 判断是否大于7
 In[63]: imdb_score > 7
Out[63]: 0        True
         1        True
         2       False
               ...  
         4913    False
         4914    False
         4915    False
         Name: imdb_score, Length: 4916, dtype: bool
# 判断是否等于字符串
 In[64]: director = movie['director_name']
 In[65]: director == 'James Cameron'
Out[65]: 0        True
         1       False
         2       False
               ...  
         4913    False
         4914    False
         4915    False
         Name: director_name, Length: 4916, dtype: bool

更多

# 利用通用函数实现加法
 In[66]: imdb_score.add(1)              # imdb_score + 1
Out[66]: 0       8.9
         1       8.1
         2       7.8
              ... 
         4913    7.3
         4914    7.3
         4915    7.6
Name: imdb_score, Length: 4916, dtype: float64
# 利用通用函数实现乘法
 In[67]: imdb_score.mul(2.5)            # imdb_score * 2.5
Out[67]: 0       19.75
         1       17.75
         2       17.00
               ...  
         4913    15.75
         4914    15.75
         4915    16.50
         Name: imdb_score, Length: 4916, dtype: float64
# 利用通用函数实现底除
 In[68]: imdb_score.floordiv(7)         # imdb_score // 7
Out[68]: 0       1.0
         1       1.0
         2       0.0
              ... 
         4913    0.0
         4914    0.0
         4915    0.0
         Name: imdb_score, Length: 4916, dtype: float64
# 利用通用函数实现大于
 In[69]: imdb_score.gt(7)               # imdb_score > 7
Out[69]: 0        True
         1        True
         2       False
               ...  
        4913    False
        4914    False
        4915    False
        Name: imdb_score, Length: 4916, dtype: bool
# 利用通用函数实现等于
 In[70]: director.eq('James Cameron')   # director == 'James Cameron'
Out[70]: 0        True
         1       False
         2       False
               ...  
         4913    False
         4914    False
         4915    False
         Name: director_name, Length: 4916, dtype: bool
# 利用通用函数实现取模
 In[71]: imdb_score.astype(int).mod(5)
Out[71]: 0       2
         1       2
         2       1
              ..
         4913    1
         4914    1
         4915    1
         Name: imdb_score, Length: 4916, dtype: int64
# a是int对象
 In[72]: a = type(1)
 In[73]: type(a)
Out[73]: type
# a是pandas.core.series.Series对象
 In[74]: a = type(imdb_score)
 In[75]: a([1,2,3])
Out[75]: 0    1
         1    2
         2    3
         dtype: int64

7. 串联Series方法

# value_counts().head(3),计数,查看前三
 In[76]: movie = pd.read_csv('data/movie.csv')
         actor_1_fb_likes = movie['actor_1_facebook_likes']
         director = movie['director_name']
 In[77]: director.value_counts().head(3)
Out[77]: Steven Spielberg    26
         Woody Allen         22
         Clint Eastwood      20
         Name: director_name, dtype: int64
# 统计缺失值的数量
 In[78]: actor_1_fb_likes.isnull().sum()
Out[78]: 7
# actor_1_fb_likes的数据类型
 In[79]: actor_1_fb_likes.dtype
Out[79]: dtype('float64')
# 缺失值填充为0、转换为整型、查看前五
 In[80]: actor_1_fb_likes.fillna(0)\
                .astype(int)\
                .head()
Out[80]: 0     1000
         1    40000
         2    11000
         3    27000
         4      131
         Name: actor_1_facebook_likes, dtype: int64

更多

# 缺失值的比例
 In[81]: actor_1_fb_likes.isnull().mean()
Out[81]: 0.0014239218877135883
# 使用括号串联
 In[82]: (actor_1_fb_likes.fillna(0)
                 .astype(int)
                 .head())
Out[82]: 0     1000
         1    40000
         2    11000
         3    27000
         4      131
         Name: actor_1_facebook_likes, dtype: int64

8. 使索引有意义

# set_index()给行索引命名
 In[83]: movie = pd.read_csv('data/movie.csv')
 In[84]: movie.shape
Out[84]: (4916, 28)
 In[85]: movie2 = movie.set_index('movie_title')
         movie2
Out[85]: 
# 通过index_col参数命名
 In[86]: pd.read_csv('data/movie.csv', index_col='movie_title')
Out[86]: 

更多

# 复原索引
 In[87]: movie2.reset_index()

9. 重命名行名和列名

# 通过rename()重命名
 In[88]: movie = pd.read_csv('data/movie.csv', index_col='movie_title')
 In[89]: idx_rename = {'Avatar':'Ratava', 'Spectre': 'Ertceps'} 
         col_rename = {'director_name':'Director Name', 
                       'num_critic_for_reviews': 'Critical Reviews'} 
 In[90]: movie.rename(index=idx_rename, 
                      columns=col_rename).head()
Out[90]: 

更多

 In[91]: movie = pd.read_csv('data/movie.csv', index_col='movie_title')
         index = movie.index
         columns = movie.columns

         index_list = index.tolist()
         column_list = columns.tolist()

         index_list[0] = 'Ratava'
         index_list[2] = 'Ertceps'
         column_list[1] = 'Director Name'
         column_list[2] = 'Critical Reviews'

 In[92]: print(index_list[:5])
         ['Ratava', "Pirates of the Caribbean: At World's End", 'Ertceps', 'The Dark Knight Rises', 'Star Wars: Episode VII - The Force Awakens']

 In[93]: print(column_list)
         ['color', 'Director Name', 'Critical Reviews', 'duration', 'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name', 'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name', 'num_voted_users', 'cast_total_facebook_likes', 'actor_3_name', 'facenumber_in_poster', 'plot_keywords', 'movie_imdb_link', 'num_user_for_reviews', 'language', 'country', 'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes', 'imdb_score', 'aspect_ratio', 'movie_facebook_likes']

 In[94]: movie.index = index_list
         movie.columns = column_list

 In[95]: movie.head()
Out[95]:

10. 创建、删除列

# 通过[列名]添加新列
 In[96]: movie = pd.read_csv('data/movie.csv')
 In[97]: movie['has_seen'] = 0
 In[98]: movie.columns
Out[98]: Index(['color', 'director_name', 'num_critic_for_reviews', 'duration',
                'director_facebook_likes', 'actor_3_facebook_likes', 'actor_2_name',
                'actor_1_facebook_likes', 'gross', 'genres', 'actor_1_name',
                'movie_title', 'num_voted_users', 'cast_total_facebook_likes',
                'actor_3_name', 'facenumber_in_poster', 'plot_keywords',
                'movie_imdb_link', 'num_user_for_reviews', 'language', 'country',
                'content_rating', 'budget', 'title_year', 'actor_2_facebook_likes',
                'imdb_score', 'aspect_ratio', 'movie_facebook_likes', 'has_seen'],
               dtype='object')
# 给新列赋值
 In[99]: movie['actor_director_facebook_likes'] = (movie['actor_1_facebook_likes'] + 
                                              movie['actor_2_facebook_likes'] + 
                                              movie['actor_3_facebook_likes'] + 
                                              movie['director_facebook_likes'])
 In[100]: movie['actor_director_facebook_likes'].isnull().sum()
Out[100]: 122
# 用all()检查是否所有的布尔值都为True
 In[101]: movie['actor_director_facebook_likes'] = movie['actor_director_facebook_likes'].fillna(0)
 In[102]: movie['is_cast_likes_more'] = (movie['cast_total_facebook_likes'] >= 
                                  movie['actor_director_facebook_likes'])
 In[103]: movie['is_cast_likes_more'].all()
Out[103]: False

 In[104]: movie = movie.drop('actor_director_facebook_likes', axis='columns')
 In[105]: movie['actor_total_facebook_likes'] = (movie['actor_1_facebook_likes'] + 
                                       movie['actor_2_facebook_likes'] + 
                                       movie['actor_3_facebook_likes'])

          movie['actor_total_facebook_likes'] = movie['actor_total_facebook_likes'].fillna(0)
 In[106]: movie['is_cast_likes_more'] = movie['cast_total_facebook_likes'] >= \
                                  movie['actor_total_facebook_likes']
    
          movie['is_cast_likes_more'].all()
Out[106]: True
 In[107]: movie['pct_actor_cast_like'] = (movie['actor_total_facebook_likes'] / 
                                movie['cast_total_facebook_likes'])
 In[108]: movie['pct_actor_cast_like'].min(), movie['pct_actor_cast_like'].max() 
Out[108]: (0.0, 1.0)

 In[109]: movie.set_index('movie_title')['pct_actor_cast_like'].head()
Out[109]: movie_title
          Avatar                                        0.577369
          Pirates of the Caribbean: At World's End      0.951396
          Spectre                                       0.987521
          The Dark Knight Rises                         0.683783
          Star Wars: Episode VII - The Force Awakens    0.000000
          Name: pct_actor_cast_like, dtype: float64

更多

# 用insert()方法原地插入列
 In[110]: profit_index = movie.columns.get_loc('gross') + 1
          profit_index
 In[111]: movie.insert(loc=profit_index,
                 column='profit',
                 value=movie['gross'] - movie['budget'])
 In[112]: movie.head()
Out[112]: 

第01章 Pandas基础
第02章 DataFrame运算
第03章 数据分析入门
第04章 选取数据子集
第05章 布尔索引
第06章 索引对齐
第07章 分组聚合、过滤、转换
第08章 数据清理
第09章 合并Pandas对象
第10章 时间序列分析
第11章 用Matplotlib、Pandas、Seaborn进行可视化


Pandas Cookbook 总结
Gupao