elasticsearch 入门篇

介绍

elasticsearch是一个高效的、可扩展的全文搜索引擎

基本概念

  • Near Realtime(NRT): es是一个接近实时查询平台,意味从存储一条数据到可以索引到数据时差很小,通常在1s内
  • Cluster: es是一个分布式、可扩展的平台, 可由一个或多个服务器通过定义的cluster.name(默认为elasticsearch)标识共建同一个集群
  • Node: 通常一台服务器上部署一台es node,作为集群的一部分,用于数据的存储和提供搜索功能,在一个集群中节点通过node.name区分,默认在node启动时随机生成一个的字符串做为节点名称,可配置
  • Index: 类似于关系型数据库中的database,用于组织一类功能相似的数据,在一个集群中可以定义任意个索引,索引的名称只能由小写字母组成,在数据索引,更新,搜索,删除时作为数据标识的一部分
  • Type: 类似于关系型数据库中的table,在Index中可以定义多个Type,原则上一个Type是由相同属性组成的数据集合
  • Document: 类似于关系型数据库中的record,是数据的最基本存储单元,使用json形式表示,Document在物理上存储在Index下,但是在逻辑上会分配到具体的Type下
  • Shards & Replica:
    一个Index可能存储大量的数据(超过单个节点的硬件限制),不管是数据存储还是数据索引,为解决数据单节点存储并提高并发,es将每一个Index物理分为多个片,从而水平扩展存储容量,提高并发(可以同时对个shard进行索引和搜索)
    为防止某个存储单元出现故障后数据不能索引的情况,es提供将shard进行复制功能,将主shard出现故障后,复制shard替代主shard进行数据索引操作,已此方式实现其高可用性,因为在搜索时可以使用复制shard,从而提高的数据搜索的并发性
    在Index创建时可以进行分片数量和复制数量的设置,默认创建每个Index设置5个shard和1个Replica,表示该Index由5个逻辑存储单元进行存储,每个逻辑存储单元具有一个复制节点进行备灾,注意,shard只能在创建Index时进行设置,shard数量与document分配到哪个shard上存储有关(通常使用hash(document _id) % shard num计算 document存储在哪个shard上)
    在es将主shard和replic分片在不同的Node上

安装

  • elasticsearch使用java语言实现,在使用时必须安装java虚拟机(目前es1.6和1.7版本均可选择1.8版本java)
  • 下载地址
  • 解压到安装目录 C:\Program Files\elasticsearch
  • 运行 cd "C:\Program Files\elasticsearch\bin" && elasticsearch.bat
  • 安装到服务 service install elasticsearch
  • 启动服务 net start elasticsearch
  • 停止服务 net stop elasticsearch
  • 测试
    访问地址: http://localhost:9200
    访问结果:
  {
    status: 200,
    name: "Smart Alec",
    cluster_name: "elasticsearch",
    version: {
      number: "1.6.0",
      build_hash: "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",
      build_timestamp: "2015-06-09T13:36:34Z",
      build_snapshot: false,
      lucene_version: "4.10.4"
    },
    tagline: "You Know, for Search"
  }

接口

es对外提供标准RESTAPI接口,使用他进行集群的所有操作:

  • 集群、节点、索引的状态和统计信息查看
  • 管理集群、节点、索引和类型
  • 执行CURD操作(创建,更新,读取,删除)和索引
  • 执行高级搜索功能,比如排序,分页,筛选,聚合,js脚本执行等

格式:curl -X<REST verb> <Node>:<Port>/<Index>/<Type>/<ID>

使用marvel插件

  • 运行 cd "C:\Program Files\elasticsearch\bin" && plugin -i elasticsearch/marvel/latest
  • 访问地址
  • marvel提供sense工具调用es的RESTAPI借口, 访问地址, 以下操作使用sense或使用linux curl命令行练习

状态查询

  • 集群状态查询
    输入: GET _cat/health?v
    输出:
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks
1442227489 18:44:49  elasticsearch yellow          1         1     50  50    0    0       50             0

说明:
status:表示集群的健康状态,值可能为green,yellow,red, green表示主shard和replica(至少一个)正常,yellow表示主shard正常但replica都不正常,red表示有的主shard和replica都有问题
node.total:表示集群中节点的数量

  • 节点状态查询
    输入: GET /_cat/nodes?v
    输出:
host      ip             heap.percent ram.percent load node.role master name
silence   192.168.1.111            30          51      d         *      Thunderbird

查询所有索引

输入: GET /_cat/indices?v
输出:

health status index              pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .marvel-2015.09.02   1   1      93564            0     78.4mb         78.4mb
yellow open   .marvel-2015.09.01   1   1      39581            0     45.9mb         45.9mb

创建索引

输入: PUT /test1?pretty
输出:

{
  "acknowledged" : true
}

查询所有索引:

health status index              pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test1                5   1          0            0       575b           575b

说明:
health:由于只运行一个节点,replica不能与主shard在同一node中,因此replica不正常,该index的状态为yellow
index:为索引名称
pri:表示主shard个数
rep:表示每个shard的复制个数
docs.count:表示index中document的个数

索引、读取、删除文档

索引文档

  • 方法1:
    输入:
PUT /test1/user/1?pretty
{"name": "silence1"}

输出:

{
  "_index" : "test1
  "_type" : "user",
  "_id" : "1",
  "_version" : 1,
  "created" : true
}
  • 方法2:
    输入:
POST /test1/user/2?pretty
{"name": "silence2"}

输出:

{
  "_index" : "test1",
  "_type" : "user",
  "_id" : "2",
  "_version" : 1,
  "created" : true
}
  • 方法3:
    输入:
POST /test1/user?pretty
{"name": "silence3"}

输出:

{
  "_index" : "test1",
  "_type" : "user",
  "_id" : "AU_MdQoXRYiHSIs7UGBQ",
  "_version" : 1,
  "created" : true
}

说明: 在索引文档时若需要指定文档ID值则需要使用PUT或者POST提交数据并显示指定ID值,若需要由es自动生成ID,则需要使用POST提交数据

读取文档:
输入: GET /test1/user/1?pretty
输出:

{
  "_index" : "test1",
  "_type" : "user",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source":{"name": "silence1"}
}

说明:
_index,_type:表示文档存储的Index和Type信息
_id:表示文档的编号
_version:表示文档的版本号,主要用于并发处理时使用乐观锁防止脏数据
found:表示请求的文档是否存在
_souce:格式为json,为文档的内容

注意:在之前我们并未创建user的Type,在进行文档索引时自动创建了user,在es中可以不显示的创建Index和Type而使用默认参数或者根据提交数据自定义,但不建议这么使用,在不清楚可能导致什么情况时显示创建Index和Type并设置参数

删除文档:
输入: DELETE /test1/user/1?pretty
输出:

{
  "found" : true,
  "_index" : "test1",
  "_type" : "user",
  "_id" : "1",
  "_version" : 2
}

再次读取文档输出:

{
  "_index" : "test1",
  "_type" : "user",
  "_id" : "1",
  "found" : false
}

删除索引

输入: DELETE /test1?pretty
输出:

{
  "acknowledged" : true
}

修改文档

初始化文档输入:

PUT /test1/user/1?pretty
{"name" : "silence2", "age":28}

修改文档输入:

PUT /test1/user/1?pretty
{"name" : "silence1"}

读取文档输出:

{
  "_index" : "test1",
  "_type" : "user",
  "_id" : "1",
  "_version" : 2,
  "found" : true,
  "_source":{"name" : "silence1"}
}

更新文档

更新数据输入:

POST /test1/user/1/_update?pretty
{"doc" : {"name" : "silence3", "age":28}}

读取数据输出:

{
  "_index" : "test1",
  "_type" : "user",
  "_id" : "1",
  "_version" : 3,
  "found" : true,
  "_source":{"name":"silence3","age":28}
}

更新文档输入:

POST /test1/user/1/_update?pretty
{"script" : "ctx._source.age += 1"}

读取文档输出:

{
  "_index" : "test1",
  "_type" : "user",
  "_id" : "1",
  "_version" : 4,
  "found" : true,
  "_source":{"name":"silence3","age":29}
}

说明:需要POST使用script则必须在elasticsearch/config/elasticsearch.yml配置script.groovy.sandbox.enabled: true
修改(PUT)和更新(POST+_update)的区别在于修改使用提交的文档覆盖es中的文档,更新使用提交的参数值覆盖es中文档对应的参数值

根据查询删除文档

输入:

DELETE /test1/user/_query?pretty
{"query" : {"match" : {"name" : "silence3"}}}

输出:

{
  "_indices" : {
    "test1" : {
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "failed" : 0
      }
    }
  }
}

获取文档数量

输入: GET /test1/user/_count?pretty
输出:

{
  "count" : 0,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  }
}

批量操作

输入:

POST /test1/user/_bulk?pretty
{"index" : {"_id" : 1}}
{"name" : "silence1"}
{"index" : {"_id" : 2}}
{"name" : "silence2"}
{"index" : {}}
{"name" : "silence3"}
{"index" : {}}
{"name" : "silence4"}

输入:

POST /test1/user/_bulk?pretty
{"update" : {"_id" : 1}}
{"doc" : {"age" : 28}}
{"delete" : {"_id" : 2}}

通过文件导入数据: curl -XPOST "localhost:9200/test1/account/_bulk?pretty" --data-binary @accounts.json

Query查询

查询可以通过两种方式进行,一种为使用查询字符串进行提交参数查询,一种为使用RESTAPI提交requesbody提交参数查询

获取所有文档输入: GET /test1/user/_search?q=*&pretty

POST /test1/user/_search?pretty
{
  "query" : {"match_all" : {}}
}

输出:

{
   "took": 2,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 3,
      "max_score": 1,
      "hits": [
         {
            "_index": "test1",
            "_type": "user",
            "_id": "1",
            "_score": 1,
            "_source": {
               "name": "silence1",
               "age": 28
            }
         },
         {
            "_index": "test1",
            "_type": "user",
            "_id": "AU_M2zgwLNdQvgqQS3MP",
            "_score": 1,
            "_source": {
               "name": "silence3"
            }
         },
         {
            "_index": "test1",
            "_type": "user",
            "_id": "AU_M2zgwLNdQvgqQS3MQ",
            "_score": 1,
            "_source": {
               "name": "silence4"
            }
         }
      ]
   }
}

说明:
took: 执行查询的时间(单位为毫秒)
timed_out: 执行不能超时
_shards: 提示有多少shard参与查询以及查询成功和失败shard数量
hits: 查询结果
hits.total: 文档总数
_score, max_score: 为文档与查询条件匹配度和最大匹配度

Query SDL

输入:

POST /test1/account/_search?pretty
{
  "query" : {"match_all":{}},
  "size": 2,
  "from" : 6,
  "sort" : {
    "age" : {"order" : "asc"} 
  }
}

说明:
query: 用于定义查询条件过滤
match_all: 表示查询所有文档
size: 表示查询返回文档数量,若未设置默认为10
from: 表示开始位置, es使用0作为开始索引,常与size组合进行分页查询,若未设置默认为0
sort: 用于设置排序属性和规则

  • 使用_source设置查询结果返回的文档属性
    输入:
POST /test1/account/_search?pretty
{
  "query": {
    "match_all": {}
  },
  "_source":["firstname", "lastname", "age"]
}

输出:

{
   "took": 5,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1000,
      "max_score": 1,
      "hits": [
         {
            "_index": "test1",
            "_type": "account",
            "_id": "4",
            "_score": 1,
            "_source": {
               "firstname": "Rodriquez",
               "age": 31,
               "lastname": "Flores"
            }
         },
         {
            "_index": "test1",
            "_type": "account",
            "_id": "9",
            "_score": 1,
            "_source": {
               "firstname": "Opal",
               "age": 39,
               "lastname": "Meadows"
            }
         }
      ]
   }
}
  • 使用match设置查询匹配值
    输入:
POST /test1/account/_search?pretty
{
  "query": {
    "match": {"address" : "986 Wyckoff Avenue"}
  },
  "size" : 2
}

输出:

{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 216,
      "max_score": 4.1231737,
      "hits": [
         {
            "_index": "test1",
            "_type": "account",
            "_id": "4",
            "_score": 4.1231737,
            "_source": {
               "account_number": 4,
               "balance": 27658,
               "firstname": "Rodriquez",
               "lastname": "Flores",
               "age": 31,
               "gender": "F",
               "address": "986 Wyckoff Avenue",
               "employer": "Tourmania",
               "email": "rodriquezflores@tourmania.com",
               "city": "Eastvale",
               "state": "HI"
            }
         },
         {
            "_index": "test1",
            "_type": "account",
            "_id": "34",
            "_score": 0.59278774,
            "_source": {
               "account_number": 34,
               "balance": 35379,
               "firstname": "Ellison",
               "lastname": "Kim",
               "age": 30,
               "gender": "F",
               "address": "986 Revere Place",
               "employer": "Signity",
               "email": "ellisonkim@signity.com",
               "city": "Sehili",
               "state": "IL"
            }
         }
      ]
   }
}

说明:根据查询结果可见在查询结果中并非只查询address包含"986 Wyckoff Avenue"的文档,而是包含986,wychoff,Avenue三个词中任意一个,这就是es分词的强大之处
可见查询结果中_score(与查询条件匹配度)按从大到小的顺序排列
此时你可能想要值查询address包含"986 Wyckoff Avenue"的文档,怎么办呢?使用match_phrase
输入:

POST /test1/account/_search?pretty
{
  "query": {
    "match_phrase": {"address" : "986 Wyckoff Avenue"}
  }
}

可能你已经注意到, 以上query中只有一个条件,若存在多个条件,我们必须使用bool query将多个条件进行组合
输入:

POST /test1/account/_search?pretty
{
  "query": {
    "bool" : {
      "must":[
        {"match_phrase": {"address" : "986 Wyckoff Avenue"}},
        {"match" : {"age" : 31}}  
      ]
    }
  }
}

说明: 查询所有条件都满足的结果

输入:

POST /test1/account/_search
{
  "query": {
    "bool" : {
      "should":[
        {"match_phrase": {"address" : "986 Wyckoff Avenue"}},
        {"match_phrase": {"address" : "963 Neptune Avenue"}}
      ]
    }
  }
}

说明: 查询有一个条件满足的结果
输入:

POST /test1/account/_search
{
  "query": {
    "bool" : {
      "must_not":[
        {"match": {"city" : "Eastvale"}},
        {"match": {"city" : "Olney"}}
      ]
    }
  }
}

说明: 查询有条件都不满足的结果

在Query SDL中可以将must, must_not和should组合使用
输入:

POST /test1/account/_search
{
  "query": {
    "bool" : {
      "must": [{
        "match" : {"age":20}
      }],
      "must_not":[
        {"match": {"city" : "Steinhatchee"}}
      ]
    }
  }
}

Filters 查询

在使用Query 查询时可以看到在查询结果中都有_score值, _score值需要进行计算, 在某些情况下我们并不需要_socre值,在es中提供了Filters查询,它类似于Query查询,但是效率较高,原因:

  1. 不需要对查询结果进行_score值的计算
  2. Filters可以被缓存在内存中,可被重复搜索从而提高查询效率
  • range 过滤器, 用于设置条件在某个范围内
    输入:
POST /test1/account/_search?pretty
{
  "query": {
    "filtered":{
      "query": {
        "match_all" : {}
      },
      "filter": {
        "range" : {
          "age" : {
            "gte" : 20,
            "lt" : 28
          }
        }
      }
    }
  }
}

判断使用filter还是使用query的最简单方法就是是否关注_score值,若关注则使用query,若不关注则使用filter

聚合分析

es提供Aggregations支持分组和聚合查询,类似于关系型数据库中的GROUP BY和聚合函数,在ES调用聚合RESTAPI时返回结果包含文档查询结果和聚合结果,也可以返回多个聚合结果,从而简化API调用和减少网络流量使用
输入:

POST /test1/account/_search?pretty
{
  "size" : 0,
  "aggs" : {
    "group_by_gender" : {
      "terms" : {"field":"gender"}
    }
  }
}

输出:

{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1000,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "group_by_gender": {
         "doc_count_error_upper_bound": 0,
         "sum_other_doc_count": 0,
         "buckets": [
            {
               "key": "m",
               "doc_count": 507
            },
            {
               "key": "f",
               "doc_count": 493
            }
         ]
      }
   }
}

说明:
size: 返回文档查询结果数量
aggs: 用于设置聚合分类
terms: 设置group by属性值

输入:

POST /test1/account/_search?pretty
{
  "size" : 0,
  "aggs" : {
    "group_by_gender" : {
      "terms" : {
        "field":"state",
        "order" : {"avg_age":"desc"},
        "size" : 3
      },
      "aggs" : {
        "avg_age" : {
          "avg" : {"field" : "age"}
        },
        "max_age" : {
          "max" : {"field": "age"}
        },
        "min_age" : {
          "min": {"field":"age"}
        }
      }
    }
  }
}

输出:

{
   "took": 9,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1000,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "group_by_gender": {
         "doc_count_error_upper_bound": -1,
         "sum_other_doc_count": 992,
         "buckets": [
            {
               "key": "de",
               "doc_count": 1,
               "max_age": {
                  "value": 37
               },
               "avg_age": {
                  "value": 37
               },
               "min_age": {
                  "value": 37
               }
            },
            {
               "key": "il",
               "doc_count": 3,
               "max_age": {
                  "value": 39
               },
               "avg_age": {
                  "value": 36.333333333333336
               },
               "min_age": {
                  "value": 32
               }
            },
            {
               "key": "in",
               "doc_count": 4,
               "max_age": {
                  "value": 39
               },
               "avg_age": {
                  "value": 36
               },
               "min_age": {
                  "value": 34
               }
            }
         ]
      }
   }
}

说明:根据state进行分类,并查询每种分类所有人员的最大,最小,平均年龄, 查询结果按平均年龄排序并返回前3个查询结果
若需要按照分类总数进行排序时可以使用_count做为sort的field值
在聚合查询时通过size设置返回的TOP数量,默认为10

在聚合查询中可任意嵌套聚合语句进行查询
输入:

POST /test1/account/_search?pretty
{
  "size" : 0,
  "aggs" : {
    "group_by_age" : {
      "range" : {
        "field": "age",
        "ranges" : [{
          "from" : 20,
          "to" : 30
        }, {
          "from": 30,
          "to" : 40
        },{
          "from": 40,
          "to": 50
        }]
      },
      "aggs":{
        "group_by_gender" : {
          "terms" : {"field": "gender"},
          "aggs" : {
            "group_by_balance" :{
              "range" : {
                "field":"balance",
                "ranges" : [{
                  "to" : 5000
                }, {
                  "from" : 5000
                }
                ]
              }
            }
          }
        }
      }
    }
  }
}

输出:

{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 5,
      "successful": 5,
      "failed": 0
   },
   "hits": {
      "total": 1000,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "group_by_age": {
         "buckets": [
            {
               "key": "20.0-30.0",
               "from": 20,
               "from_as_string": "20.0",
               "to": 30,
               "to_as_string": "30.0",
               "doc_count": 451,
               "group_by_gender": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 0,
                  "buckets": [
                     {
                        "key": "m",
                        "doc_count": 232,
                        "group_by_balance": {
                           "buckets": [
                              {
                                 "key": "*-5000.0",
                                 "to": 5000,
                                 "to_as_string": "5000.0",
                                 "doc_count": 9
                              },
                              {
                                 "key": "5000.0-*",
                                 "from": 5000,
                                 "from_as_string": "5000.0",
                                 "doc_count": 223
                              }
                           ]
                        }
                     },
                     {
                        "key": "f",
                        "doc_count": 219,
                        "group_by_balance": {
                           "buckets": [
                              {
                                 "key": "*-5000.0",
                                 "to": 5000,
                                 "to_as_string": "5000.0",
                                 "doc_count": 20
                              },
                              {
                                 "key": "5000.0-*",
                                 "from": 5000,
                                 "from_as_string": "5000.0",
                                 "doc_count": 199
                              }
                           ]
                        }
                     }
                  ]
               }
            },
            {
               "key": "30.0-40.0",
               "from": 30,
               "from_as_string": "30.0",
               "to": 40,
               "to_as_string": "40.0",
               "doc_count": 504,
               "group_by_gender": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 0,
                  "buckets": [
                     {
                        "key": "f",
                        "doc_count": 253,
                        "group_by_balance": {
                           "buckets": [
                              {
                                 "key": "*-5000.0",
                                 "to": 5000,
                                 "to_as_string": "5000.0",
                                 "doc_count": 26
                              },
                              {
                                 "key": "5000.0-*",
                                 "from": 5000,
                                 "from_as_string": "5000.0",
                                 "doc_count": 227
                              }
                           ]
                        }
                     },
                     {
                        "key": "m",
                        "doc_count": 251,
                        "group_by_balance": {
                           "buckets": [
                              {
                                 "key": "*-5000.0",
                                 "to": 5000,
                                 "to_as_string": "5000.0",
                                 "doc_count": 21
                              },
                              {
                                 "key": "5000.0-*",
                                 "from": 5000,
                                 "from_as_string": "5000.0",
                                 "doc_count": 230
                              }
                           ]
                        }
                     }
                  ]
               }
            },
            {
               "key": "40.0-50.0",
               "from": 40,
               "from_as_string": "40.0",
               "to": 50,
               "to_as_string": "50.0",
               "doc_count": 45,
               "group_by_gender": {
                  "doc_count_error_upper_bound": 0,
                  "sum_other_doc_count": 0,
                  "buckets": [
                     {
                        "key": "m",
                        "doc_count": 24,
                        "group_by_balance": {
                           "buckets": [
                              {
                                 "key": "*-5000.0",
                                 "to": 5000,
                                 "to_as_string": "5000.0",
                                 "doc_count": 3
                              },
                              {
                                 "key": "5000.0-*",
                                 "from": 5000,
                                 "from_as_string": "5000.0",
                                 "doc_count": 21
                              }
                           ]
                        }
                     },
                     {
                        "key": "f",
                        "doc_count": 21,
                        "group_by_balance": {
                           "buckets": [
                              {
                                 "key": "*-5000.0",
                                 "to": 5000,
                                 "to_as_string": "5000.0",
                                 "doc_count": 0
                              },
                              {
                                 "key": "5000.0-*",
                                 "from": 5000,
                                 "from_as_string": "5000.0",
                                 "doc_count": 21
                              }
                           ]
                        }
                     }
                  ]
               }
            }
         ]
      }
   }
}

使用head插件

  • 运行 cd "C:\Program Files\elasticsearch\bin" && plugin -install mobz/elasticsearch-head
  • 访问地址

推荐阅读更多精彩内容