All data nodes of One ES Cluster have read-only issue

Diagnose

  1. There's one ES Cluster combined with VMS with volumes attached to it as below:
Node Role
es5 data
es6 data
es7 data
es8 master
es9 master
es10 master
  1. All the 3 data nodes are attached with one cinder volume "/data1" as one of their data folder.
  2. However due to one Network issue happened last Friday, all of the 3 volumes became "Read-Only File System", which means they're not able to write new documents in.
  3. Then it impacts the whole ES Cluster to do operation on shards (distribute/replica). So the whole ES Cluster is in red status since some indices having shard issues.
  4. Then restart the Elasticsearch service on es5, and see exceptions as below:
Caused by: java.io.FileNotFoundException: /data1/instance00/stackstash-elasticsearch/nodes/0/indices/alert_08052014/3/index/_8h5j.fdx (Read-only file system)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:165)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:384)
    at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:277)
    at org.apache.lucene.store.FileSwitchDirectory.createOutput(FileSwit
  1. The attached volume is still in "Read-Only" mode.
  2. Then reboot the data node, and "Read-Only" issue is resolved.
  3. However Elasticsearch service can’t start on es5 node because "failed to read local state". There may be some broken state files due to read-only issue.
  4. The same thing happened on es6 and es7.
  5. So till now, the ES Cluster only has 3 master nodes working normally. While all of the 3 data nodes can not start up Elasticsearch service.

How to recover

  1. The replica count for each index was set to 2 which means there was a backup of all the data on the remaining 2 data node anyways.
  2. Shutdown the data node with the corrupt Lucene shards, which is es5.
  3. Start up Elasticsearch service on es6 and es7 by disable shard-allocation (can do it on any master node):
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.enable": "none"}}'
  1. Above steps bring up the data nodes, and can see all is ok. So enable shards allocation as below, and the cluster took a bit of time to turned green:
curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{"cluster.routing.allocation.enable": "all"}}'
  1. Change the replica count all for indices to 1 (original is 2) in order to speed up recovery and rebalancing.
  2. Wait for shards to rebalance (we’re still red because there are unassigned primaries).
  3. Purge the data filesystem on the corrupt data node.
  4. Restart the corrupt data node (it’s fine now) and allow rebalancing to start.
  5. Identify remaining primaries that are not automatically being initialised by ES.
  6. Manually assign remaining primaries to the newly recovered data node.
for index in $(curl -s -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $1}' | sort -u); do
 for shard in $(curl -s -XGET http://localhost:9200/_cat/shards/$index | grep UNASSIGNED | awk '{print $2}' | sort -u); do
   curl -s -XPOST 'localhost:9200/_cluster/reroute' -d "{
       \"commands\" : [ {
             \"allocate\" : {
                 \"index\" : \"$index\",
                 \"shard\" : $shard,
                 \"node\" : \"es5\",
                 \"allow_primary\" : true
             }
           }
       ]
   }"
   sleep 5
 done
  1. Wait for cluster to turn green.
  2. Change the replica count for all indices back to 2 (cluster goes yellow).
  3. Wait for cluster to turn green.

What we get

  1. Move away from volumes as local disk is plentiful on most VM flavors and instead rely on replicating shards across data nodes at some point.
  2. During recover, we can set replica to "1" to speed up rebalancing. Then set replica back after recovering.

Reference

  1. Shard allocation disable/enable

推荐阅读更多精彩内容

  • 兵荒马乱,烽烟四起。 围城内外,杳无消息。 绕城三匝,万念千记。 有志不违,誓入城内。 到头来,不过是,空城计。 ...
    蕊梦蝶想阅读 45评论 4 2
  • 《门开了》 作者:明至 终于打开了门, 你要进来坐么? 喝上一杯热茶, 再讲一个故事。 (材料:速写本、8B铅笔、...
    作家明至阅读 159评论 0 2
  • (一)穿越 倘若今天存在时光机器,你能一瞬间穿越到古代,那么你会像寻秦记中的项少龙一样辉煌么?你也能寻找到乱世中的...
    一生无忌阅读 622评论 9 8
  • 再不疯狂我们就老了,再不表白人都走了。 下个夏天,教室又坐满了人,可却不再是我们。 我们的最后一个夏天,叫离别。 ...
    3d00e54e86fa阅读 98评论 1 5