がべーじこれくしょん

技術系とかいろいろ

CentOS7にfluentd+Elasticsearch+Kibanaでデータ分析・可視化環境を整えるメモ

TODO: この辺を読んでやったのであとでまとめる

OpenJDKのインストール

yum -y install java-1.8.0-openjdk

Elasticsearchの導入


以下はメモ

Docker環境まとめ

qiita.com

やっぱDocker使ったほうがよくね?? ↓

qiita.com

Docker @ Elastic

Dockerイメージも公式が配布してるっぽい

その他

qiita.com

qiita.com

qiita.com

qiita.com

www.elastic.co

Time4VPS(RAM:2gb)の場合の特例

github.com

# systemctl status elasticsearch -l
* elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Tue 2017-10-31 17:54:27 UTC; 1min 2s ago
     Docs: http://www.elastic.co
  Process: 15977 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=killed, signal=KILL)
  Process: 15974 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 15977 (code=killed, signal=KILL)

といってelasticsearchが死ぬ。どうやらJVMの設定でヒープが足りないゆえにメモリリーク起こしてるみたい。

なので-Xms2g-Xmx2gを1gに変更

## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g

すると今度はStatusID78で死ぬように。なんでやねん。

* elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2017-10-31 18:02:16 UTC; 3min 44s ago
     Docs: http://www.elastic.co
  Process: 16164 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=78)
  Process: 16161 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 16164 (code=exited, status=78)

今度はログが吐かれてたので見てみる

...
[2017-10-31T18:02:16,139][INFO ][o.e.b.BootstrapChecks    ] [VEcDTY1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-10-31T18:02:16,148][ERROR][o.e.b.Bootstrap          ] [VEcDTY1] node validation exception
[2] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2017-10-31T18:02:16,186][INFO ][o.e.n.Node               ] [VEcDTY1] stopping ...
[2017-10-31T18:02:16,337][INFO ][o.e.n.Node               ] [VEcDTY1] stopped
[2017-10-31T18:02:16,337][INFO ][o.e.n.Node               ] [VEcDTY1] closing ...
[2017-10-31T18:02:16,418][INFO ][o.e.n.Node               ] [VEcDTY1] closed

ん…?またメモリー足りない?

コレに関しては関連してそうなIssueが

github.com

# sysctl -w vm.max_map_count=262144
sysctl: permission denied on key 'vm.max_map_count'

…は?

どうやらこれはOpenVZの仕様っぽい。

github.com

Time4VPSは仮想化にOpenVZを使ってるのねふーん

OpenVZは、Linuxカーネルをベースに開発された Red Hat Enterprise Linux 用のオペレーティングシステム レベルのサーバ仮想化ソフト。 Wikipedia

github.com

A limit on mmap counts equal to 262,144 or more

!! This is the most frequent reason for Elasticsearch failing to start since Elasticsearch version 5 was released.

On Linux, use sysctl vm.max_map_count on the host to view the current value, and see Elasticsearch's documentation on virtual memory for guidance on how to change this value. Note that the limits must be changed on the host; they cannot be changed from within a container.

If using Docker for Mac, then you will need to start the container with the MAX_MAP_COUNT environment variable (see Overriding start-up variables) set to at least 262144 (using e.g. docker's -e option) to make Elasticsearch set the limits on mmap counts at start-up time.

http://elk-docker.readthedocs.io/#prerequisites

おん…

ていうかカーネルパラメータ変更できないのってどう考えてもVPSの仕様でしょこれ…

superuser.com

したがってtime4vpsでelasticsearchを起動するのは不可能?

X-pack

www.elastic.co

export ES_HOME=/usr/share/elasticsearch
export KB_HOME=/usr/share/kibana
export PATH=$PATH:$ES_HOME/bin:$KB_HOME/bin
elasticsearch-plugin install x-pack
kibana-plugin install x-pack

Kibanaへのx-packインストールは死ぬほど時間かかるので「は?死んだ?」って思わず気長に待つべし Optimizing and caching browser bundles...で止まりやすい

discuss.elastic.co

To verify X-Pack installation, point your web browser at http://localhost:5601/ to open Kibana. You should be prompted to log in to Kibana. To log in, you can use the built-in elastic user and the password changeme.

www.elastic.co

あとLogStashもX-Pack対応してる(よくわからん)

knowledge.sakura.ad.jp

www.elastic.co

CentOSファイアウォールの設定とか

qiita.com

qiita.com