https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html
中文指南： https://www.elastic.co/guide/cn/elasticsearch/guide/current/inside-a-shard.html
虚拟机打开文件数设置
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
临时：
sysctl -w vm.max_map_count=262144
永久：
vim /etc/sysctl.conf vm.max_map_count=262144 # append
初始化目录，不能使用 root 启动
chown -R chenglin:ifengdev elasticsearch-5.4.2 mkdir -p /data/elk /data/logs/elk chown -R chenglin:ifengdev /data/elk chown -R chenglin:ifengdev /data/logs/elk
编辑配置配置文件
vim config/elasticsearch.yml
cluster.name: jx_dianshang
node.name: ${HOSTNAME}
path.data: /data/elk
path.logs: /data/logs/elk
network.host: 10.90.32.47
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.90.32.47","10.90.32.48", "10.90.32.49"]
discovery.zen.minimum_master_nodes: 2
编辑 java 启动内存
vim config/jvm.options # 原来为 2g 改为30g -Xms30g -Xmx30g
启动
# ./bin/elasticsearch-plugin install x-pack # 装监控插件 su - chenglin ./bin/elasticsearch -d
x-pack安装后，就会发现ES的访问和Kibana的访问都受到限制，那么默认的用户名和密码是多少呢
ElasticSearch
用户名：elastic
密码：changeme x-pack1234qwer
停止
jps | grep Elasticsearch kill -SIGTERM 16649
安装 Kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.2-linux-x86_64.tar.gz sha1sum kibana-5.4.2-linux-x86_64.tar.gz tar -xzf kibana-5.4.2-linux-x86_64.tar.gz cd kibana-5.4.2-linux-x86_64
配置文件
vim config/kibana.yml server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://10.90.32.47:9200"
启动
# ./bin/kibana-plugin install x-pack # 装监控插件 nohup ./bin/kibana serve &
访问
http://10.90.32.47:5601 账号密码 加上面 elasticsearch 插件
Nginx
nginx 配置文件
http
{
log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_x_forwarded_for" "$http_user_agent"';
}
Logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.tar.gz tar -xf logstash-5.5.0.tar.gz cd logstash-5.5.0
配置文件
vim config/logstash.conf // 新建文件
input {
file {
path => "/data/logs/nginx/analys/stat_access.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:logtimestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:browser_agent}" }
}
kv {
source => "request"
field_split => "&?"
value_split => "="
#remove_field => [ "path", "ident", "auth","message","httpversion","bytes" ]
}
urldecode {
all_fields => true
}
geoip {
source => "clientip"
fields => ["country_name","city_name","real_region_name","latitude","longitude","ip","location"]
}
}
output {
elasticsearch {
hosts => ["10.32.25.170:9200"]
index => "testlog-%{+YYYY.MM.dd}"
template_overwrite => true
template => "/data/soft/install/logstash/outputs/test.json"
}
}启动
./bin/logstash -f config/logstash.conf
Can't start up: not enough memory
升级到 java 1.8即可
[ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '403' contacting Elasticsearch at URL 'http://192.168.33.11:9200/_template/logstash'"
密码错误
output {
elasticsearch {
hosts => ["192.168.33.11"]
index => "testlog-%{+YYYY.MM.dd}"
user => logstash_client
password => hidden
}
}[logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://10.90.32.48:9200/_template/logstash'
curl -user elastic:changeme -XPUT 10.90.32.48:9200/_template/template_1 -d@/usr/local/logstash-5.5.0/config/output.json # 或者 curl -u elastic:changeme -XPUT 10.90.32.48:9200/_template/template_1 -d@/usr/local/logstash-5.5.0/config/output.json
通过以上命令找到错误
Since elasticsearch 5.x index level settings can NOT be set on the nodes
configuration like the elasticsearch.yaml, in system properties or command line
arguments.In order to upgrade all indices the settings must be updated via the
/${index}/_settings API. Unless all settings are dynamic all indices must be closed
in order to apply the upgradeIndices created in the future should use index templates
to set default values.
Please ensure all required values are updated on all indices by executing:
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
"index.number_of_replicas" : "0",
"index.number_of_shards" : "5"
}'
elasticsearch 5.x不支持配置文件内，修改索引配置，比如复制集数，分片数，需要通过命令行执行。
Saved "field" parameter is now invalid. Please select a new field.
Discover: "field" is a required parameter
解决方法：
针对配置的索引，至少添加一条有效的数据， 添加完成后，在如图位置刷新一下 field， 然后将发现索引下的 field searchable 被勾选了，这时就可以顺利的搜索啦。
http://www.tech06.com/2017/06/15/kibana-discover-field-required-parameter/
Discover: Fielddata is disabled on text fields by default. Set fielddata=true on [logtimestamp] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
# vim /etc/sysctl.conf // 追加 vm.max_map_count=655360 # sysctl -p