独行漫步者
我自独行,漫步天下
ELK 5.0安装使用

https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html

中文指南: https://www.elastic.co/guide/cn/elasticsearch/guide/current/inside-a-shard.html

虚拟机打开文件数设置

[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

临时:

sysctl -w vm.max_map_count=262144

永久:

vim /etc/sysctl.conf 

vm.max_map_count=262144   # append

初始化目录,不能使用 root 启动

chown -R chenglin:ifengdev elasticsearch-5.4.2
mkdir -p /data/elk /data/logs/elk
chown -R chenglin:ifengdev /data/elk
chown -R chenglin:ifengdev /data/logs/elk

编辑配置配置文件

vim config/elasticsearch.yml

cluster.name: jx_dianshang
node.name: ${HOSTNAME}
path.data: /data/elk
path.logs: /data/logs/elk
network.host: 10.90.32.47
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.90.32.47","10.90.32.48", "10.90.32.49"]
discovery.zen.minimum_master_nodes: 2

编辑 java 启动内存

vim config/jvm.options

# 原来为 2g 改为30g
-Xms30g
-Xmx30g

启动

# ./bin/elasticsearch-plugin  install x-pack    # 装监控插件
su - chenglin
./bin/elasticsearch -d

x-pack安装后,就会发现ES的访问和Kibana的访问都受到限制,那么默认的用户名和密码是多少呢

ElasticSearch

用户名:elastic

密码:changeme   x-pack1234qwer

停止

jps | grep Elasticsearch
kill -SIGTERM 16649

安装 Kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.2-linux-x86_64.tar.gz
sha1sum kibana-5.4.2-linux-x86_64.tar.gz 
tar -xzf kibana-5.4.2-linux-x86_64.tar.gz
cd kibana-5.4.2-linux-x86_64

配置文件

vim config/kibana.yml 
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://10.90.32.47:9200"

启动

# ./bin/kibana-plugin install x-pack    # 装监控插件
nohup ./bin/kibana serve & 

访问

http://10.90.32.47:5601 账号密码 加上面 elasticsearch 插件


Nginx

nginx 配置文件

http
{
            log_format  main  '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_x_forwarded_for" "$http_user_agent"';
}

Logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.tar.gz
tar -xf logstash-5.5.0.tar.gz
cd logstash-5.5.0

配置文件

vim config/logstash.conf  // 新建文件

input {
    file {
        path => "/data/logs/nginx/analys/stat_access.log"    
        start_position => "beginning"          
    }

}
filter { 
    
    grok {
           match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:logtimestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:browser_agent}"   }
                            
    }
    
    kv { 
        source => "request"  
        field_split => "&?"
        value_split => "="
        #remove_field => [ "path", "ident", "auth","message","httpversion","bytes" ]
    }
    urldecode {
        all_fields => true 
    } 
    geoip {
        source => "clientip"    
        fields => ["country_name","city_name","real_region_name","latitude","longitude","ip","location"]      
    }
    
}
output {
    elasticsearch {
        hosts => ["10.32.25.170:9200"]
        index => "testlog-%{+YYYY.MM.dd}"
        template_overwrite => true
        template => "/data/soft/install/logstash/outputs/test.json"
    }

}

启动

 ./bin/logstash -f config/logstash.conf

Can't start up: not enough memory

升级到 java 1.8即可

[ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '403' contacting Elasticsearch at URL 'http://192.168.33.11:9200/_template/logstash'"

密码错误

output {
    elasticsearch {
        hosts => ["192.168.33.11"]
        index => "testlog-%{+YYYY.MM.dd}"
        user => logstash_client
        password => hidden
    }
}

[logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://10.90.32.48:9200/_template/logstash'

curl -user elastic:changeme -XPUT 10.90.32.48:9200/_template/template_1 -d@/usr/local/logstash-5.5.0/config/output.json
# 或者
curl -u elastic:changeme -XPUT 10.90.32.48:9200/_template/template_1 -d@/usr/local/logstash-5.5.0/config/output.json

通过以上命令找到错误

Since elasticsearch 5.x index level settings can NOT be set on the nodes 

configuration like the elasticsearch.yaml, in system properties or command line 

arguments.In order to upgrade all indices the settings must be updated via the 

/${index}/_settings API. Unless all settings are dynamic all indices must be closed 

in order to apply the upgradeIndices created in the future should use index templates 

to set default values. 

Please ensure all required values are updated on all indices by executing: 

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{

  "index.number_of_replicas" : "0",

  "index.number_of_shards" : "5"

}'

elasticsearch 5.x不支持配置文件内,修改索引配置,比如复制集数,分片数,需要通过命令行执行。

Saved "field" parameter is now invalid. Please select a new field.

Discover: "field" is a required parameter

解决方法:

针对配置的索引,至少添加一条有效的数据, 添加完成后,在如图位置刷新一下 field, 然后将发现索引下的 field searchable 被勾选了,这时就可以顺利的搜索啦。

http://www.tech06.com/2017/06/15/kibana-discover-field-required-parameter/

Discover: Fielddata is disabled on text fields by default. Set fielddata=true on [logtimestamp] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

#  vim /etc/sysctl.conf 

  // 追加
  vm.max_map_count=655360

#  sysctl -p
<< 上一篇 elasticsearch 扩展安装 及 head ELK logstash分析 nginx 日志 下一篇 >>
文章标签
随意 | Created At 2014 By William Clinton | Contact Me: 1619488373@qq.com | 蜀ICP备14002619号 |