ELK环境搭建及测试数据导入

ELK一些前期准备

前面已经搭建过elasticsearch了，和它配套的，L和K：Logstash，Kibana
三者关系：Logstash搜集日志，elasticsearch存储，kibana展示

搭建Logstash、kibana

首先是官网下载安装包：

elasic.co

下载对应的安装包：
kibana-6.3.2-linux-x86_64.tar.gz
logstash-6.3.2.tar.gz

安装logstash

解压缩下载的安装包logstash-6.3.2.tar.gz，进入logstash-6.3.2/config目录，创建配置文件logstash.conf

vim logstash.conf

内容：

input {
     file {
        type => "log"
        path => "/logs/*.log"
        start_position => "beginning"
    }
}

output {
  stdout {
   codec => rubydebug { }
  }

  elasticsearch {
    hosts => "es地址IP，例如192.168.0.91"
    index => "log-%{+YYYY.MM.dd}"
  }
}

保存退出，进入bin目录，启动：

./logstash -f ../config/logstash.conf

安装Kibana

解压缩安装包，修改config下配置文件：kibana.yml

server.host: "192.168.0.91"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://192.168.0.91:9200"

需要更改的是两个地方：server.host和elasticsearch.url，更改保存退出~
进bin目录，启动服务~

添加测试数据

官方教程

ElasticSearch的sample data：

account.zip
shakespeare.json
logs.json1.gz

首先加载account数据：

curl -H "Content-Type: application/json" -XPOST 'localhost:9200/bank/account/_bulk?pretty&refresh' --data-binary "@accounts.json"

shaekspeare和logs，先做mapping

curl -X PUT "192.168.0.91:9200/shakespeare" -H 'Content-Type: application/json' -d'
{
 "mappings": {
  "doc": {
   "properties": {
    "speaker": {"type": "keyword"},
    "play_name": {"type": "keyword"},
    "line_id": {"type": "integer"},
    "speech_number": {"type": "integer"}
   }
  }
 }
}
'

logs

curl -X PUT "localhost:9200/logstash-2015.05.18" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "log": {
      "properties": {
        "geo": {
          "properties": {
            "coordinates": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}
'

curl -X PUT "localhost:9200/logstash-2015.05.19" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "log": {
      "properties": {
        "geo": {
          "properties": {
            "coordinates": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}
'

curl -X PUT "localhost:9200/logstash-2015.05.20" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "log": {
      "properties": {
        "geo": {
          "properties": {
            "coordinates": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}
'

导入数据一样：

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/doc/_bulk?pretty' --data-binary @shakespeare_6.0.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl

结束~