ES01# FileBeat與Elasticsearch集成

引言

對Elasticsearch體系化學習梳理，本文為第一篇，filebeat與Elasticsearch的集成部署，文章主要內(nèi)容有：

• Elasticsearch安裝與部署
• Kibana安裝與部署
• FileBeat與Elasticsearch集成

//?下載地址?本文為Elasticsearch?7.10.2
https://www.elastic.co/cn/downloads/past-releases#elasticsearch

2.安裝包目錄

3.集群部署

部署一個由三個幾點組成的ES集群，下面為詳細步驟。

安裝文檔

https://www.elastic.co/guide/en/elasticsearch/reference/7.10/targz.html

參數(shù)說明

bin/elasticsearch?-h
Option????????????????Description???????????????????????????????????????????????
------????????????????-----------???????????????????????????????????????????????
-E??????Configure?a?setting???????????????????????????????????????
-V,?--version?????????Prints?Elasticsearch?version?information?and?exits????????
-d,?--daemonize???????Starts?Elasticsearch?in?the?background????????????????????
-h,?--help????????????Show?help?????????????????????????????????????????????????
-p,?--pidfile???Creates?a?pid?file?in?the?specified?path?on?start?????????
-q,?--quiet???????????Turns?off?standard?output/error?streams?logging?in?console
-s,?--silent??????????Show?minimal?output???????????????????????????????????????
-v,?--verbose?????????Show?verbose?output??

安裝命令

bin/elasticsearch?-d?-Ecluster.name=melon_cluster?-Enode.name=node_1??-Epath.data=node1_data

bin/elasticsearch?-d?-Ecluster.name=melon_cluster?-Enode.name=node_2??-Epath.data=node2_data

bin/elasticsearch?-d?-Ecluster.name=melon_cluster?-Enode.name=node_3??-Epath.data=node3_data

備注：通過-E來指定K/V存儲參數(shù)，cluster.name集群名稱，node.name節(jié)點名稱，path.data數(shù)據(jù)存儲目錄

集群節(jié)點

curl?http://localhost:9200/_cat/nodes
127.0.0.1?33?100?30?4.40???cdhilmrstw?*?node_1
127.0.0.1?22?100?30?4.40???cdhilmrstw?-?node_3
127.0.0.1?24?100?30?4.40???cdhilmrstw?-?node_2

備注：集群三個節(jié)點構(gòu)成

集群健康

curl?http://localhost:9200/_cluster/health
{
????"cluster_name":"melon_cluster",
????"status":"green",
????"timed_out":false,
????"number_of_nodes":3,
????"number_of_data_nodes":3,
????"active_primary_shards":0,
????"active_shards":0,
????"relocating_shards":0,
????"initializing_shards":0,
????"unassigned_shards":0,
????"delayed_unassigned_shards":0,
????"number_of_pending_tasks":0,
????"number_of_in_flight_fetch":0,
????"task_max_waiting_in_queue_millis":0,
????"active_shards_percent_as_number":100
}

備注：集群狀態(tài)為健康green

//?下載地址，?本文為?Kibana?7.10.2
https://www.elastic.co/cn/downloads/past-releases#kibana

2.啟動kibana

bin/kibana
...
log???[14:08:09.911]?[info][plugins][watcher]?Your?basic?license?does?not?support?watcher.?Please?upgrade?your?license.
log???[14:08:09.915]?[info][kibana-monitoring][monitoring][monitoring][plugins]?Starting?monitoring?stats?collection
log???[14:08:10.984]?[info][listening]?Server?running?at?http://localhost:5601
log???[14:08:11.936]?[info][server][Kibana][http]?http?server?running?at?http://localhost:5601
...

3.界面顯示

瀏覽器訪問：http://localhost:5601，根據(jù)引導導入一些測試數(shù)據(jù)

4.狀態(tài)檢查

http://localhost:5601/status

5.執(zhí)行ES語法

檢查集群狀況情況。

通常FileBeat不直接寫入Elasticsearch，先寫入Kafka削峰填谷，再消費數(shù)據(jù)寫入Elasticsearch。本文FileBeat直接寫入Elasticsearch，通過Kibana查詢寫入的數(shù)據(jù)。

1.下載安裝包

本文以Filebeat?7.15.2為例
https://www.elastic.co/cn/downloads/past-releases#filebeat

2.官方安裝文檔

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html

3.配置修改

3.1 配置輸入目錄

type:?log
??#?Change?to?true?to?enable?this?input?configuration.
??enabled:?true
??#?Paths?that?should?be?crawled?and?fetched.?Glob?based?paths.
??paths:
????-?/Users/admin/logs/csp/*.log
????
filebeat.config.modules:
??#?Glob?pattern?for?configuration?loading
??path:?${path.config}/modules.d/*.yml

??#?Set?to?true?to?enable?config?reloading
??reload.enabled:?true

??#?Period?on?which?files?under?path?should?be?checked?for?changes
??reload.period:?10s
?

備注：在filebeat.yml將enable設(shè)置為TRUE，指定收集目錄path。

3.2 配置kibana

setup.kibana:

??#?Kibana?Host
??#?Scheme?and?port?can?be?left?out?and?will?be?set?to?the?default?(http?and?5601)
??#?In?case?you?specify?and?additional?path,?the?scheme?is?required:?http://localhost:5601/path
??#?IPv6?addresses?should?always?be?defined?as:?https://[2001:db8::1]:5601
??host:?"localhost:5601"

備注：在filebeat.yml指定kibana部署地址。

3.3 配置輸出elasticsearch

output.elasticsearch:
??#?Array?of?hosts?to?connect?to.
??hosts:?["localhost:9200"]

備注：在filebeat.yml指定elasticsearch的地址。

4.部署啟動

sudo?chown?root?filebeat.yml
sudo?filebeat?-e

5.檢索日志