Elasticsearch + kibana + fluentdで通信監視
ルータのSyslogを用い、LAN内の通信を可視化。 OSはCentOS7(Minimal Install)。 ルータはRTX1210。
・ElasticsearchにはJDK8が必要。
yum install -y java-1.8.0-openjdk-devel yum install -y java-1.8.0-openjdk-debuginfo --enablerepo=*debug*
Elasticsearchのインストール準備
/etc/yum.repos.d/elastic.repo に以下の内容を記載。
[elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
Elasticsearchのインストール
yum install elasticsearch logstash kibana metricbeat packetbeat filebeat heartbeat -y systemctl start elasticsearch
fluentdのインストール
curl -L http://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh yum group install "Development Tools" -y yum install epel-release -y yum install geoip-devel --enablerepo=epel -y yum install libmaxminddb-devel --enablerepo=epel -y td-agent-gem install fluent-plugin-rewrite-tag-filter fluent-plugin-elasticsearch fluent-plugin-geoip fluent-plugin-multi-format-parser fluent-plugin-parser fluent-plugin-with-extra-fields-parser
rsyslogの設定 (/etc/rsyslog.d/rtx1210.confを作成)
local6.* @127.0.0.1:5140
rsyslogの設定 (UDP経由のSYSLOG受信許可, ll.13-14)
$ModLoad imudp $UDPServerRun 514
fluentdの設定 (/etc/td-agent/td-agent.confに追記)
<source> type syslog tag raw.rtx1210 format none bind 127.0.0.1 port 5140 </source> #### ## Output descriptions: ## <match raw.rtx1210.**> type parser format multi_format key_name message remove_prefix raw add_prefix parsed <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway \[INSPECT\]\s+(?<target>.+)\[(?<direction>.+)\]\[(?<filter_num>\d+)\]\s+(?<proto>.+)\s+(?<src_ip>.+):(?<src_port>.+)\s+>\s+(?<dest_ip>.+):(?<dest_port>.+)\s+\((?<time>.+)\)$/ time_format '%Y/%m/%d %H:%M:%S' extra_fields { "log_type": "inspect" } </pattern> <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway (?<target>.+)\s+Rejected\s+at\s+(?<direction>.+)\((?<filter_num>\d+|default)\)\s+filter:\s+(?<proto>.+)\s+(?<src_ip>.+):(?<src_port>.+)\s+>\s+(?<dest_ip>.+):(?<dest_port>.+)$/ extra_fields { "log_type": "reject" } </pattern> <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway Logout\s+from\s+(?<proto>.+):\s+(?<ip>.+)$/ extra_fields { "log_type": "console_logout" } </pattern> <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway Login\s+succeeded\s+for\s+(?<proto>.+):\s+(?<ip>.+)$/ extra_fields { "log_type": "console_login" } </pattern> <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway \[(?<proto>.+)\]\s+(?<tunnel>.+)\s+connected\s+from\s+(?<src_ip>.+)$/ extra_fields { "log_type": "tunnel_connect" } </pattern> <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway \[(?<proto>.+)\]\s+(?<tunnel>.+)\s+disconnect\s+tunnel\s+\d+\s+complete$/ extra_fields { "log_type": "tunnel_disconnect" } </pattern> <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway \[IKE\]\s+(?<msg>.+)$/ extra_fields { "log_type": "ike" } </pattern> <pattern> format with_extra_fields base_format /^[a-zA-Z]{3}\s+\d+\s+\d+:\d+:\d+ gateway (?<msg>.+)$/ extra_fields { "log_type": "other" } </pattern> </match> <match parsed.rtx1210.**> type rewrite_tag_filter rewriterule1 log_type ^inspect$ temp.rtx1210.inspect rewriterule2 log_type ^reject$ temp.rtx1210.reject rewriterule3 log_type ^console_(.+)$ rtx1210.console.$1 rewriterule4 log_type ^tunnel_(.+)$ rtx1210.tunnel.$1 rewriterule5 log_type ^other$ rtx1210.other </match> <match temp.rtx1210.inspect.**> type geoip geoip_lookup_key dest_ip <record> geo_location '{ "lat" : ${latitude["dest_ip"]}, "lon" : ${longitude["dest_ip"]} }' country_code ${country_code["dest_ip"]} </record> skip_adding_null_record true remove_tag_prefix temp. flush_interval 1s </match> <match rtx1210.inspect.**> type elasticsearch logstash_format true logstash_prefix rtx1210-inspect include_tag_key true tag_key @log_name hosts localhost:9200 buffer_type memory num_threads 1 flush_interval 60 retry_wait 1.0 retry_limit 17 </match> <match temp.rtx1210.reject.**> type geoip geoip_lookup_key src_ip <record> geo_location '{ "lat" : ${latitude["src_ip"]}, "lon" : ${longitude["src_ip"]} }' country_code ${country_code["src_ip"]} </record> remove_tag_prefix temp. skip_adding_null_record true flush_interval 1s </match> <match rtx1210.reject.**> type elasticsearch logstash_format true logstash_prefix rtx1210-reject include_tag_key true tag_key @log_name hosts localhost:9200 buffer_type memory num_threads 1 flush_interval 60 retry_wait 1.0 retry_limit 17 </match> <match rtx1210.console.**> type elasticsearch logstash_format true logstash_prefix rtx1210-console include_tag_key true tag_key @log_name hosts localhost:9200 buffer_type memory num_threads 1 flush_interval 60 retry_wait 1.0 retry_limit 17 </match> <match rtx1210.tunnel.**> type elasticsearch logstash_format true logstash_prefix rtx1210-tunnel include_tag_key true tag_key @log_name hosts localhost:9200 buffer_type memory num_threads 1 flush_interval 60 retry_wait 1.0 retry_limit 17 </match> <match rtx1210.other.**> type elasticsearch logstash_format true logstash_prefix rtx1210-other include_tag_key true tag_key @log_name hosts localhost:9200 buffer_type memory num_threads 1 flush_interval 60 retry_wait 1.0 retry_limit 17 </match>
ElasticSearchのテンプレート
{ "template":"rtx1210-*", "mappings":{ "_default_":{ "dynamic_templates":[ ], "properties":{ "@timestamp":{ "type":"date", "index":"not_analyzed" }, "geo_location":{ "type":"geo_point" }, "src_ip":{ "type":"ip" }, "dest_ip":{ "type":"ip" }, "dest_port":{ "type":"integer" }, "src_port":{ "type":"integer" }, "log_type":{ "type":"keyword" }, "src_port":{ "type":"integer" }, "direction":{ "type":"keyword" }, "filter_num":{ "type":"keyword" }, "proto":{ "type":"keyword" }, "msg":{ "type":"string" }, "target":{ "type":"keyword" } } } } }
elasticsearchのtemplate追加
curl -XPUT localhost:9200/_template/rtx1210-*/ -d "`cat [テンプレートファイル]`"
SYSLOGポート開放
firewall-cmd --add-port 514/udp --permanent firewall-cmd --reload
サービス開始
systemctl start rsyslog systemctl start elasticsearch systemctl start td-agent systemctl start kibana
RTX1210のsyslog設定
この操作はRTX1210で行う。
syslog host [送信先のIPアドレス] syslog facility local6 syslog notice on syslog debug on
以上。
