Hi All,
Currently I run both filebeat and heartbeat in docker and would pass data to Logstash to filter before those data will be forwarded again to elasticsearch.
Also I would separate data from filebeat and heartbeat to be stored in elasticesearch with different index.
However look like data from heartbeat cannot be sent to Logstash with some reason but data from filebeat can be sent as usual.
Here is some configuration from filebeat.yml. it will monitor log from specific folder and send log information to Logstash
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
- input_type: log
paths:
- c:\logfiles\*.log
fields:
application: Filebeat_test
fields_under_root: true
multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
multiline.negate: true
multiline.match: after
#----------- Logstash output ----------
output.logstash:
# The Logstash hosts
hosts: ["logstash:5044"]
Here are some configuration from heartbeat.yml. it will keep monitoring the webapp and keep polling one SOAP web service and send information to Logstash
######## Heartbeat ##################
heartbeat.config.monitors:
heartbeat.monitors:
- type: http
application: Heartbeat_Test
urls: ["http://192.168.40.84:81/"]
schedule: '@every 15s'
timeout: 16s
- type: http
application: Heartbeat_Test
urls: ["http://192.168.40.84:89/WebService1.asmx?WSDL/"]
check.request.method: POST
check.request.header:
soapaction: 'http://tempuri.org/CloseBoat'
content-type: 'text/xml; charset=utf-8'
check.request.body: '<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<CloseBoat xmlns="http://tempuri.org/">
<Boat>TEST3333</Boat>
</CloseBoat>
</soap:Body>
</soap:Envelope>'
schedule: '@every 30s'
#----------- Logstash output ----------
output.logstash:
# The Logstash hosts
hosts: ["logstash:5044"]
Here are some configuration from heartbeat.yml. it will keep monitoring the webapp and keep polling one SOAP web service and send information to Logstash
#----------- Logstash.conf----------
input {
beats {
port => 5044
}
}
filter {
if [application] == "Filebeat_Test" {
grok {
match => { "message" => "(?<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<task>\[\d+\]) (?<level>\w+) (?<class>\w+) - \[(?<batch>\w+)\]%{GREEDYDATA:message}" }
match => { "message" => "(?<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<task>\[\d+\]) (?<level>\w+) (?<class>\w+) - %{GREEDYDATA:message}" }
overwrite => ["message"]
}
}
date {
match => ["datetime", "YYYY-MM-dd HH:mm:ss,SSS"]
target => "@timestamp"
}
}
output {
if [application] == "Filebeat_Test" {
elasticsearch {
hosts => [ "elasticsearch-node1:9200" ]
manage_template => false
index => "filebeat-test-%{+YYYY.MM.dd}"
}
}
if [application] == "Heartbeat_Test" {
elasticsearch {
hosts => [ "elasticsearch-node1:9200" ]
manage_template => false
index => "heartbeat-test-%{+YYYY.MM.dd}"
}
}
}
So for filebeat everything work fine as expected but for heartbeat i'm not really sure how it work.
because look like lofstash cannot get any information from heartbeat.
- Also i'm not really sure how to config heartbeat to poll request to SOAP webservice because look like current configuration cannot work.
- How to set application name for group data from heartbeat, because seem like current setting cannot work.
- How to filter data from heartbeat? please give me some example. thanks in advance.