Send both filebeats and heartbeat to logstash did not work

Elastic Stack Beats
1

Hi All,
Currently I run both filebeat and heartbeat in docker and would pass data to Logstash to filter before those data will be forwarded again to elasticsearch.
Also I would separate data from filebeat and heartbeat to be stored in elasticesearch with different index.
However look like data from heartbeat cannot be sent to Logstash with some reason but data from filebeat can be sent as usual.

Here is some configuration from filebeat.yml. it will monitor log from specific folder and send log information to Logstash

#=========================== Filebeat prospectors =============================

filebeat.prospectors:
- input_type: log
  paths:
    - c:\logfiles\*.log
  fields:
    application: Filebeat_test
  
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after

#----------- Logstash output ----------
output.logstash:
# The Logstash hosts
hosts: ["logstash:5044"]

Here are some configuration from heartbeat.yml. it will keep monitoring the webapp and keep polling one SOAP web service and send information to Logstash

######## Heartbeat ##################
    heartbeat.config.monitors:
    heartbeat.monitors:
    - type: http
      application: Heartbeat_Test
      urls: ["http://192.168.40.84:81/"]
      schedule: '@every 15s'
      timeout: 16s
    - type: http
      application: Heartbeat_Test
      urls: ["http://192.168.40.84:89/WebService1.asmx?WSDL/"]
      check.request.method: POST
      check.request.header:
        soapaction: 'http://tempuri.org/CloseBoat'
        content-type: 'text/xml; charset=utf-8'
      check.request.body: '<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
                              <soap:Body>
                                <CloseBoat xmlns="http://tempuri.org/">
                                  <Boat>TEST3333</Boat>
                                </CloseBoat>
                              </soap:Body>
                            </soap:Envelope>'
      schedule: '@every 30s'    
#----------- Logstash output ----------
        output.logstash:
            # The Logstash hosts
            hosts: ["logstash:5044"]

Here are some configuration from heartbeat.yml. it will keep monitoring the webapp and keep polling one SOAP web service and send information to Logstash

#----------- Logstash.conf----------
input {
    beats {
        port => 5044
    }
}
filter {
  if [application] == "Filebeat_Test" {
	  grok {
		match => { "message" => "(?<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<task>\[\d+\]) (?<level>\w+) (?<class>\w+) - \[(?<batch>\w+)\]%{GREEDYDATA:message}" }
		match => { "message" => "(?<datetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<task>\[\d+\]) (?<level>\w+) (?<class>\w+) - %{GREEDYDATA:message}" }
		overwrite => ["message"]
		}
  }
  date {
	match => ["datetime", "YYYY-MM-dd HH:mm:ss,SSS"]
	target => "@timestamp"
  }
}

output {
  if [application] == "Filebeat_Test" {
	  elasticsearch {
		hosts => [ "elasticsearch-node1:9200" ]
		manage_template => false
		index => "filebeat-test-%{+YYYY.MM.dd}" 
	  }
  }
  if [application] == "Heartbeat_Test" {
	  elasticsearch {
		hosts => [ "elasticsearch-node1:9200" ]
		manage_template => false
		index => "heartbeat-test-%{+YYYY.MM.dd}" 
	  }
 }   
}

So for filebeat everything work fine as expected but for heartbeat i'm not really sure how it work.
because look like lofstash cannot get any information from heartbeat.

  • Also i'm not really sure how to config heartbeat to poll request to SOAP webservice because look like current configuration cannot work.
  • How to set application name for group data from heartbeat, because seem like current setting cannot work.
  • How to filter data from heartbeat? please give me some example. thanks in advance.
2

Are you trying to monitor the uptime of the SOAP service, or index its data? Heartbeat is only for uptime. If you need to pull the actual data you may want the HTTP Poller Input for Logstash

3

Hi Andrew,
I want to monitor the uptime for both SOAP web service and web app whether both application still up and would like to send result status to Logstash to filter and extract some data. (for example HTTP status code)
Appreciate your help if you can guide me to achieve what I want to do.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.