Topbeat not talking to Logstash

Elastic Stack Beats
1

Hey, I am just getting going with Topbeat and am having trouble getting it to talk to logstash. I have installed to plugin and followed all the instructions, but am still not getting data. Topbeat says it is connected and sending data successfully, but even running in debug, logstash outputs nothing. Here is my topbeat.yml

input:
  # In seconds, defines how often to read server statistics
  period: 30 

  # Regular expression to match the processes that are monitored
  # By default, all the processes are monitored
  procs: [".*"]

  # Statistics to collect (all enabled by default)
  stats:
    # per system statistics, by default is true
    system: true

    # per process statistics, by default is true
    proc: true

    # file system information, by default is true
    filesystem: true

    # cpu usage per core, by default is false
    cpu_per_core: false

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:
  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["10.100.3.49:8989"]


############################# Shipper #########################################

shipper:
  # Configure local GeoIP database support.
  # If no paths are not configured geoip is disabled.
  geoip:
    paths:
      - "/usr/share/GeoIP/GeoLiteCity.dat"
      # - "/usr/local/var/GeoIP/GeoLiteCity.dat"


############################# Logging #########################################

# There are three options for the log output: syslog, file, stderr.
# Under Windows systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

  # Send all logging output to syslog. On Windows default is false, otherwise
  # default is true.
  to_syslog: false

  # Write all logging output to files. Beats automatically rotate files if rotateeverybytes
  # limit is reached.
  #to_files: false

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    #path: /var/log/mybeat

    # The name of the files where the logs are written to.
    #name: mybeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    #keepfiles: 7

  # Enable debug output for selected components. To enable all selectors use ["*"]
  # Other available selectors are beat, publish, service
  # Multiple selectors can be chained.
  #selectors: [ ]

  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  level: debug

And here is my logstash config:

input {
	udp { 
		port => 5000 
		codec => json 
		type => "log"
		} 
	tcp { 
		port => 5050 
		codec => json
		type => "log"
		}
	beats {
		port => 8989
		type => "metric"
	}
}

## Add your filters here

output {
	if [type] == "log" {
		elasticsearch {
			hosts => "es-central:9200"
			index =>"app-logs-%{+YYYY.MM.dd}"
		} 
	} else if [type] == "metric" {
  		stdout { codec => rubydebug }
		elasticsearch {
			hosts => "es-central:9200"
    			manage_template => false
    			index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    			document_type => "%{[@metadata][type]}"
		}
	}    

}

Can anyone gander what may be the problem here? Thanks

2

Start with a simple Logstash configuration and incrementally add features/complexity.

logstash.conf:

input {
  beats {
    port => 5044
  }
}

output {
   stdout { codec => rubydebug { metadata => true } }
}

Start Logstash in the foreground:

/opt/logstash/bin/logstash -f logstash.conf

Be sure to read the logstash-input-beats documentation around type. Basically if type is already set by the Beat then you cannot override it.

3

Ok, so that works. Where am I going wrong? From reading the docs it looked like the if else statement was the way to go, but now that appears to be the problem.

4

You could tag the data is Topbeat with:

shipper:
  tags: ["metric"]

Then in Logstash use if "metric" in [tags] for your conditional.

5

Awesome, thank you. This has made me wonder if there is any point in sending the data to logstash. Will topbeat time index these logs on its own? And will it multi field string values?

6

I would just route Topbeat directly into Elasticsearch. Topbeat uses daily indexes. It will not create multi fields. Why do you need them? You can change the topbeat.template.json file to customize the index template if you need multi fields.

7

Yeah I just don't like dealing with the mappings, and I need direct matches on some strings that contain forward slashes.

8

What do you mean by "direct matches"? Exact matches? That would imply non_analyzed fields to me, which is what the provided Topbeat index template uses for all strings.

9

Oh so by default it doesn't analyze strings?

10

Correct, it uses this dynamic template which defaults to non_analyzed.

11

That is awesome, thank you for all the help. You guys are the best in the business

1 Like