How to set input filter logstash?

Elastic Stack Logstash
1

Im bit new to ELK, I need to monitor my application logs(Java tomcat) using ELK, how do I setup a filter in logstash? Should I need to add any this think extra in filebeat.yml other than input path and output hostname and port

2

@rohith3e you've asked this question in the beats forum. I recommend you ask this in the Logstash forum instead. You can find that here: https://discuss.elastic.co/c/logstash

3

Quick lesson, Logstash has three phases of data handling; Input, Filter, and Output. All three phases handle data using plugins, such as beats, elasticsearch, or mutate.

  • Input is where you configure Logstash how to listen for data and do some structuring of the data as it comes in.
  • Filter is where Logstash takes that data and parses it into usable information, this is where you'll likely spend a large portion of your time configuring the pipeline, depending on your needs.
  • Output is where you tell Logstash what to do with that information once its been parsed, such as sending it to Elasticsearch for indexing.

If you are using the Filebeat agent, you have to configure the filebeat.yml of filebeat on the client machine to harvest the log file. For example:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    #Collect all .log file formats in directory
    - /var/log/*.log
    #Collect any file named log.log
    - C:\example\log.log
    #Collect any file starting with logs
    - C:\example\logs*
    #Collect all files in directory
    - C:\example\*
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 5s
setup.template.settings:
  index.number_of_shards: 1
tags: ["service-X", "web-tier"]
setup.kibana:
  host: "localhost:5601"
output.logstash:
  hosts: ["localhost:5044"]

Then, on your ElasticStack host, you'll have to configure a pipeline to listen for the beats agent data and manipulate/output it however you like. For example:

input {
  beats {
    port => 5044
  }
}
filter {
#Insert desired filter plugin configs here
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

There isn't necessarily a correct way for data to be filtered as it is generally up to the individual to determine what's valuable. If you have a more specific question about how to configure a certain plugin or you have some example data and can explain how you'd like it to be presented, there's plenty of people here willing to guide you along.

4

Thank for the inputs they are helpful.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.