Beat and Logstash

Elastic Stack Logstash
1

I am new to coding and logstash.
I am receiving many files through filebeat on single port in my logstash.conf.

How can I differentiate the files basis on their naming conventions and then write a filter for them.

Anyone can share the sample of logstash.conf file.

Thanks
Siddharth

2

Update your Filebeat configuration to set fields on events from each file. Those fields could e.g. describe the application name and the environment name. Based on that you can apply custom filtering on the Logstash side.

1 Like
3

Thanks, I would really appreciate if you can share a way that i can do something to differentiate the files in logstash.conf. Is it possible? Would be a good practice?

Thanks

4

I would really appreciate if you can share a way that i can do something to differentiate the files in logstash.conf. Is it possible?

Probably. Do you have any examples?

Would be a good practice?

I think it's a better idea to classify logs at the source (in this case Filebeat) where you know the truth instead of guessing afterwards.

5

Thanks as always, for every advise.
You mean, i should tag the file in filebeat.conf and then use that tag in logstash.conf for filtering.

filebeat example 1:

filebeat.prospectors:

  • paths: ["/var/log/smtp.log"]
  • tags: ["smtp"]

logstash example1:

filter{
if "smtp" in [tags] {
processing.....
}
}

filebeat example2:

filebeat.prospectors:

  • paths: ["/var/log/smtp.log"]
  • document_type: smtp
  • input_type: log

logstash example2:
filter{
if [type] == "smtp" {
processing....
}
}

can i use either way to differentiate the file in filebeat.conf?
both are the correct ways to differentiate the file in filebeat.conf?

Thanks

6

You mean, i should tag the file in filebeat.conf and then use that tag in logstash.conf for filtering.

Yes.

can i use either way to differentiate the file in filebeat.conf?

Yes.

both are the correct ways to differentiate the file in filebeat.conf?

Yes.

7

Thankyou so much Magnus.
I honestly appreciate your help.

8

Dear Magnus,

Can you review it as i m getting an error in filebeat "Failed to Publish events: write: connection reset by peer" while pushing the file from Filebeat to Logstash.

filebeat.prospectors:

  • type: log
    enabled: true
    paths:
    • /app/uce/uce.csv
      document_type: spam

logstash.conf
input {
port => "10011"
[type] => "spam"
}
filter {
if [type] == "spam" {
}
}
output {
if [type] == "spam" {
}
}

Thanks

9

What does Filebeat's output configuration look like?

Always format configuration snippets as preformatted text.

11

here is my filebeat.yml configuration.

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /app/uprotect/uprotect-*.csv
      #- c:\programdata\elasticsearch\logs*
      document_type: spam

    Exclude lines. A list of regular expressions to match. It drops the lines that are

    matching any regular expression from the list.

    #exclude_lines: ['^DBG']

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["10.224.225.18:10001"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

Thanks

12

I repeat: Always format configuration snippets as preformatted text. There's a toolbar button for it.

input {
port => "10011"
[type] => "spam"
}

That's invalid syntax so I'm surprised Logstash even starts.

input {
  beats {
    port => 10011
    type => "spam"
  }
}
1 Like
13

Thanks, It is now working.
I can see the harvester is running and sending data to my logstash and logstash is tagging it as i want.

But I am getting something like below, actually its many.

at io.netty.channel.MultipthreadEvenLoopGroup.register
at io.netty.bootstrap.AbstractBootstrap.initAndRegister
at io.netty.bootstrap.AbstractBootstrap.doBind
at io.netty.bootstrap.AbstractBootstrap.bind
at sun.reflect.GeneratedMethodAccessor41
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling
at org.jruby.javasupport.JavaMethod.invokeDirect

Thanks

14

Please post the full error message.

15

Thanks Magnus for always be with me.
:slight_smile:
Everything is now placed and working.

V soon i will share my conf files, which i think will be helpful for others.

Thanks
Siddharth

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.