Remove header from log file

Elastic Stack Logstash
1

Hi,

Can somebody please check below problem:

I have message file in which only one line is there "DateTime,Session,url,errormessage,stacktrace,body". I don't want this line to be indexed in ES.

So, I have written below filter pattern in logstash file but it's not working somehow
filter{
if [message] =~ /^DateTime/ {
drop {}
}
}

It would really be helpful if somebody can tell what I am missing here.

Regards,
Sweta

2

What you have looks fine and should work. If you use a stdout { codec => rubydebug } output, what does an event that should've been dropped look like?

3

Can you tell me where to insert the statement you provided in my logstash conf file?? Also, The pattern I have given not working when I am writing it in to logstash conf file.

4

Also Magnus, I want my error file to indexed in ES but there message field should not contain anything. Is this possible?

5

Can you tell me where to insert the statement you provided in my logstash conf file??

In the output block.

Also, The pattern I have given not working when I am writing it in to logstash conf file.

Always show

  • exactly what you've tried and
  • exactly what happened when you tried that.

I want my error file to indexed in ES but there message field should not contain anything.

What error file? If you want to remove the message field you can e.g. use a mutate filter.

6

Thanks Magnus for quick replying

Error file content which I am trying to filter is as below: DateTime,Session,url,errormessage,stacktrace,body

The below is my logstash.conf file
input {
beats {
port => 5044
}
}
filter
{
if [fields][log_type] == "errorlogs" {
if [message] =~ /^DateTime/ {
drop {}
}
}
}

output {

if [fields][log_type] == "errorlogs" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "errorlogs"
document_type => "%{[@metadata][type]}"
}
}
}

But after executing this, In ES errorlogs indices is not getting created. any idea?

Regards,
Sweta

7

How do you know that [fields][log_type] contains "errorlogs"?

8

I have given it in my filebeat.yml as
fields: {log_type: errorlogs}

9

Sure, but that doesn't prove anything. For example, if fields_under_root is true the field will end up as plain log_type. You need to challenge all assertions. What does the event really look like?