Parsing specific values from logs in logstash configuration

Elastic Stack Logstash
1

Hi,

I am currently using default logstash configuration to parse the apache server access log file and index it to elasticsearch.

However, I don't want to parse all fields in log but say specific ones like IP, Referrer, status code, etc.

How can I selectively specify in configuration of logstash to parse only those fields.

My current configuration is as follows:

input {
beats {
port => 5044
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}

output {
elasticsearch {
hosts => "localhost:9200"
#manage_template => false
#index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
#document_type => "%{[@metadata][type]}"
}

stdout { codec => rubydebug }
}

Can someone throw light on it?

Thanks.
Saurabh

2

You need to parse them all, then use remove_field to drop things.

3

Instead of using remove_field you can use the prune filter to delete all fields except those you want to keep.

1 Like
4

Even better!

5

Can you please give example pf prune filter on how to selectively chose the fields?

Thanks in advance.
Saurabh

6

https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html has one.

7

Thanks. Is there a documentation where I can see how each field in log is mapped? For instance; date may be referred as "datetimestamp" in logstash configuration.

I want to add performance logging in apache in httpd.conf file where i can print time taken by request on application server. How will i parse that value from log. So i want to understand how the fields are mapped and referred in filters.

Sorry for basic questions but i am newbie to this world.

Thanks a lot for your help.

8

That's https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html.

You can see what each of the patterns means here - https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

9

I want to add performance logging in apache in httpd.conf file where i can print time taken by request on application server. How will i parse that value from log.

Apache doesn't log this by default so the "how to parse" question depends on how you make Apache log that information in the first place.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.