Parsing Log With grok Filter

Elastic Stack Logstash
1

Hello EveryOne,

My name is samir and i am a trainee in a company in france.

I have a project to realize: Establish the ELK suite on logs of different applications of the company.

I need your help because at the moment I encounter difficulties with the filter grok.

I explain my problem, This is my logstash configuration file :

And this is the content of the log :

image.png1160×386 22.6 KB

As you have seen I work in mutiline. (Multiple lines in one document).

My goal is to extract only lines containing "#####".
I tried with the following filter : match => {"message" => "(?(?:^|\n).#####.(?:^\n|$))"}
But logstash sends me the whole document while I want only the line.

For example I want him to return :

image.png922×17 1.79 KB

But he return this :

image.png1001×114 4.95 KB

I hope you understand my problem and that you could help me.

Thanks so much !!!!!

2

You can try using multiline with the #### pattern.

https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html

3

Or more precisely, use the timestamp pattern for everything that does not start with it.

    codec => multiline {
       pattern => "^\d\d\.\d\d\.\d\d\.\d\d\d"
      negate => true
      what => previous
    }
4

Yess !

I succeeded. Thank you very much all.
I have just a last question, The structure of my logs is as follows :

10:00:17.307 Int 22000 ##### EI_COF_RCCAuto_v7 - 00f7029cddc0cf9f - RCCAUTO-0614304044- MAJ CFA suite
10:00:17.307_I_I_00f7029cddc0cf9f [07:48] func will be continued(0,0000000030c000c6)
10:00:17.307_M_I_00f7029cddc0cf9f [17:11] VQ 000000000478eca0 first available call: none, reason=(0)strategy
10:00:17.307_I_I_00f7029cddc0cf9f [09:04] <<<<<<<<<<<<suspend interp(JUMPING), func:CallStrategy timers:00001

Every hour it is a new document. For each document I would like to extract the time and put it in a timestamp variable.
Can you help me ?

5

You have to use grok to match the your timestamp pattern and then use the date filter to set the @timestamp .

For example:

filter{
grok {
    match => [ "message", "%{TIMESTAMP_ISO8601:event_timestamp} %{GREEDYDATA:message}" ]
  }

date {
match => ["event_timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
target => "@timestamp"
}
}
6

Ohhh Thank you very much it works !!

The time is good but it just sends me a wrong date. I think this is normal because at the beginning of each line there is only the time and not the date. The date is at the very beginning of the document.
I would like to know if we can retrieve the date and put it in the @timestamp too. So that there is the right date and the right time also.

The header of my logs urs looks like this :

So I would like to retrieve the highlighted date and put it in all the timestamp of each document.
I don't know if that's possible.

7

ES stores the @timestamp in UTC. ES does not know that your time is already in UTC because it doesn't have Z (UTC timezone) designated at the end of the time, so it did the adjustment for you. That's why you see a different date.

Not sure if you can add Z at the end of 2017-03-24T09:01:54.541 to 2017-03-24T09:01:54.541Z on the source side. Otherwise, you will have to do this in Logstash. If neither works for you, you can always do it during query time.

8

I added the Z in the UTC Time line as you said.

But the timestamp does not change, it keeps me the default date 01/01/2017.

How could I do from logstash?

9

The example I provided early did it exactly for me.

Output

{
               "path" => "/tmp/time.data",
         "@timestamp" => 2017-03-24T09:01:43.541Z,
           "@version" => "1",
               "host" => "JImmys-MacBook-Pro.local",
            "message" => "2017-03-24T09:01:43.541",
               "type" => "apache-access",
    "event_timestamp" => "2017-03-24T09:01:43.541Z"
}
10

I understand. Me my worry is that my different documents start directly by the time, there is no date. The date is only in the log header.

This is my log header :

In all the log file the date is only here.

And then the different documents of the log starts directly by the hour, like this :

image.png896×70 4.16 KB

11

Initially, I thought your log header was your full log. If you only have the time in your log, you will have to append the date to it along with the Z. You can do this in Logstash using mutate replace.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-replace

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.