Same log file contains different format of logs data. How can I parse it?

Elastic Stack Logstash
1

Hi all,
I want to parse the logfile which has a different pattern in almost every line. The main difference is actually the length of each line. Suppose my logs have a named variable, in first log it may contain three names, and in second one it may contain six, or four, and so on. I have gone through a few articles on conditional grok, but I am not able to understand what should I put in the condition for this. How are people identifying what to write in if[type] = "condition. My logs do not contain any conditional like this, it is just the length which is the issue.

2

Can you show examples of your log lines and explain what you want to extract from them?

3

Hi @Badger,
Thank you for responding. I am sharing my logs data. You can see that DependentServices have different numbers in every line. I have written the grok pattern for the logs having 4 dependent Services, if it is more or less it does not shows it in Kibana. Also it does not show _[0]grokoarsefailure. But no data is shown in Kibana. When dependent services are 4 it is coming correctly.

LOG FILES

[INFO] 2020-04-28 15:43:19,030 DetailRequest - DevStack Registration [] [] Running [] DependentService - [DL: PASS] [DB: PASS] [CPU: PASS] [Memory: FAIL]
[INFO] 2020-04-28 15:43:45,207 DetailRequest - DevStack Status [] [] Running [] DependentService - [DCS: PASS] [OpDB: PASS] [CPU: FAIL] [Memory: PASS] [SSL: PASS] [DRIVER: PASS]
[INFO] 2020-04-28 15:44:11,012 DetailRequest - DevStack Gateway [] [] Running [] DependentService - [DL: PASS] [Gateway Data: PASS] [CPU: PASS] [Memory: PASS]
[INFO] 2020-04-28 15:44:37,844 DetailRequest - DevStack Database [] [] Running [] DependentService - [Websocket: PASS] [CPU: FAIL] [Memory: FAIL]
</>

*MY LOGSTASH.CONF FILE**

input{
    file{
        path => "C:/Users/tushar/Documents/logsdata/*.log"
        start_position => "beginning"
        sincedb_path => "NULL"
    }
    #   stdin{}
}


filter{
    grok{
        match => {"message" => "\[%{LOGLEVEL:class}\] %{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:javaclass} - %{GREEDYDATA:stackname} %{GREEDYDATA:servicename} \[%{DATA:procedure}\] \[%{DATA:url}\] %{GREEDYDATA:status} \[%{GREEDYDATA:statuscode}\] %{GREEDYDATA:depservice} - \[%{DATA:dep1}\] \[%{DATA:dep2}\] \[%{DATA:dep3}\] \[%{DATA:dep4}\]"}
    }
    date {
        match => ["timestamp", "ISO8601"]
    }
}
output{
    elasticsearch{
        hosts => "http://localhost:9200"
        index => "depservice"
    }
    stdout{}
}
4

I would use grok to extract

DependentService - %{DATA:restOfLine}

and then a kv filter to parse [restOfLine].

1 Like
5

Actually I am not able to figure out how to put the conditions in it. I have gone through https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals, but it is hard to understand how they are writing the conditions. Can you help me a little or refer me something where I can learn more about these condition writing?

6

I see no reason to use conditionals for this.

7

Then how will the filter know that how much data it has to capture. Like of one line contains 6 dependent service and other contains only 3. Won't it create an issue?

8

The following configuration

    grok { match => { "message" => "DependentService - %{GREEDYDATA:[@metadata][restOfLine]}" } }
    kv { source => "[@metadata][restOfLine]" field_split_pattern => "\] \[|\]$|^\[" value_split => ":" trim_value => " " }

will turn

[INFO] 2020-04-28 15:44:37,844 DetailRequest - DevStack Database [] [] Running [] DependentService - [Websocket: PASS] [CPU: FAIL] [Memory: FAIL]

into

 "Websocket" => "PASS",
    "Memory" => "FAIL",
       "CPU" => "FAIL",

and turn

[INFO] 2020-04-28 15:43:45,207 DetailRequest - DevStack Status [] [] Running [] DependentService - [DCS: PASS] [OpDB: PASS] [CPU: FAIL] [Memory: PASS] [SSL: PASS] [DRIVER: PASS]

into

       "DCS" => "PASS",
    "Memory" => "PASS",
    "DRIVER" => "PASS",
       "CPU" => "FAIL",
      "OpDB" => "PASS",
       "SSL" => "PASS"
9

Thanks a lot @Badger for helping me out.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.