Same log file contains different format of logs data. How can I parse it?

Hi all,
I want to parse the logfile which has a different pattern in almost every line. The main difference is actually the length of each line. Suppose my logs have a named variable, in first log it may contain three names, and in second one it may contain six, or four, and so on. I have gone through a few articles on conditional grok, but I am not able to understand what should I put in the condition for this. How are people identifying what to write in if[type] = "condition. My logs do not contain any conditional like this, it is just the length which is the issue.

Can you show examples of your log lines and explain what you want to extract from them?

Hi @Badger,
Thank you for responding. I am sharing my logs data. You can see that DependentServices have different numbers in every line. I have written the grok pattern for the logs having 4 dependent Services, if it is more or less it does not shows it in Kibana. Also it does not show _[0]grokoarsefailure. But no data is shown in Kibana. When dependent services are 4 it is coming correctly.

LOG FILES

[INFO] 2020-04-28 15:43:19,030 DetailRequest - DevStack Registration [] [] Running [] DependentService - [DL: PASS] [DB: PASS] [CPU: PASS] [Memory: FAIL]
[INFO] 2020-04-28 15:43:45,207 DetailRequest - DevStack Status [] [] Running [] DependentService - [DCS: PASS] [OpDB: PASS] [CPU: FAIL] [Memory: PASS] [SSL: PASS] [DRIVER: PASS]
[INFO] 2020-04-28 15:44:11,012 DetailRequest - DevStack Gateway [] [] Running [] DependentService - [DL: PASS] [Gateway Data: PASS] [CPU: PASS] [Memory: PASS]
[INFO] 2020-04-28 15:44:37,844 DetailRequest - DevStack Database [] [] Running [] DependentService - [Websocket: PASS] [CPU: FAIL] [Memory: FAIL]
</>

*MY LOGSTASH.CONF FILE**

input{
    file{
        path => "C:/Users/tushar/Documents/logsdata/*.log"
        start_position => "beginning"
        sincedb_path => "NULL"
    }
    #   stdin{}
}


filter{
    grok{
        match => {"message" => "\[%{LOGLEVEL:class}\] %{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:javaclass} - %{GREEDYDATA:stackname} %{GREEDYDATA:servicename} \[%{DATA:procedure}\] \[%{DATA:url}\] %{GREEDYDATA:status} \[%{GREEDYDATA:statuscode}\] %{GREEDYDATA:depservice} - \[%{DATA:dep1}\] \[%{DATA:dep2}\] \[%{DATA:dep3}\] \[%{DATA:dep4}\]"}
    }
    date {
        match => ["timestamp", "ISO8601"]
    }
}
output{
    elasticsearch{
        hosts => "http://localhost:9200"
        index => "depservice"
    }
    stdout{}
}

I would use grok to extract

DependentService - %{DATA:restOfLine}

and then a kv filter to parse [restOfLine].

1 Like

Actually I am not able to figure out how to put the conditions in it. I have gone through https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals, but it is hard to understand how they are writing the conditions. Can you help me a little or refer me something where I can learn more about these condition writing?

I see no reason to use conditionals for this.

Then how will the filter know that how much data it has to capture. Like of one line contains 6 dependent service and other contains only 3. Won't it create an issue?

The following configuration

    grok { match => { "message" => "DependentService - %{GREEDYDATA:[@metadata][restOfLine]}" } }
    kv { source => "[@metadata][restOfLine]" field_split_pattern => "\] \[|\]$|^\[" value_split => ":" trim_value => " " }

will turn

[INFO] 2020-04-28 15:44:37,844 DetailRequest - DevStack Database [] [] Running [] DependentService - [Websocket: PASS] [CPU: FAIL] [Memory: FAIL]

into

 "Websocket" => "PASS",
    "Memory" => "FAIL",
       "CPU" => "FAIL",

and turn

[INFO] 2020-04-28 15:43:45,207 DetailRequest - DevStack Status [] [] Running [] DependentService - [DCS: PASS] [OpDB: PASS] [CPU: FAIL] [Memory: PASS] [SSL: PASS] [DRIVER: PASS]

into

       "DCS" => "PASS",
    "Memory" => "PASS",
    "DRIVER" => "PASS",
       "CPU" => "FAIL",
      "OpDB" => "PASS",
       "SSL" => "PASS"

Thanks a lot @Badger for helping me out.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.