# Same log file contains different format of logs data. How can I parse it?

**URL:** <https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742>\
**Category:** Logstash\
**Created:** [June 19, 2020, 7:04am UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742 "2020-06-19T07:04:27Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![TUSHAR\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tushar_sinha/32/69782_2.png) [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\
**Post date:** [June 19, 2020, 7:04am UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/1 "2020-06-19T07:04:28Z")

</div>

Hi all,  
I want to parse the logfile which has a different pattern in almost every line. The main difference is actually the length of each line. Suppose my logs have a named variable, in first log it may contain three names, and in second one it may contain six, or four, and so on. I have gone through a few articles on conditional grok, but I am not able to understand what should I put in the condition for this. How are people identifying what to write in **if[type] = "condition**. My logs do not contain any conditional like this, it is just the length which is the issue.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2020, 11:56am UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/2 "2020-06-19T11:56:33Z")

</div>

Can you show examples of your log lines and explain what you want to extract from them?

---

<div class="post-metadata">

**Author:** ![TUSHAR\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tushar_sinha/32/69782_2.png) [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\
**Post date:** [June 19, 2020, 12:06pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/3 "2020-06-19T12:06:44Z")

</div>

Hi @Badger,  
Thank you for responding. I am sharing my logs data. You can see that DependentServices have different numbers in every line. I have written the grok pattern for the logs having 4 dependent Services, if it is more or less it does not shows it in Kibana. Also it does not show \_[0]grokoarsefailure. But no data is shown in Kibana. When dependent services are 4 it is coming correctly.

**LOG FILES**

```auto
[INFO] 2020-04-28 15:43:19,030 DetailRequest - DevStack Registration [] [] Running [] DependentService - [DL: PASS] [DB: PASS] [CPU: PASS] [Memory: FAIL]
[INFO] 2020-04-28 15:43:45,207 DetailRequest - DevStack Status [] [] Running [] DependentService - [DCS: PASS] [OpDB: PASS] [CPU: FAIL] [Memory: PASS] [SSL: PASS] [DRIVER: PASS]
[INFO] 2020-04-28 15:44:11,012 DetailRequest - DevStack Gateway [] [] Running [] DependentService - [DL: PASS] [Gateway Data: PASS] [CPU: PASS] [Memory: PASS]
[INFO] 2020-04-28 15:44:37,844 DetailRequest - DevStack Database [] [] Running [] DependentService - [Websocket: PASS] [CPU: FAIL] [Memory: FAIL]
</>

*MY LOGSTASH.CONF FILE**

input{
    file{
        path => "C:/Users/tushar/Documents/logsdata/*.log"
        start_position => "beginning"
        sincedb_path => "NULL"
    }
    # stdin{}
}

filter{
    grok{
        match => {"message" => "\[%{LOGLEVEL:class}\] %{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:javaclass} - %{GREEDYDATA:stackname} %{GREEDYDATA:servicename} \[%{DATA:procedure}\] \[%{DATA:url}\] %{GREEDYDATA:status} \[%{GREEDYDATA:statuscode}\] %{GREEDYDATA:depservice} - \[%{DATA:dep1}\] \[%{DATA:dep2}\] \[%{DATA:dep3}\] \[%{DATA:dep4}\]"}
    }
    date {
        match => ["timestamp", "ISO8601"]
    }
}
output{
    elasticsearch{
        hosts => "http://localhost:9200"
        index => "depservice"
    }
    stdout{}
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2020, 12:16pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/4 "2020-06-19T12:16:03Z")

</div>

I would use grok to extract

```
DependentService - %{DATA:restOfLine}

```

and then a kv filter to parse [restOfLine].

---

<div class="post-metadata">

**Author:** ![TUSHAR\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tushar_sinha/32/69782_2.png) [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\
**Post date:** [June 19, 2020, 12:18pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/5 "2020-06-19T12:18:15Z")

</div>

Actually I am not able to figure out how to put the conditions in it. I have gone through [https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals), but it is hard to understand how they are writing the conditions. Can you help me a little or refer me something where I can learn more about these condition writing?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2020, 12:50pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/6 "2020-06-19T12:50:09Z")

</div>

I see no reason to use conditionals for this.

---

<div class="post-metadata">

**Author:** ![TUSHAR\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tushar_sinha/32/69782_2.png) [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\
**Post date:** [June 19, 2020, 1:29pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/7 "2020-06-19T13:29:03Z")

</div>

Then how will the filter know that how much data it has to capture. Like of one line contains 6 dependent service and other contains only 3. Won't it create an issue?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 19, 2020, 2:46pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/8 "2020-06-19T14:46:34Z")

</div>

The following configuration

```
    grok { match => { "message" => "DependentService - %{GREEDYDATA:[@metadata][restOfLine]}" } }
    kv { source => "[@metadata][restOfLine]" field_split_pattern => "\] \[|\]$|^\[" value_split => ":" trim_value => " " }

```

will turn

```
[INFO] 2020-04-28 15:44:37,844 DetailRequest - DevStack Database [] [] Running [] DependentService - [Websocket: PASS] [CPU: FAIL] [Memory: FAIL]

```

into

```
 "Websocket" => "PASS",
    "Memory" => "FAIL",
       "CPU" => "FAIL",

```

and turn

```
[INFO] 2020-04-28 15:43:45,207 DetailRequest - DevStack Status [] [] Running [] DependentService - [DCS: PASS] [OpDB: PASS] [CPU: FAIL] [Memory: PASS] [SSL: PASS] [DRIVER: PASS]

```

into

```
       "DCS" => "PASS",
    "Memory" => "PASS",
    "DRIVER" => "PASS",
       "CPU" => "FAIL",
      "OpDB" => "PASS",
       "SSL" => "PASS"
```

---

<div class="post-metadata">

**Author:** ![TUSHAR\_Sinha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tushar_sinha/32/69782_2.png) [@TUSHAR\_Sinha](https://discuss.elastic.co/u/TUSHAR_Sinha)\
**Post date:** [June 19, 2020, 2:49pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/9 "2020-06-19T14:49:19Z")

</div>

Thanks a lot @Badger for helping me out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2020, 2:49pm UTC](https://discuss.elastic.co/t/same-log-file-contains-different-format-of-logs-data-how-can-i-parse-it/237742/10 "2020-07-17T14:49:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.