# Help with config file for given syntax

**URL:** <https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304>\
**Category:** Logstash\
**Created:** [March 21, 2019, 12:18pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304 "2019-03-21T12:18:29Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shubham\_Yadav1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_yadav1/32/41659_2.png) [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\
**Post date:** [March 21, 2019, 12:18pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/1 "2019-03-21T12:18:29Z")

</div>

i have a file with data in the format as given below, i have tried several times to index it using logstash conf file but failed. Please help me how can i index in two cases:  
1: all key value pairs  
2: some key value pairs.

**-------------------------------**  
**a=b**  
**c=d**  
**e=3**  
**EOE**  
**-------------------------------**  
**a=z**  
**c=l**  
**e=4**  
**EOE**

---

<div class="post-metadata">

**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\
**Post date:** [March 21, 2019, 1:08pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/2 "2019-03-21T13:08:54Z")

</div>

You will want to start with a file input that uses [the multiline codec](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html).  
Try specifying pattern as EOE, and what as previous.

Use [the kv filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) to extract the key-value pairs. You might need to use two kv filters, the first splitting your multiline on the linebreak, and the second one then reading the individual fields to split the actual key-value pairs.

Test this with [a simple stdout output](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html).

---

<div class="post-metadata">

**Author:** ![Shubham\_Yadav1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_yadav1/32/41659_2.png) [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\
**Post date:** [March 21, 2019, 6:04pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/3 "2019-03-21T18:04:44Z")

</div>

I have tried doing something like this, but i think those "dash" lines aren't taken care of. Those dash lines are also in the input. is there any flaw in the code?

input {  
 file {  
 path =\> "/usr/local/Cellar/logstash/6.6.1/libexec/config/apache-daily-access.log"  
 start\_position =\> "beginning"  
 sincedb\_path =\> "/dev/null"  
 codec =\> multiline {  
 pattern =\> "^EOE"  
# negate = "true"  
 what =\> "previous"  
 }  
 }  
}

filter {  
 mutate {  
 gsub =\> [  
 "message", "\n", ";" ] }  
 kv {  
 allow\_duplicate\_values =\> false  
 field\_split =\> ";"  
 value\_split =\> "="  
 include\_keys =\> ["a", "c", "e"]  
 }

mutate {  
 convert =\> [  
 "f", "integer"  
 ]  
}

}

output {  
 elasticsearch {  
 hosts =\> ["localhost:9200"]  
 # region =\> "us-east-1"  
 index =\> "service\_index"

}  
 stdout { codec =\> json }  
}

---

<div class="post-metadata">

**Author:** ![Shubham\_Yadav1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_yadav1/32/41659_2.png) [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\
**Post date:** [March 22, 2019, 11:47am UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/4 "2019-03-22T11:47:37Z")

</div>

how do i stop multiline filter for not including the dash lines and EOE in previous line?  
this entire data is indexed as one line.  
**EOE**  
**-------------------------------**  
**a=z**  
**c=l**  
**e=4**

---

<div class="post-metadata">

**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\
**Post date:** [March 22, 2019, 11:49am UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/5 "2019-03-22T11:49:53Z")

</div>

You might need to set negate=true, and what=next.  
This will group the events from ------ to EOE.

Then you would remove the ----- line and the EOE line from the message with a filter.

---

<div class="post-metadata">

**Author:** ![Shubham\_Yadav1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_yadav1/32/41659_2.png) [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\
**Post date:** [March 24, 2019, 4:21pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/7 "2019-03-24T16:21:54Z")

</div>

thanks Benny it worked. i have 1 more question.  
i have a file with kv pairs in which startTime key has output in two formats one is in epoch(1553438267.250) and other in unix time format(2019-03-24T14:37:48.096Z),  
how do i convert epoch format to unix format during parsing and then how do i create @timestamp index on startTime values?

---

<div class="post-metadata">

**Author:** ![BennyInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bennyinc/32/21751_2.png) [@BennyInc](https://discuss.elastic.co/u/BennyInc)\
**Post date:** [March 25, 2019, 9:40am UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/8 "2019-03-25T09:40:21Z")

</div>

With the [date filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html) you can actually [match multiple patterns](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match).

`UNIX` will match epoch date, and your other one looks like `ISO8601` to me.

So you might want to use a date filter like this:

```
filter {
  date {
    match => ["startTime","UNIX","ISO8601"]
    target => "@timestamp"
  }
}

```

The default target is already @timestamp, so you can skip that line.

---

<div class="post-metadata">

**Author:** ![Shubham\_Yadav1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_yadav1/32/41659_2.png) [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\
**Post date:** [March 25, 2019, 12:37pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/9 "2019-03-25T12:37:35Z")

</div>

i have posted another problem on different thread, please help me with that.

---

<div class="post-metadata">

**Author:** ![Shubham\_Yadav1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shubham_yadav1/32/41659_2.png) [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\
**Post date:** [March 25, 2019, 4:34pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/10 "2019-03-25T16:34:24Z")

</div>

thanks benny. you rocks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 22, 2019, 4:34pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304/11 "2019-04-22T16:34:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.