# Parsing log with logstash

**URL:** <https://discuss.elastic.co/t/parsing-log-with-logstash/212280>\
**Category:** Logstash\
**Created:** [December 18, 2019, 8:29am UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280 "2019-12-18T08:29:16Z")\
**Posts on this page:** 16\
**Page:** 2

<div class="post-metadata">

**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\
**Post date:** [December 19, 2019, 9:36am UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/21 "2019-12-19T09:36:15Z")

</div>

i have the same problem please kindly help me, here are my scenarios  
i currently learning using logstash and beat, and my first lesson is ingesting csv file using filebeat to logstash and visualize it with kibana on elasticsearch. It works, the config file i use is logstash.conf  
now i try to ingesting file with grok filter, on the **same** config file by editing the previous config file that i use to ingesting csv file. and this problem appears.  
questions are :

1. how do logstash pipeline works? I mean i've deleted the previous config and according to the error it seems there is still a running pipeline?
2. how to fix this error.

please kindly help me with my silly questions. thank you so much

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [December 19, 2019, 2:00pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/22 "2019-12-19T14:00:23Z")

</div>

yes you can remove the space function is called split, read about it what it does.

mutate { strip =\> ["field\_name"] }

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [December 19, 2019, 2:04pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/23 "2019-12-19T14:04:33Z")

</div>

actually it would be good if you open another thread.

but pipeline is something if you want to run config all the time with schedule  
command line something if you want to test it.

you have not got the funda down yet.

when you start the logstash daemon, it reads the /etc/logstash/pipeline.yml file and runs all the config listed under that pileline file. and it keeps running. basically it is logstash daemon stays in memory and runs it with all the parameter you have in pipeline.yml file

if you have that daemon running that means you won't be able to run another logstash command on same machine.

once you run logstash daemon it reads all the config and all the setting and puts it in memory. that means after starting daemon if you change your config file it will not take effect of your changes. you have to restart your daemon.  
There is a option in logstash.yml which let you do it. but I have not try it

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [December 19, 2019, 4:48pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/24 "2019-12-19T16:48:24Z")

</div>

> [@elasticforme](#):
>
> strip =\> ["field\_name"]

I tried this . But spaces is still there....result below

```
{
       "message" => [
        [0] "myapp.myproject.notice.student.request-time = 2019-12-13 12:37:01.4 ",
        [1] " myapp.myproject.notice.student.response-time = 2019-12-13 12:37:19.276"
    ],
         "partA" => "myapp.myproject.notice.student.request-time ",
         "partD" => " 2019-12-13 12:37:19.276",
    "@timestamp" => 2019-12-19T16:45:18.527Z,
      "@version" => "1",
         "part1" => [
        [0] "myapp.myproject.notice.student.request-time ",
        [1] " 2019-12-13 12:37:01.4 "
    ],
         "partC" => " myapp.myproject.notice.student.response-time ",
         "part2" => [
        [0] " myapp.myproject.notice.student.response-time ",
        [1] " 2019-12-13 12:37:19.276"
    ],
         "partB" => " 2019-12-13 12:37:01.4 ",
         "host" => "ip-xx-0-0-xx"
}

```

I added strip in below conf file

## conf file

```
 mutate {
       split => ["message","#"]
       add_field => {"part1" =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}
       strip => ["part1"]
       strip => ["part2"]
       }

           mutate {
           split => ["part1","="]
           add_field => {"partA" =>"%{[part1][0]}"}
           add_field => {"partB" =>"%{[part1][1]}"}
           strip => ["partA"]
           strip => ["partB"]
           }

           mutate {
           split => ["part2","="]
           add_field => {"partC" =>"%{[part2][0]}"}
           add_field => {"partD" =>"%{[part2][1]}"}
           strip => ["partC"]
           strip => ["partD"]
           }

```

Is there anything missing ? why spaces are still there ?

---

<div class="post-metadata">

**Author:** ![Transrian](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/transrian/32/58436_2.png) [@Transrian](https://discuss.elastic.co/u/Transrian)\
**Post date:** [December 19, 2019, 5:40pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/25 "2019-12-19T17:40:22Z")

</div>

This seems way too complicated for me. Why not simply :

```ruby
filter {

  kv {
    source => "message"
    value_split => "="
    field_split_pattern => "\s*#\s*"
  }
  
  date {
    match => ["myapp.myproject.notice.student.request-time", "ISO8601"]
    target => "myapp.myproject.notice.student.request-time"
  }
  
  date {
    match => ["myapp.myproject.notice.student.response-time", "ISO8601"]
    target => "myapp.myproject.notice.student.response-time"
  }
  
}

```

Output :

```json
{
  "@timestamp": "2019-12-19T17:37:42.646Z",
  "@version": "1",
  "host": "localhost",
  "message": "myapp.myproject.notice.student.request-time = 2019-12-13 12:37:01.4 # myapp.myproject.notice.student.response-time = 2019-12-13 12:37:19.276",
  "myapp.myproject.notice.student.request-time": "2019-12-13T12:37:01.400Z",
  "myapp.myproject.notice.student.response-time": "2019-12-13T12:37:19.276Z"
}

```

---

<div class="post-metadata">

**Author:** ![Charles\_Yuliansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_yuliansen/32/45664_2.png) [@Charles\_Yuliansen](https://discuss.elastic.co/u/Charles_Yuliansen)\
**Post date:** [December 20, 2019, 1:07am UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/26 "2019-12-20T01:07:33Z")

</div>

im sorry but i thought i have the pretty same problem which showing the same error. so how do we restart the daemon?  
and is the daemon still running even if i run another config with -f flag? i mean i ever run logstash.conf and it successfully sending log to my elastic, then i want to send different log so i edit the whole logstash.conf file, then run it with -f flag.  
thank you for the responds.

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [December 21, 2019, 4:17pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/27 "2019-12-21T16:17:26Z")

</div>

@Transrian  
This is not working ...... I get this error logstash log

][main] Exception while parsing KV {:exception=\>"Invalid FieldReference: ` **2019-12-21 15:33:50.055**

Is it the issue with ISODate which is used in date match ?

How to fix this ?

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [December 21, 2019, 4:19pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/28 "2019-12-21T16:19:41Z")

</div>

@elasticforme

Is there any way to remove the spaces ? strip did not work . result posted.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [December 23, 2019, 3:01pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/29 "2019-12-23T15:01:34Z")

</div>

filter for removing space is strip

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html)

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [December 23, 2019, 6:46pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/30 "2019-12-23T18:46:31Z")

</div>

@elasticforme  
yes. I checked it. I also did the same thing but it did not work,

Is there anything missing? Could you please check at your end ?

**my conf file**

```
mutate {
       split => ["message","#"]
       add_field => {"part1" =>"%{[message][0]}"}
       add_field => {"part2" =>"%{[message][1]}"}
       strip => ["part1"]
       strip => ["part2"]
       }

           mutate {
           split => ["part1","="]
           add_field => {"partA" =>"%{[part1][0]}"}
           add_field => {"partB" =>"%{[part1][1]}"}
           strip => ["partA"]
           strip => ["partB"]
           }

           mutate {
           split => ["part2","="]
           add_field => {"partC" =>"%{[part2][0]}"}
           add_field => {"partD" =>"%{[part2][1]}"}
           strip => ["partC"]
           strip => ["partD"]
           }

```

**Result** : It did not remove spaces.  
{  
"message" =\> [  
[0] "myapp.myproject.notice.student.request-time = 2019-12-13 12:37:01.4 ",  
[1] " myapp.myproject.notice.student.response-time = 2019-12-13 12:37:19.276"  
],  
"partA" =\> "myapp.myproject.notice.student.request-time ",  
"partD" =\> " 2019-12-13 12:37:19.276",  
"@timestamp" =\> 2019-12-19T16:45:18.527Z,  
"@version" =\> "1",  
"part1" =\> [  
[0] "myapp.myproject.notice.student.request-time ",  
[1] " 2019-12-13 12:37:01.4 "  
],  
"partC" =\> " myapp.myproject.notice.student.response-time ",  
"part2" =\> [  
[0] " myapp.myproject.notice.student.response-time ",  
[1] " 2019-12-13 12:37:19.276"  
],  
"partB" =\> " 2019-12-13 12:37:01.4 ",  
"host" =\> "ip-xx-0-0-xx"

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [December 23, 2019, 6:54pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/31 "2019-12-23T18:54:03Z")

</div>

> [@volcano](#):
>
> @Transrian

Is there any solution to error ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 24, 2019, 1:19pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/32 "2019-12-24T13:19:07Z")

</div>

mutate processes operations in a [fixed order](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-proc_order). strip comes before add\_field, so when it tries to do the strip the fields do not exist, so strip is a no-op. split them each into two mutate filters.

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [December 24, 2019, 3:46pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/33 "2019-12-24T15:46:07Z")

</div>

> [@Badger](#):
>
> **split** them each into two mutate filters.

got confused now. ,,, I already have a " **split**" there. ......my issue is with strip...not split....right ?

please clarify

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 24, 2019, 4:11pm UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/34 "2019-12-24T16:11:21Z")

</div>

> [@volcano](#):
>
> mutate { split =\> ["part1","="] add\_field =\> {"partA" =\>"%{[part1][0]}"} add\_field =\> {"partB" =\>"%{[part1][1]}"} strip =\> ["partA"] strip =\> ["partB"] }

You have to use more than one mutate to make things happen in the right order

```
mutate {
    split => ["part1","="]
    add_field => {"partA" =>"%{[part1][0]}" "partB" =>"%{[part1][1]}" }
}
mutate { strip => ["partA", "partB"] }

```

---

<div class="post-metadata">

**Author:** ![volcano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/volcano/32/46287_2.png) [@volcano](https://discuss.elastic.co/u/volcano)\
**Post date:** [December 27, 2019, 11:59am UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/35 "2019-12-27T11:59:25Z")

</div>

@Badger @elasticforme  
I have tested this now. This works okay in logstash but Kibana is not allowing **array** types i.e part1 , part2 as used here.

How do you fix this ?

end goal is to show the data in Kibana visualization. but Kibana is not supporting array types.

Kibana is showing this

 ![](https://us1.discourse-cdn.com/elastic/original/3X/b/c/bc850058f2295ca61097985b4b89df518a2cae4d.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 24, 2020, 11:59am UTC](https://discuss.elastic.co/t/parsing-log-with-logstash/212280/36 "2020-01-24T11:59:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/parsing-log-with-logstash/212280.md?page=1)