# Logstash Filter help - Newbie question

**URL:** <https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580>\
**Category:** Logstash\
**Created:** [January 17, 2022, 12:14pm UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580 "2022-01-17T12:14:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![PSFletchTheTek](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@PSFletchTheTek](https://discuss.elastic.co/u/PSFletchTheTek)\
**Post date:** [January 17, 2022, 12:14pm UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580/1 "2022-01-17T12:14:33Z")

</div>

Hi All,  
Sorry I'm just a bit lost on where to start with this one, I was hoping someone would be kind enough to point me in the right direction please?  
I have a log that looks like this  
{"host":"192.168.221.11","message":"Clock advanced by 527 ticks","@version":"1","port":57429,"@timestamp":"2022-01-14T12:12:41.053Z"}  
It starts and end's with {} its comma delimited (but other examples have shown that is can have comma's inside of the "" Sections! so a simple CSV filter doesn't seem perfect)  
But helpfully it has the data or column header already in each message. IE host, message, @Version, port and @timestamp. So in theory all I need to do is read the first part and that's the name the second is the variable, continue until i hit the end character which is a }

Could I ask for help in the best way to filter these variables out please ready to be sent to Elasticsearch? I'm just not sure on which filter or patterns would make a good start with this one.

Thanks in advance,

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 17, 2022, 12:26pm UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580/2 "2022-01-17T12:26:36Z")

</div>

Hi,

Your log is valid json format, why dont you use the json logstash filter ?

> **[JSON filter plugin | Logstash Reference \[7.16\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html)**

---

<div class="post-metadata">

**Author:** ![PSFletchTheTek](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@PSFletchTheTek](https://discuss.elastic.co/u/PSFletchTheTek)\
**Post date:** [January 18, 2022, 1:20pm UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580/3 "2022-01-18T13:20:10Z")

</div>

Cool, thanks i'll give that a go.  
Would you put that as a codec in the input or run it in the filter part?  
Thanks

---

<div class="post-metadata">

**Author:** ![PSFletchTheTek](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@PSFletchTheTek](https://discuss.elastic.co/u/PSFletchTheTek)\
**Post date:** [January 18, 2022, 3:38pm UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580/4 "2022-01-18T15:38:29Z")

</div>

Hi,  
I got, {"tags":["\_jsonparsefailure"]," when putting in the codec into the input.  
I'll try the same using the filter now and report back.

---

<div class="post-metadata">

**Author:** ![PSFletchTheTek](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@PSFletchTheTek](https://discuss.elastic.co/u/PSFletchTheTek)\
**Post date:** [January 18, 2022, 5:47pm UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580/5 "2022-01-18T17:47:27Z")

</div>

Hi, So i moved the json command to the filter.  
I have a little more debug information but its still not liking it, any ideas?

```auto
input {
 # udp {
 # port => 5141
    #type => 'hsl'
 # }
  tcp {
    port => 5141
    id => "TCP_INPUT"
    type => "TCP_INPUT"
  }
}

filter {
  if [type] == "TCP_INPUT" {
    json {
      source => "message"
    }
  }
}
type or paste code here

```

Logstash-plain.log shows this

```auto
t
[2022-01-18T16:50:05,892][WARN][logstash.filters.json][main][e929e8c178d8ddd954aa9f2eeb0726d914d7f7df9e7c9f8d2b7020fe33d277a0] Error parsing json {:source=>"message", :raw=>"other mcp_obj_delete (tag=15299) messages", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'other': was expecting ('true', 'false' or 'null')
ype or paste code here

```

And the output to file shows something like this

```auto
{"tags":["_jsonparsefailure"],"port":25326,"@timestamp":"2022-01-18T16:50:05.779Z","message":"[NAT] end_transaction ignored","@version":"1","type":"TCP_INPUT","host":"192.168.221.12"}
{"tags":["_jsonparsefailure"],"port":25326,"@timestamp":"2022-01-18T16:50:05.779Z","message":"end_transaction -> running 1 ig_commit 1 od_compile 0 od_deploy 0","@version":"1","type":"TCP_INPUT","host":"192.168.221.12"}
{"tags":["_jsonparsefailure"],"port":25326,"@timestamp":"2022-01-18T16:50:05.779Z","message":"end_transaction calls device_actions()","@version":"1","type":"TCP_INPUT","host":"192.168.221.12"}
{"tags":["_jsonparsefailure"],"port":25326,"@timestamp":"2022-01-18T16:50:05.779Z","message":"device_actions: on demand feature is not enabled.","@version":"1","type":"TCP_INPUT","host":"192.168.221.12"}
{"tags":["_jsonparsefailure"],"port":25326,"@timestamp":"2022-01-18T16:50:05.779Z","message":"[NAT] end_transaction -> ig_commit 1","@version":"1","type":"TCP_INPUT","host":"192.168.221.12"}type or paste code here

```

and the raw file before the Json config was added looked like this

```auto

{"message":"[NAT] end_transaction -> ig_commit 1","@version":"1","port":25243,"@timestamp":"2022-01-18T10:37:39.719Z","host":"192.168.221.12"}
{"message":"[NAT] end_transaction ignored","@version":"1","port":25243,"@timestamp":"2022-01-18T10:37:39.719Z","host":"192.168.221.12"}
type or paste code here

```

---

<div class="post-metadata">

**Author:** ![PSFletchTheTek](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@PSFletchTheTek](https://discuss.elastic.co/u/PSFletchTheTek)\
**Post date:** [January 20, 2022, 11:05am UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580/6 "2022-01-20T11:05:18Z")

</div>

Hi,  
I've just worked out, that its valid json as Logstash has done that!  
The message field is actually what came in, and logstash has already wrapped other parameters like host and @timestamp around it ready for Elasticsearch to ingest.  
So actually i don't need to do anything else with it at all!!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 17, 2022, 11:05am UTC](https://discuss.elastic.co/t/logstash-filter-help-newbie-question/294580/7 "2022-02-17T11:05:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.