# Removing fields from logstash **URL:** \ **Category:** Logstash\ **Created:** [August 28, 2023, 7:16am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782 "2023-08-28T07:16:58Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 4, 2023, 6:39am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/23 "2023-09-04T06:39:44Z")
```auto input { beats { port => 5044 } } filter { if "/var/log/abc.log" in [log][file][path]{ mutate { add_field => { "employee_id" => "abc123" } } } } ``` ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 4, 2023, 6:40am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/24 "2023-09-04T06:40:18Z")
this is my input configuration for logstash ---
**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\ **Post date:** [September 4, 2023, 10:10am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/25 "2023-09-04T10:10:28Z")
I don't see any real question in your last comments, so not sure what you are looking for.. For logstash, you could do something like: ```auto filter { if [@metadata][env] { if [@metadata][aix] { if [@metadata][env] == "int"{ mutate{ add_field => { "environment" => "dev" } } } if [@metadata][env] == "tst"{ mutate{ add_field => { "environment" => "int" } update => { "[@metadata][domain]" => "vaja_tst2" } } } if [@metadata][env] != "int" and [@metadata][env] != "tst" { mutate{ add_field => { "environment" => "[@metadata][env]" } } } } } #If variable exists if [@metadata][rhel] { mutate{ add_field => { "[@metadata][domain]" => "%{environment}justxdom" } } } } ``` ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 4:53am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/26 "2023-09-05T04:53:20Z")
actually, i want to fetch the log details in fields [Debug] : 2023-09-04T11:22:24 -\> [Request - {"ABC":{"Service":{"Channel":{"HostIP":{"Type":"IP","Value":"0.0.0.0"},"Name":"[ABC.COM](http://ABC.COM)","Type":"AGENT"},"Device":{"IP":{},"OS":"web","Version":"116"},"Name":"ABCD","Type":"XYZ","Code":"IN" I tried the grok function too but unable to implement ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 5, 2023, 6:00am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/27 "2023-09-05T06:00:43Z")
Are you sure this is the full log line? There is begging `[Request` but no square bracket] at the end. Also JSON structure is not complete. ---
**Author:** ![Samuele\_Lolli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samuele_lolli/32/107848_2.png) [@Samuele\_Lolli](https://discuss.elastic.co/u/Samuele_Lolli)\ **Post date:** [September 5, 2023, 6:01am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/28 "2023-09-05T06:01:34Z")
Hi @bharti, have you thought about using a different strategy? Maybe you can use the ingest pipeline instead. Here the [wiki](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) Let me know if this can be a helpfull. 🙂 ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 6:35am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/29 "2023-09-05T06:35:59Z")
yes yes I just put a small part of the JSON just to show ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 6:38am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/30 "2023-09-05T06:38:37Z")
Hi Samuele ...unable to understand 😅 ---
**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\ **Post date:** [September 5, 2023, 6:52am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/31 "2023-09-05T06:52:44Z")
You are on the right way: - Split the logline with [grok](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html) You can test your grok via [the tester in Kibana](https://www.elastic.co/guide/en/kibana/current/xpack-grokdebugger.html) - Parse the [json message](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) you get via grok splitting You can operate on the fields you get afterwards.. PS: Logstash is a dedicated product for logs ingestion and way easier to operate than the elasticsearch ingestion pipelines. So I discourage the suggestion from [Samuele\_Lolli](https://discuss.elastic.co/u/Samuele_Lolli). ---
**Author:** ![Samuele\_Lolli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samuele_lolli/32/107848_2.png) [@Samuele\_Lolli](https://discuss.elastic.co/u/Samuele_Lolli)\ **Post date:** [September 5, 2023, 6:55am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/32 "2023-09-05T06:55:49Z")
Sorry for not being clear. Instead of removing field directly in logstash you can use the ingest pipeline. The use is something like that: In the output part of logstash you put the name of the ingest pipeline and then in kibana you create the ingest pipeline. You can use different processors to do a lot of different operation Let me know if now is clear ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 6:59am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/33 "2023-09-05T06:59:30Z")
```auto input { beats { port => 5044 } } filter { if "/var/log/abc.log" in [log][file][path] { grok { match => { "message" => "status: %{WORD:Status}" } } } } ``` ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 6:59am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/34 "2023-09-05T06:59:54Z")
my logstash input configuration ---
**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\ **Post date:** [September 5, 2023, 7:06am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/35 "2023-09-05T07:06:13Z")
Can you add this output plugin and post a sample output-message? So we now how your full messages looks like? It's quite hard to help you with parsing, if we don't know how everything looks like.. ```auto output { file { path => "//logstash_debug/incomingMessages.log" codec => rubydebug {metadata => true} } } ``` ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 7:07am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/36 "2023-09-05T07:07:24Z")
sorry chouben ...but the message part is confidential ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 7:12am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/37 "2023-09-05T07:12:34Z")
just think of a simple json message ...my data just look like that only [Debug] : 2023-09-05T08:59:40 -\> [Response - {"abc":{"Service":{"Channel":{"HostIP":{"Status":"APPROVED"}, "Name":"[abc.COM](http://abc.COM)","Type":"AGENT"}}}}] ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 7:12am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/38 "2023-09-05T07:12:53Z")
sent one customised message ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 7:13am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/39 "2023-09-05T07:13:58Z")
here I want that status part in a separate field ....so how can I achieve that ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 7:15am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/40 "2023-09-05T07:15:54Z")
```auto input { beats { port => 5044 } } filter { if "/var/log/abc.log" in [log][file][path] { grok { match => { "message" => "status: %{WORD:Status}" } } } } ``` am trying to fetch the status part like status APPROVED in the event ---
**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\ **Post date:** [September 5, 2023, 7:19am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/41 "2023-09-05T07:19:28Z")
My assumption is that this part ends up in the "message" field: > [@bharti](#): > > [Request - {"ABC":{"Service":{"Channel":{"HostIP":{"Type":"IP","Value":"0.0.0.0"},"Name":"[ABC.COM](http://ABC.COM)","Type":"AGENT"},"Device":{"IP":{},"OS":"web","Version":"116"},"Name":"ABCD","Type":"XYZ","Code":"IN" That implies your parsing won't work. You see the grok pattern won't match the log line above.. > [@bharti](#): > > `match => { "message" => "status: %{WORD:Status}" }` [Take a look at my Filebeat configuration above. I do parse the logline already in multiple fields.](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/20) You should do something similar. ---
**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\ **Post date:** [September 5, 2023, 7:23am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/42 "2023-09-05T07:23:20Z")
so you are saying that i have to parse the whole json in the match part? [Previous page](https://discuss.elastic.co/t/removing-fields-from-logstash/341782.md?page=1) [Next page](https://discuss.elastic.co/t/removing-fields-from-logstash/341782.md?page=3)