# Removing fields from logstash

**URL:** <https://discuss.elastic.co/t/removing-fields-from-logstash/341782>\
**Category:** Logstash\
**Created:** [August 28, 2023, 7:16am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782 "2023-08-28T07:16:58Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 5, 2023, 7:26am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/43 "2023-09-05T07:26:02Z")

</div>

Can you please send me the syntax....am very new to elastic 😅

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [September 5, 2023, 7:29am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/44 "2023-09-05T07:29:25Z")

</div>

If my assumption is correct that your logline ends up in message field:

> [@bharti](#):
>
> `match => { "message" => "status: %{WORD:Status}" }`

You could also try something like:

```auto
match => { "message" => '^%{GREEDYDATA}"Status":"%{WORD:Status}"%{GREEDYDATA}$' }

```

You could try to parse the json messages "manually" via grok, but [as indicated I would advise to use the json filter on the json message..](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/31) That implies you get the fields for free.

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 5, 2023, 7:30am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/45 "2023-09-05T07:30:45Z")

</div>

instead of grok right?

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [September 5, 2023, 7:30am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/46 "2023-09-05T07:30:59Z")

</div>

As indicated, you need to provide the required inputs. If you can't share them because they are confidential. This is the best I can do 🙂

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 5, 2023, 7:34am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/47 "2023-09-05T07:34:46Z")

</div>

beats\_input\_codec\_plain\_applied, \_grokparsefailure

in the tag field the above output is showing

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 5:20am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/48 "2023-09-07T05:20:33Z")

</div>

its working  
thank you

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 5:21am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/49 "2023-09-07T05:21:27Z")

</div>

one more question  
i want to fetch multiple fields from message like status

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 5:21am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/50 "2023-09-07T05:21:44Z")

</div>

how can i achieve this

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 5:24am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/51 "2023-09-07T05:24:40Z")

</div>

[Debug] : 2023-09-05T08:59:40 -\> [Response - {"abc":{"Service":{"Channel":{"HostIP":{"Status":"APPROVED" ,"Type":"AGENT"},  
"Name":"[abc.COM](http://abc.com/)","Type":"AGENT"}}}}]

I want both status and Type in a separate fields  
like:  
Status APPROVED  
Type AGENT

and so on

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 5:26am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/52 "2023-09-07T05:26:24Z")

</div>

I am trying multiple ways but when it starts implementing, the logs stop loading on Kibana

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [September 7, 2023, 6:56am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/53 "2023-09-07T06:56:25Z")

</div>

Can you share your current approach? (filters in logstash)

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 7:08am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/54 "2023-09-07T07:08:15Z")

</div>

```auto
input {
  beats {
    port => 5044
  }
}
filter {
    if "/var/log/abc.log" in [log][file][path] {
      grok {
         match => { "message" => '^%{GREEDYDATA}"Status":"%{WORD:Status}"%{GREEDYDATA}$' }
      }

     grok {
        match => { "message" => '^%{GREEDYDATA}"type":"%{NUMBER:Type}"%{GREEDYDATA}$' }

     }
   }
}

```

currently am trying this ...but its not working

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [September 7, 2023, 7:32am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/55 "2023-09-07T07:32:09Z")

</div>

You are matching:

> [@bharti](#):
>
> "Type":"AGENT"

2 issues in your config:

- "type" is not the same as "Type"
- "AGENT" is not a number

Beside that, your sample also contains multiple "Type" fields. So might be tricky to get the right one with the grok approach..

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 7:39am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/56 "2023-09-07T07:39:56Z")

</div>

multiple "Type" fields as in?

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 7, 2023, 7:50am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/57 "2023-09-07T07:50:08Z")

</div>

hey... it's working thanks  
Tell me one thing....if the type is both number and word,so what will be the type  
like for text we use WORD, for integers we use NUMBER  
So what will be the type of varchar

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [September 7, 2023, 7:57am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/58 "2023-09-07T07:57:55Z")

</div>

> [@bharti](#):
>
> [Debug] : 2023-09-05T08:59:40 -\> [Response - {"abc":{"Service":{"Channel":{"HostIP":{"Status":"APPROVED" ,"Type":"AGENT"},  
> "Name":"[abc.COM](http://abc.com/)","Type":"AGENT"}}}}]

Your sample contains:  
{"Status":"APPROVED" , **"Type":"AGENT"** }  
"Name":"[abc.COM](http://abc.com/)", **"Type":"AGENT"** }

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [September 7, 2023, 8:00am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/59 "2023-09-07T08:00:38Z")

</div>

Take a look at:

> **[Grokking grok | Elasticsearch Guide \[8.9\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/grok.html#grok-syntax)**

Which points via the predefined grok patterns link to:

> <https://github.com/elastic/elasticsearch/blob/main/libs/grok/src/main/resources/patterns/legacy/grok-patterns>

The second link contains all predefined grok patterns.  
If you need to match alphanumeric characters, you need to chose the appropriate pattern OR you define a custom pattern yourself.

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 8, 2023, 5:08am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/60 "2023-09-08T05:08:29Z")

</div>

The link is really helping me a lot.....  
one question -  
My logs' timestamp format is DD-MM-YYYY HH:MM:SS (08-09-2023 10:38:43AM)

so what will be the type  
I looked in link but its not in the above format

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 8, 2023, 5:16am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/61 "2023-09-08T05:16:34Z")

</div>

{"Name":"txn\_no","Value":"FT23217XT7DM"}

and for this JSON I need a field as  
txn\_no FT23217XT7DM  
like this  
How can I do this

---

<div class="post-metadata">

**Author:** ![bharti](https://avatars.discourse-cdn.com/v4/letter/b/73ab20/32.png) [@bharti](https://discuss.elastic.co/u/bharti)\
**Post date:** [September 8, 2023, 5:37am UTC](https://discuss.elastic.co/t/removing-fields-from-logstash/341782/62 "2023-09-08T05:37:52Z")

</div>

{"Name":"txn\_no","Value":"FT23217XT7DM"}

and like this, there is multiple name and values  
like  
{"Name":"txn\_no","Value":"FT23217XT7DM"},{"Name":"txn\_channel","Value":"AGENT"},{"Name":"debit\_account\_number"},{"Name":"credit\_account\_number"}

so I want to fetch each value in their respective names

[Previous page](https://discuss.elastic.co/t/removing-fields-from-logstash/341782.md?page=2)

[Next page](https://discuss.elastic.co/t/removing-fields-from-logstash/341782.md?page=4)
