Removing fields from logstash

Elastic Stack Logstash
43

Can you please send me the syntax....am very new to elastic :sweat_smile:

44

If my assumption is correct that your logline ends up in message field:

You could also try something like:

match => { "message" => '^%{GREEDYDATA}"Status":"%{WORD:Status}"%{GREEDYDATA}$' }

You could try to parse the json messages "manually" via grok, but as indicated I would advise to use the json filter on the json message.. That implies you get the fields for free.

45

instead of grok right?

46

As indicated, you need to provide the required inputs. If you can't share them because they are confidential. This is the best I can do :slight_smile:

47

beats_input_codec_plain_applied, _grokparsefailure

in the tag field the above output is showing

48

its working
thank you

49

one more question
i want to fetch multiple fields from message like status

50

how can i achieve this

51

[Debug] : 2023-09-05T08:59:40 -> [Response - {"abc":{"Service":{"Channel":{"HostIP":{"Status":"APPROVED" ,"Type":"AGENT"},
"Name":"abc.COM","Type":"AGENT"}}}}]

I want both status and Type in a separate fields
like:
Status APPROVED
Type AGENT

and so on

52

I am trying multiple ways but when it starts implementing, the logs stop loading on Kibana

53

Can you share your current approach? (filters in logstash)

54 
input {
  beats {
    port => 5044
  }
}
filter {
    if "/var/log/abc.log"  in [log][file][path] {
      grok {
         match => { "message" => '^%{GREEDYDATA}"Status":"%{WORD:Status}"%{GREEDYDATA}$' }
      }

     grok {
        match => { "message" => '^%{GREEDYDATA}"type":"%{NUMBER:Type}"%{GREEDYDATA}$' }

     }
   }
}

currently am trying this ...but its not working

55

You are matching:

2 issues in your config:

  • "type" is not the same as "Type"
  • "AGENT" is not a number

Beside that, your sample also contains multiple "Type" fields. So might be tricky to get the right one with the grok approach..

56

multiple "Type" fields as in?

57

hey... it's working thanks
Tell me one thing....if the type is both number and word,so what will be the type
like for text we use WORD, for integers we use NUMBER
So what will be the type of varchar

58

Your sample contains:
{"Status":"APPROVED" ,"Type":"AGENT"}
"Name":"abc.COM","Type":"AGENT"}

59

Take a look at:

Which points via the predefined grok patterns link to:

The second link contains all predefined grok patterns.
If you need to match alphanumeric characters, you need to chose the appropriate pattern OR you define a custom pattern yourself.

60

The link is really helping me a lot.....
one question -
My logs' timestamp format is DD-MM-YYYY HH:MM:SS (08-09-2023 10:38:43AM)

so what will be the type
I looked in link but its not in the above format

61

{"Name":"txn_no","Value":"FT23217XT7DM"}

and for this JSON I need a field as
txn_no FT23217XT7DM
like this
How can I do this

62

{"Name":"txn_no","Value":"FT23217XT7DM"}

and like this, there is multiple name and values
like
{"Name":"txn_no","Value":"FT23217XT7DM"},{"Name":"txn_channel","Value":"AGENT"},{"Name":"debit_account_number"},{"Name":"credit_account_number"}

so I want to fetch each value in their respective names