Extract the value from CSV field and add new field

Elastic Stack Logstash
1

Dear Team

Wondering if you can help please . We are parsing the CSV using the below code , One of the field (exec) in CSV file can have either one of below values

(T= some integer value |D= some integer value |N=some integer value);

I want to be able to first check which value was add on the field . For example if its (T=5462) then I want logstash to create a new field called Throttling and send the value to elastic search . So elasticserch should see the Throttling as 5462 . If the value is N then should add new field "Network" orif its D then "Disk" and their respective values .

T= Throttling
N = Network
D= Disk

input {
beats {
port => "5044"
}

}
filter {
csv {

  columns => ["time","exec","latency","orderid","src_ip","src_port","userid","obid","conn","msgtype"]
  separator => ","
  skip_empty_columns => "true"

}
if[msgtype]=="D" {
mutate {
replace => ["msgtype","Single"]
}
}
if[msgtype]=="G" {
mutate {
replace=>["msgtype","Order Can"]
}
}
if[msgtype]=="F" {
mutate {
replace=>["msgtype","Order Can req"]
}
}

}

output {

elasticsearch {
hosts => [ "127.0.0.1:9200" ]
index => "logs"
}
stdout {}
}

Regards

Mussa

2

wondering if i can get any help on above please

3

you can use a grok pattern to extract both sides of the "" in that field with

filter {
  grok { match => ["msgtype", "%{WORD:msg_type_letter}=%{NUMBER:msg_type_value}" }
}

then you can use "if/else if/else if" instead of just "if/if/if":

if [msg_type_letter] == "T" {
   mutate { rename => { "msg_type_value" => "Throttling" } }
} else if [msg_type_letter] == "D" {
   mutate { rename => { "msg_type_value" => "Disk" } }
} else if [msg_type_letter] == "N" {
   mutate { rename => { "msg_type_value" => "Network" } }
}
4

Hi jsvd

Thanks for above info manage to get the value with some changes to above code

grok {

match => ["excuses", "%{WORD:excuses_type_letter}=%{NUMBER:excuses_type_value}"]

}

if [excuses_type_letter] == "T" {

mutate { add_field => { "Throttling" => "%{excuses_type_value}"} }

}

Now seeing one more issue , Logstash is sending all 3 fields value to elasticsearch instead I only want Throttling value to be send to Elasticsearch and ignore all other values .

What code would I need to stop Logstash from sending excuses , excuses_type_letter and excuses_type_value?

image.png1082×567 22.4 KB

5

u can use

if [excuses_type_letter] == "T" {

mutate {
add_field => { "Throttling" => "%{excuses_type_value}"}
remove_field => [ "%{excuses_type_letter}", "%{excuses_type_value}" ]
}
}

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field

1 Like
6

Hi Tatdat

Thanks for the reply , i have added the code , however Logstash still sending excuses_type_letter and {excuses_type_value values to elastisearch .

Regards

Mussa

7

Hi

It worked after changing to following code . Thanks for the assistance .

mutate {

remove_field => [ "[excuses_type_letter]" ]
remove_field => [ "[excuses_type_value]" ]
}

2 Likes
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.