How to use split filter on the field using logstash

Elastic Stack Logstash
1

Hi everyone ,
i am new at logstash here i am trying to fatch data from .csv file and i have 6 fields total and in that 6 fields one field have actually data i need to split that field as multiple fields ,
here is my config file
input{
file{
path => "/home/.csv"
start_position => "beginning"
sincedb_path => "/div/null"
}
}
filter{
csv{
separator => ","
columns =>["log_id","log_datetime","log_type","log_messagge","log_componet","log_user"]
}
date{
match => ["log_datetime","yyyy-MM-dd HH:MM:ss]
}
mutate { convert => ["log_id","integer"]}
}
split{ field => "log_message"}

output{
elasticsearch{
hosts => "http://localhost:9200"
index => "logs_data"
}
stdout{}
}

in log_message field my data is like "data": [{ "field1":"string","field2":"string"...},{ "field1":"string","field2":"string"...}
and another row is like (field->"value") (field->"value")

here my problem is how can i split this log_message field into multiple fields ,
help me please ,thanks you

2

A split filter is used to split a message into multiple message, each containing one element from an array. If you want to split a string into an array then use the split option on a mutate filter. However, I don't think either is useful here.

It looks like in some cases log_message is almost a JSON array, and in some cases it is key-value pairs. If so, the kv case could be parsed using

kv { field_split_pattern => "[\( \)]+" value_split_pattern => "->" }

In the other case you can use mutate to convert it into valid JSON, and then use a json filter.

mutate { gsub => [ "message", "^", "{", "message", "$", "}" ] }
1 Like
3

Hi @Badger,
Thank you so much for your reply,

I am getting the data when i use
kv { field_split_pattern => "[( )]+" value_split_pattern => "->" }
but in my log_message field some rows are like this
"(Institution Id->77) (data Id->127) QUERY->SQL query ERROR->Table 'tablename' doesn't exist"
and some are
"QUERY-> update set tablename column=value some conditions
ERROR: Table 'Table name' doesn't exist"

if i apply your "kv" filter plugins i am getting the results like this ,
{
"ERROR" => "Table", //here i am getting only first word
"QUERY" => "insert", //here also
"Id"=>[
"77", //here i need field name i need
"127" //here also
]
},
{
"ERROR" => "Table", //here i am getting only first word
"QUERY" => "update", //here also
}
but here my expected result is
{
"ERROR" => "Table 'table name or column name' doesn't exist",
"QUERY" => "insert query", //i need total query here not only insert command
"institution Id"=>77,
"data Id" => 127
}
please help me thank you

4

The problem is that space is used to separate fields but also occurs within fields. I don't see any way to unambiguously parse that.

5

Hi @Badger ,
Thank you very much for your reply,
i do not know how to solve this problem ,i am new to this logstash ,can you explain what should i do .

Thanks
Sandeep

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.