caixukun
(caixukun)
July 3, 2023, 9:04am
1
my data is:
123456@gmail.com----john----password
123456@gmail.com----john----p@ssword
123456@gmail.com----john----123456
123456@gmail.com----john----123456
123456@gmail.com----john----123----456
123456@gmail.com----john----123456
my separator is :
separator => "----"
After I use logstash to import data, an error will appear,Because the same thing as separator appears in the data
So I would like to ask you how to solve this error
Thanks
Rios
(Rios)
July 3, 2023, 9:22am
2
Welcome to the community!
Can you show the pattern or all filter section?
1 Like
caixukun
(caixukun)
July 3, 2023, 9:28am
3
this is all
input {
file {
path => ["/test.txt"]
start_position => beginning
sincedb_path => "/test_.txt"
codec => plain{
charset=>"UTF-8"
}
}
}
filter {
mutate{
gsub => [ "message", "\\r", "" ]
}
csv {
separator => "----"
columns => ["email","name","password"]
remove_field => ["host", "tags", "path", "message","@timestamp","@version"]
}
}
output {
elasticsearch {
hosts => ["https://localhost:9200"]
index => "test"
user => "elastic"
password => "123456"
cacert => "/opt/kibana-8.8.0/data/ca_1685641330385.crt"
}
}
Rios
(Rios)
July 3, 2023, 9:32am
4
Will not work.
Try with dissect or grok.
dissect {
mapping => {
"message" => "%{email}----%{name}----%{password}"
}
}
Btway, I wouldn't remove fields: "tags", "@timestamp ","@version "
1 Like
caixukun
(caixukun)
July 3, 2023, 9:58am
5
Thank you sir, by the way, why don't you remove "tags", "@timestamp ", "@version " , if you don't remove it, will it take up more storage?
Rios
(Rios)
July 3, 2023, 11:44am
6
"tags" are useful for error handling and to know what is wrong as _dissectfailure
or _grokparsefailure
for grok plugin in tags
Error handling:
output {
if "_dissectfailure" in [tags] {
elasticsearch {
hosts => ["https://localhost:9200"]
index => "error_-%{+YYYY.MM.dd}"
cacert => "/opt/kibana-8.8.0/data/ca_1685641330385.crt"
}
}
else {
elasticsearch {
hosts => ["https://localhost:9200"]
index => "test"
cacert => "/opt/kibana-8.8.0/data/ca_1685641330385.crt"
}
}
}
"@timestamp " to know when something has been processed
"@version " it's not mandatory, but it's useful. There is a discussion here .
system
(system)
Closed
July 31, 2023, 11:45am
7
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.