A question about separator in logstash

Elastic Stack Logstash
1

my data is:

123456@gmail.com----john----password
123456@gmail.com----john----p@ssword
123456@gmail.com----john----123456
123456@gmail.com----john----123456
123456@gmail.com----john----123----456
123456@gmail.com----john----123456

my separator is :
separator => "----"
After I use logstash to import data, an error will appear,Because the same thing as separator appears in the data
So I would like to ask you how to solve this error
Thanks

2

Welcome to the community!

Can you show the pattern or all filter section?

1 Like
3

this is all

input {
  file {
		path => ["/test.txt"]
		start_position => beginning
		sincedb_path => "/test_.txt"
    codec => plain{
		charset=>"UTF-8"
		}
	}
}
filter {
	mutate{
		gsub => [ "message", "\\r", "" ]
	}
	csv {
		separator => "----"
		columns => ["email","name","password"]
		remove_field => ["host", "tags", "path", "message","@timestamp","@version"]
	}
}
output {
    elasticsearch {
    hosts => ["https://localhost:9200"]
    index => "test"
    user => "elastic"
    password => "123456"
    cacert => "/opt/kibana-8.8.0/data/ca_1685641330385.crt"
    }


 


}
4

Will not work.

Try with dissect or grok.

    dissect {
      mapping => {
        "message" => "%{email}----%{name}----%{password}"
      }
    }

Btway, I wouldn't remove fields: "tags", "@timestamp","@version"

1 Like
5

Thank you sir, by the way, why don't you remove "tags", "@timestamp", "@version" , if you don't remove it, will it take up more storage?

6
  • "tags" are useful for error handling and to know what is wrong as _dissectfailure or _grokparsefailure for grok plugin in tags
    Error handling:
output {
  if "_dissectfailure" in [tags] {
    elasticsearch {
      hosts => ["https://localhost:9200"]
      index => "error_-%{+YYYY.MM.dd}"
      cacert => "/opt/kibana-8.8.0/data/ca_1685641330385.crt"
    }

  }
  else {
    elasticsearch {
      hosts => ["https://localhost:9200"]
      index => "test"
      cacert => "/opt/kibana-8.8.0/data/ca_1685641330385.crt"
    }
  }
}

  • "@timestamp" to know when something has been processed

  • "@version" it's not mandatory, but it's useful. There is a discussion here.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.