John Doe;Debrah ;19,95;13-07-2016
M.C. Hammer;Susan;19;14-07-2015
The first represents a customer name, which sometimes exist of one name, two names, or names with periods (.)
The second is the name of the sales, which sometimes does have a space after the name, see Debrah.
The third is a sales price which sometimes has 2 digits after the comma but sometimes no decimals at all.
The fourth is a date with the format dd-mm-yyyy
I tried to use CSV filter but this does not give me the 2 decimal numbers (95), but 00 in stead, so 19,00
$ sudo /usr/share/logstash/bin/logstash -e 'input{stdin{}} filter{csv {
columns => ["customer","sales","price","date"]
separator => ";"
}}
output{stdout{codec=>rubydebug}}'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
The stdin plugin is now waiting for input:
02:34:24.229 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
02:34:24.259 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
02:34:24.461 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
John Doe;Debrah ;19,95;13-07-2016
{
"date" => "13-07-2016",
"@timestamp" => 2016-12-13T02:34:43.191Z,
"price" => "19,95",
"@version" => "1",
"host" => "elastic5",
"message" => "John Doe;Debrah ;19,95;13-07-2016",
"sales" => "Debrah ",
"tags" => [],
"customer" => "John Doe"
}
^C02:34:46.963 [SIGINT handler] WARN logstash.runner - SIGINT received. Shutting down the agent.
02:34:46.981 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"}
Thanks for your answer/reply!
This works fine. But when using Kibana, it shows my prices as 19,00 instead of 19,95. I think it has probably to do with the fact that we in the Netherlands are using a comma as a decimal sign, this instead of a period.
So, I guess that I have to tell Kibana somehow that my locale=nl.
In my CSV file I replaced the comma's (,) with periods (.) and this solves the problem, but I do not want to alter my source file just to work around this problem.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.