Csv to float from logstash to elasticsearch

Elastic Stack Logstash
1

I have some problems to read CSV and convert the values as float in Array

My Demo Dataset:

215900;216100;216300;216430;
216750;216950;217230;217500;
217780;217930;218100;218250;

My logstash Config:

input {
  file {
    path => "csvfile.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
    mutate {
        convert => { "message" => "float" }
   }
   mutate {
        split => { "message" => ";" }
    }
}
output {
  elasticsearch {
    hosts => ["http://10.14.100.100:9200","http://10.14.100.101:9200","http://10.14.100.103:9200"]
    index => "myindex"
  }
stdout {}
}

My Output in ES is:

{
   ...
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "_doc",
        "_id" : "wOcTfsdfssJ8PBIntss",
        "_score" : 1.0,
        "_source" : {
          "message" : [
            "222780",
            "222930",
            "223050",
            "223180",
            "223330",
...

I would like to have the array as float without quotes, like:

{
   ...
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "_doc",
        "_id" : "wOcTfsdfssJ8PBIntss",
        "_score" : 1.0,
        "_source" : {
          "message" : [
            222780.0,
            222930.0,
            223050.0,
            223180.0,
            223330.0,
...

Where ist the issue in my configuration?

Thanks :slight_smile:

2

That does not convert every number in the field, it just call .to_f on the string. You could replace the two mutate filters with

    ruby {
        code => '
            m = event.get("message").split(";")
            m.each_index { |x| m[x] = m[x].to_f }
            event.set("[message]", m)
        '
    }
1 Like
3

Thanks @Badger

how can i take each row in new field. In my solution i create new Id in elasticsearch with one row as message.

my csv
215900;216100;216300;216430;
216750;216950;217230;217500;
217780;217930;218100;218250;

i would like this output in elasticsearch:

{
   ...
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "_doc",
        "_id" : "wOcTfsdfssJ8PBIntss",
        "_score" : 1.0,
        "_source" : {
              "row1" : [215900;216100;216300;216430;],
              "row2" : [216750;216950;217230;217500;],
              "row3" : [217780;217930;218100;218250;]
4

that seems huge record. you trying to put whole csv as one document in ELK? you need to explain little more. as this does not seems correct design.

When you read csv. each message is single line. and that is one document (one record) in elk.

5

I have over 5000 values in my csv file. i would like to visialize the values as Heatmap in Kibana.

What is the best design for this task?

One way is: store each value with timestamp as dokument to ES and visualize it. Can i do that with logstash?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.