# Csv to float from logstash to elasticsearch

**URL:** https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105
**Category:** Logstash
**Created:** [March 12, 2021, 4:02pm UTC](https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105 "2021-03-12T16:02:25Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![ofitz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofitz/32/47071_2.png) [@ofitz](https://discuss.elastic.co/u/ofitz)
#### Post date: [March 12, 2021, 4:02pm UTC](https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105/1 "2021-03-12T16:02:25Z")

</div>

I have some problems to read CSV and convert the values as float in Array

My Demo Dataset:

```
215900;216100;216300;216430;
216750;216950;217230;217500;
217780;217930;218100;218250;

```

My logstash Config:

```
input {
  file {
    path => "csvfile.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
    mutate {
        convert => { "message" => "float" }
   }
   mutate {
        split => { "message" => ";" }
    }
}
output {
  elasticsearch {
    hosts => ["http://10.14.100.100:9200","http://10.14.100.101:9200","http://10.14.100.103:9200"]
    index => "myindex"
  }
stdout {}
}

```

My Output in ES is:

```
{
   ...
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "_doc",
        "_id" : "wOcTfsdfssJ8PBIntss",
        "_score" : 1.0,
        "_source" : {
          "message" : [
            "222780",
            "222930",
            "223050",
            "223180",
            "223330",
...

```

I would like to have the array as float without quotes, like:

```
{
   ...
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "_doc",
        "_id" : "wOcTfsdfssJ8PBIntss",
        "_score" : 1.0,
        "_source" : {
          "message" : [
            222780.0,
            222930.0,
            223050.0,
            223180.0,
            223330.0,
...

```

Where ist the issue in my configuration?

Thanks 🙂

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 12, 2021, 6:56pm UTC](https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105/2 "2021-03-12T18:56:00Z")

</div>

> [@ofitz](#):
>
> ```auto
> mutate { convert => { "message" => "float" } }
> 
> ```

That does not convert every number in the field, it just call .to\_f on the string. You could replace the two mutate filters with

```
    ruby {
        code => '
            m = event.get("message").split(";")
            m.each_index { |x| m[x] = m[x].to_f }
            event.set("[message]", m)
        '
    }

```

---

<div class="post-metadata">

### Author: ![ofitz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofitz/32/47071_2.png) [@ofitz](https://discuss.elastic.co/u/ofitz)
#### Post date: [March 13, 2021, 7:45pm UTC](https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105/3 "2021-03-13T19:45:58Z")

</div>

Thanks @Badger

how can i take each row in new field. In my solution i create new Id in elasticsearch with one row as message.

my csv  
215900;216100;216300;216430;  
216750;216950;217230;217500;  
217780;217930;218100;218250;

i would like this output in elasticsearch:

```
{
   ...
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "_doc",
        "_id" : "wOcTfsdfssJ8PBIntss",
        "_score" : 1.0,
        "_source" : {
              "row1" : [215900;216100;216300;216430;],
              "row2" : [216750;216950;217230;217500;],
              "row3" : [217780;217930;218100;218250;]
```

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [March 14, 2021, 1:32am UTC](https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105/4 "2021-03-14T01:32:24Z")

</div>

that seems huge record. you trying to put whole csv as one document in ELK? you need to explain little more. as this does not seems correct design.

When you read csv. each message is single line. and that is one document (one record) in elk.

---

<div class="post-metadata">

### Author: ![ofitz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ofitz/32/47071_2.png) [@ofitz](https://discuss.elastic.co/u/ofitz)
#### Post date: [March 14, 2021, 3:48pm UTC](https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105/5 "2021-03-14T15:48:39Z")

</div>

I have over 5000 values in my csv file. i would like to visialize the values as Heatmap in Kibana.

What is the best design for this task?

One way is: store each value with timestamp as dokument to ES and visualize it. Can i do that with logstash?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 11, 2021, 3:49pm UTC](https://discuss.elastic.co/t/csv-to-float-from-logstash-to-elasticsearch/267105/6 "2021-04-11T15:49:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
