# Converting more than one column destroys data

**URL:** <https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161>\
**Category:** Logstash\
**Created:** [September 28, 2020, 9:54am UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161 "2020-09-28T09:54:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Christoph1](https://avatars.discourse-cdn.com/v4/letter/c/85f322/32.png) [@Christoph1](https://discuss.elastic.co/u/Christoph1)\
**Post date:** [September 28, 2020, 9:54am UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/1 "2020-09-28T09:54:47Z")

</div>

converting more than one column from string \> integer completely invalidates the logstash output.

The values are interpreted as keys somehow

What am I missing when converting?

regardless of:  
CSV { convert =\> {}}  
mutate{ convert =\> {}}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 28, 2020, 12:27pm UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/2 "2020-09-28T12:27:02Z")

</div>

> [@Christoph1](#):
>
> The values are interpreted as keys somehow

What do you mean by this?

---

<div class="post-metadata">

**Author:** ![Christoph1](https://avatars.discourse-cdn.com/v4/letter/c/85f322/32.png) [@Christoph1](https://discuss.elastic.co/u/Christoph1)\
**Post date:** [September 28, 2020, 1:13pm UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/3 "2020-09-28T13:13:07Z")

</div>

Thank you for your reply.

I am quite new to the logstash pipeline, which means I am struggling to identify causes and problems:

my scenario is as follows:

i have +.csv s that are read via filebeat, and parsed via filter csv plugin.

```
  csv {
    autodetect_column_names => true
    skip_header => true
    separator =>";"
    convert => {
 "key1"=>"integer"
 "key2"=>"integer"
 ...
 }

```

When converting any or a number a colums, the output gets strange:

the original strings

```
 "key1":"1"
 "key2":"2"

```

that should look like

```
 "key1":1
 "key2":2

```

may become  
`"1":2`  
or  
`"key1":1`  
or  
`column2:"2"`

I have a single worker and I (maybe) depending on the amount of data sometimes the conversion works without problem, sometime it adds a random "column28".

Is there some additional synchronizing to be done when handling many large files?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 28, 2020, 1:42pm UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/4 "2020-09-28T13:42:19Z")

</div>

> [@Christoph1](#):
>
> sometime it adds a random "column28".

autogenerate\_column\_names is true by default. If the header row has 27 entries and a later row has 28 then that is what it will be called.

Which version of logstash are you running?

---

<div class="post-metadata">

**Author:** ![Christoph1](https://avatars.discourse-cdn.com/v4/letter/c/85f322/32.png) [@Christoph1](https://discuss.elastic.co/u/Christoph1)\
**Post date:** [September 28, 2020, 2:17pm UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/5 "2020-09-28T14:17:53Z")

</div>

Thank you for your reply.  
I am aware of the auto generation feature.  
I am also using autodetect names  
The columns match and the pipeline succeeds sometimes, hence my confusion.

I am using the latest version

Mit freundlichen Grüßen

Dr.-Ing. **Christoph Weber**

IT-Consultant

ASTRUM IT GmbH

Am Wolfsmantel 2  
D-91058 Erlangen  
Tel.: +49 91 31 94 08-309

[www.astrum-it.de](http://www.astrum-it.de/)

Geschäftsführer:  
Gerhard Pölz  
Walter Greul  
Amtsgericht Fürth, HRB 6549  
UST.-Id.: DE186320885

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 28, 2020, 3:11pm UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/6 "2020-09-28T15:11:28Z")

</div>

> [@Christoph1](#):
>
> I am using the latest version

The only thing I can think of is that pipeline.ordered is set to false, so that sometimes it picks up a data row as a header.

---

<div class="post-metadata">

**Author:** ![Christoph1](https://avatars.discourse-cdn.com/v4/letter/c/85f322/32.png) [@Christoph1](https://discuss.elastic.co/u/Christoph1)\
**Post date:** [September 28, 2020, 4:26pm UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/7 "2020-09-28T16:26:11Z")

</div>

Unfortunately the pipeline.workers are already set to 1.

Ordering is then set by default.

![image001.png](https://us1.discourse-cdn.com/elastic/original/3X/3/0/30b241525e203063b644595fa395f35831648823.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2020, 4:26pm UTC](https://discuss.elastic.co/t/converting-more-than-one-column-destroys-data/250161/8 "2020-10-26T16:26:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.