# Csv filter can only be used for one file? (multiple files messes up fields/columns)

**URL:** <https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995>\
**Category:** Logstash\
**Created:** [September 11, 2017, 4:20am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995 "2017-09-11T04:20:40Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\
**Post date:** [September 11, 2017, 4:20am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/1 "2017-09-11T04:20:40Z")

</div>

Hi,

I'm having a big problem with the csv filter. I got four .csv files, each with a separate .conf.

When I run those configs in stdout they are fine. When I run logstash and have only one of them active, data gets imported fine as well.

However as soon as I got two or more .conf files using the csv filter it messes up the data.

E.g. Index1 looks OK but index2 has some fields from index1 and some column names have become field values etc. It is very strange.

I also noticed that both of those two indexes also have the reverse dns lookup field from by netflow.conf.

Looks like Logstash is doing something strange somewhere or something is wrong with the way I got logstash set up. How can I fix this?

Index1 csv headers

```auto
Date	Duration	KBytes	Service	Label	APN	Code	Lat	Lon	CN0	Message	TxKBytes	RxKBytes
```

Index2 csv headers

```auto
Date	Block	Message	Code	Lat	Lon	CN0
```

index1 json

```auto
{
  "_index": "data-2017.09",
  "_type": "data",
  "_id": "AV5vIhbX0t3riYLIY8_W",
  "_version": 1,
  "_score": null,
  "_source": {
    "TxKBytes": "73876",
    "Message": "Power supply was turned off",
    "RxKBytes": "487211",
    "CNO": "67.8",
    "Label": "Default",
    "Service": "Standard",
    "Data": "2017-09-01 13:05:05",
    "Duration": "20:03:46",
    "Lon": "lonvalue",
    "type": "data",
    "Code": "errorcode",
    "path": "/home/test/Desktop/test/test2/data/data.csv",
    "netflow": {
      "ipv4_src_host": "%{[netflow][ipv4_src_addr]}",
      "ipv4_dst_host": "%{[netflow][ipv4_dst_addr]}"
    },
    "@timestamp": "2017-09-11T04:10:58.551Z",
    "KBytes": "1",
    "@version": "1",
    "host": "ELK-test",
    "Lat": "latvalue",
    "APN": "myapn"
  },
  "fields": {
    "@timestamp": [
      1505103058551
    ]
  },
  "sort": [
    1505103058551
  ]
}
```

index2 json

```auto
{
  "_index": "event-2017.09",
  "_type": "event",
  "_id": "AV5vIhYL0t3riYLIY89s",
  "_version": 1,
  "_score": null,
  "_source": {
    "Label": "latvalue",
    "Service": "errorcode",
    "Data": "2017-09-10 11:38:23",
    "Duration": "ADE",
    "type": "event",
    "Code": CNOvalue",
    "path": "/home/test/Desktop/test/test2/event/event.csv",
    "netflow": {
      "ipv4_src_host": "%{[netflow][ipv4_src_addr]}",
      "ipv4_dst_host": "%{[netflow][ipv4_dst_addr]}"
    },
    "@timestamp": "2017-09-11T04:10:58.354Z",
    "KBytes": "Notice: Status (Signal).",
    "@version": "1",
    "host": "ELK-test",
    "APN": "lonvalue"
  },
  "fields": {
    "@timestamp": [
      1505103058354
    ]
  },
  "sort": [
    1505103058354
  ]
}
```

As you can see the second index has values that A) should not be there and B) are in the incorrect field. Both have the netflow fields as well which should no be there either.

Index1 conf

```auto
input {
  file {
    type => "data"
    path => "/home/test/Desktop/test/test2/data/data.csv"
    sincedb_path => "/dev/null"
    start_position => "beginning"
  }
}

filter {
    csv {
    columns => ["Data","Duration","KBytes","Service","Label","APN","Code","Lat","Lon","CNO","Message","TxKBytes","RxKBytes"]
    }
    date {
    match => ["Date", "yyyy-MM-dd HH:mm:ss"]
    timezone => "UTC"
    target => "@timestamp"
    }
    mutate {
    remove_field => ["message", "Date"]
    }
}

output {
if [type] == "data" {
elasticsearch {
hosts => localhost
index => "data-%{+YYYY.MM}"
}
}
}
```

Index2 conf

```auto
input {
  file {
    type => "event"
    path => "/home/test/Desktop/test/test2/event/event.csv"
    sincedb_path => "/dev/null"
    start_position => "beginning"
  }
}

filter {
    csv {
    columns => ["Date","Block","Message","Code","Lat","Lon","CNO"]
    }
    date {
    match => ["Date", "yyyy-MM-dd HH:mm:ss"]
    timezone => "UTC"
    target => "@timestamp"
    }
    mutate {
    remove_field => ["message", "Date"]
    }
}

output {
if [type] == "event" {
elasticsearch {
hosts => localhost
index => "event-%{+YYYY.MM}"
}
}
}
```

---

<div class="post-metadata">

**Author:** ![kmsasidhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmsasidhar/32/21838_2.png) [@kmsasidhar](https://discuss.elastic.co/u/kmsasidhar)\
**Post date:** [September 11, 2017, 4:33am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/2 "2017-09-11T04:33:52Z")

</div>

Instead of splitting conf files on Index basis, pls splitting conf files like this  
1\_input.conf  
2\_filter.conf  
3\_output.conf

Add appropriate sections of each index in each conf file.

---

<div class="post-metadata">

**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\
**Post date:** [September 11, 2017, 4:37am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/3 "2017-09-11T04:37:59Z")

</div>

Thanks but I'm not sure I understand. That sounds very different from what i've seen/read so far. Do you have any examples?

My understand is that you could use the type field to separate config files.

---

<div class="post-metadata">

**Author:** ![kmsasidhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmsasidhar/32/21838_2.png) [@kmsasidhar](https://discuss.elastic.co/u/kmsasidhar)\
**Post date:** [September 11, 2017, 4:48am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/4 "2017-09-11T04:48:01Z")

</div>

In my use case, I had to do that, will share more.  
On mobile right now, will get back in a couple of hours.

---

<div class="post-metadata">

**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\
**Post date:** [September 11, 2017, 5:06am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/5 "2017-09-11T05:06:41Z")

</div>

Thank you.

Maybe @magnusbaeck could also chime in as to why the [type] filter apparently cannot be used to separate configs?

---

<div class="post-metadata">

**Author:** ![kmsasidhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmsasidhar/32/21838_2.png) [@kmsasidhar](https://discuss.elastic.co/u/kmsasidhar)\
**Post date:** [September 11, 2017, 5:15am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/6 "2017-09-11T05:15:03Z")

</div>

> [@\[solved\] Multiple logstash config file](https://discuss.elastic.co/t/solved-multiple-logstash-config-file/51692/6):
>
> I Have tried the option . Yes it is working to post the data but all the data is getting posted for every index i.e Index 1 & Index2 is having mixed data. Please suggest where I am missing. Regards, Prateek

Pl refer to the above link.

Type can still be used.  
Magnus of course, can solve it in a sec.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 11, 2017, 5:32am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/7 "2017-09-11T05:32:17Z")

</div>

You can create a separate file input for each file type and assign a tag there. Then use conditionals based on this tag to select the appropriate csv filter.

---

<div class="post-metadata">

**Author:** ![kmsasidhar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmsasidhar/32/21838_2.png) [@kmsasidhar](https://discuss.elastic.co/u/kmsasidhar)\
**Post date:** [September 11, 2017, 5:37am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/8 "2017-09-11T05:37:28Z")

</div>

In your case, apply if [type] in your filter section of both confs appropriately. It would take care I think.

---

<div class="post-metadata">

**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\
**Post date:** [September 11, 2017, 6:49am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/9 "2017-09-11T06:49:56Z")

</div>

How is that any different from input type =\> and output if type == data I got there already?

input  
Tag =\> something

filter  
as is

output  
If "something" in [tags]

Should work? Should I keep type in input and output?

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [September 11, 2017, 6:57am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/10 "2017-09-11T06:57:24Z")

</div>

Apart from separating outputs based on input, you have to separate filter processes as well. Like

```auto
filter {
    if [type] == "event" {
        # do stuff
    }
    else if [type] == "data" {
        # do other stuff
    }
}
```

Having different configurations in different files has no effect on filter segregation since Logstash just concatenates them in one big config internally.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 11, 2017, 7:00am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/11 "2017-09-11T07:00:08Z")

</div>

It is the same, but you should use conditionals in your filter block too.

---

<div class="post-metadata">

**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\
**Post date:** [September 11, 2017, 8:51am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/12 "2017-09-11T08:51:40Z")

</div>

Seems this is doing the trick 🙂

Will test further.

Though I'm seeing problems with converting gps locations to an integer. Logstash is cutting off everything after a comma (,) or dot (.). Is there any way to avoid this?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 11, 2017, 8:59am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/13 "2017-09-11T08:59:12Z")

</div>

Please don't ping people that aren't part of the thread like that.

Magnus volunteers his time here 🙂

---

<div class="post-metadata">

**Author:** ![Sjaak01](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\
**Post date:** [September 13, 2017, 4:33am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/14 "2017-09-13T04:33:48Z")

</div>

I'm sorry, wont do it again. I appreciate all the work people put in here trying to help others.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2017, 4:34am UTC](https://discuss.elastic.co/t/csv-filter-can-only-be-used-for-one-file-multiple-files-messes-up-fields-columns/99995/15 "2017-10-11T04:34:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.