# Import csv files to two different index

**URL:** <https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456>\
**Category:** Logstash\
**Created:** [July 24, 2020, 6:15am UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456 "2020-07-24T06:15:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![peretz\_shlomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peretz_shlomi/32/54701_2.png) [@peretz\_shlomi](https://discuss.elastic.co/u/peretz_shlomi)\
**Post date:** [July 24, 2020, 6:15am UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456/1 "2020-07-24T06:15:32Z")

</div>

hello

I want to import two different CSV file types into two different index.

What is the right way to do it ?

---

<div class="post-metadata">

**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\
**Post date:** [July 27, 2020, 8:48am UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456/2 "2020-07-27T08:48:54Z")

</div>

you need to ensure the output sends to two indexes. So have to tag the CSV files and then ensure each goes to its own indexes

something of below style.. You could mutate your fields to match an index-style and then output it to that

```auto
# pipelines.yml
- pipeline.id: employees_index
  path.config: "pipelines/employees_index.conf"

```

```auto
# employees_index.conf
input {
    file {
        path => "/tmp/csv/employees_it.csv"
        tags => ["employees_it"]
    }
    file {
        path => "/tmp/csv/employees_field.csv"
        tags => ["employees_field"]
    }
}

filter {
  if "it_employees" in [tags] {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "employees_it" }
    }
  } else if "field_employees" in [tags] {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "employees_field" }
    }
  } else {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "employees_unknown" }
    }
  }
  csv {
      separator => ","
    }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    user => "elastic"
    password => "somepassword"
    index => "%{[@metadata][index_prefix]}-%{+YYYY.MM}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![peretz\_shlomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peretz_shlomi/32/54701_2.png) [@peretz\_shlomi](https://discuss.elastic.co/u/peretz_shlomi)\
**Post date:** [July 31, 2020, 3:53am UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456/3 "2020-07-31T03:53:21Z")

</div>

hi  
Thanks for the help

Attached is my settings file , can you help me with the filter?  
all the file in c:\input goes to index aoi\_false  
all the file in c:\inputppm gose to index aoi\_ppm

the logstash.conf file :  
input {  
file {  
path =\> "c:/input/_.csv"  
tags =\> ["aoi\_false"]  
start\_position =\> "beginning"  
}  
file {  
path =\> "c:/inputppm/_.csv"  
tags =\> ["aoi\_ppm"]  
start\_position =\> "beginning"  
}

}

filter {  
csv {  
separator =\> ","  
columns =\> ["time","file type","Product name","Side","board result","Machine NAME","Barcode","Barcode slave","Component Qty","Real Component Qty","NG Amount","NG Qty","Operator","Working Order","module number","location","Matiral NO","NG name","NG result"]  
}  
mutate {convert =\> ["Component Qty" , "integer"] }  
mutate {convert =\> ["Real Component Qty" , "integer"] }  
mutate {convert =\> ["NG Amount" , "integer"] }  
mutate {convert =\> ["NG Qty" , "integer"] }  
date {  
match =\> ["time", "yyyyMMdd HHmmss"]  
timezone =\> "UTC"  
}  
}

output {

elasticsearch {  
hosts =\> ["127.0.0.1:9200"]  
user =\> "elastic"  
password =\> "sLf7r0eAsdKD4XPxoWpO"  
index =\> "aoi\_false"

}  
stdout {}  
}

---

<div class="post-metadata">

**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\
**Post date:** [July 31, 2020, 8:00am UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456/4 "2020-07-31T08:00:47Z")

</div>

> [@kelk](#):
>
> `index => "%{[@metadata][index_prefix]}-%{+YYYY.MM}"`

Please try (PS: I haven't tested this as I don't have sample data. You may need to tweak here and there)

```auto
input {
  file {
    path => "c:/input/*.csv"
    tags => ["aoi_false"]
    start_position => "beginning"
  }
  file {
    path => "c:/inputppm/*.csv"
    tags => ["aoi_ppm"]
    start_position => "beginning"
  }
}

filter {
  csv {
    separator => ","
    columns => ["time","file type","Product name","Side","board result","Machine NAME","Barcode","Barcode slave","Component Qty","Real Component Qty","NG Amount","NG Qty","Operator","Working Order","module number","location","Matiral NO","NG name","NG result"]
  }
  mutate {convert => ["Component Qty" , "integer"] }
  mutate {convert => ["Real Component Qty" , "integer"] }
  mutate {convert => ["NG Amount" , "integer"] }
  mutate {convert => ["NG Qty" , "integer"] }
  date {
    match => ["time", "yyyyMMdd HHmmss"]
    timezone => "UTC"
  }
  
  if "aoi_false" in [tags] {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "aoi_false" }
    }
  } else if "aoi_ppm" in [tags] {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "aoi_ppm" }
    }
  } else {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "aoi_unknown" }
    }
  }  
  
}

output {
  elasticsearch {
  hosts => ["127.0.0.1:9200"]
  user => "elastic"
  password => "sLf7r0eAsdKD4XPxoWpO"
  index => "%{[@metadata][index_prefix]}-%{+YYYY.MM}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![peretz\_shlomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peretz_shlomi/32/54701_2.png) [@peretz\_shlomi](https://discuss.elastic.co/u/peretz_shlomi)\
**Post date:** [August 2, 2020, 10:26am UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456/5 "2020-08-02T10:26:29Z")

</div>

pipelines.yml - What exactly do I need to change?

---

<div class="post-metadata">

**Author:** ![peretz\_shlomi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peretz_shlomi/32/54701_2.png) [@peretz\_shlomi](https://discuss.elastic.co/u/peretz_shlomi)\
**Post date:** [August 3, 2020, 1:09pm UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456/6 "2020-08-03T13:09:25Z")

</div>

the two config file insaid the config directory and the pipelines.yml olso.

in the pipelines.yml i update

- pipelines.id: logstash1  
path.config: "logstash1"
- pipelines.id: logstash2  
path.config: "logstash2"

I'm now stuck running the software logstash  
I enter the command:  
logstash --path.settings c:/kibana/logstash/config/

i get error " failed to read pipelines yaml file

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 31, 2020, 1:09pm UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456/7 "2020-08-31T13:09:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.