Multiple CSV files need to upload for same column names

Elastic Stack Logstash
1

Hi,

I need to upload monthly wise csv data(which contains the same columns) using logstash. So how to upload that and create an index.

Thanks in Advance.

2

Hi Hari ,

You can use csv filter plugin for the same.
Read the multiple csv file (in your case monthly basis) from the same location using file input plugin and use csv filter to process the data as per columns and send it to output to elasticsearch (if are storing it in es)

input {
  file {
    path => "/path_to_your_csv_files/*.csv"
    start_position => "beginning"
  }
}
filter {
  csv {
      separator => ","
#Your column names here
     columns => ["column1","column2".......]
  }
}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "your_index_name_here"
  }

}
3

Hi Rohan,

Thanks for replied. I have already tried this option, But it won't work. So, Please tell some other option. Actually, I'm using elastic in Windows. So, Is any problem?

4

It should work , what is the issue that your facing here? and there is no problem at all if you are using elasticsearch on windows.

5

Hi Rohan,

I have uploaded images which i am getting issues. First image is config file and second screen is error when running the by that format.

cog
cog.png919×593 28.9 KB
drop
drop.png1366×768 96.4 KB

6

Is the data getting indexed into elasticsearch?

as output screenshot only shows document_type deprecation warnings
Kindly validate your config file

7

No, It's not indexed. Because I have used path like="\path*.csv" . So, It's not getting indexed. I have checked the config already. If I mention single csv file name in path, It works, But when I using *.csv, Its showing error. So, Help me to resolve.

8

If I mention single csv file name in path, It works, But when I using *.csv, Its showing error. So, Help me to resolve.

Have you tried using forward slashes instead of backslashes in the filename pattern? Also, make sure you include the drive letter.

9

Hi,
Thanks for reply. I have tried that also, It won't work. I'm using windows. Here how to define path?

10

Finally, I got the solution. I have used "\foldername1\foldername2/*". It works fine.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.