Load Several CSV Files to several indices in Elastic cloud based on date field

Elastic Stack Logstash
1

I have several CSV I need to load them to elastic cloud based on date field such that each index will contain only one day. is it possible?

2

Put into the output as below:

output {
   elasticsearch { 
      hosts => ["blah:9200"]
      index => "csv-%{+YYYY.MM.dd}"
   }
}

Regards,
Fadjar Tandabawana

3

Thanks Fadjar,

What I mean that I have history data split in the following based on a date field inside the data
csv-20200112
Csv-20200114
Csv-20200115
.
.
Csv-20200325
And I want to load each csv to a separate index on elastic search corresponding to the input csv

4

Do your process per file

ex:
CSV-20200112

input {
  file => "csv-20200112"
  tags => "20200112"
  .
  .
}
filter {
  if [tags] == "20200112"
  process something here
}
output {
  if [tags] == "20200112"
  elasticsearch {
    host => "localhost:9200"
    index => "blah-%tags"
  }
}

save as 20200112.conf

do to another csv
Don't forget the tags of each process.

You can build the conf file using bash script and using template and sed

Regards,
Fadjar Tandabawana

5

Thanks Fadjar340

What is the usage of tags if I am going to create conf for each file file (date)

6

I think you can use script to do it, like below:

#!/bin/bash

shopt -s extglob nullglob

basedir=/home/csv
first=10
increment=1

omitdir=cmmdm

if [[ -z $omitdir ]]; then
   cdarray=( "$basedir"/*/ )
else
   cdarray=( "$basedir"/!($omitdir)/ )
fi

cdarray=( "${cdarray[@]#"$basedir/"}" )

cdarray=( "${cdarray[@]%/}" )

length=${#cdarray[@]}

vals=($(seq -w -s ' '10 $increment $length))

for ((i=0; i<${length}; i++)); do
   # echo "First parameter : ${cdarray[$i]} -- second parameter : ${vals[$i]}"
    sed s/%file%/${cdarray[$i]}/g ./template.txt > ./${vals[$i]}-${cdarray[$i]}.conf
done

The template should follow the standard conf then just add %file% as tags that you want to use

Regards,
Fadjar Tandabawana

7

Thanks for you help

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.