How to push file from github to logstash

[dheeru11 (dheerendrakumar)]

I have file which will be uploaded to github daily. I want push the same file to logstash. How can i do that ?

[ptamba]

logstash can read file from filesystem

[Luca_Belluccini]

You can use the HTTP Poller input to download the file periodically.

You can grab the URL of the file from GitHub using the "Raw" button (the url will be something like https://raw.githubusercontent.com/user/repository/branch/filename)

[dheeru11 (dheerendrakumar)]

Thanks.

But i have file format like filename_{Date}.csv
I will get one file everyday in .csv format and i need to upload that to logstash. Kindly share any conf sample file for the same.

[Luca_Belluccini]

If you have to change the URL every day, then I think you can use one of the following:

1. An exec input (doc), where you put a script in your preferred language which clones/syncs the Github repository and you sort by latest file and you print the file to stdout

2. An exec input which generates the date as string, plus an http filter (doc) to download the file using the date generated in the exec.

[dheeru11 (dheerendrakumar)]

Hi Luca,

If i want to use the same file how can i modify my below conf file. Getting error as PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

input {
http_poller {
urls => {
urlname => "https://path to csv file"
}
request_timeout => 60
schedule => {every => "30s"}
codec => "line"
}
}

filter {
csv {
separator => ","
columns => ["Transaction Name","Minimum"]
}
mutate {convert => ["Transaction Name","string"]}
mutate {convert => ["Minimum","float"]}

}

output {
elasticsearch {
hosts => ["http://host:9200"]
index => "results"
}
stdout {codec => rubydebug}
}

[Luca_Belluccini]

Hello @dheeru11

Please follow the documentation. In particular, there's a section about certificates called Using the HTTP poller with custom a custom CA or self signed cert.

In particular:

1. Download and generate the JKS containing the CA Root certificate:

   openssl s_client -showcerts -connect MYURL:MYPORT </dev/null 2>/dev/null|openssl x509 -outform PEM > downloaded_cert.pem; keytool -import -alias test -file downloaded_cert.pem -keystore downloaded_truststore.jks
   

2. Add to the http_poller

      truststore => "/path/to/downloaded_truststore.jks"
      truststore_password => "mypassword"