Logstash import file from url

I am new to Elasticsearch and Logstash, i just need to parse json data from url. Json link is : https://www.circl.lu/doc/misp/feed-osint/0165e5d7-51e6-4c2e-a382-1dd1e706f7bb.json

this is my logstash conf file:

input {

    http {
        url => "https://www.circl.lu/doc/misp/feed-osint/0165e5d7-51e6-4c2e-a382-1dd1e706f7bb.json" 
        codec => json


filter {


output {
    elasticsearch {
        hosts => "http://localhost:9200"
        index => "indexlogstash"
    stdout { codec => rubydebug }

error i get :


url is an option that does not exist for http input plugin.

what is the best way to parse jso file via url using logstash.

You can try the http_poller input plugin.

hi @leandrojmp Thank for your reply,
http_poller use a schedule.
in my case i want to send it just once.


Is it not possible for you to run a curl command, print the result in a file and read this file with logstash ?

yes i can do that


It seems that the http input plugin, the http_poller input plugin and the exec input plugin are not compatible with the single reading of a URL.

So if you can do a curl and print the result in a file, try this :

  1. Do a curl and print the result in a file
  2. Configure your logstash to read the file
    input => use the multiline codec plugin to read the entire file in one time example here.
    filter => use the json plugin to get all your data.
    output => do what you want.

You can even configure logstash before the curl command and configure it to make it read the file when it is edited.



i tried something like this but i get a type date to text error dunno why

input {
  exec {
        command => "curl https://www.circl.lu/doc/misp/feed-osint/0165e5d7-51e6-4c2e-a382-1dd1e706f7bb.json"
        interval => 30
        codec => "json"
output {
    hosts => ""
    index => "zzzzzzzz-%{+YYYY-MM-dd}"

i get this error mostly need to convert something

:response=>{"index"=>{"_index"=>"zzzzzzzz-2021-08-13", "_type"=>"_doc", "_id"=>"3rNpP3sBaVcWztqJKPzH", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [Event.Object.Attribute.value] cannot be changed from type [date] to [text]"}}}}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.