Logstash import file from url

Elastic Stack Logstash
1

I am new to Elasticsearch and Logstash, i just need to parse json data from url. Json link is : https://www.circl.lu/doc/misp/feed-osint/0165e5d7-51e6-4c2e-a382-1dd1e706f7bb.json

this is my logstash conf file:

input {

    http {
        url => "https://www.circl.lu/doc/misp/feed-osint/0165e5d7-51e6-4c2e-a382-1dd1e706f7bb.json" 
        codec => json
    }

}

filter {

}

output {
    elasticsearch {
        hosts => "http://localhost:9200"
        index => "indexlogstash"
    }
    stdout { codec => rubydebug }
    
}

error i get :

logstash
logstash1920×1080 225 KB

2

Hi,

url is an option that does not exist for http input plugin.

3

Hi,
what is the best way to parse jso file via url using logstash.

4

You can try the http_poller input plugin.

5

hi @leandrojmp Thank for your reply,
http_poller use a schedule.
in my case i want to send it just once.

6

Hi,

Is it not possible for you to run a curl command, print the result in a file and read this file with logstash ?

7

yes i can do that

8

Hi,

It seems that the http input plugin, the http_poller input plugin and the exec input plugin are not compatible with the single reading of a URL.

So if you can do a curl and print the result in a file, try this :

  1. Do a curl and print the result in a file
  2. Configure your logstash to read the file
    input => use the multiline codec plugin to read the entire file in one time example here.
    filter => use the json plugin to get all your data.
    output => do what you want.

You can even configure logstash before the curl command and configure it to make it read the file when it is edited.

Cad.

9

Hi

i tried something like this but i get a type date to text error dunno why

input {
  exec {
        command => "curl https://www.circl.lu/doc/misp/feed-osint/0165e5d7-51e6-4c2e-a382-1dd1e706f7bb.json"
        interval => 30
        codec => "json"
        }
}
filter{
        json{
         source=>"message"}
        }       
output {
        elasticsearch{
    hosts => "127.0.0.1:9200"
    index => "zzzzzzzz-%{+YYYY-MM-dd}"
        }
}

i get this error mostly need to convert something

:response=>{"index"=>{"_index"=>"zzzzzzzz-2021-08-13", "_type"=>"_doc", "_id"=>"3rNpP3sBaVcWztqJKPzH", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [Event.Object.Attribute.value] cannot be changed from type [date] to [text]"}}}}
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.