Good afternoon all,
I want to preface this post that I am completely new to Elastic and all its components. I am currently working on getting Logstash to produce an API call to our web proxy server. I have searched the interwebs for any information on anyone else that has completed this process with our particular vendor, but I have not been lucky in that aspect. So I will try to explain what I have done here.
I installed the "http_poller" plugin on my logstash server. Built a pipeline in in Kibana. I can see the index in Kibana, but I am getting errors.
Errors from Logstash:
[2020-04-20T14:08:06,030][WARN ][logstash.outputs.elasticsearch][Netskope] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netskope-alerts", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x44131609], :response=>{"index"=>{"_index"=>"netskope-alerts", "_type"=>"_doc", "_id"=>"VPH9mHEBiQ9nb_9agQ9N", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [data.file_size] of different type, current_type [long], merged_type [text]"}}}}
[2020-04-20T14:09:04,608][WARN ][logstash.outputs.elasticsearch][Netskope] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netskope-alerts", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x3aaff582], :response=>{"index"=>{"_index"=>"netskope-alerts", "_type"=>"_doc", "_id"=>"p57-mHEBYBKL1gEnZrc9", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [data.file_size] of different type, current_type [text], merged_type [long]"}}}}
[2020-04-20T14:28:06,006][WARN ][logstash.outputs.elasticsearch][Netskope] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netskope-alerts", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x22e5f2cf], :response=>{"index"=>{"_index"=>"netskope-alerts", "_type"=>"_doc", "_id"=>"lfMPmXEBiQ9nb_9azYSV", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [data.file_size] of different type, current_type [long], merged_type [text]"}}}}
I have also added my pipeline code as well:
input {
http_poller {
id => "Netskope_poller"
urls => {
alerts => {
method => "GET"
url => "https://destination.com/api/v1/alerts?token=sometoken&timeperiod=86400"
headers => {
Accept => "application/json"
}
}
}
# Maximum amount of time to wait for a request to complete
request_timeout => 30
# How far apart requests should be
schedule => { cron => "* * * * * UTC"}
# Decode the results as JSON
codec => "json"
}
}
filter {
}
output {
elasticsearch {
index => "netskope-alerts"
hosts => ["https://*********.aws.found.io:9243"]
user => "SomeUser"
password => "SomePassword"
}
}
I know the API call works due to running it in postman, I get back the correct json output.