Logstash could not index event to Elasticsearch

Good afternoon all,

I want to preface this post that I am completely new to Elastic and all its components. I am currently working on getting Logstash to produce an API call to our web proxy server. I have searched the interwebs for any information on anyone else that has completed this process with our particular vendor, but I have not been lucky in that aspect. So I will try to explain what I have done here.

I installed the "http_poller" plugin on my logstash server. Built a pipeline in in Kibana. I can see the index in Kibana, but I am getting errors.

Errors from Logstash:

[2020-04-20T14:08:06,030][WARN ][logstash.outputs.elasticsearch][Netskope] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netskope-alerts", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x44131609], :response=>{"index"=>{"_index"=>"netskope-alerts", "_type"=>"_doc", "_id"=>"VPH9mHEBiQ9nb_9agQ9N", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [data.file_size] of different type, current_type [long], merged_type [text]"}}}}
[2020-04-20T14:09:04,608][WARN ][logstash.outputs.elasticsearch][Netskope] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netskope-alerts", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x3aaff582], :response=>{"index"=>{"_index"=>"netskope-alerts", "_type"=>"_doc", "_id"=>"p57-mHEBYBKL1gEnZrc9", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [data.file_size] of different type, current_type [text], merged_type [long]"}}}}
[2020-04-20T14:28:06,006][WARN ][logstash.outputs.elasticsearch][Netskope] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netskope-alerts", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x22e5f2cf], :response=>{"index"=>{"_index"=>"netskope-alerts", "_type"=>"_doc", "_id"=>"lfMPmXEBiQ9nb_9azYSV", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [data.file_size] of different type, current_type [long], merged_type [text]"}}}}

I have also added my pipeline code as well:

    input {
      http_poller {
        id => "Netskope_poller"
        urls => {
          alerts => {
            method => "GET"
            url => "https://destination.com/api/v1/alerts?token=sometoken&timeperiod=86400"
            headers => {
              Accept => "application/json"
            }
        }
        }
        # Maximum amount of time to wait for a request to complete
        request_timeout => 30
        # How far apart requests should be
        schedule => { cron => "* * * * * UTC"}
        # Decode the results as JSON
        codec => "json"
      }
    }
    filter {
    }
    output {
    elasticsearch {
        index => "netskope-alerts"
        hosts => ["https://*********.aws.found.io:9243"]
        user => "SomeUser"
        password => "SomePassword"
    }
    }

I know the API call works due to running it in postman, I get back the correct json output.

A field on a document in elasticsearch has a type. That is set by either an index template or the first document index that contains the field. In your index the field data.file_size (which would be [data][file_size] in logstash) has type long. You need to look at one of the documents you are trying to index and see what the value and type of the field is. Try

output { stdout { codec => rubydebug } }

I applied the change to my pipeline.yml file and restarted the logstash service on the server. When I tail the log now all I see is:

[2020-04-20T15:58:11,551][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch bulk_path=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", password=>, hosts=>[https://fa13ba698bcb4cdc98e51b13e421c92e.us-east-1.aws.found.io:9243], sniffing=>false, manage_template=>false, id=>"dce8ad510ab7216c602dd1438b946e1109089f2cd09cce1985e8507e73fb202a", user=>"logstash_system", document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_10f62eca-8366-48ff-abbb-01990bc58f4f", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2020-04-20T15:58:11,612][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://logstash_system:xxxxxx@fa13ba698bcb4cdc98e51b13e421c92e.us-east-1.aws.found.io:9243/]}}
[2020-04-20T15:58:11,677][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@fa13ba698bcb4cdc98e51b13e421c92e.us-east-1.aws.found.io:9243/"}
[2020-04-20T15:58:11,734][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] ES Output version determined {:es_version=>7}
[2020-04-20T15:58:11,735][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2020-04-20T15:58:11,787][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://fa13ba698bcb4cdc98e51b13e421c92e.us-east-1.aws.found.io:9243"]}
[2020-04-20T15:58:11,801][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x7b230b2d run>"}
[2020-04-20T15:58:11,847][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2020-04-20T15:58:11,890][INFO ][logstash.agent ] Pipelines running {:count=>3, :running_pipelines=>[:DCFW, :".monitoring-logstash", :Netskope], :non_running_pipelines=>}
[2020-04-20T15:58:12,202][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Which is normally what I see when everything is working properly, but I do not have an output configured now. Everything is going to stdout. Which if I remember correctly is contained on the logstash server? But where would it go then?

Thanks,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.