Could not index event to Elasticsearch in ES 6.0.0

Elastic Stack Elasticsearch
1

I'm using ES 6.0 and beat 6.0.

I check in logstash-plain.log, i got many error like this

[2017-12-12T18:30:25,655][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"fb-fim-dns-6.0.1-2017.50", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x113962d4], :response=>{"index"=>{"_index"=>"fb-fim-dns-6.0.1-2017.50", "_type"=>"doc", "_id"=>"DMF9SmAB_Dli1qCHfj8b", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [geoip.location] tried to parse field [null] as object, but found a concrete value"}}}}

[2017-12-13T01:14:27,072][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"fb-fim-mail-6.0.1-2017.50", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x69b5aa0d], :response=>{"index"=>{"_index"=>"fb-fim-mail-6.0.1-2017.50", "_type"=>"doc", "_id"=>"rh_vS2ABGJRfL5NeixeC", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [geoip.location]", "caused_by"=>{"type"=>"i_o_exception", "reason"=>"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@58d69e2; line: 1, column: 905]"}}}}}

[2017-12-13T00:51:04,074][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"wb-fim-terminal-5.6.0-2017.50", :_type=>"wineventlog", :_routing=>nil}, #LogStash::Event:0x5002cce5], :response=>{"index"=>{"_index"=>"wb-fim-terminal-5.6.0-2017.50", "_type"=>"wineventlog", "_id"=>"CBTaS2ABGJRfL5NeE75m", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [wb-fim-terminal-5.6.0-2017.50] as the final mapping would have more than 1 type: [doc, wineventlog]"}}}}

What should i do?. How to slove my problem? Thanks!

I try to add mapping for geoip but not work

Here is my index template Filebeat teamplate

2

logstash 6.0.1 should fix this issue

3

Thank @ericrichter, i will try to update V6.0.1 and report againt.

4

Have you had a look at the data you are sending through?

1 Like
5

Sure,
My index template worked in ES 5.6.x but when i update to 6.0.0, i got this issue.
I used geoip filter plugin, translate filter plugin. Some data of geoip will be pushed to geoip field

6

Can someone help me.

I updated ES, LS to 6.0.1, still got error

[2017-12-13T14:26:49,427][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"fb-fim-mail-6.0.1-2017.50", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x8e3cde7], :response=>{"index"=>{"_index"=>"fb-fim-mail-6.0.1-2017.50", "_type"=>"doc", "_id"=>"dPbFTmAB8SyTVy-3c_PI", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [geoip.location]", "caused_by"=>{"type"=>"i_o_exception", "reason"=>"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@24cd9e54; line: 1, column: 833]"}}}}}
[2017-12-13T14:26:49,427][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"fb-fim-mail-6.0.1-2017.50", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x22890712], :response=>{"index"=>{"_index"=>"fb-fim-mail-6.0.1-2017.50", "_type"=>"doc", "_id"=>"dfbFTmAB8SyTVy-3c_PI", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [geoip.location]", "caused_by"=>{"type"=>"i_o_exception", "reason"=>"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@38b1c1ca; line: 1, column: 799]"}}}}}

7

Did you follow @mujtabahussain advice?

Could you print with a stdout output and json or rubydebug codec what is supposed to be indexed by elasticsearch?

8

Some output here

{
           "source" => "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex171205.log",
             "type" => "iis-access",
       "http.value" => "exchangecookie=7e1608b9803a40c1ac80e497a023dad4;+dtCookie=D3845DF09C18E5F80F291479A07643AB|X2RlZmF1bHR8MQ",
         "hostname" => "HCAS01-SRV",
         "beattype" => "doc",
         "clientip" => "10.3.50.215",
         "@version" => "1",
             "beat" => {
            "name" => "HCAS01-srv",
        "hostname" => "HCAS01-srv",
         "version" => "6.0.1"
    },
             "host" => "HCAS01-srv",
    "http.response" => "200",
      "http.method" => "POST",
       "user_agent" => {
              "os" => "Mac OS X",
           "major" => "5",
           "minor" => "0",
           "build" => "",
        "os_minor" => "10",
        "os_major" => "10",
            "name" => "ExchangeWebServices",
         "os_name" => "Mac OS X",
          "device" => "Other"
    },
        "timestamp" => "2017-12-05 06:38:25",
     "http.request" => "/autodiscover/autodiscover.xml",
            "geoip" => {},
           "offset" => 1181493910,
        "http.port" => "443",
         "beatname" => "fb-fim-mail",
        "http.site" => "W3SVC1",
       "prospector" => {
        "type" => "log"
    },
          "message" => "2017-12-05 06:38:25 W3SVC1 HCAS01-SRV 10.4.11.62 POST /autodiscover/autodiscover.xml - 443 company.com\\tuanna20 10.3.50.215 HTTP/1.1 Mac+OS+X/10.10.5+(14F2511);+ExchangeWebServices/5.0+(213);+Mail/8.2+(2104) exchangecookie=7e1608b9803a40c1ac80e497a023dad4;+dtCookie=D3845DF09C18E5F80F291479A07643AB|X2RlZmF1bHR8MQ 200",
             "tags" => [],
       "@timestamp" => 2017-12-13T02:22:38.513Z,
           "locaIp" => "10.4.11.62",
         "username" => "company.com\\tuanna20",
       "http.agent" => "Mac+OS+X/10.10.5+(14F2511);+ExchangeWebServices/5.0+(213);+Mail/8.2+(2104)"
}
{
           "source" => "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex171205.log",
             "type" => "iis-access",
       "http.value" => "-",
         "hostname" => "HCAS01-SRV",
         "beattype" => "doc",
         "clientip" => "59.153.244.39",
         "@version" => "1",
             "beat" => {
            "name" => "HCAS01-srv",
        "hostname" => "HCAS01-srv",
         "version" => "6.0.1"
    },
             "host" => "HCAS01-srv",
    "http.response" => "200",
      "http.method" => "POST",
       "user_agent" => {
           "name" => "Mobile Safari UI/WKWebView",
        "os_name" => "Other",
             "os" => "Other",
         "device" => "Other",
          "build" => ""
    },
        "timestamp" => "2017-12-05 06:38:26",
     "http.request" => "/Microsoft-Server-ActiveSync/default.eas",
            "geoip" => {
             "city_name" => "Hanoi",
              "timezone" => "Asia/Ho_Chi_Minh",
                    "ip" => "59.153.244.39",
              "latitude" => 21.0333,
          "country_name" => "Vietnam",
         "country_code2" => "VN",
        "continent_code" => "AS",
         "country_code3" => "VN",
           "region_name" => "Thanh Pho Ha Noi",
              "location" => {
            "lon" => 105.85,
            "lat" => 21.0333
        },
           "region_code" => "HN",
             "longitude" => 105.85
    },
           "offset" => 1181500150,
        "http.port" => "443",
         "beatname" => "fb-fim-mail",
        "http.site" => "W3SVC1",
       "prospector" => {
        "type" => "log"
    },
          "message" => "2017-12-05 06:38:26 W3SVC1 HCAS01-SRV 10.4.11.62 POST /Microsoft-Server-ActiveSync/default.eas User=lucqv@fpt.com.vn&DeviceId=341IARRU2961T2Q5G4H7GMCH5O&DeviceType=iPhone&Cmd=Sync&Log=V141_Fc1_Fid:4_Ty:Em_Filt5_St:S_Sk:2047955523_Sst5_SsCmt5_Cli:0a0c0d1f0e_BR1_BPR0_LdapC1_LdapL15_RpcC20_RpcL59_Pk820662046_S1_As:AllowedG_Mbx:MB-02-SRV.HO.FPT.VN_Throttle0_Budget:(A)Conn%3a0%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f0%25%2cCAS%3a%24null%2f%24null%2f1%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f1%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5F56c53095-e0c2-4ad3-a65a-c54c057050bf%2cNorm_ 443 lucqv@company.com 59.153.244.39 HTTP/1.1 Apple-iPhone7C2/1502.202 - 200",
             "tags" => [],
       "@timestamp" => 2017-12-13T02:22:38.515Z,
           "locaIp" => "10.4.11.62",
         "username" => "lucqv@company.com",
       "http.agent" => "Apple-iPhone7C2/1502.202"
}
9

Could you do the same with a json codec so we can try to index it manually and see why this is failing?

10

It looks like geoip is an empty object, which is causing problems. What does your full configuration look like?

11

Hi @dadoonet, some log with a json codec

https://pastebin.com/HqsG99ja

For IIS access (type=iis-access)

filter {
if [type] == "iis-access" {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:agent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:bytes}" }
    }
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:http.site} %{NOTSPACE:hostname} %{IPORHOST:locaIp} %{WORD:http.method} %{URIPATH:http.request} %{NOTSPACE:http.msg} %{INT:http.port} %{DATA:username} %{IPORHOST:clientip} %{DATA} %{DATA:http.agent} %{DATA:http.value} %{NUMBER:http.response}" }
    }
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:http.method} %{URIPATH:http.request} %{NOTSPACE:http.msg} %{DATA:username} %{IPORHOST:clientip} %{DATA:http.agent} %{DATA:http.value} %{NUMBER:http.response}" }
    }
    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLite2-City.mmdb"
    }
    useragent {
      source=> "agent"
      target=> "user_agent"
    }
    useragent {
      source=> "http.agent"
      target=> "user_agent"
    }
    translate {
      regex => true
      dictionary_path => "/etc/logstash/translates/internal-ip.yaml"
      field => "clientip"    
    }
    json {
      source => "translation"
      remove_field => ["translation"]
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
    mutate {
        remove_tag => ["beats_input_codec_plain_applied","_grokparsefailure","_geoip_lookup_failure","_dateparsefailure"]
	remove_field => ["log_timestamp","http.msg"]
    }
  }
}

With public IP, im using GeoIP filter, with internal IP, im using transale with content look like: internal-ip.yaml

'10.1.11.164': '{"geoip": {"unit": "FESFB01", "unit_desc": "FESFB01", "city_name": "Hanoi", "country_name": "Vietnam", "latitude": 21.033, "longitude": 105.85, "location": [105.85, 21.033]}}'
'10.1.11.165': '{"geoip": {"unit": "FESFB02", "unit_desc": "FESFB02", "city_name": "Hanoi", "country_name": "Vietnam", "latitude": 21.033, "longitude": 105.85, "location": [105.85, 21.033]}}'

I think there is a problem with my transate file? but in version 5.x, it worked without problem.

12

Yeah, im still waiting for you.
Because , i can't get log with log problem. :frowning:

13

I would recommend configuring a stdout output with a rubydebug codec, so we can clearly see the structure of the event as it leaves Logstash.

14

@Christian_Dahlqvist
I sent logs output with rubydebug and json codec in previous post, check that pls.

Update: I try to disable translate plugin, but still got error with geoip.location index

15

Have you tried using conditionals to select either group plugin or translate plugin depending on what the IP address starts with? Do you have a matching entry in your translate file?

16

Some case i have, with internal IP, i'll use translate file. But some case, i dont use filter for that.
But this is example of translate file

'10.1.11.164': '{"geoip": {"unit": "FESFB01", "unit_desc": "FESFB01", "city_name": "Hanoi", "country_name": "Vietnam", "latitude": 21.033, "longitude": 105.85, "location": [105.85, 21.033]}}'
'10.1.11.165': '{"geoip": {"unit": "FESFB02", "unit_desc": "FESFB02", "city_name": "Hanoi", "country_name": "Vietnam", "latitude": 21.033, "longitude": 105.85, "location": [105.85, 21.033]}}'
17

Solved!

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.