Filebeat WARN 400 please help to decipher

Elastic Stack Beats
1

Can Somoene please help me understand the message below? What I did is convert the puppetlogs go json format, in hope to avoid parsing and send it directly to elastic search instead of sending to logstash.

I do see the results in elastic, but I'm not sure if I'm seeing all results because of message below, plus it generates about 20 of these per second.

2019-06-27T18:00:43.571Z WARN elasticsearch/client.go:527 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf3d5ef2a10ed73f, ext:290519073867, loc:(*time.Location)(0x2bcc360)}, Meta:common.MapStr(nil), Fields:common.MapStr{"@version":1, "agent":common.MapStr{"ephemeral_id":"e76d148b-b1df-4c8b-81ea-007cb14fd455", "hostname":"myhostnamehere", "id":"5f24a946-e052-4dda-9071-2d1abcb5c0d8", "type":"filebeat", "version":"7.2.0"}, "auth":"-", "bytes":305, "clientip":"10.192.14.160", "ecs":common.MapStr{"version":"1.0.0"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"myhostnamehere", "id":"9577872c547c4c349d7f4dc18016252e", "name":"myhostnamehere", "os":common.MapStr{"codename":"Core", "family":"redhat", "kernel":"3.10.0-229.el7.x86_64", "name":"CentOS Linux", "platform":"centos", "version":"7 (Core)"}}, "input":common.MapStr{"type":"log"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/log/puppetlabs/puppetserver/puppetserver-access.log.json"}, "offset":128218614}, "puppet.client-cert":"-", "puppet.client-dn":"-", "puppet.client-verify":"-", "rawrequest":"GET /puppet/v3/file_metadata/modules/node_hubs/ssh_keys/hubpushprod.authorized_keys?environment=test&links=manage&checksum_type=md5&source_permissions=ignore HTTP/1.1", "referrer":"-", "request":"myhostnamehere:8140/puppet/v3/file_metadata/modules/node_hubs/ssh_keys/hubpushprod.authorized_keys", "request.accept":"pson, binary", "request.accept-encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "request.connection":"-", "request.host":"myhostnamehere:8140", "requestprotocol":"HTTP/1.1", "response":200, "response.connection":"null", "response.content-length":"null", "response.content-type":"text/pson; charset=ISO-8859-1", "response.server":"null", "total_service_time":18, "verb":"GET"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc0003ea750), Source:"/var/log/puppetlabs/puppetserver/puppetserver-access.log.json", Offset:128219520, Timestamp:time.Time{wall:0xbf3d5eaa054db96d, ext:53432913, loc:(*time.Location)(0x2bcc360)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x12039, Device:0xfd03}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"mapper [request] of different type, current_type [keyword], merged_type [ObjectMapper]"}

2

It looks as though you're trying to send the request field as an object:

e.g.

"response": {
  "accept": "pson, binary",
  "accept-encoding": "gzip;q=1.0,deflate;q=0.6,identity;q=0.3",
  "host": "myhostnamehere:8140"
}

However the index you're trying to write to already has that field mapped as a keyword type.

If you query the API you'll be able to see what Elastic has your fields mapped as:
GET /INDEX_NAME/_mapping

You might be able to use the reindex API to change the existing field type to be what you want (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html).

Other options are to delete the existing index if you don't care about the data (and change the field type in the index template if you're using one) or perhaps write to a new index and create an alias that can read across both (But Kibana mightcomplain about seeing two types for one field, but you'll at least have the documents stored)

3

First of, thank you so much for your response. It's a dedicated index for one server only. I do understand the part of changing the mapping for request filed and reindexing.

I don't care about the data, but I don't have the template outside of default. It's a first beat I installed and trying to figure out how to get it working properly. So forgive me if the questions are naive.

4

If there's no template for the index you should be OK just deleting it. Elasticsearch should auto detect what type it believes the field to be when it first encounters it; and in your case it should now see it as an object.

5

I deleted the index, without restarting filebeat. New index got created immidiatly, but I still see these errors.

Just incase for more context I've used this documenthttps://puppet.com/docs/puppetserver/5.1/config_logging_advanced.html

The only difference is I'm sending the json files directcly to elastic search skipping logstash.

6

Do you see any templates that match your index if you do GET /_template?

7

I think I see the problem.

In your document you have a field called just request which will be a keyword value, but then you also have fields like request.accept, which would be nested like:

request:
  accept: value

That's where your conflicting types are coming from; you'll need to change the names of one of the fields or perhaps remap request to something like request.url

8

Yes, I see a template. I'm assuming it auto created.

9

Thank you so much for your help. Will try that.

10

this is what I have
"request":"http://%header{Host}%requestURI",
"request.host":"%header{Host}",
"request.accept":"%header{Accept}",
"request.accept-encoding":"%header{Accept-Encoding}",
"request.connection":"%header{Connection}",

will try to change the firs one to "request.url":"http://%header{Host}%requestURI",

And keep the others the same.

11

Thank you So MUCH!!! it looks like it fixed this issue!!!!

I have this warning as well:

2019-06-27T23:07:51.043Z WARN elasticsearch/client.go:527 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf3d70f1a9c7aae3, ext:8261498103593, loc:(*time.Location)(0x2bcc360)}, Meta:common.MapStr(nil), Fields:common.MapStr{"@version":1, "agent":common.MapStr{"ephemeral_id":"8059cc0e-5e46-4c79-86fe-3cb16643060f", "hostname":"myhostname", "id":"5f24a946-e052-4dda-9071-2d1abcb5c0d8", "type":"filebeat", "version":"7.2.0"}, "auth":"-", "bytes":288, "clientip":"10.192.14.160", "ecs":common.MapStr{"version":"1.0.0"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"myhostname", "id":"9577872c547c4c349d7f4dc18016252e", "name":"myhostname", "os":common.MapStr{"codename":"Core", "family":"redhat", "kernel":"3.10.0-229.el7.x86_64", "name":"CentOS Linux", "platform":"centos", "version":"7 (Core)"}}, "input":common.MapStr{"type":"log"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/log/puppetlabs/puppetserver/puppetserver-access.log.json"}, "offset":164477908}, "puppet.client-cert":"-", "puppet.client-dn":"-", "puppet.client-verify":"-", "rawrequest":"GET /puppet/v3/file_metadata/modules/splunkforwarder/splunkforwarder.init?environment=test&links=manage&checksum_type=md5&source_permissions=ignore HTTP/1.1", "referrer":"-", "request.accept":"pson, binary", "request.accept-encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "request.connection":"-", "request.host":"myhostname:8140", "request.url":"http://myhostname:8140/puppet/v3/file_metadata/modules/splunkforwarder/splunkforwarder.init", "requestprotocol":"HTTP/1.1", "response":200, "response.connection":"null", "response.content-length":"null", "response.content-type":"text/pson; charset=ISO-8859-1", "response.server":"null", "total_service_time":21, "verb":"GET"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc0005deea0), Source:"/var/log/puppetlabs/puppetserver/puppetserver-access.log.json", Offset:164478798, Timestamp:time.Time{wall:0xbf3d68e04e146ea7, ext:33372451, loc:(*time.Location)(0x2bcc360)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x12039, Device:0xfd03}}, TimeSeries:false}, Flags:0x1} (status=400): {"type":"illegal_argument_exception","reason":"mapper [response] of different type, current_type [long], merged_type [ObjectMapper]"}

I tried using the same method. renaming

                   "response":"#asLong{%statusCode}",
                     to
                    "response.status":"#asLong{%statusCode}"
                    
                    These are the other response that are there. 
                   "response.content-type":"%responseHeader{Content-Type}",
                    "response.content-length":"%responseHeader{Content-Length}",
                    "response.server":"%responseHeader{Server}",
                    "response.connection":"%responseHeader{Connection}"

But in this case it didn't do the trick.

12

Glad to hear it worked.

On your second issue, perhaps the original response field is being duplicated to response.status instead of being renamed, leaving the old one in place? It does look to be the same issue, so I imagine a response field is creeping in somewhere

13

It's all good. On the second one it was an error on my part, I was restarting puppet instead of restarting puppetserver by accident. Again, thank you so much for your help!!!!.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.