Metricbeat error unable to send metrics on elastic

Elastic Stack Beats
1

Hi Team,

kindly help I am not able to export metrics on Elasticsearch. Getting this error.

WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbe3cd54bab923e, ext:48740163, loc:(*time.Location)(0x3a47960)}, Meta:common.MapStr(nil), Fields:common.MapStr{"system":common.MapStr{"fsstat":common.MapStr{"total_size":common.MapStr{"used":0x3c46eb000, "total":0x2a7acf1000, "free":0x26b6606000}, "count":7, "total_files":0xa0eac00}}, "tags":[]string{"api"}, "host":common.MapStr{"architecture":"x86_64", "os":common.MapStr{"family":"redhat", "name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)"}, "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059"}, "beat":common.MapStr{"version":"6.8.2", "name":"172.24.30.59", "hostname":"dmpblnk3059"}, "metricset":common.MapStr{"name":"fsstat", "module":"system", "rtt":185}, "event":common.MapStr{"dataset":"system.fsstat", "duration":185253}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.2-2020.07.22] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-22T23:18:09.285+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbe3cd54bad43f5, ext:48851207, loc:(*time.Location)(0x3a47960)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"rtt":9, "name":"uptime", "module":"system"}, "event":common.MapStr{"dataset":"system.uptime", "duration":9651}, "tags":[]string{"api"}, "host":common.MapStr{"architecture":"x86_64", "os":common.MapStr{"platform":"centos", "version":"7 (Core)", "family":"redhat", "name":"CentOS Linux", "codename":"Core"}, "id":"df2686d0a97c487385d98baaf801134e", "name":"dmpblnk3059", "containerized":false}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.2"}, "system":common.MapStr{"uptime":common.MapStr{"duration":common.MapStr{"ms":72168527000}}}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.2-2020.07.22] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-22T23:18:09.285+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbe3cd54bad3ad1, ext:48848850, loc:(*time.Location)(0x3a47960)}, Meta:common.MapStr(nil), Fields:common.MapStr{"event":common.MapStr{"dataset":"system.filesystem", "duration":215341}, "system":common.MapStr{"filesystem":common.MapStr{"free":0x18facd9000, "available":0x18facd9000, "free_files":0x63ffbfd, "used":common.MapStr{"pct":0.0003, "bytes":0x2028000}, "device_name":"/dev/sdb1", "files":0x63ffc00, "type":"xfs", "total":0x18fcd01000, "mount_point":"/data"}}, "tags":[]string{"api"}, "beat":common.MapStr{"hostname":"dmpblnk3059", "version":"6.8.2", "name":"172.24.30.59"}, "host":common.MapStr{"id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059", "architecture":"x86_64", "os":common.MapStr{"codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat", "name":"CentOS Linux"}}, "metricset":common.MapStr{"module":"system", "rtt":215, "name":"filesystem"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.2-2020.07.22] as the final mapping would have more than 1 type: [doc, aggregation]"}
2

Metricbeat Version is 6.8.2

3

waiting for response.

4

That would be why.
See Removal of mapping types | Elasticsearch Guide [8.11] | Elastic for more.

Please note that we don't provide any assurance of SLA or resolution here. We try our best to answer things, but given you asked this right before a weekend, you should have some patience :slight_smile:

5

Thanks warkolm:

got your point. can you easily suggest what changes require to get out of this error.

6 
"2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088df6b0251, ext:47870769106, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"name":"dmpblnk3059", "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "architecture":"x86_64", "os":common.MapStr{"name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat"}}, "metricset":common.MapStr{"module":"system", "rtt":373, "name":"cpu"}, "event":common.MapStr{"duration":373846, "dataset":"system.cpu"}, "system":common.MapStr{"cpu":common.MapStr{"nice":common.MapStr{"pct":0}, "steal":common.MapStr{"pct":0}, "softirq":common.MapStr{"pct":0.001}, "system":common.MapStr{"pct":0.049}, "idle":common.MapStr{"pct":15.8989}, "iowait":common.MapStr{"pct":0}, "cores":16, "user":common.MapStr{"pct":0.051}, "irq":common.MapStr{"pct":0}, "total":common.MapStr{"pct":0.1011}}}, "tags":[]string{"api"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088df9d3b9b, ext:47874060471, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"tags":[]string{"api"}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"name":"dmpblnk3059", "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "architecture":"x86_64", "os":common.MapStr{"version":"7 (Core)", "family":"redhat", "name":"CentOS Linux", "codename":"Core", "platform":"centos"}}, "metricset":common.MapStr{"rtt":1890, "name":"memory", "module":"system"}, "event":common.MapStr{"dataset":"system.memory", "duration":1890522}, "system":common.MapStr{"memory":common.MapStr{"free":0x9301e8000, "actual":common.MapStr{"free":0x9934ea000, "used":common.MapStr{"pct":0.3948, "bytes":0x63ec49000}}, "swap":common.MapStr{"total":0xecfff000, "used":common.MapStr{"pct":0, "bytes":0x0}, "free":0xecfff000}, "hugepages":common.MapStr{"surplus":0x0, "default_size":0x200000, "total":0x0, "used":common.MapStr{"bytes":0x0, "pct":0}, "free":0x0, "reserved":0x0}, "total":0xfd2133000, "used":common.MapStr{"bytes":0x6a1f4b000, "pct":0.4192}}}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}"

:common.MapStr{"soft":0xfde8, "hard":0xfde8}}, "pgid":1842, "state":"sleeping", "cpu":common.MapStr{"total":common.MapStr{"value":930, "pct":0.021, "norm":common.MapStr{"pct":0.0013}}, "start_time":common.Time{wall:0x0, ext:63731450226, loc:(*time.Location)(nil)}}, "ppid":126449, "cmdline":"/usr/share/metricbeat/bin/metricbeat -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat -e --setup", "pid":1842}}, "metricset":common.MapStr{"name":"process", "module":"system", "rtt":86025}, "tags":[]string{"api"}, "host":common.MapStr{"architecture":"x86_64", "os":common.MapStr{"name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat"}, "name":"dmpblnk3059", "id":"df2686d0a97c487385d98baaf801134e", "containerized":false}, "beat":common.MapStr{"hostname":"dmpblnk3059", "version":"6.8.10", "name":"172.24.30.59"}, "event":common.MapStr{"dataset":"system.process", "duration":86025777}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088e468334c, ext:47954471071, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"event":common.MapStr{"dataset":"system.process", "duration":86060829}, "system":common.MapStr{"process":common.MapStr{"memory":common.MapStr{"size":0x108e000, "rss":common.MapStr{"bytes":0x1ce000, "pct":0}, "share":0xd1000}, "cwd":"/", "cpu":common.MapStr{"total":common.MapStr{"value":1.7279454e+08, "pct":0.016, "norm":common.MapStr{"pct":0.001}}, "start_time":common.Time{wall:0x0, ext:63700674631, loc:(*time.Location)(nil)}}, "pgid":71536, "fd":common.MapStr{"open":0xd, "limit":common.MapStr{"soft":0x400, "hard":0x1000}}, "state":"sleeping", "name":"ossec-syscheckd", "username":"root", "cmdline":"/var/ossec/bin/ossec-syscheckd", "pid":71538, "ppid":1}}, "tags":[]string{"api"}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"os":common.MapStr{"version":"7 (Core)", "family":"redhat", "name":"CentOS Linux", "codename":"Core", "platform":"centos"}, "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059", "architecture":"x86_64"}, "metricset":common.MapStr{"module":"system", "rtt":86060, "name":"process"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088e468334c, ext:47954471071, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"name":"process", "module":"system", "rtt":86065}, "event":common.MapStr{"dataset":"system.process", "duration":86065255}, "system":common.MapStr{"process":common.MapStr{"state":"sleeping", "memory":common.MapStr{"size":0xf59ef5000, "rss":common.MapStr{"bytes":0x56e92e000, "pct":0.3433}, "share":0xfb3000}, "cwd":"/home/ansible", "fd":common.MapStr{"limit":common.MapStr{"hard":0xfde8, "soft":0xfde8}, "open":0x202}, "cpu":common.MapStr{"start_time":common.Time{wall:0x0, ext:63731189740, loc:(*time.Location)(nil)}, "total":common.MapStr{"norm":common.MapStr{"pct":0.0009}, "value":8.87279e+06, "pct":0.014}}, "ppid":1, "pgid":12234, "name":"java", "cmdline":"java -Xmx50g -server -XX:+UseG1GC -XX:MaxDirectMemorySize=5g -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Xloggc:/opt/colombiaapi-server/log/gc.log -Dserver.port=80 -Dserver.tomcat.max-threads=1200 -Dspring.profiles.active=prod -Djava.io.tmpdir=/opt/colombiaapi-server/tmp -javaagent:/opt/newrelic/newrelic.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8008 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.password.file=/opt/jmx/jmxremote.password -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.24.30.59 -Dendpoint=172.24.30.59:81 -jar /opt/colombiaapi-server/jar/colombia-api-server-0.16.0.war", "pid":12261, "username":"root"}}, "tags":[]string{"api"}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"os":common.MapStr{"name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat"}, "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059", "architecture":"x86_64"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}
7

Is it a fresh cluster or did you put there some data and mappings before? any version update?

8

there was some app data. system metrics are now being pushed.

9

it will be great if you can help with debugging

10

I suspect that you may using older version that we do not support anymore.

There is an article on removal of mappings: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.