Metricbeat error unable to send metrics on elastic

Hi Team,

kindly help I am not able to export metrics on Elasticsearch. Getting this error.

WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbe3cd54bab923e, ext:48740163, loc:(*time.Location)(0x3a47960)}, Meta:common.MapStr(nil), Fields:common.MapStr{"system":common.MapStr{"fsstat":common.MapStr{"total_size":common.MapStr{"used":0x3c46eb000, "total":0x2a7acf1000, "free":0x26b6606000}, "count":7, "total_files":0xa0eac00}}, "tags":[]string{"api"}, "host":common.MapStr{"architecture":"x86_64", "os":common.MapStr{"family":"redhat", "name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)"}, "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059"}, "beat":common.MapStr{"version":"6.8.2", "name":"172.24.30.59", "hostname":"dmpblnk3059"}, "metricset":common.MapStr{"name":"fsstat", "module":"system", "rtt":185}, "event":common.MapStr{"dataset":"system.fsstat", "duration":185253}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.2-2020.07.22] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-22T23:18:09.285+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbe3cd54bad43f5, ext:48851207, loc:(*time.Location)(0x3a47960)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"rtt":9, "name":"uptime", "module":"system"}, "event":common.MapStr{"dataset":"system.uptime", "duration":9651}, "tags":[]string{"api"}, "host":common.MapStr{"architecture":"x86_64", "os":common.MapStr{"platform":"centos", "version":"7 (Core)", "family":"redhat", "name":"CentOS Linux", "codename":"Core"}, "id":"df2686d0a97c487385d98baaf801134e", "name":"dmpblnk3059", "containerized":false}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.2"}, "system":common.MapStr{"uptime":common.MapStr{"duration":common.MapStr{"ms":72168527000}}}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.2-2020.07.22] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-22T23:18:09.285+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbe3cd54bad3ad1, ext:48848850, loc:(*time.Location)(0x3a47960)}, Meta:common.MapStr(nil), Fields:common.MapStr{"event":common.MapStr{"dataset":"system.filesystem", "duration":215341}, "system":common.MapStr{"filesystem":common.MapStr{"free":0x18facd9000, "available":0x18facd9000, "free_files":0x63ffbfd, "used":common.MapStr{"pct":0.0003, "bytes":0x2028000}, "device_name":"/dev/sdb1", "files":0x63ffc00, "type":"xfs", "total":0x18fcd01000, "mount_point":"/data"}}, "tags":[]string{"api"}, "beat":common.MapStr{"hostname":"dmpblnk3059", "version":"6.8.2", "name":"172.24.30.59"}, "host":common.MapStr{"id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059", "architecture":"x86_64", "os":common.MapStr{"codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat", "name":"CentOS Linux"}}, "metricset":common.MapStr{"module":"system", "rtt":215, "name":"filesystem"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.2-2020.07.22] as the final mapping would have more than 1 type: [doc, aggregation]"}

Metricbeat Version is 6.8.2

waiting for response.

That would be why.
See https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html for more.

Please note that we don't provide any assurance of SLA or resolution here. We try our best to answer things, but given you asked this right before a weekend, you should have some patience :slight_smile:

Thanks warkolm:

got your point. can you easily suggest what changes require to get out of this error.

"2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088df6b0251, ext:47870769106, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"name":"dmpblnk3059", "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "architecture":"x86_64", "os":common.MapStr{"name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat"}}, "metricset":common.MapStr{"module":"system", "rtt":373, "name":"cpu"}, "event":common.MapStr{"duration":373846, "dataset":"system.cpu"}, "system":common.MapStr{"cpu":common.MapStr{"nice":common.MapStr{"pct":0}, "steal":common.MapStr{"pct":0}, "softirq":common.MapStr{"pct":0.001}, "system":common.MapStr{"pct":0.049}, "idle":common.MapStr{"pct":15.8989}, "iowait":common.MapStr{"pct":0}, "cores":16, "user":common.MapStr{"pct":0.051}, "irq":common.MapStr{"pct":0}, "total":common.MapStr{"pct":0.1011}}}, "tags":[]string{"api"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088df9d3b9b, ext:47874060471, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"tags":[]string{"api"}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"name":"dmpblnk3059", "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "architecture":"x86_64", "os":common.MapStr{"version":"7 (Core)", "family":"redhat", "name":"CentOS Linux", "codename":"Core", "platform":"centos"}}, "metricset":common.MapStr{"rtt":1890, "name":"memory", "module":"system"}, "event":common.MapStr{"dataset":"system.memory", "duration":1890522}, "system":common.MapStr{"memory":common.MapStr{"free":0x9301e8000, "actual":common.MapStr{"free":0x9934ea000, "used":common.MapStr{"pct":0.3948, "bytes":0x63ec49000}}, "swap":common.MapStr{"total":0xecfff000, "used":common.MapStr{"pct":0, "bytes":0x0}, "free":0xecfff000}, "hugepages":common.MapStr{"surplus":0x0, "default_size":0x200000, "total":0x0, "used":common.MapStr{"bytes":0x0, "pct":0}, "free":0x0, "reserved":0x0}, "total":0xfd2133000, "used":common.MapStr{"bytes":0x6a1f4b000, "pct":0.4192}}}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}"

:common.MapStr{"soft":0xfde8, "hard":0xfde8}}, "pgid":1842, "state":"sleeping", "cpu":common.MapStr{"total":common.MapStr{"value":930, "pct":0.021, "norm":common.MapStr{"pct":0.0013}}, "start_time":common.Time{wall:0x0, ext:63731450226, loc:(*time.Location)(nil)}}, "ppid":126449, "cmdline":"/usr/share/metricbeat/bin/metricbeat -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat -e --setup", "pid":1842}}, "metricset":common.MapStr{"name":"process", "module":"system", "rtt":86025}, "tags":[]string{"api"}, "host":common.MapStr{"architecture":"x86_64", "os":common.MapStr{"name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat"}, "name":"dmpblnk3059", "id":"df2686d0a97c487385d98baaf801134e", "containerized":false}, "beat":common.MapStr{"hostname":"dmpblnk3059", "version":"6.8.10", "name":"172.24.30.59"}, "event":common.MapStr{"dataset":"system.process", "duration":86025777}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088e468334c, ext:47954471071, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"event":common.MapStr{"dataset":"system.process", "duration":86060829}, "system":common.MapStr{"process":common.MapStr{"memory":common.MapStr{"size":0x108e000, "rss":common.MapStr{"bytes":0x1ce000, "pct":0}, "share":0xd1000}, "cwd":"/", "cpu":common.MapStr{"total":common.MapStr{"value":1.7279454e+08, "pct":0.016, "norm":common.MapStr{"pct":0.001}}, "start_time":common.Time{wall:0x0, ext:63700674631, loc:(*time.Location)(nil)}}, "pgid":71536, "fd":common.MapStr{"open":0xd, "limit":common.MapStr{"soft":0x400, "hard":0x1000}}, "state":"sleeping", "name":"ossec-syscheckd", "username":"root", "cmdline":"/var/ossec/bin/ossec-syscheckd", "pid":71538, "ppid":1}}, "tags":[]string{"api"}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"os":common.MapStr{"version":"7 (Core)", "family":"redhat", "name":"CentOS Linux", "codename":"Core", "platform":"centos"}, "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059", "architecture":"x86_64"}, "metricset":common.MapStr{"module":"system", "rtt":86060, "name":"process"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}
2020-07-27T18:07:56.599+0530    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfbfd088e468334c, ext:47954471071, loc:(*time.Location)(0x3a49260)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"name":"process", "module":"system", "rtt":86065}, "event":common.MapStr{"dataset":"system.process", "duration":86065255}, "system":common.MapStr{"process":common.MapStr{"state":"sleeping", "memory":common.MapStr{"size":0xf59ef5000, "rss":common.MapStr{"bytes":0x56e92e000, "pct":0.3433}, "share":0xfb3000}, "cwd":"/home/ansible", "fd":common.MapStr{"limit":common.MapStr{"hard":0xfde8, "soft":0xfde8}, "open":0x202}, "cpu":common.MapStr{"start_time":common.Time{wall:0x0, ext:63731189740, loc:(*time.Location)(nil)}, "total":common.MapStr{"norm":common.MapStr{"pct":0.0009}, "value":8.87279e+06, "pct":0.014}}, "ppid":1, "pgid":12234, "name":"java", "cmdline":"java -Xmx50g -server -XX:+UseG1GC -XX:MaxDirectMemorySize=5g -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Xloggc:/opt/colombiaapi-server/log/gc.log -Dserver.port=80 -Dserver.tomcat.max-threads=1200 -Dspring.profiles.active=prod -Djava.io.tmpdir=/opt/colombiaapi-server/tmp -javaagent:/opt/newrelic/newrelic.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8008 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.password.file=/opt/jmx/jmxremote.password -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.24.30.59 -Dendpoint=172.24.30.59:81 -jar /opt/colombiaapi-server/jar/colombia-api-server-0.16.0.war", "pid":12261, "username":"root"}}, "tags":[]string{"api"}, "beat":common.MapStr{"name":"172.24.30.59", "hostname":"dmpblnk3059", "version":"6.8.10"}, "host":common.MapStr{"os":common.MapStr{"name":"CentOS Linux", "codename":"Core", "platform":"centos", "version":"7 (Core)", "family":"redhat"}, "id":"df2686d0a97c487385d98baaf801134e", "containerized":false, "name":"dmpblnk3059", "architecture":"x86_64"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.8.10-2020.07.27] as the final mapping would have more than 1 type: [doc, aggregation]"}

Is it a fresh cluster or did you put there some data and mappings before? any version update?

there was some app data. system metrics are now being pushed.

it will be great if you can help with debugging

I suspect that you may using older version that we do not support anymore.

There is an article on removal of mappings: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.