Failed to parse [kubernetes.container.start_time]

Elastic Stack Beats
1

Hello,

There seems to be a problem with the mapping of kubernetes.container.start_time. I find the following errors in my Elastic cluster logs every minute:

Elasticsearch cluster log error:

[2018-07-25T16:06:44,953][DEBUG][o.e.a.b.TransportShardBulkAction] [metricbeat-6.2.4-2018.07.25][0] failed to execute bulk item (index) BulkShardRequest [[metricbeat-6.2.4-2018.07.25][0]] containing [17] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [kubernetes.container.start_time]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:407) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:500) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:482) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:500) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:394) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
        at 
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.IllegalArgumentException: Invalid format: ""
        at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[joda-time-2.9.9.jar:2.9.9]
        at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:826) ~[joda-time-2.9.9.jar:2.9.9]
        at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:248) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:456) ~[elasticsearch-6.2.4.jar:6.2.4]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:297) ~[elasticsearch-6.2.4.jar:6.2.4]
        ... 65 more

This is related to Kubernetes containers which are in error or didn't start yet...

Not sure what the solution for this problem is? Maybe Metricbeat should check if kubernetes.container.start_time is "" and enter a valid date if not?

Grtz

Willem

2

Hi @willemdh,

Looks like you have indeed uncovered a bug here. Thank you!

We track bugs in our GitHub repository at https://github.com/elastic/beats/issues. If you have a GitHub account, I encourage you to create an issue in our repository to report this bug. That way you can stay on top of progress made on the issue as well.

If you do not have a GitHub account, just let me know here and I'll create the issue for you.

Thanks,

Shaunak

3

Created failed to parse [kubernetes.container.start_time] · Issue #7736 · elastic/beats · GitHub

Grtz

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.