Filebeat collect log and output to es but occur time parse error

lucasyu · October 21, 2020, 10:55am

Dear all

I use filebeat to collect log and parse to json output to es

but the "timestamp" field can not parse

the value of timestamp is "2020-10-21 17:10:51.963"

error description is "failed to parse date field [2020-10-21 17:10:51.963] with format [strict_date_optional||epoch_millis]"

and these is my filebeat.yaml

what should i do to solve this problem ..........


filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/filebeat/logs/*.log
  json.key_under_root: true
  json.overwrite_keys: true
  json.message_keys: log

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "indexname-*"
setup.template.pattern: "indexname-*"

output.elasticseartch:
.......

kvch · October 21, 2020, 12:57pm

This seems like an error in Elasticsearch, not in Filebeat. Could you please share the whole error message you are getting? Also, the mapping of the index you are sending to would be useful as well.

lucasyu · October 22, 2020, 3:16am

Hi

i setup filebeat and es in docker , these is the log i collect from docker

Filebeat (part of)

2020-10-22T01:18:35.257Z	WARN	elasticsearch/client.go:517	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfdc555a85734d7b, ext:860191013075732, loc:(*time.Location)(0x50026a0)}, Meta:null, 
Fields:{
"agent":{"ephemeral_id":"dfc370f3-ade6-4d6c-b683-44ab96619b8b","hostname":"0894df2db561","id":"c792b8c0-6f63-4437-a382-5085132cf894","type":"filebeat","version":"7.6.1"},
"ecs":{"version":"1.4.0"},
"host":{"name":"0894df2db561"},
"input":{"type":"log"},
"json":{"@timestamp":"2020-10-22 09:18:33.940","class":"************","level":"INFO","log":"","logger_name":"***********","message":"************","sessionId":"***********","stackTrace":""},
"log":{"file":{"path":"/usr/share/filebeat/logs/***********.log"},"offset":36492124}
}, 
Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc00093b1e0), 
Source:"/usr/share/filebeat/logs/***********..log", 
Offset:36492428, 
Timestamp:time.Time{wall:0xbfdc55320b45949b, ext:860029115262547, loc:(*time.Location)(0x50026a0)}, 
TTL:-1, 
Type:"log", 
Meta:map[string]string(nil), 
FileStateOS:file.StateOS{Inode:0x4de18c8, Device:0x57}}, 
TimeSeries:false}, 

Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): 

{"type":"mapper_parsing_exception","reason":"failed to parse field [json.@timestamp] of type [date] in document with id 'LUDiTXUBsI3miD783rnv'. Preview of field's value: '2020-10-22 09:18:33.940'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-10-22 09:18:33.940] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"Failed to parse with all enclosed parsers"}}}

lucasyu · October 22, 2020, 3:17am

ES (part of)


{"type": "server", "timestamp": "2020-10-22T01:18:35,254Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "docker-cluster", "node.name": "ee837b9fa50e", "message": "[index-2020.10.22][0] failed to execute bulk item (create) index {[index-2020.10.22][_doc][LUDiTXUBsI3miD783rnv], source[{\"@timestamp\":\"2020-10-22T01:18:34.091Z\",\"ecs\":{\"version\":\"1.4.0\"},\"host\":{\"name\":\"0894df2db561\"},\"agent\":{\"hostname\":\"0894df2db561\",\"id\":\"c792b8c0-6f63-4437-a382-5085132cf894\",\"version\":\"7.6.1\",\"type\":\"filebeat\",\"ephemeral_id\":\"dfc370f3-ade6-4d6c-b683-44ab96619b8b\"},\"log\":{\"offset\":36492124,\"file\":{\"path\":\"/usr/share/filebeat/logs/****.log\"}},\"json\":{\"sessionId\":\"**\",\"log\":\"\",\"@timestamp\":\"2020-10-22 09:18:33.940\",\"level\":\"INFO\",\"class\":\"***\",\"message\":\"*****\",\"logger_name\":\"***\",\"stackTrace\":\"\"},\"input\":{\"type\":\"log\"}}]}", "cluster.uuid": "_kKr5lLWQuWT37ezytSIAw", "node.id": "Zvlo5K8_Rc-uHNCdaHMItg" ,

"stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [json.@timestamp] of type [date] in document with id 'LUDiTXUBsI3miD783rnv'. Preview of field's value: '2020-10-22 09:18:33.940'",

"at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:306) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:488) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:614) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:427) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:505) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:418) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:793) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:770) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:202) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:81) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:895) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:109) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:374) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:297) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$24(IndexShard.java:2791) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2765) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:836) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:293) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:256) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:750) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]",

"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",

"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",

"at java.lang.Thread.run(Thread.java:830) [?:?]",

"Caused by: java.lang.IllegalArgumentException: failed to parse date field [2020-10-22 09:18:33.940] with format [strict_date_optional_time||epoch_millis]",

"at org.elasticsearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:169) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:356) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:584) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:284) ~[elasticsearch-7.6.1.jar:7.6.1]",

"... 41 more",

"Caused by: java.time.format.DateTimeParseException: Failed to parse with all enclosed parsers",

"at org.elasticsearch.common.time.JavaDateFormatter.doParse(JavaDateFormatter.java:196) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.common.time.JavaDateFormatter.parse(JavaDateFormatter.java:167) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parse(DateFieldMapper.java:356) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:584) ~[elasticsearch-7.6.1.jar:7.6.1]",

"at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:284) ~[elasticsearch-7.6.1.jar:7.6.1]",

"... 41 more"] }

lucasyu · October 22, 2020, 3:18am

about this part , i am new guy use es , i just follow the guide in internet to use filebeat collect log and output log into es . i am not sure about these text is "ES-mapping" or not ....and i did not remember i create es-mapping or something else.....

ES-mapping


{
    "mapping": {
        "_doc": {
            "properties": {
                "@timestamp": {
                    "type": "date"
                },
                "agent": {
                    "properties": {
                        "ephemeral_id": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "hostname": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "id": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "type": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "version": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                },
                "ecs": {
                    "properties": {
                        "version": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                },
                "host": {
                    "properties": {
                        "name": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                },
                "input": {
                    "properties": {
                        "type": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                },
                "json": {
                    "properties": {
                        "@timestamp": {
                            "type": "date"
                        },
                        "@version": {
                            "type": "long"
                        },
                        "class": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "exception": {
                            "properties": {
                                "exception_class": {
                                    "type": "text",
                                    "fields": {
                                        "keyword": {
                                            "type": "keyword",
                                            "ignore_above": 256
                                        }
                                    }
                                },
                                "exception_message": {
                                    "type": "text",
                                    "fields": {
                                        "keyword": {
                                            "type": "keyword",
                                            "ignore_above": 256
                                        }
                                    }
                                },
                                "stacktrace": {
                                    "type": "text",
                                    "fields": {
                                        "keyword": {
                                            "type": "keyword",
                                            "ignore_above": 256
                                        }
                                    }
                                }
                            }
                        },
                        "file": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "level": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "line_number": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "log": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "logger_name": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "mdc": {
                            "type": "object"
                        },
                        "message": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "method": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "sessionId": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "source_host": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "thread_name": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "ignore_above": 256
                                }
                            }
                        }
                    }
                },
                "log": {
                    "properties": {
                        "file": {
                            "properties": {
                                "path": {
                                    "type": "text",
                                    "fields": {
                                        "keyword": {
                                            "type": "keyword",
                                            "ignore_above": 256
                                        }
                                    }
                                }
                            }
                        },
                        "offset": {
                            "type": "long"
                        }
                    }
                },
                "message": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                }
            }
        }
    }
}

lucasyu · October 22, 2020, 4:00am

i use kibana devtools and input

GET _mapping

i can get the lots of item
such as

{
 "indexname-2020.10.16" : {......},
 "indexname-2020.10.19" : {......},
 ".kibana_task_manager_1" : {...},
 "indexname-2020.10.20" : {......},
 "indexname-2020.10.21" : {......},
 ".kibana_1" : {....},
 ".apm-agent-configuration" : {....},
 "indexname-2020.10.15" : {...}
}

i thinks these "indexname-XXXX.XX.XX" mapping is created by filebeat.....

and if i want to fix these timestamp parse error , i must edit the mapping ....

so does it mean i should edit all of these "indexname-XXXX.XX.XX" mapping ?

lucasyu · October 23, 2020, 1:38am

@kvch could you give me some advice

system · November 20, 2020, 3:38am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.