# Failed to parse \[timestamp\] **URL:** \ **Category:** Logstash\ **Created:** [April 11, 2017, 11:20am UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966 "2017-04-11T11:20:11Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![mirnuj\_atom](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mirnuj\_atom](https://discuss.elastic.co/u/mirnuj_atom)\ **Post date:** [April 11, 2017, 11:20am UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/1 "2017-04-11T11:20:11Z")
Hey Guys Hope you all are doing great! I am having an issue parsing date in tomcat logs. Elasticsearch throws errors in debug and stops accepting new messages. Here is an example of log lines: > 2017-02-27 13:53:14 INFO SAMLDefaultLogger:127 - AuthNRequest;SUCCESS;1111.11.114.239;[https://myhost](https://myhost) And here is an error message seen in elasticsearch.log > [2017-04-11 03:36:17,476][DEBUG][action.bulk] [node-0] [logstash-2017.02.27][1] failed to execute bulk item (index) index {[logstash-2017.02.27][SP][AVtcld6y5SO0hmeim\_Sn], source[{"message":"2017-02-27 13:53:14 INFO SAMLDefaultLogger:127 - AuthNRequest;SUCCESS;111.111.11.19;[https://myhost:443/saml/somedata;;","@version":"1","@timestamp":"2017-02-27T21:53:14.000Z","type":"SP","file":"/myfile/catalina.out","host":"myhost","offset":"17851028","token":"prodLogs","purpose":"app","timestamp":"2017-02-27](https://myhost:443/saml/somedata;;%22,%22@version%22:%221%22,%22@timestamp%22:%222017-02-27T21:53:14.000Z%22,%22type%22:%22SP%22,%22file%22:%22/myfile/catalina.out%22,%22host%22:%22myhost%22,%22offset%22:%2217851028%22,%22token%22:%22prodLogs%22,%22purpose%22:%22app%22,%22timestamp%22:%222017-02-27) 13:53:14","severity":"INFO","className":"SAMLDefaultLogger:127","logmessage":"AuthNRequest;SUCCESS;137.91.114.239;[https://hostm:443/saml/meta;;;](https://hostm:443/saml/meta;;;)"}]} > MapperParsingException[failed to parse [timestamp]]; nested: IllegalArgumentException[Invalid format: "2017-02-27 13:53:14" is malformed at " 13:53:14"]; > at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329) > at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:311) > at org.elasticsearch.index.mapper.DocumentParser.parseAndMergeUpdate(DocumentParser.java:740) > at org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:627) Here is how I parse date in logstash: > ``` > date { > match => ["timestamp", "yyyy-MM-dd HH:mm:ss", "dd-MMM-yyyy HH:mm:ss.SSS"] > locale => "en" > } > > ``` It looks like it parses the date fine but then it just doesn't like the format. Am I missing something obvious? Thanks Andrii ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [April 11, 2017, 7:02pm UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/2 "2017-04-11T19:02:21Z")
Your date filter shows a period between ss.SSS. The log shows a comma ss,SSS. ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [April 11, 2017, 7:04pm UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/3 "2017-04-11T19:04:32Z")
Disregard my previous comment, I was looking at the wrong log. ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [April 11, 2017, 7:15pm UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/4 "2017-04-11T19:15:50Z")
I used your log input and date filter and I didn't get this error message. Please provide the full Logstash configuration file. What version of Logstash and ES are you using? ---
**Author:** ![mirnuj\_atom](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mirnuj\_atom](https://discuss.elastic.co/u/mirnuj_atom)\ **Post date:** [April 12, 2017, 1:58pm UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/5 "2017-04-12T13:58:25Z")
Hey Jimmy Thank you for the response! I am using logstash 1:2.2.2-1 and ES 2.4.4-1 Here is the config: > filter { > if [type] == "SP" { > multiline { > patterns\_dir =\> ["/etc/logstash/patterns/"] > pattern =\> "(^%{MY\_TOMCAT\_DATESTAMP})|(^%{UNKNOWN\_DATESTAMP})" > negate =\> true > what =\> "previous" > } > > # if "\_grokparsefailure" in [tags] { > > # drop { } > > # } > > ``` > grok { > break_on_match => true > patterns_dir => ["/etc/logstash/patterns/"] > match => ["message", "%{MYTOMCATLOG}"] > match => ["message", "%{UNKNOWNLOG}"] > } > date { > match => ["timestamp", "yyyy-MM-dd HH:mm:ss", "dd-MMM-yyyy HH:mm:ss.SSS"] > locale => "en" > } > > ``` > > } > } And here is the patterns I use UNKNOWN\_DATESTAMP %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) MY\_TOMCAT\_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) > MYTOMCATLOG %{MY\_TOMCAT\_DATESTAMP:timestamp} %{LOGLEVEL:severity} +%{NOTSPACE:className} +- +%{JAVALOGMESSAGE:logmessage} > UNKNOWNLOG %{UNKNOWN\_DATESTAMP:timestamp} +%{LOGLEVEL:severity} +%{NOTSPACE:action} +%{NOTSPACE:className} +%{GREEDYDATA:message} > JAVALOGMESSAGE (.\*) There are two kinds of date in my log: 2016-09-15 05:27:54 and 15-Sep-2016 05:27:42.567 When I start harvesting logs it works for some time and then logstash stops accepting new data. ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [April 12, 2017, 8:34pm UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/6 "2017-04-12T20:34:38Z")
What made you aware that Logstash stop accepting new data? You see the log file itself grow in size, but the data wasn't getting ingested into ES? ---
**Author:** ![mirnuj\_atom](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mirnuj\_atom](https://discuss.elastic.co/u/mirnuj_atom)\ **Post date:** [April 12, 2017, 9:09pm UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/7 "2017-04-12T21:09:35Z")
> [@jkuang](#): > > What made you aware that Logstash stop accepting new data? You see the log file itself grow in size, but the data wasn't getting ingested into ES? Logstash-forwarder starts showing messages like the following, and after this no new entries in ES log but there is still data to be parsed, it simply stops working... > 2017/04/12 09:28:20.947736 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt > 2017/04/12 09:28:20.948149 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) > 2017/04/12 09:28:21.011529 Connected to 111.111.111.111 > 2017/04/12 09:28:21.019767 Read error looking for ack: EOF > 2017/04/12 09:28:21.019868 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt > 2017/04/12 09:28:21.020183 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) > 2017/04/12 09:28:36.020538 Failed to tls handshake with 111.111.111.111 read tcp 111.111.111.111:5000: i/o timeout > 2017/04/12 09:28:37.020932 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) > 2017/04/12 09:28:51.335528 Connected to 111.111.111.111 > 2017/04/12 09:28:56.348703 Read error looking for ack: EOF > 2017/04/12 09:28:56.348798 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt > 2017/04/12 09:28:56.349145 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) > 2017/04/12 09:28:56.437510 Connected to 111.111.111.111 > 2017/04/12 09:28:56.443756 Read error looking for ack: EOF > 2017/04/12 09:28:56.443837 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt > 2017/04/12 09:28:56.444092 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) > 2017/04/12 09:29:11.444418 Failed to tls handshake with 111.111.111.111 read tcp 111.111.111.111:5000: i/o timeout > 2017/04/12 09:29:12.444777 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) > 2017/04/12 09:29:26.753509 Connected to 111.111.111.111 > 2017/04/12 09:29:31.763917 Read error looking for ack: EOF > 2017/04/12 09:29:31.764016 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt > 2017/04/12 09:29:31.764346 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) > 2017/04/12 09:29:31.827521 Connected to 111.111.111.111 > 2017/04/12 09:29:31.834425 Read error looking for ack: EOF > 2017/04/12 09:29:31.834501 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt > 2017/04/12 09:29:31.834760 Connecting to [111.111.111.111]:5000 ([myhost.com](http://myhost.com)) I have just installed new ELK 5 (previously I was working with ELK 4\*) stack with the same config and it worked fine, all the data was parsed as it should without any error messages, so I think that I'm fine now. Thank you very much for help! Andrii. ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [April 18, 2017, 10:32pm UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/8 "2017-04-18T22:32:15Z")
I see an error in the log. ```auto Failed to tls handshake with 111.111.111.111 read tcp 111.111.111.111:5000: i/o timeout ``` I would recommend checking your network and disk partitions as the tls handshake failed and i/o timeout. There should be some OS logs that tell you what happened. ---
**Author:** ![mirnuj\_atom](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mirnuj\_atom](https://discuss.elastic.co/u/mirnuj_atom)\ **Post date:** [April 19, 2017, 8:18am UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/9 "2017-04-19T08:18:02Z")
Hi Logstash-forwarder and logstash are on the same server (as well as elasticsearch) so there should be no traffic limitation. It simply stops accepting new messages after the errors about timestamp appear in elasticsearch's log. But looks like the problem gone after upgrade to ELK 5. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 17, 2017, 8:26am UTC](https://discuss.elastic.co/t/failed-to-parse-timestamp/81966/10 "2017-05-17T08:26:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.