# Extract timestamp from the logline **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 14, 2019, 10:36pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277 "2019-08-14T22:36:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![justin1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin1/32/45428_2.png) [@justin1](https://discuss.elastic.co/u/justin1)\ **Post date:** [August 14, 2019, 10:36pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/1 "2019-08-14T22:36:20Z")
I am trying to index log files to Elastic search. All the log entries are being indexed into a field named _message_. @timestamp field shows the time the entry was indexed and not the timestamp from log entry. I created a ingest pipeline with grok processor to define the pattern of the log entry. I have tried several patterns and am unable to get this working, particularly because i am new to grok. All i want is the ability to extract the timestamp from the log message and everything else can be ignored or wildcarded or stored in just one variable like _message_ Any help would be appreciated **Log sample** 2019-08-05 00:04:06 error [error.js]: No authorization token was found 2019-08-05 00:04:06 info [index.js]: Request: HTTP GET / 2019-08-05 00:04:06 error [error.js]: No authorization token was found **Ingest pipeline with grok & date processor** { "description" : "Extracting date from log line" , "processors": [ { "grok": { "field": "message", "patterns": ["%{yyyy-mm-dd HH:mm:ss:logtime} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"] }, "date": { "field": "logtime", "target\_field": "@timestamp", "formats": ["yyyy-mm-dd HH:mm:ss"] } } ] } ---
**Author:** ![michaelberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelberg/32/51063_2.png) [@michaelberg](https://discuss.elastic.co/u/michaelberg)\ **Post date:** [August 15, 2019, 5:29am UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/2 "2019-08-15T05:29:41Z")
Justin ... I ran into something similar trying to get JSON output from TShark into ElasticSearch .... for the life of me I couldn't get the timestamp to pull from the JSON log and translate properly ... I found this article extremely helpful ... [https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#\_typeless\_apis\_in\_7\_0](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_typeless_apis_in_7_0) Here's what I did step by step to make the timestamp show correctly ... 1. From the Kibana console I created the Elasticsearch index first ... 2. From the console I then added a document to the new index using the "PUT" command and tagging the end with ?pipeline=\ 3. Tested the display of the timestamp using Kibana ... I'm by no means at all an Elasticsearch/Kibana guru but I wonder if perhaps what you're missing is the creation of the index first and properly defining the "timestamp" field in the index prior to ingesting ... I ran into all sorts of issues until I created the index first ... Some possible food for thought ... cheers! ---
**Author:** ![justin1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin1/32/45428_2.png) [@justin1](https://discuss.elastic.co/u/justin1)\ **Post date:** [August 15, 2019, 5:56pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/3 "2019-08-15T17:56:05Z")
Hi @michaelberg Thanks for taking the time to respond. Yes i did try creating an index before ingesting any data to it, the mapping is created etc but the enter log line is indexed into a filed named _message_. That is when i realized the filebeat cannot really process or parse the data. Do not want to use logstash for such a simple task. I created ingest pipeline with grok and date processor to possibly just extract the timestamp and leave the rest of the log message in the _message_ field. I am using the \_simulate pipeline API which is a great feature to test how the data is getting ingested and investigate and fix any issues. ``` post _ingest/pipeline/redate/_simulate { "docs":[ { "_source":{ "message":"2019-08-04 12:02:39 info [index.js]: Request: HTTP GET /" } } ] } ``` I get this below error > "type": "exception", > "reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [2019-08-04 12:02:39 info [index.js]: Request: HTTP GET /]", Clearly indicating that the grok pattern i am trying to use is not correct ---
**Author:** ![michaelberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelberg/32/51063_2.png) [@michaelberg](https://discuss.elastic.co/u/michaelberg)\ **Post date:** [August 15, 2019, 10:18pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/4 "2019-08-15T22:18:39Z")
Just curious ... have you tried using the GROK Debugger in Kibana to test the GROK Pattern against a sample line from the log? ---
**Author:** ![justin1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin1/32/45428_2.png) [@justin1](https://discuss.elastic.co/u/justin1)\ **Post date:** [August 15, 2019, 10:55pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/5 "2019-08-15T22:55:05Z")
Yes i did, found the grok pattern that actually works finally! > PUT \_ingest/pipeline/redate > { > "description" : "Extracting date from log line" > , "processors": [ > { > "grok": { > "field": "message", > "patterns": ["%{TIMESTAMP\_ISO8601:logtime} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"] > }, > "date": { > "field": "logtime", > "target\_field": "@timestamp", > "formats": ["yyyy-mm-dd HH:mm:ss","ISO8601"] > } > } > ] > } Now i am able to simulate different log messages and get desired result but filebeat is unable to recognize the ingest pipeline. I am getting this error > ERROR pipeline/output.go:121 Failed to publish events: temporary bulk send failure In the filebeat elasticsearch output configuration, i have the following config > output.elasticsearch: >     hosts: ["host"] >     index: "index-name-%{+yyyy.MM}" >     pipeline: "redate" If i remove the pipeline from the config then the logs are getting indexed, if pipeline entry is added to filebeat config then i get those errors. Any thoughts ---
**Author:** ![justin1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin1/32/45428_2.png) [@justin1](https://discuss.elastic.co/u/justin1)\ **Post date:** [August 15, 2019, 11:16pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/6 "2019-08-15T23:16:39Z")
I made this below change and the log messages are getting indexed. Although i do not understand how, appreciate if someone can shed some light on it I had the pipeline: "pipelinename" setting in _Elasticsearch output section_ of the filebeat config file. I moved that line to _filebeat inputs section_ right under file path section, like so > filebeat.inputs: > -type: log > paths: > - D:\home\site\wwwroot\logs\*.log > pipeline: "redate" And the log messages are getting indexed now. ---
**Author:** ![michaelberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michaelberg/32/51063_2.png) [@michaelberg](https://discuss.elastic.co/u/michaelberg)\ **Post date:** [August 15, 2019, 11:52pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/7 "2019-08-15T23:52:37Z")
Trying taking the quotes off the name of the pipeline in the Config file and retry ... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 12, 2019, 11:52pm UTC](https://discuss.elastic.co/t/extract-timestamp-from-the-logline/195277/8 "2019-09-12T23:52:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.