# Ingest (from filebeat): How to include date from log file header in each log line?

**URL:** <https://discuss.elastic.co/t/ingest-from-filebeat-how-to-include-date-from-log-file-header-in-each-log-line/180797>\
**Category:** Elasticsearch\
**Created:** [May 13, 2019, 11:32am UTC](https://discuss.elastic.co/t/ingest-from-filebeat-how-to-include-date-from-log-file-header-in-each-log-line/180797 "2019-05-13T11:32:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sockit](https://avatars.discourse-cdn.com/v4/letter/s/a183cd/32.png) [@sockit](https://discuss.elastic.co/u/sockit)\
**Post date:** [May 13, 2019, 11:32am UTC](https://discuss.elastic.co/t/ingest-from-filebeat-how-to-include-date-from-log-file-header-in-each-log-line/180797/1 "2019-05-13T11:32:39Z")

</div>

I am using filebeat to send data directly to an elasticsearch ingest pipeline.  
The (daily) log files each have a date for the file in a header line at the beginning of the file:

```
` **Application log** Current day: 02/18/19 **`

```

`(18:44:30.379)(04332)Message_type_1: Descriptive text of log message`  
`(18:44:30.399)(04333)Message_type_2: Log msg received (001) 02 30 31 36 37 34 30 38 31 36 33 30 32 30 37 38 38 35 30 35`

When the ingest pipeline is configured as below, I get the msg\_time (as expected) as:

```
`"msg_time" : "23:56:18.974"`

```

I would like to have each msg\_time value include the log file date, like:

```
`"msg_time" : "2019-02-18 23:56:18.974"`

```

How can this be done (preferably without using logstash as an intermediary)?

Thanks in advance.

======  
Ingest Pipeline definition:  
#!/bin/sh  
curl -XPUT "[http://localhost:9200/\_ingest/pipeline/test](http://localhost:9200/_ingest/pipeline/test)" -H 'Content-Type: application/json' -d'  
{  
"description" : "PUT \_ingest/pipeline/test: Convert test log data to indexed data",  
"version" : "1",  
"processors" : [  
{ "grok": {  
"field": "message",  
"patterns": ["\(%{TIME:msg\_time}\)\(%{NUMBER:seqnum}\)%{DATA:msg\_type}:%{GREEDYDATA:msg\_text}"]  
}  
},  
{  
"convert": {  
"field" : "seqnum",  
"type": "integer"  
}  
}  
],  
"on\_failure" : [  
{  
"set" : {  
"field" : "error",  
"value" : "{{ \_ingest.on\_failure\_message }}"  
}  
}  
]  
}'

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 13, 2019, 12:21pm UTC](https://discuss.elastic.co/t/ingest-from-filebeat-how-to-include-date-from-log-file-header-in-each-log-line/180797/2 "2019-05-13T12:21:48Z")

</div>

I don't think you can.

---

<div class="post-metadata">

**Author:** ![sockit](https://avatars.discourse-cdn.com/v4/letter/s/a183cd/32.png) [@sockit](https://discuss.elastic.co/u/sockit)\
**Post date:** [May 13, 2019, 7:22pm UTC](https://discuss.elastic.co/t/ingest-from-filebeat-how-to-include-date-from-log-file-header-in-each-log-line/180797/3 "2019-05-13T19:22:43Z")

</div>

Thanks for the response. Do you mean :

1. it can't be done at all?
2. it can't be done without using logstash? (If so, how to use logstash for this?)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 13, 2019, 9:40pm UTC](https://discuss.elastic.co/t/ingest-from-filebeat-how-to-include-date-from-log-file-header-in-each-log-line/180797/4 "2019-05-13T21:40:00Z")

</div>

Even with logstash it might be hard to do this.  
Because each line is basically a new document without a real "context".

My best guess is to extract the first line "manually" (script shell may be) and start filebeat to ingest that file (skip the first line) and add as a variable may be the content that you extracted manually.

I don't know if it's doable but you could ask in #beats:filebeat for help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 10, 2019, 9:40pm UTC](https://discuss.elastic.co/t/ingest-from-filebeat-how-to-include-date-from-log-file-header-in-each-log-line/180797/5 "2019-06-10T21:40:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.