# Combination of logfiles with time-only and datetime fields

**URL:** <https://discuss.elastic.co/t/combination-of-logfiles-with-time-only-and-datetime-fields/143048>\
**Category:** Logstash\
**Created:** [August 5, 2018, 8:11am UTC](https://discuss.elastic.co/t/combination-of-logfiles-with-time-only-and-datetime-fields/143048 "2018-08-05T08:11:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![carstengehling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carstengehling/32/34118_2.png) [@carstengehling](https://discuss.elastic.co/u/carstengehling)\
**Post date:** [August 5, 2018, 8:11am UTC](https://discuss.elastic.co/t/combination-of-logfiles-with-time-only-and-datetime-fields/143048/1 "2018-08-05T08:11:26Z")

</div>

On a single server, I have multiple logfiles, that I need to send to LogStash, the problem is, that logfiles from one application has a datetime column:

2018-06-29 13:44:22,569 [26] DEBUG yadayadayada

and logfiles from another application has only time column:

07:06:27.105 [26] DEBUG yadayadayada

The way I understand FileBeat, I cannot send these files into seperate pipelines (one FileBeat service is running on the server) - is that correct?

If so: Is there a way to configure a LogStash pipeline, so in both logfile cases, the time/datetime column gets used for the @timestamp - but in the second case, the time first gets "enriched" with current date?

Or another solution: Can FileBeat add the date itself to this single input, before it sends it to LogStash?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 5, 2018, 9:39pm UTC](https://discuss.elastic.co/t/combination-of-logfiles-with-time-only-and-datetime-fields/143048/2 "2018-08-05T21:39:27Z")

</div>

There are several options. You could use one prospector for each application, and add a tag using the prospector, then conditionalize the parsing, possibly using a routing pipeline to send the two tags to different processing pipelines.

However, if the rest of the processing is common, then just grok the date conditionally using

```
grok { match => { "message" => ["^%{TIMESTAMP_ISO8601:date},%{NUMBER:ms} \[", "^%{TIME:time} \["] } }
```

---

<div class="post-metadata">

**Author:** ![carstengehling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carstengehling/32/34118_2.png) [@carstengehling](https://discuss.elastic.co/u/carstengehling)\
**Post date:** [August 6, 2018, 9:26am UTC](https://discuss.elastic.co/t/combination-of-logfiles-with-time-only-and-datetime-fields/143048/3 "2018-08-06T09:26:01Z")

</div>

Thanks for your reply, that gets me part of the way. From one logfile, I will now have:

"datetime" =\> "2017-12-17 17:08:36",  
"ms" =\> "911",

and from another I have:

"time" =\> "06:04:53.809",

What if I want both of these to end up with - let's call it "final\_datetime"? In pseudocode computed like this:

```
if <datetime and ms exists> then
    add_field => {"final_datetime" => "%{datetime}.%{ms}"}
else if <time exists> then
    add_field => {"final_datetime" => "%{<date-part of @timestamp>} %{{time}"}
end

```

So that no matter what logline comes in, final\_datetime is the same format? (ie. YYYY-MM-DD hh:mm:ss.mss)

---

<div class="post-metadata">

**Author:** ![carstengehling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carstengehling/32/34118_2.png) [@carstengehling](https://discuss.elastic.co/u/carstengehling)\
**Post date:** [August 6, 2018, 9:44am UTC](https://discuss.elastic.co/t/combination-of-logfiles-with-time-only-and-datetime-fields/143048/4 "2018-08-06T09:44:04Z")

</div>

I figured it out - turned out that I needed to make two different groks, that is where the conditional parsing is split:

```
filter {
  grok {
    match => {
      "message" => "^%{TIMESTAMP_ISO8601:datetime},%{NUMBER:ms} \["
    }
    add_field => {
      "final_datime" => "%{datetime}.%{ms}"
    }
  }
  grok {
    match => {
      "message" => "^%{TIME:time} \["
    }
    add_field => {
      "final_datime" => "%{+YYYY-MM-dd} %{time}"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 3, 2018, 9:44am UTC](https://discuss.elastic.co/t/combination-of-logfiles-with-time-only-and-datetime-fields/143048/5 "2018-09-03T09:44:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.