# Logstash - problems with date
**URL:** https://discuss.elastic.co/t/logstash-problems-with-date/129084
**Category:** Logstash
**Created:** [April 23, 2018, 11:14am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084 "2018-04-23T11:14:57Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 23, 2018, 11:14am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/1 "2018-04-23T11:14:57Z")
Log example:
[2018-03-28 12:00:21] Processed: shop\Export\Jobs\ExportOrders
[2018-03-28 12:00:22] Processed: shop\Export\Jobs\ExportOrders
Logstash configuration:
input {
file {
path =\> "path/to/test/file.log"
start\_position =\> "beginning"
}
}
```
filter {
grok {
match => { "message" => "\[%{GREEDYDATA:logtime}\] %{WORD:status}: %{GREEDYDATA:process}" }
}
date {
timezone => "Europe/Prague"
match => ["sourcestamp", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
}
output {
elasticsearch {
hosts => ["xxx.xxx.xxx.xxx:xxxx"]
}
stdout { codec => rubydebug }
}
```
Elastisearch output (partial, that matters):
"\_source": {
"process": "shop\Export\obs\ExportOrders",
"@timestamp": "2018-04-23T11:12:05.458Z",
"message": "[2018-03-28 12:00:18] Processed: shop\Export\Jobs\ExportOrders",
"path": "/data/queue/timestamp\_test1.log",
"status": "Processed",
"host": "kibana0",
"logtime": "2018-03-28 12:00:18",
"@version": "1"
}
The problem i have is, that I am unable to replace @timestamp with logtime. I know, there are many topics about this, but no solution there helped me, so i decided, to make new topic, with my exact configuration and log. I do not know, what should I do. Can you help me ?
Thanks, David
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 23, 2018, 11:17am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/2 "2018-04-23T11:17:51Z")
> [@piccasso007](#):
>
> match =\> ["sourcestamp", "yyyy-MM-dd HH:mm:ss"]
The field name should be `logtime`, not `sourcestamp`, as that is what is extracted.
---
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 23, 2018, 11:31am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/3 "2018-04-23T11:31:24Z")
Problem is, when I use the same variable, I have no error in logstash log, but elastisearch output is then empty
```
error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "logstash-2018.04.23",
"index_uuid": "_na_",
"index": "logstash-2018.04.23"
}
],
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "logstash-2018.04.23",
"index_uuid": "_na_",
"index": "logstash-2018.04.23"
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 23, 2018, 11:34am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/4 "2018-04-23T11:34:23Z")
Have you by any chance disabled [automatic index creation in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-index_.html#index-creation)?
---
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 23, 2018, 11:39am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/5 "2018-04-23T11:39:22Z")
No, so far as i know. Only changes i did to elastisearch config are hosts and port changes.
Actually when I ran with the configuration, you suggested (the variable logtime), it created folder in "indices" (elastisearch), like it always do, so i suggest, it recieved the data from logstash but perhaps corrupted. But then i wonder why i have no parse error from logstash.
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [April 23, 2018, 12:22pm UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/6 "2018-04-23T12:22:22Z")
What does the rubydebug output look like?
---
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 23, 2018, 1:07pm UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/7 "2018-04-23T13:07:53Z")
So far I am unable to capture rubydebug output directly. I am trying to do this with:
```
output {
file {
path => "/tmp/rubydebug_output.txt"
codec => rubydebug
}
}
```
But the file remains empty. I am not sure, if it´s supposed to be like that.
---
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 25, 2018, 9:09am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/8 "2018-04-25T09:09:06Z")
Ok, I have some new informations. When I use the config Christian suggested (the one, that makes elastisearch 404 error), then the rudydebug output is empty.
When I try different config, that gives some results.
```
input {
file {
path => "/data/queue/timestamp_test2.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{WORD:status}: %{GREEDYDATA:process}" }
}
date {
timezone => "Europe/Prague"
match => ["logtime", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
}
output {
file {
path => "/tmp/rubydebug_output.txt"
codec => rubydebug
}
}
```
Then the output of rubydebug looks like:
```
"host" => "xxx",
"@timestamp" => 2018-04-25T09:00:00.529Z,
"process" => "shop\\Export\\Jobs\\ExportOrders",
"@version" => "1",
"path" => "/data/queue/timestamp_test2.log",
"message" => "[2018-03-28 12:00:22] Processed: shop\\Export\\Jobs\\ExportOrders",
"status" => "Processed"
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 25, 2018, 9:18am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/9 "2018-04-25T09:18:13Z")
You are basing the date filter on the `logtime` field, which you are not extracting in your grok filter (which does not match what your `message` field looks like anyway).
---
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 25, 2018, 9:46am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/10 "2018-04-25T09:46:49Z")
That is why I am confused, that it doesn´t work the way you said.
Ofcourse in the last example it can´t work, because, there is no _logtime_ field. I just wanted to show, that when I change the config to something else (or having different variables), I have working output.
When I posted the issue, I had different variables. _GREEDYDATA:logtime_ was there as a string looking like this _"logtime": "2018-03-28 12:00:18"_
So then I thought, the date must process it, using same variables, I had no parse error or anything like that, the output was simply emtpy.
Later I also tried
```
filter {
grok {
match => { "message" => "\[%{GREEDYDATA:logtime}\] %{WORD:status}: %{GREEDYDATA:process}" }
}
date {
timezone => "Europe/Prague"
match => ["%{logtime}", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
}
```
But ouptut is still empty. I don´t know, what should I try next.
---
### Author:  [@Jenni](https://discuss.elastic.co/u/Jenni)
#### Post date: [April 25, 2018, 10:16am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/11 "2018-04-25T10:16:37Z")
I did a small test:
**datefailtest.conf:**
```
input { stdin{} }
filter {
grok {
match => { "message" => "\[%{GREEDYDATA:logtime}\] %{WORD:status}: %{GREEDYDATA:process}" }
}
date {
timezone => "Europe/Prague"
match => ["logtime", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
}
output { stdout{ codec => rubydebug } }
```
**execution:**
> root@elasticsearch-vm:/# /usr/share/logstash/bin/logstash -f "/root/datefailtest.conf" --path.data="/var/lib/logstash-man"
> Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
> The stdin plugin is now waiting for input:
> [2018-03-28 12:00:21] Processed: shop\Export\Jobs\ExportOrders
> {
> "@timestamp" =\> 2018-03-28T10:00:21.000Z,
> "@version" =\> "1",
> "message" =\> "[2018-03-28 12:00:21] Processed: shop\Export\Jobs\ExportOrders",
> "process" =\> "shop\Export\Jobs\ExportOrders",
> "host" =\> "elasticsearch-vm",
> "status" =\> "Processed",
> "logtime" =\> "2018-03-28 12:00:21"
> }
Please check your ES configuration again and make sure that **_[this](https://discuss.elastic.co/t/mutate-function-not-working-logstash-6-2-4/129067/17)_** is not the reason for your empty output right now.
---
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 25, 2018, 11:31am UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/12 "2018-04-25T11:31:20Z")
Wow, thanks alot, now I am getting somewhere. When I tried to run it, as you suggested, with minor changes.
```
input {
file {
path => "/data/queue/timestamp_test2.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "\[%{GREEDYDATA:logtime}\] %{WORD:status}: %{GREEDYDATA:process}" }
}
date {
timezone => "Europe/Prague"
match => ["logtime", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
}
output {
file {
path => "/tmp/rubydebug_output.txt"
codec => rubydebug
}
}
```
And run it with command:
`/usr/share/logstash/bin/logstash -f "/etc/logstash/conf.d/logstash-queue.conf"`
Then I get the result i wanted, and file `/tmp/rubydebug_output.txt` contains:
```
{
"message" => "[2018-03-28 12:00:22] Processed: Inspishop\\Export\\JoeAlex\\Jobs\\ExportOrders",
"@timestamp" => 2018-03-28T10:00:22.000Z,
"status" => "Processed",
"@version" => "1",
"host" => "kibana0",
"process" => "Inspishop\\Export\\JoeAlex\\Jobs\\ExportOrders",
"path" => "/data/queue/timestamp_test2.log",
"logtime" => "2018-03-28 12:00:22"
}
```
But, when I run it as service ( service logstash start ) then it doesn´t do anything.
I suppose then I have some other config wrong.
---
### Author:  [@piccasso007](https://discuss.elastic.co/u/piccasso007)
#### Post date: [April 25, 2018, 12:00pm UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/13 "2018-04-25T12:00:07Z")
Ok all problems solved. It was all my fault, like it always is. Firstly it is OK as service, i just had to add the data after I started the service. Then I still couldnt find it in elastisearch, when I redirected the output to ES. But it was there, only I had to search index, by the old date, not the new one.
Thank you very much for your help Jenni and Christian.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 23, 2018, 12:00pm UTC](https://discuss.elastic.co/t/logstash-problems-with-date/129084/14 "2018-05-23T12:00:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.