# Logstash deletes source files and not creating index - date parsing issue
**URL:** https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243
**Category:** Logstash
**Tags:** docker
**Created:** [November 15, 2023, 5:39pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243 "2023-11-15T17:39:53Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@derekmizak](https://discuss.elastic.co/u/derekmizak)
#### Post date: [November 15, 2023, 5:39pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/1 "2023-11-15T17:39:53Z")
I am using Elastic and logstash 8.11 running in docker.
When logstash starts it deletes log files from the source directory but nothing is passed to elastic - no index is created.
I am not sure why logstash is deleting source files in the first place.
I would apreciate if someone could help to diagnose this.
My logstash.conf is:
```auto
input {
file {
mode => "read"
path => "/usr/share/logstash/ingest_data/ **/** /*.json"
codec => "json_lines"
start_position => "beginning"
sincedb_path => "/dev/null"
file_chunk_size => 1048576
}
}
filter {
json {
source => "message"
target => "parsed_data" # Using 'parsed_data' as a namespace to avoid conflicts
}
date {
match => ["[fields][@timestamp]", "UNIX_MS"]
target => "@timestamp"
}
mutate {
copy => { "[fields][source]" => "source" }
copy => { "[fields][eventType]" => "eventType" }
copy => { "[fields][category]" => "category" }
# remove_field => ["fields"] # Optional: remove the original 'fields' object if it's no longer needed
}
output {
elasticsearch {
index => "logstash-%{+YYYY.MM.dd}"
hosts => "${ELASTIC_HOSTS}"
user => "${ELASTIC_USER}"
password => "${ELASTIC_PASSWORD}"
cacert => "certs/ca/ca.crt"
}
}
```
I am trying to parse logs like this:
`{"_index":".ds-logs-dsf-2023.07.31-000018","_id":"24-12115677111-1690555439","_score":6.2382417,"fields":{"eventId":[100],"@timestamp":["1690888439226"],"source":["manager"],"eventType":["information"],"category":["rtp.document.Service.main.checks.Validator"],"message":["Basic validation passed. ."]}}`
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 15, 2023, 6:00pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/2 "2023-11-15T18:00:18Z")
> [@derekmizak](#):
>
> I am not sure why logstash is deleting source files in the first place.
Because you are using it in the _read_ mode, and when `mode` is set to _read_, it will use use the setting `file_completed_action`, which as the default will **delete** the file after processing.
Also, in _read_ mode, the setting `start_position` is ignored, as described in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-mode).
> [@derekmizak](#):
>
> but nothing is passed to elastic - no index is created.
Did you check Logstash logs? Do you have anything in Logstash logs? please share the logs.
---
### Author:  [@derekmizak](https://discuss.elastic.co/u/derekmizak)
#### Post date: [November 15, 2023, 6:20pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/3 "2023-11-15T18:20:56Z")
@leandrojmp - yes you right about read `mode` - I have chnaged it to tail now. It is not deleting anything.
In the logs I have:
It can connect to elastic
```auto
023-11-15 18:14:24 [2023-11-15T18:14:24,863][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@es01:9200/"}
2023-11-15 18:14:24 [2023-11-15T18:14:24,874][INFO][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.11.0) {:es_version=>8}
2023-11-15 18:14:24 [2023-11-15T18:14:24,874][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
2023-11-15 18:14:25 [2023-11-15T18:14:25,026][INFO][logstash.codecs.jsonlines][main][98e3ef207a4ae9d98a067a1a59b68f9c428884fa2315628551b7969ffef13295] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
2023-11-15 18:14:25 [2023-11-15T18:14:25,713][INFO][logstash.codecs.jsonlines][main][98e3ef207a4ae9d98a067a1a59b68f9c428884fa2315628551b7969ffef13295] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
```
I have also this and nothing more:
```auto
2023-11-15 18:14:40 [2023-11-15T18:14:40,394][INFO][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
2023-11-15 18:16:02 [2023-11-15T18:16:02,119][INFO][logstash.codecs.jsonlines][main][98e3ef207a4ae9d98a067a1a59b68f9c428884fa2315628551b7969ffef13295] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 15, 2023, 6:31pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/4 "2023-11-15T18:31:54Z")
And what is the result of the following requests on Kibana Dev Tools:
```auto
GET _cat/indices?v
```
and
```auto
GET logstash-*/_search
```
Also a couple of things about your logstash configuration.
If your source file is composed by line delimited json, for example, each line is a json document, you should not use the `json_lines` codec, but the `json` codec, this is in the [documenation](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json_lines.html#_description_185) as well.
> NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or **file** **inputs**. Rather, use the json codec.
Another thing is, if you are using the codec in the input, you do not need the json filter as your message will already be parsed.
I would recommend to not use a codec in the input and rely on the `json` filter, and if you choose to do that, you need to add the top-level `[parsed_data]` to your filters, as you are parsing the json into a target field, so instead of `[fields][anything]` you need to use `[parsed_data][fields][anything]`.
---
### Author:  [@derekmizak](https://discuss.elastic.co/u/derekmizak)
#### Post date: [November 15, 2023, 6:56pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/5 "2023-11-15T18:56:56Z")
@leandrojmp - You are a star - all is working now.
I have chnaged logstash.conf as per your recommendation and it is working perfectly.
I have one more question to you if you dont mind.
I wanted to parse timestamp date which in my case is "@timestamp":["1690888439226"] - this is why I have included filter for it:
```auto
date {
match => ["[fields][@timestamp]", "UNIX_MS"]
target => "@timestamp"
}
```
However it is parsed to
```auto
parsed_data.fields.@timestamp
1690978392640
```
But it doesn't decode the timestamp so it is kind of hard searching through it. Could you point me in to a direction what can I do.
```auto
input {
file {
mode => "tail"
path => "/usr/share/logstash/ingest_data/ **/** /*.json"
start_position => "beginning"
sincedb_path => "/dev/null"
file_chunk_size => 1048576
}
}
filter {
json {
source => "message"
target => "parsed_data" # Using 'parsed_data' as a namespace to avoid conflicts
}
date {
match => ["[fields][@timestamp]", "UNIX_MS"]
target => "@timestamp"
}
mutate {
copy => { "[parsed_data][fields][source]" => "source" }
copy => { "[parsed_data][fields][eventType]" => "eventType" }
copy => { "[parsed_data][fields][category]" => "category" }
# remove_field => ["fields"] # Optional: remove the original 'fields' object if it's no longer needed
}
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 15, 2023, 7:34pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/6 "2023-11-15T19:34:01Z")
> [@derekmizak](#):
>
> `match => ["[fields][@timestamp]", "UNIX_MS"]`
Use `match => ["[parsed_data][fields][@timestamp]", "UNIX_MS"]`
---
### Author:  [@derekmizak](https://discuss.elastic.co/u/derekmizak)
#### Post date: [November 15, 2023, 7:50pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/7 "2023-11-15T19:50:45Z")
@Badger I have tried it. I have also addedd target to a diferent field:
But new firld event\_date is not created after that. If I dont specify target or add `target => @timestamp` it doesn't parse date. Not sure what is wrong with it.
```auto
filter {
json {
source => "message"
target => "parsed_data" # Using 'parsed_data' as a namespace to avoid conflicts
}
date {
match => ["[parsed_data][fields][@timestamp]", "UNIX_MS"]
target => "[parsed_data][fields][event_date]"
}
mutate {
copy => { "[parsed_data][fields][source]" => "source" }
copy => { "[parsed_data][fields][eventType]" => "eventType" }
copy => { "[parsed_data][fields][category]" => "category" }
# remove_field => ["fields"] # Optional: remove the original 'fields' object if it's no longer needed
}
}
```
---
### Author:  [@derekmizak](https://discuss.elastic.co/u/derekmizak)
#### Post date: [November 15, 2023, 7:52pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/8 "2023-11-15T19:52:37Z")
I am getting `"_dateparsefailure"` appended to the document.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 15, 2023, 8:28pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/9 "2023-11-15T20:28:05Z")
Since your source document looks like this, it seems that the field is an array.
> [@derekmizak](#):
>
> fields":{"eventId":[100],"@timestamp":["1690888439226"]
Try to use `[parsed_data][fields][@timestamp][0]` in the date filter.
---
### Author:  [@derekmizak](https://discuss.elastic.co/u/derekmizak)
#### Post date: [November 15, 2023, 8:46pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/10 "2023-11-15T20:46:52Z")
After I parse json input I have date filed stored as
```auto
"parsed_data.fields.@timestamp.keyword": [
"1690883634738"
],
```
Oryginal log is like this:
`{"_index":".logs-rtp-2023.07.31-000018","_id":"24-12116678855-1690888439","_score":6.2382417,"fields":{"eventId":[100],"@timestamp":["1690888439226"],"source":["dmanager"],"eventType":["information"],"category":["Service.Common.Validator"],"message":["Basic validation passed. "]}}`
My filter I have tried was:
date {
match =\> ["[@timestamp][0]", "UNIX\_MS"]
target =\> "human\_readable\_timestamp"
tag\_on\_failure =\> ["\_timestamp\_parse\_failed"]
}
And this one is not showing error but date is not parsed and `human_readable_timestamp` firld is not generated unfortunately.
---
### Author:  [@derekmizak](https://discuss.elastic.co/u/derekmizak)
#### Post date: [November 15, 2023, 8:54pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/11 "2023-11-15T20:54:21Z")
Once I have changed my filter to:
```auto
date {
match => ["[parsed_data][fields][@timestamp][0]", "UNIX_MS"]
target => "@timestamp"
tag_on_failure => ["_timestamp_parse_failed"]
}
```
All is working as planned - so basically my issue with this was that initial field wasn't specified correctly.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 13, 2023, 8:54pm UTC](https://discuss.elastic.co/t/logstash-deletes-source-files-and-not-creating-index-date-parsing-issue/347243/12 "2023-12-13T20:54:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.