Unable to match @timestamp to field created after creating dynamic mapping

Elastic Stack Logstash
1

Hello,

I have created dynamic mapping
{"doc": {
"date_detection": true,
"dynamic_date_formats": "yyyy-MM-dd HH:mm:ss",

             "properties": {
                "created": { "type": "date",
                  "format": "yyyy-MM-dd HH:mm:ss"
                },
                 "modified": { "type": "date",
                  "format": "yyyy-MM-dd HH:mm:ss"
                },
                 "@timestamp": { "type": "date",
                  "format": "strict_date_optional_time"
                }
             },
             "dynamic_templates": [
               {
                 "integers": {
                   "match_mapping_type": "long",
                   "mapping": {
                     "type": "integer"
                   }
                 }
               },
               {
                 "strings": {
                   "match_mapping_type": "string",
                   "mapping": {
                     "type": "keyword"
                       }
                     }
                   }
             ]
           }
         }

when I am passing data without dynamic mappings the

date {
match => [ "created", "yyyy-MM-dd HH:mm:ss" ]
target => "@timestamp"
}

It is matching properly

But, When i am using this config after creating dynamic Mapping it's not working.
@timestamp field is 11 hr back as compared to created field.

A help would be really appreciated :slight_smile:

2

All timestamps stored in Elasticsearch must be in UTC timezone. Which timezone are your timestamps expressed in?

3

hello @Christian_Dahlqvist
thankyou for the response,

I am picking the data from local system

{"received":"2019-02-01 00:00:00","ipaddress":"_","response_text":"Thank you for contacting }

my data is of this format and I am renaming received to created and after that matching it to @timestamp

Regards

4

What time zone is that time stamp in? Are you using the date filter to populate @timestamp?

5

@Christian_Dahlqvist its the browser time

6

@Christian_Dahlqvist
can you Please help me out with this

7

Have you verified that the date filter converts this into the correct UTC time stamp?

8

@Christian_Dahlqvist when i am not using dynamic mapping than this config is working pretty well and i am getting my @timestamp matched

date {
match => [ "created", "yyyy-MM-dd HH:mm:ss" ]
target => "@timestamp"
}

but when i create the dynamic mapping I am facing the error

one sample log is
April 1st 2019, 05:29:59.000 @version:
@timestamp March 31st 2019, 23:59:59.000

t @version 1
t _id guWf3WkBY7Thni_1DFyI
# _score -
t _type doc
t beat.hostname localhost.localdomain
t beat.name localhost.localdomain
t beat.version 6.2.4
created April 1st 2019, 05:29:59.000
t

9

Can you paste the JSON document as well as the mapping here?

10

@Christian_Dahlqvist thanks a lot for response

my sample log is

{"no":"12344","message":"urgent","network":"bigdata","shortcode":"1234","received":"2019-02-01 00:00:01","ipaddress":"***"}

my dynamic mapping is

PUT /index_name
{
"settings" : {
"index" : {
"number_of_shards" : 2,
"number_of_replicas" : 0
}
},
"mappings": {
"doc": {
"date_detection": true,
"dynamic_date_formats": "yyyy-MM-dd HH:mm:ss",

 "properties": {
    "created": { "type": "date",
      "format": "yyyy-MM-dd HH:mm:ss"
    }
 },
 "dynamic_templates": [
   {
     "integers": {
       "match_mapping_type": "long",
       "mapping": {
         "type": "integer"
       }
     }
   },
   {
     "strings": {
       "match_mapping_type": "string",
       "mapping": {
         "type": "keyword"
           }
         }
       }
 ]

}
}
}

my logstash config file is

input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
mutate {
rename => [ "received", "created" ]
}
mutate {
remove_field => [ "some_fields" ]
}
date {
match => [ "created", "yyyy-MM-dd HH:mm:ss" ]
target => "@timestamp"
}
}
output{
elasticsearch {
hosts => "IP:9200"
index => "index_name"
}
stdout{}
}

11

Do not specify a date format in your dynamic mapping as that does not seem to match the default that the date filter produces. What happens if you use the default mappings and do not specify any dynamic mappings?

12

@Christian_Dahlqvist than what should be the format for dynamic mapping for date format
can you please write is for me

Regards

13

If you do not specify it explicitly, the default should be used which should work.

14

@Christian_Dahlqvist should the dynamic_date_formats
and date format for field created be default format?

15

Try indexing your data into a new index without any custom template (use the dynamic defaults) and check how that works out.

16

@Christian_Dahlqvist my new dynamic mapping is

PUT /environment
{
"settings" : {
"index" : {
"number_of_shards" : 2,
"number_of_replicas" : 0
}
},
"mappings": {
"doc": {
"date_detection": true,

 "properties": {
    "created": { "type": "date"
    }
 },
 "dynamic_templates": [
   {
     "integers": {
       "match_mapping_type": "long",
       "mapping": {
         "type": "integer"
       }
     }
   },
   {
     "strings": {
       "match_mapping_type": "string",
       "mapping": {
         "type": "keyword"
           }
         }
       }
 ]

}
}
}

I am still facing issue
Please help me

17

solved..

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.