Kibana custome timestamp and @timeestamp are different

Elastic Stack Logstash
1

Hi Team,

I am using below xml/xpath statement to convert my string timestamp to date type timestamp in logstash
input - 2021-06-30-20:11:23
xpath - xpath =>[ "concat(substring(/AdMsg/AdLels/AuditLevel/TS/text(), 1,10),'T',substring(/AdMsg/AdLels/AuditLevel/TS2text(), 12),'Z')", "PUT_TIME"]

so my value is coming as date in kibana ,but 5:30 hrs more than current timestamp ,current timestamp is correct in field @timestamp but in my field its showing 5:30 hrs more .

logstash and kibana server timezone is IST

Please suggest method to resolve this in conf file.

2

Hi Team ,

Please respond.

3

You have not told us what your data looks like. It is unclear if you understand that logstash and elasticsearch store dates as UTC. Are you using a date filter or are you using dynamic mapping?

4

Hi Badger,

my i/p is in xml and i am using xpath and string function to create date format

input - 

<AdLels>

    <AuditLevel>

        <TS>2021-06-30-20:11:23</TS>

    </AdMsg>

</AdLels>

xpath - xpath =>[ "concat(substring(/AdMsg/AdLels/AuditLevel/TS/text(), 1,10),'T',substring(/AdMsg/AdLels/AuditLevel/TS2text(), 12),'Z')", "PUT_TIME"]

PUT_TIME is now string like - 2021-07-02T01:39:53Z and kibana is showing this as type date - Jul 2, 2021 @ 07:09:53.000.

so this value in kibana is 5:30 hr more than current timestamp ,what need to be done to show it to timestamp same as @timestamp.

5

OK, the Z at the end means that that is Zulu, i.e. UTC. By default kibana will display that in your local time zone, which is five and half hours ahead of UTC. Use a date filter

date {
    match => [ "PUT_TIME", "YYYY-MM-dd'T'HH:mm:ss" ]
    timezone => "Asia/Kolkata"
    target => "PUT_TIME"
}

That will change that time to 2021-07-01T20:09:53Z.

6

Hi @Badger ,

Thanks for reply!!

After using this code getting - _dateparsefailure in kibana.

in kibana @timestamp is still correct and my PUT_TIME is %:30 hrs more .

Please suggest why its not parsing.

Thanks ,
Mrunalini

7

You changed your xpath to stop adding the 'Z', right?

8

Yes , I stopped adding Z, still facing same - _dateparsefailure in kibana

xpath =>[ "concat(substring(/AdMsg/AdLels/AuditLevel/TS/text(), 1,10),'T',substring(/AdMsg/AdLels/AuditLevel/TS/text(), 12))", "PUT_TIME"]

9

Hi @Badger /Team,

Please help with the issue .

10

Please, do not ping people or bump your post.

You need to share your logstash pipeline, without seeing your logstash pipeline is not possible to see what is wrong.

Also share the result message that give you the date parse failure.

11

Input -

CONSUMER_TO_EAI
some string
REQUEST
2021-07-02-18:23:15

and Pipeline config file as below:

input {
jms {
yaml_file => "XXXX"
yaml_section => "wmq"
use_jms_timestamp => false
destination => "Queue_Name"
interval => 3
runner => async

}

}
filter {
xml {
store_xml => false
source => "message"
remove_namespaces => true
remove_field => ["message"]
xpath =>[ "/AdMsg/MtIf/msaFlowDir/text()", "SERVICE_NAME"]
xpath =>[ "/AdMsg/MtIf/level/text()", "CHANNEL_NAME"]
xpath =>[ "/AdMsg/MtIf/msaStatus/text()", "SEGMENTNAME"]
xpath =>[ "concat(substring(/AdMsg/AdLels/AuditLevel/TS/text, 1,10),'T',substring(/AdMsg/AdLels/AuditLevel/TS/text(), 12))", "PUT_TIME"]

}
if "TZ" in [PUT_TIME]
{

 mutate {
                remove_field =>"PUT_TIME"
        }

}
date {

match => ["PUT_TIME", "yyyy-MM-dd'T'HH:mm:ss.SSS","yyyy-MM-dd'T'HH:mm:ss"]
timezone => "Asia/Kolkata"
target =>["PUT_TIME"]
}
output{

elasticsearch {
hosts => XXX
cacert => XXX
index => XX

 }

stdout { codec => rubydebug }
}

12

Sorry xml Input not pasted correctly so added image:

Capture

13

Hi Team,

Can someone please respond?

14

Hello Team,

This has been pending for long now , can someone please advise

15

What is the value of [PUT_TIME] in kibana? In Discovery, expand an event and copy and paste the value of the field from the JSON tab.

16

Hi Badger,

Put time in JSON Doc is - 2021-07-07T13:44:15

but in table it shows - Jul 7, 2021 @ 19:14:15.000

17

That sounds like it is working as expected. "Jul 7, 2021 @ 19:14:15.000" is the default display format for Kibana, and it is offset from UTC by 5:30.

18

Hi

I want to show original date not the 5:30 hrs more .

Is it possible to do that?

19

I do not have issues with the display of the time , i need it to be shown the original date ,not the 5:30 hrs more.

I there possible way to do that.

20

Elasticsearch stores the date as UTC, by default Kibana transforms that to the browser's timezone. If you want to change that reconfigure Kibana:

Settings -> Advanced -> DateFormat:tz