Unable to convert date from string/ change @timestamp value

Elastic Stack Logstash
1

I've tried everything i found on the forum but nothing helps, i hope someone could kindly share their experiences with me.

newts
2019-12-01 00:19:03
here is my log config
trying to change @timestamp value doesnt work

filter
{
    grok
    {
        match =>
        {
            "message" => "%{TIMESTAMP_ISO8601:logtimestamp}\s%{DATA:S_IP}\s%{WORD:s_method}\s%{DATA:cs_uri_stem}\s%{DATA:cs_uri_query}\s%{DATA:s_port}\s%{GREEDYDATA:log_message}"
        }
    }
        mutate{
        add_field => {"newts" => "%{logtimestamp}"}
remove_field => ["logtimestamp"]
}
        date
        {
            match => ["newts" ,"yyyy-MM-dd'T'HH:mm:ss,SSSS"]
            locale => "en"
            timezone => "UTC"
            target =>"@timestamp"
        }
}

trying to change logtimestamp from string to date also doesnt work.

filter
{
    grok
    {
        match =>
        {
            "message" => "%{TIMESTAMP_ISO8601:logtimestamp}\s%{DATA:S_IP}\s%{WORD:s_method}\s%{DATA:cs_uri_stem}\s%{DATA:cs_uri_query}\s%{DATA:s_port}\s%{GREEDYDATA:log_message}"
        }
    }
        mutate{
        add_field => {"newts" => "%{logtimestamp}"}
}
        date
        {
            match => ["newts" ,"yyyy-MM-dd'T'HH:mm:ss,SSSS"]
            locale => "en"
            timezone => "UTC"
            target =>"@newts"
        }
}

using multiple format as date filter also doesnt work

date
        {
            match => ["newts" ,"yyyy-MM-dd'T'HH:mm:ss,SSSS","8601"]
            locale => "en"
            timezone => "UTC"
            target =>"@newts"
        }

any help or respond will be appreciated, thank you

2

Hi

I think you could use something like this.

The OP in that case was trying to use the local time, as converted from timestamp, to name files. The solution should be the same: you also want to convert the timestamp into your local time, if I unerstood your post corretly. Just substitute filename for your newts, play with the time format, and you should be good to go.

This will not change @timestamp itself (and apparently there are reasons why you shouldn't), but you will get a variable with the local time that you can use.

Hope this helps

3

That pattern does not match that date. Try "yyyy-MM-dd HH:mm:ss".

4

hi there thank you for your response, ive tried your suggestion but it still doesnt work. newts still a string

5

hi there, thank you for your response, but i still dont i understand about your suggestion. im trying to conver the date field which currently in string type into date but seems like your suggestion will create new file which isnt what i mean to be. Any explanation will be appreciated.

6

It would be, you have set the target to @newts, not newts.

7

hi, thank you for your quick response, ive change it without "@" and it still doesnt work

8

OK, so if you use

output { stdout { codec => rubydebug } }

what does an example event look like?

9 
{
      "@timestamp" => 2020-01-27T01:32:42.668Z,
    "cs_uri_query" => "-",
         "message" => "-",
             "log" => {
        "offset" => 128538,
          "file" => {
            "path" => "-"
        }
    },
        "@version" => "1",
            "host" => {
        "name" => "-"
    },
            "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "s_port" => "443",
           "newts" => 2019-12-01T16:56:59.000Z,
     "cs_uri_stem" => "/1pixel.gif",
        "s_method" => "GET",
           "agent" => {
        "ephemeral_id" => "-",
             "version" => "7.4.2",
            "hostname" => "-",
                "type" => "filebeat",
                  "id" => "-"
    },
            "S_IP" => "202.158.82.10",
           "input" => {
        "type" => "log"
    },
     "log_message" => "-",
             "ecs" => {
        "version" => "1.1.0"
    }
}
10

according to documentation https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html. since i use iso8601 parsing, i should use yyyy-MM-dd’T’HH:mm:ss. But somehow it always returning ["_dateparsefailure"].
Im really lost. Any explanation will be appreciated

11

i rewrite my filter and finally able to make @timestamp to be the same as my timestamp field data, even it still have different hour

{
            "S_IP" => "-",
             "log" => {
        "offset" => 128538,
          "file" => {
            "path" => "-"
        }
    },
        "@version" => "1",
    "logtimestamp" => "19-12-01 23:56:59",
        "s_method" => "GET",
           "agent" => {
            "hostname" => "D-ESS2018070164",
        "ephemeral_id" => "-",
             "version" => "7.4.2",
                "type" => "filebeat"
    },
     "cs_uri_stem" => "-",
          "s_port" => "-",
             "ecs" => {
        "version" => "1.1.0"
    },
    "cs_uri_query" => "-",
           "input" => {
        "type" => "log"
    },
     "log_message" => "-",
            "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
            "host" => {
        "name" => "-"
    },
      "@timestamp" => 0019-12-01T16:49:47.000Z,
         "message" => "-"
}

but on kibana the timestamp field still filled as current date which is the timestamp where logstash processing data from filebeat. any suggestion?

12

Hi

Indeed the original poster (OP) was tryting to create files using the time in the filename, in local time. My suggestion was for a way to convert the time in @timestamp, in UTC, to a variable in your local time. The variable was called filename for clarity in that example but, as I mentioned, in you case it could be newts. The output{} part is not relevant to you case, only the ruby{} filter.

I assumed you wanted to convert from UTC to local time. If that is not the case, then please disregard my answer completely, as I was barking up the wrong tree.

13

thank you for your explanation, i will keep this for future. But for current problem im unable to create newts as date, every filter i do never change its type from string. Since i cant create it as date i wont step any further like change the time zone. Thank you for this amazing explanation anyway. Appreciate it so much.

14

The value of newts is not surrounded by quotes, so it is not a string, it is a LogStash::Timestamp (a Java object). A date filter cannot parse that. You would need to mutate+convert it to string before parsing it.

15

Thank you for your response. Case closed

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.