How to convert time typestring into date type


(Ganesh) #1

HI,
I want to convert on string time field into date type. My log is json format and its getting index into this field,

bridgeNemoJenkinsData.time taken.keyword string

I have implemented date filter to convert this as shown below,

date{
                        match => ["bridgeNemoJenkinsData.time taken","HH:mm:SSS"]
                        target => ["bridgeNemoJenkinsData.time taken"]
                 }

and get index like this ,

bridgeNemoJenkinsData.time taken 00:08:027


(Lewis Barclay) #2

Can you please post your full config and example input lines and any errors you have? You should also not have a space in the field name, change it to bridgeNemoJenkinsData.time_taken or similar.


(Ganesh) #3

This is my filter section

 mutate{
                                add_field => {
                        "ci_cd" => "bridgeNemoDataType"
                        }
                        gsub => [
                               "message", "time taken", "time_taken"
                        ]
                 }
                 date{
                        match => ["bridgeNemoJenkinsData.time_taken","HH:mm:SSS"]
                        target => ["bridgeNemoJenkinsData.time_taken"]
                 }
                ruby {
                        init => "require 'socket'"
                        code => "event.set('logstash-hostname',Socket.gethostname)"
                }
                json{
                        source => "message"
                }

I dont have any error


(Lewis Barclay) #4

Can you show some input lines and also what ends up being in the field?


(Ganesh) #5
  kafka {
                type => "doc"
                bootstrap_servers => "xxx:443"
                topics => ["PaymentNet_500000846_xx"]
                security_protocol => "SSL"
                ssl_key_password => "kafkae1"
                ssl_keystore_location => "/tls/kafkae1-signed.keystore"
                ssl_keystore_password => "kafkae1"
                ssl_truststore_location => "/tls/kafka01-client-truststore.jks"
                ssl_truststore_password => "kafka123"
                key_deserializer_class => "org.apache.kafka.common.serialization.StringDeserializer"
                value_deserializer_class => "org.apache.kafka.common.serialization.StringDeserializer"
                group_id => "PaymentNet_group-id_500000846"
                auto_offset_reset => "earliest"
        }

this is my message

{"bridgeNemoDataType":"nemo-jenkins-data","AIM_ID":"AIM","build_url":"https://cxx","stash_Url":"https://xx/tikr.git","bridgeNemoJenkinsData":{"build_Id":"22","current_branch":"develop","user":"Rohit Mohta","node_name":"jenkins-slave-8ncrq","stage_name":"DockerPublish","stage_result":"SUCCESS","pipeline_function":"Docker_Publish","time_taken":"00:21:448"}}


(Ganesh) #6

any update on my below input


(Lewis Barclay) #7

What does the end result look like?


(Ganesh) #8

end result is look like that still in string type instead of data


(Lewis Barclay) #9

in Kibana? Did you change the mapping to date instead of string?


(Ganesh) #10

How can i change the mapping in kibana


(Lewis Barclay) #11

Delete your indexes if you don't care about the data and it should be automatically picked up, otherwise you will have to put a template:

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html


(Ganesh) #12

Actually i tried to parsed that value into new index but even though it shows as string type


(Lewis Barclay) #13

What is the index name? Can you go to the developer console in Kibana and do

GET index-name

And post the results?


(Ganesh) #14

please find my result

{
  "test-000001": {
    "aliases": {},
    "mappings": {
      "doc": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "ci_cd": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "logstash-hostname": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "message": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "tags": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1542959163031",
        "number_of_shards": "3",
        "number_of_replicas": "1",
        "uuid": "K7fNcr1eQaaFPDyHbg_wKg",
        "version": {
          "created": "6020499"
        },
        "provided_name": "test-000001"
      }
    }
  }
}

(Lewis Barclay) #15

There is no field in there called "bridgeNemoJenkinsData.time_taken"