Convert Date String Into Date Format

Elastic Stack Logstash
62

yes it works but if you see the screenshot i uploaded the date column still in string field

63

if i see the mapping it show this

image
image.png1366×768 134 KB

64

How to identify the template of current index ?? or should i create a template for the index if yes then how can i make a template with naming convention ?

65

Try this:

filter {
csv {
separator => ","
columns => ["Date","MobileNo","Msg","ServiceId","SMS","wsResponse","AKNMsgId","MMMsgId","AmountCharged"]
}
date {
match => ["Date", "MM/DD/YYYY HH:mm"]
target => "Date"
}
mutate {convert => ["Date", "date"]}
}

66

If you have a look at the list of templates

GET _cat/templates

is there another one that might match your index name?

67

filter {
csv {
separator => ","
columns => ["Date","MobileNo","Msg","ServiceId","SMS","wsResponse","AKNMsgId","MMMsgId","AmountCharged"]
}
date {
match => ["Date", "MM/DD/YYYY HH:mm"]
target => "Date"
}
mutate {convert => ["Date", "date"]}
}

Try this but it show error

68

image
image.png1366×768 93.8 KB
how i know which template using the current index pattern

69

Sorry my fault, that couldnt work, try with this one:

filter {
csv {
separator => ","
columns => ["Date","MobileNo","Msg","ServiceId","SMS","wsResponse","AKNMsgId","MMMsgId","AmountCharged"]
}
date {
match => [ "Date", "ISO8601", "YYYY-MM-dd HH:mm:ss", "YYYY-MM-dd HH:mm:ss.ZZZ" ]
target => "Date"
locale => "en"
}
}

70

Data is parse but result is still the same please check the attached screenshot i have created new index

image
image.png1366×768 101 KB

71

@dgonzalezp thank you for trying to solve the problem really appreciate it... but the problem still remains the same :slight_smile:

72

Dont worry @Syed.Ubaid I want to help you :slight_smile: Lets keep trying with this now:

filter {
if ([message] =~ "\bDate\b") {
drop { }
} else {
csv {
separator => ","
columns => ["Date","MobileNo","Msg","ServiceId","SMS","wsResponse","AKNMsgId","MMMsgId","AmountCharged"]
}
date {
match => [ "Date", "ISO8601", "YYYY-MM-dd HH:mm:ss", "YYYY-MM-dd HH:mm:ss.ZZZ" ]
target => "Date"
locale => "en"
}
}
}

73

still the problem remain same date field as string

74

any update @Jenni @dgonzalezp

75

@Jenni how would i know if my current index using any template ??

76

Good morning @Syed.Ubaid
I need investigate a little bit more, give me time I will tell you when I find something

77

Morning :slight_smile: ok waiting

78

Sorry @Syed.Ubaid Im a little bit busy today but now I have time to help you,
I want to test the same in my own Elastic so could you paste the content of one CSV file that you are using?

79

My thoughts on the topic:

  1. I think your date conversion was right back when you were using

    date {
    match => ["Date", "MM/DD/YYYY HH:mm"]
    target => "Date"
    }

because that is the only pattern that matches your date string. Trying anything else doesn't make sense to me.
2. If you used the debug output like I asked you to do, it would show us if we've got a Timestamp object or an error. But you didn't.
3. Your earlier screenshot (the template list) shows '789' in the column for the matched index patterns of 'your_template'. I don't know how that got there. But that would be the reason why it doesn't match your index name.
4. Just to make sure that you are aware of that: Your first goal is to give Elasticsearch a field with a date datatype (which can accept multiple date formats, but we'd have to have working mapping for that telling ES which formats to accept). That probably won't be formatted like your original data by default in Kibana, so don't expect it to be. As soon as it is a date, we can change the formatting. But that is nothing more than a display setting in Kibana and doesn't change your data at all.

1 Like
80

Have you create a new index after add the template?

81

Yes i created new inidex every time