Merge or combine 2 different input fields as 1

Hello,

I the input CSV file I have the timestamp file separate by a TAB, one as "date" and another as "time", is it possible load is as the timestamp one ? What is the code to doing it ?

input {
file {
path => "/Users/XXX/some_log.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["system_date","system_time","ip","host"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "data-index"
}
stdout {}
}

Regards,
Jonny

Use mutate+add_field with two sprintf references to combine the date and time, then use a date filter to parse it.

Thanks a lot for the suggestion, did it and it works, however notice that the timestamp is getting back a month in the past, i.e. "2020-02-01 10:10:00" moves to "2020-01-01T09:10:00Z", any ideas what i'm missing to put it in the right way ?

Timing is also getting an hour before but found that is because the conversion is always based in UTC so it is ok.

This is the code that i'm using:

input {
file {
path => "./logs/access*.log"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["date-event", "time-event", "time-taken"]
}
mutate {convert => ["time-taken", "float"]}
mutate {add_field => {"transaction-timestamp" => "%{date-event} %{time-event}"}}

date {
	match => ["transaction-timestamp", "YYYY-MM-DD HH:mm:ss"]
	locale => "en"
	timezone => "Europe/Paris"
	target => "@timestamp"
}

}
output {
elasticsearch {
hosts => "localhost"
index => "stats"
document_type => "trafic"
}
stdout {}
}

DD is day of the year, so 01 is January first, and that overwrites the month. Use dd.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.