Merge or combine 2 different input fields as 1

[jrodbor (Jonny)]

Hello,

I the input CSV file I have the timestamp file separate by a TAB, one as "date" and another as "time", is it possible load is as the timestamp one ? What is the code to doing it ?

input {
file {
path => "/Users/XXX/some_log.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["system_date","system_time","ip","host"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "data-index"
}
stdout {}
}

Regards,
Jonny

[Badger]

Use mutate+add_field with two sprintf references to combine the date and time, then use a date filter to parse it.

[jrodbor (Jonny)]

Thanks a lot for the suggestion, did it and it works, however notice that the timestamp is getting back a month in the past, i.e. "2020-02-01 10:10:00" moves to "2020-01-01T09:10:00Z", any ideas what i'm missing to put it in the right way ?

Timing is also getting an hour before but found that is because the conversion is always based in UTC so it is ok.

This is the code that i'm using:

input {
file {
path => "./logs/access*.log"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns => ["date-event", "time-event", "time-taken"]
}
mutate {convert => ["time-taken", "float"]}
mutate {add_field => {"transaction-timestamp" => "%{date-event} %{time-event}"}}

date {
	match => ["transaction-timestamp", "YYYY-MM-DD HH:mm:ss"]
	locale => "en"
	timezone => "Europe/Paris"
	target => "@timestamp"
}

}
output {
elasticsearch {
hosts => "localhost"
index => "stats"
document_type => "trafic"
}
stdout {}
}

[Badger]

DD is day of the year, so 01 is January first, and that overwrites the month. Use dd.