I need to concatenate two fields one from date and the other from time and send to Elasticsearc 4.6.4


(Eduardo Bertolucci) #1

Dear,

I need to concatenate two fields to send to Elasticsearc 4.6.4, one field is of date and the other field is time and that field is mapped automatically.

Example of a txt file:

0;0;0;0;0;ENTRADA;;;;;;09/06/2014;06:18:55.234;XXXX;XXXX;
0;0;0;0;0;ENTRADA;;;;;;09/06/2014;06:18:55.284;XXXX;XXXX;

I want to concatenate the date + time field (09/06/2014;06:18:55.234).


(David Pilato) #2

Could you please edit your question and ask in english?


(Eduardo Bertolucci) #3

Yes.


(David Pilato) #4

Great.

Note that Elasticsearch 4.6.4 does not exist. I assume you meant 6.4.x.

Have a look at ingest pipelines. That helps to modify the source document before it gets indexed.

May be this processor would help in your case? https://www.elastic.co/guide/en/elasticsearch/reference/6.5/set-processor.html


(Eduardo Bertolucci) #5

I need the join of the two fields to be the current time (timestamp).


(David Pilato) #6

Yes. That's what I understood. And exactly what I proposed I believe.


(Eduardo Bertolucci) #7

I am using this configuration in .conf

filter {
csv {
separator => ";"
columns => ["1","2","3","4","5","6","7","8","9","10","11","Data","Hora","14"]
}
mutate {
add_field => {
"Data2" => "%{Data} %{Hora}"
}
}
}


(David Pilato) #8

This is logstash then. Ok. Does it work?


(Eduardo Bertolucci) #9

It partially works.

I need these two fields to be sent with the date type.


(David Pilato) #10

I need these two fields to be sent with the date type.

Either change the mapping in elasticsearch and set the format. See https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html

Or send dates with this format (and they will get automatically detected): 2015-01-01T12:10:30Z. Something like %{Data}T%{Hora}.

Or use a date filter in addition to this. See https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

HTH