yodog
(Yodog)
July 4, 2018, 7:01pm
1
lets say i have an index iptables-%{+YYYY.MM}
, which holds iptables kernel generated logs.
i would like to:
1- create the doc if the log line doesn't exists
or
2- increment a field called counter
if it already exists
every doc would be saved for the first time with a counter, incrementing on every insert of the same key.
input {
beats {
add_field => { "counter" => 1 }
port => "5044"
}
}
the document_id
would be based on some fields from message
;
this would allow me to always have only one of each message type.
filter {
fingerprint {
add_tag => [ "alreadyseen" ]
concatenate_sources => true
key => "alreadyseen"
source => [ "SRC", "SPT", "DST", "DPT" ]
target => "[@metadata][fingerprint]"
}
}
output {
elasticsearch {
action => "update"
doc_as_upsert => true
document_id => "%{[@metadata][fingerprint]}"
index => "iptables-%{+YYYY.MM}"
sniffing => true
template_overwrite => true
}
}
but i have no idea on how to increment my counter.
any help?
yodog
(Yodog)
July 5, 2018, 7:05pm
2
so, after reading https://www.elastic.co/guide/en/elasticsearch/reference/6.2/docs-update.html#_scripted_updates i tried the following
elasticsearch {
action => "update"
doc_as_upsert => true
document_id => "%{[@metadata][fingerprint]}"
manage_template => false
script => "ctx._source.counter++"
}
but the counter
field always concatenate the number 1
instead of adding
on 1st execution counter: 1
on 2nd execution counter: 11
on 3rd execution counter: 111
and so on
i tried all lines below
script => "ctx._source.counter++"
script => "ctx._source.counter += 1"
script => "ctx._source.counter = ctx._source.counter + 1"
script => "ctx._source.counter = ctx._source.counter++"
add_field => { "counter" => 1 }
Unless you have an index template counter will be a string in elasticsearch. You could mutate it to be an integer (and create a new index, since the old index already has a mapping).
mutate { convert => { "counter" => "integer" } }
However, you will then hit another problem: a null pointer exception when you do the initial insert
script => "if (ctx._source.counter != null) {ctx._source.counter++}"
yodog
(Yodog)
July 9, 2018, 2:38pm
4
the strange thing is that i actually do have a index template mapping
"properties": {
"@timestamp": {
"type": "date"
},
"counter": {
"type": "long"
},
and kibana mapped it as number
, so it seems right
anyway, got it to work with
script => "if (ctx._source['counter'] == null) { ctx._source['counter'] = 1 } else { ctx._source.counter++ }"
and removing add_field
from the input
system
(system)
Closed
August 8, 2018, 2:35pm
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.