Logstash ruby filter not working

Elastic Stack Logstash
1

Hi Team,

I am trying to get the number from the underscore in the _id, so I tried using ruby filter to get the number below is the following details,

Sample post data,

POST test_item/item/149927_1
{
"ITEM_ID": 149927,
"ITEM_CODE": "149927",
"RELATIONSHIP": [
{
"REL_TYP_REF_ID": -999,
"REL_TYPE": "NO RELATIONSHIP",
"REL_CTGRY": "NIL"
}
],
"ITEM_MISUSED_GTIN_FLG": "N",
"CRT_DTTM": "2009-03-25 15:15:32",
"XCD": []
}

If we execute the GET test_item/item/149927_1 we will get the _id field, Since _id = 149927_1 is a default field in elasticsearch and it is string type

Below is the logstash config file which I used,

input {

We read from the "old" index

elasticsearch {
hosts => ["10.7.147.21:9200"]
user => "esadmin"
password => "dev01"
index => "test_item"
size => 500
scroll => "5m"
docinfo => true
}
}

filter {
mutate {
remove_field => [ "@timestamp", "@version" ]
}
ruby {
code => "document_id = event['%{[@metadata][id]}']
event['PARTITION_ID'] = document_id.split('').first"
}
}

output {

We write to the "new" index

elasticsearch {
host => "10.7.147.21:9200"
protocol => "http"
user => "esadmin"
password => "dev01"
index => "test_item1"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}

We print dots to see it in action

stdout {
codec => rubydebug
}
}

I am getting the error when I execute the logstash command, bin/logstash -f logstash.conf

Ruby exception occurred: undefined method `split' for nil:NilClass {:level=>:error}
Logstash startup completed
{
"ITEM_ID" => 149927,
"ITEM_CODE" => "149927",
"RELATIONSHIP" => [
[0] {
"REL_TYP_REF_ID" => -999,
"REL_TYPE" => "NO RELATIONSHIP",
"REL_CTGRY" => "NIL"
}
],
"ITEM_MISUSED_GTIN_FLG" => "N",
"CRT_DTTM" => "2009-03-25 15:15:32",
"XCD" => [],
"tags" => [
[0] "_rubyexception"
]
}
Logstash shutdown completed

Please help me out to resolve this and correct me if am doing anything wrong in filter section.

Thanks,
Ganeshbabu R

2

code => "document_id = event['%{[@metadata][_id]}']

Here you're asking for a field named %{[@metadata][_id]} (literally) and there's obviously no such field. One of these should work better:

code => "document_id = event['@metadata']['_id']
code => "document_id = event['[@metadata][_id]']

Which one you should use might depend on your version of Logstash.

3

Hi @magnusbaeck

We are using ES 1.7.3 with logstash 1.5.5 and yes we didn't indexed "_id" field and by default elasticsearch itself will have.

In the _id, we are trying to get the number after the underscore ( _ ) and you have given two suggestions and I will try it both, which is suitable for logstash 1.5.5.

Thanks,
Ganeshbabu R

4

You should really upgrade to 2.X, 1.5 is pretty old these days.