Elasticsearch filter no longer working in 2.3.1

Elastic Stack Logstash
1

I was using logstash-all-plugins-2.2.0 package before and now I am switching to logstash-all-plugins-2.3.1, the same configuration goes against a ES 2.3.1 works using 2.2.0 version, but not 2.3.1 version.

The elasticsearch filter configuration is straightfoward

elasticsearch {
hosts => ["192.168.3.3:9200"]
query => "MCC:242 AND MNC:1 AND LAC:12201 AND CELL:7157"
fields => ["path", "ppp"]
}

Using 2.2.0 package, I am able to get result, but with 2.3.1 I am getting following error

:error=>#NoMethodError: undefined method `start_with?' for nil:NilClass, :level=>:warn, :file=>"logstash/filters/elasticsearch.rb", :line=>"99", :method=>"filter"}

Any clue? I see change log regarding dependency updates, maybe not fully tested?

2

And now I am having another similar problem with translate plugin. I have been using translate filter using logstash 2.3.1 and it was working fine. like

translate {
field => "xxxx"
dictionary_path => "/yyyy/zzzz.csv"
destination => "wwww"
}

The configuration works in logstash 2.3.1, but when running with logstash 2.2.0, I got this error

The error reported is:
LogStash::Filters::Translate: Bad Syntax in dictionary file /yyyy/zzzz.csv

Combining this issue and the issue above, I cannot run using either version which has both elasticsearch and translate filter. :confounded:

As I did not see any coding change in these two plugin but dependency changes, I will appreciate any help that can pointing me to the direction to solve this problem.

1 Like
3

did you get anywhere with this. I am looking to downgrade to 2.2 just to have plug in working. Critical to my PoC but if thats not fixed in the future I dont want to build the dependency on unsupported plugin :frowning:

4

I end up by looking at the source code and even try to enhance it. There is a pull request https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/31 that enhance it with the index support and talk about a problem with specifying the fields option, the documentation is wrong. it should be like this

fields => [ ["@timestamp", "started"] ]

or fields => { "a" => "b" "c" => "d" } worked for me also.

1 Like
5

thanks that did the trick the documentation was wrong...

6

Thanks a lot,
I spend several hours trying to understand why:

fields => [ ["data"] ]

doesn't work...till I saw your post...first element of array is a source field name from old, but second one are destination:

fields => [ ["src_doc_fieldname","desc_doc_fieldname"] ]

or this are more clear then:

fields => { "src" => "desc" }

1 Like