Logstash not indexing in Solr

Yes, it feels very lonely trying to use the SILK logging stack instead of the much more popular ELK stack. Here's what I needed to do for Logstash to Solr on CentOS 7:

Assuming you've already installed Logstash on CentOS or similar
sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-solr_http
# Fix for Solr6 & Logstash 5: timestamp error
sudo sed -i.bak -e 's/document["@timestamp"]/#document["@timestamp"]/g' /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-solr_http-3.0.1/lib/logstash/outputs/solr_http.rb

Fix for https://github.com/logstash-plugins/logstash-output-solr_http/issues/3

Note, I could instead have added a soft commit below the add: @solr.commit :commit_attributes => {:softCommit => true}

sudo sed -i -e 's/@solr.add(documents)/@solr.add(documents, :add_attributes => {:commitWithin=>1000})/g' /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-solr_http-3.0.1/lib/logstash/outputs/solr_http.rb\

Basically, the version of this plugin in Github fails to ever commit so no log events are ever searchable.

Hope this helps,
Guy

1 Like