JSON > Logstash > ElasticSearch

Elastic Stack Logstash
1

I have had some good info from a user in answer to my question here and am having issues with my config.

Basically I would like to import / index my pretty JSON file map one field to not_analized to my remote ElasticSearch index.

I am using this config to test locally:

input {
  file {
    path => "/Users/taylorsuk/Desktop/Exercises_Single.json"
    type => "exercises"
    codec => "json"
  }
}
output {
  stdout { codec => rubydebug }
  elasticsearch { 
  host => "localhost" 
  protocol => "http"
  }
}

and getting this error:

Simons-MacBook-Pro:logstash-1.5.2 taylorsuk$ bin/logstash -f logstash-guru.conf
Jul 14, 2015 7:25:13 PM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] version[1.5.1], pid[91394], build[5e38401/2015-04-09T13:41:35Z]
Jul 14, 2015 7:25:13 PM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] initializing ...
Jul 14, 2015 7:25:13 PM org.elasticsearch.plugins.PluginsService <init>
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] loaded [], sites []
Jul 14, 2015 7:25:15 PM org.elasticsearch.node.internal.InternalNode <init>
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] initialized
Jul 14, 2015 7:25:15 PM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] starting ...
Jul 14, 2015 7:25:15 PM org.elasticsearch.transport.TransportService doStart
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.11:9300]}
Jul 14, 2015 7:25:15 PM org.elasticsearch.discovery.DiscoveryService doStart
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] elasticsearch/CHxctaujSuWMn4MUz2xmEg
Jul 14, 2015 7:25:45 PM org.elasticsearch.discovery.DiscoveryService waitForInitialState
WARNING: [logstash-Simons-MacBook-Pro.local-91394-13456] waited for 30s and no initial state was set by the discovery
Jul 14, 2015 7:25:45 PM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-Simons-MacBook-Pro.local-91394-13456] started
Failed to install template: waited for [30s] {:level=>:error}

Many thanks in advance.

Simon

2

What does curl localhost:9200/ return?

3

@warkolm thanks for your reply.
it returns :

Simons-MacBook-Pro:logstash-1.5.2 taylorsuk$ curl localhost:9200/
{
  "status" : 200,
  "name" : "Guru_Node-dev",
  "cluster_name" : "rehab_guru-dev",
  "version" : {
    "number" : "1.6.0",
    "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",
    "build_timestamp" : "2015-06-09T13:36:34Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}
4

Hey, just curious, if you figured out a solution for the issue you were facing, I seem to be having the same issue.

(yes, elasticsearch works, I can connect using curl, kibana, marvel)

Your help would be appreciated!

Thanks.

5

I didn't manage to get this sorted - I just run my imports manually and don't bother with logstash.

Sorry I cannot be any help. I use searchly.com it has good visualisation in the dashboard which makes things easier for me.

Simon

6

I think your logstash is not able to find your elasticsearch cluster.
Try to put your cluster name in logstash's elasticsearch output.
Default elasticsearch cluster name for logstash is "ElasticSearch"
e.g -

output {
stdout { codec => rubydebug }
elasticsearch {
host => "localhost"
protocol => "http"
cluster => "rehab_guru-dev"
}