Hi,
I am trying to ingest data from salesforce to elasticsearch using logstash in Macbook Pro (macOS Mojave Version 10.14.2). But it's not working and gives the below error in my terminal:
*[2020-10-22T08:37:06,130][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.212-b03 on 1.8.0_212-b03 +indy +jit [darwin-x86_64]"}*
*[2020-10-22T08:37:06,348][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified*
*[2020-10-22T08:37:07,967][INFO ][org.reflections.Reflections] Reflections took 32 ms to scan 1 urls, producing 22 keys and 45 values *
*[2020-10-22T08:37:09,408][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}*
*[2020-10-22T08:37:09,570][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}*
*[2020-10-22T08:37:09,612][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}*
*[2020-10-22T08:37:09,616][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}*
*[2020-10-22T08:37:09,659][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}*
*[2020-10-22T08:37:09,714][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}*
*[2020-10-22T08:37:09,756][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/Volumes/Macintosh HD2/RnDAndLearning/ElasticStack/logstash/salesforce/logstash.config"], :thread=>"#<Thread:0xc9a9688 run>"}*
*[2020-10-22T08:37:09,781][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}*
*[2020-10-22T08:37:10,511][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.75}*
*[2020-10-22T08:37:15,714][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}*
*[2020-10-22T08:37:15,912][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}*
*[2020-10-22T08:37:21,012][INFO ][logstash.runner ] Logstash shut down.*
*[2020-10-22T08:37:21,024][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit*
Below is my configuration in config file (filename is logstash.config)
input {
salesforce {
client_id => "Consumer Key"
client_secret => "Consumer Secret"
username => "xxxxxxxxxxx@force.com"
password => "xxxxxxxxxxx"
security_token => "Security token"
sfdc_object_name => "Account"
}
}
filter {}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "sforce"
}
stdout {}
}
Credentials are correct because I am able to login successfully using postman. And I am able to set source and export as CSV using Jitterbit.
Logstash works fine because I am able to ingest data from CSV and JSON to elasticsearch.
Please help me resolving this.
Logstash version is logstash-7.9.2
