Logstash-input-salesforce plugin logstash error

Elastic Stack Logstash
1

I've tried reinstalling logstash and the logstash-input-salesforce plugin after changing from Java 11 to 1.8.0. I restart the service. I never see it in Kibana.

ERROR RECEIVED:

[2020-10-12T16:05:14,151][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-10-12T16:05:19,674][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Not really sure what to try next.

$ java -version
    openjdk version "1.8.0_262"
    OpenJDK Runtime Environment (build 1.8.0_262-b10)
    OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

$ curl http://localhost:9200
{
  "name" : "[REDACTED]",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "Ws02dlCKQHmO8IdDVLK3IQ",
  "version" : {
    "number" : "7.9.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "d34da0ea4a966c4e49417f2da2f244e3e97b4e6e",
    "build_date" : "2020-09-23T00:45:33.626720Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

$ sudo /usr/share/logstash/bin/logstash --version
logstash 7.9.2

$ /usr/share/kibana/bin/kibana --version
7.9.2

$ sudo /usr/share/elasticsearch/bin/elasticsearch --version
Version: 7.9.2, Build: default/rpm/d34da0ea4a966c4e49417f2da2f244e3e97b4e6e/2020-09-23T00:45:33.626720Z, JVM: 15

Centos 7

"/etc/logstash/conf.d/salesforce.conf"
input {
   salesforce {
        id => 'Salesforce'
        type => 'Salesforce'
      client_id => '[REDACTED]'
      client_secret => '[REDACTED]'
      username => '[REDACTED]'
      password => '[REDACTED]'
      security_token => '[REDACTED]'
      sfdc_object_name => 'EmailMessage'
      use_test_sandbox => true
   }
}

output {
        elasticsearch {
                index => "salesforce"
                hosts => "localhost"
        }
}

ERROR RECEIVED:

[2020-10-12T16:05:14,151][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-10-12T16:05:19,674][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
2

Enable log.level debug and see if you get a message that explains what the exception is.

3 
[2020-10-12T16:52:03,898][DEBUG][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main"}
[2020-10-12T16:52:03,969][DEBUG][logstash.outputs.elasticsearch][main] Normalizing http path {:path=>nil, :normalized=>nil}
[2020-10-12T16:52:04,450][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-10-12T16:52:04,464][DEBUG][logstash.outputs.elasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2020-10-12T16:52:04,686][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-10-12T16:52:04,750][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-10-12T16:52:04,761][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to
 determine the document _type {:es_version=>7}
[2020-10-12T16:52:04,863][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :ho
sts=>["//localhost"]}
[2020-10-12T16:52:04,986][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:di
sabled}
[2020-10-12T16:52:05,037][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batc
h.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/salesforce.conf"], :thread=>"
#<Thread:0x4daf21cf run>"}
[2020-10-12T16:52:05,092][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"log
stash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_fi
eld"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*",
"match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}],
 "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"},
 "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-10-12T16:52:05,123][DEBUG][logstash.outputs.elasticsearch][main] Found existing Elasticsearch template. Skipping template management {:nam
e=>"logstash"}
[2020-10-12T16:52:05,993][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
 P[output-elasticsearch{"index"=>"salesforce", "hosts"=>"localhost"}|[file]/etc/logstash/conf.d/salesforce.conf:16:2:```
elasticsearch {
                index => "salesforce"
                hosts => "localhost"
        }
```]
 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@3fdc2b9c
[2020-10-12T16:52:05,993][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
 P[output-elasticsearch{"index"=>"salesforce", "hosts"=>"localhost"}|[file]/etc/logstash/conf.d/salesforce.conf:16:2:```
elasticsearch {
                index => "salesforce"
                hosts => "localhost"
        }
```]
 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@3fdc2b9c
[2020-10-12T16:52:05,993][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
 P[output-elasticsearch{"index"=>"salesforce", "hosts"=>"localhost"}|[file]/etc/logstash/conf.d/salesforce.conf:16:2:```
elasticsearch {
                index => "salesforce"
                hosts => "localhost"
        }
```]
 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@3fdc2b9c
[2020-10-12T16:52:05,994][DEBUG][org.logstash.config.ir.CompiledPipeline][main] Compiled output
 P[output-elasticsearch{"index"=>"salesforce", "hosts"=>"localhost"}|[file]/etc/logstash/conf.d/salesforce.conf:16:2:```
elasticsearch {
                index => "salesforce"
                hosts => "localhost"
        }
```]
 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@3fdc2b9c
[2020-10-12T16:52:06,485][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2020-10-12T16:52:06,524][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2020-10-12T16:52:06,686][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.64}
[2020-10-12T16:52:09,447][DEBUG][logstash.javapipeline    ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x18718c6f run>"}
[2020-10-12T16:52:09,505][DEBUG][logstash.javapipeline    ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x348e763 dead>"}
[2020-10-12T16:52:09,507][DEBUG][logstash.javapipeline    ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x715ee600 run>"}
[2020-10-12T16:52:09,510][DEBUG][logstash.javapipeline    ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x6d875f47 dead>"}
[2020-10-12T16:52:09,515][DEBUG][logstash.outputs.elasticsearch][main] Closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
[2020-10-12T16:52:09,527][DEBUG][logstash.outputs.elasticsearch][main] Stopping sniffer
[2020-10-12T16:52:09,531][DEBUG][logstash.outputs.elasticsearch][main] Stopping resurrectionist
[2020-10-12T16:52:09,863][DEBUG][logstash.outputs.elasticsearch][main] Waiting for in use manticore connections
[2020-10-12T16:52:09,873][DEBUG][logstash.outputs.elasticsearch][main] Closing adapter #<LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x755be20c>
[2020-10-12T16:52:09,882][DEBUG][logstash.pluginmetadata  ][main] Removing metadata for plugin 441d418d00bc2c8f32e4bff29c485fc22e0501582c13f5324d3ccf308da0946d
[2020-10-12T16:52:09,891][DEBUG][logstash.javapipeline    ][main] Pipeline terminated by worker error {:pipeline_id=>"main", :exception=>#<Restforce::AuthenticationError: invalid_grant: authentication failure (400)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-4.2.2/lib/restforce/middleware/authentication.rb:31:in `authenticate!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-4.2.2/lib/restforce/middleware/authentication.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-0.14.0/lib/faraday_middleware/request/encode_json.rb:24:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-4.2.2/lib/restforce/middleware/multipart.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-4.2.2/lib/restforce/middleware/mashify.rb:8:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:143:in `build_response'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/connection.rb:387:in `run_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-0.15.4/lib/faraday/connection.rb:138:in `get'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-4.2.2/lib/restforce/concerns/verbs.rb:37:in `block in define_verb'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-4.2.2/lib/restforce/concerns/verbs.rb:63:in `block in define_api_verb'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-4.2.2/lib/restforce/concerns/api.rb:128:in `describe'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-salesforce-3.0.6/lib/logstash/inputs/salesforce.rb:90:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:226:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:225:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:359:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:183:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/salesforce.conf"], :thread=>"#<Thread:0x4daf21cf run>"}
[2020-10-12T16:52:09,919][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-10-12T16:52:09,975][DEBUG][logstash.agent           ] Starting puma
[2020-10-12T16:52:09,996][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2020-10-12T16:52:09,994][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2020-10-12T16:52:10,012][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2020-10-12T16:52:10,016][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2020-10-12T16:52:10,019][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2020-10-12T16:52:10,030][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2020-10-12T16:52:10,035][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2020-10-12T16:52:10,063][DEBUG][logstash.api.service     ] [api-service] start
[2020-10-12T16:52:10,246][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-10-12T16:52:15,127][INFO ][logstash.runner          ] Logstash shut down.
[2020-10-12T16:52:15,172][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
4

There is a problem with the authentication data.

5

Thanks, Badger.
I'm looking at my Salesforce Connected App, but don't see anything misconfigured.

On our firewall I see traffic flow freely to Salesforce, but I don't see any failed OAuth/login attempts for the user.

6

In Salesforce:
I removed all special characters from my password as per RFC6750.
I tried recreating the Connected App.
I relaxed IP Restrictions.
I allowed all users to self-authorize.
I added the machine's public IP address to the trusted list.

On Linux:
I tried .conf with and without token appended to my password.

I don't remember using token for any other OAuth connected app, is this plugin still working for other people?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.