Logstash pipeline terminated with logstash salesforce input plugin without any error logs

Elastic Stack Logstash
1

Hello
I am trying to integrate salesforce logs to ELK platform.
I used logstash salesforce input plugin to load the logs and output to a file. The logstash pipeline is getting terminated within within seconds without any error logs.
My logstash pipeline configuration as follows

input {
   salesforce {
      client_id => 'xxxx' #Salesforce consumer key
      client_secret => 'xxxx' #Salesforce consumer secret
      username => 'xxxx' #the email address used to authenticate
      password => 'xxxx' #the password used to authenticate
      security_token => 'xxxxx' #Salesforce security token
      sfdc_object_name => 'xxxx' #Salesforce object to track
      sfdc_instance_url => 'xxxxx'
   }  
}
output {
  file {
    path => "/tmp/salesforce.log"
  }
  stdout {
    codec => rubydebug
  }
}

In the logstash logs, I am getting below log
"[2024-05-15T12:58:59,232][INFO ][logstash.javapipeline ][salesforce-pipe] Pipeline terminated {"pipeline.id"=>"salesforce-pipe"}"

Please help me to debug the issue.

Thanks in advance,
Pavani

2

Enable log.level debug and see if anything useful gets logged.

3

Hello Badger,
Greetings of the day!
Thanks for the response.

I could see the below logs.

[2024-05-31T11:34:32,375][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:"salesforce-logstash-v2"}
[2024-05-31T11:34:36,081][INFO ][logstash.javapipeline    ][salesforce-logstash-v2] Starting pipeline {:pipeline_id=>"salesforce-logstash-v2", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["central pipeline management"], :thread=>"#<Thread:xxxxxx run>"}
[2024-05-31T11:34:36,079][WARN ][logstash.javapipeline    ][salesforce-logstash-v2] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary	
[2024-05-31T11:34:36,085][INFO ][logstash.javapipeline    ][salesforce-logstash-v2] Pipeline Java execution initialization time {"seconds"=>0.0}
[2024-05-31T11:34:40,082][INFO ][logstash.javapipeline    ][salesforce-logstash-v2] Pipeline terminated {"pipeline.id"=>"salesforce-logstash-v2"}
[2024-05-31T11:34:40,082][ERROR][logstash.javapipeline    ][salesforce-logstash-v2] Pipeline error {:pipeline_id=>"salesforce-logstash-v2", :exception=>#<Restforce::NotFoundError response={:status=>404, :headers=>{"date"=>"Fri, 31 May 2024 11:34:38 GMT", "content-type"=>"application/json;charset=UTF-8", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "strict-transport-security"=>"max-age=63072000; includeSubDomains", "x-content-type-options"=>"nosniff", "x-robots-tag"=>"none", "cache-control"=>"no-cache,must-revalidate,max-age=0,no-store,private", "set-cookie"=>"CookieConsentPolicy=0:1; path=/; expires=Sat, 31-May-2025 11:34:38 GMT; Max-Age=31536000; secure, LSKey-c$CookieConsentPolicy=0:1; path=/; expires=Sat, 31-May-2025 11:34:38 GMT; Max-Age=31536000; secure, BrowserId=w2WrTx9BEe-oNeFNz2MfUQ; domain=.salesforce.com; path=/; expires=Sat, 31-May-2025 11:34:38 GMT; Max-Age=31536000; secure; SameSite=Lax", "warning"=>"299 - \"This API is deprecated and will be removed by Summer '25. Please see https://help.salesforce.com/s/articleView?id=000389618&type=1 for details.\"", "sforce-limit-info"=>"api-usage=4/31345000", "server"=>"sfdcedge", "x-sfdc-request-id"=>"93c121255d5d2f246b1393646a52df11"}, :body=>[{"errorCode"=>"NOT_FOUND", "message"=>"The requested resource does not exist"}]}>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/middleware/raise_error.rb:16:in `on_complete'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-1.10.3/lib/faraday/middleware.rb:19:in `block in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-1.10.3/lib/faraday/response.rb:61:in `on_complete'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-1.10.3/lib/faraday/middleware.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-1.2.0/lib/faraday_middleware/response/follow_redirects.rb:79:in `perform_with_redirection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-1.2.0/lib/faraday_middleware/response/follow_redirects.rb:67:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/middleware/instance_url.rb:13:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/middleware/authorization.rb:11:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/middleware/authentication.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday_middleware-1.2.0/lib/faraday_middleware/request/encode_json.rb:26:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/middleware/multipart.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/middleware/mashify.rb:8:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-1.10.3/lib/faraday/rack_builder.rb:154:in `build_response'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-1.10.3/lib/faraday/connection.rb:516:in `run_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/faraday-1.10.3/lib/faraday/connection.rb:202:in `get'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/concerns/verbs.rb:37:in `block in define_verb'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/concerns/verbs.rb:63:in `block in define_api_verb'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/restforce-5.0.6/lib/restforce/concerns/api.rb:128:in `describe'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-salesforce-3.2.1/lib/logstash/inputs/salesforce.rb:101:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:391:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:316:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["central pipeline management"], :thread=>"#<Thread:0xxxxx run>"}
[2024-05-31T11:34:40,091][ERROR][logstash.agent           ] Failed to execute action {:id=>:"salesforce-logstash-v2", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Reload<salesforce-logstash-v2>, action_result: false", :backtrace=>nil}

Thanks,
Pavani

4

When the input is registered (i.e. the pipeline is initialized) it runs .describe against the object of interest to get all the metadata for it (a list of fields etc.).

If this gets a 404 NOT FOUND error then I think it is telling you that the object pointed to by sfdc_object_name => 'xxxx' does not exist at the instance pointed to by sfdc_instance_url => 'xxxxx'.

5

Hello Badger,
Greetings of the day!

Please help me with syntax

describe against the object of interest to get all the metadata for it (a list of fields etc.).

How to specify list of metadata and fields in the input.

Thanks,
Pavani

6

You cannot do that. You give the input an object name and the input asks the salesforce instance what metadata is available.