Undefined method `get' for

Elastic Stack Logstash
1

I am facing bellow error while using http_poller. I am trying to pull cloudflare API logs in json format. Here is my .conf file. I am using logstash 7.9.2 and logstash-input-http_poller (5.0.2)
Can anyone please help ?

configuration file :

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
#beats {
#port => "5044"
        http_poller
        {
                urls => {	
			cloudflare => " https://api.cloudflare.com/client/v4/user/audit_logs"
                {
			method => get
                    headers => {
                        "X-Auth-Email" => "me@company.com"
                        "X-Auth-Key" => "api_key"
                        "Content-Type" => "application/json"
                    }
                }
		}
        }
#}
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
        json 
        {
            source => "message"
        }
}
output {
        stdout { codec => rubydebug }
}

Error message :

/usr/share/logstash/bin/logstash -f cloudflare.conf --config.reload.automatic
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
    Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
    [INFO ] 2020-11-12 12:38:38.566 [main] runner - Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.272-b10 on 1.8.0_272-8u272-b10-0ubuntu1~16.04-b10 +indy +jit [linux-x86_64]"}
    [WARN ] 2020-11-12 12:38:39.160 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [ERROR] 2020-11-12 12:38:40.581 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `get' for #<Java::OrgLogstashConfigIrImperative::PluginStatement:0x48942d22>", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:227:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:219:in `expr'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:150:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:114:in `expr_attributes'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:92:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:85:in `expr'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:64:in `block in compile'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:62:in `compile'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:36:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}
    [INFO ] 2020-11-12 12:38:40.839 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
    [ERROR] 2020-11-12 12:38:43.917 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"NoMethodError", :message=>"undefined method `get' for #<Java::OrgLogstashConfigIrImperative::PluginStatement:0x3edffe50>", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:227:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:219:in `expr'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:150:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:114:in `expr_attributes'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:92:in `expr'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:85:in `expr'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:64:in `block in compile'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/compiler/lscl.rb:62:in `compile'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:36:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}
2

Where is your logstash.yml located? You probably need to add --path.settings SETTINGS_DIR to your command to give it the path of it.

/usr/share/logstash/bin/logstash -f cloudflare.conf --config.reload.automatic --path.settings /etc/elasticsearch

3

I ran command /usr/share/logstash/bin/logstash -f cloudflare.conf --config.reload.automatic --path.settings /etc/logstash/. getting following error. I am running logstash from command line only for now until I get the API data in my terminal. Also, logstash.yml is located at /etc/logstash

Also, just for information, Elasticsearch,Logstash and Kibana all are running individually on separate instances.

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2020-11-12T14:06:03,302][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.272-b10 on 1.8.0_272-8u272-b10-0ubuntu1~16.04-b10 +indy +jit [linux-x86_64]"}
[2020-11-12T14:06:03,502][ERROR][logstash.configmanagement.bootstrapcheck] There are config files (1) in the 'cloudflare.conf' folder. Elasticsearch is configured as the config store so configs cannot be sourced via the command line with -f or via logstash.yml with path.config
ERROR: There are config files (1) in the 'cloudflare.conf' folder. Elasticsearch is configured as the config store so configs cannot be sourced via the command line with -f or via logstash.yml with path.config`

I was wondering in your command if you wrote following by typo or intentionally.

4

I said /etc/elasticsearch because that was the most likely location. So now your pipeline is actually running.

Do you have centralized pipeline enabled? If so I don't think you can run this via command line according to this.

5

No,centralized pipeline is not enabled.

6

Not sure what directory you are running this from so try adding the full path to your -f cloudflare.conf

7

I am running it from correct directory. In fact, I tried ingesting log file with following configuration and it is working ok and showing me json data on screen. I am facing problem with http_poller only.

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
	file 
	{
		path => "/home/ubuntu/cloudflare/cloudflare.log"
		start_position => beginning
		codec => json{ charset => "UTF-8" }
	}
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
        json 
        {
            source => "result"
        }
}
output {
        stdout { codec => rubydebug }
}
8

Got it. Does the below work for you?

input {
   http_poller {
      urls => {
         urlname => "http://www.google.com"
      }
      request_timeout => 20
      schedule => { every =>"5s"}  
      codec => "line"
   }
}
9

yes it is working fine.

10

Try removing the space between " https

11

@ankitdevnalkar this works for me. But of course it returns an authentication error.

input {
  http_poller {
    urls => {
      test2 => {
        method => get
        url => "https://api.cloudflare.com/client/v4/user/audit_logs"
        headers => {
            "X-Auth-Email" => "me@company.com"
            "X-Auth-Key" => "api_key"
            "Content-Type" => "application/json"
        }
     }
    }
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
  }
}
output {
        stdout { codec => rubydebug }
}
12

hey @aaron-nimocks it worked :rocket: !! I was missing test2 part in my code.
Thanks buddy ! made my day :grin:

13

No problem. You can rename that to whatever you want. Glad it's working! :slight_smile:

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.