POST method http_poller

m.sereda · July 27, 2018, 12:11pm

Trying to make a POST method to login but smth is going wrong. To make a login I have to fill "login" and "password" poles which is in JSON. Where is a mistake?

input {
  http_poller {
    urls => {
      soap_request => {
        method => post
        url => "http://localhost:8080/api/ses/login"
        headers => {
          Content-Type => "application/json"
        }
        body => '{"login": "login", "password": "password"}'
      }
    }
    request_timeout => 10
    interval => 10
    codec => "plain"
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

After I need to take a cookie and send it to GET method http://localhost:8080/api/my/destination , how to do that?

magnusbaeck · July 27, 2018, 6:49pm

Trying to make a POST method to login but smth is going wrong.

Well, what happens?

After I need to take a cookie and send it to GET method http://localhost:8080/api/my/destination , how to do that?

I don't think there's any convenient way of doing that.

m.sereda · July 27, 2018, 6:52pm

Hmm... is it possible to write any script or somehow to do that?

magnusbaeck · July 27, 2018, 6:53pm

Sure, you could make the HTTP call in a separate script that you invoke with an exec plugin.

m.sereda · July 27, 2018, 6:55pm

Could you please help me to do that or is there any tutorial?

How this script should looks like? what type ?

magnusbaeck · July 27, 2018, 9:25pm

You can write such a script in any language. I don't have an example at hand.

m.sereda · July 28, 2018, 8:13am

Well, what happens?

Here is my logstash.conf with POST method and it is not working. Do I put login /password in a good way? The format which I should put is JSON

input {
  http_poller {
    urls => {
      test1 => {
        url => "http://localhost:8080/api/sess/login"
        method => "post"
        headers => {
          Accept => "application/json"
        }
        body => '{"login":"login", 
        "password":"pass"}'
      
    }

magnusbaeck · July 28, 2018, 11:11am

Here is my logstash.conf with POST method and it is not working.

That doesn't answer the question "what happens?". Are you getting any response at all from the server? If yes, what does it look like?

m.sereda · July 29, 2018, 8:06am

I got a 415 error

@version:1 message:<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>Payara Micro #badassfish - Error report</title><style type="text/css"><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:whit

magnusbaeck · July 29, 2018, 6:46pm

415 is "Unsupported media type". Perhaps you need to send "Content-type: application/json" as an additional HTTP header in your request? Otherwise I don't know. Try using e.g. Wireshark to capture a successful request and compare it to the one Logstash makes.

m.sereda · July 30, 2018, 10:31am

OK, I've made a python script with saving a cookie in a file and it is working

import os
from cookielib import LWPCookieJar

import requests

s = requests.Session()
s.cookies = LWPCookieJar('cookiejar')
if not os.path.exists('cookiejar'):
    # Create a new cookies file and set our Session's cookies
    print('setting cookies')
    s.cookies.save()
    r = s.post('http://localhost:8080/api/ses/login', json={'login': 'login', 'password': 'pass'})
else:
    # Load saved cookies from the file and use them in a request
    print('loading saved cookies')
    s.cookies.load(ignore_discard=True)
    r = s.post('http://localhost:8080/api/ses/login', json={'login': 'login', 'password': 'pass'})
#print(r.text)
# Save the session's cookies back to the file
s.cookies.save(ignore_discard=True)

print(r.status_code)
#print(r.json())
print(r.cookies)

Cookie file

#LWP-Cookies-2.0
Set-Cookie3: JSESSIONID=aae34175e7cea354b77b29020288; path="/"; domain="localhost.local"; path_spec; discard; HttpOnly=None; version=0

and trying to use an exec plugin. Don't understand how to send cookies from that scrypt to logstash

input {
  exec {
        command => "python /home/maksym/PathFolder/pythonpractice/postsample.py"
        interval => 20
        id => "plugin_id"
        }
  http_poller {
    urls => {
       test2 => {
       url => "http://localhost:8080/api/dash/space"
       method => "get"
       headers => {
         Accept => "application/json"
         Cookie => "/home/maksym/PathFolder/pythonpractice/cookiejar"
       }
    }
   }
   request_timeout => 60
   # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
   schedule => { cron => "* * * * * UTC"}
   codec => "json"
   # A hash of request metadata info (timing, response headers, etc.) will be sent here
   metadata_target => "http_poller_metadata"
   }
}

output {
   stdout {
     codec => rubydebug
   }
}

For now the output is

Appreciate any help. Thanks

magnusbaeck · July 30, 2018, 8:24pm

Do everything from the Python script. Don't try to pass the cookies from the Python script to the http_poller input.

m.sereda · July 31, 2018, 8:03am

OK, now I think I understand. I changed my logstash.conf as the following

input {
  exec {
        command => "python /home/maksym/PathFolder/pythonpractice/postsample.py"
        interval => 20
        id => "plugin_id"  
        }
   request_timeout => 60
   schedule => { cron => "* * * * * UTC"}
   codec => "json"
}
      
output {
   stdout { 
     codec => rubydebug
   }
}

and python.py like this:

import os
os.chdir("/home/maksym/PathFolder")
from cookielib import LWPCookieJar

import requests

s = requests.Session()
s.cookies = LWPCookieJar('cookiejar')
if not os.path.exists('cookiejar'):
    # Create a new cookies file and set our Session's cookies
    print('setting cookies')
    s.cookies.save()
    r = s.post('http://localhost:8080/api/sess/login', json={'login': 'login', 'password': 'pass'})
    print(r.text)
else:
    # Load saved cookies from the file and use them in a request
    print('loading saved cookies')
    s.cookies.load(ignore_discard=True)
    r = s.get('http://localhost:8080/api/dash/space')
print(r.text)
# Save the session's cookies back to the file
s.cookies.save(ignore_discard=True)

print(r.status_code)
#print(r.json())
print(r.cookies)

and when I running the script from the console it is working, but from logstash no - nothing showing in Kibana. Could you tell me where I have made a mistake?

magnusbaeck · August 1, 2018, 6:00am

Does your stdout output produce the expected results?

m.sereda · August 1, 2018, 7:19am

no, could you tell me how to fix it?

magnusbaeck · August 1, 2018, 9:04am

Well, until the stdout output produces the desired results there's no point in looking in Kibana (and right now you don't even have an elasticsearch output). What output are you getting from Logstash? Does the log indicate that the script is being run? Is the cookie file being created?

m.sereda · August 1, 2018, 9:28am

I add an elasticsearch output

input {
  exec {
        command => "/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py"
        interval => 20
        data_format => "json"

        }
   request_timeout => 60
   schedule => { cron => "* * * * * UTC"}
}
   
output {
   elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
}
   stdout {
     codec => rubydebug 
   }  
}

but still no result

when I run the python script for the first and second time the outputs are like those and it is the same as in Postman

Cookie file created after the first running the script

magnusbaeck · August 1, 2018, 9:37am

Make sure the script output only contains the JSON data you want to load into Logstash. I repeat: What output are you getting from Logstash? Does the log indicate that the script is being run?

m.sereda · August 1, 2018, 9:50am

due to logstash-plain.log I have an error

[2018-08-01T11:43:14,097][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-01T11:43:15,633][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 130, column 20 (byte 2388) after input {\n  exec {\n\tcommand => \"/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py\"\n        interval => 20\n\t#id => \"myid\"\n\tdata_format => \"json\"\n\n\t}\n   request_timeout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-01T11:43:16,100][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-01T11:43:56,166][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-01T11:43:58,223][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 130, column 20 (byte 2388) after input {\n  exec {\n\tcommand => \"/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py\"\n        interval => 20\n\t#id => \"myid\"\n\tdata_format => \"json\"\n\n\t}\n   request_timeout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-01T11:43:58,744][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-01T11:44:37,468][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-01T11:44:39,641][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 130, column 20 (byte 2388) after input {\n  exec {\n\tcommand => \"/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py\"\n        interval => 20\n\t#id => \"myid\"\n\tdata_format => \"json\"\n\n\t}\n   request_timeout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-01T11:44:40,182][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Please tell me what do you mean/how to do that, I thought that I could see an output in Kibana

What output are you getting from Logstash?

And I see message=>"Expected one of #, { at line 130, column 20 (byte 2388)but I don't have 130 lines

tgaudin · August 1, 2018, 10:05am

Your input configuration is wrong, so the pipeline doesn't start at all.

The exec filter takes either interval or schedule configuration, and doesn't have a data_format option.
Also, you have pieces of your old http_poller configuration left, which you should remove.

A correct input block would be

input {
  exec {
    command => "/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py"
    interval => 20
    codec => "json"
  }
}

Also, regarding

Logstash combines all of your configuration files, so line numbers aren't accurate. You should look at the text of the error, "after input ... request_timeout " to know where the error is.

Topic		Replies	Views
How to post the http request with Authorization header , header content type and Body using logstash HTTP poller input plugin? Logstash	2	1406	June 7, 2019
HTTP Poller authentication: encode token Logstash	3	1647	September 5, 2018
Multiple questions about http_poller input plugin Logstash	4	144	April 11, 2024
Http_poller input fails authentication Logstash	1	11	August 6, 2024
Post data to logstash using http input Logstash	11	28569	January 17, 2017

POST method http_poller

Related Topics