POST method http_poller

Elastic Stack Logstash
1

Trying to make a POST method to login but smth is going wrong. To make a login I have to fill "login" and "password" poles which is in JSON. Where is a mistake?

input {
  http_poller {
    urls => {
      soap_request => {
        method => post
        url => "http://localhost:8080/api/ses/login"
        headers => {
          Content-Type => "application/json"
        }
        body => '{"login": "login", "password": "password"}'
      }
    }
    request_timeout => 10
    interval => 10
    codec => "plain"
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

After I need to take a cookie and send it to GET method http://localhost:8080/api/my/destination , how to do that?

2

Trying to make a POST method to login but smth is going wrong.

Well, what happens?

After I need to take a cookie and send it to GET method http://localhost:8080/api/my/destination , how to do that?

I don't think there's any convenient way of doing that.

3

Hmm... is it possible to write any script or somehow to do that?

4

Sure, you could make the HTTP call in a separate script that you invoke with an exec plugin.

5

Could you please help me to do that or is there any tutorial?

How this script should looks like? what type ?

6

You can write such a script in any language. I don't have an example at hand.

7

Well, what happens?

Here is my logstash.conf with POST method and it is not working. Do I put login /password in a good way? The format which I should put is JSON

input {
  http_poller {
    urls => {
      test1 => {
        url => "http://localhost:8080/api/sess/login"
        method => "post"
        headers => {
          Accept => "application/json"
        }
        body => '{"login":"login", 
        "password":"pass"}'
      
    }
8

Here is my logstash.conf with POST method and it is not working.

That doesn't answer the question "what happens?". Are you getting any response at all from the server? If yes, what does it look like?

9

I got a 415 error

@version:1 message:<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>Payara Micro #badassfish - Error report</title><style type="text/css"><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:whit

10

415 is "Unsupported media type". Perhaps you need to send "Content-type: application/json" as an additional HTTP header in your request? Otherwise I don't know. Try using e.g. Wireshark to capture a successful request and compare it to the one Logstash makes.

11

OK, I've made a python script with saving a cookie in a file and it is working

import os
from cookielib import LWPCookieJar

import requests

s = requests.Session()
s.cookies = LWPCookieJar('cookiejar')
if not os.path.exists('cookiejar'):
    # Create a new cookies file and set our Session's cookies
    print('setting cookies')
    s.cookies.save()
    r = s.post('http://localhost:8080/api/ses/login', json={'login': 'login', 'password': 'pass'})
else:
    # Load saved cookies from the file and use them in a request
    print('loading saved cookies')
    s.cookies.load(ignore_discard=True)
    r = s.post('http://localhost:8080/api/ses/login', json={'login': 'login', 'password': 'pass'})
#print(r.text)
# Save the session's cookies back to the file
s.cookies.save(ignore_discard=True)

print(r.status_code)
#print(r.json())
print(r.cookies)

Cookie file

#LWP-Cookies-2.0
Set-Cookie3: JSESSIONID=aae34175e7cea354b77b29020288; path="/"; domain="localhost.local"; path_spec; discard; HttpOnly=None; version=0

and trying to use an exec plugin. Don't understand how to send cookies from that scrypt to logstash

input {
  exec {
        command => "python /home/maksym/PathFolder/pythonpractice/postsample.py"
        interval => 20
        id => "plugin_id"
        }
  http_poller {
    urls => {
       test2 => {
       url => "http://localhost:8080/api/dash/space"
       method => "get"
       headers => {
         Accept => "application/json"
         Cookie => "/home/maksym/PathFolder/pythonpractice/cookiejar"
       }
    }
   }
   request_timeout => 60
   # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
   schedule => { cron => "* * * * * UTC"}
   codec => "json"
   # A hash of request metadata info (timing, response headers, etc.) will be sent here
   metadata_target => "http_poller_metadata"
   }
}

output {
   stdout {
     codec => rubydebug
   }
}

For now the output is

Screenshot%20from%202018-07-30%2012-30-52
Screenshot from 2018-07-30 12-30-52.png1176×548 58.4 KB

Appreciate any help. Thanks

12

Do everything from the Python script. Don't try to pass the cookies from the Python script to the http_poller input.

13

OK, now I think I understand. I changed my logstash.conf as the following

input {
  exec {
        command => "python /home/maksym/PathFolder/pythonpractice/postsample.py"
        interval => 20
        id => "plugin_id"  
        }
   request_timeout => 60
   schedule => { cron => "* * * * * UTC"}
   codec => "json"
}
      
output {
   stdout { 
     codec => rubydebug
   }
}

and python.py like this:

import os
os.chdir("/home/maksym/PathFolder")
from cookielib import LWPCookieJar

import requests

s = requests.Session()
s.cookies = LWPCookieJar('cookiejar')
if not os.path.exists('cookiejar'):
    # Create a new cookies file and set our Session's cookies
    print('setting cookies')
    s.cookies.save()
    r = s.post('http://localhost:8080/api/sess/login', json={'login': 'login', 'password': 'pass'})
    print(r.text)
else:
    # Load saved cookies from the file and use them in a request
    print('loading saved cookies')
    s.cookies.load(ignore_discard=True)
    r = s.get('http://localhost:8080/api/dash/space')
print(r.text)
# Save the session's cookies back to the file
s.cookies.save(ignore_discard=True)

print(r.status_code)
#print(r.json())
print(r.cookies)

and when I running the script from the console it is working, but from logstash no - nothing showing in Kibana. Could you tell me where I have made a mistake?

14

Does your stdout output produce the expected results?

15

no, could you tell me how to fix it?

16

Well, until the stdout output produces the desired results there's no point in looking in Kibana (and right now you don't even have an elasticsearch output). What output are you getting from Logstash? Does the log indicate that the script is being run? Is the cookie file being created?

17

I add an elasticsearch output

input {
  exec {
        command => "/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py"
        interval => 20
        data_format => "json"

        }
   request_timeout => 60
   schedule => { cron => "* * * * * UTC"}
}
   
output {
   elasticsearch {
        hosts => ["localhost:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
}
   stdout {
     codec => rubydebug 
   }  
}

but still no result

Screenshot%20from%202018-08-01%2011-18-42
Screenshot from 2018-08-01 11-18-42.png939×462 35.3 KB

when I run the python script for the first and second time the outputs are like those and it is the same as in Postman

Screenshot%20from%202018-08-01%2011-27-51
Screenshot from 2018-08-01 11-27-51.png1277×371 271 KB

Cookie file created after the first running the script

18

Make sure the script output only contains the JSON data you want to load into Logstash. I repeat: What output are you getting from Logstash? Does the log indicate that the script is being run?

19

due to logstash-plain.log I have an error

[2018-08-01T11:43:14,097][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-01T11:43:15,633][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 130, column 20 (byte 2388) after input {\n  exec {\n\tcommand => \"/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py\"\n        interval => 20\n\t#id => \"myid\"\n\tdata_format => \"json\"\n\n\t}\n   request_timeout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-01T11:43:16,100][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-01T11:43:56,166][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-01T11:43:58,223][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 130, column 20 (byte 2388) after input {\n  exec {\n\tcommand => \"/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py\"\n        interval => 20\n\t#id => \"myid\"\n\tdata_format => \"json\"\n\n\t}\n   request_timeout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-01T11:43:58,744][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-08-01T11:44:37,468][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
[2018-08-01T11:44:39,641][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 130, column 20 (byte 2388) after input {\n  exec {\n\tcommand => \"/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py\"\n        interval => 20\n\t#id => \"myid\"\n\tdata_format => \"json\"\n\n\t}\n   request_timeout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
[2018-08-01T11:44:40,182][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Screenshot%20from%202018-08-01%2011-51-21
Screenshot%20from%202018-08-01%2011-51-211366×768 188 KB

Please tell me what do you mean/how to do that, I thought that I could see an output in Kibana

What output are you getting from Logstash?

And I see message=>"Expected one of #, { at line 130, column 20 (byte 2388)but I don't have 130 lines

20

Your input configuration is wrong, so the pipeline doesn't start at all.

The exec filter takes either interval or schedule configuration, and doesn't have a data_format option.
Also, you have pieces of your old http_poller configuration left, which you should remove.

A correct input block would be

input {
  exec {
    command => "/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py"
    interval => 20
    codec => "json"
  }
}

Also, regarding

Logstash combines all of your configuration files, so line numbers aren't accurate. You should look at the text of the error, "after input ... request_timeout " to know where the error is.