POST method http_poller

[m.sereda]

Should I change codec to "json_lines"?

Here is a log with comment out of codec

This one with codec "json_line"

[Badger]

OK, so your script is producing no output. And if there is no output a json codec consumes the event without emitting anything.

Thus

input { exec { command => "echo ''" codec => json interval => 10 } }
output { stdout { codec => rubydebug } }

does not produce an event, although

input { exec { command => "echo ''" interval => 10 } }
output { stdout { codec => rubydebug } }

does.

What do you get when you run '/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py' on the command line?

[m.sereda]

The output of python script is the following

[m.sereda]

Any recomendations?

[m.sereda]

Do you have any recomendation how to fix?

[magnusbaeck (Magnus Bäck)]

You can change the command being run to

/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py 2>&1

so that stderr output is included in the output. What happens?

Also, how are you starting Logstash?

[m.sereda]

input {
  exec {
        command => "/usr/bin/python /home/maksym/PathFolder/pythonpractice/postsample.py 2>&1"
        interval => 20
        codec => "json_lines"
        } 
}

output {
  # elasticsearch {
#       hosts => ["localhost:9200"]
#       index => "logstash-%{+YYYY.MM.dd}"
#}
   stdout { 
     codec => rubydebug
   }
}

Here are a logstash.conf and log output
I am starting logstash by systemctl start logstash

[magnusbaeck (Magnus Bäck)]

Well, there you have it. The logstash user can't run the script in your home directory.

[m.sereda]

I have changed directory for python script and cookiejar file to /etc/logstash. Now I have a such log

[tgaudin]

As you showed us here POST method http_poller, your script outputs the string "setting cookies" or "loading saved cookies", which is not JSON. You should not print this in your script.

Also, if it still looks like the script you shared in this post POST method http_poller, you probably will need to not print r.status_code, r.cookies, or anything that isn't the JSON response from your server.

[m.sereda]

OK, sir I've made some changes with the path and now I have a such log, cookiejar was created by logstash

[m.sereda]

Ok I have changed it:

import os
os.chdir("/var/lib/logstash")
from cookielib import LWPCookieJar

import requests

s = requests.Session()
s.cookies = LWPCookieJar('cookiejar')
if not os.path.exists('cookiejar'):
    # Create a new cookies file and set our Session's cookies
    #print('setting cookies')
    s.cookies.save()
    r = s.post('http://localhost:8080/api/session/login', json={'login': 'login', 'password': 'pass'})

else:
    # Load saved cookies from the file and use them in a request
    #print('loading saved cookies')
    s.cookies.load(ignore_discard=True)
    r = s.get('http://localhost:8080/api/dashboard/staging-space')
print(r.text)
# Save the session's cookies back to the file
s.cookies.save(ignore_discard=True)

Is everything is OK for now?

[tgaudin]

That looks OK, if your API is supposed to send availableSpace or node: {...} information it's in the event. If I remember correctly, the stdout output can be seen in journalctl -fu logstash if you want to make sure the events look like you want. Then you can uncomment the Elasticsearch output and see if Kibana picks up the documents.

(Also, copy-pasting logs is easier to read and is indexable for search engines, better than screenshots)

[m.sereda]

OMG, finally it is working! You tgaudin and magnusbaeck are my heroes! Thank you guys!

p.s. sorry i have to sent a screenshot due to maximum number of characters.