katara
(Katara)
May 16, 2020, 10:37am
1
Hi,
Im trying to send my data to a php input, but i have a few questions:
Here's my config:
> input {
> tcp {
> port => 8443
> codec => json_lines { charset => CP1252 }
> }
> }
> filter
> {
> ruby { code => 'event.set("logmessage", Base64.encode64(event.get("Message")))' }
> ruby { code => 'event.set("device", Base64.encode64(event.get("host")))' }
> ruby { code => 'event.set("sev", Base64.encode64(event.get("Severity")))' }
> ruby { code => 'event.set("tagpath", Base64.encode64(event.get("tag_filepath")))' }
> ruby { code => 'event.set("appname", Base64.encode64(event.get("ApplicationName")))' }
> }
> output {
> if [ApplicationName] == "OASIS"
> {
> elasticsearch {
> hosts => ["10.16.5.2:9200"]
> user => "elastic"
> password => "elk@123"
> index => "apps"
> }
> }
> else
> {
> elasticsearch {
> hosts => ["10.16.5.2:9200"]
> user => "elastic"
> password => "changeme"
> index => "errors"
> }
> }
> if [Severity] == "Error"
> {
> exec
> {
> command => "php tcp.php %{logmessage} %{$device} %{sev} %{tagpath} %{appname}"
> }
> }
> stdout { codec => rubydebug }
> }
My php send these ina json format to an api url.
I tried to encode the fields with base64 in ruby.
since my command is
> php /etc/logstash/conf.d/tcp.php %{logmessage} %{$device} %{sev} %{tagpath} %{appname}
if the message is something like this:
> 2020-01-01 23:45:45,034 Failed to reload application URL
then the spaces might be taken wrongly as next element as
> logmessage: 2020-01-01
> device: 23:45:45:,034
> and so on., which is not my expectation.
Is this a fair expectation and if yes, is my encoding method right?
After i did this, no data has been sent to the url.
I tested my php on command line with sample inputs, which worked fine.
What am i missing?
Please provide me your suggestions and help.
Can you show an example of an event output using rubydebug?
BTW, you set the field [device] but reference [$device]
katara
(Katara)
May 16, 2020, 7:56pm
3
@badger ,
Noted, Thank you!
My logstash output is something like this,
I mapped my output to a file and got this. The messages may always vary, but here it goes.
> {"@version":"1","host":"strproelk03","message":"Retrying 99.61 - - [12/May/2020:23:24:06 -0600] \"GET /robots.txt HTTP/1.1\" 404 443 208","ApplicationName":"Oasis","Severity":"ERROR","@timestamp":"2020-05-16T19:46:25.908Z","tagfile_path":"/etc/logstash/conf.d/nodejs/node.log","EventReceivedTime":"2020-05-16T19:46:25.908Z"}
Firstly, "ERROR" != "Error", so that message would not go to the exec output. Secondly, that event has a [message] field, but not a [Message] field.
katara
(Katara)
May 16, 2020, 9:29pm
5
@Badger thank you very much! Corrected ERROR, device and Message to match the same as my inputs. Is the ruby filter done right?
Badger
May 16, 2020, 10:53pm
6
It looks reasonable to me. Although I would do it all in one filter...
ruby {
code => '
event.set("logmessage", Base64.encode64(event.get("Message")))
event.set("device", Base64.encode64(event.get("host")))
event.set("sev", Base64.encode64(event.get("Severity")))
event.set("tagpath", Base64.encode64(event.get("tag_filepath")))
event.set("appname", Base64.encode64(event.get("ApplicationName")))
'
}
katara
(Katara)
May 18, 2020, 4:51am
8
@Badger ,
I think the data in the ruby fields is not getting added. i dont see them in Es.
Should the calling of the column be any different as [message] or anything?
is there a different usage for my data push? I think something is wrong here in this case.
`
command => "php tcp.php %{logmessage} %{$device} %{sev} %{tagpath} %{appname}"
`
I tried sending data directly from my php file,
as
php tcp.php messagesample deveice1 error /elk logstash
and this sent the message successfully.
Please help me resolve this.
katara
(Katara)
May 18, 2020, 9:46am
9
Update: I see the data in ES, but sill not passed to the php file.
@Badger
Badger
May 18, 2020, 3:07pm
10
Enable loglevel debug and check the logstash logs to see what stderr and stdout look like when the command is exec'd.
katara
(Katara)
May 18, 2020, 3:10pm
11
@Badger I think i found the issue,
For some reason, it is not taking up un manipulated fields.
So I did a mutate gsub just as with no reason to remove some slashes, etc.
Once It flowed through it, i was able to get this successful.
weirdly, that's how it worked out for me!
Thank you for helping me out figure this all out!
Katara!
system
(system)
Closed
June 15, 2020, 3:10pm
12
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.