How to get the json api data into message field in http poller using logsatsh config file

Elastic Stack Logstash
1

Hi,
Im using http poller to get API data output to be ingested into elastic search. I'm not getting any error but the message field is empty.

My config file is

input {
	stdin{}
	http_poller {
		urls => {
			q1 => {
				method => get
				user => "xxxx"
				password => "xxxxx"
				url => "https://xxxxxx?SchemaName=Dialogs_ChatTranscript&sessionId=2c4679fc-1335-453f-a5be-3b6d93eb165"
				headers => {
					Accept => "application/json"
				}
			}
		}
		request_timeout => 60
		schedule => { cron => "* * * * * UTC"}
		codec => multiline {
           pattern => "^{"
           negate => "true"
           what => "next"
       }
		metadata_target => "http_poller_metadata"
	}
}
filter {
	kv {
      source => message
      target => parsedjson
      field_split => "\n"
      value_split => ":\ "      
    }
	
}
output{
	stdout {
		codec => rubydebug
	}
}

My output is:

[2020-03-30T19:04:11,102][INFO ][logstash.inputs.http_poller][main] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2020-03-30T19:04:11,320][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2020-03-30T19:04:11,631][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
C:/elastic_stack/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
[2020-03-30T19:04:14,782][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
C:/elastic_stack/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
                "@version" => "1",
                 "message" => "\"\"",
    "http_poller_metadata" => {
                    "code" => 200,
                    "name" => "q1",
                    "host" => "LE000217",
                 "request" => {
               "auth" => {
                "eager" => true,
                 "user" => "xxxx",
                 "pass" => "xxxxx"
            },
             "method" => "get",
            "headers" => {
                "Accept" => "application/json"
            },
                "url" => "https://xxxxxxxxxxx?SchemaName=Dialogs_ChatTranscript&sessionId=2c4679fc-1335-453f-a5be-3b6d93eb165"
        },
         "runtime_seconds" => 31.616,
        "response_headers" => {
                     "expires" => "-1",
            "x-aspnet-version" => "4.0.30319",
                      "pragma" => "no-cache",
                "content-type" => "application/json; charset=utf-8",
                "x-powered-by" => "ASP.NET",
                        "date" => "Mon, 30 Mar 2020 13:35:32 GMT",
                      "server" => "Microsoft-IIS/8.5",
              "content-length" => "2",
               "cache-control" => "no-cache"
        },
        "response_message" => "OK",
           "times_retried" => 0
    },
              "@timestamp" => 2020-03-30T13:35:31.952Z
}

Please help me to get the api output in elastic search

2

If the response headers include that then there is no body in the response.

3

Hi Badger,

If the response has no body then how i will get it from API.
My API output which i got from postman tool is:

"[{\"attachment\":null,\"sessionId\":\"2c4679fc-1335-453f-a5be-3b6d93eb165b\",\"conversationId\":\"\",\"channel\":\"msteams\",\"from\":\"Bot\",\"fromName\":\"tryviumteamsdev\",\"fromId\":\"28:ce992a7e-aaed-4564-9b5d-c44d259cea9b\",\"to\":\"Agent\",\"toName\":\"User11 Agent\",\"toId\":\"29:19uGT1lGVMUu1JQPstwKKJ8rZi2JYXI3ngKREi534zFxlqddhDy_lI5OYV7ysfq0PhG6jVRPh8c1M5cwnS43AeA\",\"messageId\":\"c652e302-23c9-4a77-83fc-ae30c3e123bb\",\"messageText\":\"Your session has been ended. Thank you for contacting Service desk\",\"messageType\":\"text\",\"nlp\":{\"nlpEngine\":null,\"intent\":null,\"score\":null,\"entities\":null,\"category\":null,\"detectedBy\":null},\"qnaMaker\":{\"result\":null,\"score\":null},\"errorMessage\":null,\"createdOn\":\"2020-03-25T07:24:34.718\"}]"

Is my config file provided is correct?
Please help me in fixing this as i'm new to logstash

4

I think you should remove the multiline codec. If the content-type is application/json I believe a json codec will automatically get applied.

5

Hi Badger,

Thank you for your reply.
If i remove multiline codec then i'm getting the jsonparsefailure error.
Please see the output below:

[2020-03-31T10:50:01,348][ERROR][logstash.codecs.json     ][main] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>"\"\""}
C:/elastic_stack/logstash-7.6.0/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
                "@version" => "1",
              "@timestamp" => 2020-03-31T05:20:01.394Z,
                 "message" => "\"\"",
                    "tags" => [
        [0] "_jsonparsefailure"
    ],
    "http_poller_metadata" => {
         "runtime_seconds" => 1.106,
                    "code" => 200,
           "times_retried" => 0,
                    "name" => "q1",
        "response_headers" => {
                      "pragma" => "no-cache",
                     "expires" => "-1",
                        "date" => "Tue, 31 Mar 2020 05:20:01 GMT",
            "x-aspnet-version" => "4.0.30319",
              "content-length" => "2",
                "x-powered-by" => "ASP.NET",
                      "server" => "Microsoft-IIS/8.5",
               "cache-control" => "no-cache",
                "content-type" => "application/json; charset=utf-8"
        },
        "response_message" => "OK",
                    "host" => "LE000217",
                 "request" => {
                "url" => myurl,
             "method" => "get",
            "headers" => {
                "Content-Type" => "application/json",
                      "Accept" => "application/json"
            },
               "auth" => {
                 "user" => me,
                "eager" => true,
                 "pass" => mypassword
            }
        }
    }
}

Please help me out on this

6

The response from the .NET app your are calling is just "". There is a problem with your request, you would have to ask the folks who run the .NET app what that problem is.

7

Thank you very much Badger.
As you said the problem is with API only. I'm looking into that.
Thanks once again for your time to reply me.
cheers!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.