Logstash http poller- custom header : error

Elastic Stack Logstash
1

I am trying to use an API url to get data from a source.

My api has a custom Header for aw-tenant-code, which works well with a curl.

Here is my sample data:

input {
  http_poller {
    urls => {
    snowsr =>  {
    url =>  "https://url.com/API/system/users/enrolleddevices/search"
    user => "elk"
    password => "********"
    headers => {
        Accept => "application/json"
        aw-tenant-code => "TN37b6rpCi7hJZkdGINQkDukaxccv5bRYOELNm0xB2I="
}
}
}
request_timeout => 60
metadata_target => "http_poller_metadata"
schedule => { cron => "* * * * *"}
#codec => "json"
}
}

output {
  elasticsearch {
    hosts =>  ["16.75.729:9200"]
    index => "workspace"
}
        stdout { codec => rubydebug }
}

Sample data in data source:

{
   "EnrolledDeviceInfoList":[
      {
         "DeviceID":24,
         "SerialNumber":"FVJ1WT",
         "AssetNumber":"8D1ACE6812",
         "FriendlyName":"MacBook Air",
         "UserName":"UM2",
         "EnrolledDate":"03/02/2020 16:05:22",
         "RegistrationDate":"01/01/0001 00:00:00",
         "LastSeen":"06/14/2020 07:59:01",
         "Platform":"AppleOsX",
         "OrganizationGroup":"Corp",
         "CustomAttributes":[

         ]
      },
      {
         "DeviceID":104,
         "SerialNumber":"HT3",
         "AssetNumber":"",
         "FriendlyName":"Desktop Windows Desktop",
         "UserName":"UWI",
         "EnrolledDate":"02/11/2020 19:26:58",
         "RegistrationDate":"01/01/0001 00:00:00",
         "LastSeen":"06/16/2020 13:29:24",
         "Platform":"WinRT",
         "OrganizationGroup":"Corp",
         "CustomAttributes":[

         ]
      }
   ]
}

I get an error as below:

[2020-06-16T10:51:33,198][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:workspace, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of
#, => at line 10, column 4 (byte 237) after input {\n  http_poller {\n    urls => {\n    snowsr =>  {\n    url =>  \"https://url.com/API/system/users/enrolleddevices/search\"\n    user
=> \"elk\"\n    password => \"*********\"\n    headers => {\n\tAccept => \"application/json\"\n\taw", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

What am i missing here? How do i split and get index for each device here. I am not sure why not one row of got loaded. Please help me.

2

Try putting quotes around aw-tenant-code.

3

@Badger
the below worked,

input {
  http_poller {
    urls => {
    snowsr =>  {
    url =>  "https://url.com/API/system/users/enrolleddevices/search"
    user => "elkuem"
    password => "**********"
 headers => {"aw-tenant-code" => "TN5bRYOELNm0xB2I="}
}
}
request_timeout => 60
metadata_target => "http_poller_metadata"
schedule => { cron => "* * * * *"}
#codec => "json"
}
}
filter
{
 json {
                source => "EnrolledDeviceInfoList"
        }

   split
        {
        field => "EnrolledDeviceDetails"
}
}

However the data isnt processed properly.

I get the below error:

2020-06-16T12:52:02,175][ERROR][logstash.codecs.json     ] stomAttributes
</EnrolledDeviceDetails><EnrolledDeviceDetails><DeviceID>19</DeviceID><SerialNumber>C02ZW6R</SerialNumber><AssetNumber>0B2DF2AC</AssetNumber><FriendlyName>MacBook Pro</FriendlyName><UserName>UA</UserName><EnrolledDate>06/15/2020 11:45:58</EnrolledDate><RegistrationDate>01/01/0001 00:00:00</RegistrationDate><LastSeen>06/16/2020 16:22:26</LastSeen><Platform>AppleOsX</Platform><OrganizationGroup>Corp</OrganizationGroup><CustomAttributes /></EnrolledDeviceDetails><EnrolledDeviceDetails><DeviceID>18</DeviceID><SerialNumber>C02Z</SerialNumber><AssetNumber>E04306632</AssetNumber><FriendlyName>MacBook Pro </FriendlyName><UserName>UJC</UserName><EnrolledDate>06/16/2020 16:07:37</EnrolledDate><RegistrationDate>01/01/0001 00:00:00</RegistrationDate><LastSeen>06/16/2020 16:33:03</LastSeen><Platform>AppleOsX</Platform><OrganizationGroup>Prod_Corp</OrganizationGroup><CustomAttributes /></EnrolledDeviceDetails></EnrolledDeviceInfoList></EnrolledDevices>"

They seem to load as xml here, however, when i curl, the data is in json.

How can i correct this?
Please help me out.

Katara

4

That error message does not look right, can you edit your post, select the error message and click on </> in the toolbar?

5

Done! @Badger.
Please check now.

6

The default codec for http_poller is json. To debug this I suggest changing the code on the input to plain, the changing the output to

output { stdout { codec => rubydebug } }

and see if the message looks like what you get from curl, and whether it is properly formed JSON.

7

Hi @Badger,
I found the issue to be with accept => "/application/json"

Thank you very much for helping out!
Katara

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.