[LOGSTASH] Input http-poller POST

Bonjour tout le monde,

J'essaye dans Logstash, de contacter une API REST au travers d'une méthode POST quoi doit me retourner un document JSON.

Voici ma configuration:

input {
      http_poller {
        urls => {
         getIssueForGZLProjectVersion => {
            method => post
            user => "bcu"
            password => "changeit"
            url => "https://URL/jira/rest/api/2/search"
            body => '{"jql":"project = GZL && fixVersion = 19088"}'
            headers => {
              Accept => "application/json"
            }
          }
        }
        request_timeout => 60
        schedule => { cron => "* * * * * UTC"}
        codec => "json"
        metadata_target => "http_poller_metadata"
      }
    }

Voici le message qui est retourné à Logstash:

{
              "@timestamp" => 2019-07-10T06:31:00.397Z,
                "@version" => "1",
    "http_poller_metadata" => {
                    "code" => 415,
           "times_retried" => 0,
        "response_headers" => {
                 "x-seraph-loginreason" => "OK",
                     "x-xss-protection" => "1; mode=block",
                           "set-cookie" => [
                [0] "JSESSIONID=F6A8F064F0B411EADD5B8489C2C2A33B;path=/jira;Secure;HttpOnly",
                [1] "atlassian.xsrf.token=ALTQ-FBWF-E57B-SBIP_90f1f981930daf2276fdd632d4b847e6444f27c5_lin;path=/jira;Secure"
            ],
                          "x-ausername" => "bcu",
              "content-security-policy" => "frame-ancestors 'self'",
               "x-content-type-options" => "nosniff",
                           "connection" => "Keep-Alive",
                               "x-asen" => "SEN-457387",
                                 "vary" => "User-Agent",
                           "keep-alive" => "timeout=5, max=300",
                               "server" => "Apache",
                         "x-asessionid" => "16kd6yx",
                         "x-arequestid" => "511x1069620x1",
                                 "date" => "Wed, 10 Jul 2019 06:31:00 GMT",
                      "x-frame-options" => "sameorigin",
                         "content-type" => "text/html;charset=UTF-8",
            "strict-transport-security" => "max-age=63072000; includeSubdomains;",
                    "transfer-encoding" => "chunked"
        },
        "response_message" => "415",
                    "host" => "logstash-jira-pipeline",
                 "request" => {
               "auth" => {
                "eager" => true,
                 "pass" => "changeit",
                 "user" => "bcu"
            },
                "url" => "https://URL/jira/rest/api/2/search",
               "body" => "{\"jql\":\"project = GZL && fixVersion = 19088\"}",
            "headers" => {
                "Accept" => "application/json"
            },
             "method" => "post"
        },
         "runtime_seconds" => 0.201462,
                    "name" => "getIssueForGZLProjectVersion"
    }

Je reçois une erreur HTTP de type 415.

Cependant, quand j'utilise la commande suivante à l'aide de curl:

curl -D- -u bcu:changeit -X POST --header "Content-Type: application/json" --data '{"jql":"project = GZL && fixVersion = 19088"}' https://URL/jira/rest/api/2/search

Je reçois bien une réponse avec un document JSON:

{
  "expand": "schema,names",
  "startAt": 0,
  "maxResults": 50,
  "total": 10,
  "issues": [
    {
      "expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
      "id": "35559",
      "self": "https://URL/jira/rest/api/2/issue/35559",
      "key": "GZL-4870",
      "fields": {
        "issuetype": {
          "self": "https://URL/jira/rest/api/2/issuetype/1",
          "id": "1",
          "description": "",
          "iconUrl": "https:/URL/jira/secure/viewavatar?size=xsmall&avatarId=11273&avatarType=issuetype",
          "name": "Bogue",
          "subtask": false,
          "avatarId": 11273
        },
        [...]

Une idée ?
Merci pour votre aide

Je pense qu'il te manque l'équivalent de Content-Type: application/json...
Du coup je mettrais:

headers => {
              Accept => "application/json"
              Content-Type => "application/json"
            }

Merci @dadoonet pour ton retour.

Alors, lorsque que je rajoute:

headers => {
              Accept => "application/json"
              Content-Type => "application/json"
            }

J'ai l'erreur suivante:

[2019-07-10T07:09:54,525][ERROR][logstash.agent           ] Failed to execute action 

{:action=>LogStash::PipelineAction::Create/pipeline_id:main,exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 12, column 18 (byte 369) after input {\n http_poller {\n urls => {\n getIssueForGZLProjectVersion => {\n method => post\n user => "bcu"\n password => "changeme"\n url => "https://URL/jira/rest/api/2/search"\n body => '{"jql":"project = GZL && fixVersion = 19088"}'\n headers => {\n Accept => "application/json"\n Content", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}

Il semblerait qu'il manque une virgule donc j'ai ajouté cette dernière:

headers => {
          Accept => "application/json",
          Content-Type => "application/json"
        }

Mais le problème persiste:

[2019-07-10T07:12:54,906][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 11, column 39 (byte 351) after input {\n  http_poller {\n    urls => {\n     getIssueForGZLProjectVersion => {\n        method => post\n        user => \"bcu\"\n        password => \"changeme\"\n        url => \"https://URL/jira/rest/api/2/search\"\n        body => '{\"jql\":\"project = GZL && fixVersion = 19088\"}'\n        headers => {\n          Accept => \"application/json\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

Si je ne mets que:

headers => {
          Content-Type => "application/json"
        }

J'ai aussi une erreur:

[2019-07-10T07:16:35,355][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 11, column 18 (byte 330) after input {\n  http_poller {\n    urls => {\n     getIssueForGZLProjectVersion => {\n        method => post\n        user => \"bcu\"\n        password => \"changeme\"\n        url => \"https://URL/jira/rest/api/2/search\"\n        body => '{\"jql\":\"project = GZL && fixVersion = 19088\"}'\n        headers => {\n          Content", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

Par contre, si je ne mets que:

headers => {
          Accept => "application/json"
        }

Je n'ai pas d'erreur

J'ai trouvé @dadoonet, il faillait mettre entre double quote:

headers => {
          Accept => "application/json"
          "Content-Type" => "application/json"
        }

Je reçois correctement mon document JSON.

Merci et bonne journée

Ha ! C'est à cause du tiret en fait.
Du coup, je pense qu'il faudrait être rigoureux et mettre:

headers => {
  "Accept" => "application/json"
  "Content-Type" => "application/json"
}

@dadoonet: Ha oui en effet :wink:

J'aurai une autre petite question.

J'essaye d'utiliser deux input de type http_poller car j'aurai besoin d'ajouter des field en fonction des urls requêtées:

input {
  http_poller {
    urls => {
     getIssuesForFHIRValidatorProjectAndFixVersion => {
        method => post
        user => "bcu"
        password => "changeit"
        url => "https://URL/jira/rest/api/2/search"
        body => '{"jql":"project = FHIRVAL AND fixVersion = 3.0.3","fields":["id","key","self","priority","issuetype"]}'
        headers => {
          Accept => "application/json"
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
    add_field => {
       "service.name" => "FHIR Validator"
       "service.id" => "13111"
       "service.key" => "FHIRVAL"
       "service.version" => "3.0.3"
    }
    #metadata_target => "http_poller_metadata"
  }
  http_poller {
    urls => {
     getIssuesForGazelleTMProjectAndFixVersion => {
        method => post
        user => "bcu"
        password => "changeit"
        url => "https://URL/jira/rest/api/2/search"
        body => '{"jql":"project = GZL AND fixVersion = 5.1.1","fields":["id","key","self","priority","issuetype"]}'
        headers => {
          Accept => "application/json"
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
    add_field => {
       "service.name" => "Gazelle Test Management"
       "service.id" => "10013"
       "service.key" => "GZL"
       "service.version" => "5.1.1"
    }
    #metadata_target => "http_poller_metadata"
  }

Je n'ai pas d'erreur quand je lance logstash et j'ai l'impression qu'il comprend bien qu'il doit ouvrir deux input de type http_poller:

[2019-07-10T09:32:41,062][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2019-07-10T09:32:41,548][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}

Cependant, il n'y a que le http_poller portant le nom getIssuesForFHIRValidatorProjectAndFixVersion qui est exécuté.

Aurais-tu une idée ?

Aucune idée. C'est peut être un bug. @colinsurprenant ton avis ?

Néanmoins suivant ton cas, tu peux aussi faire 2 pipelines LS différents, chacun avec son input.

Oui j'ai pensé aussi à ça de faire deux pipelines.

Je vais testé pour voir

@dadoonet

Autant pour moi, c'était un élément dans le body qui n'était pas correct, ce qui avait pour conséquence de me retourner une réponse vide.

Du coup tout fonctionne.

Désolé pour le dérangement et merci pour ton aide

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.