[LOGSTASH] Input http-poller POST

Benjamin_Carriou · July 10, 2019, 6:35am

Bonjour tout le monde,

J'essaye dans Logstash, de contacter une API REST au travers d'une méthode POST quoi doit me retourner un document JSON.

Voici ma configuration:

input {
      http_poller {
        urls => {
         getIssueForGZLProjectVersion => {
            method => post
            user => "bcu"
            password => "changeit"
            url => "https://URL/jira/rest/api/2/search"
            body => '{"jql":"project = GZL && fixVersion = 19088"}'
            headers => {
              Accept => "application/json"
            }
          }
        }
        request_timeout => 60
        schedule => { cron => "* * * * * UTC"}
        codec => "json"
        metadata_target => "http_poller_metadata"
      }
    }

Voici le message qui est retourné à Logstash:

{
              "@timestamp" => 2019-07-10T06:31:00.397Z,
                "@version" => "1",
    "http_poller_metadata" => {
                    "code" => 415,
           "times_retried" => 0,
        "response_headers" => {
                 "x-seraph-loginreason" => "OK",
                     "x-xss-protection" => "1; mode=block",
                           "set-cookie" => [
                [0] "JSESSIONID=F6A8F064F0B411EADD5B8489C2C2A33B;path=/jira;Secure;HttpOnly",
                [1] "atlassian.xsrf.token=ALTQ-FBWF-E57B-SBIP_90f1f981930daf2276fdd632d4b847e6444f27c5_lin;path=/jira;Secure"
            ],
                          "x-ausername" => "bcu",
              "content-security-policy" => "frame-ancestors 'self'",
               "x-content-type-options" => "nosniff",
                           "connection" => "Keep-Alive",
                               "x-asen" => "SEN-457387",
                                 "vary" => "User-Agent",
                           "keep-alive" => "timeout=5, max=300",
                               "server" => "Apache",
                         "x-asessionid" => "16kd6yx",
                         "x-arequestid" => "511x1069620x1",
                                 "date" => "Wed, 10 Jul 2019 06:31:00 GMT",
                      "x-frame-options" => "sameorigin",
                         "content-type" => "text/html;charset=UTF-8",
            "strict-transport-security" => "max-age=63072000; includeSubdomains;",
                    "transfer-encoding" => "chunked"
        },
        "response_message" => "415",
                    "host" => "logstash-jira-pipeline",
                 "request" => {
               "auth" => {
                "eager" => true,
                 "pass" => "changeit",
                 "user" => "bcu"
            },
                "url" => "https://URL/jira/rest/api/2/search",
               "body" => "{\"jql\":\"project = GZL && fixVersion = 19088\"}",
            "headers" => {
                "Accept" => "application/json"
            },
             "method" => "post"
        },
         "runtime_seconds" => 0.201462,
                    "name" => "getIssueForGZLProjectVersion"
    }

Je reçois une erreur HTTP de type 415.

Cependant, quand j'utilise la commande suivante à l'aide de curl:

curl -D- -u bcu:changeit -X POST --header "Content-Type: application/json" --data '{"jql":"project = GZL && fixVersion = 19088"}' https://URL/jira/rest/api/2/search

Je reçois bien une réponse avec un document JSON:

{
  "expand": "schema,names",
  "startAt": 0,
  "maxResults": 50,
  "total": 10,
  "issues": [
    {
      "expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
      "id": "35559",
      "self": "https://URL/jira/rest/api/2/issue/35559",
      "key": "GZL-4870",
      "fields": {
        "issuetype": {
          "self": "https://URL/jira/rest/api/2/issuetype/1",
          "id": "1",
          "description": "",
          "iconUrl": "https:/URL/jira/secure/viewavatar?size=xsmall&avatarId=11273&avatarType=issuetype",
          "name": "Bogue",
          "subtask": false,
          "avatarId": 11273
        },
        [...]

Une idée ?
Merci pour votre aide

dadoonet · July 10, 2019, 6:55am

Je pense qu'il te manque l'équivalent de Content-Type: application/json...
Du coup je mettrais:

headers => {
              Accept => "application/json"
              Content-Type => "application/json"
            }

Benjamin_Carriou · July 10, 2019, 7:15am

Merci @dadoonet pour ton retour.

Alors, lorsque que je rajoute:

headers => {
              Accept => "application/json"
              Content-Type => "application/json"
            }

J'ai l'erreur suivante:

[2019-07-10T07:09:54,525][ERROR][logstash.agent           ] Failed to execute action

{:action=>LogStash::PipelineAction::Create/pipeline_id:main,exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 12, column 18 (byte 369) after input {\n http_poller {\n urls => {\n getIssueForGZLProjectVersion => {\n method => post\n user => "bcu"\n password => "changeme"\n url => "https://URL/jira/rest/api/2/search"\n body => '{"jql":"project = GZL && fixVersion = 19088"}'\n headers => {\n Accept => "application/json"\n Content", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:ininitialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}

Il semblerait qu'il manque une virgule donc j'ai ajouté cette dernière:

headers => {
          Accept => "application/json",
          Content-Type => "application/json"
        }

Mais le problème persiste:

[2019-07-10T07:12:54,906][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 11, column 39 (byte 351) after input {\n  http_poller {\n    urls => {\n     getIssueForGZLProjectVersion => {\n        method => post\n        user => \"bcu\"\n        password => \"changeme\"\n        url => \"https://URL/jira/rest/api/2/search\"\n        body => '{\"jql\":\"project = GZL && fixVersion = 19088\"}'\n        headers => {\n          Accept => \"application/json\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

Benjamin_Carriou · July 10, 2019, 7:17am

Si je ne mets que:

headers => {
          Content-Type => "application/json"
        }

J'ai aussi une erreur:

[2019-07-10T07:16:35,355][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 11, column 18 (byte 330) after input {\n  http_poller {\n    urls => {\n     getIssueForGZLProjectVersion => {\n        method => post\n        user => \"bcu\"\n        password => \"changeme\"\n        url => \"https://URL/jira/rest/api/2/search\"\n        body => '{\"jql\":\"project = GZL && fixVersion = 19088\"}'\n        headers => {\n          Content", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

Par contre, si je ne mets que:

headers => {
          Accept => "application/json"
        }

Je n'ai pas d'erreur

Benjamin_Carriou · July 10, 2019, 7:23am

J'ai trouvé @dadoonet, il faillait mettre entre double quote:

headers => {
          Accept => "application/json"
          "Content-Type" => "application/json"
        }

Je reçois correctement mon document JSON.

Merci et bonne journée

dadoonet · July 10, 2019, 8:44am

Ha ! C'est à cause du tiret en fait.
Du coup, je pense qu'il faudrait être rigoureux et mettre:

headers => {
  "Accept" => "application/json"
  "Content-Type" => "application/json"
}

Benjamin_Carriou · July 10, 2019, 9:34am

@dadoonet: Ha oui en effet

J'aurai une autre petite question.

J'essaye d'utiliser deux input de type http_poller car j'aurai besoin d'ajouter des field en fonction des urls requêtées:

input {
  http_poller {
    urls => {
     getIssuesForFHIRValidatorProjectAndFixVersion => {
        method => post
        user => "bcu"
        password => "changeit"
        url => "https://URL/jira/rest/api/2/search"
        body => '{"jql":"project = FHIRVAL AND fixVersion = 3.0.3","fields":["id","key","self","priority","issuetype"]}'
        headers => {
          Accept => "application/json"
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
    add_field => {
       "service.name" => "FHIR Validator"
       "service.id" => "13111"
       "service.key" => "FHIRVAL"
       "service.version" => "3.0.3"
    }
    #metadata_target => "http_poller_metadata"
  }
  http_poller {
    urls => {
     getIssuesForGazelleTMProjectAndFixVersion => {
        method => post
        user => "bcu"
        password => "changeit"
        url => "https://URL/jira/rest/api/2/search"
        body => '{"jql":"project = GZL AND fixVersion = 5.1.1","fields":["id","key","self","priority","issuetype"]}'
        headers => {
          Accept => "application/json"
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
    add_field => {
       "service.name" => "Gazelle Test Management"
       "service.id" => "10013"
       "service.key" => "GZL"
       "service.version" => "5.1.1"
    }
    #metadata_target => "http_poller_metadata"
  }

Je n'ai pas d'erreur quand je lance logstash et j'ai l'impression qu'il comprend bien qu'il doit ouvrir deux input de type http_poller:

[2019-07-10T09:32:41,062][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2019-07-10T09:32:41,548][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}

Cependant, il n'y a que le http_poller portant le nom getIssuesForFHIRValidatorProjectAndFixVersion qui est exécuté.

Aurais-tu une idée ?

dadoonet · July 10, 2019, 9:49am

Aucune idée. C'est peut être un bug. @colinsurprenant ton avis ?

Néanmoins suivant ton cas, tu peux aussi faire 2 pipelines LS différents, chacun avec son input.

Benjamin_Carriou · July 10, 2019, 10:05am

Oui j'ai pensé aussi à ça de faire deux pipelines.

Je vais testé pour voir

Benjamin_Carriou · July 10, 2019, 10:21am

@dadoonet

Autant pour moi, c'était un élément dans le body qui n'était pas correct, ce qui avait pour conséquence de me retourner une réponse vide.

Du coup tout fonctionne.

Désolé pour le dérangement et merci pour ton aide

system · August 7, 2019, 10:24am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Probéme avec logstash Discussions en français	4	1159	July 6, 2017
Problème pour parser un fichier JSON à l'aide de Logstash Discussions en français	6	1887	July 6, 2017
Logstash http_poller POST Syntax Logstash	2	4436	July 6, 2017
[LogStash] Comment ignorer les _grokparsefailure? Discussions en français	3	4856	July 6, 2017
Http_poller input Logstash	3	251	May 26, 2020

[LOGSTASH] Input http-poller POST

Related topics