Query does not find an existent register

Hi Guys,

I am trying to execute a query in Kibana but it does not find any register. Does anyone can help me to figure out what i am doing wrong ?

  • Query used (not working):
    http.request.body.content:*CONTR0020246023* AND http.response.body.content:*OtherError*

  • Bellow the json code (I attached a screenshot of kibana, i think is better than the json bellow) :

{
  "_index": "packetbeat-7.6.1-2020.03.19-000001",
  "_type": "_doc",
  "_id": "qlSH-HABGbxWlr-UEC43",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-03-20T15:19:23.563Z",
    "host": {
      "name": "ecmsdb2pe2"
    },
    "query": "POST /admx_ecms/services/ws_cma3",
    "type": "http",
    "status": "Error",
    "method": "post",
    "client": {
      "ip": "10.110.11.85",
      "port": 44819,
      "bytes": 2124
    },
    "server": {
      "ip": "10.188.183.11",
      "port": 80,
      "bytes": 561
    },
    "event": {
      "start": "2020-03-20T15:19:23.563Z",
      "end": "2020-03-20T15:19:24.576Z",
      "kind": "event",
      "category": "network_traffic",
      "dataset": "http",
      "duration": 1013006000
    },
    "ecs": {
      "version": "1.4.0"
    },
    "source": {
      "ip": "10.110.11.85",
      "port": 44819,
      "bytes": 2124
    },
    "agent": {
      "ephemeral_id": "ea9a1a3d-715a-43ac-b480-086ecf40a98f",
      "hostname": "ecmsdb2pe2",
      "id": "c0c85ac4-6dfc-4492-a659-787feb7c1e87",
      "version": "7.6.1",
      "type": "packetbeat"
    },
    "url": {
      "domain": "10.188.183.11",
      "path": "/admx_ecms/services/ws_cma3",
      "full": "http://10.188.183.11/admx_ecms/services/ws_cma3",
      "scheme": "http"
    },
    "destination": {
      "bytes": 561,
      "ip": "10.188.183.11",
      "port": 80
    },
    "user_agent": {
      "original": "Apache CXF 2.7.5"
    },
    "network": {
      "type": "ipv4",
      "transport": "tcp",
      "protocol": "http",
      "direction": "inbound",
      "community_id": "1:Dhzr73qoVhTYixGyFyF5IQFVMAE=",
      "bytes": 2685
    },
    "http": {
      "version": "1.1",
      "request": {
        "bytes": 2124,
        "headers": {
          "content-length": 1847,
          "content-type": "text/xml; charset=UTF-8"
        },
        "method": "post",
        "body": {
          "bytes": 1847,
          "content": "<soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\"><soap:Header><wsse:Security xmlns:wsse=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\" xmlns:wsu=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\" soap:mustUnderstand=\"1\"><wsse:UsernameToken wsu:Id=\"UsernameToken-2511304\"><wsse:Username>ODA</wsse:Username><wsse:Password Type=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText\">123passwd@</wsse:Password></wsse:UsernameToken></wsse:Security></soap:Header><soap:Body><con:contractServiceParametersWriteRequest xmlns:con=\"http://ericsson.com/services/ws_cma3/contractserviceparameterswrite\" xmlns:ses=\"http://ericsson.com/services/ws_cma3/sessionchange\" xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\"><con:inputAttributes><con:contract><con:publicKey>CONTR0020246023</con:publicKey>\n            </con:contract><con:profileId>0</con:profileId><con:service><con:publicKey>CSAHR</con:publicKey>\n            </con:service><con:paramValues><con:serviceParameterValueNode><con:action>m</con:action><con:targetParameterValues><con:targetParameterValue><con:parameterValues><con:parameterValue><con:publicKey>81</con:publicKey>\n                           </con:parameterValue>\n                        </con:parameterValues>\n                     </con:targetParameterValue>\n                  </con:targetParameterValues>\n               </con:serviceParameterValueNode>\n            </con:paramValues>\n         </con:inputAttributes><con:sessionChangeRequest><ses:values><ses:item><ses:key>BU_ID_PUB</ses:key><ses:value>CSGBU</ses:value>\n               </ses:item>\n            </ses:values>\n         </con:sessionChangeRequest>      \n      </con:contractServiceParametersWriteRequest></soap:Body></soap:Envelope>"
        }
      },
      "response": {
        "body": {
          "content": "<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://schemas.xmlsoap.org/soap/envelope/\"><SOAP-ENV:Header/><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode xmlns:ns0=\"http://ericsson.com/services/fault\">ns0:AIR.A100.OtherError</faultcode><faultstring xml:lang=\"en\">Other Error. </faultstring></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope>",
          "bytes": 327
        },
        "bytes": 561,
        "headers": {
          "content-length": 327,
          "content-type": "text/xml;charset=UTF-8"
        },
        "status_code": 500
      }
    }
  },
  "fields": {
    "event.end": [
      "2020-03-20T15:19:24.576Z"
    ],
    "@timestamp": [
      "2020-03-20T15:19:23.563Z"
    ],
    "event.start": [
      "2020-03-20T15:19:23.563Z"
    ]
  },
  "highlight": {
    "http.response.body.content": [
      "@kibana-highlighted-field@<SOAP-ENV:Envelope xmlns:SOAP-ENV=\"http://schemas.xmlsoap.org/soap/envelope/\"><SOAP-ENV:Header/><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode xmlns:ns0=\"http://ericsson.com/services/fault\">ns0:AIR.A100.OtherError</faultcode><faultstring xml:lang=\"en\">Other Error. </faultstring></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope>@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1584717563563
  ]
}

Hi @Claudio_Ract_Costa,

It's a bit difficult to read your post without having the code formatted, but based on the query you've written it seems it should be working.

I just indexed the document you provided in a clean Kibana deployment (I'm running the master branch locally). This is what I put in the dev tools console:

POST test/_doc
{
"@timestamp": "2020-03-20T15:19:23.563Z",
"host": {
"name": "ecmsdb2pe2"
},
"query": "POST /admx_ecms/services/ws_cma3",
"type": "http",
"status": "Error",
"method": "post",
"client": {
"ip": "10.110.11.85",
"port": 44819,
"bytes": 2124
},
"server": {
"ip": "10.188.183.11",
"port": 80,
"bytes": 561
},
"event": {
"start": "2020-03-20T15:19:23.563Z",
"end": "2020-03-20T15:19:24.576Z",
"kind": "event",
"category": "network_traffic",
"dataset": "http",
"duration": 1013006000
},
"ecs": {
"version": "1.4.0"
},
"source": {
"ip": "10.110.11.85",
"port": 44819,
"bytes": 2124
},
"agent": {
"ephemeral_id": "ea9a1a3d-715a-43ac-b480-086ecf40a98f",
"hostname": "ecmsdb2pe2",
"id": "c0c85ac4-6dfc-4492-a659-787feb7c1e87",
"version": "7.6.1",
"type": "packetbeat"
},
"url": {
"domain": "10.188.183.11",
"path": "/admx_ecms/services/ws_cma3",
"full": "http://10.188.183.11/admx_ecms/services/ws_cma3",
"scheme": "http"
},
"destination": {
"bytes": 561,
"ip": "10.188.183.11",
"port": 80
},
"user_agent": {
"original": "Apache CXF 2.7.5"
},
"network": {
"type": "ipv4",
"transport": "tcp",
"protocol": "http",
"direction": "inbound",
"community_id": "1:Dhzr73qoVhTYixGyFyF5IQFVMAE=",
"bytes": 2685
},
"http": {
"version": "1.1",
"request": {
"bytes": 2124,
"headers": {
"content-length": 1847,
"content-type": "text/xml; charset=UTF-8"
},
"method": "post",
"body": {
"bytes": 1847,
"content": """<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">soap:Header<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" soap:mustUnderstand="1"><wsse:UsernameToken wsu:Id="UsernameToken-2511304">wsse:UsernameODA</wsse:Username><wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">123passwd@</wsse:Password></wsse:UsernameToken></wsse:Security></soap:Header>soap:Body<con:contractServiceParametersWriteRequest xmlns:con="http://ericsson.com/services/ws_cma3/contractserviceparameterswrite" xmlns:ses="http://ericsson.com/services/ws_cma3/sessionchange" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">con:inputAttributescon:contractcon:publicKeyCONTR0020246023</con:publicKey>\n </con:contract>con:profileId0</con:profileId>con:servicecon:publicKeyCSAHR</con:publicKey>\n </con:service>con:paramValuescon:serviceParameterValueNodecon:actionm</con:action>con:targetParameterValuescon:targetParameterValuecon:parameterValuescon:parameterValuecon:publicKey81</con:publicKey>\n </con:parameterValue>\n </con:parameterValues>\n </con:targetParameterValue>\n </con:targetParameterValues>\n </con:serviceParameterValueNode>\n </con:paramValues>\n </con:inputAttributes>con:sessionChangeRequestses:valuesses:itemses:keyBU_ID_PUB</ses:key>ses:valueCSGBU</ses:value>\n </ses:item>\n </ses:values>\n </con:sessionChangeRequest> \n </con:contractServiceParametersWriteRequest></soap:Body></soap:Envelope>"""
}
},
"response": {
"body": {
"content": """<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">SOAP-ENV:Header/SOAP-ENV:BodySOAP-ENV:Fault<faultcode xmlns:ns0="http://ericsson.com/services/fault">ns0:AIR.A100.OtherError<faultstring xml:lang="en">Other Error. </SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope>""",
"bytes": 327
},
"bytes": 561,
"headers": {
"content-length": 327,
"content-type": "text/xml;charset=UTF-8"
},
"status_code": 500
}
}
}

Then I created a test* index pattern in Kibana, and was able to query using http.request.body.content:*CONTR0020246023* AND http.response.body.content:*OtherError*. It correctly highlighted both fields in the document.

Based on the logs, looks like you are running 7.6.1? Are you certain the timerange you have selected includes the document you are expecting to be retrieved? If you inspect your network requests in your browser dev tools, what query do you see going out to the es endpoint?

Thanks for help Luke !

i've just edited the post. I think it is now correctly formatted.

Yes, my Kibana version is 7.6.1.

Follow bellow the tcpdump screenshot about this request

http.request.body.content: *CONTR0020246023* AND http.response.body.content:*OtherError*

ps.: i used as timerange the last 7 days and this event happened today. It should appears

Do you see something wrong ?

Hi Luke,

I got ... This query now works if I use the field http.request.body.content.text instead http.request.body.content

Do you know why ?

That makes a bit more sense -- when indexing string fields in Elasticsearch, by default strings are mapped both as text and keyword datatypes.

The text datatype is what is used when you need to do a full-text search as you are doing here.

That said, it feels odd to me that you are needing to explicitly include .text in the query. In most cases it works without this. What do your mappings look like for these two fields?