Duplicate Fields in JSON Response payload when querying Kibana Dashboard Using Discover Tab

Elastic Stack Logstash
1

Hi,

I am running the latest version of Elastic Stack Elasticsearch version 8.14.2 | Elasticsearch Guide [8.14] | Elastic on Red Hat Enterprise Linux OS version 8.10 (8.10 Release Notes | Red Hat Product Documentation). I have configured the Logstash to push API logs, which include the request payload from the client and the response payload from backend services, to the Elastic Stack using the Message Logging Policy as described in the https://docs.apigee.com/api-platform/reference/policies/message-logging-policy#syslog

cat /etc/logstash/conf.d/apigee-logstash_new.conf
input {
  syslog {
    port => 5044
    grok_pattern => "%{GREEDYDATA} Apigee-Edge - - %{GREEDYDATA:message}"
    add_field => { "sourceapp" => "api_access" }
  }
}

filter {
  kv {
    source => "message"
    field_split => "||"
  }

  ruby {
    code => "
      require 'base64'

      ['proxyRequestPayload', 'targetRequestPayload', 'targetResponsePayload', 'clientResponsePayload'].each do |field|
        value = event.get(field)
        if value
          event.set(field, Base64.decode64(value))
        end
      end
    "
  }

  mutate {
    remove_field => ["text", "message", "event"]
  }
}

output {
  elasticsearch {
    hosts => ["192.168.0.109:9200"]
    manage_template => true
    index => "new-apigee-log-%{+YYYY.MM.dd}"
    action => "index"
  }
}

When I run the Discover tab in the Kibana dashboard,

image
image3454×526 101 KB

I see duplicate fields in the response payload in JSON format. For example, proxyRequestPayload is repeated twice, and so on. An example of the response is shown below in JSON format.

{
  "_index": "new-apigee-log-2024.07.02",
  "_id": "idOGcpABlF81N5hzz7b9",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@version": "1",
    "proxyRequestTime": "2024-07-02T08:19:15.891Z",
    "proxyClientIp": "172.16.16.13",
    "status": "200",
    "@timestamp": "2024-07-02T08:19:15.900672942Z",
    "apigee-request-id": "apigeeapigatewaytraining-27099-679342-2",
    "eol": "yes\u0000\n",
    "environment": "test",
    "requestDecryptedPayload ": "null",
    "proxyRequestHeaderLists": "{\"Content-Length\":\"641\"},{\"Content-Type\":\"application/xml\"},{\"Host\":\"apigee-tr-test.mydomain.com\"},{\"x-api-interaction-id\":\"e7c60fd3-a722-4726-bd45-16fc89d79333\"},{\"X-Forwarded-For\":\"172.16.16.13\"},{\"X-Forwarded-Port\":\"443\"},{\"X-Forwarded-Proto\":\"https\"},{\"x-timestamp\":\"1719908356\"}",
    "proxyResponseHeaderList": "{\"Content-Security-Policy\":\"default-src\"},{\"Content-Type\":\"application/xml\"},{\"Host\":\"apigee-tr-test.mydomain.com\"},{\"Strict-Transport-Security\":\"max-age=31536000\"},{\"X-Api-Interaction-Id\":\"e7c60fd3-a722-4726-bd45-16fc89d79333\"},{\"X-Content-Type-Options\":\"nosniff\"},{\"X-Forwarded-For\":\"172.16.16.13\"},{\"X-XSS-Protection\":\"1; mode=block\"}",
    "proxy-url": "https://apigee-tr-test.mydomain.com/cb-van/mock",
    "proxyRequestPayload": "<FCDB_REQ_ENV><FCDB_HEADER><SOURCE>APIBANKING</SOURCE><FCDBCOMP>FCDB</FCDBCOMP><IDUSER>APIBANKING</IDUSER><SERVICE></SERVICE><SOURCE_USERID>APIBANKINGUSER</SOURCE_USERID><DESTINATION>FCDB</DESTINATION><COUNTRYCODE>IN</COUNTRYCODE><USERTYPE></USERTYPE><LANGID>eng</LANGID><USERID/><IDCHANNELUSER>SYEDAPIAUTH</IDCHANNELUSER><CHANNELID>01</CHANNELID><MSGSTAT>SUCCESS</MSGSTAT>                 </FCDB_HEADER><FCDB_BODY><ACCOUNTNOINPUTINFO><NBRACCOUNT>2774201000198</NBRACCOUNT><IDCUSTOMER>13961989</IDCUSTOMER><CODBRANCH>2774</CODBRANCH>                     </ACCOUNTNOINPUTINFO>                 </FCDB_BODY>             </FCDB_REQ_ENV>         ",
    "responseDecryptedPayload": "null",
    "fault": "false",
    "proxy-method": "POST",
    "organization": "apigee-dac-training",
    "client-message-id": "e7c60fd3-a722-4726-bd45-16fc89d79333",
    "log": {
      "syslog": {
        "priority": 0,
        "severity": {
          "code": 0,
          "name": "Emergency"
        },
        "facility": {
          "code": 0,
          "name": "kernel"
        }
      }
    },
    "service": {
      "type": "system"
    },
    "proxyName": "cb-van-mock-response",
    "host": {
      "ip": "192.168.0.146"
    },
    "clientResponsePayload": "\"<Response><VANCreationResponse><Status><ExtendedReply><MessagesArray/></ExtendedReply><ErrorCode>1749</ErrorCode><ExternalReferenceNo>ExternalReferenceNo888188800011111</ExternalReferenceNo><IsOverriden>false</IsOverriden><IsServiceChargeApplied>false</IsServiceChargeApplied><Memo/><ReplyCode>0</ReplyCode><ReplyText/><SpReturnValue>0</SpReturnValue></Status><AccountNo>9833111000032</AccountNo><CorporateID/><AccountNumDetailsDTO><VanNumber>00792000000266555</VanNumber><CustomerName>NAVNEETKUMARJAIN</CustomerName><StartDate>20230701</StartDate><EndDate>20231104</EndDate></AccountNumDetailsDTO><AccountNumDetailsDTO><VanNumber>00792000000266556</VanNumber><CustomerName>NAVNEETKUMARJAIN</CustomerName><StartDate>20230701</StartDate><EndDate>20231104</EndDate></AccountNumDetailsDTO><ExternalSystemID>systemid1</ExternalSystemID></VANCreationResponse></Response>\"",
    "requestHost": "apigee-tr-test.mydomain.com",
    "proxyResponseTimetaken": "2024-07-02T08:19:15.895Z",
    "sourceapp": "api_access"
  },
  "fields": {
    "eol": [
      "yes\u0000\n"
    ],
    "proxyName": [
      "cb-van-mock-response"
    ],
    "proxy-method": [
      "POST"
    ],
    "proxyResponseTimetaken": [
      "2024-07-02T08:19:15.895Z"
    ],
    "requestHost": [
      "apigee-tr-test.mydomain.com"
    ],
    "service.type": [
      "system"
    ],
    "host.ip": [
      "192.168.0.146"
    ],
    "client-message-id": [
      "e7c60fd3-a722-4726-bd45-16fc89d79333"
    ],
    "log.syslog.facility.name": [
      "kernel"
    ],
    "responseDecryptedPayload": [
      "null"
    ],
    "@version": [
      "1"
    ],
    "log.syslog.severity.name": [
      "Emergency"
    ],
    "log.syslog.priority": [
      0
    ],
    "proxyResponseHeaderList": [
      "{\"Content-Security-Policy\":\"default-src\"},{\"Content-Type\":\"application/xml\"},{\"Host\":\"apigee-tr-test.mydomain.com\"},{\"Strict-Transport-Security\":\"max-age=31536000\"},{\"X-Api-Interaction-Id\":\"e7c60fd3-a722-4726-bd45-16fc89d79333\"},{\"X-Content-Type-Options\":\"nosniff\"},{\"X-Forwarded-For\":\"172.16.16.13\"},{\"X-XSS-Protection\":\"1; mode=block\"}"
    ],
    "proxy-url": [
      "https://apigee-tr-test.mydomain.com/cb-van/mock"
    ],
    "apigee-request-id": [
      "apigeeapigatewaytraining-27099-679342-2"
    ],
    "sourceapp": [
      "api_access"
    ],
    "log.syslog.severity.code": [
      0
    ],
    "proxyRequestTime": [
      "2024-07-02T08:19:15.891Z"
    ],
    "fault": [
      "false"
    ],
    "proxyRequestHeaderLists": [
      "{\"Content-Length\":\"641\"},{\"Content-Type\":\"application/xml\"},{\"Host\":\"apigee-tr-test.mydomain.com\"},{\"x-api-interaction-id\":\"e7c60fd3-a722-4726-bd45-16fc89d79333\"},{\"X-Forwarded-For\":\"172.16.16.13\"},{\"X-Forwarded-Port\":\"443\"},{\"X-Forwarded-Proto\":\"https\"},{\"x-timestamp\":\"1719908356\"}"
    ],
    "requestDecryptedPayload .keyword": [
      "null"
    ],
    "environment": [
      "test"
    ],
    "@timestamp": [
      "2024-07-02T08:19:15.900Z"
    ],
    "clientResponsePayload": [
      "\"<Response><VANCreationResponse><Status><ExtendedReply><MessagesArray/></ExtendedReply><ErrorCode>1749</ErrorCode><ExternalReferenceNo>ExternalReferenceNo888188800011111</ExternalReferenceNo><IsOverriden>false</IsOverriden><IsServiceChargeApplied>false</IsServiceChargeApplied><Memo/><ReplyCode>0</ReplyCode><ReplyText/><SpReturnValue>0</SpReturnValue></Status><AccountNo>9833111000032</AccountNo><CorporateID/><AccountNumDetailsDTO><VanNumber>00792000000266555</VanNumber><CustomerName>NAVNEETKUMARJAIN</CustomerName><StartDate>20230701</StartDate><EndDate>20231104</EndDate></AccountNumDetailsDTO><AccountNumDetailsDTO><VanNumber>00792000000266556</VanNumber><CustomerName>NAVNEETKUMARJAIN</CustomerName><StartDate>20230701</StartDate><EndDate>20231104</EndDate></AccountNumDetailsDTO><ExternalSystemID>systemid1</ExternalSystemID></VANCreationResponse></Response>\""
    ],
    "proxyRequestPayload": [
      "<FCDB_REQ_ENV><FCDB_HEADER><SOURCE>APIBANKING</SOURCE><FCDBCOMP>FCDB</FCDBCOMP><IDUSER>APIBANKING</IDUSER><SERVICE></SERVICE><SOURCE_USERID>APIBANKINGUSER</SOURCE_USERID><DESTINATION>FCDB</DESTINATION><COUNTRYCODE>IN</COUNTRYCODE><USERTYPE></USERTYPE><LANGID>eng</LANGID><USERID/><IDCHANNELUSER>SYEDAPIAUTH</IDCHANNELUSER><CHANNELID>01</CHANNELID><MSGSTAT>SUCCESS</MSGSTAT>                 </FCDB_HEADER><FCDB_BODY><ACCOUNTNOINPUTINFO><NBRACCOUNT>2774201000198</NBRACCOUNT><IDCUSTOMER>13961989</IDCUSTOMER><CODBRANCH>2774</CODBRANCH>                     </ACCOUNTNOINPUTINFO>                 </FCDB_BODY>             </FCDB_REQ_ENV>         "
    ],
    "organization": [
      "apigee-dac-training"
    ],
    "proxyClientIp": [
      "172.16.16.13"
    ],
    "log.syslog.facility.code": [
      0
    ],
    "requestDecryptedPayload ": [
      "null"
    ],
    "status": [
      "200"
    ]
  }
}

Please guide me. Thanks in advance.

Best Regards,

Kaushal

3

Hi @kaushalshriyan

What you are showing is not not repeated data...

The top parts should the _source json for the document that arrived at elastic...
That is stored in elastic as a json structure.

The 2nd time is the fields data ... the data that is actually indexed into elastic and searchable ...

Elastic stores both of these part of the data store. You can read about that if you like.

Here is a good article

Elasticsearch Stored Fields: Comparison to _source field and more..

4

Thanks @stephenb. So in short I do not have to modify /etc/logstash/conf.d/apigee-logstash_new.conf logstash file or do any changes?

Please suggest further. Thanks in advance.

Best Regards,

Kaushal

5

I can't really comment if you want to do more parsing or not. I'm just explaining what you asked where you thought you had duplicate Fields which you do not.

Yes, the data looks pretty well. Parsed so if you're happy with it, I would move on to your next task :slight_smile:

Did you create a mapping?

If not, I would use the new data stream naming convention and name it

logs-apagee-new

And let the data stream manage everything else.

In the output you would put

output {
  elasticsearch {
    hosts => ["192.168.0.109:9200"]
    manage_template => true
    index => "logs-apigee-new"
    action => "create"
  }
}

This should create a data stream which you can control with an ILM policy

Just a thought

Data stream in general

logs data stream. This document talks about using elastic agent, but if you to the data stream above, be able to take advantage of some of these features as well