# Parsing nested json not happening properly

**URL:** <https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282>\
**Category:** Logstash\
**Created:** [May 6, 2021, 11:18am UTC](https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282 "2021-05-06T11:18:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [May 6, 2021, 11:18am UTC](https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282/1 "2021-05-06T11:18:37Z")

</div>

Hi Folks,

I am trying to parse modsecurity audit logs which are natively being logged in JSON format. However when I am using logstash to ingest in elastic stack those appears like below and actual needed fields are not being parsed in their respective fields. I am not sure if any further config needed in logstash?

Here is my logstash config

```auto
input {
      file {
        type => "json"
        path => "/var/log/modsec_audit.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
      }
    }
filter {
        json {
          source => "message"
                skip_on_invalid_json => "true"
          tag_on_failure => ["_jsonparsefailure"]
        }
      }

```

And here are the messages being parsed -

```auto
{
  "_index": "applox-2021.05.06",
  "_type": "_doc",
  "_id": "gldWQXkBPJxogvJE-ilc",
  "_score": 1,
  "_source": {
    "host": "cwaf",
    "type": "json",
    "@version": "1",
    "path": "/var/log/modsec_audit.log",
    "transaction": {
      "host_port": 80,
      "unique_id": "16202916142.116840",
      "messages": [
        {
          "message": "Method is not allowed by policy",
          "details": {
            "reference": "v0,6",
            "maturity": "9",
            "tags": [
              "application-multi",
              "language-multi",
              "platform-multi",
              "attack-generic",
              "OWASP_CRS/POLICY/METHOD_NOT_ALLOWED",
              "WASCTC/WASC-15",
              "OWASP_TOP_10/A6",
              "OWASP_AppSensor/RE1",
              "PCI/12.1"
            ],
            "ruleId": "911100",
            "file": "/etc/nginx/modsec/crs/owaspcrs/rules/REQUEST-911-METHOD-ENFORCEMENT.conf",
            "rev": "2",
            "lineNumber": "27",
            "ver": "OWASP_CRS/3.0.0",
            "accuracy": "9",
            "severity": "2",
            "match": "Matched \"Operator `Within' with parameter `GET HEAD POST OPTIONS' against variable `REQUEST_METHOD' (Value: `DELETE' )",
            "data": "DELETE"
          }
        },
        {
          "message": "Inbound Anomaly Score Exceeded (Total Score: 1000)",
          "details": {
            "reference": "",
            "maturity": "0",
            "tags": [
              "application-multi",
              "language-multi",
              "platform-multi",
              "attack-generic"
            ],
            "ruleId": "949110",
            "file": "/etc/nginx/modsec/crs/owaspcrs/rules/REQUEST-949-BLOCKING-EVALUATION.conf",
            "rev": "",
            "lineNumber": "44",
            "ver": "",
            "accuracy": "0",
            "severity": "2",
            "match": "Matched \"Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `1000' )",
            "data": ""
          }
        }
      ],
      "response": {
        "http_code": 403,
        "body": "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.20.0</center>\r\n</body>\r\n</html>\r\n",
        "headers": {
          "Server": "nginx/1.20.0",
          "Connection": "keep-alive",
          "Content-Length": "153",
          "Date": "Thu, 06 May 2021 09:00:14 GMT",
          "Content-Type": "text/html"
        }
      },
      "client_ip": "127.0.0.1",
      "host_ip": "127.0.0.1",
      "server_id": "30ef83bdf67cbb820de6ae6410fba661c0a34ea0",
      "request": {
        "uri": "/test.php",
        "method": "DELETE",
        "http_version": 1.1,
        "headers": {
          "Accept": "*/*",
          "Host": "localhost",
          "User-Agent": "curl/7.58.0"
        }
      },
      "producer": {
        "secrules_engine": "Enabled",
        "modsecurity": "ModSecurity v3.0.4 (Linux)",
        "components": [
          "OWASP_CRS/3.0.2\""
        ],
        "connector": "ModSecurity-nginx v1.0.1"
      },
      "client_port": 39504,
      "time_stamp": "Thu May 6 14:30:14 2021"
    },
    "message": "{\"transaction\":{\"client_ip\":\"127.0.0.1\",\"time_stamp\":\"Thu May 6 14:30:14 2021\",\"server_id\":\"30ef83bdf67cbb820de6ae6410fba661c0a34ea0\",\"client_port\":39504,\"host_ip\":\"127.0.0.1\",\"host_port\":80,\"unique_id\":\"16202916142.116840\",\"request\":{\"method\":\"DELETE\",\"http_version\":1.1,\"uri\":\"/test.php\",\"headers\":{\"Host\":\"localhost\",\"User-Agent\":\"curl/7.58.0\",\"Accept\":\"*/*\"}},\"response\":{\"body\":\"<html>\\r\\n<head><title>403 Forbidden</title></head>\\r\\n<body>\\r\\n<center><h1>403 Forbidden</h1></center>\\r\\n<hr><center>nginx/1.20.0</center>\\r\\n</body>\\r\\n</html>\\r\\n\",\"http_code\":403,\"headers\":{\"Server\":\"nginx/1.20.0\",\"Date\":\"Thu, 06 May 2021 09:00:14 GMT\",\"Content-Length\":\"153\",\"Content-Type\":\"text/html\",\"Connection\":\"keep-alive\"}},\"producer\":{\"modsecurity\":\"ModSecurity v3.0.4 (Linux)\",\"connector\":\"ModSecurity-nginx v1.0.1\",\"secrules_engine\":\"Enabled\",\"components\":[\"OWASP_CRS/3.0.2\\\"\"]},\"messages\":[{\"message\":\"Method is not allowed by policy\",\"details\":{\"match\":\"Matched \\\"Operator `Within' with parameter `GET HEAD POST OPTIONS' against variable `REQUEST_METHOD' (Value: `DELETE' )\",\"reference\":\"v0,6\",\"ruleId\":\"911100\",\"file\":\"/etc/nginx/modsec/crs/owaspcrs/rules/REQUEST-911-METHOD-ENFORCEMENT.conf\",\"lineNumber\":\"27\",\"data\":\"DELETE\",\"severity\":\"2\",\"ver\":\"OWASP_CRS/3.0.0\",\"rev\":\"2\",\"tags\":[\"application-multi\",\"language-multi\",\"platform-multi\",\"attack-generic\",\"OWASP_CRS/POLICY/METHOD_NOT_ALLOWED\",\"WASCTC/WASC-15\",\"OWASP_TOP_10/A6\",\"OWASP_AppSensor/RE1\",\"PCI/12.1\"],\"maturity\":\"9\",\"accuracy\":\"9\"}},{\"message\":\"Inbound Anomaly Score Exceeded (Total Score: 1000)\",\"details\":{\"match\":\"Matched \\\"Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `1000' )\",\"reference\":\"\",\"ruleId\":\"949110\",\"file\":\"/etc/nginx/modsec/crs/owaspcrs/rules/REQUEST-949-BLOCKING-EVALUATION.conf\",\"lineNumber\":\"44\",\"data\":\"\",\"severity\":\"2\",\"ver\":\"\",\"rev\":\"\",\"tags\":[\"application-multi\",\"language-multi\",\"platform-multi\",\"attack-generic\"],\"maturity\":\"0\",\"accuracy\":\"0\"}}]}}",
    "@timestamp": "2021-05-06T11:01:29.155Z"
  },
  "fields": {
    "@timestamp": [
      "2021-05-06T11:01:29.155Z"
    ]
  }
}

```

The Most interested fields are

```auto
transaction.messages

```

Can someone pls help?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 6, 2021, 2:54pm UTC](https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282/2 "2021-05-06T14:54:46Z")

</div>

That very much looks as though the [message] field was successfully parsed. What do you not like about the result.

---

<div class="post-metadata">

**Author:** ![Blason](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blason/32/42284_2.png) [@Blason](https://discuss.elastic.co/u/Blason)\
**Post date:** [May 6, 2021, 3:20pm UTC](https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282/3 "2021-05-06T15:20:51Z")

</div>

Oh I guess it did not appear properly in Json Field let me show you GUI. Somehow those nested transaction.messages field not getting parsed.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/a/daca7859f4c02f6fc93635b272c5bc2d541cac2f.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/6/56ebe01e466142eca1122f8aefb00046c925a747.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fabe4bb2f49bfaeb608830787b1605c00e6bbcf0.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 3, 2021, 3:21pm UTC](https://discuss.elastic.co/t/parsing-nested-json-not-happening-properly/272282/4 "2021-06-03T15:21:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.