Ingest with condition issue

Elastic Stack Kibana
1

met strange issue.

created one ingest with condition by Kibana, and tested pass with my data in ingest edit by 'Add documents'. but failed when I use 'Dev Tools' in kibana, 'PUT' same data. why?

will be failed, paste the follow data in 'Dev Tools'.

POST filebeat-up-plus-app1-7.12.0-2021.04.25-000001/_doc?pipeline=test_ingest-copy
{
  "message": """2021-04-15 00:04:49.322 [INFO ][my-threadpool-executor-6] c.p.p.b.m.service.impl.MaterialsLeaveServiceImpl getBaseUserInfo: {"empAuthStatus":"1","enterpriseId":XXX,"enterpriseName":"XXXXXX","mobilephone":"XXXXX","realname":"XXX","username":XXXXXXXX}"""
}

and error msg:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "mapper_parsing_exception",
        "reason" : "object mapping for [jsondata] tried to parse field [jsondata] as object, but found a concrete value"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "object mapping for [jsondata] tried to parse field [jsondata] as object, but found a concrete value"
  },
  "status" : 400
}

but when I change data from 'getBaseUserinfo' to 'getBaseUserinfo map' will be passed test in 'Dev Tool'. I don't know why.

2

can you share your ingestion pipeline, in particular any grok or dissect processor which splits the message up, and the json processor?

3

my pipe

{
  "test_ingest" : {
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            """^(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) \[(?<result.txt>\w+ ?)\]\[(?<http_process>.*?)\]\s(?<classname>\S*)\s%{GREEDYDATA:upplus.message}"""
          ]
        }
      },
      {
        "date" : {
          "field" : "timestamp",
          "formats" : [
            "yyyy-MM-dd HH:mm:ss.SSS"
          ],
          "timezone" : "UTC+8"
        }
      },
      {
        "grok" : {
          "field" : "upplus.message",
          "patterns" : [
            """^(?<usermap>[^:]*)[^\{]*(?<jsondata>.*)"""
          ],
          "if" : """ctx.classname == 'c.p.p.b.m.service.impl.MaterialsLeaveServiceImpl'
""",
          "tag" : "json grok",
          "ignore_failure" : true
        }
      },
      {
        "json" : {
          "field" : "jsondata",
          "if" : "ctx.usermap == 'getBaseUserInfo map'",
          "tag" : "json_script"
        }
      }
    ]
  }
}
4

This seemed to work okay. I had to add some escaped double quotes around two of the redacted XXXXX values. And I did not use the triple-quoted message, but rather, I escaped the message contents with this website: Free Online JSON Escape / Unescape Tool - FreeFormatter.com

POST _ingest/pipeline/tbs-test/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
            "message": "2021-04-15 00:04:49.322 [INFO ][my-threadpool-executor-6] c.p.p.b.m.service.impl.MaterialsLeaveServiceImpl getBaseUserInfo: {\"empAuthStatus\":\"1\",\"enterpriseId\":\"XXX\",\"enterpriseName\":\"XXXXXX\",\"mobilephone\":\"XXXXX\",\"realname\":\"XXX\",\"username\":\"XXXXXXXX\"}"
      }
    }
  ]
}

result:

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "result" : {
            "txt" : "INFO "
          },
          "upplus" : {
            "message" : """getBaseUserInfo: {"empAuthStatus":"1","enterpriseId":"XXX","enterpriseName":"XXXXXX","mobilephone":"XXXXX","realname":"XXX","username":"XXXXXXXX"}"""
          },
          "jsondata" : {
            "empAuthStatus" : "1",
            "enterpriseId" : "XXX",
            "enterpriseName" : "XXXXXX",
            "mobilephone" : "XXXXX",
            "realname" : "XXX",
            "username" : "XXXXXXXX"
          },
          "@timestamp" : "2021-04-15T00:04:49.322+08:00",
          "classname" : "c.p.p.b.m.service.impl.MaterialsLeaveServiceImpl",
          "http_process" : "my-threadpool-executor-6",
          "usermap" : "getBaseUserInfo",
          "message" : """2021-04-15 00:04:49.322 [INFO ][my-threadpool-executor-6] c.p.p.b.m.service.impl.MaterialsLeaveServiceImpl getBaseUserInfo: {"empAuthStatus":"1","enterpriseId":"XXX","enterpriseName":"XXXXXX","mobilephone":"XXXXX","realname":"XXX","username":"XXXXXXXX"}""",
          "timestamp" : "2021-04-15 00:04:49.322"
        },
        "_ingest" : {
          "timestamp" : "2021-04-26T08:29:09.984546632Z"
        }
      }
    }
  ]
}
5

Hi Jason_Slater,

this message is from my log file by filebeat, In fact, met error message in filebeat log. I picked error message log for ‘POST' test. for your testing result, that mean, need change index template?

and another thing, If I change message .... c.p.p.b.m.service.impl.MaterialsLeaveServiceImpl getBaseUserInfo: {..... to .... c.p.p.b.m.service.impl.MaterialsLeaveServiceImpl getBaseUserInfo map: {..... then do 'POST', that will be OK, why?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.