JSON Line Splitting inside string logs

[sbienert]

Hello,
I have normal string logs from Log4j. Some of them contain JSON queries inside but Filebeat writes them to Elasticsearch as if each line of the JSON is a separate log because as of right now I only have the default line splitting activated.

So how can Filebeat know that the JSON which already starts in the normal string line belongs to the entire log?

This is a sample of one log output containing a json query inside:

2018-09-14 10:02:59.553 -testClient admin DEBUG 2500 --- [http-nio-7044-exec-7] c.n.elasticsearch.search.Service   : Search query to be executed {
  "_source": [
    "id",
    "client",
    "version",
    "name",
    "@class"
  ],
  "query": {
    "bool" : {
      "must" : [
        {
          "wildcard" : { "name" : "*test*" }
        },
        {
          "term" : { "client" : "testClient" }
        }
      ]
    }
  }
}.

Please notice that the first bracket is already in the string line and the JSON ends with a dot.

Thank you very much

[ruflin]

You can use mutiline to make sure everything ends up as 1 event in Elasticsearch (including everything around json). If you also want to extract the json, you could try your luck with using the dissect processor to extract the json part into a value and then use the decode_json processor. Could work. Otherwise use the ingest in Elasticsearch with grok and json processor.