JSON field expansion

[DPattee]

Over the past couple days I've read a bunch of posts here and on some other forums, plus official documentation, but there is something that just isn't "clicking"...

Goal: Expand/explode/parse a document field that contains json key value pairs in to individual fields, with the key as the name and the value as the value.

Example of existing field in ES:

foo.bar {
"randomnumber": "12",
"fruit": "banana",
}

What I want, two separate parsed fields:

asdf.randomnumber 12
asdf.fruit banana

Note: I do not want to know every possible field (I.e. maybe the json has 4 fields, maybe down the line it has 15 fields, I don't want to have to hardcode every one of those or keep the pipeline up-to-date with all the possibilities)

I have a custom pipeline that runs for all messages received from filebeat, and I know that system 'works' because I have multiple branched pipelines that correctly process other messages. So I just added a branch that executes for this specific type of message. Further, I know that the correct pipeline is being run because I add a field "custompipeline" with a value of the pipeline name.

My pipeline is being kept very small to figure out how to do what I want, it has one 'set' (for teh pipeline name tag), an 'on_failure' (to surface an error message), and just the little blob I'm working through here.

What I've tried:
I've tried a few different combinations of the json and foreach processors.

  "json": {
    "field": "foo.bar",
    "target_field": ""
  }

Unexpected character ('c' (code 99)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@737189ee; line: 1, column: 4]

  "foreach": {
    "field": "foo.bar",
    "processor": {
      "json": {
        "field": "_ingest._value",
        "target_field": "asdf._ingest._value"
      }
    }
  }

I was hopeful based on some random forum posts I found suggested this would iterate through, but that only results in an error message of Unexpected character ('c' (code 99)): was expecting double-quote to start field name\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@2d5a039; line: 1, column: 3]

My best guess about the "c" is that that's the actual first letter of the first key in the real data (i.e. where the "r" in "randomnumber" is in this example)

  "foreach": {
    "field": "foo.bar",
    "processor": {
      "json": {
        "field": "_ingest._value.vendor",
        "target_field": "asdf.vendor"
      }
    }
  }

Unrecognized token 'Apple': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@149061ab; line: 1, column: 7]

  "foreach": {
    "field": "foo.bar",
    "processor": {
      "uppercase": {
        "field": "_ingest._value.vendor",
        "target_field": "asdf.vendor"
      }
    }
  }

I tried switching to a really simply processor, uppercase, rather than json decoder. This one got the value of a single key out and made it its own field, but I have to know the name ('vendor') ahead of time and hardcode it, and it clearly isn't something that is iterating through all the key value pairs.

So what obvious and probably documented thing am I missing

[DPattee]

Oh, and I'm pretty sure I can't use the decode_json_fields processor in the modules's config yml based on experiences a year or so back with how the osquery module works... All my other osquery-related pipelines had to be done on the ingest node.

[DPattee]

Ohhh, I was overthinking the heck out of this problem.

There is no need to use the json processor at all. All you have to do is foreach rename...

"foreach": {
"field": "foo.bar",
"processor": {
"rename": {
"field": "_ingest._value",
"target_field": "asdf"
}
}
}

That fully iterates through all the key value pairs stored within foo.bar and creates asdf. fields with the correct values.