Ingestion Pipeline not parsing out field values

[tonyz]

Hi,

I am applying a new ingestion pipeline through Kibana Dev Tools to parse a specific field. However, when I created the pipeline the fields are not parsed out at all. I tried this in _ingest/pipeline/_simulate before submitting the put pipeline API and it works in simulate.

Below is my ingest pipeline.

  PUT _ingest/pipeline/1
{
    "description": "parse json object",
    "processors": [
      {

        "grok": {
          "field": "log",
          "patterns": [
            "%{RUBY_LOGGER}\\[%{BACULA_JOB:job}\\]%{GREEDYDATA:log}"
          ],
          "ignore_failure":true
        },
        "json": {
          "field": "log",
          "target_field": "log-json",
          "ignore_failure": true
        }
        }
      
    ]
}

_ingest/pipeline/_simulate

POST _ingest/pipeline/_simulate
    {
      "pipeline": {
        "description": "parse json object",
        "processors": [
          {

            "grok": {
              "field": "log",
              "patterns": [
                "%{RUBY_LOGGER}\\[%{BACULA_JOB:job}\\]%{GREEDYDATA:log}"
              ],
              "ignore_failure":true
            },
            "json": {
              "field": "log",
              "target_field": "log-json",
              "ignore_failure": true
            }
            }
          
        ]
      },
      "docs": [
        {
          "_source": {
            "log": "I, [2018-03-02T14:37:24.048385 #7]  INFO -- : [96621855-c4f1-4c8e-b9c9-2bbb4e50171c]{\"key\":\"value\",\"key2\":\"value2\",\"key3\":1,\"key4\":1,\"key5\":1,\"key6\":\"value\"}\n"
          }
        }
      ]
    }

Any guidance would be greatly appreciated.

Thanks,
Tony