Duplicate field 'field'\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@xxxxx; line: 7, column: 18

[Shiv18 (Siva)]

Hi Team,

I'm trying to create ingest pipeline in elasticsearch for lowercase few fields. below is my api,

PUT _ingest/pipeline/lowercase_pipeline
{
"description" : "lowercases the incoming field values",
"processors" : [
{
"lowercase" : {
"field" : "process.name",
"field" : "process.args"
}
}
]
}

and below is the error i get ,

{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "Failed to parse content to map"
}
],
"type": "parse_exception",
"reason": "Failed to parse content to map",
"caused_by": {
"type": "json_parse_exception",
"reason": "Duplicate field 'field'\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@xxxxx; line: 7, column: 18]"
}
},
"status": 400
}

But when i pass single field, it accepts and acknowledge the api. when i pass more than 1 field, it shows the above error.

[dadoonet (David Pilato)]

You need to apply 2 lowercase processor, one for each field.

[Shiv18 (Siva)]

@dadoonet
Thank you very much for quick response. below is the syntax i tried and only first field is getting lowercase conversion not the second. sorry im new to elasticsearch,

PUT _ingest/pipeline/lowercase_pipeline
{
"description" : "lowercases the incoming field values",
"processors" : [
{
"lowercase" : {
"field" : "process.args"
}
},
{
"lowercase" : {
"field" : "process.name"
}
}
]
},

it acknowledge with true. but only process.args converted into lowercase not process.name.

[dadoonet (David Pilato)]

Could you share an example of the problem using the ingest _simulate endpoint?

[Shiv18 (Siva)]

@dadoonet, below is the result of the simulate for pipeline,

{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "request body or source parameter is required"
}
],
"type": "parse_exception",
"reason": "request body or source parameter is required"
},
"status": 400
}

[dadoonet (David Pilato)]

What did you call exactly ?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

[Shiv18 (Siva)]

hey @dadoonet, sorry for not using markdown.

PUT _ingest/pipeline/lowercase_pipeline
{
    "description" : "lowercases the incoming field values",
    "processors" : 
    [
      {
        "lowercase" : 
        {
          "field" : "process.args"
        }
      },
      {
        "lowercase" : 
        {
          "field" : "process.name"
        }
      }
    ]
  }

result for above api is,

{
  "acknowledged" : true
}

When i query for,

GET _ingest/pipeline/lowercase_pipeline/_simulate

it results as below,

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "request body or source parameter is required"
      }
    ],
    "type": "parse_exception",
    "reason": "request body or source parameter is required"
  },
  "status": 400
}

[dadoonet (David Pilato)]

I guess you didn't read the documentation. Here it is: https://www.elastic.co/guide/en/elasticsearch/reference/7.5/simulate-pipeline-api.html

[Shiv18 (Siva)]

@dadoonet, sorry again for wasting your time. please find the below details results as you want,

pipeline i created for lowercase 2 fields,

PUT _ingest/pipeline/lowercase_pipeline/
{
    "description" : "lowercases the incoming field values",
    "processors" : 
    [
      {
        "lowercase" : 
        {
          "field" : "process.name"
        }
      },
      {
        "lowercase" : 
        {
          "field" : "process.args"
        }
      }
    ]
  } 

result for the above api,

{
  "acknowledged" : true
}

and when i run _simulate for the same as below,

POST _ingest/pipeline/lowercase_pipeline/_simulate
{
  "docs": [
    {
      "_source":{
        "message":"Process Create:\nRuleName: \nUtcTime: 2019-12-13 05:30:55.019\nProcessGuid: {3331cfa9-220f-5df3-0000-00104f804300}\nProcessId: 7956\nImage: C:\\Windows\\System32\\svchost.exe\nFileVersion: 10.0.18362.1 (WinBuild.160101.0800)\nDescription: Host Process for Windows Services\nProduct: Microsoft® Windows® Operating System\nCompany: Microsoft Corporation\nOriginalFileName: svchost.exe\nCommandLine: C:\\Windows\\System32\\svchost.exe -k LocalServiceNetworkRestricted -p -s lmhosts\nCurrentDirectory: C:\\Windows\\system32\\\nUser: NT AUTHORITY\\LOCAL SERVICE\nLogonGuid: {3331cfa9-1ddc-5df3-0000-0020e5030000}\nLogonId: 0x3E5\nTerminalSessionId: 0\nIntegrityLevel: System\nHashes: MD5=9520A99E77D6196D0D09833146424113,SHA256=DD191A5B23DF92E12A8852291F9FB5ED594B76A28A5A464418442584AFD1E048\nParentProcessGuid: {3331cfa9-1ddc-5df3-0000-001039b20000}\nParentProcessId: 608\nParentImage: C:\\Windows\\System32\\services.exe\nParentCommandLine: C:\\Windows\\system32\\services.exe"
      }
    }
  ]
}

and the result is ,

{
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "ingest_processor_exception",
            "reason" : "java.lang.IllegalArgumentException: field [process] not present as part of path [process.name]"
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : "field [process] not present as part of path [process.name]"
      }
    }
  ]
}

to give full picture, im shipping windows sysmon logs from winlogbeat to elasticsearch. let me know if you require any additional information.

Thanks!

[dadoonet (David Pilato)]

Your document has only a field message.
So this can not work as you are trying to lowercase non existing fields.