Duplicate field 'field'\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@xxxxx; line: 7, column: 18

Elastic Stack Elasticsearch
1

Hi Team,

I'm trying to create ingest pipeline in elasticsearch for lowercase few fields. below is my api,

PUT _ingest/pipeline/lowercase_pipeline
{
"description" : "lowercases the incoming field values",
"processors" : [
{
"lowercase" : {
"field" : "process.name",
"field" : "process.args"
}
}
]
}

and below is the error i get ,

{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "Failed to parse content to map"
}
],
"type": "parse_exception",
"reason": "Failed to parse content to map",
"caused_by": {
"type": "json_parse_exception",
"reason": "Duplicate field 'field'\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@xxxxx; line: 7, column: 18]"
}
},
"status": 400
}

But when i pass single field, it accepts and acknowledge the api. when i pass more than 1 field, it shows the above error.

2

You need to apply 2 lowercase processor, one for each field.

3

@dadoonet
Thank you very much for quick response. below is the syntax i tried and only first field is getting lowercase conversion not the second. sorry im new to elasticsearch,

PUT _ingest/pipeline/lowercase_pipeline
{
"description" : "lowercases the incoming field values",
"processors" : [
{
"lowercase" : {
"field" : "process.args"
}
},
{
"lowercase" : {
"field" : "process.name"
}
}
]
},

it acknowledge with true. but only process.args converted into lowercase not process.name.

5

Could you share an example of the problem using the ingest _simulate endpoint?

6

@dadoonet, below is the result of the simulate for pipeline,

{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "request body or source parameter is required"
}
],
"type": "parse_exception",
"reason": "request body or source parameter is required"
},
"status": 400
}

7

What did you call exactly ?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

8

hey @dadoonet, sorry for not using markdown.

PUT _ingest/pipeline/lowercase_pipeline
{
    "description" : "lowercases the incoming field values",
    "processors" : 
    [
      {
        "lowercase" : 
        {
          "field" : "process.args"
        }
      },
      {
        "lowercase" : 
        {
          "field" : "process.name"
        }
      }
    ]
  }

result for above api is,

{
  "acknowledged" : true
}

When i query for,

GET _ingest/pipeline/lowercase_pipeline/_simulate

it results as below,

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "request body or source parameter is required"
      }
    ],
    "type": "parse_exception",
    "reason": "request body or source parameter is required"
  },
  "status": 400
}
9

I guess you didn't read the documentation. Here it is: https://www.elastic.co/guide/en/elasticsearch/reference/7.5/simulate-pipeline-api.html

10

@dadoonet, sorry again for wasting your time. please find the below details results as you want,

pipeline i created for lowercase 2 fields,

PUT _ingest/pipeline/lowercase_pipeline/
{
    "description" : "lowercases the incoming field values",
    "processors" : 
    [
      {
        "lowercase" : 
        {
          "field" : "process.name"
        }
      },
      {
        "lowercase" : 
        {
          "field" : "process.args"
        }
      }
    ]
  }

result for the above api,

{
  "acknowledged" : true
}

and when i run _simulate for the same as below,

POST _ingest/pipeline/lowercase_pipeline/_simulate
{
  "docs": [
    {
      "_source":{
        "message":"Process Create:\nRuleName: \nUtcTime: 2019-12-13 05:30:55.019\nProcessGuid: {3331cfa9-220f-5df3-0000-00104f804300}\nProcessId: 7956\nImage: C:\\Windows\\System32\\svchost.exe\nFileVersion: 10.0.18362.1 (WinBuild.160101.0800)\nDescription: Host Process for Windows Services\nProduct: Microsoft® Windows® Operating System\nCompany: Microsoft Corporation\nOriginalFileName: svchost.exe\nCommandLine: C:\\Windows\\System32\\svchost.exe -k LocalServiceNetworkRestricted -p -s lmhosts\nCurrentDirectory: C:\\Windows\\system32\\\nUser: NT AUTHORITY\\LOCAL SERVICE\nLogonGuid: {3331cfa9-1ddc-5df3-0000-0020e5030000}\nLogonId: 0x3E5\nTerminalSessionId: 0\nIntegrityLevel: System\nHashes: MD5=9520A99E77D6196D0D09833146424113,SHA256=DD191A5B23DF92E12A8852291F9FB5ED594B76A28A5A464418442584AFD1E048\nParentProcessGuid: {3331cfa9-1ddc-5df3-0000-001039b20000}\nParentProcessId: 608\nParentImage: C:\\Windows\\System32\\services.exe\nParentCommandLine: C:\\Windows\\system32\\services.exe"
      }
    }
  ]
}

and the result is ,

{
  "docs" : [
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "ingest_processor_exception",
            "reason" : "java.lang.IllegalArgumentException: field [process] not present as part of path [process.name]"
          }
        ],
        "type" : "illegal_argument_exception",
        "reason" : "field [process] not present as part of path [process.name]"
      }
    }
  ]
}

to give full picture, im shipping windows sysmon logs from winlogbeat to elasticsearch. let me know if you require any additional information.

Thanks!

11

Your document has only a field message.
So this can not work as you are trying to lowercase non existing fields.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.