"Could not index event" due to "field already exists"

Elastic Stack Elasticsearch
1

Data path: Windows Sysmon --> Winlogbeat --> Logstash --> Elasticsearch

Problem: Certain new events are unable to be indexed:

"status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"field [process.pe.description] already exists"

The value of process.pe.description in the failed event is:

"Microsoft® Group Policy Update Utility"

Here is the index mapping for the field.

           "pe": {
              "properties": {
                "description": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 32765
                    },
                    "security": {
                      "type": "text",
                      "analyzer": "es_security_analyzer"
                    }
                  }
                }

My assumption is the problem is that the index type is "text" and the ® makes the field not text, thus it tries to create a new field, but the field already exists.

Can anyone confirm or deny that and help me figure out what next steps I can take to resolve this? Thanks.

For searchability: This is event id 1 in sysmon.

2

So as a test i threw a string with the ® character into es to see what mapping it would give it.

PUT testindex/_doc/1
{
"somefield"  : "Microsoft® Group Policy Update Utility"
}

GET testindex/_mapping
{
  "testindex" : {
    "mappings" : {
      "properties" : {
        "somefield" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

seems that that character has no problem being text, and in my experience you would get an error about it being the wrong type or maybe get a mapping conflict.

This might be a dumb question but are there two description fields? Thats what this link seems to say

https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventid=90001

1 Like
3

Thanks for doing that test. I just looked at the data and confirmed that this is the only instance of process.pe.description in the event. After looking at Kibana I have just now removed a 3rd party tool that is sending sysmon logs to elasticsearch in the same index because only sysmon events originating from that 3rd party tool are successfully being indexed. Tomorrow I will be able to see if that resolves the issue.

4

Well,
That did not clear it up and I am still getting the same error today. Here is the full sanitized event. Maybe you can find a duplicate field that I didn't?

[
    2021-08-12T00: 04: 18,
    688
][WARN
][logstash.outputs.elasticsearch
] Could not index event to Elasticsearch. {
    :status=>400,
    :action=>[
        "index",
        {
            :_id=>nil,
            :_index=>"so-beats",
            :routing=>nil,
            :pipeline=>"beats.common"
        },
        {
            "related"=>{
                "hash"=>[
                    "118729dd62b9422d21afe5cebd564f8a",
                    "b76ce2bba63bd2949fa6e36fba963379b9d682f7642cd3782d9818fcd30a3e00",
                    "d2466cd4c38ecef8aab0ee78b79fc8b8"
                ],
                "user"=>"NETWORK SERVICE"
            },
            "log"=>{
                "level"=>"information"
            },
            "tags"=>[
                "beat-ext",
                "beats_input_codec_plain_applied"
            ],
            "type"=>"redis-input",
            "metadata"=>{
                "type"=>"_doc",
                "beat"=>"winlogbeat",
                "version"=>"7.14.0",
                "ip_address"=>"..."
            },
            "process"=>{
                "args"=>[
                    "gpupdate.exe",
                    "/target:computer"
                ],
                "pe"=>{
                    "company"=>"Microsoft Corporation",
                    "original_file_name"=>"GPUpdate.exe",
                    "imphash"=>"d2466cd4c38ecef8aab0ee78b79fc8b8",
                    "description"=>"Microsoft® Group Policy Update Utility",
                    "product"=>"Microsoft® Windows® Operating System",
                    "file_version"=>"10.0.19041.572 (WinBuild.160101.0800)"
                },
                "pid"=>14480,
                "parent"=>{
                    "args"=>[
                        "C:\\WINDOWS\\system32\\svchost.exe",
                        "-k",
                        "netsvcs",
                        "-p",
                        "-s",
                        "Schedule"
                    ],
                    "pid"=>2764,
                    "command_line"=>"C:\\WINDOWS\\system32\\svchost.exe -k netsvcs -p -s Schedule",
                    "executable"=>"C:\\Windows\\System32\\svchost.exe",
                    "name"=>"svchost.exe",
                    "entity_id"=>"{a3a021f9-4e58-60f0-2b00-000000000b00}"
                },
                "hash"=>{
                    "md5"=>"118729dd62b9422d21afe5cebd564f8a",
                    "sha256"=>"b76ce2bba63bd2949fa6e36fba963379b9d682f7642cd3782d9818fcd30a3e00"
                },
                "command_line"=>"gpupdate.exe /target:computer",
                "executable"=>"C:\\Windows\\System32\\gpupdate.exe",
                "name"=>"gpupdate.exe",
                "entity_id"=>"{a3a021f9-657f-6114-a9de-d94d01000000}",
                "working_directory"=>"C:\\WINDOWS\\system32\\"
            },
            "host"=>{
                "hostname"=>"...",
                "mac"=>[
                    "...",
                    "...",
                    "...",
                    "...",
                    "..."
                ],
                "os"=>{
                    "family"=>"windows",
                    "platform"=>"windows",
                    "build"=>"19042.1110",
                    "type"=>"windows",
                    "kernel"=>"10.0.19041.1110 (WinBuild.160101.0800)",
                    "version"=>"10.0",
                    "name"=>"Windows 10 Enterprise"
                },
                "name"=>"...",
                "ip"=>[
                    "...",
                    "...",
                    "...",
                    "...",
                    "...",
                    "...",,
                    "...",
                    "...",
                    "...",
                    "..."
                ],
                "architecture"="...",
                "id"=>"..."
            },
            "winlog"=>{
                "opcode"=>"Info",
                "provider_name"=>"Microsoft-Windows-Sysmon",
                "computer_name"=>"...",
                "version"=>5,
                "task"=>"Process Create (rule: ProcessCreate)",
                "process"=>{
                    "thread"=>{
                        "id"=>15204
                    },
                    "pid"=>10112
                },
                "record_id"=>43577,
                "channel"=>"Microsoft-Windows-Sysmon/Operational",
                "event_id"=>"1",
                "event_data"=>{
                    "Description"=>"Microsoft® Group Policy Update Utility",
                    "Product"=>"Microsoft® Windows® Operating System",
                    "FileVersion"=>"10.0.19041.572 (WinBuild.160101.0800)",
                    "LogonGuid"=>"{a3a021f9-4e57-60f0-e403-000000000000}",
                    "TerminalSessionId"=>"0",
                    "LogonId"=>"0x3e4",
                    "Company"=>"Microsoft Corporation",
                    "IntegrityLevel"=>"System"
                },
                "api"=>"wineventlog",
                "user"=>{
                    "type"=>"User",
                    "name"=>"SYSTEM",
                    "identifier"=>"S-1-5-18",
                    "domain"=>"NT AUTHORITY"
                },
                "provider_guid"=>"...",
            },
            "event"=>{
                "module"=>"sysmon",
                "category"=>[
                    "process"
                ],
                "created"=>"2021-08-12T00:04:17.126Z",
                "type"=>[
                    "start",
                    "process_start"
                ],
                "provider"=>"Microsoft-Windows-Sysmon",
                "kind"=>"event",
                "action"=>"Process Create (rule: ProcessCreate)",
                "code"=>"1"
            },
            "@timestamp"=>2021-08-12T00: 04: 15.445Z,
            "agent"=>{
                "hostname"=>"...",
                "ephemeral_id"=>"...",
                "type"=>"winlogbeat",
                "version"=>"7.14.0",
                "name"=>"...",
                "id"=>"..."
            },
            "hash"=>{
                "md5"=>"118729dd62b9422d21afe5cebd564f8a",
                "sha256"=>"b76ce2bba63bd2949fa6e36fba963379b9d682f7642cd3782d9818fcd30a3e00",
                "imphash"=>"d2466cd4c38ecef8aab0ee78b79fc8b8"
            },
            "ecs"=>{
                "version"=>"1.10.0"
            },
            "user"=>{
                "name"=>"NETWORK SERVICE",
                "domain"=>"NT AUTHORITY",
                "id"=>"S-1-5-18"
            },
            "message"=>"Process Create:\nRuleName: -\nUtcTime: 2021-08-12 00:04:15.445\nProcessGuid: {a3a021f9-657f-6114-a9de-d94d01000000}\nProcessId: 14480\nImage: C:\\Windows\\System32\\gpupdate.exe\nFileVersion: 10.0.19041.572 (WinBuild.160101.0800)\nDescription: Microsoft® Group Policy Update Utility\nProduct: Microsoft® Windows® Operating System\nCompany: Microsoft Corporation\nOriginalFileName: GPUpdate.exe\nCommandLine: gpupdate.exe /target:computer\nCurrentDirectory: C:\\WINDOWS\\system32\\\nUser: NT AUTHORITY\\NETWORK SERVICE\nLogonGuid: {a3a021f9-4e57-60f0-e403-000000000000}\nLogonId: 0x3E4\nTerminalSessionId: 0\nIntegrityLevel: System\nHashes: MD5=118729DD62B9422D21AFE5CEBD564F8A,SHA256=B76CE2BBA63BD2949FA6E36FBA963379B9D682F7642CD3782D9818FCD30A3E00,IMPHASH=D2466CD4C38ECEF8AAB0EE78B79FC8B8\nParentProcessGuid: {a3a021f9-4e58-60f0-2b00-000000000b00}\nParentProcessId: 2764\nParentImage: C:\\Windows\\System32\\svchost.exe\nParentCommandLine: C:\\WINDOWS\\system32\\svchost.exe -k netsvcs -p -s Schedule",
            "@version"=>"1"
        }
    ],
    :response=>{
        "index"=>{
            "_index"=>"so-beats",
            "_type"=>"_doc",
            "_id"=>nil,
            "status"=>400,
            "error"=>{
                "type"=>"illegal_argument_exception",
                "reason"=>"field [process.pe.description] already exists"
            }
        }
    }
}
5

Yea that doesn't look like theres anything inherently wrong with the data.

Could be a problem with the pipeline?

GET /_ingest/pipeline/beats.common
1 Like
6

You are correct sir! There is an ingest pipeline in Elasticsearch that is modifying a whole bunch of the field names for sysmon data.

{ "rename":      { "field": "winlog.event_data.description",            "target_field": "process.pe.description",         "ignore_missing": true  } },

Once I got rid of this one for testing, it started erroring on process.pe.product. At this point I am going to dig into Elasticsearch documentation to see if I can figure out why this is happening. If you have ideas and you want to keep helping that's great, but if not I now have a direction to go with this and will keep updating this post until I figure it out.

I guess I should have mentioned this in the original post, but the system I am running is called Security Onion, in case that helps anyone out.

7

Yea I figured you were using security onion from "so-beats" as an index name. I haven't used security onion in quite a while. I'm guessing beats.common is some custom pipeline they have setup? All my beats pipelines have names that seem to mean they are specific to one beat.

If you haven't adjusted anything with pipelines yourself and this is just how it's working "out of the box" you might want to open an issue on their github or checkout whatever they're support channels are.

I'm not super experienced with ingest pipelines but i'm guessing somehow two different ones are trying to work on your data and that is causing these conflicts

8

Good thoughts. Thanks again!

9

Update: I determined that it is all of the process.pe.* fields and only those fields that are the issue. After removing all of the renames for those fields everything works correctly. I will keep on investigating and will update as I get answers.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.