Saved "field" parameter is now invalid. Please select a new field

ManuelF · November 23, 2020, 10:01pm

Hi,

Currently using ELK 7.9.2.

I have been using Winlogbeat for a while now. I created some custom visualizations and dashboards that read from winlogbeat-* indexes. Everything was working fine just a few days ago, until now that many visualizations are showing the error message Saved "field" parameter is now invalid. Please select a new field..... I have read many posts related to this issue, that seems to be very common and almost sure related to wrong indexing, where ILM may be involved. I have tried everything, including removing indexes, templates, ILM policies, and even removing the beat from the host, then start from scratch, but the issue persist. Logstash is not part of the chain as the beat ships data directly to ES

Steps

Stopped Winlogbeat service
Uninstalled service
Removed installation folder
Closed/removed all Winlogbeat indexes
Removed Winlogbeat index template
Removed Winlogbeat ILM policy
Removed Winlogbeat index pattern
Made sure a new index it is not being created automatically due to an endpoint sending data
Reinstalled Winlogbeat following documented steps
- Ran command winlogbeat setup. Command is always successful
Confirmed Index, index template, index pattern, ILM policy was added
Started Winlogbeat service

I could share part of my Winlogbeat config, but there is not a single line related to ILM (not to setting or modifying anything), as I prefer the auto-config to do all changes for me. I am using the basic beat settings, but connecting via ssl (which is working fine). I have spent hours breaking my brain trying to locate where the root issue is, so I can fix this.

I would appreciate some help here since I am stuck!

Thanks

matw · November 24, 2020, 8:19am

The error message sounds a bit like in this issue:

So there is an issue with your index pattern, I think you're refreshed the fields in the index pattern management? Which version of Winlogbeat are you using?

Thx & Best,
Matthias

ManuelF · November 24, 2020, 2:13pm

Hi @matw,

Thank you for jump in to assist me on this issue. Winlogbeat match ELK version. Both are v7.9.2. I am destroying everything I could find that is related to Winlogbeat: index, index template, index pattern, ILM policy, so I can have a fresh start from scratch. Can you please tell if I am missing anything else? Before removing the index I stop Winlogbeat in the endpoint, then on ES I refresh the index, clean index cache, and close the index.

What would you recommend me to try next?

Thank you

matw · November 24, 2020, 2:47pm

Could you export the index pattern saved object of winlogbeat-*, and paste the JSON here?
many thx!
Best,
Matthias

ManuelF · November 24, 2020, 3:05pm

I cleaned up everything one more time.

Ran command winlogbeat setup from the endpoint (no errors):

Setup Winlogbeat

Went to check winlogbeat-* in Kibana/Saved Objects

Pulled out Winlogbeat mapping data from Dev Console

Winlogbeat index is also empty

ndjson file

{"attributes":{"fieldFormatMap":"{\"client.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"client.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"client.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"destination.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"destination.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"destination.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"event.duration\":{\"id\":\"duration\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"},\"inputFormat\":\"nanoseconds\",\"outputFormat\":\"asMilliseconds\",\"outputPrecision\":1}},\"event.sequence\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"event.severity\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.request.body.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.request.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.response.body.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.response.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.response.status_code\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"log.syslog.facility.code\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"log.syslog.priority\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"network.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"package.size\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.pgid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.pid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.ppid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.thread.id\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.pgid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.pid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.ppid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.thread.id\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"server.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"server.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"server.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"source.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"source.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"source.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"url.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}}}","fields":"[{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false}]","timeFieldName":"@timestamp","title":"winlogbeat-*"},"id":"winlogbeat-*","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2020-11-24T14:52:35.546Z","version":"WzQ3MDU0MjEsNTld"}
{"exportedCount":1,"missingRefCount":0,"missingReferences":[]}

So this seems to be the issue. Winlogbeat is not experiencing a wrong mapping, but instead is not mapping at all. Setup command is failing?

ManuelF · November 24, 2020, 5:03pm

Hi @matw,

I enabled Winlogbeat at the endpoint to have some sample data. Mappings is now showing more fields, but most are classified as "text", which I am not sure is the expected result.

Here you can check the mapping fields. Text is too big to post it here

I tried to recreate one of the visualizations from scratch, but all terms are now available only in the .keyword form. In another ELK node I have for testing I don't have to use .keyword values to have working the same visualization.

So, where could be the issue? In the index template, or perhaps the data is is not arriving in the right format? Just in case I downloaded a fresh copy of Winlogbeat 7.9.2 and did the deployment from scratch, but the issue persist.

ManuelF · November 24, 2020, 5:49pm

@matw,

I think the Dynamic Mapping is taking precedence over the regular mapping. Perhaps because from the start not all the fields are correctly mapped, and then all the fields are classified as new/unknown fields. I think the last couple of times I deployed Winlogbeat, I found around 800 fields ready to receive data and now I have less than 250, even sending data.

ManuelF · November 24, 2020, 9:36pm

After several hours troubleshooting without being able to resolve this issue, I am sure the problem is still wrong indexing, but I can't find where or when is happening.....

matw · November 25, 2020, 5:07am

think you're right, that's the issue, will ask around if this is a known issue or new. and how to fix it.

ManuelF · November 25, 2020, 1:49pm

Hi @matw,

Where you able to find anything? Honestly I have no clue. I am experiencing the same behavior on v7.9.2 and v7.10. I was hopping that Winlogbeat 7.10 and its infrastructure would not have this issue, but I was wrong. The v7.10 it is not a fresh install, but an upgrade from v7.9.2, so it might be inheriting whatever the issue is. If you need further information about my node config, please let me know. I hope you and your team can help me to resolve this issue.

Thanks

matw · November 25, 2020, 2:54pm

Hi

Can you check if the winlogbeat index pattern is installed? E.g, here:

Normally this should be installed when you run the setup command, but let's check if it's there

Best,
Matthias

ManuelF · November 25, 2020, 3:18pm

Both Index Template and Index Pattern are installed after running the setup command from PowerShell console.

As I posted earlier, the Index Pattern is created with only 5 fields. The Index Pattern is added with the expected exported fileds.

matw · November 25, 2020, 3:51pm

Ok thanks, keep on asking (this is more a Beats question than a Kibana one, which is my territory
). What you could do is opening an issue in the beats repository

You've already done very good research, it certainly sounds like a bug, that was maybe not reported
There is something about ilm that creates invalid mappings:

It may be related to it?

Best,
Matthias

ManuelF · November 25, 2020, 4:45pm

Hi @matw,

Thank you for staying. So, summarizing: there's no solution for this issue at the moment? I agree with you. I'm almost sure the wrong mapping involves ILM somehow, as I already had similar issue sometime ago. In fact very close to the issue described above. I also had to remove setup.ilm.check_exists: false from my config file, which I added due to the same PowerShell output after setup command. I though that issue was resolved by now.

What is weird is that due to the previous experience, I am not including setup.ilm.check_exists: false in my configs anymore, but still the issue appeared.

If there's nothing else you or your team can do for me at this moment, I will submit a bug report.

If we can test/try anything else in the meantime, please let me know.

Thank you

ManuelF · November 25, 2020, 4:52pm

Hi @matw,

If you can, please invite a beat or Winlogbeat expert to this thread so he/she can help me narrow this issue. Perhaps a deep check to my steps or configs.

Thank you

matw · November 26, 2020, 5:00pm

Thx for opening the issue, just linking this here as a follow up!