Saved "field" parameter is now invalid. Please select a new field

Elastic Stack Kibana
1

Hi,

Currently using ELK 7.9.2.

I have been using Winlogbeat for a while now. I created some custom visualizations and dashboards that read from winlogbeat-* indexes. Everything was working fine just a few days ago, until now that many visualizations are showing the error message Saved "field" parameter is now invalid. Please select a new field..... I have read many posts related to this issue, that seems to be very common and almost sure related to wrong indexing, where ILM may be involved. I have tried everything, including removing indexes, templates, ILM policies, and even removing the beat from the host, then start from scratch, but the issue persist. Logstash is not part of the chain as the beat ships data directly to ES

Steps

  • Stopped Winlogbeat service
  • Uninstalled service
  • Removed installation folder
  • Closed/removed all Winlogbeat indexes
  • Removed Winlogbeat index template
  • Removed Winlogbeat ILM policy
  • Removed Winlogbeat index pattern
  • Made sure a new index it is not being created automatically due to an endpoint sending data
  • Reinstalled Winlogbeat following documented steps
    • Ran command winlogbeat setup. Command is always successful
  • Confirmed Index, index template, index pattern, ILM policy was added
  • Started Winlogbeat service

I could share part of my Winlogbeat config, but there is not a single line related to ILM (not to setting or modifying anything), as I prefer the auto-config to do all changes for me. I am using the basic beat settings, but connecting via ssl (which is working fine). I have spent hours breaking my brain trying to locate where the root issue is, so I can fix this.

I would appreciate some help here since I am stuck!

Thanks

2

The error message sounds a bit like in this issue:

So there is an issue with your index pattern, I think you're refreshed the fields in the index pattern management? Which version of Winlogbeat are you using?

Thx & Best,
Matthias

3

Hi @matw,

Thank you for jump in to assist me on this issue. Winlogbeat match ELK version. Both are v7.9.2. I am destroying everything I could find that is related to Winlogbeat: index, index template, index pattern, ILM policy, so I can have a fresh start from scratch. Can you please tell if I am missing anything else? Before removing the index I stop Winlogbeat in the endpoint, then on ES I refresh the index, clean index cache, and close the index.

What would you recommend me to try next?

Thank you

4

Could you export the index pattern saved object of winlogbeat-*, and paste the JSON here?
many thx!
Best,
Matthias

5

I cleaned up everything one more time.

Ran command winlogbeat setup from the endpoint (no errors):

Setup Winlogbeat

Went to check winlogbeat-* in Kibana/Saved Objects

Winlogbeat Saved Object
Winlogbeat Saved Object1191×576 17.4 KB

Pulled out Winlogbeat mapping data from Dev Console

Winlogbeat mapping
Winlogbeat mapping1366×246 11.7 KB

Winlogbeat index is also empty

Winlogbeat index
Winlogbeat index1021×689 37.9 KB

ndjson file

{"attributes":{"fieldFormatMap":"{\"client.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"client.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"client.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"destination.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"destination.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"destination.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"event.duration\":{\"id\":\"duration\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"},\"inputFormat\":\"nanoseconds\",\"outputFormat\":\"asMilliseconds\",\"outputPrecision\":1}},\"event.sequence\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"event.severity\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.request.body.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.request.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.response.body.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.response.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"http.response.status_code\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"log.syslog.facility.code\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"log.syslog.priority\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"network.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"package.size\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.pgid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.pid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.ppid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.parent.thread.id\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.pgid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.pid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.ppid\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"process.thread.id\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"server.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"server.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"server.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"source.bytes\":{\"id\":\"bytes\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"source.nat.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"source.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}},\"url.port\":{\"id\":\"string\",\"params\":{\"parsedUrl\":{\"origin\":\"https://192.168.1.190:5601\",\"pathname\":\"/app/dashboards\",\"basePath\":\"\"}}}}","fields":"[{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false}]","timeFieldName":"@timestamp","title":"winlogbeat-*"},"id":"winlogbeat-*","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2020-11-24T14:52:35.546Z","version":"WzQ3MDU0MjEsNTld"}
{"exportedCount":1,"missingRefCount":0,"missingReferences":[]}

So this seems to be the issue. Winlogbeat is not experiencing a wrong mapping, but instead is not mapping at all. Setup command is failing?

6

Hi @matw,

I enabled Winlogbeat at the endpoint to have some sample data. Mappings is now showing more fields, but most are classified as "text", which I am not sure is the expected result.

Here you can check the mapping fields. Text is too big to post it here

I tried to recreate one of the visualizations from scratch, but all terms are now available only in the .keyword form. In another ELK node I have for testing I don't have to use .keyword values to have working the same visualization.

So, where could be the issue? In the index template, or perhaps the data is is not arriving in the right format? Just in case I downloaded a fresh copy of Winlogbeat 7.9.2 and did the deployment from scratch, but the issue persist.

7

@matw,

I think the Dynamic Mapping is taking precedence over the regular mapping. Perhaps because from the start not all the fields are correctly mapped, and then all the fields are classified as new/unknown fields. I think the last couple of times I deployed Winlogbeat, I found around 800 fields ready to receive data and now I have less than 250, even sending data.

8

After several hours troubleshooting without being able to resolve this issue, I am sure the problem is still wrong indexing, but I can't find where or when is happening.....

9

think you're right, that's the issue, will ask around if this is a known issue or new. and how to fix it.

10

Hi @matw,

Where you able to find anything? Honestly I have no clue. I am experiencing the same behavior on v7.9.2 and v7.10. I was hopping that Winlogbeat 7.10 and its infrastructure would not have this issue, but I was wrong. The v7.10 it is not a fresh install, but an upgrade from v7.9.2, so it might be inheriting whatever the issue is. If you need further information about my node config, please let me know. I hope you and your team can help me to resolve this issue.

Thanks

11

Hi

Can you check if the winlogbeat index pattern is installed? E.g, here:

Bildschirmfoto 2020-11-25 um 15.49.17
Bildschirmfoto 2020-11-25 um 15.49.172766×1796 485 KB

Normally this should be installed when you run the setup command, but let's check if it's there

Best,
Matthias

12

Both Index Template and Index Pattern are installed after running the setup command from PowerShell console.

As I posted earlier, the Index Pattern is created with only 5 fields. The Index Pattern is added with the expected exported fileds.

13

Ok thanks, keep on asking (this is more a Beats question than a Kibana one, which is my territory
). What you could do is opening an issue in the beats repository


You've already done very good research, it certainly sounds like a bug, that was maybe not reported
There is something about ilm that creates invalid mappings:

It may be related to it?

Best,
Matthias

14

Hi @matw,

Thank you for staying. So, summarizing: there's no solution for this issue at the moment? I agree with you. I'm almost sure the wrong mapping involves ILM somehow, as I already had similar issue sometime ago. In fact very close to the issue described above. I also had to remove setup.ilm.check_exists: false from my config file, which I added due to the same PowerShell output after setup command. I though that issue was resolved by now.

What is weird is that due to the previous experience, I am not including setup.ilm.check_exists: false in my configs anymore, but still the issue appeared.

If there's nothing else you or your team can do for me at this moment, I will submit a bug report.

If we can test/try anything else in the meantime, please let me know.

Thank you

15

Hi @matw,

If you can, please invite a beat or Winlogbeat expert to this thread so he/she can help me narrow this issue. Perhaps a deep check to my steps or configs.

Thank you

16

Thx for opening the issue, just linking this here as a follow up!