I'm trying to rename user fields from several different sources to user.name to match ECS but none will work. If I leave it as user then the field shows up but when I try to rename it to user.name, I see all the other fields in ElasticSearch except this one and I don't get any errors from logstash. Below is my mutate function and I have tried variations on this from dropping the [ ] brackets to other values like username but none work. I have other values that I can rename such as src to source.ip but the user.name one doesn't take. Any ideas why I can't rename it this?
Would it work if you send the event to a new index rather than an already created one?
My guess is it doesn't work since the user field is already mapped as a text field in your current index, while you're trying to treat it as a object field (by nesting the name field in it).
Please try to send the event you're renaming the field in to a different (new) index and see if it works.
I believe you are on the right track. I'm not able to send this to a new index however we have a * index that is supposed to be for searching all indexes and this one had the user.name field in it. I can delete this index mapping and when I did, the user.name kind of gets created.
As you can see below, it creates the user but puts the .name portion in { } brackets and not as user.name.
Mhmh I don't really get what you wrote. Guess you are confusing the index with the index-pattern. The latter is only a "mask" to see your data. If you delete it, you're not changing the mapping of the index. The only way to change a mapping for a field that already exists in a index is to reindex your data (applying a mapping BEFORE reindexing the events).
Now, what you posted above (the table view and the JSON view) are exactly the same thing, a user field which is of type object and has a nested field name in it.
If you explain here what exactly you would like to achieve, I could help you get it done.
Would you be ok for you to have the value of that field in another field user_name or username? If that's the case, you can avoid the reindex. If you want to have a root field user.name, you'll need to reindex the data.
Though, I sincerely recommend you to avoid such a choice.
Yes, I must have confused index with index-pattern although I'm still not sure why deleting the * index pattern allowed me to see the user in the correct field. Other fields that I've renamed don't show in this pattern, they show like source.ip instead of source { "ip" : 1.2.3.4 } in the Kibana table view. The json views look correct for all the renamed fields.
Ideally we would want all usernames from each index mapped to the ECS field of user.name. We need to reindex other older indexes that aren't mapped properly (and that's beyond the scope for this question). I'm trying to map this new data source to ECS so that it will eventually align with the others when they are reindexed.
I think you are still making a bit of confusion. I'll try to make things as clear as possible down below and give you couple of tips.
First of all, I sincerely recommend you not to use such an index-pattern like *. It totally makes it useless the whole concept of having an index pattern. It forces the system to make searches over ALL the indices and the risk of having conflicting fields is extremely high.
The general concept in Elastic is to use indices and index-patterns so as to create separate domains and contexts. If you use a * index pattern, you are destroying this system (unless you use it for very peculiar goals).
Now:
I'm still not sure why deleting the * index pattern allowed me to see the user in the correct field
Where were you able to see such fields? In the Discover section? In a visualization? Can you take a screenshot of it and post it here? My guess is you either didn't refresh the page after deleting the pattern or you are using a different pattern on that page.
they show like source.ip instead of source { "ip" : 1.2.3.4 } in the Kibana table view
Kibana will always show you the fields with the dot notation in the table view of the Discover section. It's the way it represents both nested fields and root fields with dots in their names. That's why you have to check the JSON view of the Discover section (or the mapping of the index in the Management>Elasticsearch>Index Management section) to see how that field is actually ingested and mapped in the index.
Ideally we would want all usernames from each index mapped to the ECS field of user.name
Be careful here, because the ECS does use the dot notation to create NESTED objects. It means that the fields with dots you see in the ECS compliant indices, are object fields, containing other fields. Check in the JSON view to have a better understanding of it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
