How to add a new field to all my log indices

[Goishin]

I'm a new kibana user and you guys have been invaluable to me over the past two months while I get my feet under me. Now I've got a question about logging data. I upload my testpass logs to cloudwatch which then shuffles the logs over to my kibana cluster. Almost all of this was done using the AWS-provided tooling. I did not modify anything with regards to how the log stuff is setup. In the @message field that comes in, I have two fields that are prepended to each message. Now, I want to use those two fields to add those log messages to a dashboard. But Kibana doesn't know they're fields. They just show up in the @message field. How do I tell Kibana to use them as fields, and not just have them as part of the @message field? To make matters worse, I've ended up with a bunch of log streams, each log stream ended up as a separate index in kibana, and all the indexes are addressed as one, "cwl-*". How do I add those two new fields to all the cloudwatch indices?

[nickpeihl]

So you don't see the new fields in Kibana? Perhaps you need to refresh your index pattern. https://www.elastic.co/guide/en/kibana/current/managing-fields.html#_manage_your_index_pattern

[Goishin]

Hey Nick! (that's my name too!)
Thanks for the response. However, the index doesn't know that I've done anything yet. The way that I added the fields I'm interested in is I just modified the log event message string (the string you find in @message) to include my terms. Here's an example:
Before:
@message = "I'm a log statement!"
After:
@message = "[special.variable:13423-2342-23422-12132] [second special variable:123456] I'm a log statement!"
How do I tell Kibana about my special variable and second special variable fields? Is this something I do in cloudwatch? Or in Kibana using a scripted field?

[nickpeihl]

Hi Other Nick ()

It sounds like you will want to split the @message field into separate special.variable fields when ingesting them into Elasticsearch. I'm not familiar with your Cloudwatch setup, but maybe you can do this with an ingest processor like grok?

[Goishin]

That's it! That's it!!!! Yes! That's how I do it. Ok, cool. Thanks a bunch, Nick

Let me run off and figure out how to get this doing what I want it to.