Hopefully, someone can help:
In the SIEM section of Kibana, in the events section, I wish to customise the columns shown (I have parsed these logs using Logstash and wish to use these filters in the plugin).
It appears that it is based on categories and there is only one category there: Default ECS. Is there anyway I can create other categories or put my own filters in the columns?
Welcome to the community clearedskies!
Yes, you can create your own categories by using dotted notation in the field names, e.g. category1.fieldA, category1.fieldB.
To illustrate this, I created two new categories, category1 and category2 shown in the screenshot below:
Let's walk through the steps taken to create the screenshot above:
First, I created a new index using the Kibana Dev Tools with the following mappings:
PUT /deleteme_category_test
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"properties" : {
"field1" : { "type" : "text" },
"category1.fieldA" : { "type" : "text" },
"category1.fieldB" : { "type" : "text" },
"category2.fieldA" : { "type" : "text" }
}
}
}
In the mappings above:
category1 contains two fields (category1.fieldA and category1.fieldB)category2 contains one field (category2.fieldA)field1 does not use dotted notation, so it doesn't have a category, but I added it to show that we'll still be able see it in the Customize Columns view in the SIEM appNext, in Kibana Advanced settings, I added our new index, (deleteme_category_test), to the siem:defaultIndex setting:
apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, packetbeat-*, winlogbeat-*, deleteme_category_test
as shown in the screenshot below:
After saving the change to the siem:defaultIndex setting (to add our newly-created index) and navigating back to the Customize Columns view in the SIEM app, the new categories appear, as shown in the following screenshot:
The screenshot above illustrates:
field1 in our example) still appears in the Customize Columns view (just below category2)Feel free to let us know if that answers your question, or if you need more help. Thanks!
Hi @Andrew_G and thanks for the welcome.
I understand what you mean regarding the use of mappings and the principle idea behind it. I have already pointed SIEM to the correct indexes.
However, I have structured my output so Logstash outputs the logs to a new index each day, in the format of '''logstash-logs-%{+YYYY.MM.dd}''' .
This raises a problem of configuring the mappings. I would rather not have to manually, everyday, adjust the mappings of each index. Is there any solution that you know of?
Am I right in thinking that I would have to use index templates so these category fields are implemented and if so is it possible to use the Kibana interface to set these up and any idea on the notation needed?
Thanks
Following on from my previous response, I have had a play around with templates but I am not getting the desired response from what I have done.
Without any templates, and the index being created automatically the mapping structure is as follows:
{
"mapping": {
"_doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"agent": {
"type": "object"
},
"detail": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ecs": {
"type": "object"
},
"eventtime": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"host": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"hostname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"input": {
"type": "object"
},
"log": {
"properties": {
"file": {
"properties": {
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"process": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"program": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
When creating a template, should I be writing the fields in this section?
Also, when I do the same steps you did in creating the deleteme_category_test index, the category, category1, does not show in the Categories section in events:
As well as this another error appears:
Thanks for your replies @clearedskies!
With regards to your questions:
However, I have structured my output so Logstash outputs the logs to a new index each day, in the format of '''logstash-logs-%{+YYYY.MM.dd}''' .
This raises a problem of configuring the mappings. I would rather not have to manually, everyday, adjust the mappings of each index. Is there any solution that you know of?
Am I right in thinking that I would have to use index templates so these category fields are implemented and if so is it possible to use the Kibana interface to set these up and any idea on the notation needed?
and
Yes, it's possible to use Kibana (or the index template API) to define the mappings once, so they are automatically applied to every new index created that matches a pattern, like logstash-logs-%{+YYYY.MM.dd} in your example.
Let's adapt our previous example to your use case. We will build an index template in Kibana, and get the custom categories in the template to appear in the Customize Columns view in the SIEM app:
siem:defaultIndex settingFirst, in Kibana Advanced Settings, let's reset the siem:defaultIndex to the default value by clicking the Reset to default button:
After restoring the default, I reloaded the SIEM app and verified the custom categories from our previous example no longer appear:
In the next section, we'll create an Index Template using Kibana.
On the Index Templates tab of the Kibana Index Management page, click the Create a template button shown in the screenshot below:
In Step 1 (as shown in the screenshot below), enter logstash-logs-* in the Index patterns section so it matches the logstash-logs-%{+YYYY.MM.dd} indexes in your use case:
After entering the index pattern in the screenshot above, click Next > twice to advance to Step 3, Mappings, shown in the screenshot below:
Let's create the following (example) mappings:
"mappings" : {
"properties" : {
"field1" : { "type" : "text" },
"category1.fieldA" : { "type" : "text" },
"category1.fieldB" : { "type" : "text" },
"category2.fieldA" : { "type" : "text" }
}
}
Per the screenshot below, enter the mappings for our template:
After specifying the mappings in the screenshot above, click Next > twice to advance to Step 5, Review template, shown in the screenshot below:
Click the Create template button shown in the screenshot above to create the template.
We just created a new index template. In the next section, we'll verify the template is correctly applied to newly created indexes.
To verify the index template is correctly applied to newly-created indexes (note: it will not retroactively be applied to existing indexes) , I created a new index via Kibana Dev Tools to match the logstash-logs-%{+YYYY.MM.dd} format in your example:
PUT /logstash-logs-2020.05.28
After creating the above index, I filtered the Indices tab on the Index Management page to find the newly created index, as shown in the screenshot below:
As shown in the screenshot below, clicking on the newly-created index and selecting the Mappings tab reveals that our template was correctly applied to the index:
Now that we verified our index pattern is correctly being applied to newly-created indexes, we're ready to add the logstash-logs-* index pattern to the SIEM app in the next section.
logstash-logs-* to the siem:defaultIndex settingLet's add our logstash-logs-* index pattern to the SIEM app.
In Kibana Advanced settings, add , logstash-logs-* to the siem:defaultIndex setting:
apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, packetbeat-*, winlogbeat-*, logstash-logs-*
as shown in the screenshot below:
NOTE: Be sure to click the Save changes button after editing the above
We're now ready to view our custom categories in the SIEM app (in the next section).
Customize Columns viewAfter saving the changes in the previous section and reloading the SIEM app, we can now see our custom categories and fields in the Customize Columns view, as shown in the screenshot below:
The steps above accomplish our goal of getting custom categories and fields to appear in the Customize Columns view, which is (IMO) an advanced use case. That said, I think it's worth noting that the SIEM app was built from the ground up to use the Elastic Common Schema.
To unlock the full potential of the SIEM app, consider using the Beats specified in the siem:defaultIndex setting, i.e. Auditbeat, Filebeat, Packetbeat, Winlogbeat.
These Beats all conform to the Elastic Common Schema, so ingesting data with them will, for example, enable SIEM features like the Timeline to render events as shown in the screenshot below:
Consider getting started by ingesting some data with Auditbeat by clicking the Add data button in the screenshot below, and then choosing Auditbeat:
From there, Kibana will guide you through the process of downloading Auditbeat, configuring it, and ingesting data.
I would agree the method you have outlined should work, yet mine apparently will not. I understand that inputting directly through Beats would mean an easier integration, however, the logs I am using would need to be parsed so my set up is currently, Filebeat --> Logstash --> Elasticsearch --> Kibana.
Going back to the original issue, can you also show the configuration of how you mapped the template for Dynamic Templates and advanced options. The format of my index is very different to yours and despite the steps you have outlined my SIEM Categories continue to only show 'default ECS'.
(Appreciate the help on this small use case)
I'm wondering:
Is the index template (containing the custom categories) being associated with newly-created indexes?
Does the index pattern specified in the siem:defaultIndex setting match the logstash-logs-* indexes?
To answer the first question, "Is the index template (containing the custom categories) being associated with newly-created indexes?", would you be willing to click on an index that was created after the index template (with the custom fields) existed, and post a screenshot that shows both the full index name (e.g. logstash-logs-2020.05.28) and the Mappings tab in the same screenshot, like the example below?
To answer the second question, "Does the index pattern specified in the siem:defaultIndex setting match the logstash-logs-* indexes?", would you be willing to copy-paste the value of that setting, like the example below?
apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, packetbeat-*, winlogbeat-*, logstash-logs-*
Thanks @clearedskies!
Apologies for late response @Andrew_G
I have done the process another time. I am perplexed but for some reason the categories are now appearing along with others as well. Very odd, and I am going to try and find out what I did differently. This has been very useful; thanks
Thanks for the update @clearedskies, and happy to hear that the custom categories are appearing! 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.