X-pack not generating indices

[Sjaak01 (Sjaak)]

Hi,

I installed the X-pack but it is not generating the reporting etc. indices even though I can generate and download reports.

My user has superuser rights.

I've got a default 5.4.1 installation, no strange settings or anything. Also tried adding action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history* to the elasticsearch.yml but no change.

So I can generate reports but there is no way to delete them.

[bohyun (Bohyun Kim)]

Hi @Sjaak01

Are you trying to view those indices in the X-Pack monitoring views? If so, we hide the system indices from the UI by default.

Can you go to dev tools > console then run GET _cat/indices and let me know what you see as a result?

Thanks,
Bohyun

[Sjaak01 (Sjaak)]

Hi bohyun,

I'm looking at the discovery and management pages.

green open .security D64Mr9eWSD-pn8NytYbnjQ 1 0 5 0 18.2kb 18.2kb yellow open .monitoring-logstash-2-2017.06.12 DH8SDIcSQIiczTXRmO1k7A 1 1 761 0 290.4kb 290.4kb yellow open .monitoring-data-2 zjQZzK6zTb2qcMPcGwDviA 1 1 4 0 14.8kb 14.8kb yellow open .monitoring-kibana-2-2017.06.13 J3Uq22MCQbuy38GS3fWbZw 1 1 800 0 580.5kb 580.5kb yellow open netflow-2017.06.13 yiclbiejQ_KG3YB_9y0-Iw 5 1 2025 0 2.2mb 2.2mb yellow open netflow-2017.06.07 M6DsBa11RBOJIXoJ_2hAXw 5 1 12009 0 4.6mb 4.6mb yellow open fortinet-2017.05.31 n7IqKl6SQ1i7OLwB80ynrA 5 1 0 0 800b 800b yellow open netflow-2017.06.10 5kdaguubRxikcHMI5mrv2w 5 1 6933 0 2.3mb 2.3mb yellow open netflow-2017.06.02 JijN_irNQWmSHOsjIYbJwQ 5 1 7543 0 3.1mb 3.1mb yellow open netflow-2017.06.11 tpVld786RrSr4RJH3xZP5A 5 1 7273 0 2.6mb 2.6mb yellow open netflow-2017.06.05 0JV09I3AScyPB_reN2nbPQ 5 1 4150 0 1.7mb 1.7mb yellow open .watcher-history-3-2017.06.12 LUFw5626R-ybBI6_M70y0w 1 1 6580 0 5.2mb 5.2mb yellow open netflow-2017.06.04 aerjxWttSeWf3Q-lxKo-uA 5 1 3456 0 1.3mb 1.3mb yellow open .monitoring-kibana-2-2017.06.12 UfQT7wJ5QXuOvLM64hdX0g 1 1 7742 0 1.7mb 1.7mb yellow open .triggered_watches 5bY_PshnTKquBrHxv45iMw 1 1 0 0 9.5kb 9.5kb yellow open netflow-2017.06.03 yQLKPWVWRPGNuRrn3wIAyg 5 1 3236 0 1.2mb 1.2mb yellow open .monitoring-es-2-2017.06.13 T4sAsyRyS_CoehpOcZOqDA 1 1 29141 1050 37.3mb 37.3mb yellow open .monitoring-alerts-2 kad4BP3sRD2g50BRHyetmQ 1 1 1 0 13kb 13kb yellow open .reporting-2017.06.11 5oTVBgDQQ8Sbu5UW3FyYXg 5 1 3 0 1.6mb 1.6mb yellow open netflow-2017.06.06 mKcuCZLmTz-8tzRkM_Mbiw 5 1 10111 0 3.8mb 3.8mb yellow open .monitoring-es-2-2017.06.12 U3050LO_RB2FyJPyBm0dPw 1 1 230535 1192 147.5mb 147.5mb yellow open netflow-2017.06.09 k2aO-kSyS6WV9w6PFcqTkQ 5 1 8151 0 3.1mb 3.1mb yellow open netflow-2017.06.08 2nS4Gq3PQdSINEcuQALkkQ 5 1 12833 0 5.2mb 5.2mb yellow open .watches nzqRJ02mTm29PG65MWrw8g 1 1 4 0 23.3kb 23.3kb yellow open netflow-2017.06.12 MbVNcYB7RdKW--FvEaA1Dw 5 1 8758 0 3.4mb 3.4mb yellow open netflow-2017.06.01 rCLBvL2mR9eHcICt4b_NLw 5 1 4274 0 1.9mb 1.9mb yellow open .watcher-history-3-2017.06.13 C9gCiG7tTtu2Apes6l5Bpw 1 1 660 0 1.3mb 1.3mb yellow open .kibana Dhsw8v4YSgaYKOK0ReXKGA 1 1 57 2 81.7kb 81.7kb

Edit: Above shows a reporting index for the 11th, I generated some reports on the 13th.

[bohyun (Bohyun Kim)]

Hello @Sjaak01

X-Pack reporting generates weekly indices so you will not see .reporting-2017.06.13 as a result.

For more details about reporting index, please go to https://www.elastic.co/guide/en/x-pack/current/reporting-settings.html#reporting-advanced-settings

Hope this helps,
Bohyun

[Sjaak01 (Sjaak)]

I see. So if I want to delete a report I can't delete a single report but instead will have to delete at least one week?

Is there any way to make these indices visible so I can look at them and maybe search through them? I'm new to X-pack so not entire sure what is possible yet but for a watch I intend to make I will probably need to look at the watch history as well to avoid sending double alerts.

[bohyun (Bohyun Kim)]

So if I want to delete a report I can't delete a single report but instead will have to delete at least one week?

Correct.

Is there any way to make these indices visible so I can look at them and maybe search through them?

To view the raw objects in your .reporting index, run GET .reporting-2017.06.11/_search and you will be able to see all of the reports that were generated from 2017.06.11 to today.

I'm new to X-pack so not entire sure what is possible yet but for a watch I intend to make I will probably need to look at the watch history as well to avoid sending double alerts.

What Watch are you trying to make?

[Sjaak01 (Sjaak)]

@bohyun

To view the raw objects in your .reporting index, run GET .reporting-2017.06.11/_search and you will be able to see all of the reports that were generated from 2017.06.11 to today.

Thanks.

Something like this but because I don't know what I'm doing its not really going anywhere at the moment.

Alert on SUM of field exeeding certain value?

Hey, this sounds possible to me. The important part here is to write a proper search query (independent from the watch). The search query needs to do the following have a range filter in your query that filters from the first of the month until now have a term filter in your query (or a match query), that filters for the host you are interested in have a sum aggregation in your request, that counts up your bytes (use the sum aggregation Now you got the correct data (a single aggregation re…

[bohyun (Bohyun Kim)]

Hey @Sjaak01

I can go comment on that other post on Watcher question. Can you flag this post as resolved so we can mark it as completed?

Thanks,
Bohyun