How to manage elasticsearch logs: disk almost full

Elastic Stack Elasticsearch
1

Dear Users,

I'm a newbie and this is the first time I use ELK stack.
I was able to deploy ELK stack and capture needed information. I created my first dashboard and visualizations. They are working as expected: we are capturing logs from two data delivery servers (let's say they are similar to FTP server).
So, in visualizations, we can see a lot of interesting statistics about: total amount of transferred data (daily, monthly, yearly and so on).

Unfortunately, I noticed that the used disk space is growing and disk is going to be full.
I read that the logs retention time can be set, some logs can be removed, ilm policy can be set and elasticsearch curator is a valid solution.

My questions are: do you have some particular suggestions about the right procedure to solve my problem without losing statistics already captured ?! what is the best procedure to reach the target? What happens to the "total amount of transferred data" value if I remove the logs?

My ELK stack version should be v.6.2.4 ( I read that in Kibana dashboard).

Could you please help me to detect the right way to solve the issue?

Many thanks to all of you.
Regards,
Mauro

2

You may consider first to summarize your raw data using rollup, then ceate vizualisations & dashboards from summarized data, then appy ILM to raw indices

1 Like
3

Many thanks, ylasri. I think it is a very hard task for a newbie, but I can search on google some example that could help me.

Do you think that this procedure will delete all information and statistics already collected and saved in the visualizations? in other words, I will lose the total amount of transferred files and sizes?

Thank you,
Mauro

4

Sorry, I think rollup feature is not included in v.6.2.4.
Since I can't change the version (I'm using a customized docker container), what is the alternative solution? Thanks.

5

I used to be in the same situation where I used curator to purge my indices and python script to save my aggregates into an other indices...
Now I use more ILM, Roll up and Transform...

I would suggest to write external script that summarize data as per your requirements and save the summary into new indices and use them to build viz

1 Like
6

Thank you. Last question, I promise.
I noticed that the most important part of big files is in /var/log/logstash.
What happen if I remove the files with ending with the date? Where I can find the retention and rotation time of these log files?

[root@imon logstash]# ll
total 17078132
-rw-r--r-- 1 polkitd input 736998825 Oct 30 00:59 logstash-plain-2020-10-29.log
-rw-r--r-- 1 polkitd input 9976000451 Oct 31 00:59 logstash-plain-2020-10-30.log
-rw-r--r-- 1 polkitd input 1730356293 Nov 1 00:59 logstash-plain-2020-10-31.log
-rw-r--r-- 1 polkitd input 2201159922 Nov 2 00:59 logstash-plain-2020-11-01.log
-rw-r--r-- 1 polkitd input 1796597232 Nov 2 20:03 logstash-plain.log

7

You're welcome
Feel free to ask
Yes you can clean these logs
Are you running logstash In debug mode? Or there are errors in logs?

1 Like
8

Thank you for the time you are spending for me.
No, logstash is not in debug mode, but I checked the files and there are errors in logs :frowning:

This is a line I extracted from log file:
[2020-11-02T00:00:04,166][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('\' (code 92)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: (String)"\u0000Sw\xB1\u0000\u0000\u00061__BEGIN_JSON__{\"time_stamp\": \"1604275204164\", \"file_descriptor\": \"13\", \"repl_requested\": \"-1\", \"user_user_other_i nfo_user_info\": \"\", \"hostname\": \"irs01.cmcc.scc\", \"proxy_auth_info_auth_str\": \"\", \"rule_name\": \"audit_pep_resource_write_post\", \"dataId\": \"0\", \"proxy_auth_info _flag\": \"0\", \"pid\": \"35868\", \"std_string\": \"LtfsCache01\", \"logical_path\": \"/idas/home/sp1/CMCC-SPS3.5/07/2005/031/sps3.5_200507_031.cam.h.2005-11.tar\"[truncated 131 6 chars]; line: 1, column: 2]>, :data=>"\\u0000Sw\\xB1\\u0000\\u0000\\u00061__BEGIN_JSON__{\\\"time_stamp\\\": \\\"1604275204164\\\", \\\"file_descriptor\\\": \\\"13\\\", \\\"repl _requested\\\": \\\"-1\\\", \\\"user_user_other_info_user_info\\\": \\\"\\\", \\\"hostname\\\": \\\"irs01.cmcc.scc\\\", \\\"proxy_auth_info_auth_str\\\": \\\"\\\", \\\"rule_name\\ \": \\\"audit_pep_resource_write_post\\\", \\\"dataId\\\": \\\"0\\\", \\\"proxy_auth_info_flag\\\": \\\"0\\\", \\\"pid\\\": \\\"35868\\\", \\\"std_string\\\": \\\"LtfsCache01\\\", \\\"logical_path\\\": \\\"/idas/home/sp1/CMCC-SPS3.5/07/2005/031/sps3.5_200507_031.cam.h.2005-11.tar\\\", \\\"l1_desc_idx\\\": \\\"-1\\\", \\\"auth_scheme\\\": \\\"native\\\", \\ \"file_size\\\": \\\"0\\\", \\\"proxy_user_other_info_user_info\\\": \\\"\\\", \\\"client_addr\\\": \\\"10.0.1.76\\\", \\\"proxy_rods_zone\\\": \\\"idas\\\", \\\"proxy_user_type\\ \": \\\"\\\", \\\"proxy_user_other_info_user_create\\\": \\\"\\\", \\\"dataType\\\": \\\"\\\", \\\"flags_kw\\\": \\\"0\\\", \\\"proxy_sys_uid\\\": \\\"0\\\", \\\"proxy_user_name\\ \": \\\"rods\\\", \\\"in_pdmo\\\": \\\"\\\", \\\"user_user_name\\\": \\\"sp1\\\", \\\"mode_kw\\\": \\\"0\\\", \\\"physical_path\\\": \\\"/ltfscache/idas/home/sp1/CMCC-SPS3.5/07/20 05/031/sps3.5_200507_031.cam.h.2005-11.tar\\\", \\\"proxy_auth_info_auth_flag\\\": \\\"5\\\", \\\"proxy_auth_info_auth_scheme\\\": \\\"\\\", \\\"proxy_auth_info_host\\\": \\\"\\\" , \\\"proxy_auth_info_ppid\\\": \\\"0\\\", \\\"proxy_user_other_info_user_comments\\\": \\\"\\\", \\\"user_auth_info_auth_str\\\": \\\"\\\", \\\"proxy_user_other_info_user_modify\ \\": \\\"\\\", \\\"user_user_other_info_user_comments\\\": \\\"\\\", \\\"resc_hier\\\": \\\"LtfsCache01\\\", \\\"user_auth_info_auth_flag\\\": \\\"3\\\", \\\"user_auth_info_auth_s cheme\\\": \\\"\\\", \\\"user_auth_info_flag\\\": \\\"0\\\", \\\"user_auth_info_host\\\": \\\"\\\", \\\"int\\\": \\\"41943040\\\", \\\"user_auth_info_ppid\\\": \\\"0\\\", \\\"user _rods_zone\\\": \\\"idas\\\", \\\"user_user_other_info_user_modify\\\": \\\"\\\", \\\"user_sys_uid\\\": \\\"0\\\", \\\"user_user_other_info_user_create\\\": \\\"\\\", \\\"user_use r_type\\\": \\\"\\\", \\\"std_string_ptr\\\": \\\"\\\", \\\"ERROR\\\": \\\"[void*] not supported\\\"}__END_JSON__"}

I'm trying to check iRODS data transfer. Do you know iRODS!?

9

No idea about iRODS
But you have to fix those errors
If not runned in debug and no error is generated logstash does not generate logs...

10

Do you have some ideas about the meaning of this message ?

Unexpected character ('\' (code 92)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: (String)"\u0000Sw\xB1\u0000\u0000\u00061__BEGIN_JSON_

So, the problem is in the source host?
Thanks

11

Looks like your logstash pipeline is configured to parse json events but some of them are not good serialized...

12

So, if I'm not wrong, if I solve the errors related to logstash I can solve also the disk full issue...it seems that elasticsearch doesn't require a lot of space in my deploy.
If I understand what I read, elasticsearch is a database, right? And I should work on it if I need to save only information related to the last 3 months (for example)...

Could you please say me if my understanding is correct?

If yes, I will try to understand the content of this thread Rotating ES Logs

13

Yes!

1 Like
14

Many many thanks for your availability :slight_smile:
Let's try to understand what is "logstash pipeline" :smiley:

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.