How to rotate logstash.log in ELK

kartikelastic · October 24, 2015, 9:56pm

Hi team,

How to rotate /var/log/logstash/logstash.log? It fills up the drive and then everything breaks. Simply doing a
server# > /var/log/logstash/logstash.log does not work
What I have done is downloaded "curator" https://github.com/elasticsearch/curator
and I do this (NOT AT ALL the best approach):

/var/log/logstash/logstash.log
curator delete indices --all-indices
Create an index with:
curl -XPUT 'http://localhost:9200/twitter/' -d '{
"settings" : {
"index" : {
"numberofshards" : 3,
"numberofreplicas" : 2
}
}
}'
Someone please advise me on best practices in this regard.

magnusbaeck · October 25, 2015, 9:10am

How to rotate /var/log/logstash/logstash.log?

Use the standard logrotate tool. Elastic's RPM and Debian packages include a configuration for it:

github.com

elastic/logstash/blob/main/pkg/logrotate.conf

/var/log/logstash/*.log /var/log/logstash/*.err /var/log/logstash/*.stdout {
        daily
        rotate 7
        copytruncate
        compress
        delaycompress
        missingok
        notifempty
}

It fills up the drive and then everything breaks.

Logstash doesn't log much data, so unless you're

running it with --verbose or --debug or
still have a stdout output enabled or
are experiencing tons of errors

it shouldn't fill any disks. You should look into why the log is growing so much.

kartikelastic · October 25, 2015, 11:50am

well the reason /var/log/logstash/logstash.log is increasing in size is that it is taking all windows server event logs. Could I have set those logs to include "information" level logging? I guess I will have to check the nglog.conf on the windows servers

magnusbaeck · October 25, 2015, 11:58am

If you give a few example lines from /var/log/logstash/logstash.log we can advise you how to reduce the amount of logging.

kartikelastic · October 25, 2015, 12:03pm

at [Source: [B@6b831825; line: 1, column: 10]>, :level=>:warn}
{:timestamp=>"2015-10-24T13:35:39.755000-0400", :message=>"Trouble parsing json", :source=>"message", :raw=>"Auditing settings on object were changed.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK-HP-I3$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nObject:\r\n\tObject Server:\tSecurity\r\n\tObject Type:\tFile\r\n\tObject Name:\tC:\Windows\SysWOW64\en-US\adtschema.dll.mui\r\n\tHandle ID:\t0x18\r\n\r\nProcess Information:\r\n\tProcess ID:\t0x145c\r\n\tProcess Name:\tC:\Windows\System32\poqexec.exe\r\n\r\nAuditing Settings:\r\n\tOriginal Security Descriptor:\t\r\n\tNew Security Descriptor:\t\tS:ARAI(AU;SAFA;DCLCRPCRSDWDWO;;;WD)", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Auditing': was expecting ('true', 'false' or 'null')
at [Source: [B@6dba1b94; line: 1, column: 10]>, :level=>:warn}
{:timestamp=>"2015-10-24T13:35:39.756000-0400", :message=>"Trouble parsing json", :source=>"message", :raw=>"Auditing settings on object were changed.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK-HP-I3$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nObject:\r\n\tObject Server:\tSecurity\r\n\tObject Type:\tFile\r\n\tObject Name:\tC:\$Windows.~BT\Sources\SafeOS\SafeOS.Mount\Program Files (x86)\desktop.ini\r\n\tHandle ID:\t0xc10\r\n\r\nProcess Information:\r\n\tProcess ID:\t0x1118\r\n\tProcess Name:\tC:\$Windows.~BT\Sources\SetupHost.exe\r\n\r\nAuditing Settings:\r\n\tOriginal Security Descriptor:\t\r\n\tNew Security Descriptor:\t\tS:ARAI", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Auditing': was expecting ('true', 'false' or 'null')
at [Source: [B@56d9c1f9; line: 1, column: 10]>, :level=>:warn}
{:timestamp=>"2015-10-24T13:35:39.757000-0400", :message=>"Trouble parsing json", :source=>"message", :raw=>"Auditing settings on object were changed.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK-HP-I3$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nObject:\r\n\tObject Server:\tSecurity\r\n\tObject Type:\tFile\r\n\tObject Name:\tC:\Windows\SysWOW64\en-US\msaudite.dll.mui\r\n\tHandle ID:\t0x18\r\n\r\nProcess Information:\r\n\tProcess ID:\t0x145c\r\n\tProcess Name:\tC:\Windows\System32\poqexec.exe\r\n\r\nAuditing Settings:\r{:timestamp=>"2015-10-24T13:35:39.758000-0400", :message=>"Trouble parsing json", :source=>"message", :raw=>"Auditing settings on object were changed.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK-HP-I3$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nObject:\r\n\tObject Server:\tSecurity\r\n\tObject Type:\tFile\r\n\tObject Name:\tC:\Windows\SysWOW64\en-US\auditpol.exe.mui\r\n\tHandle ID:\t0x18\r\n\r\nProcess Information:\r\n\tProcess ID:\t0x145c\r\n\tProcess Name:\tC:\Windows\System32\poqexec.exe\r\n\r\nAuditing Settings:\r\n\tOriginal Security Descriptor:\t\r\n\tNew Security Descriptor:\t\tS:ARAI(AU;SAFA;DCLCRPCRSDWDWO;;;WD)", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Auditing': was expecting ('true', 'false' or 'null')
at [Source: [B@b2a4f66; line: 1, column: 10]>, :level=>:warn}
{:timestamp=>"2015-10-24T13:35:39.759000-0400", :message=>"Trouble parsing json", :source=>"message", :raw=>"Auditing settings on object were changed.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK-HP-I3$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nObject:\r\n\tObject Server:\tSecurity\r\n\tObject Type:\tFile\r\n\tObject Name:\tC:\Windows\System32\credui.dll\r\n\tHandle ID:\t0x18\r\n\r\nProcess Information:\r\n\tProcess ID:\t0x145c\r\n\tProcess Name:\tC:\Windows\System32\poqexec.exe\r\n\r\nAuditing Settings:\r\n\tOriginal Security Descriptor:\t\r\n\tNew Security Descriptor:\t\tS:ARAI(AU;SAFA;DCLCRPCRSDWDWO;;;WD)", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Auditing': was expecting ('true', 'false' or 'null')
at [Source: [B@1bb3f946; line: 1, column: 10]>, :level=>:warn}

kartikelastic · October 25, 2015, 12:15pm

at [Source: [B@56d9c1f9; line: 1, column: 10]>, :level=>:warn}
{:timestamp=>"2015-10-24T13:35:39.757000-0400", :message=>"Trouble parsing json", :source=>"message", :raw=>"Auditing settings on object were changed.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK-HP-I3$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nObject:\r\n\tObject Server:\tSecurity\r\n\tObject Type:\tFile\r\n\tObject Name:\tC:\Windows\SysWOW64\en-US\msaudite.dll.mui\r\n\tHandle ID:\t0x18\r\n\r\nProcess Information:\r\n\tProcess ID:\t0x145c\r\n\tProcess Name:\tC:\Windows\System32\poqexec.exe\r\n\r\nAuditing Settings:\r{:timestamp=>"2015-10-24T13:35:39.758000-0400", :message=>"Trouble parsing json", :source=>"message", :raw=>"Auditing settings on object were changed.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK-HP-I3$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\tLogon ID:\t\t0x3e7\r\n\r\nObject:\r\n\tObject Server:\tSecurity\r\n\tObject Type:\tFile\r\n\tObject Name:\tC:\Windows\SysWOW64\en-US\auditpol.exe.mui\r\n\tHandle ID:\t0x18\r\n\r\nProcess Information:\r\n\tProcess ID:\t0x145c\r\n\tProcess Name:\tC:\Windows\System32\poqexec.exe\r\n\r\nAuditing Settings:\r\n\tOriginal Security Descriptor:\t\r\n\tNew Security Descriptor:\t\tS:ARAI(AU;SAFA;DCLCRPCRSDWDWO;;;WD)", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Auditing': was expecting ('true', 'false' or 'null')
at [Source: [B@b2a4f66; line: 1, column: 10]>, :level=>:warn}

magnusbaeck · October 25, 2015, 7:41pm

It looks like you're attempt to parse non-JSON data with the json codec or filter.

kartikelastic · October 29, 2015, 2:25pm

Okay, I apologize for the delay in replying to you, and thank you very much for replying to my posts:
This procedure works:
null logstash.log > /var/log/logstash.log
curator delete indices --all-indices
curl -XPUT 'http://localhost:9200/twitter/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 2
}
}
}'

Then reboot, I then see this in the web interface:

October 29th 2015, 10:19:35.336

message:
Cryptographic operation. Subject: Security ID: S-1-5-18 Account Name: KARTIK1$ Account Domain: WORKGROUP Logon ID: 0x3E7 Cryptographic Parameters: Provider Name: Microsoft Software Key Storage Provider Algorithm Name: RSA Key Name: 51a92691-66f1-280f-d0db-59fad4f73491 Key Type: User key. Cryptographic Operation: Operation: Open Key. Return Code: 0x80090016
@version:
1
@timestamp:
October 29th 2015, 10:19:35.336
host:
%{host2}
type:
WindowsEventLog
tags:
_grokparsefailure
FileName:
source_host:
eventlog_severity:
AccountName:
eventlog_channel:
EventTime:
2015-10-29 09:19:32
Hostname:
Kartik1
Keywords:
-9218868437227405000
EventType:
AUDIT_FAILURE
SeverityValue:
4
Severity:
ERROR
EventID:
5061
SourceName:
Microsoft-Windows-Security-Auditing
ProviderGuid:
{54849625-5478-4994-A5BA-3E3B0328C30D}
Version:
0
Task:
12290
OpcodeValue:
0
RecordNumber:
13096
ActivityID:
{C7F2A81E-05CC-0006-24A8-F2C7CC05D101}
ProcessID:
772
ThreadID:
49896
Channel:
Security
Category:
System Integrity
Opcode:
Info
SubjectUserSid:
S-1-5-18
SubjectUserName:
KARTIK1$
SubjectDomainName:
WORKGROUP
SubjectLogonId:
0x3e7
ProviderName:
Microsoft Software Key Storage Provider
AlgorithmName:
RSA
KeyName:
51a92691-66f1-280f-d0db-59fad4f73491
KeyType:
%%2500
Operation:
%%2480
ReturnCode:
0x80090016
EventReceivedTime:
1446128374
SourceModuleName:
eventlog
SourceModuleType:
im_msvistalog
_source:
{"message":"Cryptographic operation.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-18\r\n\tAccount Name:\t\tKARTIK1$\r\n\tAccount Domain:\t\tWORKGROUP\r\n\