Stdout cause syslog file to be to big

Elastic Stack Logstash
1

sometime I have to use

stdout { codec => rubydebug { metadata => true } }

to do some troubleshooting on a pipeline.

I noticed that when I do the syslog file is taking up GB of storage.

is there a way to have it write to a separate file instead so that I can easy delete it when I am done with it?

2

Instead of using a stdout output, use a file output

file { codec => rubydebug { metadata => true } path => "/path/to/log.log" }

If the output keeps the file open then deleting it will not free up disk space.

3

Just to add...

path => "/path/to/log.log"

Since you have the output files in GB, maybe is useful to use files on daily base.
path => "/path/to/log_%{+YYYY-MM-dd}.log"

1 Like
4

That helped a lot. Thanks ! Is there a way to name based on the pipeline and host it is coming from? Or have it auto generate sub folders based on them?

5

You can reference any field on the event in the path option of the file output, although there are a couple of restrictions.

You could use a ruby filter to add the pipeline_id and hostname to the event. I would add them inside the [@metadata] field, which you can reference in the output using %{}, but they will not be sent to the destination with the rest of the fields on the event.

Something like

ruby {
    init => 'require "socket"'
    code => '
        # Or save in @class variable in init to avoid repeated call
        event.set("[@metadata][hostname]", Socket.gethostname)

        event.set("[@metadata][pipeline_id]", execution_context.pipeline.pipeline_id)"
    '
 }

...

file {
    path => "/TheFixedBit/%{[@metadata][hostname]/%{@metadata][pipeline_id]}-%{+YYYY-MM-dd}.log"
    ...
6

will this give me the host name of the logstash server or the sever the beats agent is send data from?

7

It would be the hostname of the logstash server. If you want the hostname of the beat then have the beat add host metadata to the event.

8

I ended up using this

file { codec => rubydebug { metadata => true } path => "/logstash/logs/file_output/%{[@metadata][beat]}/%{[host.hostname]}.log" }

It worked but it returned {[host.hostname]}.log as the file name and not the actual host name

9 
file { codec => rubydebug { metadata => true } path => "/logstash/logs/file_output/%{[@metadata][beat]}/%{[host][hostname]}.log" }

This one worked.