Appending Random number in csv file name

Elastic Stack Logstash
1

Hi,
I want to append random number after the file name.

output {
	stdout { codec => rubydebug }
  csv {
    # elastic field name
    fields =>  ["@timestamp","requestid","ngnix.responsebytes","ngnix.cpu.usage"]
	
    # This is path where we store output.   
    path => "C:/Users/M1056317/ELK/csv/try6/csv-export-(random number).csv"
		
  }
 
}

Please guide me how can I do it.

2

You could use the event fields in the path. So, you could use the timestamp. Something like the below should work for you.

path => "C:/Users/M1056317/ELK/csv/try6/csv-export-%{+YYYY-MM-dd-HHmmSS}.csv"

Alternatively you could generate a random number using ruby filter in the logstash filter , assign it to a field (randomNum) and use the same field (randomNum) in the path.

path => "C:/Users/M1056317/ELK/csv/try6/csv-export-%{randomNum}.csv"

3

I tried the first solution as I dont want an extra field in my document, even for 5 minute scheduler it is giving 1 file per second. Can you make me understand how does the name effects opening and closing of file?

4

I tried second solution as well as I already have one unique field for all the logs, but again instead of five minutes , it is creating one csv for each unique requestId. Where am I going wrong?

5

You would have to write code in a ruby filter to add a field (under [@metadata]) that contains the time rounded to five minutes and then use a sprintf reference to it in the path option.

6

Hi Badger, I am very new to Ruby and tried what you said,

filter{
ruby {
		code => '
			t = Time.at(event.get("@timestamp").floor_to(15*60))	
		'
	
	}
}


output {
	stdout { codec => rubydebug }
  csv {
    # elastic field name
    fields =>  ["@timestamp","requestid","ngnix.responsebytes","ngnix.cpu.usage"]

    path => "C:/Users/M1056317/ELK/csv/try7/csv-export-%{+YYYY-MM-dd_hh}%{t}.csv"
		
  }
 
}

I am getting this exception:

[2021-02-10T19:20:05,154][ERROR][logstash.filters.ruby    ][main][7c96a8d6d717052e84a4d23fab6fa33c245ac64b1497bbcce0de457a118e444b] Ruby exception occurred: undefined method `floor_to' for 2021-02-10T13:48:40.249Z:LogStash::Timestamp
Did you mean?  Float

Can you please help?

7

I would do it using

    ruby {
        code => '
            t = (event.get("@timestamp").to_f/300).floor*300
            event.set("[@metadata][suffix]", Time.at(t).strftime("%Y-%m-%d-%H-%M"))
        '
    }

then use "C:/Users/M1056317/ELK/csv/try7/csv-export-%{[@metadata][suffix]}.csv" in the path option.

1 Like
8

Thanks Badger. Its working.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.