Add directoryname and filename as different field to elasticsearch

Elastic Stack Logstash
1

I am using file plugin to read mutiple files in directory . I would like to add file name and directory name as fields. how could I do that.

Following is a sample directory structure and content inside it

	D:/test/shahtest
		c12345/logs/linux_x64-fastdebug.log
		c12345/logs/windows_x64-fastdebug.log
		c12345/logs/windows_x64-product.log
		
		a567888/logs/linux_x64-fastdebug.log
		a567888/logs/windows_x64-fastdebug.log
		a567888/logs/windows_x64-product.log

Sample input

	input {
		file {
			path => ["D:/test/shahtest/*/logs/*.*-fastdebug.log"]		
			start_position => "beginning"
			type => "fastdebuglogs"
			add_tag => "fastdebuglogs"
			
		}
		
		file {
			path => ["D:/test/shahtest/*/logs/*.*-product.log"]		
			start_position => "beginning"
			type => "productlogs"
			add_tag => "productlogs"				
		}

What I am looking for is.

  1. When I read one of the fastdebug file , all the event written from that file should contain directoryname and filename as different field for eg: all the event from "a567888/logs/linux_x64-fastdebug.log" should contain field "dir_name" as "a567888" and "file_name" as "linux_x64-fastdebug.log"

  2. All the logs files are of each 120 Mb each so I need to grab one string "JOBID" from that specific log and add a field named JOBID with the respective value from the log. All the event happening from a specific log should have the same JOBID. I planned of using grok, but grok is event\line specific so I am not sure how could I assign this JOBID for the entire event\log from a single file.

Any help is appreciated

2

I was able to do this using a grok as below.

        grok {
	patterns_dir => "./patterns"
	match => ["path","D:/test/shahtest/%{DATA:sjptjobid}/logs/%{DATA:sjptlogtype}.log" ]
	add_tag => "pathsyslogs"
	}