Shipping log files

dhaval1 · November 21, 2015, 5:04pm

This is my first post. If it is not the right place, kindly redirect me to the correct place.

I want to first ship the log files from the remote server to the central sever "as it is" with out any formatting using logstash. Later on i will think of for Elastic search and generate dash board using Kibana .

To achieve that i did the following changes. Below config is coping all the logs under directory /var/log/*.log to /home/logstash/logs/test.log.

logstash-forwarder.conf as follows.

"servers": [ "192.168.0.100:5043" ],
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},

The list of files configurations

"files": [
{
"paths": [
"/var/log/*.log"
],
"fields": { "type": "syslog" }
}

and logstash.conf

input {
  lumberjack {
    port => 5043
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {}
output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/test.log"
}
}

I want all the log files under directory /var/log/*.log should come under directory /home/logstash/logs/, instead of test.log but it is not working. For that i change the output section as below. Kindly correct me, how do i achieve this.

output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/*.log"
}
}

vtst2412 · November 21, 2015, 7:50pm

Add a grok filter to capture the filename in a field, something like:

filter {
grok {
match => ["path","/var/log/%{DATA:filename}.log"]
}
}

Then use that field as the dynamic string for the output:

output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/%{filename}.log"
}
}

V.

dhaval1 · November 21, 2015, 8:48pm

After change, I got the following output under /home/logstash/logs/

%{filename}.log

at this stage my config is like below.

logstash-forwarder.conf as follows.

"servers": [ "192.168.0.100:5043" ],
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},

The list of files configurations

"files": [
{
"paths": [
"/var/log/*.log"
],
"fields": { "type": "syslog" }
}

and logstash.conf is like

input {
lumberjack {
port => 5043
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

filter {
grok {
match => ["path","/var/log/%{DATA:filename}.log"]
}
}

output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/%{filename}.log"
}
}

following is the output of logstash-forwader.log file

2015/11/22 02:11:28.082132 harvest: "/var/log/dnf.log" position:3177754 (offset snapshot:3177754)
2015/11/22 02:11:28.082190 Registrar will re-save state for /var/log/dnf.rpm.log
2015/11/22 02:11:28.082203 All prospectors initialised with 3 states to persist
2015/11/22 02:11:28.082236 harvest: "/var/log/dnf.rpm.log" position:217854 (offset snapshot:217854)
2015/11/22 02:11:28.082286 harvest: "/var/log/test.log" (offset snapshot:0)
2015/11/22 02:11:28.082624 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/11/22 02:11:28.082895 Connecting to [192.168.0.100]:5043 (192.168.0.100)
2015/11/22 02:11:28.215599 Connected to 192.168.0.100

vtst2412 · November 21, 2015, 10:33pm

Syntax error. I'm surprised it didn't throw an error when you ran.

  filter {
          grok {
              match => ["path" => "/var/log/%{DATA:filename}.log"]
      }

https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

Add this to your logstash output config and run it again.

 stdout { codec => rubydebug}

I want to see the fields of this document after the grok filter.

dhaval1 · November 22, 2015, 12:10pm

Following is the output of file and its content

tail -f %{filename}.log

[697874.511] (II) intel(0): EDID vendor "LGD", prod id 952
[697874.511] (II) intel(0): Printing DDC gathered Modelines:
[697874.511] (II) intel(0): Modeline "1366x768"x0.0 69.30 1366 1398 1430 1470 768 771 776 786 -hsync -vsync (47.1 kHz eP)
[698888.139] (II) intel(0): EDID vendor "LGD", prod id 952
[698888.182] (II) intel(0): Printing DDC gathered Modelines:
[698888.182] (II) intel(0): Modeline "1366x768"x0.0 69.30 1366 1398 1430 1470 768 771 776 786 -hsync -vsync (47.1 kHz eP)

vtst2412 · November 22, 2015, 8:04pm

I meant the stdout output, not the file output.

magnusbaeck · November 22, 2015, 8:54pm

 match => ["path" => "/var/log/%{DATA:filename}.log"]

No... that's not a documented syntax. It might work but

match => ["path", "/var/log/%{DATA:filename}.log"]

or

match => { "path" => "/var/log/%{DATA:filename}.log" }

are the two accepted forms.

dhaval1 · November 23, 2015, 10:49am

I changed both the option in filter but still i am getting out file %{filename}.log instead of all files which specified in logstash-forwarder.conf "/var/log/*.log"

My output section as below.

output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/%{filename}.log"
}
}

dhaval1 · November 23, 2015, 10:51am

ok.

I changed both the option in filter but still i am getting out file
%{filename}.log instead of all files which specified in
logstash-forwarder.conf "/var/log/*.log" I need all log files specified
under directory /var/log/ to ouput directory. /home/logstash/logs/

My output section as below.

output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/%{filename}.log"
}
}

magnusbaeck · November 23, 2015, 1:40pm

The name of the field containing the filename is file in the logstash-forwarder case (see below). It's Logstash that uses path. Hence, the grok filter to parse the file field instead.

github.com

elastic/logstash-forwarder/blob/v0.4.0/publisher1.go#L233




func writeDataFrame(event *FileEvent, sequence uint32, output io.Writer) {
	//emit("event: %s\n", *event.Text)
	// header, "1D"
	output.Write([]byte("1D"))
	// sequence number
	binary.Write(output, binary.BigEndian, uint32(sequence))
	// 'pair' count
	binary.Write(output, binary.BigEndian, uint32(len(*event.Fields)+4))


	writeKV("file", *event.Source, output)
	writeKV("host", hostname, output)
	writeKV("offset", strconv.FormatInt(event.Offset, 10), output)
	writeKV("line", *event.Text, output)
	for k, v := range *event.Fields {
		writeKV(k, v, output)
	}
}


func writeKV(key string, value string, output io.Writer) {
	//emit("kv: %d/%s %d/%s\n", len(key), key, len(value), value)

dhaval1 · November 23, 2015, 2:18pm

I am new to logstash and not able to understand the code sorry for that.

Could you please help me where is the change required to get the desired out put.

Whether it is possible or not.

magnusbaeck · November 23, 2015, 2:27pm

In the grok filter that extracts the filename field from the path field, change "path" to "file".

dhaval1 · November 23, 2015, 6:32pm

i tried that, not working.

magnusbaeck · November 24, 2015, 6:41am

We can try to help if you post additional details, including your configuration, the messages you do get, and what you expected to get. Just posting "it didn't work" is not the best way to get quick help.

dhaval1 · November 24, 2015, 3:34pm

apology for that.

Here is my initial configuration to achieve all the logs under directory /var/log/.log to ship /home/logstash/logs/.log. directory as it is without any formatting.

logstash-forwarder.conf

"servers": [ "192.168.0.100:5043" ],
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},

The list of files configurations

"files": [
{
"paths": [
"/var/log/*.log"
],
"fields": { "type": "syslog" }
}

and logstash.conf

input {
lumberjack {
port => 5043
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {}
output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/test.log"
}
}

Post advise, removed filename field from the path field, change "path" to "file". Still i am getting output %{filename}.log under /home/logstash/logs/ instead of /home/logstash/logs/.log e.g on client server file /var/log/test.log should come as it is /home/logstash/logs/test.log on central server.

input {
lumberjack {
port => 5043
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

filter {
grok {
match => { "file" => "/var/log/%{DATA:}.log" } == it was match => { "path" => "/var/log/%{DATA:filename}.log" }
}
}

output {
file {
message_format => " %{[message]}"
path => "/home/logstash/logs/%{filename}.log"
}
}

magnusbaeck · November 24, 2015, 9:11pm

Replace the file output with a stdout { codec => rubydebug } output and report what you get.

dhaval1 · November 25, 2015, 6:32pm

after change my output config is like this
output {
stdout { codec => rubydebug}
}

and i got the following logs under /var/log/logstash/logstash.stdout.

{
"message" => "Jul 14 09:45:19 Installed: putty-0.64-1.fc20.x86_64",
"@version" => "1",
"@timestamp" => "2015-11-25T18:27:06.468Z",
"type" => "syslog",
"file" => "/var/log/bak.log",
"host" => "localhost.localdomain",
"offset" => "23729",
"tags" => [
[0] "_grokparsefailure"
]
}
{
"message" => "Oct 28 23:18:26 Updated: clamav-data-empty-0.98.7-1.fc20.noarch",
"@version" => "1",
"@timestamp" => "2015-11-25T18:27:06.484Z",
"type" => "syslog",
"file" => "/var/log/bak.log",
"host" => "localhost.localdomain",
"offset" => "23781",
"tags" => [
[0] "_grokparsefailure"
]
}

magnusbaeck · November 25, 2015, 6:52pm

Change

match => { "file" => "/var/log/%{DATA:}.log" }

to

match => { "file" => "/var/log/%{DATA:filename}\.log" }

I don't believe we suggested that you change %{DATA:filename} to %{DATA:}.

dhaval1 · November 26, 2015, 6:07pm

That's the point.
It works !!
Thanks for your help.