Have couple questions about Logstash configuration

Elastic Stack Logstash
21 
sincedb_path => "/var/log/logstash"

Hmm. sincedb_path should point to a file, not a directory. Does /var/log/logstash exist on your system? If it does, is it a directory or a file? Logstash should fail upon startup if it exists as a directory.

i mean how can i create that logstash will send data to Kibana from 2 different devices with 2 different index pattern ?

See elasticsearch - Make logstash add different inputs to different indices - Stack Overflow.

22

  1. i have this path as a folder but logstash running and working without any errors , i will remove this line , but how can i delete all fields with time and that it wont create them again ?

  2. after reading this article : http://stackoverflow.com/a/27147688/414355.

i have changed my config

input {
file {
path => ["/var/log/network.log"]

start_position => "beginning"
type => "firewall"
tags => [ "netsyslog" ]
}

file {
path => ["/var/log/Windows.log"]

start_position => "beginning"
type => "Windows"
tags => [ "netsyslog" ]
}

}

filter {

         kv {
        field_split => ","
}

}

output {
if [type] == "firewall" {

elasticsearch {
protocol => "node"
host => "localhost"

index => "firewall-%{+YYYY.MM.dd}"
}
}

else {
elasticsearch {
protocol => "node"
host => "localhost"

  index => "Windows-%{+YYYY.MM.dd}"
}

}
}
and still doesn't work

23
  1. Sorry, I don't understand this question.
  2. The configuration looks sane but Kibana complains about there not being any indexes matching Windows-*. Are there such indexes?
24

  1. how can i delete all fields with those messages ? and how can i make that it wont create them again ?

  2. well i entered to kibana path and i can see that indices created , but with other name than Windows/Firewall

and why its separated ? traffic,event,app-ctrl they are all Firewall messages , how can i make them to be in 1 folder under 1 index pattern

25

how can i delete all fields with those messages ?

Oh, right. Just delete all messages with such fields. You can use Elasticsearch's delete by query API. But since you're obviously still in the testing phase, why not just start over again or ignore these fields

and how can i make that it wont create them again ?

If you've fixed your configuration it won't happen again. This is a good example why it's a good idea to inspect messages with e.g. the stdout output before you actually start sending data to Elasticsearch.

well i entered to kibana path

Kibana doesn't have any local storage so I don't know what you mean. Are you actually talking about Elasticsearch? Don't inspect Elasticsearch's state by looking at the file system. Use a cluster dashboard plugin like kopf or ElasticHQ instead.

26
  1. the problem is that at the begging of my log i have this : 2015-09-15T14:00:18+03:00

example :

2015-09-15T14:00:18+03:00 10.1.1.200 date=2015-09-15,time=13:04:41,devname=FG200B,device_id=FG200B3911602199,log_id=0021000002,type=traffic,subtype=allowed,pri=notice,vd=root,src=50.50.50.1,src_port=65143,src_int="ssl.root",dst=10.1.1.240,dst_port=53,dst_int="port13",SN=54725607,status=accept,policyid=23,dir_disp=org,tran_disp=snat,proto=17,duration=180,sent=72,rcvd=265,sent_pkt=1,rcvd_pkt=1

how i can make that it will ignore this message ?

i tried

filter {

         kv {
        field_split => ","
  remove_field => [ "@timestamp" ]

}

but then nothing works..

  1. Kibana doesn't have any local storage so I don't know what you mean. Are you actually talking about Elasticsearch? Don't inspect Elasticsearch's state by looking at the file system. Use a cluster dashboard plugin like kopf or ElasticHQ instead.

i mean the path of elasticsearch

/var/lib/logstash/data/elasticsearch/nodes/1/

so why when i configured type for Firewall and Windows i have those folders ? when all of them are from fortigate . why its not take traffic,event,app-ctrl under 1 folder with the name Firewall-*

27

You need to isolate the key/value pairs in a field, then feed that field to the kv filter.

grok {
  match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IP:ip} %{GREEDYDATA:kv}"]
}
kv {
  source => "kv"
  ...
}

(I haven't tested if TIMESTAMP_ISO8601 actually matches your timestamp. This config snippet is just an example.)

so why when i configured type for Firewall and Windows i have those folders ? when all of them are from fortigate . why its not take traffic,event,app-ctrl under 1 folder with the name Firewall-*

There are no folders in Elasticsearch. You're better off ignoring the on-disk storage structure.

The configuration you showed earlier posts messages to either firewall-* or Windows-*. Under no circumstances will it post to an index that begins with any of the other prefixes you mentioned. Make sure you don't have another configuration file in /etc/logstash/conf.d with an output section that posts to those indexes.