Unable to fetch mapping

Elastic Stack Kibana
1

Hi,

I have set a Kibana server and I am trying to get it up and running
I must admit, I am very new to kibana and am in learning mode.
I have the following events sent via syslog using port 514 (I am using the root user for now)

I have written the following input

input {
        udp {
                port => 514
                type => syslog
            }

        }

filter {
        if [type] == "syslog" {

                                grok {
                                        match => {
                                                   message => '(%{NUMBER:pre_number}).*?(%{TIMESTAMP_ISO8601:time}).*?(%{IPV4:server_ip}).*?(%{HOSTNAME:server_name}).*?(%{NOTSPACE:par_srv_version}).*?(%{WORD:evt_type}).*?endpoint=(%{HOSTNAME:ep_host_name}).*?(%{IPV4:ep_ip}).*?type=(%{GREEDYDATA:Protection_module})\,.*?caller=(%{GREEDYDATA:caller})\,.*?callee=\'(%{GREEDYDATA:callee})\,.*?MD5=(%{GREEDYDATA:md5})\,.*?description=(%{GREEDYDATA:description})'
                                                 }
                                     }
                                grok {
                                        match => {
                                                  message => '(%{NUMBER:pre_number}).*?(%{TIMESTAMP_ISO8601:time}).*?(%{IPV4:server_ip}).*?(%{HOSTNAME:server_name}).*?(%{NOTSPACE:par_srv_version}).*?(%{WORD:evt_type}).*?endpoint=(%{HOSTNAME:ep_host_name}).*?(%{IPV4:ep_ip}).*?type=(%{GREEDYDATA:Protection_module})\,.*?caller=(%{GREEDYDATA:caller})\,.*?callee=\'(%{IPV4:Destination_ip}).*(%{NUMBER:Destination_port}).*MD5=(%{GREEDYDATA:md5})\,.*description=(%{GREEDYDATA:description})'
                                                 }
                                     }
                                grok {
                                        match => {
                                                  message => '(%{NUMBER:pre_number}).*?(%{TIMESTAMP_ISO8601:time}).*?(%{IPV4:server_ip}).*?(%{HOSTNAME:server_name}).*?(%{NOTSPACE:par_srv_version}).*?(%{WORD:evt_type}).*?(?<performed_action>\b\w+\b\s*\b\w+\b).*?(%{HOSTNAME:ep_host_name}).*?(%{IPV4:ep_ip})\)\s*.*?(%{GREEDYDATA:action})'
                                                 }
                                     }
                                grok {
                                        match => {
                                                  message => '(%{NUMBER:pre_number}).*?(%{TIMESTAMP_ISO8601:time}).*?(%{IPV4:server_ip}).*?(%{HOSTNAME:server_name}).*?(%{NOTSPACE:par_srv_version}).*?(%{WORD:evt_type}).*?Agent.*?(%{HOSTNAME:ep_host_name}).*?(%{IPV4:ep_ip})\)\s*.*?(%{GREEDYDATA:action})'
                                                 }
                                     }
                                grok {
                                        match => {
                                                  message => '(%{NUMBER:pre_number}).*?(%{TIMESTAMP_ISO8601:time}).*?(%{IPV4:server_ip}).*?(%{HOSTNAME:server_name}).*?(%{NOTSPACE:par_srv_version}).*?(%{WORD:evt_type}).*?ip=(%{IPV4:ep_ip})\s*.*?(%{GREEDYDATA:action})'
                                                 }
                                     }
                                grok {
                                        match => {
                                                  message => '(%{NUMBER:pre_number}).*?(%{TIMESTAMP_ISO8601:time}).*?(%{IPV4:server_ip}).*?(%{HOSTNAME:server_name}).*?(%{NOTSPACE:par_srv_version}).*?(%{WORD:user_for_installation})\@(%{HOSTNAME:ep_host_name}).*?(%{GREEDYDATA:action})'
                                                 }
                                     }
								grok {
                                        match => {
                                                  message => '(%{NUMBER:pre_number}).*?(%{TIMESTAMP_ISO8601:time}).*?(%{IPV4:server_ip}).*?(%{HOSTNAME:server_name}).*?(%{NOTSPACE:par_srv_version}).*?(%{WORD:user_for_installation})\@(%{HOSTNAME:ep_host_name}).*?(%{GREEDYDATA:action})'
                                                 }
                                     }		
                                }
        }

I have restarted the logstash service.

When I try to discover and create an index I get the "Unable to fetch mapping"

What am I missing?

Thanks
Yoram

2

Have you configured an elasticsearch output or do you only have inputs and filters?

3

The only output I have configured is the one that was in the installation guide

output {
        elasticsearch { host => localhost }
        stdout {codec => rubydebug }
       }
4

Okay. Disable the elasticsearch output for now. Do you get anything to stdout if you send anything to port 514?

5

By disabling the elasticsearch output you mean put everything in comment or delete the file?

6

Make sure elasticsearch { host => localhost } isn't read by Logstash. Doesn't matter if you comment out the line or delete it.

7

Hi,

I have commented the line. restarted the logstash & elasticsearch services.
I am still not getting any events into Kibana.

Run the following in my server http://:9200/_cat/indices?v
and got the following:

health status index   pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   .kibana   1   1          1            0      2.5kb          2.5kb

In Kibana the status is as seen in the attached pic

8

If you comment out the elasticsearch output there will obviously not be any events sent to Elasticsearch. That's the point. We remove one source of errors to ease debugging.

Now. Do you get anything to stdout if you send anything to port 514? If yes, do your filters work as expected?

9

So....
now I have the events coming in.
for some reason I couldn't receive events in IP4 and also I had to change the port to a higher port to resolve it.

THanks for pointing me in the right direction