Enable logstash to read on port number

Elastic Stack Logstash
1

Hi There,

I need some help in testing this. Below is the logstash config file[logstash-6.5.4/config/logstash-sample-4.conf]. I would like to feed the logstash with elasticsearch logs, which runs on port 9200.

input {
  tcp {
    type => "logs"
    codec => "json"
    port => 9200
  }
    
  file {
        path => "/var/log/system.log"
        type => "syslog"
   }
}

output {
   if [type]=="syslog" {
  elasticsearch {
             hosts => [ "localhost:9200" ]
             index => "syslog-debug-%{+YYYY.MM.dd}"
        }
   }   
   else if [type]=="logs" {
        elasticsearch {
             hosts => [ "localhost:9200" ]
             index => "elasticlogs-debug-%{+YYYY.MM.dd}"
       stdout {
             codec => rubydebug
     }
        }
   }
   else {
        elasticsearch {
             hosts => [ "localhost:9200" ]
             index => "logs-debug-%{+YYYY.MM.dd}"
        }
   }
}

Ran logstash with logstash -f <logstash_file>
But it is failing.

[2019-04-02T13:23:07,899][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 25, column 15 (byte 493) after output {\n   if [type]==\"syslog\" {\n  elasticsearch {\n             hosts => [ \"localhost:9200\" ]\n             index => \"syslog-debug-%{+YYYY.MM.dd}\"\n        }\n   }   \n   else if [type]==\"logs\" {\n        elasticsearch {\n             hosts => [ \"localhost:9200\" ]\n             index => \"elasticlogs-debug-%{+YYYY.MM.dd}\"\n       stdout ", :backtrace=>["/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in `initialize'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/pipeline_action/create.rb:42:in `block in execute'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/agent.rb:92:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/agent.rb:92:in `exclusive'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "/Users/mouliveera/Desktop/tools/logstash-6.5.4/logstash-core/lib/logstash/agent.rb:317:in `block in converge_state'"]}

I tried with apache port 80 as well. I see the similar behaviour. Can someone help me in fixing it.
Thanks

2

First, with the above config you would have both Logstash and Elasticsearch listening on port 9200 on localhost. That will not work. I would let Elasticsearch use its default port 9200 and set Logastash to use some other non standard port. E.g. I use ports 551X for Logstash inputs.

Second, Elsticsearch doesn't push logs that Logstash could just receive on a port. I use Filebeat to tail the Elasticsearch log files and ship Elasticsearch logs to Logstash.

Then, for the Logstash error message which says

Expected one of #, => at line 25

You have to close the elasticsearch output config before you define the stdout output.

3

Thanks for your response. How about apache2/https ports.

input {
  tcp {
    type => "logs"
    codec => "json"
    port => 80
  }
output {
  elasticsearch {
             hosts => [ "localhost:9200" ]
             index => "apache-debug-%{+YYYY.MM.dd}"
        }
4

I would not recommend using well known or common ports as ports for Logstash inputs.

Again, Logstash is not pulling any logs from e.g. Apache. Unless you setup some sort of log shipper the webserver logs are written to a file on the filesystem where the webserver process runs. Containers is a slightly different story but even there you would probably have to tail a log file on disk at some stage of the log export.

For a Logstash TCP input that excepts JSON to work the way you want, you have to have something that actually sends those messages as JSON to the desired port on the machine where Logstash runs.

You can test this with netcat. The below is a bash example for a UDP input.

#!/bin/bash

  BLAH=$1

   DATE=$(date)
   echo "{ \"log\": \"Testing JSON logs $BLAH - $DATE\"}" | nc -u -w2 logstash.example.com 12346
exit

If you use software from Elastic to do this then Filebeat would be the one to use.

5

Also i tried with one of our application. It runs in my local and forms logs in JSON format.

input {
  file {
    path => "/var/log/system.log"
    type => "syslog"
 }

 beats{
      port => 8081
      type => "localMeru"
 }
}
output {
if[type] == "syslog"{
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "system-logs-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
  } else if[type] == "localMeru"{
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "local-meru-logs-%{+YYYY.MM.dd}"
  }
  }

I see system logs creating indices, but not app localMeru which runs on port 8081

Do I need to trim the inputs?

6

Do you have any beats running anywhere sending data to Logstash?

A brief explanation of exactly what you want to do would be good.

7

No. No beats running on local instance. I am experimenting with ELK stack. I tried with files, they are creating indices.

I would like to understand, if we have a service and it is running on a port 1234. How can we use the port to create indices.

Thanks

8

You can't directly do that. As you describe it above, the port for the service is a listening port. It is not sending anything to Logstash.

The same way as you have Logstash reading a local log file in your config above, you can have Filebeat reading a log file on another systems and send it to Logstash.