Rsyslog is not forwarding logs to elasticsearch

Aksel · January 1, 2023, 3:05pm

I'm trying to configure rsyslog to send logs to logstash and then forward them to elasticsearch.

I have create a config file /etc/rsyslog.d/60-output.conf with the following content:

*.* @localhost:10514;json-template

and a template file /etc/rsyslog.d/01-json-template.conf with the following content:

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}

Then I restarted rsyslog service. And for logstash I created a config file /etc/logstash/conf.d/logstash.conf with the following content:

input {
  udp {
    port => 10514
    codec => "json"
    type => "rsyslog"
  }
}
filter { }
output {
  if [type] == "rsyslog" {
    elasticsearch {
      hosts => [ "localhost:9200" ]
    }
  }
}

Then I restarted logstash.

When I run sudo netstat -tulpn | grep 10514 I get this:

user@rsyslog-server:/var/log$ sudo netstat -tulpn | grep 10514
udp        0      0 0.0.0.0:10514           0.0.0.0:*                           5327/java

so Logstash is listening on port 10514.

To verify the elasticsearch input I run curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty' but this doesn't return any results:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

I'm using:

rsyslogd 8.2208.0 (aka 2022.08)
logstash 7.17.8
elastisearch 7.17.8

How can I solve this ?

stephenb · January 1, 2023, 5:19pm

Hi @Aksel

For debug...Try taking this out...

Also you could add stdout{} in the output section outside the if to see if you are actually getting any messages

Aksel · January 1, 2023, 6:01pm

Thnaks for your response,

I changed my configuration to:

Then I started logstash with this config:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --verbose

This is the output:


Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-01-01 18:50:48.417 [main] runner - DEPRECATION WARNING: The flag ["--verbose"] has been deprecated, please use "--log.level=info" instead.
[INFO ] 2023-01-01 18:50:48.423 [main] runner - Starting Logstash {"logstash.version"=>"7.17.8", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.17+8 on 11.0.17+8 +indy +jit [linux-x86_64]"}
[INFO ] 2023-01-01 18:50:48.426 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2023-01-01 18:50:48.689 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-01-01 18:50:49.682 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2023-01-01 18:50:50.739 [Converge PipelineAction::Create<main>] Reflections - Reflections took 50 ms to scan 1 urls, producing 119 keys and 419 values 
[WARN ] 2023-01-01 18:50:51.350 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.396 [Converge PipelineAction::Create<main>] udp - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.438 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.479 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-01-01 18:50:51.658 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2023-01-01 18:50:51.953 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2023-01-01 18:50:52.174 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2023-01-01 18:50:52.199 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.17.8) {:es_version=>7}
[WARN ] 2023-01-01 18:50:52.200 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2023-01-01 18:50:52.276 [[main]-pipeline-manager] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[INFO ] 2023-01-01 18:50:52.362 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x332001c0 run>"}
[INFO ] 2023-01-01 18:50:52.380 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2023-01-01 18:50:53.026 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.66}
[INFO ] 2023-01-01 18:50:53.093 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[WARN ] 2023-01-01 18:50:53.158 [[main]<udp] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:53.186 [[main]<udp] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-01-01 18:50:53.188 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2023-01-01 18:50:53.206 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:10514"}
[INFO ] 2023-01-01 18:50:53.244 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:10514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}

It seems that nothing is happening

stephenb · January 1, 2023, 6:08pm

Yup So it seems like your rsyslog is not actually sending anything because anything you receive you should show up...

You could just try to netcat a line or two to the UDP port and That should show up in your logstash put.

Aksel · January 1, 2023, 6:10pm

I'm sorry, but how can I do that ?

stephenb · January 1, 2023, 6:24pm

Google is your friend

But make sure you use 127.0.0.1 or your IP not localhost

$ nc -u 127.0.0.1 10514
Hello from UDP Land

Then type in / paste in a couple line... they should show up in the logstash logs.. and actually even in elasticsearch

On the logstash side

{
         "event" => {
        "original" => "Hello from UDP Land\n"
    },
          "host" => {
        "ip" => "127.0.0.1"
    },
    "@timestamp" => 2023-01-01T18:24:15.154811Z,
       "message" => "Hello from UDP Land\n",
      "@version" => "1",
          "type" => "rsyslog"
}

this is the logstash config that produced that with Version 8.5.3 your with 7.17.x the index name will be different.. other than that it should be same

input {
  udp {
    port => 10514
    type => "rsyslog"
  }
}
filter { }
output {
  if [type] == "rsyslog" {
    elasticsearch {
      hosts => [ "localhost:9200" ]
    }
  }
	stdout {}
}

And the logs showed up in Elasticsearch

GET logs-generic-default/_search

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 8,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-generic-default-2023.01.01-000001",
        "_id": "FRmTboUBhuzPI0SRwbCj",
        "_score": 1,
        "_source": {
          "event": {
            "original": """Hello from UDP Land
"""
          },
          "host": {
            "ip": "127.0.0.1"
          },
          "@timestamp": "2023-01-01T18:24:15.154811Z",
          "message": """Hello from UDP Land
""",
          "@version": "1",
          "type": "rsyslog",
          "data_stream": {
            "type": "logs",
            "dataset": "generic",
            "namespace": "default"
          }
        }
      }
....

stephenb · January 1, 2023, 6:37pm

Oh and @Aksel I forgot to welcome you to the community...so Welcome!

system · January 29, 2023, 6:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.