Mantain source IP TCP Output

SecRobe · April 16, 2022, 3:30pm

Hello, I'm absolutely newbie in Logstash.
I'm using Logstash 6.2.3 with a simple input filter (TCP and UDP) and Syslog output, the problem with syslog output (as we know) it's it adds the "logstash" header, I'm trying to use the TCP output, but when I forward this logs to an external server, it's sent with the IP from logstash server, so if I use logstash to forward from several servers, all of them goes to external syslog server with the logstash server IP addres (not the source IP from the server that it's sending logs to logstash).
Is there anything I could do to solve this?

Event (example) received via "syslog output":

<13>Apr 16 15:08:30 192.168.0.58 LOGSTASH[-]: <38>Apr 16 17:08:30 howard systemd-logind: New session 75 of user robe.

as you can see, the timestamp and priority changes from original event and the "logstash" it's added.

Event (example) received via TCP Output:
<38>Apr 16 16:57:43 howard systemd-logind: Removed session 61.

as you can see, I receive the event in a correct way, but the source of the sender it's the logstash server (not the IP address from "howard" as I would like receive) this give me problems when I create the logsources based on these events.

Any help or anyone with a problem like this?

Thanks very much and sorry for my english

stephenb · April 16, 2022, 6:32pm

Hi @SecRobe Welcome to the community.

Please share your logstash configuration... That will help us help you.

SecRobe · April 16, 2022, 6:44pm

Hello, this is with syslog output (the if conditional it's for testing too, it's a lab actually),

[root@pajaroto conf.d]# cat logstash-syslog.conf
### Input Section
input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }

}


### Filter Section
filter {
  if [type] == "syslog" {
        grok {
         match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
         add_field => [ "received_at", "%{@timestamp}" ]
         add_field => [ "received_from", "%{host}" ]
         }
        syslog_pri { }
        date {
        match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
  }
}


### Output Section
output {
  stdout { codec => rubydebug }
  if "Hola" in [message] {
  elasticsearch { hosts => ["localhost:9200"] }
  }
  else {
  elasticsearch { hosts => ["localhost:9200"] }
  syslog {
   host => "192.168.0.30"
   port => 514
   protocol => "tcp"
  }
}
}

all it's ok, but sends the logstash header and "changes" the piority of event (example:
<13>Apr 16 18:32:46 192.168.0.58 LOGSTASH[-]: <86>Apr 16 20:32:46 howard sshd[3072]: pam_unix(sshd:session): session closed for user robe

this event has a prioriy of 86 (info/auth) but it's change to 13 (user/notice) and timestamp, host and LOGSTASH it's added too

and this is the TCP output configuration:

### Input Section
input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }

}


### Filter Section
filter {
  if [type] == "syslog" {
        grok {
         match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
         add_field => [ "received_at", "%{@timestamp}" ]
         add_field => [ "received_from", "%{host}" ]
         }
        syslog_pri { }
        date {
        match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
  }
}



### Output Section
output {
  stdout { codec => rubydebug }
  if "Hola" in [message] {
  elasticsearch { hosts => ["localhost:9200"] }
  }
  else {
  elasticsearch { hosts => ["localhost:9200"] }
  tcp{
   host => "192.168.0.30"
   port => 514

   codec => line {
   format => " %{message}"
   }
	}

}

the event it's ok, but the source IP in this case it's not 192.168.0.58 (the server sending logs to LOGSTASH, it's 192.168.0.57, the LOGSTASH server it selfs)

<77>Apr 14 10:06:01 howard anacron[10849]: Normal exit (1 job run)

perhaps the main poblem when I use TCP it's in the server receiving from LOGSTASH, the question it's in syslog output is there anyway to remove "logstash header" or in TCP output, is there anyway to matain source IP from server generating event?

thanks for your help

stephenb · April 17, 2022, 2:14am

HI @SecRobe

So if I use this simple example

logstash.conf

input {
  tcp {
    port => 5514
    type => syslog
  }
}

filter {
 mutate {
   # Do your real logic to get the host
   add_field => { "host" => "192.168.55.43" } 
   add_field => { "my_syslog_pri" => "77" } 
   add_field => { "my_procid" => "myapp" } 
 }
}


### Output Section
output {
  stdout { codec => rubydebug }

  tcp {
    host => "127.0.0.1"
    port => 5614
    codec => line {
      format => "%{message}"
    }
	}

  syslog {
   host => "127.0.0.1"
   port => 5615
   protocol => "tcp"
   priority => "%{my_syslog_pri}"
   procid => "%{my_procid}"
  }
}

And I set up a tcp listener on both the simple tcp and syslog
hyperion:~ sbrown$ nc -l 5614
hyperion:~ sbrown$ nc -l 5615

and I start logstash....

and then echo in one of your lines...
echo '<77>Apr 14 10:06:01 howard anacron[10849]: Normal exit (1 job run)' | nc 127.0.0.1 5514

For the TCP Output, The message passes through logstash without any added information... I do not see any added "logststash header" with the tcp output.

hyperion:~ sbrown$  nc -l 5614
 <77>Apr 14 10:06:01 howard anacron[10849]: Normal exit (1 job run)

No changes ... just acts as a passthrough... No "Logastash Header"

With the syslog I get this seems to work except the priority I can not get it set to set ... with the "Logstash header"

hyperion:~ sbrown$ nc -l 5615
<13>Apr 17 02:13:12 192.168.55.43 LOGSTASH[myapp]: 2022-04-17T02:13:12.569418Z 192.168.55.43 <77>Apr 14 10:06:01 howard anacron[10849]: Normal exit (1 job run)

Perhaps Mr @Badger might have some insight...

Badger · April 17, 2022, 2:49am

Hi, @stephenb, I did not understand what was being asked when I first read this, but now...

@SecRobe, if I understand correctly, you are receiving syslog messages over a tcp input, such as

<38>Apr 16 17:08:30 howard systemd-logind: New session 75 of user robe.

and sending them to a syslog output. That results in fields being prepended to that message

<13>Apr 16 15:08:30 192.168.0.58 LOGSTASH[-]: <38>Apr 16 17:08:30 howard systemd-logind: New session 75 of user robe.

and you are calling the <13>Apr 16 15:08:30 192.168.0.58 LOGSTASH[-]: the "logstash header". If you are going to use a syslog output there is no way to avoid having it prepend fields to make it either RFC 3164 or RFC 5424 compliant.

Since your messages are already valid syslog messages, it may make more sense to use a tcp output. However, there is no way to forge another server's IP address in the packets outbound from the logstash server, which I think is what you are asking for.

Instead of using logstash to forward the message, can you configure syslog on "howard" to send two copies of each message, one to logstash and one to 192.168.0.30? Most syslog daemons can do this.

SecRobe · April 17, 2022, 8:59am

Hello again, thanks you two for the help. The reasson to use Logstash it's drop non-valid and unwanted events (for example comming from ESXi (verbose) and other data sources).
With syslog output plugin it's working fine, but the header (added with logstash) add size to every event, so if I have to retain events increase the storage requeriments. This is the unique (let's say) "problem".
I will continue working with syslog output.
Again, tahnk you very much for your help, I appreciate your recomendations

stephenb · April 17, 2022, 3:26pm

@SecRobe

I figured out how to pass through your priority the key is to set `use_labels => "false"

use_labels

Value type is boolean

Default value is true

use label parsing for severity and facility levels use priority field if set to false

...
filter {
 mutate {
   # Do your real logic to get the host, app priority etc... 
   add_field => { "host" => "192.168.55.43" } 
   add_field => { "syslog_pri" => "77" } 
   add_field => { "my_procid" => "myapp" } 
 }
}

### Output Section
output {
  stdout { codec => rubydebug }

  tcp {
    host => "127.0.0.1"
    port => 5614
    codec => line {
      format => "%{message}"
    }
	}

  syslog {
   host => "127.0.0.1"
   port => 5615
   use_labels => "false"  <!--- set to false
   protocol => "tcp"
   procid => "%{my_procid}"
  }
}

sample out put

hyperion:tcp-syslog-logstash sbrown$ nc -l 5615
<77>Apr 17 15:23:10 192.168.55.43 LOGSTASH[myapp]: 2022-04-17T15:23:10.437464Z 192.168.55.43 <77>Apr 14 10:06:01 howard anacron[10849]: Normal exit (1 job run)

system · May 15, 2022, 3:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
TCP syslog output not working properly in logstash Logstash	1	1491	December 21, 2017
Logstash syslog issue Logstash	9	1098	April 23, 2018
Logstash - Syslog input and output Logstash	1	400	February 27, 2018
Logstash syslog output: Weird message processing Logstash	3	428	December 12, 2019
Syslog filter for IP Logstash	4	841	March 7, 2021

Mantain source IP TCP Output

use_labels

Related topics

`use_labels`