Sending logging data to ES using tcp input plugin, no use of file

Elastic Stack Logstash
1

Hi All

We get very good log file in LS, but i want to mute this file logging and send the logging data directly to ES
I am using following configuration, but it is not working. It does not show anything in console and does not create index

input {
tcp {
host => localhost
port => 3456
codec => json
ssl_enable => false
}
}
filter {
date {
match => [ "timeMillis", "UNIX_MS" ]
}
}

I have updated the log4j.properties file like this in LS config folder

log4j.rootLogger=DEBUG,tcp

log4j.appender.tcp=org.apache.log4j.net.SocketAppender
log4j.appender.tcp.Port=3456
log4j.appender.tcp.RemoteHost=localhost
log4j.appender.tcp.ReconnectionDelay=10000
log4j.appender.tcp.Application=test
log4j.appender.tcp.append=true

while i am running other pipeline in parallel in port 9006

I saw, some guys asked similar que, but no one replied those. Did anyone tried this?

2

The following works in 7.2.0

rootLogger.appenderRef.tcp.ref = JsonOverTcp

appender.json_tcp.name = JsonOverTcp
appender.json_tcp.layout.type = JSONLayout
appender.json_tcp.layout.compact = true
appender.json_tcp.layout.eventEol = true
appender.json_tcp.type = Socket
appender.json_tcp.protocol = TCP
appender.json_tcp.port = 3456
appender.json_tcp.host = localhost
appender.json_tcp.reconnectionDelayMillis = 10000

That will get you messages like

{
"@timestamp" => 2019-06-29T23:05:55.946Z,
     "level" => "WARN",
      "port" => 56570,
      "host" => "localhost",
    "thread" => "SIGINT handler",
"loggerName" => "logstash.runner",
"timeMillis" => 1561849555604,
  "logEvent" => {
    "message" => "SIGINT received. Shutting down."
}
[...]
}

That said, I do not like the setup very much. You miss all the messages at startup, because the tcp input is not listening, and log4j appears not to buffer if the output is unavailable. I could easily see you missing messages during shutdown, because the input gets closed. (For example, if logstash crashes, you may miss the reason why.) Finally if you are sending the log of an instance back through the same instance that created them there could be weird cases where processing a message results in a message, and it goes in to an infinite loop (or even worse an exponential increase in volume). If you are going to do this I strongly recommend that you have a dedicated logstash instance (not a pipeline, an instance) that just feeds the tcp input into elasticsearch (a date filter should be OK :slight_smile: ).

1 Like
3

Hi, thanks for the properties file, with the given property file, its working.
I see whats the limitation with this solution, that you described above

Will it be possible to extend the field value.
How can i expand this for
pipeline.id
node.name
etc.

{
"@timestamp" => 2019-06-29T23:05:55.946Z,
"level" => "WARN",
"port" => 56570,
"host" => "localhost",
"thread" => "SIGINT handler",
"loggerName" => "logstash.runner",
"timeMillis" => 1561849555604,
"logEvent" => {
"message" => "SIGINT received. Shutting down."
}

4

Hi @Badger

"port" => 56570,

and the port we get in LS 9600, both are not same
Can you please let me know this 56570 port is for what and from where it is coming

5

I believe 56570 is the port on the client side of the TCP connection. I cannot think of a use for it, but I guess it does no harm.

6

The reason i have asked because

i run multiple pipeline, by using --path.data from the same instance and it does create different port for each pipeline like 9600,9601....

But with this port# 56570, it is difficult to identify the response is for which pipeline

That is the reason , i was asking for how to change the tcp response or extend the response by pipeline.id, so that can identify the log message is for which pipeline

7

If you are using a multiple tcp inputs you can use the tags option to label the events.

8

i am using one pipeline whose input is tcp and output is ES

And multiple other pipelines to do logging of all the events to ES( from above pipeline). In ES i want to identify which logs events for which pipeline.

Tags may add other errors tags as well, so may not be useful. I am looking for ways to extended the response from tcp input

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.