OK, I'm relatively new to Logstash; we're looking at it as a possible replacement for what we're using now. I've been tasked with setting it up in such a way that it redirects the data along to our current system and also sends a copy to elasticsearch so we can "play with it" without losing the data in our current system before we decide to fully commit. I thought it'd be easy enough to list an output first before the filter, but that doesn't work since Logstash groups the inputs/filters/outputs together. I tried separating into two pipelines, but it seems the filter in the one pipeline still affects the other. Using the "clone" tag won't work unless there's a way to remove the cloned type for one output; ideally the packets sent to our current solution will appear to be the same as they used to be, with the exception of coming from a different host of course. Any solutions that I'm missing, or should I turn to some third party product to duplicate the traffic before it gets to Logstash?
Or to perhaps look at this differently, why did the pipelines crossover? Having one make the copy, use a file for an output, then having the other use that file for an input seemed like it would work until the "just copy" pipeline was being filtered too....
I don't understand why you need to duplicate anything. A single Logstash pipeline with two outputs should do. Unless conditionals are used all outputs receive all events.
I tried separating into two pipelines, but it seems the filter in the one pipeline still affects the other.
Then you probably weren't using two pipelines but just split your single-pipeline configuration in two files.
Using the "clone" tag won't work unless there's a way to remove the cloned type for one output;
You can use conditionals to route different events to different outputs.
The need for duplication is because there's one syslog stream coming out of DeviceX, and that stream needs to go to both elasticsearch and OldSystemY. Before going to ES, Logstash needs to transform the data into chunks that are more useful than a flat syslog entry.... but the old system would be confused if it were to receive the already transformed string, since it's expecting to transform it itself (and is hosted in the cloud so, no, I can't get my hands on it.)
I thought I had my pipelines screwed up too, but monitoring in Kibana insisted I had two pipelines. I since put them back into one pipeline, but apparently am not getting the conditionals right either. I'm quite positive the error is something I've done -- again, newbie here -- but if someone had a tip or trick that might help, well, that's what I was fishing for.
Well, what does your configuration look like so far? And what does a sample event produced by Logstash look like?
My pipeline.yml:
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf" - pipeline.id: redirect
path.config: "/etc/logstash/redirectpipe/*.conf"
The only file under /etc/logstash/redirectpipe:
input {
udp {
port => 5555
}
}
output {
tcp {
host => ["10.253.13.56"]
port => 5555
}
file {
path => "/home/logstash/redirectthis"
create_if_deleted => true
}
}
My other pipeline's config file:
input {
file {
path => "/home/logstash/redirectthis"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:sy slog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA :syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
output {
elasticsearch { hosts => ["10.226.1.94:9200"] }
stdout { codec => rubydebug }
}
}
The situation: I am generating traffic from another server using:
ncat -w1 -u 10.1.1.98 5555 <<< "This is testing my redirection"
Logstash is kicking back to the file and to the syslog server I'm catching it at 10.1.1.98:
{"@version":"1","@timestamp":"2018-05-07T13:31:56.641Z","message":"This is testing my redirection\n","host":"10.1.1.97"}
However, it was the redirect pipeline that caught it at UDP 5555, so it should be hitting those outputs first and without filtering... then the filtering pipeline should pick it up from the file, grok it, and send it to elasticstash. As it is, it's not even given me the standard output.
Thoughts? Other config areas you'd like to see? Thanks in advance!
So the tcp output is sending the event in the wrong format, JSON, instead of just echoing the udp plugin input? Make sure the tcp output uses the plain codec. You won't like the default format so you need to adjust that too. This should be an improvement:
tcp {
...
codec => plain {
format => "%{message}"
}
}Poking through my logs, I'm now seeing an error "[2018-05-07T09:45:06,751][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"undefined method `system?' for nil:NilClass", :error=>"NoMethodError"}" coming up.... not sure if it's related...
I will try that momentarily. The weird thing is when I took the filter out of the other pipeline friday, then it echoed out the UDP just fine. Weird. I'll keep ya posted though.
Progress! It's now passing along as it received, which is past the first hurdle. I'm still not seeing the entry in elasticstash or standard output. Is there a better method to hand from one pipeline to another than writing to a file and having the other pipeline read it?
OK, belay my last request about pipelines. The utility of what you suggested struck me, and I put everything back into one pipeline, using one output based on your suggestion to spit out just what it gets, another for stdout for testing and another for elasticsearch; it worked perfectly. THANK YOU!!!!
I would use tcp in preference to file. Others may prefer other options. I have this input in one pipeline fed by this output in the other.
output { tcp { host => "127.0.1.34" port => 7777 } }
input { tcp { host => "127.0.1.34" port => 7777 } }
I think I have solved this for now in this case by having one output that puts out the raw message, but this may come in handy down the road so I'll make a point to remember. Thank you!
Are you using a file output as shown earlier in the thread? Do you like the result you get when you restart logstash?
Once it hit me that I could send an unfiltered output using Magnus' trick, I went with that. I currently have two different inputs (there will be quite a few more), and I tag them coming in, but my current output config (with a little sanitizing for IPs and tag names) is as follows:
output {
if "[@metadata][xxxx0002]" in [tags] {
udp {
host => ["10.1.1.71"]
port => 1037
codec => plain {
format => "%{message}"
}
}
} else if "[@metadata][xxxx0001]" in [tags] {
udp {
host => ["10.1.1.71"]
port => 1038
codec => plain {
format => "%{message}"
}
}
}
elasticsearch { hosts => ["10.1.1.94:9200"] }
stdout { codec => rubydebug }
}
I'll create a new input file for each input with a unique tag and just extend this out for outputs. Once I get to other types of devices I may need to revisit the pipeline question at that point, but this will keep me going for a while.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.