Logstash Forwarder not supported on AIX - what now?!

Elastic Stack Logstash
1

I just found out that Logstash Forwarder is not supported on the AIX platform - currently that's our application platform from which we need to source our logs and send them to the ELK stack on a Linux box.

Our application writes SystemOut.log files to the local file system. Are there any other possible solution in lieu of the Logstash Forwarder that we could consider?!

2

We don't support any AIX sorry.
Can you ship it via rsyslog or similar?

3

Not familiar with rsyslog yet. Does rsyslog allow forwarding log data like LS Forwarder would? Will it handle our application SystemOut.log files - they are not in a SysLog format...

4

We have been working with rsyslog trying to get it to send data to our remote Logstash service.

Our architecture is this:
Application server running on AIX machine, the ELK stack running on RHEL server running remotely.

We can get some logs to the ELK server, but our application produces log files with dynamic names (including order number), so we need to setup a log file monitoring in rsyslog with wildcard in the file name. This introduces additional requirements for rsyslog: specific version of rsyslog that has to be compiled from the source code and the enablement of the inotify feature at the kernel level - we are not sure we can have that turned on.

Just in case we run into a roadblock here - is there another option out there for sending logs to ELK?!

Is Beats supported on AIX?!

1 Like
5

There's no go support on AIX, so there are no beats.

6

Hi James,

Did you have any progress on this? Did you manage to solve your issue with getting the AIX logs into Elastic? I have been tasked to do the same.

The issue on my part is the the log I need to tail has a very high volume so it will impact other infrastructure. So, I need to run filter before getting it on the the network.

Regards Sanjay Mstry

7

We tried the rsyslog and it would have worked, except our log names are dynamic: tracelog_order2324.txt, etc and it required firmware upgrade to get to the next level of the rsyslog, making it unfeasible, so we looked elsewhere.

We found rsynch utility which works for us just fine.

Good luck -
James

8

I have a websphere application server running in a AIX machine, and all the log file are configured to fall under (usual) path /Appserver/logs/SystemOut.log.... Now I have installed ELK in one of my windows machine.

Can I have some basic overview of how to integrate logstash with my Websphere running in AIX? or Is it even possible to accomplish that?

9

We are using a utility program called rsynch to ship data from AIX machine where our application runs over to a remote ELK server. It works ok so far.

We configured rsynch to watch a certain source directory and synch them out to the remote server. The remote server runs Logstash instances that consume the data and record it in the ElasticSearch cluster.

10

Hi James,

So glad to see your posts. I'm also working on shipping application log data to remote server for ELK analysis and search.

You said you were using rsync to sync up log data between AIX machine and the remote ELK server. I read the post you recommended, but still not familiar with it. I have a question about this tool. Does it support file rotation? Does it ship data like what filebeat does? Thanks for your help.

11

Hello Caoping,

Thanks for your interest! I work with James on the infra part of the ELK setup. =)

Rsync makes delta transfer of recently-changed log files from our source AIX server. We scheduled it to throw the delta every minute.

In terms of FIle Rotation? Well there's no function to weed out old log files yet, at least we did not use any rsync args to do some rotation.

In theory, rsync works best as a filesystem synchronization tool via ssh, so we maximize it to sync our log file directory with the remote directory (where Logstash can listen). Not as lightweight as FIlebeat I believe, but works for now. Still looking for an AIX-friendly solution =)

.

12

Hi Raymund,

Thanks for your quick response.

I'm quite new to AIX, and got stuck in the first step - installing rsync in AIX. I've installed the AIX toolbox, but don't know how to install rsync from it. By searching from google, it told me to use rpm to install that. However, I don't know what's the package name of rsync. Also I didn't find any package with text rsync.

I know it's a stupid problem, but I really need your help on the installation. Thanks so much.

13

Hi Raymund,

Please ignore my previous question about how to install rsync in AIX platform since I've done it. But I still have a question about how you're using rsync.

The requirement is I need to sync files on server A with server B. So at B, I need to run rsync as a daemon, which will listen to a particular port to see if any incoming rsync requests from a remote server, right? Then how about server A? How to let rsync at A to automatically ship log data to server B at a interval?

I'm still new to rsync, but I'll dig it deeper. Thanks for your help! Very appreciated.