Need to send data from .log file to AWS opensearch

Elastic Stack Logstash
1

Hello all,

Could you please help me on this.

I am quite new to the Elasticsearch ocean.

My requirement is, I need to send data from .log file to AWS opensearch. Whether the below option will work for me.

Or i have to forcefully choose the follwing option download "Logstash OSS with OpenSearch Output Plugin" from the website (Opensearch 2.11.1 · OpenSearch).
which have a pre loaded opensearch output plugin.

What is the maximum Logstash version supports for opensearch for below two conditions.

  1. Logstash downloaded from official elasticsearch site (Download Logstash Free | Get Started Now | Elastic)

  2. Logstash OSS with OpenSearch Output Plugin" from the website (Opensearch 2.11.1 · OpenSearch)

In the opensearch document its mentioned like below

Install Logstash
To install Logstash on OpenSearch, first install Logstash on your cluster, then the OpenSearch Logstash plugin, as described in the following steps.

Tarball

*Make sure you have Java Development Kit (JDK) version 8 or 11 installed.

*Download the Logstash tarball from Logstash downloads(Download Logstash Free | Get Started Now | Elastic).

*Navigate to the downloaded folder in the terminal and extract the files. Make sure that your version of Logstash and platform matches the one downloaded:

tar -zxvf logstash-8.8.2-linux-x86_64.tar.gz
*Navigate to the logstash-8.8.2 directory.

*Use the following command to install the plugin:

bin/logstash-plugin install logstash-output-opensearch

Thanks & Regards,
Anu

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

I am aware that opensearch is AWS product. But i am checking if any suggestions can be provided from the experts over here.

4

Your LS host should have the internet connection. I have installed the opensearch plugin without problems.

  1. Install the pluging:
    logstash-plugin install logstash-output-opensearch
Validating logstash-output-opensearch
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Installing logstash-output-opensearch
Installation successful

  1. Check does the plugin exit:
    logstash-plugin.bat list

  2. Make LS .conf file to process your data.
    You need input, filter, output. Is your log in a file, syslog or how do you receive?
    The filter section give you ability to transform data by grok, json or similar plugin.

  3. Run LS, and monitor

  4. Validate data in OS.

1 Like
5

Thanks for the quick response. As per the OpenSearch document i need to install Java Development Kit (JDK) version 8 or 11.

If i am using Logstash 8.2 or later version is it mandatory to install the Java. From which version we don't want to install the Java. Please help me.

Did you installed Java separately. Which version of Logstash you are using.

Thanks & regards,
Anu

6

AFAIK LS will work without installed Java because JDK is included. When you run it, 1st sentence is:
"Using bundled JDK: /path/logstash/jdk/bin/java.exe"

Which version of Logstash you are using

The latest 8.11.1. AFAIK 8.x and 7.something both have JDK included.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.