Configure es with logstash

Elastic Stack Logstash
1

input {
s3 {
access_key_id => ""
secret_access_key => ""
bucket => "dumpsampleperigon"
region => "us-west-1"
}
}

output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "logs-%{+YYYY.MM.dd}"
xpack.monitoring.enabled => true
}
}

2

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-05-24T09:50:55,443][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2023-05-24T09:50:55,450][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[2023-05-24T09:50:56,005][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set xpack.monitoring.enabled: true in logstash.yml
[2023-05-24T09:50:56,007][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Metricbeat to monitor Logstash. Documentation can be found at:

[2023-05-24T09:50:56,308][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-05-24T09:50:56,670][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://elasticsearch:9200/]}}
[2023-05-24T09:50:56,926][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2023-05-24T09:50:56,987][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>7}
[2023-05-24T09:50:56,990][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2023-05-24T09:50:57,111][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2023-05-24T09:50:57,111][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2023-05-24T09:50:57,529][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2023-05-24T09:50:57,758][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [0-9], [ \t\r\n], "#", "}" at line 4, column 20 (byte 55) after input {\n beats {\n port => 5044\n version => 7.12", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in block in converge_state'"]}
[2023-05-24T09:50:58,621][INFO ][org.reflections.Reflections] Reflections took 40 ms to scan 1 urls, producing 23 keys and 47 values
[2023-05-24T09:50:58,725][WARN ][deprecation.logstash.outputs.elasticsearchmonitoring] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-05-24T09:50:58,810][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://elasticsearch:9200/]}}
[2023-05-24T09:50:58,823][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2023-05-24T09:50:58,831][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ES Output version determined {:es_version=>7}
[2023-05-24T09:50:58,831][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2023-05-24T09:50:58,879][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elasticsearch:9200"]}
[2023-05-24T09:50:58,887][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2023-05-24T09:50:58,956][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x2420b739 run>"}
[2023-05-24T09:50:59,689][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.73}
[2023-05-24T09:50:59,715][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2023-05-24T09:51:00,895][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2023-05-24T09:51:01,887][INFO ][logstash.runner ] Logstash shut down.

3

The pipeline you shared is not the one logstash is trying to run, this says that you have a beats input for example.

Please share the pipeline you are trying to run, use the preformatted text button, the </> button, when sharing log errors and configurations.

4

thank you for ur reply.
version: "3.7"
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
container_name: elasticsearch
restart: always
environment:
- xpack.security.enabled=false
- discovery.type=single-node
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- elasticsearch-data-volume:/usr/share/elasticsearch/data
ports:
- "9200:9200"
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.12.0
restart: always
environment:
SERVER_NAME: kibana
ELASTICSEARCH_HOSTS: http://elasticsearch:9200
ports:
- "5601:5601"
depends_on:
- elasticsearch
logstash:
container_name: logstash
image: docker.elastic.co/logstash/logstash:7.12.0
restart: always
volumes:
- ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
ports:
- "5044:5044"
depends_on:
- elasticsearch
filebeat:
container_name: filebeat
image: docker.elastic.co/beats/filebeat:7.12.0
restart: always
user: root
volumes:
- ./filebeat.yml:/usr/share/filebeat/filebeat.yml
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock
command: filebeat -e -strict.perms=false
depends_on:
- logstash

volumes:
elasticsearch-data-volume:
driver: local
this is my docker compose file wer in m truing filebeat conatiner off along weith that i have logstash.conf file which i shared earlier and both the file is in same folder, except this i have not created any pipeline. do i need to create a pipeline?

5

You need to share this file, the configuration you shared in your first post is not the configuration logstash is trying to run.

You are getting this erro:

[2023-05-24T09:50:57,758][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [0-9], [ \t\r\n], "#", "}" at line 4, column 20 (byte 55) after input {\n beats {\n port => 5044\n version => 7.12"

Which indicates that you have a beats input, but the configuration you shared does not have a beats input, without seeing your configuration is impossible to know what the issue can be.

Also, as already asked shared logs and configurations using the preformatted text option, the </> button.

6

Also, a beats input does not have a version option.

7

thank you for your reply, this is realted to my old config file which i chnges into new one and now also i am getting same error. i just have run compose file and the written logstash.config file we i have shared earlier, thn how come i am getting error regarding to old config file

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.