# Configure es with logstash **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [May 29, 2023, 4:31pm UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604 "2023-05-29T16:31:51Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![sujata\_g](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujata_g/32/118222_2.png) [@sujata\_g](https://discuss.elastic.co/u/sujata_g)\ **Post date:** [May 29, 2023, 4:31pm UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/1 "2023-05-29T16:31:51Z")
input { s3 { access\_key\_id =\> "" secret\_access\_key =\> "" bucket =\> "dumpsampleperigon" region =\> "us-west-1" } } output { elasticsearch { hosts =\> ["[http://elasticsearch:9200](http://elasticsearch:9200)"] index =\> "logs-%{+YYYY.MM.dd}" xpack.monitoring.enabled =\> true } } ---
**Author:** ![sujata\_g](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujata_g/32/118222_2.png) [@sujata\_g](https://discuss.elastic.co/u/sujata_g)\ **Post date:** [May 29, 2023, 4:32pm UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/2 "2023-05-29T16:32:17Z")
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties [2023-05-24T09:50:55,443][INFO][logstash.runner] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties [2023-05-24T09:50:55,450][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"7.12.0", "jruby.version"=\>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86\_64]"} [2023-05-24T09:50:56,005][WARN][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml [2023-05-24T09:50:56,007][WARN][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version. Please configure Metricbeat to monitor Logstash. Documentation can be found at: > **[Collect Logstash monitoring data with Metricbeat | Logstash Reference \[8.8\] |...](https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html)** [2023-05-24T09:50:56,308][WARN][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [2023-05-24T09:50:56,670][INFO][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://elasticsearch:9200/](http://elasticsearch:9200/)]}} [2023-05-24T09:50:56,926][WARN][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=\>"[http://elasticsearch:9200/](http://elasticsearch:9200/)"} [2023-05-24T09:50:56,987][INFO][logstash.licensechecker.licensereader] ES Output version determined {:es\_version=\>7} [2023-05-24T09:50:56,990][WARN][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>7} [2023-05-24T09:50:57,111][INFO][logstash.monitoring.internalpipelinesource] Monitoring License OK [2023-05-24T09:50:57,111][INFO][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline. [2023-05-24T09:50:57,529][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2023-05-24T09:50:57,758][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [0-9], [\t\r\n], "#", "}" at line 4, column 20 (byte 55) after input {\n beats {\n port =\> 5044\n version =\> 7.12", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge\_state'"]} [2023-05-24T09:50:58,621][INFO][org.reflections.Reflections] Reflections took 40 ms to scan 1 urls, producing 23 keys and 47 values [2023-05-24T09:50:58,725][WARN][deprecation.logstash.outputs.elasticsearchmonitoring] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [2023-05-24T09:50:58,810][INFO][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://elasticsearch:9200/](http://elasticsearch:9200/)]}} [2023-05-24T09:50:58,823][WARN][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=\>"[http://elasticsearch:9200/](http://elasticsearch:9200/)"} [2023-05-24T09:50:58,831][INFO][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ES Output version determined {:es\_version=\>7} [2023-05-24T09:50:58,831][WARN][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>7} [2023-05-24T09:50:58,879][INFO][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=\>["[http://elasticsearch:9200](http://elasticsearch:9200)"]} [2023-05-24T09:50:58,887][WARN][logstash.javapipeline][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary [2023-05-24T09:50:58,956][INFO][logstash.javapipeline][.monitoring-logstash] Starting pipeline {:pipeline\_id=\>".monitoring-logstash", "pipeline.workers"=\>1, "pipeline.batch.size"=\>2, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>2, "pipeline.sources"=\>["monitoring pipeline"], :thread=\>"#\"} [2023-05-24T09:50:59,689][INFO][logstash.javapipeline][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=\>0.73} [2023-05-24T09:50:59,715][INFO][logstash.javapipeline][.monitoring-logstash] Pipeline started {"pipeline.id"=\>".monitoring-logstash"} [2023-05-24T09:51:00,895][INFO][logstash.javapipeline][.monitoring-logstash] Pipeline terminated {"pipeline.id"=\>".monitoring-logstash"} [2023-05-24T09:51:01,887][INFO][logstash.runner] Logstash shut down. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 29, 2023, 5:36pm UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/3 "2023-05-29T17:36:46Z")
> [@sujata\_g](#): > > [2023-05-24T09:50:57,758][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [0-9], [\t\r\n], "#", "}" at line 4, column 20 (byte 55) after input {\n beats {\n port =\> 5044\n version =\> 7.12" The pipeline you shared is not the one logstash is trying to run, this says that you have a `beats` input for example. Please share the pipeline you are trying to run, use the preformatted text button, the `` button, when sharing log errors and configurations. ---
**Author:** ![sujata\_g](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujata_g/32/118222_2.png) [@sujata\_g](https://discuss.elastic.co/u/sujata_g)\ **Post date:** [May 30, 2023, 4:26am UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/4 "2023-05-30T04:26:40Z")
thank you for ur reply. version: "3.7" services: elasticsearch: image: [docker.elastic.co/elasticsearch/elasticsearch:7.12.0](http://docker.elastic.co/elasticsearch/elasticsearch:7.12.0) container\_name: elasticsearch restart: always environment: - xpack.security.enabled=false - discovery.type=single-node ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 cap\_add: - IPC\_LOCK volumes: - elasticsearch-data-volume:/usr/share/elasticsearch/data ports: - "9200:9200" kibana: container\_name: kibana image: [docker.elastic.co/kibana/kibana:7.12.0](http://docker.elastic.co/kibana/kibana:7.12.0) restart: always environment: SERVER\_NAME: kibana ELASTICSEARCH\_HOSTS: [http://elasticsearch:9200](http://elasticsearch:9200) ports: - "5601:5601" depends\_on: - elasticsearch logstash: container\_name: logstash image: [docker.elastic.co/logstash/logstash:7.12.0](http://docker.elastic.co/logstash/logstash:7.12.0) restart: always volumes: - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf ports: - "5044:5044" depends\_on: - elasticsearch filebeat: container\_name: filebeat image: [docker.elastic.co/beats/filebeat:7.12.0](http://docker.elastic.co/beats/filebeat:7.12.0) restart: always user: root volumes: - ./filebeat.yml:/usr/share/filebeat/filebeat.yml - /var/lib/docker/containers:/var/lib/docker/containers:ro - /var/run/docker.sock:/var/run/docker.sock command: filebeat -e -strict.perms=false depends\_on: - logstash volumes: elasticsearch-data-volume: driver: local this is my docker compose file wer in m truing filebeat conatiner off along weith that i have logstash.conf file which i shared earlier and both the file is in same folder, except this i have not created any pipeline. do i need to create a pipeline? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 31, 2023, 12:13pm UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/5 "2023-05-31T12:13:12Z")
> [@sujata\_g](#): > > volumes: > > - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf You need to share this file, the configuration you shared in your first post **is not** the configuration logstash is trying to run. You are getting this erro: > [2023-05-24T09:50:57,758][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [0-9], [\t\r\n], "#", "}" at line 4, column 20 (byte 55) after input {\n beats {\n port =\> 5044\n version =\> 7.12" Which indicates that you have a `beats` input, but the configuration you shared does not have a beats input, without seeing your configuration is impossible to know what the issue can be. Also, as already asked shared logs and configurations using the preformatted text option, the `` button. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 1, 2023, 1:29am UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/6 "2023-06-01T01:29:24Z")
> [@leandrojmp](#): > > input {\n beats {\n port =\> 5044\n version =\> 7.12" Also, a beats input does not have a version option. ---
**Author:** ![sujata\_g](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sujata_g/32/118222_2.png) [@sujata\_g](https://discuss.elastic.co/u/sujata_g)\ **Post date:** [June 4, 2023, 10:25am UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/7 "2023-06-04T10:25:52Z")
thank you for your reply, this is realted to my old config file which i chnges into new one and now also i am getting same error. i just have run compose file and the written logstash.config file we i have shared earlier, thn how come i am getting error regarding to old config file ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 2, 2023, 10:26am UTC](https://discuss.elastic.co/t/configure-es-with-logstash/334604/8 "2023-07-02T10:26:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.