# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=541 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 542 --- ## [Logstash Input Plugin fails](https://discuss.elastic.co/t/logstash-input-plugin-fails/182703)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 4\ **Last updated:** [May 24, 2019, 7:07pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-fails/182703 "2019-05-24T19:07:21Z")
Hi everyone, I'm trying to export AWS Cloudwatch logs to Logstash input plugin. However, I get an error message saying "Failed to execute action...". \[pipeline.conf file\]: input{ cloudwatch { namespace =\> "AWS/Logs" … --- ## [Logstash output on local partition rather than in docker](https://discuss.elastic.co/t/logstash-output-on-local-partition-rather-than-in-docker/182710)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 6:32pm UTC](https://discuss.elastic.co/t/logstash-output-on-local-partition-rather-than-in-docker/182710 "2019-05-24T18:32:56Z")
Hi Guys, I have logstash instance running in docker container and I am using Logstash-csv-output plugin to write the data in CSV file; however, the data or file is being written inside docker container. What do I do so… --- ## [Logstash Inplut plugin metrics and filters?](https://discuss.elastic.co/t/logstash-inplut-plugin-metrics-and-filters/182708)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 6:09pm UTC](https://discuss.elastic.co/t/logstash-inplut-plugin-metrics-and-filters/182708 "2019-05-24T18:09:17Z")
From the image above, we can see the metrics and filters for variety of namesspaces including EC2 and EBS, but how about logs? I don't know what to enter for metrics and filters if I want to import AWS Cloudwatch logs … --- ## [org.elasticsearch.index.mapper.MapperParsingException: object mapping for \[message\] tried to parse field \[message\] as object, but found a concrete value](https://discuss.elastic.co/t/org-elasticsearch-index-mapper-mapperparsingexception-object-mapping-for-message-tried-to-parse-field-message-as-object-but-found-a-concrete-value/182597)
**Author:** [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Replies:** 10\ **Last updated:** [May 24, 2019, 4:13pm UTC](https://discuss.elastic.co/t/org-elasticsearch-index-mapper-mapperparsingexception-object-mapping-for-message-tried-to-parse-field-message-as-object-but-found-a-concrete-value/182597 "2019-05-24T16:13:54Z")
I upload a template, and try to parse a column, but it recognized as a string, how to make it as a object, please help... Here is my template: { "index\_patterns": "springboot\*", "order": 0, "settings": { "num… --- ## [Condition on filename in the path](https://discuss.elastic.co/t/condition-on-filename-in-the-path/182679)
**Author:** [@Hyrastios](https://discuss.elastic.co/u/Hyrastios)\ **Replies:** 2\ **Last updated:** [May 24, 2019, 4:09pm UTC](https://discuss.elastic.co/t/condition-on-filename-in-the-path/182679 "2019-05-24T16:09:57Z")
Hi. I want to parse all my apache\_logs in the same time, but to do so I have to differentiate logs coming from access files, error files, or mod\_jk files. I'm using filebeat to import my logs and I send them to elastic… --- ## [Combining 2 fields in logstash](https://discuss.elastic.co/t/combining-2-fields-in-logstash/179066)
**Author:** [@satyajeet\_dutt](https://discuss.elastic.co/u/satyajeet_dutt)\ **Replies:** 1\ **Last updated:** [May 24, 2019, 3:18pm UTC](https://discuss.elastic.co/t/combining-2-fields-in-logstash/179066 "2019-05-24T15:18:54Z")
I have been trying to combine 2 fields in logstash. My first clickstream log contains {x=y, software=DEMO, language=EN} as a parameter. My second log contains clickstream log {a=b, projectId=DEMO, language=EN} as a pa… --- ## [Logstash start Issue](https://discuss.elastic.co/t/logstash-start-issue/179734)
**Author:** [@inext3211](https://discuss.elastic.co/u/inext3211)\ **Replies:** 2\ **Last updated:** [May 24, 2019, 1:58pm UTC](https://discuss.elastic.co/t/logstash-start-issue/179734 "2019-05-24T13:58:04Z")
When i am trying to start Logstash using "service logstash start" a pop-up is asking for password to start logstash. I don't know about password. Please help.. Os:- Ubuntu 16.4 Elasticsearch:- 7.0.0 --- ## [Need help removing square brackets from field name in JSON input](https://discuss.elastic.co/t/need-help-removing-square-brackets-from-field-name-in-json-input/182351)
**Author:** [@devdua](https://discuss.elastic.co/u/devdua)\ **Replies:** 3\ **Last updated:** [May 24, 2019, 1:35pm UTC](https://discuss.elastic.co/t/need-help-removing-square-brackets-from-field-name-in-json-input/182351 "2019-05-24T13:35:52Z")
This is a sample of the log that's causing parse error - { "output": "11:08:30.411631274: Warning Sensitive file opened for reading by non-trusted program (user=root program=cat command=cat \\/etc\\/shadow file=\\/etc\\/s… --- ## [Does one failed output kill / stop the whole pipeline?](https://discuss.elastic.co/t/does-one-failed-output-kill-stop-the-whole-pipeline/182209)
**Author:** [@bjosve](https://discuss.elastic.co/u/bjosve)\ **Replies:** 4\ **Last updated:** [May 24, 2019, 1:35pm UTC](https://discuss.elastic.co/t/does-one-failed-output-kill-stop-the-whole-pipeline/182209 "2019-05-24T13:35:34Z")
Hi, I have a logstash output like this, output { tcp { mode =\> "client" host =\> "server01" port =\> 5400 codec =\> "line" } tcp { mode =\> "client" host =\> "server02" port =\> 5400 codec =\> "line" } file { p… --- ## [Is PKI authentication supported for Centralized Pipeline Management?](https://discuss.elastic.co/t/is-pki-authentication-supported-for-centralized-pipeline-management/182659)
**Author:** [@marc-sensenich](https://discuss.elastic.co/u/marc-sensenich)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 1:29pm UTC](https://discuss.elastic.co/t/is-pki-authentication-supported-for-centralized-pipeline-management/182659 "2019-05-24T13:29:38Z")
I'm looking to use Centralized Pipeline Management in my Logstash deployments with PKI authentication to Elasticsearch. Here is a snippet from my Logstash environment configuration xpack.management.enabled: "true" xpack… --- ## [Grok command to parse Spring boot logs](https://discuss.elastic.co/t/grok-command-to-parse-spring-boot-logs/182636)
**Author:** [@Dheeraj\_K\_Chaitanya](https://discuss.elastic.co/u/Dheeraj_K_Chaitanya)\ **Replies:** 1\ **Last updated:** [May 24, 2019, 1:15pm UTC](https://discuss.elastic.co/t/grok-command-to-parse-spring-boot-logs/182636 "2019-05-24T13:15:00Z")
Hi, My log looks like this " 2019-05-24 12:41:19.174 DEBUG 12528 --- \[http-nio-8888-exec-2\] c.c.b.c.BankApplicationController : AccountMaster \[accountId=1355, accountType=Savings, accountBalance=500.0, accountOp… --- ## [Make Array of objects as type nested](https://discuss.elastic.co/t/make-array-of-objects-as-type-nested/182635)
**Author:** [@Sayed\_Atif](https://discuss.elastic.co/u/Sayed_Atif)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 11:37am UTC](https://discuss.elastic.co/t/make-array-of-objects-as-type-nested/182635 "2019-05-24T11:37:18Z")
I am using JDBC-input and pulling data from Postgres. My input statement looks like statement =\> "select a.id, a.deleted\_at, a.updated\_at, json\_agg(b.\*)::text as user\_hubs, json\_agg(c.\*)::text as user\_schools, json\_agg(… --- ## [Does logstash read newly created kafka topics automatically](https://discuss.elastic.co/t/does-logstash-read-newly-created-kafka-topics-automatically/182626)
**Author:** [@prabhakar\_talari](https://discuss.elastic.co/u/prabhakar_talari)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 11:16am UTC](https://discuss.elastic.co/t/does-logstash-read-newly-created-kafka-topics-automatically/182626 "2019-05-24T11:16:15Z")
All, May i know does logstash will read/load newly created kafka topics automatically or do we need to restart logstash to read newly created kafka topics Thanks --- ## [How to create different actions for different log patterns by grok](https://discuss.elastic.co/t/how-to-create-different-actions-for-different-log-patterns-by-grok/182608)
**Author:** [@rijinmp](https://discuss.elastic.co/u/rijinmp)\ **Replies:** 4\ **Last updated:** [May 24, 2019, 11:13am UTC](https://discuss.elastic.co/t/how-to-create-different-actions-for-different-log-patterns-by-grok/182608 "2019-05-24T11:13:53Z")
Hi All , I have a log set . Only the time stamp is the common factor . All other part of the log is some java errors . Doesn't have any common structure. I am categorizing the error types by particular word's presence … --- ## [Logstash.conf doesnt work when run as service but runs otherwise](https://discuss.elastic.co/t/logstash-conf-doesnt-work-when-run-as-service-but-runs-otherwise/182298)
**Author:** [@Darshan\_v\_Shah](https://discuss.elastic.co/u/Darshan_v_Shah)\ **Replies:** 4\ **Last updated:** [May 24, 2019, 11:10am UTC](https://discuss.elastic.co/t/logstash-conf-doesnt-work-when-run-as-service-but-runs-otherwise/182298 "2019-05-24T11:10:36Z")
I want to listen to beats on port 5000 and also push a local file to ES. In process of debugging, I have commented the rest and just running : input { file { path=\> "/home/ec2-user/URLTestOutput/out" codec =\> "json"} } … --- ## [Logstash upgrade dilemma?!](https://discuss.elastic.co/t/logstash-upgrade-dilemma/182589)
**Author:** [@agiorgi](https://discuss.elastic.co/u/agiorgi)\ **Replies:** 2\ **Last updated:** [May 24, 2019, 10:24am UTC](https://discuss.elastic.co/t/logstash-upgrade-dilemma/182589 "2019-05-24T10:24:23Z")
Hello there! I'd like to update my ELK stack to version 7.1. Still, i don't understand whether i have to turn off beats, forwarders etc.. When upgrading logstash. Assuming that i have 6 instances of logstash , behind… --- ## [JSON Stream from HTTP API as Input](https://discuss.elastic.co/t/json-stream-from-http-api-as-input/182601)
**Author:** [@shorty](https://discuss.elastic.co/u/shorty)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 9:29am UTC](https://discuss.elastic.co/t/json-stream-from-http-api-as-input/182601 "2019-05-24T09:29:01Z")
Hello all, I'm looking for a way to read an "endless" stream of JSON from an HTTP API. The JSON lines are newline delimited. Using the http\_plugin I always get after a lot of time the error "Could not read from stream: … --- ## [Applying analyzer in filter elasticsearch](https://discuss.elastic.co/t/applying-analyzer-in-filter-elasticsearch/182598)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 9:23am UTC](https://discuss.elastic.co/t/applying-analyzer-in-filter-elasticsearch/182598 "2019-05-24T09:23:43Z")
Hello. I am attempting to enrich my data with ES lookup when the URL appears in my log line. if \[url\]{ elasticsearch { hosts =\> \["localhost:9200"\] index =\> "url\_lookup" query\_template =\> "/etc/logstash/conf.d/quer… --- ## [Can't figure out the field to split on](https://discuss.elastic.co/t/cant-figure-out-the-field-to-split-on/182590)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 8:40am UTC](https://discuss.elastic.co/t/cant-figure-out-the-field-to-split-on/182590 "2019-05-24T08:40:41Z")
Hi, I'm using curl to query Elasticsearch. The response is in JSON and looks like the example below. But I can't figure out which field to split on to create a seperate document for each person. I tried json filter and… --- ## [JDBC Input (MSSQL) to CSV output to slow - memory no way near Xmx](https://discuss.elastic.co/t/jdbc-input-mssql-to-csv-output-to-slow-memory-no-way-near-xmx/182573)
**Author:** [@romariol](https://discuss.elastic.co/u/romariol)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 7:12am UTC](https://discuss.elastic.co/t/jdbc-input-mssql-to-csv-output-to-slow-memory-no-way-near-xmx/182573 "2019-05-24T07:12:35Z")
Hello everyone, I'm having a use case where I have a JDBC input to a MSSQL server, fetching all the rows and writing them to a CSV file. the table is quite big (+/- 90 million records) so getting the data and writing it… --- ## [Time\_file field of S3 ouptut plugin](https://discuss.elastic.co/t/time-file-field-of-s3-ouptut-plugin/182502)
**Author:** [@Mudit\_Mehta1](https://discuss.elastic.co/u/Mudit_Mehta1)\ **Replies:** 2\ **Last updated:** [May 24, 2019, 6:44am UTC](https://discuss.elastic.co/t/time-file-field-of-s3-ouptut-plugin/182502 "2019-05-24T06:44:36Z")
I am using S3 output plugin for logstash version 7.0.1 to ship logs to S3 bucket. I set up a docker container running logstash. But the plugin is creating objects in S3 after 15-20 mins delay. Is this because of the time… --- ## [Storing JSON array field from jdbc input to elastic search](https://discuss.elastic.co/t/storing-json-array-field-from-jdbc-input-to-elastic-search/182357)
**Author:** [@Sayed\_Atif](https://discuss.elastic.co/u/Sayed_Atif)\ **Replies:** 2\ **Last updated:** [May 24, 2019, 5:35am UTC](https://discuss.elastic.co/t/storing-json-array-field-from-jdbc-input-to-elastic-search/182357 "2019-05-24T05:35:53Z")
My configuration file has a filter which converts majors field filter { json { source =\> "majors" remove\_field =\> \["majors"\] } } I get following debug event { "user\_id" =\> 1529067, "schoo… --- ## [Failed to query elasticsearch issue](https://discuss.elastic.co/t/failed-to-query-elasticsearch-issue/182216)
**Author:** [@sangeetha\_sankar](https://discuss.elastic.co/u/sangeetha_sankar)\ **Replies:** 2\ **Last updated:** [May 24, 2019, 4:52am UTC](https://discuss.elastic.co/t/failed-to-query-elasticsearch-issue/182216 "2019-05-24T04:52:34Z")
Below is the query i have written in logstash config file elasticsearch { hosts =\> \["localhost:9200"\] index =\> "restricteddata" query =\> "type:restricted AND acc\_number:%{\[acc\_id\]} AND corp\_id:%{\[corp\_code\]… --- ## [Generate multiple index for diff service logs coming from one single server](https://discuss.elastic.co/t/generate-multiple-index-for-diff-service-logs-coming-from-one-single-server/182361)
**Author:** [@Eva](https://discuss.elastic.co/u/Eva)\ **Replies:** 2\ **Last updated:** [May 24, 2019, 4:16am UTC](https://discuss.elastic.co/t/generate-multiple-index-for-diff-service-logs-coming-from-one-single-server/182361 "2019-05-24T04:16:49Z")
Hello there I would like to know if it is possible to generate more than one "logstash" index for different service logs coming from different services within same server. This is because I have multiple services runnin… --- ## [AWS Cloudwatch Logs Exported directly by Logstash/Filebeat?](https://discuss.elastic.co/t/aws-cloudwatch-logs-exported-directly-by-logstash-filebeat/182538)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 0\ **Last updated:** [May 24, 2019, 2:14am UTC](https://discuss.elastic.co/t/aws-cloudwatch-logs-exported-directly-by-logstash-filebeat/182538 "2019-05-24T02:14:03Z")
Hi everyone, \[Background\]: I'm a coop student trying to learn all ELK Stack components. I've been struggling with Logstash input plugin recently. I've spent about two days trying to find a way to directly get cloudwatc… --- ## [Logstash 7.1.0 problem on Install](https://discuss.elastic.co/t/logstash-7-1-0-problem-on-install/182522)
**Author:** [@kingram](https://discuss.elastic.co/u/kingram)\ **Replies:** 3\ **Last updated:** [May 24, 2019, 12:04am UTC](https://discuss.elastic.co/t/logstash-7-1-0-problem-on-install/182522 "2019-05-24T00:04:37Z")
Hardware: Raspbberry Pi3 (arm processor) OS: Raspbian (Debian Stretch 9) Java: openjdk 10 $ bin/logstash -e 'input { stdin {} } output { stdout {} }' OpenJDK Server VM warning: Option UseConcMarkSweepGC was deprecated… --- ## [Winlogbeat - event\_data data types](https://discuss.elastic.co/t/winlogbeat-event-data-data-types/182503)
**Author:** [@raged](https://discuss.elastic.co/u/raged)\ **Replies:** 1\ **Last updated:** [May 23, 2019, 9:20pm UTC](https://discuss.elastic.co/t/winlogbeat-event-data-data-types/182503 "2019-05-23T21:20:10Z")
I am trying to get properties that are sent to logstash via our Winlogbeat data shipper to show up as an integer type if the value is indeed an integer. Here is an example of a snippet of JSON that was posted to ES: "e… --- ## [Parse custom log file to get metrics](https://discuss.elastic.co/t/parse-custom-log-file-to-get-metrics/182042)
**Author:** [@Etienne.lv](https://discuss.elastic.co/u/Etienne.lv)\ **Replies:** 4\ **Last updated:** [May 23, 2019, 8:54pm UTC](https://discuss.elastic.co/t/parse-custom-log-file-to-get-metrics/182042 "2019-05-23T20:54:44Z")
Hi ! I'm new on ELK environment, so I apologize for asking some "obvious" things. To understand my problem, this is the context : I wrote an application and deploy it on 2 VMs "server". (in the future i would like to… --- ## [Reduce the amount of shards of index (Elasticsearch)](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109)
**Author:** [@zolthar-z](https://discuss.elastic.co/u/zolthar-z)\ **Replies:** 8\ **Last updated:** [May 23, 2019, 7:16pm UTC](https://discuss.elastic.co/t/reduce-the-amount-of-shards-of-index-elasticsearch/178109 "2019-05-23T19:16:53Z")
Hi All Right now I have a elk configuration where filebeat send logs to logstash and logstash applied a filter and send the information to elasticsearch, this configurations is generating to many shards on each index an… --- ## [Process downloaded log files](https://discuss.elastic.co/t/process-downloaded-log-files/180948)
**Author:** [@Yogendra\_Hada](https://discuss.elastic.co/u/Yogendra_Hada)\ **Replies:** 10\ **Last updated:** [May 23, 2019, 6:09pm UTC](https://discuss.elastic.co/t/process-downloaded-log-files/180948 "2019-05-23T18:09:49Z")
Hi, I am downloading files from S3 on a daily basis. The number of files are quite huge. approximately~ 2,50,000. I want Logstash to process these files and send events to Elasticsearch before I encounter next download… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=540) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=542)