# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=430 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 431 --- ## [Running a pipeline with a short command](https://discuss.elastic.co/t/running-a-pipeline-with-a-short-command/212017)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 1\ **Last updated:** [December 19, 2019, 7:04am UTC](https://discuss.elastic.co/t/running-a-pipeline-with-a-short-command/212017 "2019-12-19T07:04:21Z")
I have my conf files in the etc/logstash/conf.d/ directory, so whenever I run a pipeline I have to add this path to my command. ex: bin/logstash -f /etc/logstash/conf.d/pipe.conf what do I have to modify to run a pipel… --- ## [Need to parse field name via logstash](https://discuss.elastic.co/t/need-to-parse-field-name-via-logstash/212186)
**Author:** [@shagun](https://discuss.elastic.co/u/shagun)\ **Replies:** 4\ **Last updated:** [December 19, 2019, 5:27am UTC](https://discuss.elastic.co/t/need-to-parse-field-name-via-logstash/212186 "2019-12-19T05:27:13Z")
Hi, I need to parse the field in the following input JSON via Logstash. INPUT { "xyz": { "http://com.myappliation.com/abc/def/fieldName": "value", "id": "hsakjh-uuekjn-kj48ehu, "http://com.myappliation.com/abc… --- ## [Metrics filter plugin has delay in flushing](https://discuss.elastic.co/t/metrics-filter-plugin-has-delay-in-flushing/212437)
**Author:** [@Jehutywong](https://discuss.elastic.co/u/Jehutywong)\ **Replies:** 0\ **Last updated:** [December 19, 2019, 4:27am UTC](https://discuss.elastic.co/t/metrics-filter-plugin-has-delay-in-flushing/212437 "2019-12-19T04:27:44Z")
Logstash version: 7.5.1 When in heavy load, ~1.8M events per minute. I used meter and set clear\_interval and flush\_interval to both 60 seconds. The outcome is that, i got metrics event in every ~65 seconds. Any idea? --- ## [Elasticsearch input login - read only new documents](https://discuss.elastic.co/t/elasticsearch-input-login-read-only-new-documents/212408)
**Author:** [@Zabulon](https://discuss.elastic.co/u/Zabulon)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 9:54pm UTC](https://discuss.elastic.co/t/elasticsearch-input-login-read-only-new-documents/212408 "2019-12-18T21:54:25Z")
Hi, I'm a bit noob in the elastic world and English is not my primary language.. But, I need some help !!!! How can I get only new documents from a remote elasticsearch database using the "elasticsearch input plugin" … --- ## [Error while connecting to Amazon ES using Logstash on EC2 \[403 error\]](https://discuss.elastic.co/t/error-while-connecting-to-amazon-es-using-logstash-on-ec2-403-error/212055)
**Author:** [@Neeraj\_Rajagopalan\_I](https://discuss.elastic.co/u/Neeraj_Rajagopalan_I)\ **Replies:** 2\ **Last updated:** [December 18, 2019, 7:33pm UTC](https://discuss.elastic.co/t/error-while-connecting-to-amazon-es-using-logstash-on-ec2-403-error/212055 "2019-12-18T19:33:36Z")
I am connecting to AWS ES using logstash installed on AWS EC2. I have this issue since 2 days and not able to resolve. Any help would be appreciated. Below is the error log: \[root@ip-10-0-0-7 logstash\]# /usr/share/logs… --- ## [Seperate indexes for each log path field error](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235)
**Author:** [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Replies:** 3\ **Last updated:** [December 18, 2019, 6:54pm UTC](https://discuss.elastic.co/t/seperate-indexes-for-each-log-path-field-error/212235 "2019-12-18T18:54:44Z")
I am trying to tag every log file path in filebeat and in logstash, use that tag to create separate index for each log file. But only one index is being created. Why does document\_type not work here? Here is my filebeat… --- ## [Grokparsefailure output error](https://discuss.elastic.co/t/grokparsefailure-output-error/209350)
**Author:** [@Barushka](https://discuss.elastic.co/u/Barushka)\ **Replies:** 8\ **Last updated:** [December 18, 2019, 6:26pm UTC](https://discuss.elastic.co/t/grokparsefailure-output-error/209350 "2019-12-18T18:26:12Z")
My logstash shuts down because of this error: \[2019-11-25T10:53:06,400\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::Conf… --- ## [Filebeat System Dashboard](https://discuss.elastic.co/t/filebeat-system-dashboard/212283)
**Author:** [@tanthiamhuat](https://discuss.elastic.co/u/tanthiamhuat)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 8:45am UTC](https://discuss.elastic.co/t/filebeat-system-dashboard/212283 "2019-12-18T08:45:05Z")
I have successfully installed ELK (Elasticsearch, Logstash and Kibana) on a Virtue Machine in Azure, and tested succesfully that it is able to read CSV file and link to Azure SQL Data, via some .conf file which links up … --- ## [Logstash does not output to ES](https://discuss.elastic.co/t/logstash-does-not-output-to-es/212041)
**Author:** [@Terran](https://discuss.elastic.co/u/Terran)\ **Replies:** 3\ **Last updated:** [December 18, 2019, 4:44pm UTC](https://discuss.elastic.co/t/logstash-does-not-output-to-es/212041 "2019-12-18T16:44:12Z")
I am new to logstash and have worked to configure a basic pipeline. However, this does not seems to output anything into Elasticsearch. I can see it send output to stdout though. I am hoping that someone can look over… --- ## [Windows NX logs to file to logstash](https://discuss.elastic.co/t/windows-nx-logs-to-file-to-logstash/212366)
**Author:** [@Ryan\_Swisher](https://discuss.elastic.co/u/Ryan_Swisher)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 4:08pm UTC](https://discuss.elastic.co/t/windows-nx-logs-to-file-to-logstash/212366 "2019-12-18T16:08:43Z")
we use the USM anywhere SIEM. I exported the logs from the NX log log source and spits it out to a text file. I am using filebeat to grab this file and send to logstash. I am then struggling to get things parsed in elas… --- ## [WHat Protocol does Logstash uses to pull logs from Cloudfront](https://discuss.elastic.co/t/what-protocol-does-logstash-uses-to-pull-logs-from-cloudfront/212365)
**Author:** [@OseMike](https://discuss.elastic.co/u/OseMike)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 4:08pm UTC](https://discuss.elastic.co/t/what-protocol-does-logstash-uses-to-pull-logs-from-cloudfront/212365 "2019-12-18T16:08:13Z")
Hello, I would like to know which protocol, logstash uses to pull its log from cloudfront to the instance for processing. Is it FTP? UDP? or TCP? https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-logs-… --- ## [Transform large file from postgresql to elasticsearch](https://discuss.elastic.co/t/transform-large-file-from-postgresql-to-elasticsearch/211803)
**Author:** [@pilo](https://discuss.elastic.co/u/pilo)\ **Replies:** 5\ **Last updated:** [December 18, 2019, 2:49pm UTC](https://discuss.elastic.co/t/transform-large-file-from-postgresql-to-elasticsearch/211803 "2019-12-18T14:49:21Z")
Hi everybody I have an issue when dealing with large table in postgresql. I have a table with about 1 millions rows, each row contains some text, about half of A4 page of length. I want to index this table into elastics… --- ## [My input JDBC conf is not working .... \[org.logstash.execution.PeriodicFlush\]\[main\] Pushing flush onto pipeline.\]](https://discuss.elastic.co/t/my-input-jdbc-conf-is-not-working-org-logstash-execution-periodicflush-main-pushing-flush-onto-pipeline/212184)
**Author:** [@Haalanam](https://discuss.elastic.co/u/Haalanam)\ **Replies:** 3\ **Last updated:** [December 18, 2019, 1:51pm UTC](https://discuss.elastic.co/t/my-input-jdbc-conf-is-not-working-org-logstash-execution-periodicflush-main-pushing-flush-onto-pipeline/212184 "2019-12-18T13:51:57Z")
I cannot manage to get data into my es instance. This is what input jdbc looks like :slight\_smile: jdbc { type =\> "B" jdbc\_connection\_string =\> "jdbc:oracle:thin:@//fff.ff.ff.f:1521/fff" jdbc\_d… --- ## [EventLogFile object not able to access in logstash using logstash-input-salesforce plugin](https://discuss.elastic.co/t/eventlogfile-object-not-able-to-access-in-logstash-using-logstash-input-salesforce-plugin/212318)
**Author:** [@SumitJangid92](https://discuss.elastic.co/u/SumitJangid92)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 12:15pm UTC](https://discuss.elastic.co/t/eventlogfile-object-not-able-to-access-in-logstash-using-logstash-input-salesforce-plugin/212318 "2019-12-18T12:15:46Z")
Hi, I am using logstash-input-salesforce plugin to get the data from salesforce. It is working fine for objects like 'Account, Opportunity' but when I am fetching the data from 'EventLogFile'. It's showing me error tha… --- ## [Two field mapppings created one field.keyword and another field](https://discuss.elastic.co/t/two-field-mapppings-created-one-field-keyword-and-another-field/212315)
**Author:** [@SathishPrakasam](https://discuss.elastic.co/u/SathishPrakasam)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 12:08pm UTC](https://discuss.elastic.co/t/two-field-mapppings-created-one-field-keyword-and-another-field/212315 "2019-12-18T12:08:31Z")
Hi team, i need an urgent help, below is my elastic template. The fields that gets indexed is creating two fields, one with the field name "field" and another with name "field.keyword". Having tow different mapping fo… --- ## [Aggregation resets values](https://discuss.elastic.co/t/aggregation-resets-values/212282)
**Author:** [@NewToThis](https://discuss.elastic.co/u/NewToThis)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 8:40am UTC](https://discuss.elastic.co/t/aggregation-resets-values/212282 "2019-12-18T08:40:59Z")
Hello ladies and gentlemen, I am doing a simple logstash aggregegation where I am checking specific values on a SQL table and whenever the this value is being found I am having a counter count up. In the end I am just d… --- ## [Filebeat service](https://discuss.elastic.co/t/filebeat-service/212260)
**Author:** [@patil\_mahesh](https://discuss.elastic.co/u/patil_mahesh)\ **Replies:** 0\ **Last updated:** [December 18, 2019, 5:02am UTC](https://discuss.elastic.co/t/filebeat-service/212260 "2019-12-18T05:02:31Z")
Hi I am implementing anypoint monitoring , And installed filebeat agent listening to 5044 by default.i do not see the log appeared at client. I am assuming the same port number used for other server monitoring.how can… --- ## [How to add multiple conditions in single if statement if ((A==1) && (B==( 2 || 3))](https://discuss.elastic.co/t/how-to-add-multiple-conditions-in-single-if-statement-if-a-1-b-2-3/212103)
**Author:** [@Abdulrahman\_Batis](https://discuss.elastic.co/u/Abdulrahman_Batis)\ **Replies:** 3\ **Last updated:** [December 17, 2019, 2:56pm UTC](https://discuss.elastic.co/t/how-to-add-multiple-conditions-in-single-if-statement-if-a-1-b-2-3/212103 "2019-12-17T14:56:26Z")
I want to add tags to some known issue to make it easier to identified as known issue what we want to do is to add tag to logs messages only if that message came from 2 or 3 different process , for example: if "error… --- ## [REST API using logstash](https://discuss.elastic.co/t/rest-api-using-logstash/212177)
**Author:** [@akhilsharma.in](https://discuss.elastic.co/u/akhilsharma.in)\ **Replies:** 0\ **Last updated:** [December 17, 2019, 2:51pm UTC](https://discuss.elastic.co/t/rest-api-using-logstash/212177 "2019-12-17T14:51:18Z")
Hi Team, I'm executing REST API calls via logstash. I'm using http\_poller to generate authentication token in the input block. Then I'm using http filter plugin to fetch data. However, we've seen that it does not always… --- ## [Scripted upsert fails](https://discuss.elastic.co/t/scripted-upsert-fails/149078)
**Author:** [@Nenad](https://discuss.elastic.co/u/Nenad)\ **Replies:** 1\ **Last updated:** [December 17, 2019, 2:06pm UTC](https://discuss.elastic.co/t/scripted-upsert-fails/149078 "2019-12-17T14:06:03Z")
I have below output: output { elasticsearch { index =\> "meet2" document\_type =\> "seen" document\_id =\> "%{member\_a}" hosts =\> \["s1:9200"\] # doc\_as\_upsert =\> true action =\> "update" scripted\_upsert =\> true script… --- ## [Logstash for AWS security events?](https://discuss.elastic.co/t/logstash-for-aws-security-events/212173)
**Author:** [@mrcsmonteiro](https://discuss.elastic.co/u/mrcsmonteiro)\ **Replies:** 0\ **Last updated:** [December 17, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-for-aws-security-events/212173 "2019-12-17T14:05:55Z")
Hi All, I am new to Elasticsearch, and I have been assigned to a task to ingest AWS (CloudTrail, Config, and GuardDuty logs) from S3 buckets to Elastic Cloud for Security Compliance. This is the initial scope for Elasti… --- ## [Error parsing csv file // logstash](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963)
**Author:** [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\ **Replies:** 12\ **Last updated:** [December 17, 2019, 12:55pm UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963 "2019-12-17T12:55:27Z")
Hi, I'm checking my logstash logs and see many entries with WARN "Error parsing csv" (80% cases are well parsed). ¿could u help me out? Many thanks in advance. Furthermore, "date" field is not added to timestamp index. … --- ## [How to add a date filter statement into JDBC input?](https://discuss.elastic.co/t/how-to-add-a-date-filter-statement-into-jdbc-input/212123)
**Author:** [@Haalanam](https://discuss.elastic.co/u/Haalanam)\ **Replies:** 2\ **Last updated:** [December 17, 2019, 12:42pm UTC](https://discuss.elastic.co/t/how-to-add-a-date-filter-statement-into-jdbc-input/212123 "2019-12-17T12:42:45Z")
I have a JDBC input hat looks like this: jdbc { type =\> "D" jdbc\_driver\_class =\> "Java::oracle.jdbc.driver.OracleDriver" jdbc\_connection\_string =\> "jdbc:oracle:thin:@//nn.nn.nn.nn:1521/nn" jdbc\_user =\> … --- ## [Collectd to Logstash not working as expected](https://discuss.elastic.co/t/collectd-to-logstash-not-working-as-expected/212097)
**Author:** [@csamuel](https://discuss.elastic.co/u/csamuel)\ **Replies:** 0\ **Last updated:** [December 17, 2019, 5:22am UTC](https://discuss.elastic.co/t/collectd-to-logstash-not-working-as-expected/212097 "2019-12-17T05:22:07Z")
Hi Team , From collectd to colletctd i am getting below results . \[{"values": \[1\], "dstypes": \["gauge"\], "dsnames": \["value"\], "time": 1571655625, "interval": 90.0, "host": "APLEV-UHN1TKY311002", "plugin": "exec", "plu… --- ## [Amazon elastic search connection via EC2 instance installed logstash](https://discuss.elastic.co/t/amazon-elastic-search-connection-via-ec2-instance-installed-logstash/212078)
**Author:** [@ELK\_enthu1](https://discuss.elastic.co/u/ELK_enthu1)\ **Replies:** 0\ **Last updated:** [December 17, 2019, 1:25am UTC](https://discuss.elastic.co/t/amazon-elastic-search-connection-via-ec2-instance-installed-logstash/212078 "2019-12-17T01:25:44Z")
I am connecting to AWS ES using logstash installed on AWS EC2. I have this issue since 2 days and not able to resolve. Any help would be appreciated. Below is the error log: \[root@ip-10-0-0-7 logstash\]# /usr/share/logs… --- ## [Issue observed while trying to start the Logstash in ELK stack 7.5](https://discuss.elastic.co/t/issue-observed-while-trying-to-start-the-logstash-in-elk-stack-7-5/211885)
**Author:** [@anirbandas](https://discuss.elastic.co/u/anirbandas)\ **Replies:** 3\ **Last updated:** [December 17, 2019, 12:50am UTC](https://discuss.elastic.co/t/issue-observed-while-trying-to-start-the-logstash-in-elk-stack-7-5/211885 "2019-12-17T00:50:40Z")
While trying to start the logstash in ELK Stack 7.5 , i am getting the below error related to java.lang.ClassNotFoundException. C:\\Users\\xeadmin\\Desktop\\work\>logstash -f logstash.conf Java HotSpot(TM) 64-Bit Server VM … --- ## [Compare two datasets (failed)](https://discuss.elastic.co/t/compare-two-datasets-failed/211968)
**Author:** [@carmezsa](https://discuss.elastic.co/u/carmezsa)\ **Replies:** 1\ **Last updated:** [December 16, 2019, 7:57pm UTC](https://discuss.elastic.co/t/compare-two-datasets-failed/211968 "2019-12-16T19:57:47Z")
I am configure LOGSTASH to compare two dataset but something is wrong. tail /var/log/logstash/logstash-plain.log \[2019-12-16T10:12:14,264\]\[ERROR\]\[logstash.javapipeline \]\[main\] Pipeline aborted due to error {:pipe… --- ## [Wrapping CSV Output Fields in Quotes](https://discuss.elastic.co/t/wrapping-csv-output-fields-in-quotes/212049)
**Author:** [@Mohammad\_Etemad](https://discuss.elastic.co/u/Mohammad_Etemad)\ **Replies:** 2\ **Last updated:** [December 16, 2019, 7:44pm UTC](https://discuss.elastic.co/t/wrapping-csv-output-fields-in-quotes/212049 "2019-12-16T19:44:36Z")
Hi all, I am using the csv output for my logstash, but I want to have each field wrapped in quotes. So now I have the output as: filed1Value,field2Value,field3Value but I want the output to be: "filed1Value","field2V… --- ## [Google\_cloud\_storage](https://discuss.elastic.co/t/google-cloud-storage/212043)
**Author:** [@elk3](https://discuss.elastic.co/u/elk3)\ **Replies:** 0\ **Last updated:** [December 16, 2019, 6:09pm UTC](https://discuss.elastic.co/t/google-cloud-storage/212043 "2019-12-16T18:09:13Z")
i want read file from cloud into ES index, i created file google\_cloud\_storage via input plugin into Es but problem is logstash not terminating(i.e)logstash not shutdown , i want force logstash by ctrl+c or sigint is th… --- ## [Logstash persisted queue with xpack monitoring](https://discuss.elastic.co/t/logstash-persisted-queue-with-xpack-monitoring/210898)
**Author:** [@Yungyoung\_Ok](https://discuss.elastic.co/u/Yungyoung_Ok)\ **Replies:** 7\ **Last updated:** [December 16, 2019, 5:26pm UTC](https://discuss.elastic.co/t/logstash-persisted-queue-with-xpack-monitoring/210898 "2019-12-16T17:26:21Z")
I am using x-pack monitoring to monitor the logstash. logstash.yml xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.hosts: \["node-1", "node-2", "node-3"\] my logstash command: ./bin/logstash -f my\_conf.c… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=429) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=431)