# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=578 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 579 --- ## [Logstash jdbc output is giving only one record even](https://discuss.elastic.co/t/logstash-jdbc-output-is-giving-only-one-record-even/173613)
**Author:** [@Vikram\_Sivam](https://discuss.elastic.co/u/Vikram_Sivam)\ **Replies:** 1\ **Last updated:** [March 26, 2019, 6:41pm UTC](https://discuss.elastic.co/t/logstash-jdbc-output-is-giving-only-one-record-even/173613 "2019-03-26T18:41:05Z")
Hello Team, We are using logstash jdbc plugin to inject data to ES from SQL server database. The query used has 50 records as output but the data injected in ES has only 1 record. We have tried using fingerprint and t… --- ## [Group errors messages by type](https://discuss.elastic.co/t/group-errors-messages-by-type/173943)
**Author:** [@Ebisu](https://discuss.elastic.co/u/Ebisu)\ **Replies:** 1\ **Last updated:** [March 26, 2019, 6:33pm UTC](https://discuss.elastic.co/t/group-errors-messages-by-type/173943 "2019-03-26T18:33:09Z")
So let's say we have a few dozen different types of error messages in our logs, but with no error codes by which we could easily group them. Those messages have a form that could be parsed by grok like this: Could not … --- ## [Logs not being parsed](https://discuss.elastic.co/t/logs-not-being-parsed/173629)
**Author:** [@Abed\_Bishara](https://discuss.elastic.co/u/Abed_Bishara)\ **Replies:** 1\ **Last updated:** [March 26, 2019, 5:58pm UTC](https://discuss.elastic.co/t/logs-not-being-parsed/173629 "2019-03-26T17:58:01Z")
Hi I can see some logs are not being parsed by the Kibana How can I filter those specific logs to see which logs are not being parsed ? --- ## [Need to filter below AWS API gateway logs](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796)
**Author:** [@Piyush\_Sonigra](https://discuss.elastic.co/u/Piyush_Sonigra)\ **Replies:** 4\ **Last updated:** [March 26, 2019, 5:16pm UTC](https://discuss.elastic.co/t/need-to-filter-below-aws-api-gateway-logs/173796 "2019-03-26T17:16:15Z")
{"messageType":"DATA\_MESSAGE", "owner":"144360258", "logGroup":"API-Gateway-Execution-Logs\_x63d3nk/live", "logStream":"d645920e395fedad7bbbed0eca3fe2e0","subscriptionFilters":\["API-Gateway-Execution-Logs\_x63d3nr84klive"\]… --- ## [Logstash-output-elasticsearch Connection reset while trying to send bulk request to elasticsearch](https://discuss.elastic.co/t/logstash-output-elasticsearch-connection-reset-while-trying-to-send-bulk-request-to-elasticsearch/173941)
**Author:** [@pgrange](https://discuss.elastic.co/u/pgrange)\ **Replies:** 1\ **Last updated:** [March 26, 2019, 5:13pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-connection-reset-while-trying-to-send-bulk-request-to-elasticsearch/173941 "2019-03-26T17:13:41Z")
Hello, I have a logstash node which outputs data to an elasticsearch server. Both are hosted on a gcloud platform. On the logstash server I often (several times a day) see Connection reset errors in the logs: LogStas… --- ## [SAP - Logstash connection](https://discuss.elastic.co/t/sap-logstash-connection/171526)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 3\ **Last updated:** [March 26, 2019, 4:17pm UTC](https://discuss.elastic.co/t/sap-logstash-connection/171526 "2019-03-26T16:17:08Z")
Hello, I want to ingest my SAP data to Logstash. How can I do it? Thank you, Francisca --- ## [Multiple filebeats with different log type specification on logstash](https://discuss.elastic.co/t/multiple-filebeats-with-different-log-type-specification-on-logstash/173867)
**Author:** [@cenginz](https://discuss.elastic.co/u/cenginz)\ **Replies:** 2\ **Last updated:** [March 26, 2019, 3:43pm UTC](https://discuss.elastic.co/t/multiple-filebeats-with-different-log-type-specification-on-logstash/173867 "2019-03-26T15:43:47Z")
Hi all, I've two filebeats located on different servers with different purposes. One of them I've been using with the logstash configuration I give below to ship docker logs. And I've a new filebeat for shipping hazelca… --- ## [Log stash with file-beat log file](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807)
**Author:** [@nagr](https://discuss.elastic.co/u/nagr)\ **Replies:** 12\ **Last updated:** [March 26, 2019, 2:54pm UTC](https://discuss.elastic.co/t/log-stash-with-file-beat-log-file/172807 "2019-03-26T14:54:21Z")
Hi, I am trying to load the log file with some filter condition using filebeat plugin, for example please is the raw log file content entry "2016-01-13 01:39:34, Info DPX Started DPX phase: Invent… --- ## [ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, { at line 24, column 7](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-at-line-24-column-7/173833)
**Author:** [@sblancocr](https://discuss.elastic.co/u/sblancocr)\ **Replies:** 6\ **Last updated:** [March 26, 2019, 2:47pm UTC](https://discuss.elastic.co/t/error-logstash-agent-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-at-line-24-column-7/173833 "2019-03-26T14:47:40Z")
Hi. I installed a logstash and show me next error log: \[2019-03-25T15:36:16,145\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"6.6.1"} \[2019-03-25T15:36:18,147\]\[ERROR\]\[logstash.agent … --- ## [Calculate time difference between 2 lines in log file](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file/173684)
**Author:** [@Thibaut\_M](https://discuss.elastic.co/u/Thibaut_M)\ **Replies:** 5\ **Last updated:** [March 26, 2019, 2:10pm UTC](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file/173684 "2019-03-26T14:10:46Z")
Hi everyone, I retrived a log file that look like this : 2019-03-07 04:35:57.421 19EC | INFO ROOT: (Netto2600-Brutto2608)AllocatedBlockCount=0 2019-03-07 04:35:57.421 19EC | INFO ROOT: … --- ## [Connect Firebird DB to Logstash , Beats](https://discuss.elastic.co/t/connect-firebird-db-to-logstash-beats/173718)
**Author:** [@TEJASWI\_CHUNDURI](https://discuss.elastic.co/u/TEJASWI_CHUNDURI)\ **Replies:** 1\ **Last updated:** [March 26, 2019, 1:39pm UTC](https://discuss.elastic.co/t/connect-firebird-db-to-logstash-beats/173718 "2019-03-26T13:39:34Z")
Please let me know whether there is support for firebird DB from logstash /Beats. --- ## [XML Parse Returns an Empty Array](https://discuss.elastic.co/t/xml-parse-returns-an-empty-array/173652)
**Author:** [@NativAt](https://discuss.elastic.co/u/NativAt)\ **Replies:** 3\ **Last updated:** [March 26, 2019, 1:11pm UTC](https://discuss.elastic.co/t/xml-parse-returns-an-empty-array/173652 "2019-03-26T13:11:24Z")
I have the following XML file: \ \ \7290027600007\ \001\ \001\ \9\ \8.… --- ## [JDBC Input Plugin use different input in one single SQL query](https://discuss.elastic.co/t/jdbc-input-plugin-use-different-input-in-one-single-sql-query/173571)
**Author:** [@Fazolo](https://discuss.elastic.co/u/Fazolo)\ **Replies:** 2\ **Last updated:** [March 26, 2019, 11:44am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-use-different-input-in-one-single-sql-query/173571 "2019-03-26T11:44:40Z")
Hello, I have data in two different databases. In database 1 there is products details and in database2 there is metrics related to these products. Is it possible with the JDBC input plugin to have as the output a sing… --- ## [Can I have several input listening on the same port?](https://discuss.elastic.co/t/can-i-have-several-input-listening-on-the-same-port/173912)
**Author:** [@angelicamaria](https://discuss.elastic.co/u/angelicamaria)\ **Replies:** 1\ **Last updated:** [March 26, 2019, 11:38am UTC](https://discuss.elastic.co/t/can-i-have-several-input-listening-on-the-same-port/173912 "2019-03-26T11:38:51Z")
Hello Guys, I am starting with this topic and I have the following question: Can I have several input listening on the same port? --- ## [Grokdebug says expression is ok but I get grokfailures when running live](https://discuss.elastic.co/t/grokdebug-says-expression-is-ok-but-i-get-grokfailures-when-running-live/173779)
**Author:** [@bhu73](https://discuss.elastic.co/u/bhu73)\ **Replies:** 3\ **Last updated:** [March 26, 2019, 10:58am UTC](https://discuss.elastic.co/t/grokdebug-says-expression-is-ok-but-i-get-grokfailures-when-running-live/173779 "2019-03-26T10:58:44Z")
I have a mangled syslogish message format coming from Microsoft Orchestrator :- {"message":"\<41\>Testmeddelande: jhkhuhuihuhukhukjhkjhkjhnÄÄÄ\\u0000","@version":"1","@timestamp":"2019-03-25T10:47:07.173Z","host":"192.168… --- ## [How to remove unfiltered field so won't view that on Kibana Discovery](https://discuss.elastic.co/t/how-to-remove-unfiltered-field-so-wont-view-that-on-kibana-discovery/173877)
**Author:** [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\ **Replies:** 0\ **Last updated:** [March 26, 2019, 8:13am UTC](https://discuss.elastic.co/t/how-to-remove-unfiltered-field-so-wont-view-that-on-kibana-discovery/173877 "2019-03-26T08:13:18Z")
Hi Guys, Can you help me, I've filtered the field that I want to see on my Kibana Discover. But, the thing is the other field I din't filter also still in the Discovery field on Kibana. I used two filebeats and the oth… --- ## [Collecting Json/CSV data from a URL as input](https://discuss.elastic.co/t/collecting-json-csv-data-from-a-url-as-input/173849)
**Author:** [@HaranKumar](https://discuss.elastic.co/u/HaranKumar)\ **Replies:** 0\ **Last updated:** [March 26, 2019, 2:49am UTC](https://discuss.elastic.co/t/collecting-json-csv-data-from-a-url-as-input/173849 "2019-03-26T02:49:23Z")
Hello Team, I am collecting list of IOC from various sources using minemeld and it is available in a URL. https://...My\_IP\_address.../feeds/Bad\_IPv4?tr=1&v=json How do I collect the list of IPs and send it into elasti… --- ## [Logstash MSSQL on Azure Marketplace Template](https://discuss.elastic.co/t/logstash-mssql-on-azure-marketplace-template/173836)
**Author:** [@dlaine](https://discuss.elastic.co/u/dlaine)\ **Replies:** 0\ **Last updated:** [March 25, 2019, 10:31pm UTC](https://discuss.elastic.co/t/logstash-mssql-on-azure-marketplace-template/173836 "2019-03-25T22:31:13Z")
I have the Azure Marketplace template deployed and need to setup a pipeline to MSSQL. Do I need to install the MS JDBC drivers or are they installed as part of the solution? If not, is there any documentation on setting … --- ## [Merging two lines with the same id to use as one document](https://discuss.elastic.co/t/merging-two-lines-with-the-same-id-to-use-as-one-document/173213)
**Author:** [@pbressan13](https://discuss.elastic.co/u/pbressan13)\ **Replies:** 3\ **Last updated:** [March 25, 2019, 8:45pm UTC](https://discuss.elastic.co/t/merging-two-lines-with-the-same-id-to-use-as-one-document/173213 "2019-03-25T20:45:57Z")
Hi, I'm struggling a few days in order to use aggregate filter to work (or i think it should work like this) I have two lines, as follows E.g: 08:00:45,585|73560708|ibwsin-tp-359|INFO|LoggerService||LogExecutionResul… --- ## [Logstash - Elastic search input plugin - Aggregation query doesn't seem to work](https://discuss.elastic.co/t/logstash-elastic-search-input-plugin-aggregation-query-doesnt-seem-to-work/171147)
**Author:** [@hariharnarayanan](https://discuss.elastic.co/u/hariharnarayanan)\ **Replies:** 1\ **Last updated:** [March 25, 2019, 8:26pm UTC](https://discuss.elastic.co/t/logstash-elastic-search-input-plugin-aggregation-query-doesnt-seem-to-work/171147 "2019-03-25T20:26:35Z")
I'm trying to use logstash to source data from elastic search and index them in elastic search log index. Basically, I am trying to use an aggregate query from ES input and put them in another ES log index. When I use cu… --- ## [Ingesting files via ingest plugin](https://discuss.elastic.co/t/ingesting-files-via-ingest-plugin/173619)
**Author:** [@yungnvn](https://discuss.elastic.co/u/yungnvn)\ **Replies:** 0\ **Last updated:** [March 23, 2019, 11:05pm UTC](https://discuss.elastic.co/t/ingesting-files-via-ingest-plugin/173619 "2019-03-23T23:05:21Z")
Hi We have some folks trying to ingest files(pdfs) using the ingest plugin however the data is coming in as encrypted rawContent instead of searchable text - at least in the case of PDFs. Is there a way to convert from … --- ## [Help with config file for given syntax](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304)
**Author:** [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\ **Replies:** 8\ **Last updated:** [March 25, 2019, 4:34pm UTC](https://discuss.elastic.co/t/help-with-config-file-for-given-syntax/173304 "2019-03-25T16:34:24Z")
i have a file with data in the format as given below, i have tried several times to index it using logstash conf file but failed. Please help me how can i index in two cases: 1: all key value pairs 2: some key value pa… --- ## [Filter Logs from Firewall](https://discuss.elastic.co/t/filter-logs-from-firewall/172494)
**Author:** [@david-vazquez](https://discuss.elastic.co/u/david-vazquez)\ **Replies:** 3\ **Last updated:** [March 25, 2019, 3:50pm UTC](https://discuss.elastic.co/t/filter-logs-from-firewall/172494 "2019-03-25T15:50:16Z")
Hello everybody! Im new using ELK, so I have some doubts in some stuff I´m trying to do. I´m sending logs from my Firewall to a Rsyslog server. I was sending logs throught Filebeat,but when I´m visualizing data in Kiba… --- ## [How to filter “select” queries](https://discuss.elastic.co/t/how-to-filter-select-queries/173766)
**Author:** [@angham](https://discuss.elastic.co/u/angham)\ **Replies:** 0\ **Last updated:** [March 25, 2019, 1:55pm UTC](https://discuss.elastic.co/t/how-to-filter-select-queries/173766 "2019-03-25T13:55:57Z")
How to filter “select” queries from a log file that contains over 30000 lines this is my log file "15:49:05,479 DEBUG \[org.hibernate.stat.internal.ConcurrentStatisticsImpl\] (default task-43) HHH000117: HQL: select T fr… --- ## [Encountered mapper\_parsing\_exception when working through LogStash tutorial](https://discuss.elastic.co/t/encountered-mapper-parsing-exception-when-working-through-logstash-tutorial/173746)
**Author:** [@Miao](https://discuss.elastic.co/u/Miao)\ **Replies:** 2\ **Last updated:** [March 25, 2019, 12:35pm UTC](https://discuss.elastic.co/t/encountered-mapper-parsing-exception-when-working-through-logstash-tutorial/173746 "2019-03-25T12:35:55Z")
I recently downloaded LogStash 6.6.2, and I am currently working through the tutorial here. I created a first-pipeline.conf file inside my Logstash directory: input { beats { port =\> "5044" } } filter {… --- ## [message=\>"Could not execute action: PipelineAction::Create\, action\_result: false", :backtrace=\>nil](https://discuss.elastic.co/t/message-could-not-execute-action-pipelineaction-create-main-action-result-false-backtrace-nil/171284)
**Author:** [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\ **Replies:** 4\ **Last updated:** [March 25, 2019, 11:35am UTC](https://discuss.elastic.co/t/message-could-not-execute-action-pipelineaction-create-main-action-result-false-backtrace-nil/171284 "2019-03-25T11:35:02Z")
trying to parse a file with logstash.conf present in /usr/local/cellar/logstash/6.6.1/libexec/config directory, and i am running the command logstash -f logstash.conf from bin folder. the logstash.conf file contents are… --- ## [Trouble parsing msgpack input, falling back to plain text](https://discuss.elastic.co/t/trouble-parsing-msgpack-input-falling-back-to-plain-text/173733)
**Author:** [@bjk](https://discuss.elastic.co/u/bjk)\ **Replies:** 0\ **Last updated:** [March 25, 2019, 11:01am UTC](https://discuss.elastic.co/t/trouble-parsing-msgpack-input-falling-back-to-plain-text/173733 "2019-03-25T11:01:49Z")
The data coming in through kafka is encoded in msgpack so I try to unpack it using codec. However, the following error occurs. config input { kafka { zk\_connect =\> "localhost:2181" topic\_id =\> "topic" group\_id =\> … --- ## [Logrotate](https://discuss.elastic.co/t/logrotate/173643)
**Author:** [@praveen1](https://discuss.elastic.co/u/praveen1)\ **Replies:** 4\ **Last updated:** [March 25, 2019, 10:40am UTC](https://discuss.elastic.co/t/logrotate/173643 "2019-03-25T10:40:07Z")
Hi Teams, can anyone help me in configuring logrotate in "version" : {"number" : "6.1.1", to reduce my Disk size, I want to only keep 7 days logs and in zipped format. i could see these 2 file in logstash file: log… --- ## [Jdbc\_static lookups returns more than one row](https://discuss.elastic.co/t/jdbc-static-lookups-returns-more-than-one-row/173724)
**Author:** [@chaire](https://discuss.elastic.co/u/chaire)\ **Replies:** 0\ **Last updated:** [March 25, 2019, 10:32am UTC](https://discuss.elastic.co/t/jdbc-static-lookups-returns-more-than-one-row/173724 "2019-03-25T10:32:31Z")
I'm using jdbc\_static filter to enrich data from another log. My problem is that my lookups returns more than one row. When I tried accessing the field with something like "%{\[server\]\[0\]\[description\]}" it didn't get the … --- ## [Prevent plugin restart](https://discuss.elastic.co/t/prevent-plugin-restart/173720)
**Author:** [@Rhys\_Watkin](https://discuss.elastic.co/u/Rhys_Watkin)\ **Replies:** 0\ **Last updated:** [March 25, 2019, 10:16am UTC](https://discuss.elastic.co/t/prevent-plugin-restart/173720 "2019-03-25T10:16:52Z")
Use case: Logstash ingest from JDBC but misconfigured with the wrong credentials. If it fails, it'll fail forever so there's no point in Logstash saying A plugin had an unrecoverable error. Will restart this plugin. (I… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=577) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=579)