# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=786 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 787 --- ## [Logstash Connectivity Error to Elasticsearch](https://discuss.elastic.co/t/logstash-connectivity-error-to-elasticsearch/126940)
**Author:** [@kumehta1998](https://discuss.elastic.co/u/kumehta1998)\ **Replies:** 1\ **Last updated:** [April 5, 2018, 5:42pm UTC](https://discuss.elastic.co/t/logstash-connectivity-error-to-elasticsearch/126940 "2018-04-05T17:42:52Z")
\[WARN \]\[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"http://192.168.111.33:9200/", :error\_type=\>LogStash::Outputs::ElasticSearch::HttpClient::Pool::Host… --- ## [Aggregate - concatenate events](https://discuss.elastic.co/t/aggregate-concatenate-events/126808)
**Author:** [@las](https://discuss.elastic.co/u/las)\ **Replies:** 5\ **Last updated:** [April 5, 2018, 3:39pm UTC](https://discuss.elastic.co/t/aggregate-concatenate-events/126808 "2018-04-05T15:39:11Z")
12:12:12 processed T1 data1 val1 data2 val2 dataId 1111 data7 val7 I have configured filebeat to send multiline events like above to be sent to logstash. In logstash, I am using grok and aggregate to extract the da… --- ## [Why is Logstash stalling for so long?](https://discuss.elastic.co/t/why-is-logstash-stalling-for-so-long/126951)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 0\ **Last updated:** [April 5, 2018, 2:34pm UTC](https://discuss.elastic.co/t/why-is-logstash-stalling-for-so-long/126951 "2018-04-05T14:34:09Z")
I have a pipeline set up to an S3 bucket that has ~700k files in it. When I restart a pipeline (I have auto-restart enabled) it takes hours to restart. Looking at Logstash's logs I see huge blocks of \[2018-04-05T14:30:2… --- ## [Use logstash as proxy for elasticsearch](https://discuss.elastic.co/t/use-logstash-as-proxy-for-elasticsearch/125980)
**Author:** [@tarp](https://discuss.elastic.co/u/tarp)\ **Replies:** 2\ **Last updated:** [April 5, 2018, 2:05pm UTC](https://discuss.elastic.co/t/use-logstash-as-proxy-for-elasticsearch/125980 "2018-04-05T14:05:39Z")
Hi, I deployed some proxy servers that run logstash and nginx. I do not want developers or users hitting the elasticsearch cluster directly. Logstash provides an input to handle JSON text that logstash turns around a… --- ## [Cross correlation between multiple index - logstash filter](https://discuss.elastic.co/t/cross-correlation-between-multiple-index-logstash-filter/126933)
**Author:** [@suchand](https://discuss.elastic.co/u/suchand)\ **Replies:** 0\ **Last updated:** [April 5, 2018, 1:08pm UTC](https://discuss.elastic.co/t/cross-correlation-between-multiple-index-logstash-filter/126933 "2018-04-05T13:08:16Z")
Hi, Please help me to achieve the following condition. If (SourceIP of (Winlogbeat – Login as Admin AND Winlogbeat Privilege Escalation) = Same SourceIP - say X) AND (Output of the Denied or Allowed Outbound Malicious… --- ## [How to check column quantity in a filter?](https://discuss.elastic.co/t/how-to-check-column-quantity-in-a-filter/126757)
**Author:** [@DanV](https://discuss.elastic.co/u/DanV)\ **Replies:** 2\ **Last updated:** [April 5, 2018, 12:58pm UTC](https://discuss.elastic.co/t/how-to-check-column-quantity-in-a-filter/126757 "2018-04-05T12:58:31Z")
Hello, I'm receiving logs from a firewall and i want to sort the different types of logs based on the quantity of columns (columns created using cvs with autogenerate on) they have, instead of doing it based on the colu… --- ## [Sincedb\_path is not working in CentOs](https://discuss.elastic.co/t/sincedb-path-is-not-working-in-centos/126725)
**Author:** [@Sunil\_Villuri](https://discuss.elastic.co/u/Sunil_Villuri)\ **Replies:** 6\ **Last updated:** [April 5, 2018, 12:08pm UTC](https://discuss.elastic.co/t/sincedb-path-is-not-working-in-centos/126725 "2018-04-05T12:08:09Z")
Hi All, I want to re-read the same file again from beginning in CentOs which was installed by YUM. I used the below input configuration in windows (it's working Fine ) Where as in CentOs (its not working ). i was not ab… --- ## [Error: mongodb.jdbc.MongoDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc\_driver\_library?](https://discuss.elastic.co/t/error-mongodb-jdbc-mongodriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/126896)
**Author:** [@Shubham\_Goyal](https://discuss.elastic.co/u/Shubham_Goyal)\ **Replies:** 0\ **Last updated:** [April 5, 2018, 10:22am UTC](https://discuss.elastic.co/t/error-mongodb-jdbc-mongodriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/126896 "2018-04-05T10:22:27Z")
I am using following logstash config: input{ jdbc{ jdbc\_driver\_library =\> "/home/UnityJDBC/mongodb\_unityjdbc\_full.jar" jdbc\_driver\_class =\> "mongodb.jdbc.MongoDriver" jdbc\_connection\_string =\> "jdbc:mongodb://localh… --- ## [LogStash::PipelineReporter](https://discuss.elastic.co/t/logstash-pipelinereporter/126868)
**Author:** [@daeuke](https://discuss.elastic.co/u/daeuke)\ **Replies:** 1\ **Last updated:** [April 5, 2018, 7:54am UTC](https://discuss.elastic.co/t/logstash-pipelinereporter/126868 "2018-04-05T07:54:45Z")
Hi everybody!:smiley: My logstash pipeline got stuck and giving warning logs like below {:timestamp=\>"2018-04-05T13:46:28.970000+0600", :message=\>#\ **Author:** [@joshiegy](https://discuss.elastic.co/u/joshiegy)\ **Replies:** 0\ **Last updated:** [April 5, 2018, 8:17am UTC](https://discuss.elastic.co/t/grok-failes-due-to-aao-in-syslog/126876 "2018-04-05T08:17:09Z") G'day! So I'm trying to parse our syslog at my company. Most lines work but any line with the Swedish/nordic characters Å Ä Ö failes and the log says Grok regexp threw exception {:exception=\>"incompatible encoding reg… --- ## [Logstash shards with unaassigned replicas](https://discuss.elastic.co/t/logstash-shards-with-unaassigned-replicas/126600)
**Author:** [@kostja79](https://discuss.elastic.co/u/kostja79)\ **Replies:** 3\ **Last updated:** [April 5, 2018, 7:36am UTC](https://discuss.elastic.co/t/logstash-shards-with-unaassigned-replicas/126600 "2018-04-05T07:36:41Z")
Hi to all, I have a cluster with only 1 node. However, when I start logstash, it creates several replicas of any shard: GET /\_cat/shards?h=index,shard,prirep,state,unassigned.reason logstash-2018.04.03 … --- ## [Xml to elasticsearch by logstash](https://discuss.elastic.co/t/xml-to-elasticsearch-by-logstash/126859)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 1\ **Last updated:** [April 5, 2018, 7:22am UTC](https://discuss.elastic.co/t/xml-to-elasticsearch-by-logstash/126859 "2018-04-05T07:22:43Z")
Is there anyone to help?? Please give me a example about xml filter.. I want to parse this xml source. \ \ **Author:** [@Arko](https://discuss.elastic.co/u/Arko)\ **Replies:** 4\ **Last updated:** [April 5, 2018, 6:45am UTC](https://discuss.elastic.co/t/how-to-introduce-delay-in-logstash/126813 "2018-04-05T06:45:37Z") I have quite tipical architecture: Filebit -\> Logstash -\> Logstash -\> Elasticsearch First and second logstash makes some sql queries to database during its work (via plugin). But the problem is that I need introduce 1… --- ## [Possible to specify start and end time as part of schedule?](https://discuss.elastic.co/t/possible-to-specify-start-and-end-time-as-part-of-schedule/126747)
**Author:** [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\ **Replies:** 3\ **Last updated:** [April 5, 2018, 6:28am UTC](https://discuss.elastic.co/t/possible-to-specify-start-and-end-time-as-part-of-schedule/126747 "2018-04-05T06:28:07Z")
I am trying to schedule a job that runs every 5 minutes starting at 1 a.m. and ending at 11 p.m. Here's the expression I am planning to use, 0 0/5 1-23 ? \* \* \* I have minimal knowledge in cron but based on the JDBC inp… --- ## [Http\_poller input giving read time out](https://discuss.elastic.co/t/http-poller-input-giving-read-time-out/126809)
**Author:** [@uvarisubash](https://discuss.elastic.co/u/uvarisubash)\ **Replies:** 3\ **Last updated:** [April 5, 2018, 5:58am UTC](https://discuss.elastic.co/t/http-poller-input-giving-read-time-out/126809 "2018-04-05T05:58:42Z")
Http\_poller input giving read time out when data is more 100000. please help me on this { "@timestamp" =\> 2018-04-04T19:17:10.278Z, "tags" =\> \[ \[0\] "\_http\_request\_failure", … --- ## [Netflow Installation on Centos 7](https://discuss.elastic.co/t/netflow-installation-on-centos-7/126843)
**Author:** [@Nazmi\_Shukri](https://discuss.elastic.co/u/Nazmi_Shukri)\ **Replies:** 0\ **Last updated:** [April 5, 2018, 5:45am UTC](https://discuss.elastic.co/t/netflow-installation-on-centos-7/126843 "2018-04-05T05:45:39Z")
Can someone that expert tell me how to setup netflow in centos 7 ? --- ## [Please give me a example about Logstash xml filter](https://discuss.elastic.co/t/please-give-me-a-example-about-logstash-xml-filter/126837)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 4\ **Last updated:** [April 5, 2018, 5:29am UTC](https://discuss.elastic.co/t/please-give-me-a-example-about-logstash-xml-filter/126837 "2018-04-05T05:29:38Z")
Hello everyone. I'm a newbie using a Elastic Stack. I'd like to use xml filter, so I wish get a result format please give me a easy xml filter sample. My xml source is like this : \ **Author:** [@apatel2848](https://discuss.elastic.co/u/apatel2848)\ **Replies:** 13\ **Last updated:** [April 5, 2018, 2:21am UTC](https://discuss.elastic.co/t/fingerprint-aggregate-filter/125979 "2018-04-05T02:21:32Z") How to generate a unique fingerprint for every record retrieved via JDBC plugin? For example, I am retrieving list of invoice line items via a JDBC plugin and using an aggregate filter to group those line items by the in… --- ## [Logstash debugging not writing to logs](https://discuss.elastic.co/t/logstash-debugging-not-writing-to-logs/126613)
**Author:** [@bkelley](https://discuss.elastic.co/u/bkelley)\ **Replies:** 4\ **Last updated:** [April 5, 2018, 12:38am UTC](https://discuss.elastic.co/t/logstash-debugging-not-writing-to-logs/126613 "2018-04-05T00:38:35Z")
Continuing the discussion from Beats input debugging logs not working: I seem to be having the same problem listed above where I change the logger level of a subsystem but no additional logging is written. The only way… --- ## [Update documentation for path.settings?](https://discuss.elastic.co/t/update-documentation-for-path-settings/126815)
**Author:** [@ElasticN00b](https://discuss.elastic.co/u/ElasticN00b)\ **Replies:** 2\ **Last updated:** [April 5, 2018, 12:37am UTC](https://discuss.elastic.co/t/update-documentation-for-path-settings/126815 "2018-04-05T00:37:32Z")
Greetings programs! While I was trying to set up logstash I was running into a fun error: \[2018-04-04T14:19:05,689\]\[INFO \]\[logstash.config.source.local.configpathloader\] No config files found in path {:path=\>"/etc/logs… --- ## [Can not create pipeline while running logstash](https://discuss.elastic.co/t/can-not-create-pipeline-while-running-logstash/126612)
**Author:** [@dpk3d](https://discuss.elastic.co/u/dpk3d)\ **Replies:** 3\ **Last updated:** [April 4, 2018, 10:53pm UTC](https://discuss.elastic.co/t/can-not-create-pipeline-while-running-logstash/126612 "2018-04-04T22:53:22Z")
Hi ELK Community! Hope you people are doing good. I'm getting the below error :- \[2018-04-03T22:20:19,670\]\[ERROR\]\[logstash.agent \] Cannot create pipeli ne {:reason=\>"Expected one of #, { at line 14, column … --- ## [Logstash s3 input plugin with dynamic prefix](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631)
**Author:** [@vikas-prabhakar](https://discuss.elastic.co/u/vikas-prabhakar)\ **Replies:** 3\ **Last updated:** [April 4, 2018, 10:47pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-with-dynamic-prefix/126631 "2018-04-04T22:47:30Z")
Hello, we have a special requirement where we have to read data from s3 bucket in which folders are dynamic(e.g s3bucket/2018/04/04) --- ## [Adding tags](https://discuss.elastic.co/t/adding-tags/126788)
**Author:** [@Salma](https://discuss.elastic.co/u/Salma)\ **Replies:** 2\ **Last updated:** [April 4, 2018, 7:29pm UTC](https://discuss.elastic.co/t/adding-tags/126788 "2018-04-04T19:29:20Z")
Hi, I'm trying to add some tags based on a given array. if the request was equal to one of the values that are on the tags, then I want Logstash to tag it as normal, else abnormal. Following is my code, but unfortunat… --- ## [New to logstash, Need help to fetch the value from the spcecific column, which is array of strings](https://discuss.elastic.co/t/new-to-logstash-need-help-to-fetch-the-value-from-the-spcecific-column-which-is-array-of-strings/125855)
**Author:** [@sridharb](https://discuss.elastic.co/u/sridharb)\ **Replies:** 4\ **Last updated:** [April 4, 2018, 6:50pm UTC](https://discuss.elastic.co/t/new-to-logstash-need-help-to-fetch-the-value-from-the-spcecific-column-which-is-array-of-strings/125855 "2018-04-04T18:50:03Z")
Need help to display values of specific column column18 as fields, column is array of strings Below is the column18: {"accept":\["text/html, image/gif, image/jpeg, \*; q=.2, /; q=.2"\],"host":\["xxx.xx.xxx.com"\],"pragma":\[… --- ## [Aggregation from Elasticsearch filter in Logstash](https://discuss.elastic.co/t/aggregation-from-elasticsearch-filter-in-logstash/126802)
**Author:** [@roblopes](https://discuss.elastic.co/u/roblopes)\ **Replies:** 2\ **Last updated:** [April 4, 2018, 6:47pm UTC](https://discuss.elastic.co/t/aggregation-from-elasticsearch-filter-in-logstash/126802 "2018-04-04T18:47:19Z")
I have a very tiny index (around 10 documents), having some important information, being one of them an attribute called number\_of\_days. What I am trying to do is run a simple query to retrieve document with the bigger … --- ## [Adding a copy field and set it lowercase](https://discuss.elastic.co/t/adding-a-copy-field-and-set-it-lowercase/126762)
**Author:** [@Gabriel\_Pellegrini\_M](https://discuss.elastic.co/u/Gabriel_Pellegrini_M)\ **Replies:** 2\ **Last updated:** [April 4, 2018, 5:38pm UTC](https://discuss.elastic.co/t/adding-a-copy-field-and-set-it-lowercase/126762 "2018-04-04T17:38:29Z")
Hello guys, I'm having a trouble with a mutate filter, where i need to set a field copy to lowercase. But when i try to load on elastic , the field copy won't be lowercase. My Logstash Version is 5.2 i'm using the c… --- ## [Replace default logs type with custom Type](https://discuss.elastic.co/t/replace-default-logs-type-with-custom-type/126791)
**Author:** [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Replies:** 2\ **Last updated:** [April 4, 2018, 5:22pm UTC](https://discuss.elastic.co/t/replace-default-logs-type-with-custom-type/126791 "2018-04-04T17:22:56Z")
In addition to the question I just raised, I would like to retain the type specified when calling Logstash (see curl request) - also within elastic. Right now, If I call logstash using the curl request below, Logstash In… --- ## [Using request\_path to create output filter?](https://discuss.elastic.co/t/using-request-path-to-create-output-filter/126789)
**Author:** [@titan1978](https://discuss.elastic.co/u/titan1978)\ **Replies:** 2\ **Last updated:** [April 4, 2018, 5:22pm UTC](https://discuss.elastic.co/t/using-request-path-to-create-output-filter/126789 "2018-04-04T17:22:03Z")
Hello, I'm using the following configuration to route request to my Elastic Instance. However, it looks like the request\_path header IF check is not passing and thus the elasticsearch instance is not being invoked. Can … --- ## [Extract](https://discuss.elastic.co/t/extract/126754)
**Author:** [@proximator](https://discuss.elastic.co/u/proximator)\ **Replies:** 4\ **Last updated:** [April 4, 2018, 4:31pm UTC](https://discuss.elastic.co/t/extract/126754 "2018-04-04T16:31:08Z")
Hi I would like to build a new field from my error messages using some grok patterns, the idea is to extract the class name from the first line and then from each single line that starts with "Caused by:" In the exampl… --- ## [Processing multiline events from filebeat](https://discuss.elastic.co/t/processing-multiline-events-from-filebeat/126771)
**Author:** [@las](https://discuss.elastic.co/u/las)\ **Replies:** 4\ **Last updated:** [April 4, 2018, 3:38pm UTC](https://discuss.elastic.co/t/processing-multiline-events-from-filebeat/126771 "2018-04-04T15:38:20Z")
Filebeat has been configured to read file and produce multiline events that are sent to logstash. In logstash, I want to combine multiple multiline events using grok and aggregate filters. As part of grok, I also want … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=785) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=787)