# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=741 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 742 --- ## [LS agent successfully started but ES index not created](https://discuss.elastic.co/t/ls-agent-successfully-started-but-es-index-not-created/135658)
**Author:** [@Ambuj\_Saxena](https://discuss.elastic.co/u/Ambuj_Saxena)\ **Replies:** 7\ **Last updated:** [June 14, 2018, 6:44am UTC](https://discuss.elastic.co/t/ls-agent-successfully-started-but-es-index-not-created/135658 "2018-06-14T06:44:38Z")
Hi, I am new here. The first time I ran through the LS guide, first time I push to all data into elasticsearch but my all data point are string & can't do much about it. So delete the index page and try again this time.… --- ## [How to use grok pattern file in logstash filter](https://discuss.elastic.co/t/how-to-use-grok-pattern-file-in-logstash-filter/135129)
**Author:** [@abhishek007](https://discuss.elastic.co/u/abhishek007)\ **Replies:** 3\ **Last updated:** [June 14, 2018, 6:33am UTC](https://discuss.elastic.co/t/how-to-use-grok-pattern-file-in-logstash-filter/135129 "2018-06-14T06:33:40Z")
Hi, I have log with this pattern. 07/06/2018 09:42:39.607,T,someString,pool-9-thread-4,D:007,1528335759607,120 I want to write grok pattern using grok debugger, but could not succeed . I found one grok pattern file o… --- ## [Logstash won't read file?](https://discuss.elastic.co/t/logstash-wont-read-file/135813)
**Author:** [@reswob](https://discuss.elastic.co/u/reswob)\ **Replies:** 1\ **Last updated:** [June 14, 2018, 5:47am UTC](https://discuss.elastic.co/t/logstash-wont-read-file/135813 "2018-06-14T05:47:10Z")
I'm playing with the some logs files I ..... found on the internet, which comes in JSON files. So I'm using LS to ingest them and send them to ES. But LS won't read them and I'm having a really hard time diagnosing th… --- ## [Invalid field format when writing data to influxdb from csv file using logstash-output-plugin](https://discuss.elastic.co/t/invalid-field-format-when-writing-data-to-influxdb-from-csv-file-using-logstash-output-plugin/135829)
**Author:** [@ebueno](https://discuss.elastic.co/u/ebueno)\ **Replies:** 0\ **Last updated:** [June 14, 2018, 5:07am UTC](https://discuss.elastic.co/t/invalid-field-format-when-writing-data-to-influxdb-from-csv-file-using-logstash-output-plugin/135829 "2018-06-14T05:07:24Z")
Hi there! I am trying to get vmstat output (in CSV format) and load the data into influxdb using logstash and logstash-output-plugin, but am getting error "invalid field format" when writing data to influxdb. See detail… --- ## [Logstash stopped running after first run](https://discuss.elastic.co/t/logstash-stopped-running-after-first-run/135782)
**Author:** [@mc1392](https://discuss.elastic.co/u/mc1392)\ **Replies:** 6\ **Last updated:** [June 13, 2018, 11:46pm UTC](https://discuss.elastic.co/t/logstash-stopped-running-after-first-run/135782 "2018-06-13T23:46:04Z")
The data loaded the first time, not sure what happened. I get this notification: \[2018-06-13T14:28:34,944\]\[INFO \]\[logstash.agent \] Successfully started Logstash API endpoint {:port=\>9600} Then it just sits… --- ## [Mutate+copy of @timestamp gets NullPointerException](https://discuss.elastic.co/t/mutate-copy-of-timestamp-gets-nullpointerexception/135033)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 1\ **Last updated:** [June 13, 2018, 10:54pm UTC](https://discuss.elastic.co/t/mutate-copy-of-timestamp-gets-nullpointerexception/135033 "2018-06-13T22:54:25Z")
In 6.2.4, when I try to do a mutate+copy of @timestamp it blows up with a NullPointerException. Is there another way to do this? input { generator { message =\> '2018/05/11 12:34:56' count =\> 1 } } filter { date { ma… --- ## [Best Way to Combine chain two output plugins](https://discuss.elastic.co/t/best-way-to-combine-chain-two-output-plugins/135428)
**Author:** [@adrianz](https://discuss.elastic.co/u/adrianz)\ **Replies:** 1\ **Last updated:** [June 13, 2018, 10:36pm UTC](https://discuss.elastic.co/t/best-way-to-combine-chain-two-output-plugins/135428 "2018-06-13T22:36:22Z")
Hello, I have device data currently stored in ElasticSearch, what I need to do is to export the data using Logstash (I'm using the elasticsearch input plugin) do some formatting ETL (extract days into seperate fields) an… --- ## [From twitter input to Splunk HTTP Event collector](https://discuss.elastic.co/t/from-twitter-input-to-splunk-http-event-collector/135709)
**Author:** [@drosys](https://discuss.elastic.co/u/drosys)\ **Replies:** 1\ **Last updated:** [June 13, 2018, 10:02pm UTC](https://discuss.elastic.co/t/from-twitter-input-to-splunk-http-event-collector/135709 "2018-06-13T22:02:17Z")
Dear All, I am trying to send twitter input data to HTTP Event collector of Splunk. The main issue is that twitter input plugin generate a JSON object which needs to be modified to become a "value" of the key "event". … --- ## [Numbers below zero indexed as text](https://discuss.elastic.co/t/numbers-below-zero-indexed-as-text/134035)
**Author:** [@Peter\_Boer](https://discuss.elastic.co/u/Peter_Boer)\ **Replies:** 1\ **Last updated:** [June 13, 2018, 8:29pm UTC](https://discuss.elastic.co/t/numbers-below-zero-indexed-as-text/134035 "2018-06-13T20:29:54Z")
When indexing data with numbers below zero the fields get classified as text instead of numbers. Now I cannot make value based graphs in Timelion. This json object is indexed correctly: {"sensorId": 1, "values": {"gyro… --- ## [Many http error 429s in Logstash](https://discuss.elastic.co/t/many-http-error-429s-in-logstash/135439)
**Author:** [@jbrack](https://discuss.elastic.co/u/jbrack)\ **Replies:** 3\ **Last updated:** [June 13, 2018, 6:44pm UTC](https://discuss.elastic.co/t/many-http-error-429s-in-logstash/135439 "2018-06-13T18:44:33Z")
I'm a bit new to ELK but have a environment that sends high-volume and bursty logs in the following configuration: Fluentd (around 15 instances) -\> (via HTTP) -\> LogStash (1 containerized instance) -\> AWS ElasticSearch … --- ## [How to extract index name from the uri field in logstash](https://discuss.elastic.co/t/how-to-extract-index-name-from-the-uri-field-in-logstash/135475)
**Author:** [@Santu](https://discuss.elastic.co/u/Santu)\ **Replies:** 16\ **Last updated:** [June 13, 2018, 6:00pm UTC](https://discuss.elastic.co/t/how-to-extract-index-name-from-the-uri-field-in-logstash/135475 "2018-06-13T18:00:14Z")
Hi, the api and index fields are not being populated in the request logs. for this i am using the custom pattern grok rules to extract the index and api values from the uri field like below. some how it doesn't works… --- ## [Assign default value to a field if no value is present](https://discuss.elastic.co/t/assign-default-value-to-a-field-if-no-value-is-present/135753)
**Author:** [@Hanciv](https://discuss.elastic.co/u/Hanciv)\ **Replies:** 2\ **Last updated:** [June 13, 2018, 4:47pm UTC](https://discuss.elastic.co/t/assign-default-value-to-a-field-if-no-value-is-present/135753 "2018-06-13T16:47:52Z")
Hi, I am reading a log message with KV pairs in logstash and sending them over to Elastic Search. If some of the fields dont have value, i want to assing a default value like "N/A" instead of sending them as "%FIELD\_NAM… --- ## [Flatten the json array into field in filter of logstash](https://discuss.elastic.co/t/flatten-the-json-array-into-field-in-filter-of-logstash/135750)
**Author:** [@5b0eafdfbfb313f6ed17](https://discuss.elastic.co/u/5b0eafdfbfb313f6ed17)\ **Replies:** 1\ **Last updated:** [June 13, 2018, 4:43pm UTC](https://discuss.elastic.co/t/flatten-the-json-array-into-field-in-filter-of-logstash/135750 "2018-06-13T16:43:54Z")
Now I have the filed below in Kibana. { "steps": \[ { "stepName": "\*\*\*\*", "values": { "type": "\*\*", "startTime": "\*\*", "total": 0 } }, { "st… --- ## [Logstash-ouput-zabbix - zabbix\_sender](https://discuss.elastic.co/t/logstash-ouput-zabbix-zabbix-sender/135764)
**Author:** [@vdwygaep](https://discuss.elastic.co/u/vdwygaep)\ **Replies:** 0\ **Last updated:** [June 13, 2018, 4:24pm UTC](https://discuss.elastic.co/t/logstash-ouput-zabbix-zabbix-sender/135764 "2018-06-13T16:24:03Z")
Is there already a solution for the zabbix\_sender issue? When Logstash Node is not the zabbix data/event node zabbix\_sender will fail with message from zabbix server/proxy "...rejected all items sent...". Running (All … --- ## [Set date time transformations from date filter](https://discuss.elastic.co/t/set-date-time-transformations-from-date-filter/135759)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 3\ **Last updated:** [June 13, 2018, 4:18pm UTC](https://discuss.elastic.co/t/set-date-time-transformations-from-date-filter/135759 "2018-06-13T16:18:45Z")
Here is a (partial) syslog entry (ignore IP validity they have been scrambled): Jun 15 00:51:19 139.133.7.190 (squid-1): src="172.16.0.51" src\_port="49530" dst="122.355.89.1" dst\_port="80" local\_time="15/Jun/2015:00:51:… --- ## [CSV not parsing logs](https://discuss.elastic.co/t/csv-not-parsing-logs/135605)
**Author:** [@jkevan](https://discuss.elastic.co/u/jkevan)\ **Replies:** 6\ **Last updated:** [June 13, 2018, 3:43pm UTC](https://discuss.elastic.co/t/csv-not-parsing-logs/135605 "2018-06-13T15:43:51Z")
Hi there, I'm new to the elastic stack, so please forgive me if i ask some obvious questions. I currently have logstash collecting logs from our palo alto firewall and im not able to see any new fields pop up from the f… --- ## [Verify working logstash installation](https://discuss.elastic.co/t/verify-working-logstash-installation/135737)
**Author:** [@heskez](https://discuss.elastic.co/u/heskez)\ **Replies:** 9\ **Last updated:** [June 13, 2018, 3:34pm UTC](https://discuss.elastic.co/t/verify-working-logstash-installation/135737 "2018-06-13T15:34:33Z")
Hi there, I ran a logstash instance from April and it was working. For some reason now it has stopped. Restarting all ELK services doesn't solve the problem. Is there a way to troubleshoot this? What are the steps? To ma… --- ## [Conversion two string to date](https://discuss.elastic.co/t/conversion-two-string-to-date/135739)
**Author:** [@ahmed\_charafouddine](https://discuss.elastic.co/u/ahmed_charafouddine)\ **Replies:** 3\ **Last updated:** [June 13, 2018, 2:26pm UTC](https://discuss.elastic.co/t/conversion-two-string-to-date/135739 "2018-06-13T14:26:08Z")
Hello everyone, Having two cords: date = 201808612 hour = 1234 is there a way to replace it with the @timestamp with the logstash date module. Thanks --- ## [Logstash Configuration Error -\*\*Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 14](https://discuss.elastic.co/t/logstash-configuration-error-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-at-line-14/135717)
**Author:** [@priya1](https://discuss.elastic.co/u/priya1)\ **Replies:** 2\ **Last updated:** [June 13, 2018, 1:39pm UTC](https://discuss.elastic.co/t/logstash-configuration-error-failed-to-execute-action-action-logstash-create-pipeline-id-main-exception-logstash-configurationerror-message-expected-one-of-at-line-14/135717 "2018-06-13T13:39:51Z")
Hi Team, Please find the details below, Code : input{ file { path =\> "C:\\Users\\priya\_shukla\\Desktop\\SAPLogsERR\\test2.log" start\_position =\> "beginning" sincedb\_path =\> "C:\\Users\\priya\_shukla\\Downloads\\logstash-6.2.… --- ## [Split filter](https://discuss.elastic.co/t/split-filter/135719)
**Author:** [@ahmed\_charafouddine](https://discuss.elastic.co/u/ahmed_charafouddine)\ **Replies:** 3\ **Last updated:** [June 13, 2018, 1:26pm UTC](https://discuss.elastic.co/t/split-filter/135719 "2018-06-13T13:26:10Z")
Hello everyone, How could I cut a field in a sentence? I looked, there is a filter split module but I do not know how I can use it. Example: path =\> h2dd\_AMR\_20180612-2357.log what want: fields\_1: ​​AMR date: 201806… --- ## [JMX input not starting](https://discuss.elastic.co/t/jmx-input-not-starting/135676)
**Author:** [@Amrit\_Jangid](https://discuss.elastic.co/u/Amrit_Jangid)\ **Replies:** 5\ **Last updated:** [June 13, 2018, 12:18pm UTC](https://discuss.elastic.co/t/jmx-input-not-starting/135676 "2018-06-13T12:18:00Z")
Trying to read Kafka JMX metrics data into Logstash, Here are the config- $ cat jmx.conf input { jmx { path =\> "/home/ubuntu/dev/logstash-6.2.4/conf" polling\_frequency =\> 10 type =\> "jmx" nb\_thread =\> … --- ## [Logstash date filter performs a timeshift of 3 hours](https://discuss.elastic.co/t/logstash-date-filter-performs-a-timeshift-of-3-hours/135704)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 1\ **Last updated:** [June 13, 2018, 11:56am UTC](https://discuss.elastic.co/t/logstash-date-filter-performs-a-timeshift-of-3-hours/135704 "2018-06-13T11:56:32Z")
Filtering some syslog logs (which are actually stored in a file as follows): grok { match =\> { 'message' =\> '%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:\\\[%{POSINT:syslog… --- ## [Logstash log4j2 plugins installation](https://discuss.elastic.co/t/logstash-log4j2-plugins-installation/133839)
**Author:** [@Ankita\_Ghate](https://discuss.elastic.co/u/Ankita_Ghate)\ **Replies:** 6\ **Last updated:** [June 13, 2018, 11:03am UTC](https://discuss.elastic.co/t/logstash-log4j2-plugins-installation/133839 "2018-06-13T11:03:59Z")
while starting logstash, I am getting below error, 83\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"6.2.2"} \[2018-05-30T09:14:47,157\]\[INFO \]\[logstash.agent \] Successfully started … --- ## [Docker Logstash to Elastic: connection refused](https://discuss.elastic.co/t/docker-logstash-to-elastic-connection-refused/135583)
**Author:** [@Adrien\_Renaud](https://discuss.elastic.co/u/Adrien_Renaud)\ **Replies:** 2\ **Last updated:** [June 13, 2018, 10:27am UTC](https://discuss.elastic.co/t/docker-logstash-to-elastic-connection-refused/135583 "2018-06-13T10:27:04Z")
I use the following official dockers: docker run -d -p 9200:9200 -p 9300:9300 --name myel \\ -e "discovery.type=single-node" \\ -e "XPACK\_SECURITY\_ENABLED=false" \\ -e "XPACK\_REPORTING\_E… --- ## [Intermittent ConnectionFailed errors from ElasticSearch filter](https://discuss.elastic.co/t/intermittent-connectionfailed-errors-from-elasticsearch-filter/135652)
**Author:** [@edball](https://discuss.elastic.co/u/edball)\ **Replies:** 0\ **Last updated:** [June 13, 2018, 8:36am UTC](https://discuss.elastic.co/t/intermittent-connectionfailed-errors-from-elasticsearch-filter/135652 "2018-06-13T08:36:59Z")
Hi, I've set up a Logstash pipeline with a filebeat input, a couple of ElasticSearch filters to retrieve previous events and an ElasticSearch ouput. I am getting the following errors intermittently in the logs: \[2018-0… --- ## [Unable to send all packets to elasticsearch through logstash](https://discuss.elastic.co/t/unable-to-send-all-packets-to-elasticsearch-through-logstash/135324)
**Author:** [@sahara](https://discuss.elastic.co/u/sahara)\ **Replies:** 2\ **Last updated:** [June 13, 2018, 7:46am UTC](https://discuss.elastic.co/t/unable-to-send-all-packets-to-elasticsearch-through-logstash/135324 "2018-06-13T07:46:12Z")
i am capturing packets through tshark and send events to logstash through filebeat. My logstash sends only required data after filtering to elasticsearch but i am not receiving all the events at elasticsearch. what are t… --- ## [Filebeat trans logstash for log4j error](https://discuss.elastic.co/t/filebeat-trans-logstash-for-log4j-error/135320)
**Author:** [@flanet](https://discuss.elastic.co/u/flanet)\ **Replies:** 6\ **Last updated:** [June 13, 2018, 6:17am UTC](https://discuss.elastic.co/t/filebeat-trans-logstash-for-log4j-error/135320 "2018-06-13T06:17:49Z")
\[root@localhost logstash-6.2.4\]# ./bin/logstash -f config/log4j-es.conf Sending Logstash's logs to /usr/logstash/logstash-6.2.4/logs which is now configured via log4j2.properties \[{"message\_field"=\>{"path\_match"=\>"mes… --- ## [How to completely disable logstash logging?](https://discuss.elastic.co/t/how-to-completely-disable-logstash-logging/135553)
**Author:** [@Rahulkumar](https://discuss.elastic.co/u/Rahulkumar)\ **Replies:** 7\ **Last updated:** [June 13, 2018, 6:12am UTC](https://discuss.elastic.co/t/how-to-completely-disable-logstash-logging/135553 "2018-06-13T06:12:48Z")
Hi, I am using Linux system. I do not want logstash to write any log either under location "/usr/share/logstash/logs" or under "/var/log/logstash". I want to completely disable all type of logging for logstash. Pleas… --- ## [How to define cloud id in logstash config file?](https://discuss.elastic.co/t/how-to-define-cloud-id-in-logstash-config-file/135584)
**Author:** [@fcukinyahoo](https://discuss.elastic.co/u/fcukinyahoo)\ **Replies:** 1\ **Last updated:** [June 12, 2018, 9:06pm UTC](https://discuss.elastic.co/t/how-to-define-cloud-id-in-logstash-config-file/135584 "2018-06-12T21:06:39Z")
What I am trying to accomplish is execute sql query against our mssql database, pass that data to ES stack in cloud. To my understanding I have to take following actions, install jdk on windows client where i ll be sen… --- ## [Grok pattern creation](https://discuss.elastic.co/t/grok-pattern-creation/135467)
**Author:** [@hitman](https://discuss.elastic.co/u/hitman)\ **Replies:** 2\ **Last updated:** [June 12, 2018, 8:03pm UTC](https://discuss.elastic.co/t/grok-pattern-creation/135467 "2018-06-12T20:03:16Z")
2018-05-25 10:53:15.8779 fucntionality:nil request:enter user\_id:3623 school\_id:4 result:success username:(('Reka',),) 2018-05-25 10:53:15.8888 functionality:nil request:enter user\_id:3648 school\_id:8 result:success use… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=740) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=742)