# Running logstash properly

**URL:** <https://discuss.elastic.co/t/running-logstash-properly/124020>\
**Category:** Logstash\
**Created:** [March 15, 2018, 3:30am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020 "2018-03-15T03:30:15Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 15, 2018, 3:30am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/1 "2018-03-15T03:30:15Z")

</div>

I have installed my logstash using rpm and enabled it using systemctl. I have to stash a csv file and create table in grafana daily. Do i have to run bin/logstash -f /etc/logstash/conf.d/logstash.config via cron if i want to stash sa csv file hourly?. I am not seeing any content my grafana when i am about to add a table panel.

![2018-03-15_12-12-29](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bfeae7e66bd25a74242d5ca371e5f20131912f0f.jpg)

index are getting created.

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size  
green open .kibana \_G9Bg9S8SruqNG12CBrU2g 1 0 2 1 16kb 16kb  
yellow open implementationcrq-2018.03.15 H7B50zQZQsuwHsx5XHK2pA 5 1 43 0 88.4kb 88.4kb  
yellow open pendingcrq-2018.03.15 Nnb8\_XnVQQmxPMzpBJO9tA 5 1 2 0 19.1kb 19.1kb  
yellow open completedcrq-2018.03.15 5FIT8AxsSfKry22xKEleHQ 5 1 16 0 85.8kb 85.8kb  
yellow open closedcrq-2018.03.15 ecf0hIpZSUCObNR7hDZQKg 5 1 53 0 129.7kb 129.7kb

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 17, 2018, 8:13pm UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/2 "2018-03-17T20:13:44Z")

</div>

You can keep Logstash running all the time and configure a file input to read \*.csv or whatever the files are named. When you want Logstash to process a file just copy it to the directory that Logstash is monitoring.

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 19, 2018, 2:10am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/3 "2018-03-19T02:10:39Z")

</div>

i have created a cron to run my logstash config every 3mins. Is that allowed? in my logs i am seeing this.

[2018-03-18T22:03:11,256][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2018-03-18T22:03:11,262][FATAL][logstash.runner] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.  
[2018-03-18T22:03:11,265][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit  
[2018-03-18T22:06:11,320][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 19, 2018, 2:43am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/4 "2018-03-19T02:43:46Z")

</div>

Should enable systemctl for logstash service rather than running the logstash using cron?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 19, 2018, 7:11am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/5 "2018-03-19T07:11:01Z")

</div>

Yes, run Logstash as a background service instead and follow the advice I gave earlier. Logstash doesn't shut down when it has processed all data via the file input so starting it every three minutes doesn't make sense.

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 19, 2018, 10:01am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/6 "2018-03-19T10:01:49Z")

</div>

Thanks. Follow up question. I am receiving this error in logs "the `type` event field won't be used to determine the document \_type"

what's wrong in my logstash config?

input {  
file {  
type =\> "samscrq"  
path =\> "/samba/CRQ\_reports/samscrq.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_samscrq"  
}  
file {  
type =\> "pending"  
path =\> "/samba/CRQ\_reports/pending.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_pending"  
}  
file {  
type =\> "implementation"  
path =\> "/samba/CRQ\_reports/implementation.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_implementation"  
}  
file {  
type =\> "completed"  
path =\> "/samba/CRQ\_reports/completed.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_completed"  
}  
file {  
type =\> "closed"  
path =\> "/samba/CRQ\_reports/closed.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_closed"  
}  
}  
filter {  
if [type] == "samscrq" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "completed" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "closed" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "implementation" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "pending" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
}  
output {  
if [type] == "samscrq" {  
elasticsearch {  
hosts =\> ["146.40.233.10:9200"]  
index =\> "samscrq"  
}  
}  
if [type] == "completed" {  
elasticsearch {  
hosts =\> ["146.40.233.10:9200"]  
index =\> "completed"  
}  
}  
if [type] == "closed" {  
elasticsearch {  
hosts =\> ["146.40.233.10:9200"]  
index =\> "closed"  
}  
}  
if [type] == "implementation" {  
elasticsearch {  
hosts =\> ["146.40.233.10:9200"]  
index =\> "implementation"  
}  
}  
if [type] == "pending" {  
elasticsearch {  
hosts =\> ["146.40.233.10:9200"]  
index =\> "pending"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 19, 2018, 10:42am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/7 "2018-03-19T10:42:08Z")

</div>

It means exactly what it says. The value of the `type` field will no longer by default be used to determine the type of the documents. If this really is what you want you can use

```
document_type => "%{type}"

```

in your elasticsearch output configuration. Here's the relevant code:

> <https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/v9.0.3/lib/logstash/outputs/elasticsearch/common.rb#L208-L225>

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 19, 2018, 11:09am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/8 "2018-03-19T11:09:17Z")

</div>

what should be my correct approach? this is the scenario. every 2 hours a CSV file is dumped in my samba directory. I have a python scrupt parses the CSV file and create a new CSV file (pending.csv, completed.csv and implementation.csv). From this new input files i would like to create 3 different indeces that i will be using for my table panel in grafana. I want to display this in my dashboard with table panels.

 ![2018-03-19_19-08-06](https://us1.discourse-cdn.com/elastic/original/3X/7/5/7565f83b89d3db0200ab001c3a2f6ba6f63cba7a.jpg)

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 19, 2018, 11:23am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/9 "2018-03-19T11:23:31Z")

</div>

i have update i used type instead of type. but still same error "the type event field won't be used to determine the document \_type"

input {  
file {  
type =\> "samscrq"  
path =\> "/samba/CRQ\_reports/samscrq.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_samscrq"  
}  
file {  
type =\> "pending"  
path =\> "/samba/CRQ\_reports/pending.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_pending"  
}  
file {  
type =\> "implementation"  
path =\> "/samba/CRQ\_reports/implementation.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_implementation"  
}  
file {  
type =\> "completed"  
path =\> "/samba/CRQ\_reports/completed.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_completed"  
}  
file {  
type =\> "closed"  
path =\> "/samba/CRQ\_reports/closed.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_closed"  
}  
}  
filter {  
if [type] == "samscrq" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "completed" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "closed" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "implementation" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
if [type] == "pending" {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","Status","start\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
}  
}  
output {  
if [type] == "samscrq" {  
elasticsearch {  
hosts =\> ["XXX.XX.XXX.10:9200"]  
index =\> "samscrq"  
}  
}  
if [type] == "completed" {  
elasticsearch {  
hosts =\> ["XXX.XX.XXX.10:9200"]  
index =\> "completed"  
}  
}  
if [type] == "closed" {  
elasticsearch {  
hosts =\> ["XXX.XX.XXX.10:9200"]  
index =\> "closed"  
}  
}  
if [type] == "implementation" {  
elasticsearch {  
hosts =\> ["XXX.XX.XXX.10:9200"]  
index =\> "implementation"  
}  
}  
if [type] == "pending" {  
elasticsearch {  
hosts =\> ["XXX.XX.XXX.10:9200"]  
index =\> "pending"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 19, 2018, 12:03pm UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/10 "2018-03-19T12:03:30Z")

</div>

> i have update i used type instead of type.

What?

> but still same error "the type event field won't be used to determine the document \_type"

It's not an error, it's a warning. If you don't want to see the warning you can set `document_type` to something.

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 19, 2018, 12:13pm UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/11 "2018-03-19T12:13:02Z")

</div>

I see. I thought its already an error and config was not good at all.

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 19, 2018, 12:14pm UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/12 "2018-03-19T12:14:25Z")

</div>

> [@magnusbaeck](#):
>
> What

The config earlier was using tag it was supposedly type 🙂 sorry for that

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [March 19, 2018, 9:26pm UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/13 "2018-03-19T21:26:06Z")

</div>

What is your python script doing to the incoming CSV file? Is it using the `Status` field to decide which file to write the line to?  
If so, then Logstash can do all of what you want to do in a single pass of the incoming file.  
I am sure you can use Logstash functions to send each event to the [correct index on-the-fly using string interpolation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index).  
Perhaps:

```auto
input {
  file {
    type => "samscrq"
    path => "/samba/CRQ_reports/incoming.csv"
    start_position => "beginning"
    sincedb_path => "/opt/sincedb/.sincedb_crq"
  }
}
filter {
  csv {
    separator => ","
    columns => ["change_id","summary","status","start_date","coordinator"]
    skip_empty_columns => true
    skip_empty_rows => true
  }
  mutate {
    lowercase => ["status"]
  }
}
output {
  elasticsearch {
    hosts => ["XXX.XX.XXX.10:9200"]
    index => "%{status}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 20, 2018, 1:40am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/14 "2018-03-20T01:40:49Z")

</div>

yes it is using the status field to decide which to write the line to. I'll try to use this logstash config. I'll let you know of the outcome. Thanks

---

<div class="post-metadata">

**Author:** ![retxedue](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@retxedue](https://discuss.elastic.co/u/retxedue)\
**Post date:** [March 20, 2018, 1:53am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/15 "2018-03-20T01:53:53Z")

</div>

I tried running below config. see below pic i am seeing this type of indeces.

input {  
file {  
type =\> "samscrq"  
path =\> "/samba/CRQ\_reports/\*.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/opt/sincedb/.sincedb\_crq"  
}  
}  
filter {  
csv {  
separator =\> ","  
columns =\> ["change\_id","summary","status","start\_date","end\_date","coordinator"]  
skip\_empty\_columns =\> true  
skip\_empty\_rows =\> true  
}  
mutate {  
lowercase =\> ["status"]  
}  
}  
output {  
elasticsearch {  
hosts =\> ["XXX.XX.XXX.XX:9200"]  
index =\> "%{status}"  
}  
}

 ![2018-03-19_19-08-06](https://us1.discourse-cdn.com/elastic/original/3X/f/8/f847cb6c6f2de1c719743e6851006e6c56cb8dcb.jpg)

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [March 20, 2018, 3:32am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/16 "2018-03-20T03:32:36Z")

</div>

If you replace the elastics watch output with  
stout { codec =\> { rubydebug }}  
What does one event look like, copy paste it here.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2018, 3:32am UTC](https://discuss.elastic.co/t/running-logstash-properly/124020/17 "2018-04-17T03:32:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.