[2019-12-22T23:38:10,583][DEBUG][logstash.agent] org/jruby/RubyIO.java:3551:in `select'
/home/mehak/Documents/logstash-7.4.0/vendor/bundle/jruby/2.5.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/home/mehak/Documents/logstash-7.4.0/vendor/bundle/jruby/2.5.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
^C
```
I had to force quit the command as nothing got printed after 'block in run'. What does this command show here?
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 8:48am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/10 "2019-12-23T08:48:13Z")
@Mehak_Bhargava
Please share the sample of stored data in your current index.
Above command will show your logstash log file live as it get updated. Did you change the location of the log file? by it will store inside above directory.
Did you configure index-pattern in filebeat config file?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 9:18am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/11 "2019-12-23T09:18:32Z")
@mancharagopan I have always had the logstash-plain file under that directory. This is all in Ubuntu. So when I downloaded and installed everything, my logstash is under Documents folder.
I did not configure index-patter in filebeat config file. How does that work and why in filebeat?
This is from kibana's dicover.
```auto
Dec 22, 2019 @ 23:30:20.82608/10/2019 12:15:19 579 (null) DEBUG 10 Processing start for record : 136378980959205484#115 ID: 49619767
access
logstash-2019.12.23-000001
Dec 22, 2019 @ 23:30:20.8262019-10-08 12:32:10,480 [Thread-12] INFO c.e.d.s.WorkflowInstanceManager - resumeWorkflowInstance; workflowInstanceId=24935899
dispatch
logstash-2019.12.23-000001
Dec 22, 2019 @ 23:30:20.82908/10/2019 12:15:19 599 (null) DEBUG 10 Get Incident by id for inbound Activity Start : Incident Id (CacheRepositoryProxy:start) = 24748988
access
logstash-2019.12.23-000001
```
This is from GET index/search
```auto
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstash-2019.12.23-000001",
"_type" : "_doc",
"_id" : "yCCpMW8ByDrQo4Z4_MF-",
"_score" : 1.0,
"_source" : {
"ecs" : {
"version" : "1.1.0"
},
"agent" : {
"type" : "filebeat",
"ephemeral_id" : "075b5945-9b1c-4cc4-af4b-b70092f369c4",
"hostname" : "mehak-VirtualBox",
"version" : "7.4.0",
"id" : "bad135c8-d359-4936-b515-79eb4bb24630"
},
"log" : {
"offset" : 846421,
"file" : {
"path" : "/home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/log2.log"
}
},
"tags" : [
"beats_input_codec_plain_applied"
],
"@version" : "1",
"fields" : {
"log_type" : "access"
},
"@timestamp" : "2019-12-23T07:30:20.826Z",
"host" : {
"name" : "mehak-VirtualBox"
},
"message" : "08/10/2019 12:15:19 579 (null) DEBUG 10 Processing start for record : 136378980959205484#115 ID: 49619767",
"type" : "another_test"
}
},
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 9:29am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/12 "2019-12-23T09:29:36Z")
> [@Christian\_Dahlqvist](#):
>
> I suspect this should be `index => "%{[fields][log_type]}-%{+YYYY.MM.dd}"` .
Did you try this?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 9:33am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/13 "2019-12-23T09:33:05Z")
yes, and I still got logstash-date as an index only. Should I add index pattern in filebeat.yml too?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 10:23am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/14 "2019-12-23T10:23:17Z")
@mancharagopan @Christian_Dahlqvist
Its interesting that I chnaged my entire logstash.conf to have manual name of index and still I get logstash-2019.12.23-000001. Why is this happening?
```auto
input {
file{
path: /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/log2.log
add_field => {"Differentindex", "access"}
}
file{
path: /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log
add_field => {"Differentindex", "error"}
}
file{
path: /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/dispatcher-log.log
add_field => {"Differentindex", "dispatch"}
}
}
filter {
if[add_field] =="access"{
grok {
match => {"message" => "%{COMBINEDAPACHELOG}"}
} else if [add_field] == "errors" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}else [add_field] == "dispatcher" {
grok {
match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
}
}
}
output {
if [add_field] == "access" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "access-index"
}
} else if [add_field] == "errors" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "errors-index"
}
}
stdout {
codec => rubydebug
}
}
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 10:38am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/15 "2019-12-23T10:38:17Z")
Can you share the content in one of the log file so i can test it?
Also your filter conditions are wrong. add\_field is an option to add a new field. please refer
[https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-add\_field](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-add_field)
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 23, 2019, 10:52am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/16 "2019-12-23T10:52:20Z")
As far as I can see you do not have ant `add_field` defined which would mean none of those outputs are used. Do you by any chance have any other files in the config directory that have an elasticsearch output?
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 11:00am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/17 "2019-12-23T11:00:41Z")
@Mehak_Bhargava
> [@Christian\_Dahlqvist](#):
>
> As far as I can see you do not have ant `add_field` defined which would mean none of those outputs are used. Do you by any chance have any other files in the config directory that have an elasticsearch output?
How many config files you have in logstash config folder?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 23, 2019, 2:47pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/18 "2019-12-23T14:47:05Z")
If you have a single index called logstash then you are seeing [ILM](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm) in action.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 4:18pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/19 "2019-12-23T16:18:37Z")
Yes, @mancharagopan,
log file 1
```auto
08/10/2019 12:14:48 599 (null) DEBUG 27 GetUpdatedIncident for Incident Id 24749162 on thread 04fd1833-8275-46ff-816f-9acf0c1f7724:80759 on Thread 27
08/10/2019 12:14:48 600 (null) DEBUG 19 Updating cache with activity (152775689) Add Item:True Modify Item: False
08/10/2019 12:14:48 601 (null) DEBUG 67 Applying dynamic filter
08/10/2019 12:14:48 604 (null) DEBUG 43 GetUpdatedIncidentIndicators for Incident Id 24749170 on thread 04fd1833-8275-46ff-816f-9acf0c1f7724:80752 on Thread 43 Context Id : 4ec7ffe6-431c-4280-bca1-263e20077cbc
```
In the input section, I can add type instead of add\_field
```auto
input {
file{
path: /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/log2.log
type => { "access"}
```
Is this better?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 4:24pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/20 "2019-12-23T16:24:54Z")
@Christian_Dahlqvist, So I should define add\_field outside of teh input section in pipeline.conf here?
I have first-pipeline.conf, second-pipeline.conf, logstash.conf, and pipeline. conf. Below are the contents of it-
```auto
input{}
filter{}
output{}
```
```auto
input{}
filter{}
output{}
```
logstash.conf is below
```auto
# Read input from filebeat by listening to port 5044 on which filebeat will send the data
input {
beats {
port => "5044"
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if[type] =="DispatcherApp"{
grok {
match => {"message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} %{USER:extra} %{HTTPDATE:timestamp} \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})"}
}
} else if [type] == "IncidentAgent" {
grok {
match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} %{USER:extra} %{HTTPDATE:timestamp} \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})" }
}
}
}
output {
stdout {
codec => rubydebug
}
# Sending properly parsed log events to elasticsearch
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "DispApp"
#index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
}
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "InciAgent"
}
```
pipeline.conf is the one I have copied before in comments.
[Next page](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806.md?page=2)