FileBeat Not starting due to config in yml file


(Aditya Soni) #1

HI experts,
This is my filebeat.yml file ..
i am getting error :1053 whenever i am starting filebeat service.
may be some mistake i am doing in this file, please correct me where i am wrong.

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.



  # Paths that should be crawled and fetched. Glob based paths.
 paths:
- E:\ELK-STACK\logstash-tutorial-dataset.log
input_type: log
document_type: apachelogs
  # document_type: apachelogs
 
   
   
   #paths:
   # - E:\ELK-STACK\mylogs.log
  #fields: {log_type: mypersonal-logs}
       #- C:\Logs\GatewayService\GatewayService-Processor.Transactions-20170810
   # - C:\ECLIPSE WORKSPACE\jcgA1\jcgA1\logs-logstash.*
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after


#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
 # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5043"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

(Truongdqse03303) #2

i think your syntax not correct, try

- input_type: log
   paths:
   - E:\ELK-STACK\logstash-tutorial-dataset.log
  document_type: apachelogs

(Aditya Soni) #3

HI @truongdqse03303 tried your solution but it doesn't worked, Filebeat service is not getting started.
Here is the updated .yml.

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log
   paths:
   - E:\ELK-STACK\logstash-tutorial-dataset.log
  document_type: apachelogs

(Truongdqse03303) #4

what is the filebeat log tell you


#5

document_type is deprecated in 5.5.

Here the documentation:
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html

Which version are you running?


(Aditya Soni) #6

Hi @truongdqse03303 ,
i am getting this on publishing .yml from console.

filebeat2017/08/23 07:11:24.290841 beat.go:339: CRIT Exiting: error loading conf
ig file: yaml: line 18: mapping values are not allowed in this context
Exiting: error loading config file: yaml: line 18: mapping values are not allowe
d in this context

(Aditya Soni) #7

@Mojster I am using filebeat-5.5.1-windows-x86_64.

ok as you said the document_type is depreacted i agree.

that's why i have tried using fields i.e

- input_type: log
   paths:
   - E:\ELK-STACK\logstash-tutorial-dataset.log
 fields: 
   document_type:apachelogs  

again service not started.


(Truongdqse03303) #8

Hi @aditya_soni
Your config file seem normal but i think the reason is your paths: and fields: not in the direct line.
Try to put it like this

- input_type: log
 paths:
   - E:\ELK-STACK\logstash-tutorial-dataset.log
 fields: 
   document_type:apachelogs

(Steffen Siering) #9

This is a YAML parsing error. YAML is sensitive to usage of whitespace. That is, settings should always be indented by the same amount of characters (we prefer space over tab for consistency).

You can also check you config is parseable with http://www.yamllint.com


(Aditya Soni) #10

hi @steffens yes there was a mistake in yaml as it is not properly organised, so i have corrected it to this yml.
and now i am getting this error:-

error loading states for prospector 18392878754820818519: required 'object', but found 'string' in field 'filebeat.prospectors.0.fields'

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.full.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
  fields: 
    document_type:apachelogs
  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Mutiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after


#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

please help regarding it.


(Steffen Siering) #11

might be this setting:

  fields: 
    document_type:apachelogs

Try putting a space between document_type: and apachelogs.

Why do you want to set fields.document_type, instead of just using the document_typesetting (adds a type field)?


(Aditya Soni) #12

HI all,

I have deleted my default .yml file and updated with this configuration and it worked.
thanks for your help all of you.
filebeat.prospectors:
-
paths:
- E:\ELK-STACK\logstash-tutorial-dataset.log

  input_type: log
  document_type: apachelogs
  

-
  paths:
  - C:\Logs\GatewayService\GatewayService-Processor.Transactions-20170810.slog
  
  input_type: log
  document_type: gatewaylogs  
  
 
-
  paths:
  - E:\automation frameowrk\AUTO_JUL27_Debit_Purchase_Ig3_EditError\FirstData-BuyPass-HC.log
  
  input_type: log
  document_type: authenticlogs 
  
output.logstash:
  hosts: ["localhost:5043"]

now i have one more question is that i have a unstructured authentic(our project) logs which doesn't follow same pattern so how may i apply the pattern on these type of logs?
sample of the logs is:

07:35:19.119 TRACE [SocketConnectionHandler:ListProc-Q0:I5] 	[UID:SER22VVM211:CPBridge:201707270735180832:000000000000000001, Message Timestamp:2017-07-27 07:35:18.832] Writing raw message to socket
07:35:19.119 DEBUG [ListProcessor:ListProc-Q0:I5] 	Action [AcquirerRequest.AuxRequest] executed with result [TERMINATE_ALL(2)]
07:35:19.119 DEBUG [ListMgr:SocketConnection-Executor] 	Scheduling task .... priority=false
07:35:19.120 TRACE [ListMgr:SocketConnection-Executor] 	Message queued into List Processor queue
07:35:19.120 DEBUG [SocketConnectionHandler:ListProc-Q0:I0] 	connection:/SER22VVM211/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC%231: WriteAcknowledgedEvent received on FirstData-BuyPass-HC#1
07:35:19.120 TRACE [ConnectionStatisticalInfo:ListProc-Q0:I0] 	BucketPeriod : 10 Seconds, 1 ms starting at Thu Jul 27 07:35:10 EDT 2017 will be retained until Thu Jul 27 07:36:00 EDT 2017
07:35:19.265 TRACE [SocketConnection:SelectorHandler-0] 	Message received with MLI : 119
07:35:19.268 DEBUG [ListMgr:SocketConnection-Executor] 	Scheduling task .... priority=false
07:35:19.268 TRACE [ListMgr:SocketConnection-Executor] 	Message queued into List Processor queue
07:35:19.268 WARN  [SocketConnectionHandler:ListProc-Q0:I4] 	[UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] *****SENSITIVE DATA DISPLAYED CLEAR AT THIS LEVEL - THIS MAY INVALIDATE PCI DSS COMPLIANCE, RUNNING AT THIS LEVEL MAY SLOW DOWN THE SYSTEM*******
07:35:19.268 INFO  [SocketConnectionHandler:ListProc-Q0:I4] 	[UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] connection:/SER22VVM211/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC%231: Message received from FirstData-BuyPass-HC#1
07:35:19.268 DEBUG [SocketConnectionHandler:ListProc-Q0:I4] 	Acquirer data cannot be retrieved with Acquirer ID[-1]
07:35:19.269 TRACE [SocketConnectionHandler:ListProc-Q0:I4] 	[UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] connection:/SER22VVM211/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC/FirstData-BuyPass-HC%231
Receiving incoming message from java.nio.channels.SocketChannel[connected local=/153.71.66.148:22734 remote=/153.61.241.191:7736]
	0000:000000:  41 52 50 33  30 30 39 30  31 30 32 30  30 20 39 30  | ARP3009010200 90 | 
	0010:000016:  30 36 30 30  33 30 30 30  30 30 30 30  31 30 20 20  | 06003000000010   | 
	0020:000032:  20 20 20 20  30 37 33 35  31 33 30 34  30 31 31 31  |     073513040111 | 
	0030:000048:  39 30 30 37  30 30 37 30  30 37 31 20  20 20 20 20  | 90070070071      | 
	0040:000064:  20 20 20 20  20 20 20 20  20 20 20 20  20 20 20 20  |                  | 
	0050:000080:  20 20 20 45  32 47 45 44  49 54 20 45  52 3A 4F 50  |    E2GEDIT ER:OP | 
	0060:000096:  54 20 44 41  54 41 20 20  20 20 20 20  20 31 34 20  | T DATA       14  | 
	0070:000112:  20 39 30 30  30 30 30                               |  900000          | 

07:35:19.272 TRACE [LengthVar:ListProc-Q0:I4] 	Optional Area : DataLength = 0 Max Data Length = 999
07:35:19.273 DEBUG [MsgParser:ListProc-Q0:I4] 	[UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] 
Parsed Incoming Body Fields :
EXT-ID[---]     FLD[Device ID]                 FRMT[Fixed]              LL[0]  LEN[11]   DATA[3009010200 ] 
EXT-ID[---]     FLD[Sequence Number]           FRMT[Fixed]              LL[0]  LEN[6]    DATA[900600] 
EXT-ID[---]     FLD[Transaction Code]          FRMT[Fixed]              LL[0]  LEN[2]    DATA[30]   
EXT-ID[---]     FLD[Auth Amount]               FRMT[Fixed]              LL[0]  LEN[8]    DATA[00000010] 
EXT-ID[---]     FLD[Posting Date]              FRMT[Fixed]              LL[0]  LEN[6]    DATA[      ] 
EXT-ID[---]     FLD[Transaction Time]          FRMT[Fixed]              LL[0]  LEN[6]    DATA[073513] 
EXT-ID[---]     FLD[Manual Entry Flag]         FRMT[Fixed]              LL[0]  LEN[1]    DATA[0]    
EXT-ID[---]     FLD[Card Data]                 FRMT[Fixed]              LL[0]  LEN[40]   DATA[4011190070070071                        ] 
EXT-ID[---]     FLD[Action Code]               FRMT[Fixed]              LL[0]  LEN[1]    DATA[E]    
EXT-ID[---]     FLD[Response Code]             FRMT[Fixed]              LL[0]  LEN[2]    DATA[2G]   
EXT-ID[---]     FLD[Terminal Display]          FRMT[Fixed]              LL[0]  LEN[16]   DATA[EDIT ER:OPT DATA] 
EXT-ID[---]     FLD[User Data]                 FRMT[Fixed]              LL[0]  LEN[7]    DATA[       ] 
EXT-ID[---]     FLD[Authorizer Network ID]     FRMT[Fixed]              LL[0]  LEN[2]    DATA[14]   
EXT-ID[---]     FLD[Authorizer Network Res..]  FRMT[Fixed]              LL[0]  LEN[2]    DATA[  ]   
EXT-ID[---]     FLD[Version Number]            FRMT[Fixed]              LL[0]  LEN[3]    DATA[900]  
EXT-ID[---]     FLD[Optional Area]             FRMT[LLLVAR Group]       LL[3]  LEN[0]    DATA[]     

07:35:19.273 DEBUG [Validator:ListProc-Q0:I4] 	[UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] 
VALIDATION:

07:35:19.274 DEBUG [MsgMapper:ListProc-Q0:I4] 	[UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] 
MAPPING:
SOURCE       INT[FirstData-BuyPass-HC] MSG[Authorization Response]      SEG[Authorization Response] 
DESTINATION  INT[Authentic]      MSG[Generic Message]             SEG[Generic Body]      

07:35:19.275 DEBUG [MsgMapper:ListProc-Q0:I4] 	[UID:N/A, Message Timestamp:2017-07-27 07:35:15.773] 

please reply


#13

@steffens on the documentation page is stated, that document_type field is deprecated since 5.5.


(Aditya Soni) #14

yeah okay , so i will use fields to differentiate the logs.
can you please answer my question of applying pattern on the above logs
?


#15

I'm using ES stack only for a few months, so I'm fresh.
For this type of logs I use grok filter followed by if/else.

This is my example. Hope it helps

filter {
	grok {
		match => {"message" => "%{DATA:act}\#%{DATA:interface}\#%{DATA:country}\#%{DATA:acronym}\#%{DATA:typeDB}\#%{GREEDYDATA:leftovers}"}
	}
	if [act] == "CONN"{
		drop { }
	}
	ruby {
		code => "event.set('acronym', event.get('acronym').upcase)"
	}
	
	mutate{
		split => {"leftovers" => "#"}
	}
	
	if [act] == "AUTH"{
		mutate{
			add_field => {
				"date" => "%{leftovers[1]}"
				"userID" => "%{leftovers[2]}"
			}
			remove_field => [leftovers]
		}
	}else if [act] == "DISP"{
		mutate{
			add_field => {
				"date" => "%{leftovers[0]}"
				"outputType" => "%{leftovers[1]}"
				"userID" => "%{leftovers[2]}"
			}
			remove_field => [leftovers]
		}
	}else if [act] == "SEARCH"{
...
...
...
}

(system) #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.