Make create-fields index out of range error


(Zachary Buckholz) #1

I am attempting to remake a haproxy filebeat module I did last year manually using the recommended process from https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html

make create-fileset MODULE={module} FILESET={fileset}

This first step worked fine, but then I tried to use the make create-fields command to generate a fields.yml

make create-fields MODULE={module} FILESET={fileset}

$ make create-fields MODULE=haproxy FILESET=haproxy_log
panic: runtime error: index out of range

goroutine 1 [running]:
main.newField(0xc4200763b1, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/z/go/src/github.com/elastic/beats/filebeat/scripts/generator/fields/main.go:80 +0x19a
main.getElementsFromPatterns(0xc420064810, 0x3, 0x3, 0x3, 0xa00005de300, 0x38, 0x38, 0xc420080380)
/home/z/go/src/github.com/elastic/beats/filebeat/scripts/generator/fields/main.go:122 +0xc2
main.(*processors).processFields(0xc420080380, 0x2, 0x4, 0xc420080380, 0x0, 0x0)
/home/z/go/src/github.com/elastic/beats/filebeat/scripts/generator/fields/main.go:175 +0x58
main.(*pipeline).toFieldsYml(0xc4200802c0, 0x0, 0x7ffd11d32291, 0x7, 0x7ffd11d322a3, 0xb, 0xc4200802c0)
/home/z/go/src/github.com/elastic/beats/filebeat/scripts/generator/fields/main.go:282 +0x66
main.main()
/home/z/go/src/github.com/elastic/beats/filebeat/scripts/generator/fields/main.go:327 +0x267
exit status 2
make: *** [create-fields] Error 1

The pipeline.json is pretty straight forward. I am not sure what's going on, it appears to be a []byte array and I think it's related to using the geo_point field type.

Hopefully someone can point me in the right direction.
Thanks

{
"description": "HAProxy Logging",
"on_failure": [
{
"set": {
"field": "haproxy.ingest.error",
"value": "{{ _ingest.on_failure_message }}"
}
}
],
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{HAPROXYHTTP}",
"%{HAPROXYTCP}",
"%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{GREEDYDATA:error}"
]
}
},
{
"geoip": {
"field": "client_ip",
"target_field": "geoip"
}
}
]
}


(ruflin) #2

@Zachary_Buckholz Great to see you use our new generator which was just merged a few days ago. For the error I think @kvch is best to answer.


(Noémi Ványi) #3

I am happy to see that you are using the new generator!
It seems to me that the script is not prepared for having patterns without fields names e.g %{HAPROXYHTTP}. I am opening a PR to fix it.
Thanks for the report.


(Noémi Ványi) #4

I also noticed that you escape using \. You must use \\, otherwise it cannot be escaped properly in Filebeat.


(Noémi Ványi) #5

I opened the PR: https://github.com/elastic/beats/pull/6110
Please, let me know if you are still facing issues with the generator.


(Zachary Buckholz) #6

Thanks guys I will give it a try this afternoon.


(Zachary Buckholz) #7

It did not error out, but it didn't give me a fields.yml I was expecting to see. It's a starting point I can further edit manually. Maybe my understanding of the feature is incorrect, but I was expecting to see all the HAPROXYHTTP, HAPROXYTCP, SYSLOG timestamp, and most importantly the geo_point.

I got the following

module/haproxy/haproxy_log/_meta/fields.yml

  • name: syslog_timestamp
    description: Please add description
    example: Please add example
    type: text
  • name: syslog_server
    description: Please add description
    example: Please add example
    type: keyword
  • name: client_ip
    description: Please add description
    example: Please add example
  • name: client_port
    description: Please add description
    example: Please add example
  • name: accept_date
    description: Please add description
    example: Please add example
  • name: backend_name
    description: Please add description
    example: Please add example
  • name: server_name
    description: Please add description
    example: Please add example
  • name: error
    description: Please add description
    example: Please add example
    type: text

(Noémi Ványi) #8

What would be the expected output?


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.