Hi there,
I'm currently using the following pipeline with success:
{
"ingestpipeline" : {
"description" : "...",
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
],
"on_failure" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
],
"on_failure" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
],
"ignore_failure" : true
}
}
]
}
}
]
}
}
]
}
}
After using that, I decided to add a second grok processor in the pipeline for extraction from a different source field: log. But whenever I try to use that new pipeline config, I'm given the error message:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "[processors] required property is missing",
"property_name": "processors"
}
],
"type": "parse_exception",
"reason": "[processors] required property is missing",
"property_name": "processors"
},
"status": 400
}
Here is the pipeline configuration I'm trying to use:
{
"ingestpipeline" : {
"description" : "...",
"processors" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
],
"on_failure" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
],
"on_failure" : [
{
"grok" : {
"field" : "message",
"patterns" : [
"(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|Error|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
],
"ignore_failure" : true
}
}
]
}
}
]
}
},
{
"grok" : {
"field" : "log",
"patterns" : [
"(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rror|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE)).*?(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
],
"on_failure" : [
{
"grok" : {
"field" : "log",
"patterns" : [
"(?<useremail>[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)"
],
"on_failure" : [
{
"grok" : {
"field" : "log",
"patterns" : [
"(?<loglevel>([Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nformation|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rror|ERROR|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE))"
],
"ignore_failure" : true
}
}
]
}
}
]
}
}
]
}
}
I'm using AWS Elasticsearch Service OpenDistro version 7.1