Custom fields appear on different index or globally?

Elastic Stack Elasticsearch
1

Hi, I have 2 pipelines with logstash for parsing diffent csv and on elasticsearch after indexing, custom fields which are created in one pipeline appear on every index or globally on "discover" and probably messes up the values fetched. Do I need a custom mapping on elastic or custom fields on one pipeline become globaly available thus appear on every index selected?

Thanks!!!

doing a GET /_mapping returns the following:

 "stats-2018-08-22": {
"mappings": {
  "doc": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "bytes_in/s": {
        "type": "long"
      },
      "bytes_out/s": {
        "type": "long"
      },
      "bytes_total": {
        "type": "long"
      },
      "column10": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column11": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column12": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column13": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column14": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column15": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column16": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column6": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column7": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column8": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "column9": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "delay": {
        "type": "long"
      },
      "errors_in": {
        "type": "long"
      },
      "errors_out": {
        "type": "long"
      },
      "geoip": {
        "type": "object"
      },
      "host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "interface": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "jitter": {
        "type": "long"
      },
      "loss": {
        "type": "long"
      },
      "message": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "packets_in": {
        "type": "long"
      },
      "packets_out": {
        "type": "long"
      },
      "packets_total": {
        "type": "long"
      },
      "path": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "pl": {
        "type": "float"
      },
      "rtt": {
        "type": "float"
      },
      "segment": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "tags": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "target_host": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "unix_timestamp": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

}

why my custom fields from another pipeline ("jitter" "delay "loss") are presented on every index ?? as showed above

2

Multiple files in the same directory are concatenated by Logstash, so all data will go to all output as described inthis thread.

1 Like
3

great!! thanks a lot for your rapid answer Christian!!!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.