Loading

File log receiver

Serverless Observability Stack EDOT Collector

The file log receiver ingests logs from local files and forwards them for processing and exporting. It is a versatile and widely-used component for collecting logs in file-based formats, such as container logs, system logs, or custom application logs.

The receiver supports multiline parsing, filtering, persistent tracking of file offsets, and routing based on file metadata or log content.

The filelogreceiver is included by default in the EDOT Collector's Kubernetes Helm chart. It is preconfigured to collect logs from pod log files such as /var/log/pods/*/*/*.log, and uses the file_storage extension to track read positions and avoid duplicate ingestion after restarts.

To view or customize the default configuration, refer to:

The following example shows how to ingest Kubernetes container logs with routing logic based on log format:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    exclude:
      - /var/log/pods/*/*/*.gz
    start_at: beginning
    include_file_name: true
    include_file_path: true
    fingerprint_size: 100
    max_log_size: 102_400
    storage: file_storage
    operators:
      - type: router
        routes:
          - output: parser_containerd
            expr: 'body matches "^\\d{4}-\\d{2}-\\d{2}T"'
          - output: parser_crio
            expr: 'body matches "^[A-Z][a-z]{2} [0-9]{1,2} "'

      - id: parser_containerd
        type: json_parser
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: 2006-01-02T15:04:05.000000000Z07:00

      - id: parser_crio
        type: regex_parser
        regex: '^(?P<time>[^ ]+ [^ ]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<body>.*)'
        timestamp:
          parse_from: attributes.time
          layout: '%b %d %H:%M:%S'

The following are some of the most commonly used settings when working with the file log receiver. These options help control what files are read, how logs are parsed, and how file positions are tracked between restarts:

Option Description
include List of glob patterns for files to include.
exclude Optional glob patterns for files to exclude (for example rotated or compressed files).
start_at beginning or end. Controls where to start reading files when no checkpoint exists.
operators Parsing and routing logic for logs.
storage Enables persistent tracking of file positions using a storage extension.
max_log_size Maximum size of individual log entries (in bytes).
fingerprint_size Size (in bytes) used to identify and deduplicate files.

For the full list of options, refer to the upstream filelogreceiver documentation.

These tips can help you get the most out of the file log receiver:

Use persistent storage to avoid duplicates

Without persistent storage, the receiver will not retain file read positions across restarts. This can result in either duplicate ingestion (if start_at is set to beginning) or lost logs (if set to end). Use the storage: setting and configure a persistent volume when running in Kubernetes.

The default configuration excludes rotated files, which helps prevent duplicate ingestion. If you need to include rotated logs, update the include: and exclude: patterns accordingly.

Multiline log support is not enabled by default. To handle multi-line messages such as stack traces, define a regex_parser, combine_logs, or multiline operator in your Helm chart configuration.

If your environment produces logs in multiple formats (for example containerd and CRI-O), use the router operator to apply appropriate parsers based on the log structure.

Using start_at: beginning without a storage extension will re-read all files from the start after each restart, which might lead to duplicate log entries.

Like any component, file log receiver has some trade-offs and behaviors to be aware of, especially in Kubernetes environments:

  • Persistent log tracking requires explicit storage: configuration and persistent volume support in Kubernetes.

  • Multiline logs are not parsed by default. You must customize the configuration to parse them.

  • Incorrect include/exclude globs can result in missing rotated logs or unintended ingestion.

  • High-volume directories might require tuning of max_concurrent_files.