본문으로 건너뛰기
버전: dev

Log Integration (Beta)

HertzBeat's log integration module aims to achieve unified reception, standardized processing of log data from different third-party log systems and observability platforms. As a centralized "log center", HertzBeat can efficiently integrate log information from external systems and provide real-time log monitoring and analysis capabilities.

경고

The log integration feature is currently in Beta (experimental) stage. There may be potential defects and limitations. The feature is under active development and iteration.

Core Capabilities​

  • Multi-source Log Integration: Support receiving log data from mainstream platforms such as OpenTelemetry, Filebeat, Vector, Loki
  • Log Format Standardization: Convert log data from different platforms to HertzBeat's internal unified format for subsequent processing and analysis
  • Real-time Log Processing: Provide real-time log stream processing capabilities, supporting instant storage and distribution of log data
  • Intelligent Log Analysis: Provide log search, filtering and other analysis functions

Supported Log Sources​

HertzBeat currently supports data integration from the following third-party log platforms:

  • OTLP: Support standard OpenTelemetry Log Protocol (OTLP) HTTP format, can directly receive log data from OpenTelemetry Collector and various applications that support OTLP.
  • More Protocol Support: HertzBeat is actively expanding its log integration support, including Filebeat, Vector, Loki, etc. If you can't find the integration you need temporarily, the active community can also help you add it.

You can view specific integration methods and configuration examples through HertzBeat's "Log Integration" interface.

log_integration

OpenTelemetry OTLP Protocol Integration​

API Endpoint​

HertzBeat provides the following interface for receiving OTLP log data:

POST /api/logs/otlp/v1/logs

Request Configuration​

Request Headers​

  • Content-Type: application/json or application/x-protobuf
  • Authorization: Bearer {token}

Request Body Format​

Supports standard OTLP JSON-Protobuf format or Binary Protobuf format log data:

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "my-service"
            }
          },
          {
            "key": "service.version", 
            "value": {
              "stringValue": "1.0.0"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "my-logger",
            "version": "1.0.0"
          },
          "logRecords": [
            {
              "timeUnixNano": "1640995200000000000",
              "severityNumber": 9,
              "severityText": "INFO",
              "body": {
                "stringValue": "This is a log message"
              },
              "attributes": [
                {
                  "key": "user.id",
                  "value": {
                    "stringValue": "12345"
                  }
                }
              ],
              "traceId": "12345678901234567890123456789012",
              "spanId": "1234567890123456"
            }
          ]
        }
      ]
    }
  ]
}

Configuration Examples​

OpenTelemetry Collector Configuration​

Add HertzBeat as a log export target in the OpenTelemetry Collector configuration file:

exporters:
  otlphttp:
    logs_endpoint: http://{hertzbeat_host}:1157/api/logs/otlp/v1/logs
    compression: none
    encoding: json
    headers:
      Authorization: "Bearer {token}"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Log Data Format Description​

Core Fields​

  • timeUnixNano: Log timestamp (nanosecond precision)
  • severityNumber: Log level numeric value (1-24, corresponding to TRACE to FATAL)
  • severityText: Log level text (such as "INFO", "ERROR", etc.)
  • body: Log message content
  • attributes: Log attribute key-value pairs
  • traceId: Trace ID (optional)
  • spanId: Span ID (optional)

Resource Attributes​

Application and environment information can be set through resource.attributes:

  • service.name: Service name
  • service.version: Service version
  • deployment.environment: Deployment environment (dev/test/prod)
  • host.name: Host name

Configuration Verification​

  1. Configure Log Sender: Configure external systems to send OTLP logs to HertzBeat specified interface
  2. View Received Logs: View received log data in HertzBeat real-time log module
  3. Verify Data Integrity: Verify whether log data format, timestamp, attributes and other information are correct

log_stream

Common Issues​

Log Sending Failed​

  • Network Connection Issues: Ensure HertzBeat service address can be accessed by external systems
  • Request Header Error: Verify that request header Content-Type is set to application/json

Log Format Error​

  • OTLP Format: Ensure standard OTLP JSON-Protobuf or Binary Protobuf format is sent
  • Timestamp Format: Check if timestamp format is Unix timestamp with nanosecond precision
  • Log Level: Verify severityNumber value range (1-24)
  • Data Type: Ensure data types of each field comply with OTLP specification

For more log integration methods or technical issues, feel free to communicate with the community through GitHub Issues.