Event stores configuration section

Zentral events are stored using instances of the event store backends. Multiple stores can be used at the same time. Some configuration options are common to all the store backends, and some are specific to each backend.

To define a store, a backend configuration needs to be added to the base.json stores dictionary. A unique identifier is used as the key, and the configuration is a dictionary. For example:

{
    …
    "stores": {
        "elasticseach": {
            "backend": "zentral.core.stores.backends.elasticsearch"
            …
        }
    }
}

Common backend options

backend

MANDATORY

The python module implementing the store, as a string. Currently available:

  • zentral.core.stores.backends.azure_log_analytics
  • zentral.core.stores.backends.datadog
  • zentral.core.stores.backends.elasticsearch
  • zentral.core.stores.backends.http
  • zentral.core.stores.backends.humio
  • zentral.core.stores.backends.kinesis
  • zentral.core.stores.backends.opensearch
  • zentral.core.stores.backends.snowflake
  • zentral.core.stores.backends.splunk
  • zentral.core.stores.backends.sumo_logic
  • zentral.core.stores.backends.syslog

frontend

OPTIONAL

A boolean indicating if the store is the main event store to be used to fetch events in the Zentral UI. Only one store can be set as the frontend store, and ATM, only the datadog, elasticsearch and splunk backends support fetching events for display in the UI.

excluded_event_filters

OPTIONAL

A list of filters used to exclude events from the store. Each filter is a dictionary/object. Filters can have tags, event_type and routing_key attributes. Each filter attribute is a list of strings. For example:

{
  "excluded_event_filters": [
    {"tags": ["munki", "santa"]},
    {"event_type": ["osquery_result"], "routing_key": ["important"]},
    {"event_type": ["zentral_login", "zentral_logout"]}
  ]
}

With these filters, the following events are excluded:

  • munki or santatagged events
  • osquery_result events with the important routing_key value
  • zentral_login or zentral_logout events

Boolean combinations: arrays/lists → OR, dictionaries/objects → AND.

  • Within excluded_event_filters, the different filters are combined using the OR operator.
  • Within each filter, the different attributes must all match (AND).
  • For each filter attribute, at least one value must match (OR).

The excluded_event_filters take precendence over the included_event_filters. If an event is a match for the excluded_event_filters, the included_event_filters are not evaluated, and the event is excluded.

If both excluded_event_filters and included_event_filters are not set, all events will be included in the store.

included_event_filters

OPTIONAL

A list of filters to included in the store. See excluded_event_filters for the filter syntax.

The included_event_filters are applied after the excluded_event_filters. If an event is a match for the excluded_event_filters, the included_event_filters and not evaluated, and the event is excluded.

If both excluded_event_filters and included_event_filters are not set, all events will be included in the store.

excluded_event_types

DEPRECATED

Use excluded_event_filters instead.

included_event_types

DEPRECATED

Use included_event_filters instead.

events_url_authorized_groups

OPTIONAL

A list of group names. Empty by default (i.e. all users will get the links). Can be used to display the links to the events in the store to only a subset of Zentral users, if not all users have direct access to the store.

Kinesis backend options

This store is capable of batch operation. The maximum batch_size is 500. See the kinesis:PutRecords documentation for more details.

AWS authentication and authorization

When operating in AWS, it is recommended to use a role attached to the EC2 instance or to the container to authenticate the calls to the Kinesis API.

Example of an IAM policy to allow Zentral to write to the Kinesis stream:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowKinesisPut"
            "Action": [
                "kinesis:PutRecords",
                "kinesis:PutRecord"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kinesis:<AWS_REGION>:<AWS_ACCOUNT_ID>:stream/<KINESIS_STREAM_NAME>",
        }
    ]
}

The PutRecord action can be omitted if the store is configured for batch operations.

If the authentication is not possible using the environment (standard environment variable, instance metadata service, …), you can set aws_access_key_id and aws_secret_access_key in the store configuration.

You can also configure an AWS IAM role to be assumed, using the assume_role_arn store configuration key.

stream

MANDATORY

The name of the Kinesis stream.

region_name

MANDATORY

The name of the Kinesis stream AWS region.

serialization_format

By default, the events will be serialized using the Zentral canonical serialization.

A serialization format optimized for the use with Kinesis Firehose is also available: firehose_v1.

Full example

{
    "backend": "zentral.core.stores.backends.kinesis",
    "aws_access_key_id": "XXXXXXXXXXXXX",
    "aws_secret_access_key": "YYYYYYYYYYYYY",
    "assume_role_arn": "arn:aws:iam::<ACCOUNT_ID>:role/<NAME_OF_THE_ROLE>",
    "stream": "name_of_the_stream",
    "region": "us-east-1",
    "serialization_format": "firehose_v1"
}

HTTP backend options

endpoint_url

MANDATORY

The URL where the Zentral events will be POSTed.

For example: https://acme.service-now.com/api/now/import/zentral_events.

username

OPTIONAL

Username used for Basic Authentication. If used, password MUST be set too.

password

OPTIONAL

Password used for Basic Authentication. If used, username MUST be set too.

headers

OPTIONAL

A string / string dictionary of extra headers to be set for the HTTP requests. The Content-Type header is set to application/json by default.

WARNING Basic Authentication via username and password conflicts with the configuration of the Authorization header.

concurrency

OPTIONAL

WARNING only works if the AWS SNS/SQS queues backend is used.

An integer between 1 and 20, 1 by default. The number of threads to use when posting the events. This can increase the throughput of the store worker.

Full example

{
    "backend": "zentral.core.stores.backends.http",
    "endpoint_url": "https://acme.service-now.com/api/now/import/zentral_events",
    "username": "Zentral",
    "password": "{{ env:SERVICE_NOW_API_PASSWORD }}",
    "verify_tls": true,
    "included_event_filters": [{
      "event_type": [
        "add_machine",
        "add_machine_os_version",
        "remove_machine_os_version",
        "add_machine_system_info",
        "remove_machine_system_info",
        "add_machine_business_unit",
        "remove_machine_business_unit",
        "add_machine_group",
        "remove_machine_group",
        "add_machine_disk",
        "remove_machine_disk",
        "add_machine_network_interface",
        "remove_machine_network_interface",
        "add_machine_osx_app_instance",
        "remove_machine_osx_app_instance",
        "add_machine_deb_package",
        "remove_machine_deb_package",
        "add_machine_program_instance",
        "remove_machine_program_instance",
        "add_machine_principal_user",
        "remove_machine_principal_user"
      ]
    }]
}

Snowflake backend options

The Snowflake backend is read-only. It can only be used as a frontend backend. To store the events in snowflake, you will have to setup a pipeline using the Kinesis backend, and Kinesis Firehose for example.

account

MANDATORY

The name of the Snowflake account

user

MANDATORY

The name of the Snowflake user

password

MANDATORY

The password of the Snowflake user

database

MANDATORY

The name of the Snowflake database

schema

The name of the Snowflake schema. Defaults to PUBLIC.

role

MANDATORY

The name of the Snowflake role.

warehouse

MANDATORY

The name of the Snowflake warehouse.

session_timeout

In seconds, the session timeout. After the current session has timed out, a new connection will be established if necessary. Defaults to 4 hours - 10 minutes.

Full example

{
  "backend": "zentral.core.stores.backends.snowflake",
  "frontend": true,
  "username": "Zentral",
  "password": "{{ env:SNOWFLAKE_PASSWORD }}",
  "database": "ZENTRAL",
  "schema": "ZENTRAL",
  "role": "ZENTRAL",
  "warehouse": "DEFAULTWH",
  "session_timeout": 14400
}

Splunk backend options

hec_url

MANDATORY

The base URL of the Splunk HTTP Event Collector. For example: https://splunk.example.com:8088. The path to the collector endpoint must not be included.

hec_extra_headers

OPTIONAL

A String/String dictionary of extra headers to use for the Splunk HEC requests. Empty by default. This can be used to authenticate with a proxy for example. The Authorization and Content-Type headers cannot be changed.

hec_token

MANDATORY

The HEC token. It is recommended to use the common Zentral configuration options to read the value from an environment variable "{{ env:ENV_VAR_NAME }}", a file "{{ file:FILE_PATH }}", or a GCP or AWS secret "{{ secret:NAME_OF_THE_SECRET }}".

verify_tls

OPTIONAL

A boolean value to indicate if the connection must be verified or not. Default: true.

batch_size

OPTIONAL

The number of events to write in a single request. Default: 1. A value up to 100 can be used to speed up the event storage.

source

OPTIONAL

The name of the source to use in the Splunk events. Do not use it if the source is set by the HTTP event collector.

index

OPTIONAL

The name of the Splunk index.

computer_name_as_host_sources

OPTIONAL

A list of inventory source names to use to find a hostname to set as the host value in the Splunk event. Empty by default (i.e. the machine serial number will be used as the host value).

serial_number_field

OPTIONAL

The name of the Splunk event field to use for the machine serial number. Default: machine_serial_number.

custom_host_field

OPTIONAL

If set, the event metadata host field value will be copied to this event field.

search_app_url

OPTIONAL

The URL to the Splunk search app. For example: https://splunk.example.com/en-US/app/search/search. Empty by default. If set, links will be displayed in the Zentral UI to allow users to see the events in Splunk.

search_url

OPTIONAL

The base URL of the Splunk API server. For example: https://splunk.example.com:8089. If this is set, along with an authentication_token, the store can be used as a frontend store.

search_extra_headers

OPTIONAL

A String/String dictionary of extra headers to use for the Splunk search API requests. Empty by default. This can be used to authenticate with a proxy for example. The Authorization and Content-Type headers cannot be changed.

authentication_token

OPTIONAL

The authentication token to use with the Splunk API server. If this is set, along with a search_url, the store can be used as a frontend store.

search_source

OPTIONAL

If set, a source filter will be added to the search jobs and urls. Use this for example if a single Splunk index is used for multiple Zentral instances.

search_timeout

OPTIONAL

In seconds. Defaults to 300s. The number of seconds to keep a search after processing has stopped. Only used if the store is configured as a frontend store.

Full example

{
    "backend": "zentral.core.stores.backends.splunk",
    "frontend": false,
    "hec_url": "https://splunk.example.com:8088",
    "hec_extra_headers": {
      "CF-Access-Client-Id": "123",
      "CF-Access-Client-Secret": "{{ env:SPLUNK_HEC_CF_ACCESS_CLIENT_SECRET }}"
    },
    "hec_token": "{{ env:HEC_TOKEN }}",
    "verify_tls": true,
    "batch_size": 100,
    "source": "zentral.example.com",
    "index": "zentral",
    "computer_name_as_host_sources": ["santa", "osquery"],
    "serial_number_field": "serial_number",
    "search_app_url": "https://splunk.example.com/en-US/app/search/search",
    "search_url": "https://splunk.example.com:8089",
    "search_extra_headers": {
      "CF-Access-Client-Id": "456",
      "CF-Access-Client-Secret": "{{ env:SPLUNK_SEARCH_CF_ACCESS_CLIENT_SECRET }}"
    },
    "authentication_token": "{{ env:SPLUNK_AUTH_TOKEN }}",
    "search_source": "zentral.example.com",
    "search_timeout": 300
}