Brewing in Beats: Autodiscovery with Docker

Autodiscovery – use Docker events to auto-configure Beats

The autodiscovery feature with the first provider (for Docker) has been merged. Autodiscovery allows the user to define different providers, that watch for system changes and emit events to a common bus. Then the autodiscovery module detects situations when there is something we can monitor and instantiates new modules for it.

The first provider watches for Docker events. It supports config mapping from container metadata to config templates, so new modules are created when a container starts.

metricbeat.autodiscover: providers: - type: docker templates: - condition: equals.docker.container.image: redis config: - module: redis metricsets: ["info", "keyspace"] hosts: "${data.host}:${data.port}"

The above is an example configuration that instantiates the Metricbeat Redis module every time a new Redis container (defined by having the redis image) is started. Note that the connection information (host/ports) is filled in by the autodiscovery support via a template.

This feature will be released in Metricbeat and Filebeat 6.1.

Configure the number of routing shards in the Elasticsearch template

Elasticsearch 6.1 will have an API for splitting shards. To enable this feature, applications like Beats need to set the config option index.number_of_routing_shards at index time creation. The actual number of shards must be a factor of the number of routing shards.

This PR adds configuration option in Beats for the routing shards, with a default of 30. We chose 30 as it is a multiple of 1, 3 and 5, our current number of default shards in Beats and ES. 

The new configuration option will be present in Beats 6.1.

Packetbeat: support for reading TLS envelopes

This PR adds TLS protocol support to Packetbeat, which is one of the most anticipated Packetbeat features. It doesn’t mean decrypting traffic, but it parses the initial handshake and extracts data like ciphers supported by the client and the server, the client and server certificate chains, the subject alternative name (SAN), validity dates, raw certificates, and so on. This data is super valuable for debugging TLS issues and also for intrusion detection and auditing.

The implementation also comes with support for the ALPN extension to TLS, which allows Packetbeat to detect, for example, whether HTTP/2 or HTTP/1 are used as an application protocol on top of the TLS connection.

This feature will be released in Packetbeat 6.1.

Filebeat: Docker JSON-file prospector

This PR adds an (experimental) dedicated Filebeat prospector for Docker logs written by the default JSON logging driver. Filebeat could already read Docker logs via the log prospector with JSON decoding enabled, but this new prospector makes things easier for the user. It abstracts the format, so there is no need to manually configure JSON decoding. Here is an example config, which captures the logs from a single container specified by its ID:

prospectors: - type: docker containers.ids: - c3ec7a0bd9640151a768663b7e78c115d5b1a7f87fba572666bacd8065893d41

It also parses the timestamp from the JSON file, something that wasn’t possible with Filebeat alone (it required Logstash or Ingest Node). 

This new prospector will be released with Filebeat 6.1.

Other changes

Repository: elastic/beats

Affecting all Beats

Changes in master:

  • Adjust generated CSV dependency file #5618
  • Fix console color output on Windows #5612
  • Optional pipeline and output metrics #5586
  • Add `Get` method to `cfgfile.Registry` #5581
Metricbeat

Changes in master:

  • Fix failing perfmon test #5607
  • Add Windows service dashboard #5603
  • Foundation for Logstash Metricbeat module #5540
Filebeat

Changes in master:

  • Remove spurious ID method #5583
  • Remove prospector validation as not needed anymore #5537
  • Add experimental Docker json-file prospector #5402
Heartbeat

Changes in master:

  • Fix visualization in Heartbeat dashboards #5564
Auditbeat

Changes in master:

  • Remove redundant log prefix for auditbeat module (#5533) #5539
Testing

Changes in master:

  • Fix generator tests #5622
  • Update to Golang 1.9.2 #5569
  • Improve test stability #5567
  • Test new Kafka setup for more stable builds #5549

Changes in 6.0:

  • Test new Kafka setup for more stable builds #5549
  • Fix packetbeat flaky tests #5200
  • Improve test stability #5567
Documentation

Changes in master:

  • Fix Logstash defaults in output documentation #5620
  • Add Docker docs for Auditbeat #5619
  • Update filters to processors #5616
  • Fix repo steps in the Beats platform ref #5602
  • Fix experimental and beta flags for features #5591
  • Flag experimental and beta modules in the docs #5590
  • Add link to ES docs about type removal #5588
  • Metricbeat docs fixes #5578

Changes in 5.6:

  • Clarify docs for system core percentages #5565
  • Update multiline.asciidoc #5356

Changes in 6.0:

  • Fix prerelease links in apt/yum package docs #5641
  • Backport to 6.0: Doc changes in #5588, #5590, #5591, #5602 #5624
  • Hot fix link that was using current. #5580
  • Backport to 6.0: #5513 and #5496 #5574
  • Backport to 6.0: Fix getting started (#5326) #5573
  • Set docs version to 6.0.0 and released #5561

Repository: elastic/gosigar

Changes in master:

  • Changes to support CPU usage metric in Beats #83

Removal of Mapping Types in Elasticsearch 6.0

Mapping types are going away. Elasticsearch 6.0 supports a single mapping
type per index only, and it represents the first step on the way to removing
types altogether.

This blog post explains what types are, why we are removing them, how you can
migrate your indices to a single type world, and lays out the full schedule
for type removal over the next three major versions.

  • What are mapping types?
  • Why are mapping types being removed?
  • Alternatives to mapping types
    • Custom type field
    • Index per document type
  • Parent/Child without mapping types
  • Schedule for removal of mapping types
  • Migrating multi-type indices to single-type
    • Custom type field
    • Index per document type

<tldr>

  • Indices created in Elasticsearch 6.0.0 or later may contain a
    single mapping type only.
  • Indices created in 5.x with multiple
    mapping types will continue to function as before in Elasticsearch 6.x.
  • Mapping types will be completely removed in Elasticsearch 7.0.0
    although some backwards compatibility features will only be removed in 9.0.0.

</tldr>

parent-child relationship between documents, so documents of type question could be parents to
documents of type answer.

join field.

join field replacement for parent-child is available on indices created in 5.6.

Elasticsearch 6.x

  • Indices created in 5.x will continue to function in 6.x as they did in 5.x.
  • Indices created in 6.x only allow a single-type per index. Any name can be used for the type, but there can be only one.
  • The _type name can no longer be combined with the _id to form the _uid field. The _uid field has become an alias for the _id field.
  • New indices no longer support the old-style of parent/child and should use the join field instead.
  • The _default_ mapping type is deprecated.

Elasticsearch 7.x

  • The type parameter in URLs are optional. For instance, indexing a document no longer requires a document type.
  • The GET|PUT _mapping APIs support a query string parameter ( include_type_name) which indicates whether the body should include a layer for the type name. It defaults to true. 7.x indices which don’t have an explicit type will use the dummy type name _doc.
  • The _default_ mapping type is removed.

Elasticsearch 8.x

  • The type parameter is no longer supported in URLs.
  • The include_type_name parameter defaults to false.

Elasticsearch 9.x

  • The include_in_type parameter is removed.

Reindex API can be used to convert multi-type indices to
single-type indices. The following examples can be used in Elasticsearch 5.6
or Elasticsearch 6.x. In 6.x, there is no need to specify index.mapping.single_type as that is the default.

Custom type field

This first example adds a custom type field and sets it to the value of the
original _type. It also adds the type to the _id in case there are any
documents of different types which have conflicting IDs:

PUT new_twitter
{ "mappings": { "doc": { "properties": { "type": { "type": "keyword" }, "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" }, "content": { "type": "text" }, "tweeted_at": { "type": "date" } } } }
}
POST _reindex
{ "source": { "index": "twitter" }, "dest": { "index": "new_twitter" }, "script": { "source": """ ctx._source.type = ctx._type; ctx._id = ctx._type + '-' + ctx._id; ctx._type = 'doc'; """ }
} 

Index per document type

This next example splits our twitter index into a tweets index and a users index:

PUT users
{ "settings": { "index.mapping.single_type": true }, "mappings": { "user": { "properties": { "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" } } } }
}
PUT tweets
{ "settings": { "index.mapping.single_type": true }, "mappings": { "tweet": { "properties": { "content": { "type": "text" }, "user_name": { "type": "keyword" }, "tweeted_at": { "type": "date" } } } }
}
POST _reindex
{ "source": { "index": "twitter", "type": "user" }, "dest": { "index": "users" }
}
POST _reindex
{ "source": { "index": "twitter", "type": "tweet" }, "dest": { "index": "tweets" }
}