[Agentless] Agentless policies create global add_fields processors for organization and other fields that collide with ECS

[cmacknz]

We have a user experience ingestion errors because the ingest pipeline of the Microsoft O365 integration because agentless agents include a global processor to add the organization field as a string, which collides with the ECS organization field the integration expects.

An example pipeline failure is:

Processor 'rename' with tag '' failed with message 'cannot set [id] with parent object of type [java.lang.String] as part of path [organization.id]'

Where the field causing the failure is organization below:

"division": "engineering",
"organization": "security",
"team": "security-service-integrations",

The agent policy includes these fields with add_field processors:

inputs:
    - data_stream:
        namespace: default
      id: cel-o365-abcdefgh
      meta:
        package:
            name: o365
            version: 2.15.1
      processors:
        - add_fields:
            fields:
                division: engineering
                organization: security
                team: security-service-integrations

Agentless policies add the division, organization, and team as global data tags:

kibana/x-pack/platform/plugins/shared/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/single_page_layout/hooks/setup_technology.ts

Lines 195 to 196 in 12b7429

	name: agentlessPolicyName,
	global_data_tags: getGlobaDataTags(packageInfo),

Global data tags get injected via add_fields processors when the policy is created:

kibana/x-pack/platform/plugins/shared/fleet/server/services/agent_policies/package_policies_to_agent_inputs.ts

Lines 181 to 191 in 12b7429

	const globalDataTagsToAddFields = (tags: GlobalDataTag[]): FullAgentPolicyAddFields => {
	const fields: { [key: string]: string | number } = {};
	tags.forEach((tag) => {
	fields[tag.name] = tag.value;
	});
	return {
	add_fields: {
	target: '',
	fields,

These fields need to be in a namespace that will not collide with actual integration data or simply removed from the events. It is possible there was no intention to inject these fields into every event.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[chemamartinez]

Other integrations potentially affected by this are Google Workspace and Zscaler ZIA, both integrations support Agentless and populate the organization ECS fields.

[andrewkroh]

or simply removed from the events.

👍

This data is irrelevant to the user. I believe removing the fields from the user's documents should be the solution.