Skip to content

mimecast: resolve field data type conflicts between data streams #13825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 23, 2025
Merged

mimecast: resolve field data type conflicts between data streams #13825

merged 3 commits into from
May 23, 2025

Conversation

@navnit-elastic
Copy link
Contributor

@navnit-elastic navnit-elastic commented May 7, 2025 •
edited by ShourieG
Loading

Proposed commit message

mimecast: resolve field data type conflicts between data streams. 

This introduces a breaking change by changing the field type from 'keyword' 
to 'boolean'  for the 'taggedExternal' and 'taggedMalicious' fields in the
'siem_logs' data stream to  resolve conflicts with other data streams.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Pipeline and System Tests:

--- Test results for package: mimecast - START ---
╭──────────┬───────────────────────────────┬───────────┬───────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE  │ DATA STREAM                   │ TEST TYPE │ TEST NAME                                                         │ RESULT │ TIME ELAPSED │
├──────────┼───────────────────────────────┼───────────┼───────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ mimecast │ archive_search_logs           │ pipeline  │ (ingest pipeline warnings test-archive-search-logs.log)           │ PASS   │ 306.593196ms │
│ mimecast │ archive_search_logs           │ pipeline  │ test-archive-search-logs.log                                      │ PASS   │ 146.969676ms │
│ mimecast │ audit_events                  │ pipeline  │ (ingest pipeline warnings test-audit-events.log)                  │ PASS   │ 310.830633ms │
│ mimecast │ audit_events                  │ pipeline  │ test-audit-events.log                                             │ PASS   │ 437.827176ms │
│ mimecast │ cloud_integrated_logs         │ pipeline  │ (ingest pipeline warnings test-cloud-integrated-logs.log)         │ PASS   │ 301.083562ms │
│ mimecast │ cloud_integrated_logs         │ pipeline  │ test-cloud-integrated-logs.log                                    │ PASS   │ 282.356154ms │
│ mimecast │ dlp_logs                      │ pipeline  │ (ingest pipeline warnings test-dlp-logs.log)                      │ PASS   │ 307.648848ms │
│ mimecast │ dlp_logs                      │ pipeline  │ test-dlp-logs.log                                                 │ PASS   │ 150.602666ms │
│ mimecast │ message_release_logs          │ pipeline  │ (ingest pipeline warnings test-release-logs.json)                 │ PASS   │ 319.872628ms │
│ mimecast │ message_release_logs          │ pipeline  │ test-release-logs.json                                            │ PASS   │ 182.077153ms │
│ mimecast │ siem_logs                     │ pipeline  │ (ingest pipeline warnings test-siem-docs-logs.log)                │ PASS   │ 311.991513ms │
│ mimecast │ siem_logs                     │ pipeline  │ (ingest pipeline warnings test-siem-v1-logs.log)                  │ PASS   │ 338.505978ms │
│ mimecast │ siem_logs                     │ pipeline  │ (ingest pipeline warnings test-siem-v2-logs.log)                  │ PASS   │ 325.435147ms │
│ mimecast │ siem_logs                     │ pipeline  │ test-siem-docs-logs.log                                           │ PASS   │ 212.246677ms │
│ mimecast │ siem_logs                     │ pipeline  │ test-siem-v1-logs.log                                             │ PASS   │ 217.540131ms │
│ mimecast │ siem_logs                     │ pipeline  │ test-siem-v2-logs.log                                             │ PASS   │ 761.126708ms │
│ mimecast │ threat_intel_malware_customer │ pipeline  │ (ingest pipeline warnings test-threat-intel-malware-customer.log) │ PASS   │ 298.566802ms │
│ mimecast │ threat_intel_malware_customer │ pipeline  │ test-threat-intel-malware-customer.log                            │ PASS   │ 243.381339ms │
│ mimecast │ threat_intel_malware_grid     │ pipeline  │ (ingest pipeline warnings test-threat-intel-malware-grid.log)     │ PASS   │ 300.580692ms │
│ mimecast │ threat_intel_malware_grid     │ pipeline  │ test-threat-intel-malware-grid.log                                │ PASS   │ 209.518905ms │
│ mimecast │ ttp_ap_logs                   │ pipeline  │ (ingest pipeline warnings test-ttp-ap-logs.log)                   │ PASS   │  311.30106ms │
│ mimecast │ ttp_ap_logs                   │ pipeline  │ test-ttp-ap-logs.log                                              │ PASS   │ 177.949521ms │
│ mimecast │ ttp_ip_logs                   │ pipeline  │ (ingest pipeline warnings test-ttp-ip-logs.log)                   │ PASS   │ 375.385615ms │
│ mimecast │ ttp_ip_logs                   │ pipeline  │ test-ttp-ip-logs.log                                              │ PASS   │   113.9369ms │
│ mimecast │ ttp_url_logs                  │ pipeline  │ (ingest pipeline warnings test-ttp-url-logs.log)                  │ PASS   │ 308.547008ms │
│ mimecast │ ttp_url_logs                  │ pipeline  │ test-ttp-url-logs.log                                             │ PASS   │ 192.192675ms │
╰──────────┴───────────────────────────────┴───────────┴───────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: mimecast - END   ---
Done

--- Test results for package: mimecast - START ---
╭──────────┬───────────────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE  │ DATA STREAM                   │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├──────────┼───────────────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ mimecast │ archive_search_logs           │ system    │ v1        │ PASS   │ 37.721527525s │
│ mimecast │ archive_search_logs           │ system    │ v2        │ PASS   │ 35.758943964s │
│ mimecast │ audit_events                  │ system    │ v1        │ PASS   │ 45.305248179s │
│ mimecast │ audit_events                  │ system    │ v2        │ PASS   │ 38.429750746s │
│ mimecast │ cloud_integrated_logs         │ system    │ default   │ PASS   │ 35.757686076s │
│ mimecast │ dlp_logs                      │ system    │ v1        │ PASS   │ 48.201899082s │
│ mimecast │ dlp_logs                      │ system    │ v2        │ PASS   │ 36.712307332s │
│ mimecast │ message_release_logs          │ system    │ default   │ PASS   │ 38.835097978s │
│ mimecast │ siem_logs                     │ system    │ v1        │ PASS   │ 36.605111926s │
│ mimecast │ siem_logs                     │ system    │ v2        │ PASS   │ 39.783404246s │
│ mimecast │ threat_intel_malware_customer │ system    │ v1        │ PASS   │ 39.585065846s │
│ mimecast │ threat_intel_malware_customer │ system    │ v2        │ PASS   │ 40.705041755s │
│ mimecast │ threat_intel_malware_grid     │ system    │ v1        │ PASS   │ 38.590748061s │
│ mimecast │ threat_intel_malware_grid     │ system    │ v2        │ PASS   │ 41.711608657s │
│ mimecast │ ttp_ap_logs                   │ system    │ v1        │ PASS   │ 47.704400582s │
│ mimecast │ ttp_ap_logs                   │ system    │ v2        │ PASS   │ 32.713646644s │
│ mimecast │ ttp_ip_logs                   │ system    │ v1        │ PASS   │ 49.082286029s │
│ mimecast │ ttp_ip_logs                   │ system    │ v2        │ PASS   │ 39.419874431s │
│ mimecast │ ttp_url_logs                  │ system    │ v1        │ PASS   │ 48.508968133s │
│ mimecast │ ttp_url_logs                  │ system    │ v2        │ PASS   │ 37.688238246s │
╰──────────┴───────────────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: mimecast - END   ---
Done

Related issues

Screenshots

@navnit-elastic navnit-elastic self-assigned this May 7, 2025
@navnit-elastic navnit-elastic requested a review from a team as a code owner May 7, 2025 09:34
@navnit-elastic navnit-elastic added breaking change Integration:mimecast Mimecast (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels May 7, 2025
@elasticmachine
Copy link

elasticmachine commented May 7, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

ShourieG
ShourieG reviewed May 7, 2025
View reviewed changes
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented May 7, 2025 •
edited
Loading

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@ShourieG ShourieG mentioned this pull request May 9, 2025
Open
@elasticmachine
Copy link

elasticmachine commented May 22, 2025

💚 Build Succeeded

History

cc @navnit-elastic

@elastic-sonarqube
Copy link

elastic-sonarqube bot commented May 22, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@navnit-elastic navnit-elastic requested a review from ShourieG May 23, 2025 05:08
ShourieG
ShourieG approved these changes May 23, 2025
View reviewed changes
Copy link
Contributor

@ShourieG ShourieG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShourieG ShourieG merged commit 31d18b5 into elastic:main May 23, 2025
8 checks passed
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented May 23, 2025

Package mimecast - 3.0.0 containing this change is available at https://epr.elastic.co/package/mimecast/3.0.0/

v1v added a commit to v1v/integrations that referenced this pull request May 26, 2025
@v1v
Merge branch 'main' into feature/use-google-secrets
52daeaf 
* main: (42 commits)
  [jamf_pro] Fix `flattened` field types for non-object values (elastic#13985)
  [Netskope Alerts] Add text multi-field to netskope.alerts.breach.description field (elastic#13977)
  zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams (elastic#13904)
  apm: Add config for tail-based sampling discard on write (elastic#13950)
  [CI] Add dev/coverage into backport script (elastic#13987)
  Update configuration updatecli for 8.x snapshot (elastic#13981)
  [Prometheus] Add username, password, and SSL related fields for query dataset (elastic#13969)
  o365: Ignore failures in rename processors for organization fields (elastic#13983)
  aws.firewall: Document ingested log types of AWS Network Firewall (elastic#13978)
  mimecast: resolve field data type conflicts between data streams (elastic#13825)
  [Infoblox NIOS] Handle the parsing of IPv6 address (elastic#13947)
  [Cribl] Fix handling of metric event type (elastic#13930)
  zscaler_zpa: fix handling of multiple remote IPs, and event categorisation (elastic#13755)
  Adding agentless deployment to the sublime security integration (elastic#13963)
  [integration/system] add use_performance_counters in system integration (elastic#13150)
  crowdstrike,m365_defender,microsoft_defender_{cloud,endpoint},sentinel_one: normalise severity handling (elastic#13955)
  [forgerock] Map `forgerock.response.elapsedTime` as a long not a date (elastic#13959)
  github: squelch errors from pagination ends (elastic#13965)
  cisco_secure_endpoint: squelch errors from pagination ends (elastic#13964)
  [Cloud Security] Cloud Asset Inventory:  fixed cloud formation URL (elastic#13971)
  ...
v1v added a commit that referenced this pull request May 26, 2025
@v1v
Merge branch 'feature/use-google-secrets' into test/use-google-secrets
81b8efa 
* feature/use-google-secrets: (43 commits)
  use -ci account
  [jamf_pro] Fix `flattened` field types for non-object values (#13985)
  [Netskope Alerts] Add text multi-field to netskope.alerts.breach.description field (#13977)
  zscaler_zia: add strict field template mode for tcp and http_endpoint input data streams (#13904)
  apm: Add config for tail-based sampling discard on write (#13950)
  [CI] Add dev/coverage into backport script (#13987)
  Update configuration updatecli for 8.x snapshot (#13981)
  [Prometheus] Add username, password, and SSL related fields for query dataset (#13969)
  o365: Ignore failures in rename processors for organization fields (#13983)
  aws.firewall: Document ingested log types of AWS Network Firewall (#13978)
  mimecast: resolve field data type conflicts between data streams (#13825)
  [Infoblox NIOS] Handle the parsing of IPv6 address (#13947)
  [Cribl] Fix handling of metric event type (#13930)
  zscaler_zpa: fix handling of multiple remote IPs, and event categorisation (#13755)
  Adding agentless deployment to the sublime security integration (#13963)
  [integration/system] add use_performance_counters in system integration (#13150)
  crowdstrike,m365_defender,microsoft_defender_{cloud,endpoint},sentinel_one: normalise severity handling (#13955)
  [forgerock] Map `forgerock.response.elapsedTime` as a long not a date (#13959)
  github: squelch errors from pagination ends (#13965)
  cisco_secure_endpoint: squelch errors from pagination ends (#13964)
  ...
anupratharamachandran pushed a commit to anupratharamachandran/integrations that referenced this pull request Jun 2, 2025
@navnit-elastic @anupratharamachandran
mimecast: resolve field data type conflicts between data streams (ela…
c205f10 
…stic#13825)

mimecast: resolve field data type conflicts between data streams. 

This introduces a breaking change by changing the field type from 'keyword' 
to 'boolean'  for the 'taggedExternal' and 'taggedMalicious' fields in the
'siem_logs' data stream to  resolve conflicts with other data streams.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ShourieG ShourieG ShourieG approved these changes

Assignees

@navnit-elastic navnit-elastic

Labels

breaking change Integration:mimecast Mimecast (Partner supported) Integration:ti_misp MISP Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[mimecast] Field data type conflicts between data streams

4 participants

@navnit-elastic @elasticmachine @ShourieG @andrewkroh