[Bug] DAC - Timeline rules that validate fine in Kibana fails validation in DAC

[0xBAADF0OD]

Describe the Bug

Hi

I have a few timeline rules that are running fine, but when I attempt to export via DAC, they throw errors. It looks like the DAC validation only looks for the default timeline template uuids.

TEST_TIMELINE_RULE - {'rule': [ValidationError({'type': ['Must be equal to threshold.'], 'timeline_id': ['Must be one of: db366523-f1c6-4c1f-8731-6ce5ed9e5717, 91832785-286d-4ebe-b884-1a208d111a70, 76e52245-7519-4251-91ab-262fb1a1728c, 495ad7a7-316e-4544-8a0f-9c098daee76e, 4d4c0b59-ea83-483f-b8c1-8c360ee53c5c, e70679c2-6cde-4510-9764-4823df18f7db, 300afc76-072d-4261-864d-4149714bf3f1, 3e47ef71-ebfc-4520-975c-cb27fc090799, 3e827bab-838a-469f-bd1e-5e19a2bff2fd, 4434b91a-94ca-4a89-83cb-a37cdc0532b7.'], 'timeline_title': ['Must be one of: Generic Endpoint Timeline, Generic Network Timeline, Generic Process Timeline, Generic Threat Match Timeline, Comprehensive File Timeline, Comprehensive Process Timeline, Comprehensive Network Timeline, Comprehensive Registry Timeline, Alerts Involving a Single User Timeline, Alerts Involving a Single Host Timeline.'], 'threshold': ['Missing data for required field.']}),

The docs mention limited API coverage, but is there any way to bypass these validation checks so we can at least export the rules into our repo?

To Reproduce

No response

Expected Behavior

No response

Screenshots

No response

Desktop - OS

None

Desktop - Version

No response

Additional Context

No response

[eric-forte-elastic]

Hi @0xBAADF0OD thanks for writing up this issue! 👋

Yes, this looks like an error that one could expect from our optional validation. Just checking have you tried using the bypass_optional_elastic_validation flag in your config?

Instructions ref

[0xBAADF0OD]

@eric-forte-elastic Thanks for the suggestion. We confirmed that bypass_optional_elastic_validation is set to true in config, but we're still running into very random errors. All of these rules were created in Kibana but fails to export

(these are separate errors)

{'rule': [ValidationError({'name': ['String does not match expected pattern.'], 'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'name': ['String does not match expected pattern.'], 'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'name': ['String does not match expected pattern.']}), ValidationError({'name': ['String does not match expected pattern.'], 'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'name': ['String does not match expected pattern.'], 'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'name': ['String does not match expected pattern.'], 'type': ['Must be equal to query.']}), ValidationError({'name': ['String does not match expected pattern.'], 'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}

Error at line:1,column:112 Expected 'or' but got 'OR'

'utf-8' codec can't decode byte 0x92 in position 315: invalid start byte

Error at line:2,column:163 Field not recognized for iam event sequence

[eric-forte-elastic]

@0xBAADF0OD roger. It looks like the timeline rules error is no longer happening? Could you share the export command you are using? Or are you importing from a local .ndjson file? Thanks!