Skip to content

Conversation

@jshearer
Copy link
Contributor

@jshearer jshearer commented Aug 22, 2023

Description:

  • Replace per-subschema field limit with a global complexity limit, currently set at 1000 total fields.
  • Add a --flow.enable-schema-inference flag to flowctl-go serve consumer to enable schema inference for capture tasks

This change is Reviewable

@jshearer jshearer force-pushed the fix/improve_schema_limits branch from 7d6d596 to 8166b31 Compare August 22, 2023 16:15
Copy link
Member

@jgraettinger jgraettinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments, looking good!

Copy link
Member

@jgraettinger jgraettinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % final comments!

for prop in slf.object.properties.iter_mut() {
enforce_field_count_limits(&mut prop.shape, loc.push_prop(&prop.name))
shape_to_squash.array.additional_items =
squash_addl_properties(shape_to_squash.array.additional_items);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't you just set additional_items and additional_properties to None ?

nit: if you can't and must keep this, then squash_addl_properties => squash_addl ? sometimes item, sometimes property

@jshearer jshearer force-pushed the fix/improve_schema_limits branch from ba809e8 to 8b7196c Compare August 24, 2023 19:20
@jshearer jshearer force-pushed the fix/improve_schema_limits branch from 8b7196c to 441ea5b Compare August 24, 2023 19:24
@jshearer jshearer merged commit ed8ee5f into master Aug 24, 2023
@jgraettinger jgraettinger deleted the fix/improve_schema_limits branch August 24, 2023 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants