MAGE-1083 Collection size caching

[cammonro]

Summary

This PR aims to reduce overhead related to calculating collection size. As catalogs grow in size the query becomes increasingly less performant. By eliminating redundant COUNT() operations we can shave some processing time here.

This PR includes the following changes:

• Removed rebuildEntityIds() as a relic of Algolia\AlgoliaSearch\Helper\Data::rebuildStoreProductIndex()here
  • It is not necessary to re-page an already chunked array of entityIds
  • Product set on the delta will never be higher than the page size max because of potential parent aggregation for configurables etc.
• Removed collection cloning - was only needed for in-process collection paging on deltas which we are no longer doing
• Removed collection reset ops for memory management - collection use is now limited to function scope
• Paging information is now included on deltas for observability - previously logs misreported page 1 for all jobs on a large delta
  
• Removed areParentsLoaded member boolean - this internal state was observed to be problematic when processing jobs across stores (parents would not be loaded for subsequent stores) - now should be consistent
  
• Moved emulation to encompass entire collection build to ensure full alignment with storefront behavior.
• Introduced an optional cache to store collection size in Redis
• Added a plugin to purge the collection size when products are added or deleted or edited in a way that impacts visibility on the storefront
• Added unit tests

Please note that I chose not to address all stock scenarios for cache invalidation until we more properly address MSI. That can be reevaluated at a later date.

Result

Unit tests:

Caching of collection size in Redis:

New indexing cache:

Full index job gen - EXTRA LARGE fixture - uncached:

Full index job gen - EXTRA LARGE fixture - CACHED:

[damcou]

Great job ! 🙌

Unit tests are passing and integration tests for products indexing as well:

Added my thoughts, let me know if it makes sense :)

[cammonro]

Thanks for the great feedback @damcou ! I will implement the changes you've proposed (though some might go on a different ticket).

[damcou]

Tests are passing, LGTM 🚀