Disable Spark Catalog caching for integration tests

[kevinjqliu]

Fix #482

While working on #444, I couldn't get integration tests working since using spark.sql to count the number of data files returns the wrong result. This becomes more bizarre when Python's iceberg table state is not the same as Spark's iceberg table state.

This PR adds a simple test to verify that "python's iceberg table snapshot id" is the same as "spark's iceberg table's snapshot id".

The culprit is the spark.sql.catalog.catalog-name.cache-enabled setting, which defaults to True and caches table metadata.
Spark sql calls will use the cached iceberg metadata instead of the updated one.

spark.sql.catalog.catalog-name.cache-enabled	true or false	Whether to enable catalog cache, default value is true

https://iceberg.apache.org/docs/latest/spark-configuration/#catalog-configuration

[amogh-jahagirdar]

Thanks @kevinjqliu I think this change makes sense. I don't think there's ever a reason on the Python side where we want to have the spark caching enabled. On the Iceberg Java side we do have tests which validate the caching catalog behavior when it's enabled/disabled so we don't need to test that through PyIceberg (I think). I've triggered CI if it passes, I'll go ahead and merge

[amogh-jahagirdar]

Nit: I think a better name would be test_python_writes_with_spark_snapshot_reads or something more specific than what it currently is . It's mre verbose but I think it captures the goal of the test better

[kevinjqliu]

make sense, added!

[sungwy]

Great idea @kevinjqliu ! Thanks for adding this

[amogh-jahagirdar]

Sweet, thanks @kevinjqliu! I'm going to go ahead and merge this now.