[SPARK-48766][PYTHON] Document the behavior difference of extraction between element_at and try_element_at

[zhengruifeng]

What changes were proposed in this pull request?

Document the behavior difference of extraction between element_at and try_element_at

Why are the changes needed?

when the function try_element_at was introduced in 3.5, its extraction handling was unintentionally not consistent with the element_at, which causes confusion.

This PR document this behavior difference (I don't think we can fix it since it will be a breaking change).

In [1]: from pyspark.sql import functions as sf

In [2]: df = spark.createDataFrame([({"a": 1.0, "b": 2.0}, "a")], ['data', 'b'])

In [3]: df.select(sf.try_element_at(df.data, 'b')).show()
+-----------------------+
|try_element_at(data, b)|
+-----------------------+
|                    1.0|
+-----------------------+


In [4]: df.select(sf.element_at(df.data, 'b')).show()
+-------------------+
|element_at(data, b)|
+-------------------+
|                2.0|
+-------------------+

Does this PR introduce any user-facing change?

doc changes

How was this patch tested?

ci, added doctests

Was this patch authored or co-authored using generative AI tooling?

no

[zhengruifeng]

@panbingkun @LuciferYang @HyukjinKwon

[allisonwang-db]

Is this behavior difference intentional? Is it consistent with the SQL functions element_at and try_element_at?

[zhengruifeng]

I think the SQL side also treat it as a literal:

In [2]: spark.sql("SELECT ELEMENT_AT(MAP('a', 'b'), 'a')").show()
+------------------------+
|element_at(map(a, b), a)|
+------------------------+
|                       b|
+------------------------+

[zhengruifeng]

thanks all, merged to master

[panbingkun]

Late LGTM

[panbingkun]

Do we need to add another example below this?

df.select(sf.try_element_at(df.data, df.b)).show()

Because of the grammar scene above, I took a long time to understand its intended meaning, which is so obscure, 😄