Skip to content
Closed
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
7dec5eb
[SPARK-47705][INFRA] Sort LogKey alphabetically and build a test to e…
dtenedor Apr 3, 2024
6a0555c
[SPARK-47700][SQL] Fix formatting of error messages with treeNode
jchen5 Apr 3, 2024
49eefc5
[SPARK-47722][SS] Wait until RocksDB background work finish before cl…
WweiL Apr 3, 2024
fbe6b1d
[SPARK-47721][DOC] Guidelines for the Structured Logging Framework
gengliangwang Apr 3, 2024
e3aab8c
[SPARK-47210][SQL] Addition of implicit casting without indeterminate…
mihailomilosevic2001 Apr 3, 2024
d87ac8e
[SPARK-47708][CONNECT] Do not log gRPC exception to stderr in PySpark
nemanja-boric-databricks Apr 4, 2024
447f8af
[SPARK-47720][CORE] Update `spark.speculation.multiplier` to 3 and `s…
dongjoon-hyun Apr 4, 2024
678aeb7
[SPARK-47683][PYTHON][BUILD] Decouple PySpark core API to pyspark.cor…
HyukjinKwon Apr 4, 2024
c25fd93
[SPARK-47705][INFRA][FOLLOWUP] Sort LogKey alphabetically and build a…
panbingkun Apr 4, 2024
d272a1b
[SPARK-47724][PYTHON][TESTS] Add an environment variable for testing …
HyukjinKwon Apr 4, 2024
d75c775
[SPARK-46812][PYTHON][TESTS][FOLLOWUP] Skip `pandas`-required tests i…
dongjoon-hyun Apr 4, 2024
3f6ac60
[SPARK-47577][CORE][PART1] Migrate logError with variables to structu…
gengliangwang Apr 4, 2024
f6999df
[SPARK-47081][CONNECT] Support Query Execution Progress
grundprinzip Apr 4, 2024
bffb02d
[SPARK-47565][PYTHON] PySpark worker pool crash resilience
Apr 4, 2024
3b8aea3
Revert "[SPARK-47708][CONNECT] Do not log gRPC exception to stderr in…
nemanja-boric-databricks Apr 4, 2024
5f9f5db
[SPARK-47689][SQL][FOLLOWUP] More accurate file path in TASK_WRITE_FA…
cloud-fan Apr 4, 2024
5ca3467
[SPARK-47729][PYTHON][TESTS] Get the proper default port for pyspark-…
HyukjinKwon Apr 4, 2024
25fc67f
[SPARK-47728][DOC] Document G1 Concurrent GC metrics
LucaCanali Apr 4, 2024
e3405c1
[SPARK-47610][CONNECT][FOLLOWUP] Add -Dio.netty.tryReflectionSetAcces…
pan3793 Apr 4, 2024
3fd0cd6
[SPARK-47598][CORE] MLLib: Migrate logError with variables to structu…
panbingkun Apr 4, 2024
240923c
[SPARK-46812][PYTHON][TESTS][FOLLOWUP] Check should_test_connect and …
dongjoon-hyun Apr 4, 2024
fb96b1a
[SPARK-47723][CORE][TESTS] Introduce a tool that can sort alphabetica…
panbingkun Apr 5, 2024
404d58c
[SPARK-47081][CONNECT][FOLLOW-UP] Add the `shell` module into PyPI pa…
HyukjinKwon Apr 5, 2024
b9ca91d
[SPARK-47712][CONNECT] Allow connect plugins to create and process Da…
tomvanbussel Apr 5, 2024
0107435
[SPARK-47734][PYTHON][TESTS] Fix flaky DataFrame.writeStream doctest …
JoshRosen Apr 5, 2024
d5620cb
[SPARK-47289][SQL] Allow extensions to log extended information in ex…
parthchandra Apr 5, 2024
aeb082e
[SPARK-47081][CONNECT][TESTS][FOLLOW-UP] Skip the flaky doctests for now
HyukjinKwon Apr 5, 2024
97e63ff
[SPARK-47735][PYTHON][TESTS] Make pyspark.testing.connectutils compat…
HyukjinKwon Apr 5, 2024
12d0367
[SPARK-47724][PYTHON][TESTS][FOLLOW-UP] Make testing script to inheri…
HyukjinKwon Apr 5, 2024
6bd0ccf
[SPARK-47511][SQL][FOLLOWUP] Rename the config REPLACE_NULLIF_USING_W…
cloud-fan Apr 5, 2024
c34baeb
[SPARK-47719][SQL] Change spark.sql.legacy.timeParserPolicy default t…
srielau Apr 5, 2024
18072b5
[SPARK-47577][CORE][PART2] Migrate logError with variables to structu…
gengliangwang Apr 5, 2024
1efbf43
[SPARK-47310][SS] Add micro-benchmark for merge operations for multip…
anishshri-db Apr 5, 2024
d1ace24
[SPARK-47582][SQL] Migrate Catalyst logInfo with variables to structu…
dtenedor Apr 5, 2024
11abc64
[SPARK-47094][SQL] SPJ : Dynamically rebalance number of buckets when…
szehon-ho Apr 6, 2024
42dc815
[SPARK-47743][CORE] Use milliseconds as the time unit in logging
gengliangwang Apr 6, 2024
7385f19
[SPARK-47592][CORE] Connector module: Migrate logError with variables…
panbingkun Apr 6, 2024
d69df59
[SPARK-47738][BUILD] Upgrade Kafka to 3.7.0
panbingkun Apr 6, 2024
60a3fbc
[SPARK-47727][PYTHON] Make SparkConf to root level to for both SparkS…
HyukjinKwon Apr 6, 2024
644687b
[SPARK-47709][BUILD] Upgrade tink to 1.13.0
LuciferYang Apr 6, 2024
4d9dbb3
[SPARK-46722][CONNECT][SS][TESTS][FOLLOW-UP] Drop the tables after te…
HyukjinKwon Apr 7, 2024
c11585a
[SPARK-47751][PYTHON][CONNECT] Make pyspark.worker_utils compatible w…
HyukjinKwon Apr 7, 2024
d743012
[SPARK-47753][PYTHON][CONNECT][TESTS] Make pyspark.testing compatible…
HyukjinKwon Apr 7, 2024
f7dff4a
[SPARK-47752][PS][CONNECT] Make pyspark.pandas compatible with pyspar…
HyukjinKwon Apr 7, 2024
e92e8f5
[SPARK-47744] Add support for negative-valued bytes in range encoder
neilramaswamy Apr 7, 2024
0c992b2
[SPARK-47755][CONNECT] Pivot should fail when the number of distinct …
zhengruifeng Apr 7, 2024
b299b2b
[SPARK-47299][PYTHON][DOCS] Use the same `versions.json` in the dropd…
panbingkun Apr 8, 2024
cc6c0eb
[MINOR][TESTS] Deduplicate test cases `test_parse_datatype_string`
HyukjinKwon Apr 8, 2024
ad2367c
[MINOR][PYTHON][SS][TESTS] Drop the tables after being used at `test_…
HyukjinKwon Apr 8, 2024
f576b85
[SPARK-47541][SQL] Collated strings in complex types supporting opera…
nikolamand-db Apr 8, 2024
d55bb61
[SPARK-47558][SS] State TTL support for ValueState
sahnib Apr 8, 2024
3a39ac2
[SPARK-47713][SQL][CONNECT] Fix a self-join failure
zhengruifeng Apr 8, 2024
eb8e997
[SPARK-47657][SQL] Implement collation filter push down support per f…
stefankandic Apr 8, 2024
f0d8f82
[SPARK-47750][DOCS][SQL] Postgres: Document Mapping Spark SQL Data Ty…
yaooqinn Apr 8, 2024
211afd4
[MINOR][PYTHON][CONNECT][TESTS] Enable `MapInPandasParityTests.test_d…
zhengruifeng Apr 8, 2024
f94d95d
[SPARK-47762][PYTHON][CONNECT] Add pyspark.sql.connect.protobuf into …
HyukjinKwon Apr 8, 2024
29d077f
[SPARK-47748][BUILD] Upgrade `zstd-jni` to 1.5.6-2
panbingkun Apr 8, 2024
60806c6
[SPARK-47746] Implement ordinal-based range encoding in the RocksDBSt…
neilramaswamy Apr 8, 2024
134a139
[SPARK-47681][SQL] Add schema_of_variant expression
chenhao-db Apr 8, 2024
abb7b04
[SPARK-47504][SQL] Resolve AbstractDataType simpleStrings for StringT…
mihailomilosevic2001 Apr 8, 2024
91b2331
[WIP] ListStateTTL implementation
ericm-db Apr 8, 2024
479392a
adding log lines
ericm-db Apr 8, 2024
7aab43e
test cases pass
ericm-db Apr 8, 2024
71f960d
spacing
ericm-db Apr 8, 2024
998764c
using NextIterator instead
ericm-db Apr 8, 2024
1dcb7d8
refactor feedback
ericm-db Apr 9, 2024
47867e7
undoing unnecessary change
ericm-db Apr 9, 2024
cfd30c3
refactor get_ttl_value
ericm-db Apr 9, 2024
4a19cb7
refactor test case
ericm-db Apr 9, 2024
993125c
specific doc for clearIfExpired
ericm-db Apr 9, 2024
fd5200f
moving isExpired to common place
ericm-db Apr 9, 2024
d43ffb1
refactoring to use common utils
ericm-db Apr 9, 2024
30f6094
updating interface header
ericm-db Apr 9, 2024
e9376d9
Map State TTL, Initial Commit
ericm-db Apr 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-47724][PYTHON][TESTS] Add an environment variable for testing …
…remote pure Python library

### What changes were proposed in this pull request?

This PR proposes to add an environment variable called `SPARK_CONNECT_TESTING_REMOTE` to set `remote` URL.

### Why are the changes needed?

In order to test pure Python library with a remote server.

### Does this PR introduce _any_ user-facing change?

No, test-only.

### How was this patch tested?

Manually tested.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#45868 from HyukjinKwon/SPARK-47724.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
  • Loading branch information
HyukjinKwon committed Apr 4, 2024
commit d272a1b4367edac03ba1de91551573c8903dd8d8
2 changes: 1 addition & 1 deletion python/pyspark/testing/connectutils.py
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ def conf(cls):

@classmethod
def master(cls):
return "local[4]"
return os.environ.get("SPARK_CONNECT_TESTING_REMOTE", "local[4]")

@classmethod
def setUpClass(cls):
Expand Down