Skip to content

Commit 90d3066

Browse files
committed
[SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
### What changes were proposed in this pull request? This PR proposes to avoid using subshell when it activates Conda environment. Looks like it ends up with activating the env within the subshell even if you use `conda` command. ### Why are the changes needed? If you take a close look for GitHub Actions log: ``` Installing dist into virtual env Processing ./python/dist/pyspark-3.1.0.dev0.tar.gz Collecting py4j==0.10.9 Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB) Using legacy setup.py install for pyspark, since package 'wheel' is not installed. Installing collected packages: py4j, pyspark Running setup.py install for pyspark: started Running setup.py install for pyspark: finished with status 'done' Successfully installed py4j-0.10.9 pyspark-3.1.0.dev0 ... Installing dist into virtual env Obtaining file:///home/runner/work/spark/spark/python Collecting py4j==0.10.9 Downloading py4j-0.10.9-py2.py3-none-any.whl (198 kB) Installing collected packages: py4j, pyspark Attempting uninstall: py4j Found existing installation: py4j 0.10.9 Uninstalling py4j-0.10.9: Successfully uninstalled py4j-0.10.9 Attempting uninstall: pyspark Found existing installation: pyspark 3.1.0.dev0 Uninstalling pyspark-3.1.0.dev0: Successfully uninstalled pyspark-3.1.0.dev0 Running setup.py develop for pyspark Successfully installed py4j-0.10.9 pyspark ``` It looks not properly using Conda as it removes the previously installed one when it reinstalls again. We should ideally test it with Conda environment as it's intended. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? GitHub Actions will test. I also manually tested in my local. Closes apache#29212 from HyukjinKwon/SPARK-32419. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent 8a4b221 commit 90d3066

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

dev/run-pip-tests

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ for python in "${PYTHON_EXECS[@]}"; do
8686
source "$CONDA_PREFIX/etc/profile.d/conda.sh"
8787
fi
8888
conda create -y -p "$VIRTUALENV_PATH" python=$python numpy pandas pip setuptools
89-
source activate "$VIRTUALENV_PATH" || (echo "Falling back to 'conda activate'" && conda activate "$VIRTUALENV_PATH")
89+
source activate "$VIRTUALENV_PATH" || conda activate "$VIRTUALENV_PATH"
9090
else
9191
mkdir -p "$VIRTUALENV_PATH"
9292
virtualenv --python=$python "$VIRTUALENV_PATH"
@@ -129,7 +129,7 @@ for python in "${PYTHON_EXECS[@]}"; do
129129

130130
# conda / virtualenv environments need to be deactivated differently
131131
if [ -n "$USE_CONDA" ]; then
132-
source deactivate || (echo "Falling back to 'conda deactivate'" && conda deactivate)
132+
source deactivate || conda deactivate
133133
else
134134
deactivate
135135
fi

0 commit comments

Comments
 (0)