Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
45c7bc5
Adding the proto and the dependencies
grundprinzip Jun 16, 2022
c2c6de0
Trying to fix aarch64
grundprinzip Jul 5, 2022
034a44d
Moving connect to its own module
grundprinzip Jul 7, 2022
0cedf00
[SparkConnect] Spark Connect Planner
grundprinzip Jul 4, 2022
5d4b8dd
adding the planner
grundprinzip Jul 25, 2022
506fc44
adding the python code
grundprinzip Jul 26, 2022
0f6a687
fixing tests
grundprinzip Jul 26, 2022
9adf4bd
Adding some very basic unit tests
grundprinzip Jul 29, 2022
ed2fe24
adding explain to the service
grundprinzip Jul 29, 2022
786efbb
adding apache header and small col function
grundprinzip Jul 31, 2022
8693efb
more apache headers
grundprinzip Jul 31, 2022
d336b2a
Adding some more tests
grundprinzip Jul 31, 2022
dbe3d1f
adding some more doc
grundprinzip Jul 31, 2022
3ea55ca
some basic tests for literals
grundprinzip Jul 31, 2022
127a492
Fixing the pom files
grundprinzip Aug 29, 2022
4c63000
Moving to grpc-java
grundprinzip Aug 29, 2022
b713969
Fixing some python tests
grundprinzip Aug 29, 2022
0547e07
Fixing more python tests
grundprinzip Aug 29, 2022
074e1f7
making connect build by default
grundprinzip Aug 29, 2022
e77a018
fixing sbt build
grundprinzip Aug 29, 2022
21224ec
More sbt stuff
grundprinzip Aug 30, 2022
e22975f
More sbt stuff
grundprinzip Aug 30, 2022
9601449
More sbt stuff
grundprinzip Aug 31, 2022
b3ab663
properly shaded spark connect build
grundprinzip Sep 2, 2022
8968f1e
SBT build and testing works
grundprinzip Sep 4, 2022
87a50d4
Python linting
grundprinzip Sep 4, 2022
fefc84a
Restricting enaling Spark Connect to sepcific tests only and fixing t…
grundprinzip Sep 4, 2022
193e6b0
scala 2.13 fix
grundprinzip Sep 4, 2022
e1f862e
Fixin python lint issues
grundprinzip Sep 4, 2022
d5b6002
Fixin python lint issues
grundprinzip Sep 4, 2022
8bb03d2
Marking all classes in Spark Connect as experimental
grundprinzip Sep 4, 2022
265aca3
Fixing style
grundprinzip Sep 4, 2022
d6f64e8
test infra
grundprinzip Sep 5, 2022
3d00bb0
Missing file for disabling mypy checks
grundprinzip Sep 5, 2022
3525eab
Trying to add a python package the right way
grundprinzip Sep 5, 2022
7e1dd58
Removing grpc/protobuf from pypy
grundprinzip Sep 5, 2022
fa9e85e
Disabling pypy for spark connect
grundprinzip Sep 5, 2022
cf6b19a
Adding licence files
grundprinzip Sep 5, 2022
16bf911
Fixing a bug in UDF handling for Python version selection
grundprinzip Sep 5, 2022
762104a
Trailing newlines
grundprinzip Sep 5, 2022
ca4cfbb
Adding dependency manaifest
grundprinzip Sep 5, 2022
79bbfd6
Moving to classs
grundprinzip Sep 6, 2022
ec1221b
Merge branch 'master' into spark-connect-grpc-shaded
grundprinzip Sep 6, 2022
63d8ebc
dependencies
grundprinzip Sep 6, 2022
70ad818
M2 cache bomb
grundprinzip Sep 6, 2022
ae75ca4
More m2 removal
grundprinzip Sep 6, 2022
7e615f7
Revert "More m2 removal"
HyukjinKwon Sep 7, 2022
8effefb
Revert "M2 cache bomb"
HyukjinKwon Sep 7, 2022
aa68119
Add Python 3.7
HyukjinKwon Sep 7, 2022
fcefe9e
Revert "Add Python 3.7"
HyukjinKwon Sep 7, 2022
ff8e4ad
Merge remote-tracking branch 'upstream/master' into HEAD
HyukjinKwon Sep 7, 2022
b2f6548
Python 3.7
HyukjinKwon Sep 7, 2022
feb0233
Revert "Python 3.7"
HyukjinKwon Sep 7, 2022
4b02eb3
Tweaking memory consumption of SBT
grundprinzip Sep 7, 2022
aec7e3d
Disabling paralell execution for SBT in doc build
grundprinzip Sep 7, 2022
e4485c3
Avoiding copying the shaded jar in doc build
grundprinzip Sep 8, 2022
4ef5a35
Sbt doc build still
grundprinzip Sep 8, 2022
e13b955
test package build
grundprinzip Sep 8, 2022
6c96205
only test package build
grundprinzip Sep 9, 2022
d1e3c13
using exact pyspark build options
grundprinzip Sep 9, 2022
9d43765
Moving things around
grundprinzip Sep 9, 2022
448fcc8
Desperate attempts
grundprinzip Sep 9, 2022
b3d2d8c
Java 11 anyone?
grundprinzip Sep 9, 2022
b2c6bfd
Revert "Java 11 anyone?"
grundprinzip Sep 13, 2022
11ad9b6
disable tests on assembly run
grundprinzip Sep 13, 2022
7617b82
Adding additional tests
grundprinzip Sep 14, 2022
ce4900b
format + slight python api change
grundprinzip Sep 14, 2022
28e7741
update on readme and import
grundprinzip Sep 14, 2022
8223615
Properly catching exceptions and removing stray debug user
grundprinzip Sep 18, 2022
8bf14f2
scalastyle
grundprinzip Sep 18, 2022
87ba6f2
doc fix
grundprinzip Sep 18, 2022
45f23e0
addrsessing review comments
grundprinzip Sep 20, 2022
4971980
removing embedded protos for 3p google
grundprinzip Sep 20, 2022
4aafab8
renaming
grundprinzip Sep 20, 2022
38b69ce
Adding @Since annotation
grundprinzip Sep 20, 2022
07b0ec8
Fixing python test with the right package
grundprinzip Sep 20, 2022
f47b8e9
fixing build error due to package refactoring
grundprinzip Sep 20, 2022
b57cbd2
Sql -> SQL
grundprinzip Sep 21, 2022
77470cd
Scala review comments
grundprinzip Sep 22, 2022
ee13ae2
Build file review
grundprinzip Sep 22, 2022
51be506
Python review comments
grundprinzip Sep 22, 2022
549a10e
Python review comments
grundprinzip Sep 22, 2022
b0608f3
black fmt
grundprinzip Sep 22, 2022
279faf4
Addressing review comments
grundprinzip Sep 23, 2022
e5e2347
Addressing review comments
grundprinzip Sep 23, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Trying to add a python package the right way
  • Loading branch information
grundprinzip committed Sep 5, 2022
commit 3525eabb6290e58b13f7f263e115e601eebc69aa
2 changes: 1 addition & 1 deletion dev/create-release/spark-rm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ ARG APT_INSTALL="apt-get install --no-install-recommends -y"
# We should use the latest Sphinx version once this is fixed.
# TODO(SPARK-35375): Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
ARG PIP_PKGS="sphinx==3.0.4 mkdocs==1.1.2 numpy==1.19.4 pydata_sphinx_theme==0.4.1 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==2.11.3 twine==3.4.1 sphinx-plotly-directive==0.1.3 pandas==1.1.5 pyarrow==3.0.0 plotly==5.4.0 markupsafe==2.0.1 docutils<0.17"
ARG PIP_PKGS="sphinx==3.0.4 mkdocs==1.1.2 numpy==1.19.4 pydata_sphinx_theme==0.4.1 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==2.11.3 twine==3.4.1 sphinx-plotly-directive==0.1.3 pandas==1.1.5 pyarrow==3.0.0 plotly==5.4.0 markupsafe==2.0.1 docutils<0.17 grpcio==1.48.1 protobuf==4.21.5"
ARG GEM_PKGS="bundler:2.2.9"

# Install extra needed repos and refresh.
Expand Down
2 changes: 1 addition & 1 deletion dev/infra/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ RUN mkdir -p /usr/local/pypy/pypy3.7 && \
ln -sf /usr/local/pypy/pypy3.7/bin/pypy /usr/local/bin/pypy3

RUN curl -sS https://bootstrap.pypa.io/get-pip.py | pypy3
RUN pypy3 -m pip install numpy 'pandas<=1.4.3' scipy coverage matplotlib
RUN pypy3 -m pip install numpy 'pandas<=1.4.3' scipy coverage matplotlib grpcio protobuf

RUN $APT_INSTALL gnupg ca-certificates pandoc
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/' >> /etc/apt/sources.list
Expand Down
4 changes: 2 additions & 2 deletions dev/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -47,5 +47,5 @@ PyGithub
black==22.6.0

# Spark Connect
grpcio=1.48.1
protobuf=4.21.5
grpcio==1.48.1
protobuf==4.21.5