Merge remote-tracking branch 'upstream/main' into improve_docstring_check_performance

Author: dontgoto

doc/source/reference/series.rst

@@ -335,7 +335,6 @@ Datetime properties
    Series.dt.tz
    Series.dt.freq
    Series.dt.unit
-   Series.dt.normalize
 
 Datetime methods
 ^^^^^^^^^^^^^^^^

doc/source/user_guide/scale.rst

@@ -156,7 +156,7 @@ fits in memory, you can work with datasets that are much larger than memory.
 
    Chunking works well when the operation you're performing requires zero or minimal
    coordination between chunks. For more complicated workflows, you're better off
-   :ref:`using another library <scale.other_libraries>`.
+   :ref:`using other libraries <scale.other_libraries>`.
 
 Suppose we have an even larger "logical dataset" on disk that's a directory of parquet
 files. Each file in the directory represents a different year of the entire dataset.
@@ -219,188 +219,10 @@ different library that implements these out-of-core algorithms for you.
 
 .. _scale.other_libraries:
 
-Use Dask
---------
+Use Other Libraries
+-------------------
 
-pandas is just one library offering a DataFrame API. Because of its popularity,
-pandas' API has become something of a standard that other libraries implement.
-The pandas documentation maintains a list of libraries implementing a DataFrame API
-in `the ecosystem page <https://pandas.pydata.org/community/ecosystem.html>`_.
-
-For example, `Dask`_, a parallel computing library, has `dask.dataframe`_, a
-pandas-like API for working with larger than memory datasets in parallel. Dask
-can use multiple threads or processes on a single machine, or a cluster of
-machines to process data in parallel.
-
-
-We'll import ``dask.dataframe`` and notice that the API feels similar to pandas.
-We can use Dask's ``read_parquet`` function, but provide a globstring of files to read in.
-
-.. ipython:: python
-   :okwarning:
-
-   import dask.dataframe as dd
-
-   ddf = dd.read_parquet("data/timeseries/ts*.parquet", engine="pyarrow")
-   ddf
-
-Inspecting the ``ddf`` object, we see a few things
-
-* There are familiar attributes like ``.columns`` and ``.dtypes``
-* There are familiar methods like ``.groupby``, ``.sum``, etc.
-* There are new attributes like ``.npartitions`` and ``.divisions``
-
-The partitions and divisions are how Dask parallelizes computation. A **Dask**
-DataFrame is made up of many pandas :class:`pandas.DataFrame`. A single method call on a
-Dask DataFrame ends up making many pandas method calls, and Dask knows how to
-coordinate everything to get the result.
-
-.. ipython:: python
-
-   ddf.columns
-   ddf.dtypes
-   ddf.npartitions
-
-One major difference: the ``dask.dataframe`` API is *lazy*. If you look at the
-repr above, you'll notice that the values aren't actually printed out; just the
-column names and dtypes. That's because Dask hasn't actually read the data yet.
-Rather than executing immediately, doing operations build up a **task graph**.
-
-.. ipython:: python
-   :okwarning:
-
-   ddf
-   ddf["name"]
-   ddf["name"].value_counts()
-
-Each of these calls is instant because the result isn't being computed yet.
-We're just building up a list of computation to do when someone needs the
-result. Dask knows that the return type of a :class:`pandas.Series.value_counts`
-is a pandas :class:`pandas.Series` with a certain dtype and a certain name. So the Dask version
-returns a Dask Series with the same dtype and the same name.
-
-To get the actual result you can call ``.compute()``.
-
-.. ipython:: python
-   :okwarning:
-
-   %time ddf["name"].value_counts().compute()
-
-At that point, you get back the same thing you'd get with pandas, in this case
-a concrete pandas :class:`pandas.Series` with the count of each ``name``.
-
-Calling ``.compute`` causes the full task graph to be executed. This includes
-reading the data, selecting the columns, and doing the ``value_counts``. The
-execution is done *in parallel* where possible, and Dask tries to keep the
-overall memory footprint small. You can work with datasets that are much larger
-than memory, as long as each partition (a regular pandas :class:`pandas.DataFrame`) fits in memory.
-
-By default, ``dask.dataframe`` operations use a threadpool to do operations in
-parallel. We can also connect to a cluster to distribute the work on many
-machines. In this case we'll connect to a local "cluster" made up of several
-processes on this single machine.
-
-.. code-block:: python
-
-   >>> from dask.distributed import Client, LocalCluster
-
-   >>> cluster = LocalCluster()
-   >>> client = Client(cluster)
-   >>> client
-   <Client: 'tcp://127.0.0.1:53349' processes=4 threads=8, memory=17.18 GB>
-
-Once this ``client`` is created, all of Dask's computation will take place on
-the cluster (which is just processes in this case).
-
-Dask implements the most used parts of the pandas API. For example, we can do
-a familiar groupby aggregation.
-
-.. ipython:: python
-   :okwarning:
-
-   %time ddf.groupby("name")[["x", "y"]].mean().compute().head()
-
-The grouping and aggregation is done out-of-core and in parallel.
-
-When Dask knows the ``divisions`` of a dataset, certain optimizations are
-possible. When reading parquet datasets written by dask, the divisions will be
-known automatically. In this case, since we created the parquet files manually,
-we need to supply the divisions manually.
-
-.. ipython:: python
-   :okwarning:
-
-   N = 12
-   starts = [f"20{i:>02d}-01-01" for i in range(N)]
-   ends = [f"20{i:>02d}-12-13" for i in range(N)]
-
-   divisions = tuple(pd.to_datetime(starts)) + (pd.Timestamp(ends[-1]),)
-   ddf.divisions = divisions
-   ddf
-
-Now we can do things like fast random access with ``.loc``.
-
-.. ipython:: python
-   :okwarning:
-
-   ddf.loc["2002-01-01 12:01":"2002-01-01 12:05"].compute()
-
-Dask knows to just look in the 3rd partition for selecting values in 2002. It
-doesn't need to look at any other data.
-
-Many workflows involve a large amount of data and processing it in a way that
-reduces the size to something that fits in memory. In this case, we'll resample
-to daily frequency and take the mean. Once we've taken the mean, we know the
-results will fit in memory, so we can safely call ``compute`` without running
-out of memory. At that point it's just a regular pandas object.
-
-.. ipython:: python
-   :okwarning:
-
-   @savefig dask_resample.png
-   ddf[["x", "y"]].resample("1D").mean().cumsum().compute().plot()
-
-.. ipython:: python
-   :suppress:
-
-   import shutil
-
-   shutil.rmtree("data/timeseries")
-
-These Dask examples have all be done using multiple processes on a single
-machine. Dask can be `deployed on a cluster
-<https://docs.dask.org/en/latest/setup.html>`_ to scale up to even larger
-datasets.
-
-You see more dask examples at https://examples.dask.org.
-
-Use Modin
----------
-
-Modin_ is a scalable dataframe library, which aims to be a drop-in replacement API for pandas and
-provides the ability to scale pandas workflows across nodes and CPUs available. It is also able
-to work with larger than memory datasets. To start working with Modin you just need
-to replace a single line of code, namely, the import statement.
-
-.. code-block:: ipython
-
-   # import pandas as pd
-   import modin.pandas as pd
-
-After you have changed the import statement, you can proceed using the well-known pandas API
-to scale computation. Modin distributes computation across nodes and CPUs available utilizing
-an execution engine it runs on. At the time of Modin 0.27.0 the following execution engines are supported
-in Modin: Ray_, Dask_, `MPI through unidist`_, HDK_. The partitioning schema of a Modin DataFrame partitions it
-along both columns and rows because it gives Modin flexibility and scalability in both the number of columns and
-the number of rows.
-
-For more information refer to `Modin's documentation`_ or the `Modin's tutorials`_.
-
-.. _Modin: https://github.com/modin-project/modin
-.. _`Modin's documentation`: https://modin.readthedocs.io/en/latest
-.. _`Modin's tutorials`: https://github.com/modin-project/modin/tree/master/examples/tutorial/jupyter/execution
-.. _Ray: https://github.com/ray-project/ray
-.. _Dask: https://dask.org
-.. _`MPI through unidist`: https://github.com/modin-project/unidist
-.. _HDK: https://github.com/intel-ai/hdk
-.. _dask.dataframe: https://docs.dask.org/en/latest/dataframe.html
+There are other libraries which provide similar APIs to pandas and work nicely with pandas DataFrame,
+and can give you the ability to scale your large dataset processing and analytics
+by parallel runtime, distributed memory, clustering, etc. You can find more information
+in `the ecosystem page <https://pandas.pydata.org/community/ecosystem.html#out-of-core>`_.

doc/source/whatsnew/v0.10.0.rst

@@ -242,18 +242,42 @@ labeled the aggregated group with the end of the interval: the next day).
 - Calling ``fillna`` on Series or DataFrame with no arguments is no longer
   valid code. You must either specify a fill value or an interpolation method:
 
-.. ipython:: python
-   :okwarning:
+.. code-block:: ipython
 
-   s = pd.Series([np.nan, 1.0, 2.0, np.nan, 4])
-   s
-   s.fillna(0)
-   s.fillna(method="pad")
+    In [6]: s = pd.Series([np.nan, 1.0, 2.0, np.nan, 4])
+
+    In [7]: s
+    Out[7]:
+    0      NaN
+    1      1.0
+    2      2.0
+    3      NaN
+    4      4.0
+    dtype: float64
+
+    In [8]: s.fillna(0)
+    Out[8]:
+    0      0.0
+    1      1.0
+    2      2.0
+    3      0.0
+    4      4.0
+    dtype: float64
+
+    In [9]: s.fillna(method="pad")
+    Out[9]:
+    0      NaN
+    1      1.0
+    2      2.0
+    3      2.0
+    4      4.0
+    dtype: float64
 
 Convenience methods ``ffill`` and  ``bfill`` have been added:
 
 .. ipython:: python
 
+   s = pd.Series([np.nan, 1.0, 2.0, np.nan, 4])
    s.ffill()
 
 

doc/source/whatsnew/v3.0.0.rst

@@ -201,6 +201,7 @@ Removal of prior version deprecations/changes
 - Changed the default value of ``observed`` in :meth:`DataFrame.groupby` and :meth:`Series.groupby` to ``True`` (:issue:`51811`)
 - Enforced deprecation disallowing parsing datetimes with mixed time zones unless user passes ``utc=True`` to :func:`to_datetime` (:issue:`57275`)
 - Enforced deprecation of :meth:`.DataFrameGroupBy.get_group` and :meth:`.SeriesGroupBy.get_group` allowing the ``name`` argument to be a non-tuple when grouping by a list of length 1 (:issue:`54155`)
+- Enforced deprecation of :meth:`Series.interpolate` and :meth:`DataFrame.interpolate` for object-dtype (:issue:`57820`)
 - Enforced deprecation of ``axis=None`` acting the same as ``axis=0`` in the DataFrame reductions ``sum``, ``prod``, ``std``, ``var``, and ``sem``, passing ``axis=None`` will now reduce over both axes; this is particularly the case when doing e.g. ``numpy.sum(df)`` (:issue:`21597`)
 - Enforced deprecation of passing a dictionary to :meth:`SeriesGroupBy.agg` (:issue:`52268`)
 - Enforced deprecation of string ``AS`` denoting frequency in :class:`YearBegin` and strings ``AS-DEC``, ``AS-JAN``, etc. denoting annual frequencies with various fiscal year starts (:issue:`57793`)
@@ -241,6 +242,7 @@ Removal of prior version deprecations/changes
 - Removed argument ``limit`` from :meth:`DataFrame.pct_change`, :meth:`Series.pct_change`, :meth:`.DataFrameGroupBy.pct_change`, and :meth:`.SeriesGroupBy.pct_change`; the argument ``method`` must be set to ``None`` and will be removed in a future version of pandas (:issue:`53520`)
 - Removed deprecated argument ``obj`` in :meth:`.DataFrameGroupBy.get_group` and :meth:`.SeriesGroupBy.get_group` (:issue:`53545`)
 - Removed deprecated behavior of :meth:`Series.agg` using :meth:`Series.apply` (:issue:`53325`)
+- Removed deprecated keyword ``method`` on :meth:`Series.fillna`, :meth:`DataFrame.fillna` (:issue:`57760`)
 - Removed option ``mode.use_inf_as_na``, convert inf entries to ``NaN`` before instead (:issue:`51684`)
 - Removed support for :class:`DataFrame` in :meth:`DataFrame.from_records`(:issue:`51697`)
 - Removed support for ``errors="ignore"`` in :func:`to_datetime`, :func:`to_timedelta` and :func:`to_numeric` (:issue:`55734`)
@@ -281,6 +283,7 @@ Performance improvements
 - Performance improvement in :meth:`RangeIndex.take` returning a :class:`RangeIndex` instead of a :class:`Index` when possible. (:issue:`57445`, :issue:`57752`)
 - Performance improvement in ``DataFrameGroupBy.__len__`` and ``SeriesGroupBy.__len__`` (:issue:`57595`)
 - Performance improvement in indexing operations for string dtypes (:issue:`56997`)
+- Performance improvement in unary methods on a :class:`RangeIndex` returning a :class:`RangeIndex` instead of a :class:`Index` when possible. (:issue:`57825`)
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_300.bug_fixes:

environment.yml

@@ -60,9 +60,8 @@ dependencies:
   - zstandard>=0.19.0
 
   # downstream packages
-  - dask-core<=2024.2.1
+  - dask-core
   - seaborn-base
-  - dask-expr<=0.5.3
 
   # local testing dependencies
   - moto

pandas/_config/config.py

@@ -56,10 +56,8 @@
     TYPE_CHECKING,
     Any,
     Callable,
-    Literal,
     NamedTuple,
     cast,
-    overload,
 )
 import warnings
 
@@ -69,7 +67,6 @@
 if TYPE_CHECKING:
     from collections.abc import (
         Generator,
-        Iterable,
         Sequence,
     )
 
@@ -583,12 +580,6 @@ def _get_root(key: str) -> tuple[dict[str, Any], str]:
     return cursor, path[-1]
 
 
-def _is_deprecated(key: str) -> bool:
-    """Returns True if the given option has been deprecated"""
-    key = key.lower()
-    return key in _deprecated_options
-
-
 def _get_deprecated_option(key: str):
     """
     Retrieves the metadata for a deprecated option, if `key` is deprecated.
@@ -685,54 +676,6 @@ def _build_option_description(k: str) -> str:
     return s
 
 
-@overload
-def pp_options_list(
-    keys: Iterable[str], *, width: int = ..., _print: Literal[False] = ...
-) -> str: ...
-
-
-@overload
-def pp_options_list(
-    keys: Iterable[str], *, width: int = ..., _print: Literal[True]
-) -> None: ...
-
-
-def pp_options_list(
-    keys: Iterable[str], *, width: int = 80, _print: bool = False
-) -> str | None:
-    """Builds a concise listing of available options, grouped by prefix"""
-    from itertools import groupby
-    from textwrap import wrap
-
-    def pp(name: str, ks: Iterable[str]) -> list[str]:
-        pfx = "- " + name + ".[" if name else ""
-        ls = wrap(
-            ", ".join(ks),
-            width,
-            initial_indent=pfx,
-            subsequent_indent="  ",
-            break_long_words=False,
-        )
-        if ls and ls[-1] and name:
-            ls[-1] = ls[-1] + "]"
-        return ls
-
-    ls: list[str] = []
-    singles = [x for x in sorted(keys) if x.find(".") < 0]
-    if singles:
-        ls += pp("", singles)
-    keys = [x for x in keys if x.find(".") >= 0]
-
-    for k, g in groupby(sorted(keys), lambda x: x[: x.rfind(".")]):
-        ks = [x[len(k) + 1 :] for x in list(g)]
-        ls += pp(k, ks)
-    s = "\n".join(ls)
-    if _print:
-        print(s)
-    return s
-
-
-#
 # helpers