From 97410b6715b128ed8ae20f5c476a9119df7dbeb0 Mon Sep 17 00:00:00 2001 From: Erez Rokah Date: Thu, 20 Jun 2024 14:47:27 +0100 Subject: [PATCH 01/10] chore: Create LICENSE (#188) Part of https://github.com/cloudquery/cloudquery/issues/18335, license taken from https://github.com/cloudquery/plugin-sdk/blob/main/LICENSE --- LICENSE | 353 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 353 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..82b4de9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,353 @@ +Mozilla Public License, version 2.0 + +1. Definitions + +1.1. “Contributor” + + means each individual or legal entity that creates, contributes to the + creation of, or owns Covered Software. + +1.2. “Contributor Version” + + means the combination of the Contributions of others (if any) used by a + Contributor and that particular Contributor’s Contribution. + +1.3. “Contribution” + + means Covered Software of a particular Contributor. + +1.4. “Covered Software” + + means Source Code Form to which the initial Contributor has attached the + notice in Exhibit A, the Executable Form of such Source Code Form, and + Modifications of such Source Code Form, in each case including portions + thereof. + +1.5. “Incompatible With Secondary Licenses” + means + + a. that the initial Contributor has attached the notice described in + Exhibit B to the Covered Software; or + + b. that the Covered Software was made available under the terms of version + 1.1 or earlier of the License, but not also under the terms of a + Secondary License. + +1.6. “Executable Form” + + means any form of the work other than Source Code Form. + +1.7. “Larger Work” + + means a work that combines Covered Software with other material, in a separate + file or files, that is not Covered Software. + +1.8. “License” + + means this document. + +1.9. “Licensable” + + means having the right to grant, to the maximum extent possible, whether at the + time of the initial grant or subsequently, any and all of the rights conveyed by + this License. + +1.10. “Modifications” + + means any of the following: + + a. any file in Source Code Form that results from an addition to, deletion + from, or modification of the contents of Covered Software; or + + b. any new file in Source Code Form that contains any Covered Software. + +1.11. “Patent Claims” of a Contributor + + means any patent claim(s), including without limitation, method, process, + and apparatus claims, in any patent Licensable by such Contributor that + would be infringed, but for the grant of the License, by the making, + using, selling, offering for sale, having made, import, or transfer of + either its Contributions or its Contributor Version. + +1.12. “Secondary License” + + means either the GNU General Public License, Version 2.0, the GNU Lesser + General Public License, Version 2.1, the GNU Affero General Public + License, Version 3.0, or any later versions of those licenses. + +1.13. “Source Code Form” + + means the form of the work preferred for making modifications. + +1.14. “You” (or “Your”) + + means an individual or a legal entity exercising rights under this + License. For legal entities, “You” includes any entity that controls, is + controlled by, or is under common control with You. For purposes of this + definition, “control” means (a) the power, direct or indirect, to cause + the direction or management of such entity, whether by contract or + otherwise, or (b) ownership of more than fifty percent (50%) of the + outstanding shares or beneficial ownership of such entity. + + +2. License Grants and Conditions + +2.1. Grants + + Each Contributor hereby grants You a world-wide, royalty-free, + non-exclusive license: + + a. under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or as + part of a Larger Work; and + + b. under Patent Claims of such Contributor to make, use, sell, offer for + sale, have made, import, and otherwise transfer either its Contributions + or its Contributor Version. + +2.2. Effective Date + + The licenses granted in Section 2.1 with respect to any Contribution become + effective for each Contribution on the date the Contributor first distributes + such Contribution. + +2.3. Limitations on Grant Scope + + The licenses granted in this Section 2 are the only rights granted under this + License. No additional rights or licenses will be implied from the distribution + or licensing of Covered Software under this License. Notwithstanding Section + 2.1(b) above, no patent license is granted by a Contributor: + + a. for any code that a Contributor has removed from Covered Software; or + + b. for infringements caused by: (i) Your and any other third party’s + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + + c. under Patent Claims infringed by Covered Software in the absence of its + Contributions. + + This License does not grant any rights in the trademarks, service marks, or + logos of any Contributor (except as may be necessary to comply with the + notice requirements in Section 3.4). + +2.4. Subsequent Licenses + + No Contributor makes additional grants as a result of Your choice to + distribute the Covered Software under a subsequent version of this License + (see Section 10.2) or under the terms of a Secondary License (if permitted + under the terms of Section 3.3). + +2.5. Representation + + Each Contributor represents that the Contributor believes its Contributions + are its original creation(s) or it has sufficient rights to grant the + rights to its Contributions conveyed by this License. + +2.6. Fair Use + + This License is not intended to limit any rights You have under applicable + copyright doctrines of fair use, fair dealing, or other equivalents. + +2.7. Conditions + + Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in + Section 2.1. + + +3. Responsibilities + +3.1. Distribution of Source Form + + All distribution of Covered Software in Source Code Form, including any + Modifications that You create or to which You contribute, must be under the + terms of this License. You must inform recipients that the Source Code Form + of the Covered Software is governed by the terms of this License, and how + they can obtain a copy of this License. You may not attempt to alter or + restrict the recipients’ rights in the Source Code Form. + +3.2. Distribution of Executable Form + + If You distribute Covered Software in Executable Form then: + + a. such Covered Software must also be made available in Source Code Form, + as described in Section 3.1, and You must inform recipients of the + Executable Form how they can obtain a copy of such Source Code Form by + reasonable means in a timely manner, at a charge no more than the cost + of distribution to the recipient; and + + b. You may distribute such Executable Form under the terms of this License, + or sublicense it under different terms, provided that the license for + the Executable Form does not attempt to limit or alter the recipients’ + rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + + You may create and distribute a Larger Work under terms of Your choice, + provided that You also comply with the requirements of this License for the + Covered Software. If the Larger Work is a combination of Covered Software + with a work governed by one or more Secondary Licenses, and the Covered + Software is not Incompatible With Secondary Licenses, this License permits + You to additionally distribute such Covered Software under the terms of + such Secondary License(s), so that the recipient of the Larger Work may, at + their option, further distribute the Covered Software under the terms of + either this License or such Secondary License(s). + +3.4. Notices + + You may not remove or alter the substance of any license notices (including + copyright notices, patent notices, disclaimers of warranty, or limitations + of liability) contained within the Source Code Form of the Covered + Software, except that You may alter any license notices to the extent + required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + + You may choose to offer, and to charge a fee for, warranty, support, + indemnity or liability obligations to one or more recipients of Covered + Software. However, You may do so only on Your own behalf, and not on behalf + of any Contributor. You must make it absolutely clear that any such + warranty, support, indemnity, or liability obligation is offered by You + alone, and You hereby agree to indemnify every Contributor for any + liability incurred by such Contributor as a result of warranty, support, + indemnity or liability terms You offer. You may include additional + disclaimers of warranty and limitations of liability specific to any + jurisdiction. + +4. Inability to Comply Due to Statute or Regulation + + If it is impossible for You to comply with any of the terms of this License + with respect to some or all of the Covered Software due to statute, judicial + order, or regulation then You must: (a) comply with the terms of this License + to the maximum extent possible; and (b) describe the limitations and the code + they affect. Such description must be placed in a text file included with all + distributions of the Covered Software under this License. Except to the + extent prohibited by statute or regulation, such description must be + sufficiently detailed for a recipient of ordinary skill to be able to + understand it. + +5. Termination + +5.1. The rights granted under this License will terminate automatically if You + fail to comply with any of its terms. However, if You become compliant, + then the rights granted under this License from a particular Contributor + are reinstated (a) provisionally, unless and until such Contributor + explicitly and finally terminates Your grants, and (b) on an ongoing basis, + if such Contributor fails to notify You of the non-compliance by some + reasonable means prior to 60 days after You have come back into compliance. + Moreover, Your grants from a particular Contributor are reinstated on an + ongoing basis if such Contributor notifies You of the non-compliance by + some reasonable means, this is the first time You have received notice of + non-compliance with this License from such Contributor, and You become + compliant prior to 30 days after Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent + infringement claim (excluding declaratory judgment actions, counter-claims, + and cross-claims) alleging that a Contributor Version directly or + indirectly infringes any patent, then the rights granted to You by any and + all Contributors for the Covered Software under Section 2.1 of this License + shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user + license agreements (excluding distributors and resellers) which have been + validly granted by You or Your distributors under this License prior to + termination shall survive termination. + +6. Disclaimer of Warranty + + Covered Software is provided under this License on an “as is” basis, without + warranty of any kind, either expressed, implied, or statutory, including, + without limitation, warranties that the Covered Software is free of defects, + merchantable, fit for a particular purpose or non-infringing. The entire + risk as to the quality and performance of the Covered Software is with You. + Should any Covered Software prove defective in any respect, You (not any + Contributor) assume the cost of any necessary servicing, repair, or + correction. This disclaimer of warranty constitutes an essential part of this + License. No use of any Covered Software is authorized under this License + except under this disclaimer. + +7. Limitation of Liability + + Under no circumstances and under no legal theory, whether tort (including + negligence), contract, or otherwise, shall any Contributor, or anyone who + distributes Covered Software as permitted above, be liable to You for any + direct, indirect, special, incidental, or consequential damages of any + character including, without limitation, damages for lost profits, loss of + goodwill, work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses, even if such party shall have been + informed of the possibility of such damages. This limitation of liability + shall not apply to liability for death or personal injury resulting from such + party’s negligence to the extent applicable law prohibits such limitation. + Some jurisdictions do not allow the exclusion or limitation of incidental or + consequential damages, so this exclusion and limitation may not apply to You. + +8. Litigation + + Any litigation relating to this License may be brought only in the courts of + a jurisdiction where the defendant maintains its principal place of business + and such litigation shall be governed by laws of that jurisdiction, without + reference to its conflict-of-law provisions. Nothing in this Section shall + prevent a party’s ability to bring cross-claims or counter-claims. + +9. Miscellaneous + + This License represents the complete agreement concerning the subject matter + hereof. If any provision of this License is held to be unenforceable, such + provision shall be reformed only to the extent necessary to make it + enforceable. Any law or regulation which provides that the language of a + contract shall be construed against the drafter shall not be used to construe + this License against a Contributor. + + +10. Versions of the License + +10.1. New Versions + + Mozilla Foundation is the license steward. Except as provided in Section + 10.3, no one other than the license steward has the right to modify or + publish new versions of this License. Each version will be given a + distinguishing version number. + +10.2. Effect of New Versions + + You may distribute the Covered Software under the terms of the version of + the License under which You originally received the Covered Software, or + under the terms of any subsequent version published by the license + steward. + +10.3. Modified Versions + + If you create software not governed by this License, and you want to + create a new license for such software, you may create and use a modified + version of this License if you rename the license and remove any + references to the name of the license steward (except to note that such + modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses + If You choose to distribute Source Code Form that is Incompatible With + Secondary Licenses under the terms of this version of the License, the + notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice + + This Source Code Form is subject to the + terms of the Mozilla Public License, v. + 2.0. If a copy of the MPL was not + distributed with this file, You can + obtain one at + http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular file, then +You may include the notice in a location (such as a LICENSE file in a relevant +directory) where a recipient would be likely to look for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - “Incompatible With Secondary Licenses” Notice + + This Source Code Form is “Incompatible + With Secondary Licenses”, as defined by + the Mozilla Public License, v. 2.0. From 9cafffdf22aa261a52f55d2148f008ce8d20ec5f Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Sun, 30 Jun 2024 20:58:15 -0400 Subject: [PATCH 02/10] fix(deps): Update dependency grpcio to v1.64.1 (#189) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Update | Change | |---|---|---| | [grpcio](https://grpc.io) ([source](https://github.com/grpc/grpc)) | patch | `==1.64.0` -> `==1.64.1` | --- ### Release Notes
grpc/grpc (grpcio) ### [`v1.64.1`](https://github.com/grpc/grpc/releases/tag/v1.64.1) [Compare Source](https://github.com/grpc/grpc/compare/v1.64.0...v1.64.1) This is release 1.64.1 ([grateful](https://github.com/grpc/grpc/blob/master/doc/g_stands_for.md)) of gRPC Core. For gRPC documentation, see [grpc.io](https://grpc.io/). For previous releases, see [Releases](https://github.com/grpc/grpc/releases). This release contains refinements, improvements, and bug fixes, with highlights listed below. ## Core - \[StatsPlugin] Fix use-after-free issue ([#​36664](https://github.com/grpc/grpc/issues/36664)). ([#​36670](https://github.com/grpc/grpc/pull/36670)) ## C++ - \[OTel] Generate pkg-config file for grpcpp_otel_plugin ([#​36686](https://github.com/grpc/grpc/issues/36686)). ([#​36722](https://github.com/grpc/grpc/pull/36722)) - \[OTel] Add find_package call for OTel dependency in installed gRPC configuration ([#​36674](https://github.com/grpc/grpc/issues/36674)). ([#​36688](https://github.com/grpc/grpc/pull/36688)) ## Python - \[Python Server] Remove abstractmethod notation for add_registered_method_handlers (v1.64.x backport). ([#​36696](https://github.com/grpc/grpc/pull/36696))
--- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 800230e..c1b816c 100644 --- a/setup.py +++ b/setup.py @@ -13,7 +13,7 @@ "cloudquery-plugin-pb==0.0.28", "exceptiongroup==1.2.1", "black==24.4.2", - "grpcio==1.64.0", + "grpcio==1.64.1", "grpcio-tools==1.64.0", "iniconfig==2.0.0", "Jinja2==3.1.4", From f7eea8d30a72e8f4226f75f4d0caef9be8bb36ae Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Sun, 30 Jun 2024 21:54:33 -0400 Subject: [PATCH 03/10] fix(deps): Update dependency grpcio-tools to v1.64.1 (#190) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Update | Change | |---|---|---| | [grpcio-tools](https://grpc.io) | patch | `==1.64.0` -> `==1.64.1` | --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index c1b816c..aaab84c 100644 --- a/setup.py +++ b/setup.py @@ -14,7 +14,7 @@ "exceptiongroup==1.2.1", "black==24.4.2", "grpcio==1.64.1", - "grpcio-tools==1.64.0", + "grpcio-tools==1.64.1", "iniconfig==2.0.0", "Jinja2==3.1.4", "MarkupSafe==2.1.5", From e3b240e594a046b65217b24cb34d53dcb2f9ac3d Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Sun, 30 Jun 2024 21:57:50 -0400 Subject: [PATCH 04/10] fix(deps): Update dependency pytest to v8.2.2 (#193) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Update | Change | |---|---|---| | [pytest](https://github.com/pytest-dev/pytest) ([changelog](https://docs.pytest.org/en/stable/changelog.html)) | patch | `==8.2.1` -> `==8.2.2` | --- ### Release Notes
pytest-dev/pytest (pytest) ### [`v8.2.2`](https://github.com/pytest-dev/pytest/releases/tag/8.2.2) [Compare Source](https://github.com/pytest-dev/pytest/compare/8.2.1...8.2.2) # pytest 8.2.2 (2024-06-04) ## Bug Fixes - [#​12355](https://github.com/pytest-dev/pytest/issues/12355): Fix possible catastrophic performance slowdown on a certain parametrization pattern involving many higher-scoped parameters. - [#​12367](https://github.com/pytest-dev/pytest/issues/12367): Fix a regression in pytest 8.2.0 where unittest class instances (a fresh one is created for each test) were not released promptly on test teardown but only on session teardown. - [#​12381](https://github.com/pytest-dev/pytest/issues/12381): Fix possible "Directory not empty" crashes arising from concurent cache dir (`.pytest_cache`) creation. Regressed in pytest 8.2.0. ## Improved Documentation - [#​12290](https://github.com/pytest-dev/pytest/issues/12290): Updated Sphinx theme to use Furo instead of Flask, enabling Dark mode theme. - [#​12356](https://github.com/pytest-dev/pytest/issues/12356): Added a subsection to the documentation for debugging flaky tests to mention lack of thread safety in pytest as a possible source of flakyness. - [#​12363](https://github.com/pytest-dev/pytest/issues/12363): The documentation webpages now links to a canonical version to reduce outdated documentation in search engine results.
--- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index aaab84c..25ba8af 100644 --- a/setup.py +++ b/setup.py @@ -24,7 +24,7 @@ "pluggy==1.5.0", "protobuf==5.27.0", "pyarrow==15.0.2", - "pytest==8.2.1", + "pytest==8.2.2", "python-dateutil==2.8.2", "pytz==2024.1", "six==1.16.0", From b2e8673bf4e212ca0983ef0fb1e0849859a466b5 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Sun, 30 Jun 2024 22:02:34 -0400 Subject: [PATCH 05/10] fix(deps): Update dependency protobuf to v5.27.2 (#192) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Update | Change | |---|---|---| | [protobuf](https://developers.google.com/protocol-buffers/) | patch | `==5.27.0` -> `==5.27.2` | --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 25ba8af..d56670c 100644 --- a/setup.py +++ b/setup.py @@ -22,7 +22,7 @@ "packaging==24.0", "pandas==2.2.2", "pluggy==1.5.0", - "protobuf==5.27.0", + "protobuf==5.27.2", "pyarrow==15.0.2", "pytest==8.2.2", "python-dateutil==2.8.2", From 73d18ca673a1d86de622181fb21c10ed142c9b58 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Sun, 30 Jun 2024 22:47:20 -0400 Subject: [PATCH 06/10] fix(deps): Update dependency packaging to v24.1 (#194) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Update | Change | |---|---|---| | [packaging](https://github.com/pypa/packaging) | minor | `==24.0` -> `==24.1` | --- ### Release Notes
pypa/packaging (packaging) ### [`v24.1`](https://github.com/pypa/packaging/releases/tag/24.1) [Compare Source](https://github.com/pypa/packaging/compare/24.0...24.1) #### What's Changed - pyupgrade/black/isort/flake8 → ruff by [@​DimitriPapadopoulos](https://github.com/DimitriPapadopoulos) in [https://github.com/pypa/packaging/pull/769](https://github.com/pypa/packaging/pull/769) - Add support for Python 3.13 and drop EOL 3.7 by [@​hugovk](https://github.com/hugovk) in [https://github.com/pypa/packaging/pull/783](https://github.com/pypa/packaging/pull/783) - Bump the github-actions group with 4 updates by [@​dependabot](https://github.com/dependabot) in [https://github.com/pypa/packaging/pull/782](https://github.com/pypa/packaging/pull/782) - Fix typo in `_parser` docstring by [@​pradyunsg](https://github.com/pradyunsg) in [https://github.com/pypa/packaging/pull/784](https://github.com/pypa/packaging/pull/784) - Modernise type annotations using FA rules from ruff by [@​pradyunsg](https://github.com/pradyunsg) in [https://github.com/pypa/packaging/pull/785](https://github.com/pypa/packaging/pull/785) - Document `markers.default_environment()` by [@​edgarrmondragon](https://github.com/edgarrmondragon) in [https://github.com/pypa/packaging/pull/753](https://github.com/pypa/packaging/pull/753) - Bump the github-actions group with 3 updates by [@​dependabot](https://github.com/dependabot) in [https://github.com/pypa/packaging/pull/789](https://github.com/pypa/packaging/pull/789) - Work around platform.python_version() returning non PEP 440 compliant version for non-tagged CPython builds by [@​sbidoul](https://github.com/sbidoul) in [https://github.com/pypa/packaging/pull/802](https://github.com/pypa/packaging/pull/802) #### New Contributors - [@​dependabot](https://github.com/dependabot) made their first contribution in [https://github.com/pypa/packaging/pull/782](https://github.com/pypa/packaging/pull/782) - [@​edgarrmondragon](https://github.com/edgarrmondragon) made their first contribution in [https://github.com/pypa/packaging/pull/753](https://github.com/pypa/packaging/pull/753) **Full Changelog**: https://github.com/pypa/packaging/compare/24.0...24.1
--- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index d56670c..eb92166 100644 --- a/setup.py +++ b/setup.py @@ -19,7 +19,7 @@ "Jinja2==3.1.4", "MarkupSafe==2.1.5", "numpy==1.26.4", - "packaging==24.0", + "packaging==24.1", "pandas==2.2.2", "pluggy==1.5.0", "protobuf==5.27.2", From a2835bb8b0e3d810b21060d1f46a7873d1a3ea09 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Sun, 30 Jun 2024 22:47:38 -0400 Subject: [PATCH 07/10] fix(deps): Update dependency python-dateutil to v2.9.0.post0 (#195) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Update | Change | |---|---|---| | [python-dateutil](https://github.com/dateutil/dateutil) | minor | `==2.8.2` -> `==2.9.0.post0` | --- ### Release Notes
dateutil/dateutil (python-dateutil) ### [`v2.9.0.post0`](https://github.com/dateutil/dateutil/releases/tag/2.9.0.post0) [Compare Source](https://github.com/dateutil/dateutil/compare/2.9.0...2.9.0.post0) ### Version 2.9.0.post0 (2024-03-01) #### Bugfixes - Pinned `setuptools_scm` to `<8`, which should make the generated `_version.py` file compatible with all supported versions of Python. ### [`v2.9.0`](https://github.com/dateutil/dateutil/releases/tag/2.9.0) [Compare Source](https://github.com/dateutil/dateutil/compare/2.8.2...2.9.0) ### Version 2.9.0 (2024-02-29) #### Data updates - Updated tzdata version to 2024a. (gh pr [#​1342](https://github.com/dateutil/dateutil/issues/1342)) #### Features - Made all `dateutil` submodules lazily imported using [PEP 562](https://www.python.org/dev/peps/pep-0562/). On Python 3.7+, things like `import dateutil; dateutil.tz.gettz("America/New_York")` will now work without explicitly importing `dateutil.tz`, with the import occurring behind the scenes on first use. The old behavior remains on Python 3.6 and earlier. Fixed by Orson Adams. (gh issue [#​771](https://github.com/dateutil/dateutil/issues/771), gh pr [#​1007](https://github.com/dateutil/dateutil/issues/1007)) #### Bugfixes - Removed a call to `datetime.utcfromtimestamp`, which is deprecated as of Python 3.12. Reported by Hugo van Kemenade (gh pr [#​1284](https://github.com/dateutil/dateutil/issues/1284)), fixed by Thomas Grainger (gh pr [#​1285](https://github.com/dateutil/dateutil/issues/1285)). #### Documentation changes - Added note into docs and tests where relativedelta would return last day of the month only if the same day on a different month resolves to a date that doesn't exist. Reported by [@​hawkEye-01](https://github.com/hawkEye-01) (gh issue [#​1167](https://github.com/dateutil/dateutil/issues/1167)). Fixed by [@​Mifrill](https://github.com/Mifrill) (gh pr [#​1168](https://github.com/dateutil/dateutil/issues/1168))
--- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index eb92166..cf554e6 100644 --- a/setup.py +++ b/setup.py @@ -25,7 +25,7 @@ "protobuf==5.27.2", "pyarrow==15.0.2", "pytest==8.2.2", - "python-dateutil==2.8.2", + "python-dateutil==2.9.0.post0", "pytz==2024.1", "six==1.16.0", "structlog==23.3.0", From 3193879e57fe84d5cdbffeddf25bb98cea5bef10 Mon Sep 17 00:00:00 2001 From: Mariano Gappa Date: Mon, 8 Jul 2024 09:18:55 +0100 Subject: [PATCH 08/10] feat: State Client (#198) * wio * Add gRPC connection example * Implement working StateClient. * Remove unnecessary modifications. * Remove unnecessary modifications. * Remove TODO. --------- Co-authored-by: Herman Schaaf --- .../sdk/internal/servers/plugin_v3/plugin.py | 2 +- cloudquery/sdk/scheduler/scheduler.py | 19 ++ cloudquery/sdk/scheduler/table_resolver.py | 4 + cloudquery/sdk/stateclient/__init__.py | 0 cloudquery/sdk/stateclient/stateclient.py | 207 ++++++++++++++++++ 5 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 cloudquery/sdk/stateclient/__init__.py create mode 100644 cloudquery/sdk/stateclient/stateclient.py diff --git a/cloudquery/sdk/internal/servers/plugin_v3/plugin.py b/cloudquery/sdk/internal/servers/plugin_v3/plugin.py index e8a94ae..ffbc581 100644 --- a/cloudquery/sdk/internal/servers/plugin_v3/plugin.py +++ b/cloudquery/sdk/internal/servers/plugin_v3/plugin.py @@ -63,7 +63,7 @@ def Sync(self, request, context): skip_dependent_tables=request.skip_dependent_tables, skip_tables=request.skip_tables, tables=request.tables, - backend_options=None, + backend_options=request.backend, ) for msg in self._plugin.sync(options): diff --git a/cloudquery/sdk/scheduler/scheduler.py b/cloudquery/sdk/scheduler/scheduler.py index 7ba3df7..c7e7b0c 100644 --- a/cloudquery/sdk/scheduler/scheduler.py +++ b/cloudquery/sdk/scheduler/scheduler.py @@ -10,6 +10,7 @@ SyncMigrateTableMessage, ) from cloudquery.sdk.schema import Resource +from cloudquery.sdk.stateclient.stateclient import StateClient from .table_resolver import TableResolver, Client QUEUE_PER_WORKER = 100 @@ -35,6 +36,7 @@ class Scheduler: def __init__( self, concurrency: int, queue_size: int = 0, max_depth: int = 3, logger=None ): + self._post_sync_hook = lambda: None self._queue = queue.Queue() self._max_depth = max_depth if logger is None: @@ -201,6 +203,8 @@ def sync( break continue yield message + + self._post_sync_hook() thread.shutdown(wait=True) def _send_migrate_table_messages( @@ -210,3 +214,18 @@ def _send_migrate_table_messages( yield SyncMigrateTableMessage(table=resolver.table.to_arrow_schema()) if resolver.child_resolvers: yield from self._send_migrate_table_messages(resolver.child_resolvers) + + def set_post_sync_hook(self, fn): + """ + Use this to set a function that will be called after the sync is finished, + a la `defer fn()` in Go (but for a single function, rather than a stack). + + This is necessary because plugins use this pattern on their sync method: + + ``` + return self._scheduler.sync(...) + ``` + + So if a plugin has a `state_client`, there's nowhere to call the flush method. + """ + self._post_sync_hook = fn diff --git a/cloudquery/sdk/scheduler/table_resolver.py b/cloudquery/sdk/scheduler/table_resolver.py index be13d7c..13c20c6 100644 --- a/cloudquery/sdk/scheduler/table_resolver.py +++ b/cloudquery/sdk/scheduler/table_resolver.py @@ -14,6 +14,7 @@ def __init__(self, table: Table, child_resolvers: Optional[List] = None) -> None child_resolvers = [] self._table = table self._child_resolvers = child_resolvers + self.state_client = None @property def table(self) -> Table: @@ -23,6 +24,9 @@ def table(self) -> Table: def child_resolvers(self): return self._child_resolvers + def set_state_client(self, state_client): + self.state_client = state_client + def multiplex(self, client: Client) -> List[Client]: return [client] diff --git a/cloudquery/sdk/stateclient/__init__.py b/cloudquery/sdk/stateclient/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/cloudquery/sdk/stateclient/stateclient.py b/cloudquery/sdk/stateclient/stateclient.py new file mode 100644 index 0000000..0b0dded --- /dev/null +++ b/cloudquery/sdk/stateclient/stateclient.py @@ -0,0 +1,207 @@ +from __future__ import annotations + +import grpc +from abc import ABC, abstractmethod +from typing import Generator, Optional, Tuple + +import pyarrow as pa +from cloudquery.sdk import schema +from cloudquery.sdk.plugin.plugin import BackendOptions +from cloudquery.plugin_v3 import plugin_pb2, plugin_pb2_grpc, arrow +from functools import wraps + +keyColumn = "key" +valueColumn = "value" + + +class StateClientBuilder: + """ + Provides a `build` method that creates a `ConnectedStateClient` if you pass backend_options, + or `NoOpStateClient` otherwise. + + Args: + backend_options (BackendOptions): which has `connection` & `table_name` strings. + + Returns: + `ConnectedStateClient` or `NoOpStateClient` + + """ + + @staticmethod + def build(*, backend_options: BackendOptions) -> StateClient: + if not backend_options or not backend_options.table_name: + return NoOpStateClient(backend_options=backend_options) + + return ConnectedStateClient(backend_options=backend_options) + + +class StateClient(ABC): + """ + Abstract class that defines the interface for a state client. + + It implements all methods except those that require a connection, + so it is a succinct overview of what a state client does. + """ + + def __init__(self, *, backend_options: BackendOptions): + self.mem = {} + self.changes = {} + self.connection = getattr(backend_options, "connection", None) + self.table = Table(getattr(backend_options, "table_name", None)) + + self.migrate_state_table() + self.read_all_state() + + def get_key(self, key: str) -> Optional[str]: + return self.mem.get(key) + + def set_key(self, key: str, value: str) -> None: + self.mem[key] = value + self.changes[key] = True + + def flush(self): + if not self.changes: + return + + self.write_all_state(self._changed_keys()) + + def _changed_keys(self): + for key, _ in self.changes.items(): + yield (key, self.mem[key]) + + @abstractmethod + def migrate_state_table(self): + pass + + @abstractmethod + def read_all_state(self): + pass + + @abstractmethod + def write_all_state(self, changed_keys: Generator[Tuple[str, str], None, None]): + pass + + +class NoOpStateClient(StateClient): + """ + A state client implementation that does nothing. Used when there is no backend connection. + """ + + def get_key(self, key: str) -> Optional[str]: + pass + + def set_key(self, key: str, value: str) -> None: + pass + + def migrate_state_table(self): + pass + + def read_all_state(self): + pass + + def write_all_state(self, changed_keys: Generator[Tuple[str, str], None, None]): + pass + + +def connected(func): + """ + Decorator that provides a `backend_plugin` with a gRPC connection to the decorated function. + """ + + @wraps(func) + def wrapper(self, *args, **kwargs): + with grpc.insecure_channel(self.connection) as channel: + backend_plugin = plugin_pb2_grpc.PluginStub(channel) + return func(self, backend_plugin, *args, **kwargs) + + return wrapper + + +class ConnectedStateClient(StateClient): + """ + A state client implementation that connects to a backend plugin via gRPC to read/write state. + """ + + @connected + def migrate_state_table(self, backend_plugin: plugin_pb2_grpc.PluginStub): + backend_plugin.Write(self._migrate_table_request()) + + @connected + def read_all_state(self, backend_plugin: plugin_pb2_grpc.PluginStub): + response = backend_plugin.Read( + plugin_pb2.Read.Request(table=self.table.bytes()) + ) + + for record in read_response_to_records(response): + self.mem[record[keyColumn]] = record[valueColumn] + + @connected + def write_all_state( + self, + backend_plugin: plugin_pb2_grpc.PluginStub, + changed_keys: Generator[Tuple[str, str], None, None], + ): + backend_plugin.Write(self._write_request(k, v) for k, v in changed_keys) + + def _write_request(self, key: str, value: str) -> plugin_pb2.Write.Request: + record = pa.RecordBatch.from_arrays( + [ + pa.array([key]), + pa.array([value]), + ], + schema=self.table.arrow_schema(), + ) + return plugin_pb2.Write.Request( + insert=plugin_pb2.Write.MessageInsert(record=arrow.record_to_bytes(record)) + ) + + def _migrate_table_request(self): + yield plugin_pb2.Write.Request( + migrate_table=plugin_pb2.Write.MessageMigrateTable(table=self.table.bytes()) + ) + + +def read_response_to_records(response) -> Generator[dict[str, str], None, None]: + for record in response: + record_batch = arrow.new_record_from_bytes(record.record) + for record in recordbatch_to_list_of_maps(record_batch): + yield record + + +def recordbatch_to_list_of_maps( + record_batch: pa.RecordBatch, +) -> Generator[dict[str, str], None, None]: + table = pa.Table.from_batches([record_batch]) + for row in table.to_pandas().to_dict(orient="records"): + yield row + + +class Table: + """ + Represents a state table with two columns: key and value. + Provides convenience methods for whatever the gRPC requests need. + """ + + def __init__(self, name): + self.name = name + self._arrow_schema = None + self._bytes = None + + if self.name: + self._arrow_schema = self._state_table_schema().to_arrow_schema() + self._bytes = arrow.schema_to_bytes(self._arrow_schema) + + def arrow_schema(self): + return self._arrow_schema + + def bytes(self): + return self._bytes + + def _state_table_schema(self): + return schema.Table( + name=self.name, + columns=[ + schema.Column(name=keyColumn, type=pa.string(), primary_key=True), + schema.Column(name=valueColumn, type=pa.string()), + ], + ) From a6c7a1045e5e703e4ae5200a1c1b06f68f1a11c9 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Mon, 8 Jul 2024 04:35:34 -0400 Subject: [PATCH 09/10] fix(deps): Update dependency tzdata to v2024 (#197) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Update | Change | |---|---|---| | [tzdata](https://github.com/python/tzdata) | major | `==2023.4` -> `==2024.1` | --- ### Release Notes
python/tzdata (tzdata) ### [`v2024.1`](https://github.com/python/tzdata/releases/tag/2024.1): : Release of upstream tzdata 2024a [Compare Source](https://github.com/python/tzdata/compare/2023.4...2024.1) ### Version 2024.1 Upstream version 2024a released 2024-02-01T17:28:56+00:00 #### Briefly: Kazakhstan unifies on UTC+5 beginning 2024-03-01. Palestine springs forward a week later after Ramadan. zic no longer pretends to support indefinite-past DST. localtime no longer mishandles Ciudad Juárez in 2422. #### Changes to future timestamps Kazakhstan unifies on UTC+5. This affects Asia/Almaty and Asia/Qostanay which together represent the eastern portion of the country that will transition from UTC+6 on 2024-03-01 at 00:00 to join the western portion. (Thanks to Zhanbolat Raimbekov.) Palestine springs forward a week later than previously predicted in 2024 and 2025\. (Thanks to Heba Hamad.) Change spring-forward predictions to the second Saturday after Ramadan, not the first; this also affects other predictions starting in 2039. #### Changes to past timestamps Asia/Ho_Chi_Minh's 1955-07-01 transition occurred at 01:00 not 00:00. (Thanks to Đoàn Trần Công Danh.) From 1947 through 1949, Toronto's transitions occurred at 02:00 not 00:00. (Thanks to Chris Walton.) In 1911 Miquelon adopted standard time on June 15, not May 15. #### Changes to documentation The strftime man page documents which struct tm members affect which conversion specs, and that tzset is called. (Problems reported by Robert Elz and Steve Summit.)
--- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index cf554e6..ba4afca 100644 --- a/setup.py +++ b/setup.py @@ -30,7 +30,7 @@ "six==1.16.0", "structlog==23.3.0", "tomli==2.0.1", - "tzdata==2023.4", + "tzdata==2024.1", ] url = "https://github.com/cloudquery/plugin-sdk-python" From 60e9d53381c4318881d448126b05448bd8916652 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Mon, 8 Jul 2024 04:46:26 -0400 Subject: [PATCH 10/10] chore(main): Release v0.1.27 (#191) --- .release-please-manifest.json | 2 +- CHANGELOG.md | 18 ++++++++++++++++++ setup.py | 2 +- 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/.release-please-manifest.json b/.release-please-manifest.json index 3f06a9c..fb4a2b8 100644 --- a/.release-please-manifest.json +++ b/.release-please-manifest.json @@ -1,3 +1,3 @@ { - ".": "0.1.26" + ".": "0.1.27" } diff --git a/CHANGELOG.md b/CHANGELOG.md index dd3d80d..9cb5509 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ # Changelog +## [0.1.27](https://github.com/cloudquery/plugin-sdk-python/compare/v0.1.26...v0.1.27) (2024-07-08) + + +### Features + +* State Client ([#198](https://github.com/cloudquery/plugin-sdk-python/issues/198)) ([3193879](https://github.com/cloudquery/plugin-sdk-python/commit/3193879e57fe84d5cdbffeddf25bb98cea5bef10)) + + +### Bug Fixes + +* **deps:** Update dependency grpcio to v1.64.1 ([#189](https://github.com/cloudquery/plugin-sdk-python/issues/189)) ([9cafffd](https://github.com/cloudquery/plugin-sdk-python/commit/9cafffdf22aa261a52f55d2148f008ce8d20ec5f)) +* **deps:** Update dependency grpcio-tools to v1.64.1 ([#190](https://github.com/cloudquery/plugin-sdk-python/issues/190)) ([f7eea8d](https://github.com/cloudquery/plugin-sdk-python/commit/f7eea8d30a72e8f4226f75f4d0caef9be8bb36ae)) +* **deps:** Update dependency packaging to v24.1 ([#194](https://github.com/cloudquery/plugin-sdk-python/issues/194)) ([73d18ca](https://github.com/cloudquery/plugin-sdk-python/commit/73d18ca673a1d86de622181fb21c10ed142c9b58)) +* **deps:** Update dependency protobuf to v5.27.2 ([#192](https://github.com/cloudquery/plugin-sdk-python/issues/192)) ([b2e8673](https://github.com/cloudquery/plugin-sdk-python/commit/b2e8673bf4e212ca0983ef0fb1e0849859a466b5)) +* **deps:** Update dependency pytest to v8.2.2 ([#193](https://github.com/cloudquery/plugin-sdk-python/issues/193)) ([e3b240e](https://github.com/cloudquery/plugin-sdk-python/commit/e3b240e594a046b65217b24cb34d53dcb2f9ac3d)) +* **deps:** Update dependency python-dateutil to v2.9.0.post0 ([#195](https://github.com/cloudquery/plugin-sdk-python/issues/195)) ([a2835bb](https://github.com/cloudquery/plugin-sdk-python/commit/a2835bb8b0e3d810b21060d1f46a7873d1a3ea09)) +* **deps:** Update dependency tzdata to v2024 ([#197](https://github.com/cloudquery/plugin-sdk-python/issues/197)) ([a6c7a10](https://github.com/cloudquery/plugin-sdk-python/commit/a6c7a1045e5e703e4ae5200a1c1b06f68f1a11c9)) + ## [0.1.26](https://github.com/cloudquery/plugin-sdk-python/compare/v0.1.25...v0.1.26) (2024-06-03) diff --git a/setup.py b/setup.py index ba4afca..fe1e4b2 100644 --- a/setup.py +++ b/setup.py @@ -51,7 +51,7 @@ ] setuptools.setup( name=name, - version="0.1.26", + version="0.1.27", description=description, long_description=long_description, author="CloudQuery LTD",