feat: web scraping connector task#1501

Merged

Vasilije1990 merged 58 commits intotopoteretes:devfrom

Geoff-Robin:feature/web_scraping_connector_task

Oct 11, 2025

Commits on Sep 30, 2025

Commits on Oct 2, 2025

first iteration of bs4_connector.py done
Geoff-Robin
committed

Commits on Oct 3, 2025

Changed bs4_connector.py to bs4_crawler.py
Geoff-Robin
committed

Commits on Oct 6, 2025

Commits on Oct 7, 2025

Removed staticmethod decorator from bs4_crawler.py, kwargs from the function signature in save_data_item_to_storage.py, removed unused imports in ingest_data.py and added robots_cache_ttl as a conf…
Geoff-Robin
committed
Removed print statement that I used for debugging
Geoff-Robin
committed
Added self as an argument to all previous methods that were static methods
Geoff-Robin
committed
Closed crawler instance in a finally block
Geoff-Robin
committed
ruff formatted
Geoff-Robin
committed
Added await
Geoff-Robin
committed
Made api_key field in TavilyConfig models to be Optional[str] type to allow None value
Geoff-Robin
committed
Release v0.3.5 (#1515)
borisarzentar
authored

Commits on Oct 9, 2025

Commits on Oct 10, 2025

Commits on Oct 11, 2025

Merge branch 'dev' into feature/web_scraping_connector_task
Geoff-Robin
authored