Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
7c17057
fix: https://github.com/unclecode/crawl4ai/issues/756
aravindkarnam Mar 1, 2025
5edfea2
Fix LiteLLM branding and link
joshrad-dev Mar 2, 2025
1e819cd
fixes: https://github.com/unclecode/crawl4ai/issues/774
aravindkarnam Mar 3, 2025
f14e4a4
Merge pull request #776 from jawshoeadan/patch-1
aravindkarnam Mar 3, 2025
504207f
docs: update text in llm-strategies.md to reflect new changes in LlmC…
aravindkarnam Mar 3, 2025
341b7a5
🐛 Truncate width to integer string in parse_srcset
dvschuyl Mar 11, 2025
cbb8755
Merge branch 'next' into 2025-MAR-ALPHA-1
aravindkarnam Mar 13, 2025
a3954dd
refactor: Move the checking of protocol and prepending protocol insid…
aravindkarnam Mar 14, 2025
c190ba8
refactor: Instead of custom validation of question, rely on the built…
aravindkarnam Mar 14, 2025
84883be
Merge branch 'next' into 2025-MAR-ALPHA-1
aravindkarnam Mar 18, 2025
9109ecd
chore: Raise an exception with clear messaging when body tag is missi…
aravindkarnam Mar 18, 2025
529a797
docs: remove hallucinations from docs for CrawlerRunConfig + Add chun…
aravindkarnam Mar 18, 2025
4359b12
docs + fix: Update example for full page screenshot & PDF export. Fix…
aravindkarnam Mar 18, 2025
8cecbec
Merge branch 'next' into 2025-MAR-ALPHA-1
aravindkarnam Mar 20, 2025
eedda1a
fix: Truncate long urls in middle than end since users are confused t…
aravindkarnam Mar 20, 2025
ac2f9ae
fix: streamline url status logging via single entrypoint i.e. logger.…
aravindkarnam Mar 20, 2025
e0c2a7c
chore: remove mistakenly commited deps.txt file
aravindkarnam Mar 21, 2025
8b761f2
fix: improve logged url readability by decoding encoded urls
aravindkarnam Mar 21, 2025
6740e87
fix: remove trailing slash when the path is empty. This is causing du…
aravindkarnam Mar 21, 2025
f891133
fix: Move adding of visited urls to the 'visited' set, when queueing …
aravindkarnam Mar 21, 2025
471d110
fix: url normalisation ref: https://github.com/unclecode/crawl4ai/iss…
aravindkarnam Mar 21, 2025
e01d1e7
fix: link normalisation in BestFirstStrategy
aravindkarnam Mar 21, 2025
efa7325
Merge branch 'next' into 2025-MAR-ALPHA-1
aravindkarnam Mar 24, 2025
2f0e217
Chore: Add brotli as dependancy to fix: https://github.com/unclecode/…
aravindkarnam Mar 25, 2025
e3111d0
fix: prevent session closing after each request to maintain connectio…
aravindkarnam Mar 25, 2025
585e5e5
fix: https://github.com/unclecode/crawl4ai/issues/733
aravindkarnam Mar 25, 2025
7be5427
Merge branch 'next' into 2025-MAR-ALPHA-1
aravindkarnam Mar 27, 2025
c635f6b
refactor(browser): reorganize browser strategies and improve Docker i…
unclecode Mar 27, 2025
57e0423
fix:target_element should not affect link extraction. -> https://gith…
aravindkarnam Mar 28, 2025
64f20ab
refactor(docker): update Dockerfile and browser strategy to use Chromium
unclecode Mar 28, 2025
d8cbeff
fix: https://github.com/unclecode/crawl4ai/issues/842
aravindkarnam Mar 28, 2025
3ff7eec
refactor(browser): consolidate browser strategy implementations
unclecode Mar 28, 2025
bb02398
refactor(browser): improve browser strategy architecture and lifecycl…
unclecode Mar 30, 2025
1119f2f
fix: https://github.com/unclecode/crawl4ai/issues/911
maggie-edkey Mar 31, 2025
ef1f0c4
fix:https://github.com/unclecode/crawl4ai/issues/701
aravindkarnam Mar 31, 2025
d8357e8
Merge pull request #915 from maggie-edkey/css-selector
aravindkarnam Mar 31, 2025
757e317
fix: https://github.com/unclecode/crawl4ai/issues/839
aravindkarnam Mar 31, 2025
765f856
Merge pull request #808 from dvschuyl/bug/parse-srcset-fix-float-width
aravindkarnam Mar 31, 2025
555455d
feat(browser): implement browser pooling and page pre-warming
unclecode Mar 31, 2025
c5cac2b
feat(browser): add BrowserHub for centralized browser management and …
unclecode Apr 1, 2025
9e16a4b
Merge next and resolve conflicts
aravindkarnam Apr 2, 2025
179921a
fix(crawler): update get_page call to include additional return value
unclecode Apr 2, 2025
86df202
fix(crawler): handle exceptions in get_page call to ensure page retri…
unclecode Apr 2, 2025
73fda8a
fix: address the PR review: https://github.com/unclecode/crawl4ai/pul…
aravindkarnam Apr 3, 2025
4133e54
typo-fix: https://github.com/unclecode/crawl4ai/pull/918
aravindkarnam Apr 3, 2025
7155778
chore: move from faust-cchardet to chardet
aravindkarnam Apr 3, 2025
14894b4
feat(config): set DefaultMarkdownGenerator as the default markdown ge…
unclecode Apr 3, 2025
b1693b1
Remove old quickstart files
unclecode Apr 5, 2025
591f55e
refactor(browser): rename methods and update type hints in BrowserHub…
unclecode Apr 6, 2025
5b66208
Refactor next branch
unclecode Apr 6, 2025
02e627e
fix(crawler): simplify page retrieval logic in AsyncPlaywrightCrawler…
unclecode Apr 8, 2025
9038e9a
Merge branch 'main' into next
unclecode Apr 8, 2025
6f7ab9c
fix: Revert changes to session management in AsyncHttpWebcrawler and …
aravindkarnam Apr 8, 2025
a2061bf
feat(crawler): add MHTML capture functionality
unclecode Apr 9, 2025
66ac07b
feat(crawler): add network request and console message capturing
unclecode Apr 10, 2025
108b2a8
Fixed capturing console messages for case the url is the local file. …
unclecode Apr 10, 2025
7c358a1
fix(browser): add null check for crawlerRunConfig.url
unclecode Apr 10, 2025
18e8227
feat(crawler): add console message capture functionality
unclecode Apr 10, 2025
3179d6a
fix(core): improve error handling and stability in core components
unclecode Apr 11, 2025
022f5c9
Merged next branch
aravindkarnam Apr 12, 2025
d84508b
fix: revert the old target_elms code in regular webscraping strategy
aravindkarnam Apr 12, 2025
9fc5d31
fix: revert the old target_elms code in LXMLwebscraping strategy
aravindkarnam Apr 12, 2025
7d8e81f
fix: fix target_elements, in a less invasive and more efficient way s…
aravindkarnam Apr 12, 2025
ecec53a
Docker tested on Windows machine.
unclecode Apr 13, 2025
dcc2654
fix: Add a nominal wait time for remove overlay elements since it's a…
aravindkarnam Apr 14, 2025
c56974c
feat(docs): enhance documentation UI with ToC and GitHub stats
unclecode Apr 14, 2025
cd7ff6f
feat(docs): add AI assistant interface and code copy button
unclecode Apr 14, 2025
82aa53a
Merge branch 'next-alpine-docker' into next
unclecode Apr 14, 2025
793668a
Remove parameter_updates.txt
unclecode Apr 14, 2025
230f22d
refactor(proxy): move ProxyConfig to async_configs and improve LLM to…
unclecode Apr 15, 2025
5206c6f
Modify the test file
unclecode Apr 15, 2025
94d4865
docs(tests): clarify server URL comments in deep crawl tests
unclecode Apr 15, 2025
eed7f88
Merge branch 'next' into 2025-MAR-ALPHA-1
aravindkarnam Apr 17, 2025
7db6b46
feat(markdown): add content source selection for markdown generation
unclecode Apr 17, 2025
30ec4f5
feat(docs): add comprehensive Docker API demo script
unclecode Apr 17, 2025
fd899f6
Merge branch 'next-fix-markdown-source' into next
unclecode Apr 17, 2025
921e0c4
feat(tests): implement high volume stress testing framework
unclecode Apr 17, 2025
3bf78ff
refactor(docker-demo): enhance error handling and output formatting
unclecode Apr 17, 2025
907cba1
Merge branch 'next-stress' into next
unclecode Apr 17, 2025
16b2318
feat(api): implement crawler pool manager for improved resource handling
unclecode Apr 18, 2025
c2902fd
reverse:last change in order of execution for it introduced a new iss…
aravindkarnam Apr 19, 2025
d2648ea
fix: solved with deepcopy of elements https://github.com/unclecode/cr…
aravindkarnam Apr 19, 2025
b27bb36
merge next. Resolve conflicts. Fix some import errors and error hand…
aravindkarnam Apr 19, 2025
a58c800
refactor(server): migrate to pool-based crawler management
unclecode Apr 20, 2025
5297e36
feat(mcp): Implement MCP protocol and enhance server capabilities
unclecode Apr 21, 2025
b5c2573
feat(browser): add geolocation, locale and timezone support
unclecode Apr 21, 2025
0007aea
Update changelog
unclecode Apr 21, 2025
f3ebb38
Merge PR #899 into next, resolve conflicts in server.py and docs/brow…
unclecode Apr 22, 2025
4812f08
feat(docker): update Docker deployment for v0.6.0
unclecode Apr 22, 2025
c98ffe2
Update CHANGELOG
unclecode Apr 22, 2025
b0aa8bc
Update README
unclecode Apr 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: Add a nominal wait time for remove overlay elements since it's a…
…lready controllable through delay_before_return_html
  • Loading branch information
aravindkarnam committed Apr 14, 2025
commit dcc265458cef022a6b03bcaa47686e08869bcb02
2 changes: 1 addition & 1 deletion crawl4ai/js_snippet/remove_overlay_elements.js
Original file line number Diff line number Diff line change
Expand Up @@ -116,5 +116,5 @@ async () => {

// Wait a bit for any animations to complete
document.body.scrollIntoView(false);
await new Promise((resolve) => setTimeout(resolve, 250));
await new Promise((resolve) => setTimeout(resolve, 50));
};