Add Per-Host Rate Limiting and Caching

[mre]

Currently, lychee faces challenges with rate limiting and cache effectiveness when checking links, particularly when dealing with multiple requests to the same hosts. This leads to several issues that need to be addressed:

Current Problems

• Multiple concurrent requests to the same host trigger rate limits (429 errors) (See Add custom delay inbetween requests (prevent ban) #989)
• Cache is ineffective with high concurrency due to race conditions (See The cache is ineffective with the default concurrency, for links in a website's theme #1593 (comment))
• Global concurrency settings are too coarse-grained
• Different hosts have different rate limit requirements
• Headers are applied to all hosts, causing potential security issues (Security: restrict custom HTTP request headers to specific URL patterns #1298 and custom Header not sent #1441 (comment))

Proposed Solution

We should implement a smart per-host rate limiting and caching system that would:

1. Track rate limits per host using a concurrent HashMap:

use std::collections::HashMap;
use time::OffsetDateTime;

struct HostConfig {
    rate_limit_reset: Option<OffsetDateTime>,
    request_delay: Option<Duration>,
    max_concurrent_requests: Option<u32>,
}

2. Implement smarter caching:

• Maintain separate cache states per host

3. Stretch goal: Add configuration options per host:

lychee --max-concurrency-per-host github.com=10 --delay-per-host github.com=100ms

4. Stretch goal II: Add support for per-host headers

The idea would be to maintain a HeaderMap.
See #1297 for details.

Implementation Notes

• Use the existing rate-limits crate, which is mostly useful for APIs
• Handle 429 responses with proper backoff using response headers when available

Benefits

• Prevents IP bans from aggressive checking
• More efficient resource usage
• Better compliance with API rate limits
• Improved cache effectiveness. Since the cache is per host, there would be no synchronization issues
• Faster overall execution by avoiding unnecessary retries

Examples

[hosts."github.com"]
max_concurrent_requests = 10
request_delay = "100ms"
headers = { Authorization = "token ghp_xxxx", "User-Agent" = "my-bot" }

[hosts."api.example.com"] 
max_concurrent_requests = 1
request_delay = "1s"
headers = { "X-API-Key" = "secret", Accept = "application/json" }

CLI usage example:

lychee --max-concurrency-per-host github.com=10 --delay-per-host github.com=100ms

And when adding headers:

lychee \
  --max-concurrency-per-host github.com=10 \
  --delay-per-host github.com=100ms \
  --headers-per-host 'github.com=Authorization:token ghp_xxxx,User-Agent:my-bot'

This is just a proposal. I'm not 100% certain about the naming yet.

Related issues: #989, #1593

[mre]

Also see:

• Look into tower as a replacement for deadpool + channels #36
• https://github.com/james7132/Hourai/blob/4a4b343f237f726ae1d2e2943e0bed58406f2cf6/logger/hourai-feeds/src/reddit/rate_limiter.rs

[wkaisertexas]

Could you not just add a global cache of sites visited and use just a mutex lock on this?

The cost of editing a single-line entry in a hash map vs making a network request to fetch a website is such a disparity that this global visited state likely would not slow down parallel execution that much.

[mre]

Could you not just add a global cache of sites visited and use just a mutex lock on this?

Oh yeah, we do. The overhead is indeed minimal, that is not an issue. We use DashMap under the hood, which is lockless.

The problem is that requests get executed concurrently, so in the worst case we might end up making the same request multiple times before one of them completes and updates the cache.

A simple way to fix that is to have one central point that filters out duplicate requests. That sounds easy enough, but it introduces a bottleneck that defeats the purpose of concurrent checking. The real challenge is coordinating requests efficiently without sacrificing parallelism.

It gets more complicated with rate-limiting, which is different per host. So caching alone isn't enough because it only solves the duplicate request problem. There are other reasons for per-host settings, such as:

• Different API rate limits per host
• Custom headers needed only for certain hosts (Which is relevant for auth headers for example, which may contain tokens)
• Host-specific retry strategies (one might retry on some hosts, but not on others)
• ...

Per-host rate limiting solves this more elegantly by:

1. Tracking in-flight requests per host
2. Queueing duplicate requests while one is processing
3. Sharing the result with all sources once the first request completes
4. Respecting each host's rate limits independently

This way, we avoid duplicate requests while maintaining high concurrency across different hosts. It's more complex than a global lock, but gives us the best of both worlds.

[wkaisertexas]

Ahhh…. as a note, I think indicating whether or not a request was cached in addition to a check or an x might be a good idea. Maybe using a faded green or read for these requests would be a good idea. You could also do something like ‘x(cached)’ to more explicitly indicate this, but that may get verbose.

…

On Sat, May 10, 2025 at 6:13 AM Matthias Endler ***@***.***> wrote: *mre* left a comment (lycheeverse/lychee#1605) <#1605 (comment)> Could you not just add a global cache of sites visited and use just a mutex lock on this? Oh yeah, we do <https://github.com/lycheeverse/lychee/blob/master/lychee-bin/src/cache.rs>. The overhead is indeed minimal, that is not an issue. We use DashMap under the hood, which is lockless. The problem is that requests get executed concurrently, so in the worst case we might end up making the same request multiple times before one of them completes and updates the cache. A simple way to fix that is to have one central point that filters out duplicate requests. That sounds easy enough, but it introduces a bottleneck that defeats the purpose of concurrent checking. The real challenge is coordinating requests efficiently without sacrificing parallelism. It gets more complicated with rate-limiting, which is different per host. So caching alone isn't enough because it only solves the duplicate request problem. There are other reasons for per-host settings, such as: - Different API rate limits per host - Custom headers needed only for certain hosts (Which is relevant for auth headers for example, which may contain tokens) - Host-specific retry strategies (one might retry on some hosts, but not on others) - ... Per-host rate limiting solves this more elegantly by: 1. Tracking in-flight requests per host 2. Queueing duplicate requests while one is processing 3. Sharing the result with all sources once the first request completes 4. Respecting each host's rate limits independently This way, we avoid duplicate requests while maintaining high concurrency across different hosts. It's more complex than a global lock, but gives us the best of both worlds. — Reply to this email directly, view it on GitHub <#1605 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGUB4RQWDQVLZJBRSPVBI6L25XNPNAVCNFSM6AAAAABUVXMSQCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNRYG42DSNBSG4> . You are receiving this because you commented.Message ID: ***@***.***>

[thomaseizinger]

The problem is that requests get executed concurrently, so in the worst case we might end up making the same request multiple times before one of them completes and updates the cache.

You might want to check out moka: https://docs.rs/moka/latest/moka/

It has a feature of deduplicating requests for the same cache key until the first one resolves.