[graveler] A single failed range and metarange operation can fail a commit or merge

[arielshaqed]

[Observed by a customer]

What happened

A DNS error on a major cloud provider brought down a large commit at a user site. This could have been retried and would have succeeded.

User received this error. A human can deduce that the issue is an unresponsive DNS (UDP port 53) that caused an operation to fail. However a machine cannot, and has no idea what to do with an ISE.

[2023-10-99 99:99:99.999] {foo.py} ERROR - Internal Server Error: {"message":"commit: close writer ns=s3://AWS-BUCKET/ metarange id=fff555: failed closing metarange writer: sstable store (fff555): adapter put s3://AWS-BUCKET/ fff555: Put \"https://AWS-BUCKET.s3.amazonaws.com/_lakefs/fff555\": dial tcp: lookup AWS-BUCKET.s3.amazonaws.com on 172.31.99.99:53: read udp 10.0.99.99:48835-\u003e172.31.99.99:53: i/o timeout"}

What we should do

We should either:

• Retry operations; or
• Give sufficient information on the API response to allow the caller to retry operation.

Preferences

I prefer adding information to the API response: an optional response field (JSON body or header) as well as a readable message that user code can use to trigger a retry. As a server, lakeFS has no idea how hard it should try vs. how quickly it should fail. By passing the problem onto the user, we give them more information. This could be handled automatically -- by wrapping expensive commit operations in a backoff -- or it could be handled manually -- by the user reading the message and deciding that it may be worth retrying this operation.

[arielshaqed]

PS

We could just wrap the error inside the block adaptor inside a "RetriableError", and then add the relevant text and fields in handleError on the controller.

Alternatively, we do not need to do this all at once! We can start by changing just commit/merge, and continue from there to report on more operations. It really is a ladder.

[arielshaqed]

Why retry?

Both apps and users are confused by a message that says some underlying component failed. Even as developers we are not sure whether to recommend retrying after a DNS failure: would we still request a retry if it turned out that the DNS server configuration is broken? However some cases can be analyzed for risk. I believe that:

1. If a metadata operation failed because an infrastructure dependency failed, it is worth retrying.
2. If a basic enough infrastructure dependency fails (e.g. DNS), it is worth retrying.

Most infrastructure never retries, and it never recommends that you retry. As a result application code makes uninformed decisions. lakeFS is used by applications where a single run can be expensive; this changes the cost of falsely claiming that an operation could be retried.

• The Spark behaviour is essentially always to retry. This is actually a mistake: Spark can only retry huge chunks of computation, so it cannot afford to perform many retries. Indeed, we end up adding API retry loops to every Spark component that we develop. Having an actual bit on the response that says whether to retry would improve matters!
• The Airflow behaviour (adapted also by the lakeFS provider, but this is copied from every other provider that I have seen) is a bit better: BaseOperator defines a retries property. But hardly anyone sets it. Again, having an actual bit on the response would help our provider know whether to fail with or without allowing retries.

We will decide on the issue when we can recommend a retry.

[github-actions]

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.

[arielshaqed]

Still relevant.