Add e2e pytest test for pull command #2227
Conversation
Summary of ChangesHello @telemaco, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly upgrades the testing suite by migrating the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
Reviewer's GuideAdds a new end-to-end pytest suite for the File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request is a great addition, migrating the pull command tests to a comprehensive e2e pytest suite. The new OllamaServer fixture for isolated testing is a solid improvement. The tests cover a wide range of functionalities, from different transports to cache utilization and error handling.
My review includes suggestions to enhance the test code's quality and robustness. I've pointed out minor issues like unnecessary print statements and comment typos. More significantly, I've recommended changes to make a test container-engine-agnostic by removing a hardcoded command, which will improve test coverage. I've also highlighted a FIXME comment that needs attention.
test/e2e/test_pull.py
Outdated
| assert fake_model_registry_url in [model["name"] for model in model_list] | ||
|
|
||
| # Clean fake image | ||
| ctx.check_call(["podman", "rmi", fake_model_registry_url.replace("oci://", "")]) |
There was a problem hiding this comment.
The podman command is hardcoded here, which is why the test is decorated with @skip_if_docker. To make this test work with both podman and docker, you should:
- Remove
@skip_if_dockerfrom line 312. - Add the
container_enginefixture to the test signature on line 313:def test_pull_with_registry(container_registry, container_engine): - Use the
container_enginevariable here.
| ctx.check_call(["podman", "rmi", fake_model_registry_url.replace("oci://", "")]) | |
| ctx.check_call([container_engine, "rmi", fake_model_registry_url.replace("oci://", "")]) |
test/e2e/conftest.py
Outdated
| print(self.url) | ||
| print(requests.get(self.url, timeout=0.5)) |
There was a problem hiding this comment.
These print statements should be removed from the test helper as they add noise to test execution logs, making debugging more difficult. The requests.get call is sufficient on its own to check for server availability.
| print(self.url) | |
| print(requests.get(self.url, timeout=0.5)) | |
| requests.get(self.url, timeout=0.5) |
test/e2e/test_pull.py
Outdated
| with RamalamaExecWorkspace(env_vars=env_vars) as ctx: | ||
| ramalama_cli = ["ramalama", "--store", ctx.storage_dir] | ||
|
|
||
| # Ensure huggingface cache exists and is set as environment variable |
There was a problem hiding this comment.
This comment appears to have a typo. It should likely refer to ollama cache instead of huggingface cache.
| # Ensure huggingface cache exists and is set as environment variable | |
| # Ensure ollama cache exists and is set as environment variable |
test/e2e/test_pull.py
Outdated
| ctx.environ["OLLAMA_HOST"] = ollama_server.url | ||
| ctx.environ["OLLAMA_MODELS"] = ollama_server.models_dir.as_posix() | ||
|
|
||
| # Pull image using huggingface cli |
There was a problem hiding this comment.
This comment seems to have a typo. It should refer to the ollama CLI, not the huggingface CLI.
| # Pull image using huggingface cli | |
| # Pull image using ollama cli |
| @skip_if_no_container | ||
| @skip_if_docker | ||
| def test_pull_with_registry(container_registry): | ||
| # FIXME: Check with the maintainers if some of the original tests with registry are necessary |
There was a problem hiding this comment.
This FIXME comment should be addressed. Please either remove it if it's no longer relevant, or create an issue to track this work and reference the issue number in the comment.
telemaco
force-pushed
the
e2e-pytest-pull-cmd
branch
from
04966f3 to
9dbc13a
Compare
- Migrate to e2e pytest the `test/system/050-pull.bats` bat test. - Ollama fixture has been added to test on a insolated ollama instance - The tests cover a wide range of functionalities, including: - Pulling models from different transports like Ollama, Hugging Face, OCI, and local files. - Verifying the use of local caches from Ollama and Hugging Face to speed up subsequent pulls. - Testing error handling for non-existent models and endianness mismatches. - Ensuring proper interaction with authenticated OCI registries for pulling private models. - Validating the handling of models with multiple layers and references. Signed-off-by: Roberto Majadas <[email protected]>
telemaco
force-pushed
the
e2e-pytest-pull-cmd
branch
from
9dbc13a to
96395fa
Compare
1 participant
test/system/050-pull.batsbat test.Summary by Sourcery
Add end-to-end pytest coverage for the
ramalama pullcommand across multiple model transports and error cases.Enhancements:
Build:
requestsas a development dependency required by new e2e tests.Tests:
ramalama pull, covering Ollama, Hugging Face, OCI, file transports, and model listing.