Skip to content

Standard integration tests, PEP8, docker adjustments #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Jul 12, 2022
Merged

Standard integration tests, PEP8, docker adjustments #23

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
dcb062a
WIP PEP8
CBroz1 May 2, 2022
2f842b2
WIP Draft integration tests
CBroz1 May 4, 2022
bbfe140
WIP standard pytests. missing element-event. passed CLI, need docker …
CBroz1 May 4, 2022
c143548
WIP docker edits
CBroz1 May 5, 2022
7d4f879
minor docker changes
CBroz1 May 6, 2022
0cfdcf3
Revised docker from @iamamutt
CBroz1 May 13, 2022
90d0d74
Apply suggestions from code review
CBroz1 May 16, 2022
c77ac0c
WIP. Replace with dj image. TLS config statement
CBroz1 May 17, 2022
39d252d
Docker success *if detatched*
CBroz1 May 23, 2022
6a66ec4
Add contact info to Code of Conduct
CBroz1 Jun 17, 2022
a875dca
Remove order constraints in test_pipeline_generation
CBroz1 Jun 23, 2022
a777f34
WIP: Apply suggestions 1
CBroz1 Jul 8, 2022
c34d325
WIP: Apply suggestions 2
CBroz1 Jul 8, 2022
7e41a3a
WIP: Apply suggestions 3. All except Docker
CBroz1 Jul 8, 2022
217a84d
Revert dockerfile arg; adjust 05-explore
CBroz1 Jul 8, 2022
d7165ac
Apply suggestions from code review
CBroz1 Jul 12, 2022
8435df2
WIP. See details. Jupysync, verbose context, abc.Seq, etc.
CBroz1 Jul 12, 2022
b1e45a2
WIP: verbosity if to oneliner
CBroz1 Jul 12, 2022
68b04ed
Version bump
CBroz1 Jul 12, 2022
6fc72ee
Notebook fix black'd issues. Docker compose vol map
CBroz1 Jul 12, 2022
fd89eba
Revise notebook 02 - formatting issues
CBroz1 Jul 12, 2022
50fcc20
Revert nb 05
CBroz1 Jul 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revised docker from @iamamutt 
tests/test_ingest.py::test_ingest_subjects PASSED
tests/test_ingest.py::test_ingest_sessions PASSED
tests/test_ingest.py::test_paramset_insert ERROR
tests/test_pipeline_generation.py::test_generate_pipeline PASSED
tests/test_populate.py::test_recording_info_populate PASSED
tests/test_populate.py::test_processing_populate ERROR

ERRORs because AttributeError: module 'collections' has no attribute 'Mapping'
  • Loading branch information
@CBroz1
CBroz1 committed May 13, 2022
commit 0cfdcf3641e0313d5cccf761fdd9cea44d8cbee2
82 changes: 36 additions & 46 deletions docker/Dockerfile.test
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,59 +1,49 @@
FROM datajoint/djbase:py3.9-debian-8eb1715

ARG GITHUB_USERNAME=cbroz1
USER anaconda:anaconda

COPY ./workflow-miniscope/docker/apt_requirements.txt /tmp/
COPY ./docker/apt_requirements.txt /tmp/
RUN /entrypoint.sh echo "Installed dependencies."

# Install CaImAn
RUN git clone --branch master https://github.com/kabilar/CaImAn
WORKDIR /main
RUN git clone --branch master https://github.com/kabilar/CaImAn
WORKDIR /main/CaImAn
RUN conda install -n base -c conda-forge -y mamba
RUN /bin/bash -c 'mamba env update --n base --file environment.yml'
RUN /bin/bash -c 'mamba env update --n base --file environment.yml'
RUN pip install .
RUN python caimanmanager.py install --inplace

WORKDIR /main

# Option 1 - Install DataJoint's remote fork of the workflow and elements
# RUN git clone https://github.com/datajoint/workflow-miniscope.git /main/

# Option 2 - Install user's remote fork of element and workflow
# or an unreleased version of the element
# RUN pip install git+https://github.com/<user>/element-lab.git
# RUN pip install git+https://github.com/<user>/element-animal.git
# RUN pip install git+https://github.com/<user>/element-session.git
# RUN pip install "element-interface@git+https://github.com/<user>/element-interface"
# RUN pip install git+https://github.com/<user>/element-miniscope.git
# RUN git clone https://github.com/<user>/workflow-miniscope.git /main/workflow-miniscope

# Option 3 - Install user's local fork of element and workflow
RUN mkdir /main/element-lab \
/main/element-animal \
/main/element-session \
/main/element-event \
/main/element-interface \
/main/element-miniscope \
/main/workflow-miniscope

COPY --chown=anaconda:anaconda ./element-lab /main/element-lab
COPY --chown=anaconda:anaconda ./element-animal /main/element-animal
COPY --chown=anaconda:anaconda ./element-session /main/element-session
COPY --chown=anaconda:anaconda ./element-event /main/element-event
COPY --chown=anaconda:anaconda ./element-interface /main/element-interface
COPY --chown=anaconda:anaconda ./element-miniscope /main/element-miniscope
COPY --chown=anaconda:anaconda ./workflow-miniscope /main/workflow-miniscope

RUN pip install -e /main/element-lab
RUN pip install -e /main/element-animal
RUN pip install -e /main/element-session
RUN pip install -e /main/element-event
RUN pip install -e /main/element-interface
RUN pip install -e /main/element-miniscope
RUN mv -f /main/workflow-miniscope/dj_local_conf.json /main/workflow-miniscope/unused_local_conf.json

# Install the workflow
RUN pip install /main/workflow-miniscope
RUN pip install -r /main/workflow-miniscope/requirements_test.txt

WORKDIR /main/workflow-miniscope

# 1. Install Elements from github user
RUN pip install --no-deps "element-interface@git+https://github.com/${GITHUB_USERNAME}/element-interface"
RUN pip install --no-deps "djarchive-client@git+https://github.com/${GITHUB_USERNAME}/djarchive-client"
RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-lab.git
RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-animal.git
RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-session.git
RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-event.git
RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-miniscope.git
# RUN pip install git+https://github.com/${GITHUB_USERNAME}/workflow-miniscope.git
# end 1

# 2. Install local fork of element and workflow
# COPY --chown=anaconda:anaconda ../element-lab /main/
# COPY --chown=anaconda:anaconda ../element-animal /main/
# COPY --chown=anaconda:anaconda ../element-session /main/
# COPY --chown=anaconda:anaconda ../element-event /main/
# COPY --chown=anaconda:anaconda ../element-interface /main/
# COPY --chown=anaconda:anaconda ../element-miniscope /main/
COPY --chown=anaconda:anaconda . ./
# RUN pip install -e /main/element-lab
# RUN pip install -e /main/element-animal
# RUN pip install -e /main/element-session
# RUN pip install -e /main/element-event
# RUN pip install -e /main/element-interface
# RUN pip install -e /main/element-miniscope
RUN pip install -e .
# end 2

RUN rm -f ./dj_local_conf.json
RUN pip install -r ./requirements_test.txt
68 changes: 44 additions & 24 deletions docker/docker-compose-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,32 @@
# export COMPOSE_DOCKER_CLI_BUILD=0 # some machines need for smooth --build
# .env file: TEST_DATA_DIR=<local parent directory of workflow_mini_data{1,2}>
# docker-compose -f ./docker/docker-compose-test.yaml up --build
# docker-compose -f ./docker/docker-compose-test.yaml down
# docker-compose -f ./docker/docker-compose-test.yaml up --build --force-recreate --detach
# docker-compose -f ./docker/docker-compose-test.yaml down --volumes

version: "2.4"
x-net: &net
networks:
- main

services:
db:
<<: *net
image: datajoint/mysql:5.7
container_name: workflow-miniscope-test-db
networks:
miniscope:
image: mysql:5-debian
platform: linux/amd64
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
- MYSQL_ROOT_PASSWORD=simple
MYSQL_ROOT_PASSWORD: simple
MYSQL_USER: datajoint
MYSQL_PASSWORD: datajoint
volumes:
- mysql-miniscope:/var/lib/mysql
container_name: workflow-miniscope-test-db

workflow:
<<: *net
networks:
miniscope:
build:
context: ../../
dockerfile: ./workflow-miniscope/docker/Dockerfile.test
env_file: .env # TEST_DATA_DIR
context: ../.
dockerfile: ./docker/Dockerfile.test
args:
- GITHUB_USERNAME=cbroz1
image: workflow-miniscope-test:0.1.0
container_name: workflow-miniscope-test
environment:
Expand All @@ -37,16 +44,29 @@ services:
tail -f /dev/null
volumes:
- ${TEST_DATA_DIR}:/main/test_data
- ./apt_requirements.txt:/tmp/apt_requirements.txt
- ../../element-lab:/main/element-lab
- ../../element-animal:/main/element-animal
- ../../element-session:/main/element-session
- ../../element-event:/main/element-event
- ../../element-interface:/main/element-interface
- ../../element-miniscope:/main/element-miniscope
# - ./apt_requirements.txt:/tmp/apt_requirements.txt
# - ../element-lab:/main/element-lab
# - ../element-animal:/main/element-animal
# - ../element-session:/main/element-session
# - ../element-event:/main/element-event
# - ../element-interface:/main/element-interface
# - ../element-miniscope:/main/element-miniscope
- ..:/main/workflow-miniscope
depends_on:
db:
condition: service_healthy
condition: service_started

adminer:
networks:
miniscope:
image: adminer
restart: always
ports:
- 8082:8080

networks:
main:
miniscope:
name: workflow-miniscope-network

volumes:
mysql-miniscope:
Comment on lines +67 to +68
Copy link
Collaborator

@kabilar kabilar Jul 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required since the above code is commented out?

Copy link
Contributor Author

@CBroz1 CBroz1 Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @iamamutt - Could you remind me why volumes is here? I see it in the copy you drafted prior to this PR. Does networks not provide what we need on it's own?

Copy link

@iamamutt iamamutt Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CBroz1 That was there to map the data from the mysql database to a docker volume instead of bind mounting to the local filesystem. If you don't need access to the database data at all, you can delete the volumes section at the bottom, +72 to +73, and also it's mention at lines +19 to +20.

If you want a bind mounted volume, change - mysql-miniscope:/var/lib/mysql to - path/to/some/data/dir:/var/lib/mysql, and you can delete the volumes section at the end of the file.

Copy link
Contributor Author

@CBroz1 CBroz1 Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. Thanks @iamamutt!

10 changes: 5 additions & 5 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
datajoint>=0.13.0
element-lab==0.1.0b0
element-animal==0.1.0b0
element-session==0.1.0b0
element-miniscope==0.1.0
element-lab>=0.1.0b0
element-animal>=0.1.0b0
element-session>=0.1.0b0
element-miniscope>=0.1.0
element-interface @ git+https://github.com/datajoint/element-interface.git
djarchive-client @ git+https://github.com/datajoint/djarchive-client.git
jupytext==1.13.7
jupytext>=1.13.7
2 changes: 1 addition & 1 deletion workflow_miniscope/pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

from element_lab import lab
from element_animal import subject
from element_session import session_with_datetime as session
from element_session import session
from element_event import trial, event
from element_miniscope import miniscope

Expand Down