Merge pull request #23 from CBroz1/dev

Standard integration tests, PEP8, docker adjustments

Author: kabilar

CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and 
+[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
+
+
+## 0.2.0 - 2022-07-11
++ Update - Docker and Compose files
++ Update - Requirements for latest releases of Elements
++ Update - Workflow pipeline, ingestion, and populate scripts
++ Update - Pytests
++ Add - Code of conduct
+
+## 0.1.0 - 2022-04-30
++ Add - Analysis schema and Jupyter notebook for event-aligned calcium activity
++ Add - Data exploration Jupyter notebook
++ Add - Docker and Compose files
++ Update - Pytests
++ Update - README
+
+## 0.1.0a2 - 2021-10-18
++ Update - Change version to reflect release phase
+
+## 0.1.1 - 2021-04-01
++ Add - Version

CODE_OF_CONDUCT.md

@@ -0,0 +1,133 @@
+
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[[email protected]](mailto:[email protected]).
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

README.md

@@ -19,9 +19,13 @@ convention, and directory lookup methods (see
 [workflow_miniscope/ingest.py](workflow_miniscope/ingest.py))
 3. Processing results.
 
-See the [Element Miniscope documentation](https://elements.datajoint.org/description/miniscope/) for the background information and development timeline.
+See the 
+[Element Miniscope documentation](https://elements.datajoint.org/description/miniscope/)
+ for the background information and development timeline.
 
-For more information on the DataJoint Elements project, please visit https://elements.datajoint.org.  This work is supported by the National Institutes of Health.
+For more information on the DataJoint Elements project, please visit 
+https://elements.datajoint.org.  This work is supported by the National Institutes of 
+Health.
 
 ## Workflow architecture
 
@@ -41,21 +45,35 @@ DataJoint Elements ([element-lab](https://github.com/datajoint/element-lab),
 
 ## Interacting with the DataJoint workflow
 
++ Our [YouTube tutorial](https://www.youtube.com/watch?v=nWUcPFZOSVw) walks through all 
+  the key details of this workflow.
+
 + Please refer to the following workflow-specific 
  [Jupyter notebooks](/notebooks) for an in-depth explanation of how to run the 
  workflow ([03-process.ipynb](notebooks/03-process.ipynb)) and explore the data 
  ([05-explore.ipynb](notebooks/05-explore.ipynb)).
 
+
 ## Citation
 
-+ If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
++ If your work uses DataJoint and DataJoint Elements, please cite the respective
+  Research Resource Identifiers (RRIDs) and manuscripts.
 
 + DataJoint for Python or MATLAB
-    + Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
+    + Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton
+      RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or
+      Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
 
-    + DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
+    + DataJoint (
+      [RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version 
+        `<Enter version number>`)
 
 + DataJoint Elements
-    + Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
-
-    + DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Miniscope (version `<Enter version number>`)
+    + Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D,
+      Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for
+      Neurophysiology. bioRxiv. 2021 Jan 1. doi:
+      https://doi.org/10.1101/2021.03.30.437358
+
+    + DataJoint Elements (
+      [RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Miniscope 
+      (version `<Enter version number>`)

docker/Dockerfile.dev

@@ -16,16 +16,18 @@ RUN python caimanmanager.py install --inplace
 WORKDIR /main
 
 RUN mkdir /main/element-lab \
-           /main/element-animal \
-           /main/element-session \
-           /main/element-interface \
-           /main/element-miniscope \
-           /main/workflow-miniscope
+          /main/element-animal \
+          /main/element-session \
+          /main/element-event \
+          /main/element-interface \
+          /main/element-miniscope \
+          /main/workflow-miniscope
 
 # Copy user's local fork of elements and workflow
 COPY --chown=anaconda:anaconda ./element-lab /main/element-lab
 COPY --chown=anaconda:anaconda ./element-animal /main/element-animal
 COPY --chown=anaconda:anaconda ./element-session /main/element-session
+COPY --chown=anaconda:anaconda ./element-event /main/element-event
 COPY --chown=anaconda:anaconda ./element-interface /main/element-interface
 COPY --chown=anaconda:anaconda ./element-miniscope /main/element-miniscope
 COPY --chown=anaconda:anaconda ./workflow-miniscope /main/workflow-miniscope
@@ -34,11 +36,12 @@ COPY --chown=anaconda:anaconda ./workflow-miniscope /main/workflow-miniscope
 RUN pip install -e /main/element-lab
 RUN pip install -e /main/element-animal
 RUN pip install -e /main/element-session
+RUN pip install -e /main/element-event
 RUN pip install -e /main/element-interface
 RUN pip install -e /main/element-miniscope
 RUN pip install -e /main/workflow-miniscope
 RUN pip install -r /main/workflow-miniscope/requirements_test.txt
 
 WORKDIR /main/workflow-miniscope
 
-ENTRYPOINT ["tail", "-f", "/dev/null"]
+ENTRYPOINT ["tail", "-f", "/dev/null"]

docker/Dockerfile.test

@@ -1,58 +1,50 @@
 FROM datajoint/djbase:py3.9-debian-8eb1715
 
+ARG GITHUB_USERNAME=datajoint
 USER anaconda:anaconda
 
-COPY ./workflow-miniscope/docker/apt_requirements.txt /tmp/
+COPY ./docker/apt_requirements.txt /tmp/
 RUN /entrypoint.sh echo "Installed dependencies."
 
 # Install CaImAn
-RUN git clone --branch master https://github.com/kabilar/CaImAn 
+WORKDIR /main
+RUN git clone --branch master https://github.com/kabilar/CaImAn
 WORKDIR /main/CaImAn
 RUN conda install -n base -c conda-forge -y mamba
-RUN /bin/bash -c 'mamba env update --n base --file environment.yml' 
+RUN /bin/bash -c 'mamba env update --n base --file environment-minimal.yml'
 RUN pip install .
 RUN python caimanmanager.py install --inplace
 
-WORKDIR /main
-
-# Option 1 - Install DataJoint's remote fork of the workflow and elements
-# RUN git clone https://github.com/datajoint/workflow-miniscope.git /main/
-
-# Option 2 - Install user's remote fork of element and workflow
-#            or an unreleased version of the element
-# RUN pip install git+https://github.com/<user>/element-lab.git
-# RUN pip install git+https://github.com/<user>/element-animal.git
-# RUN pip install git+https://github.com/<user>/element-session.git
-# RUN pip install "element-interface@git+https://github.com/<user>/element-interface"
-# RUN pip install git+https://github.com/<user>/element-miniscope.git
-# RUN git clone https://github.com/<user>/workflow-miniscope.git /main/workflow-miniscope
-
-# Option 3 - Install user's local fork of element and workflow
-RUN mkdir /main/element-lab \
-          /main/element-animal \
-          /main/element-session \
-          /main/element-interface \
-          /main/element-miniscope \
-          /main/workflow-miniscope
-
-COPY --chown=anaconda:anaconda ./element-lab /main/element-lab
-COPY --chown=anaconda:anaconda ./element-animal /main/element-animal
-COPY --chown=anaconda:anaconda ./element-session /main/element-session
-COPY --chown=anaconda:anaconda ./element-interface /main/element-interface
-COPY --chown=anaconda:anaconda ./element-miniscope /main/element-miniscope
-COPY --chown=anaconda:anaconda ./workflow-miniscope /main/workflow-miniscope
-
-RUN pip install -e /main/element-lab
-RUN pip install -e /main/element-animal
-RUN pip install -e /main/element-session
-RUN pip install -e /main/element-interface
-RUN pip install -e /main/element-miniscope
-RUN rm -f /main/workflow-miniscope/dj_local_conf.json
-
-# Install the workflow
-RUN pip install /main/workflow-miniscope
-RUN pip install -r /main/workflow-miniscope/requirements_test.txt
-
-RUN pip uninstall datajoint
-RUN pip install git+
 WORKDIR /main/workflow-miniscope
+
+# 1. Install Elements from github user
+RUN pip install --no-deps "element-interface@git+https://github.com/${GITHUB_USERNAME}/element-interface"
+RUN pip install --no-deps "djarchive-client@git+https://github.com/${GITHUB_USERNAME}/djarchive-client"
+RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-lab.git
+RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-animal.git
+RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-session.git
+RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-event.git
+RUN pip install git+https://github.com/${GITHUB_USERNAME}/element-miniscope.git
+# RUN pip install git+https://github.com/${GITHUB_USERNAME}/workflow-miniscope.git
+# end 1
+
+# 2. Install local fork of element and workflow
+# COPY --chown=anaconda:anaconda ../element-lab /main/
+# COPY --chown=anaconda:anaconda ../element-animal /main/
+# COPY --chown=anaconda:anaconda ../element-session /main/
+# COPY --chown=anaconda:anaconda ../element-event /main/
+# COPY --chown=anaconda:anaconda ../element-interface /main/
+# COPY --chown=anaconda:anaconda ../element-miniscope /main/
+COPY --chown=anaconda:anaconda . ./
+# RUN pip install -e /main/element-lab
+# RUN pip install -e /main/element-animal
+# RUN pip install -e /main/element-session
+# RUN pip install -e /main/element-event
+# RUN pip install -e /main/element-interface
+# RUN pip install -e /main/element-miniscope
+RUN pip install -e .
+# end 2
+
+# Permission denied on rm
+RUN mv ./dj_local_conf.json ./dj_local_conf-Unused.json
+RUN pip install -r ./requirements_test.txt

docker/apt_requirements.txt

@@ -5,4 +5,4 @@ ffmpeg
 libsm6
 libxext6
 libhdf5-dev
-pkg-config
+pkg-config

docker/docker-compose-dev.yaml

@@ -35,4 +35,4 @@ services:
       db:
         condition: service_healthy
 networks:
-  main:
+  main: