Store attachments as intact files #659

dimitri-yatsenko · 2019-09-14T01:04:32Z

…nto attach

all tests pass

coveralls · 2019-09-20T16:07:25Z

Coverage decreased (-0.5%) to 91.475% when pulling 8aa1543 on dimitri-yatsenko:attach into aae57ac on datajoint:dev.

coveralls · 2019-09-20T16:07:25Z

Coverage decreased (-0.5%) to 91.475% when pulling 8aa1543 on dimitri-yatsenko:attach into aae57ac on datajoint:dev.

coveralls · 2019-09-20T16:07:25Z

Coverage decreased (-0.6%) to 91.438% when pulling 48147f1 on dimitri-yatsenko:attach into ef5543b on datajoint:dev.

…ment

guzman-raphael · 2019-09-23T15:24:19Z

Can you please remove python 3.4 from .travis.yml if we no longer plan on testing against it?

…nto attach

guzman-raphael · 2019-10-03T16:53:33Z

Please merge in the latest from dev to include new testing structure.

guzman-raphael · 2019-10-03T16:55:17Z

tests/test_filepath.py

    relpath = 'one/two/three'
-    os.makedirs(os.path.join(stage_path, relpath), exist_ok=True)
-    managed_file = os.path.join(stage_path, relpath, filename)
+    managed_file = Path(stage_path, relpath, filename)


This test failed on WIN10. For example, the result of managed_file was C:\\Users\\rguzman\\AppData\\Local\\Temp\\tmpb3gvp0ht\\one\two\threepicture.dat.

guzman-raphael · 2019-10-03T16:57:31Z

tests/test_filepath.py

+    ext.delete(delete_external_files=True)


 def test_filepath_s3():


This test failed on WIN10. For example, the result of managed_file was dj/repo/one\two\three\picture.dat.

guzman-raphael

Updated with errors from running nosetests on WIN10.

guzman-raphael · 2019-10-03T17:09:23Z

tests/test_filepath.py

+    ext.upload_filepath(managed_file)  # this is fine because the file is the same


 def test_duplicate_upload_s3():


This test failed on WIN10. For example, the result of managed_file was dj/repo/one\two\three\plot.dat.

guzman-raphael · 2019-10-03T17:11:14Z

tests/test_filepath.py

+    ext.upload_filepath(managed_file)  # this should raise exception because the file has changed


 def test_duplicate_error_s3():


This test failed on WIN10. For example, the result of managed_file was dj/repo/one\two\three\thesis.dat.

guzman-raphael · 2019-10-03T17:12:14Z

tests/test_filepath.py

    relative_path = 'one/two/three'
-    os.makedirs(os.path.join(stage_path, relative_path), exist_ok=True)
-    managed_file = os.path.join(stage_path, relative_path, filename)
+    managed_file = Path(stage_path, relative_path, 'attachment.dat')


This test failed on WIN10. For example, the result of managed_file was C:\\Users\\rguzman\\AppData\\Local\\Temp\\tmpb3gvp0ht\\one\two\threeattachment.dat.

guzman-raphael · 2019-10-03T17:16:18Z

tests/test_filepath.py

+    table.external[store].delete(delete_external_files=True)


 def test_filepath_class_again():


This test failed on WIN10 at test_filepath_class_again. Error details: datajoint.errors.DataJointError: A different version of 'one\two\three\attachment.dat' has already been placed

This test failed on WIN10 at test_filepath_class_s3. For example, the result of managed_file was dj/repo/one\two\three\attachment.dat.

This test failed on WIN10 at test_filepath_class_s3_again. For example, the result of managed_file was dj/repo/one\two\three\attachment.dat.

guzman-raphael · 2019-10-03T17:22:41Z

tests/test_filepath.py

+        managed_file.parent.mkdir(parents=True, exist_ok=True)
+        with managed_file.open('wb') as f:
+            f.write(contents)  # same in all files
        table.insert1((i, managed_file))


This test failed on WIN10. Error details: datajoint.errors.DuplicateError: ("Duplicate entry '1' for key 'PRIMARY'", 'To ignore duplicate entries in insert, set skip_duplicates=True')

this happens sometimes when the previous test run was terminated before cleanup

guzman-raphael · 2019-10-03T17:24:21Z

tests/test_filepath.py

-

 def test_filepath_cleanup_s3():
    """test deletion of filepath entries from external table """


This test failed on WIN10. For example, the result of managed_file was dj/repo/four\three\one\file.dat.

…nto attach

…ons.

guzman-raphael · 2019-10-03T21:18:40Z

.travis.yml

        - sudo pip install python-coveralls
-        - coveralls
+        - coveralls
+>>>>>>> bff58baac8da58b43be95ba1c58174cb7721411f


This line should be removed... Travis tests will no longer run on this PR until this is addressed.

Provide WIN10/Python3.7/MySQL 5.7 compatibility

Fix WIN10 filepath to store as posix and fetch as user's platform.

…able the filepath datatype

…t-python into attach

guzman-raphael

Awesome! Feel much better about this version. :)
Only have some minor suggestions for CHANGELOG.md then I will happily approve and merge.

CHANGELOG.md

Co-Authored-By: guzman-raphael <[email protected]>

dimitri-yatsenko added 6 commits September 13, 2019 06:10

attachments are saved as intact files

bcfb173

reform attachments and filepath

18366e6

complete implementation of external storage except for clean

11d7d9b

Merge branch 'dev' of https://github.com/datajoint/datajoint-python i…

22347ae

…nto attach

refactor external storage

7e7c183

complete refactoring of external storage for version 0.12

8aa1543

all tests pass

dimitri-yatsenko marked this pull request as ready for review September 20, 2019 16:07

dimitri-yatsenko added 3 commits September 20, 2019 13:39

rename attribute basename to attachment_name in external table

a2d6f9a

add __repr__ to ExternalMapping

2c11a65

external files are not copied if stage and store are the same

3a1c6ab

dimitri-yatsenko mentioned this pull request Sep 20, 2019

Issue 648 #657

Closed

dimitri-yatsenko added 4 commits September 20, 2019 18:38

make external tables require setting the delete_external_files argu…

478b36a

…ment

update CHANGELOG

bb1deab

fix Python 3.4 compatibility

e2636ed

fix Python 3.4 and 3.5 compatibility

11a1f93

dimitri-yatsenko added 3 commits September 23, 2019 10:49

dropped support for Python 3.4

165f795

Merge branch 'dev' of https://github.com/datajoint/datajoint-python i…

c18af74

…nto attach

minor changes in error messages

48147f1

guzman-raphael reviewed Oct 3, 2019

View reviewed changes

dimitri-yatsenko and others added 4 commits October 3, 2019 13:17

Merge branch 'dev' of https://github.com/datajoint/datajoint-python i…

2e26c4a

…nto attach

Update to pathlib in test init.

0c4fd1c

Update test_blob_migrate to be compatible for WIN10.

6ba86e0

Fix WIN10 compatibility with KeyboardInterrupt and SystemExit excepti…

43a5126

…ons.

guzman-raphael requested changes Oct 3, 2019

View reviewed changes

dimitri-yatsenko and others added 11 commits October 3, 2019 16:57

Merge pull request #4 from guzman-raphael/dimitri-attach

c7ca34c

Provide WIN10/Python3.7/MySQL 5.7 compatibility

Fix WIN10 filepath to store as posix and fetch as user's platform.

5c99e37

Fix relpath for Python3.5.

e2c3f23

Fix copytree for Python3.5.

93aeefc

Fix typo.

20719ae

Fix for Python3.5.

7ca0099

Update coveralls.

f3ffd63

Update coverall env vars.

bb1b40f

Merge pull request #5 from guzman-raphael/win-filepath

f05c50d

Fix WIN10 filepath to store as posix and fetch as user's platform.

add environment variable DJ_SUPPORT_FILEPATH_MANAGEMENT to enable/dis…

298efed

…able the filepath datatype

Merge branch 'attach' of https://github.com/dimitri-yatsenko/datajoin…

e609fbe

…t-python into attach

guzman-raphael reviewed Oct 7, 2019

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

dimitri-yatsenko and others added 2 commits October 7, 2019 22:03

Update CHANGELOG.md

6dda528

Co-Authored-By: guzman-raphael <[email protected]>

Update CHANGELOG.md

796dae6

Co-Authored-By: guzman-raphael <[email protected]>

guzman-raphael merged commit 92989d0 into datajoint:dev Oct 8, 2019

dimitri-yatsenko mentioned this pull request Oct 8, 2019

Incorrect path reference #650

Closed

ixcat mentioned this pull request Nov 13, 2019

Strip root from external S3 storage on insert #693

Merged

		ext.delete(delete_external_files=True)


		def test_filepath_s3():

		ext.upload_filepath(managed_file) # this is fine because the file is the same


		def test_duplicate_upload_s3():

		ext.upload_filepath(managed_file) # this should raise exception because the file has changed


		def test_duplicate_error_s3():

		table.external[store].delete(delete_external_files=True)


		def test_filepath_class_again():



		def test_filepath_cleanup_s3():
		"""test deletion of filepath entries from external table """

Store attachments as intact files #659

Store attachments as intact files #659

Uh oh!

Conversation

dimitri-yatsenko commented Sep 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Sep 20, 2019

Uh oh!

coveralls commented Sep 20, 2019

Uh oh!

coveralls commented Sep 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guzman-raphael commented Sep 23, 2019

Uh oh!

guzman-raphael commented Oct 3, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guzman-raphael left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guzman-raphael Oct 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guzman-raphael left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dimitri-yatsenko commented Sep 14, 2019 •

edited

Loading

coveralls commented Sep 20, 2019 •

edited

Loading

guzman-raphael Oct 3, 2019 •

edited

Loading