Attachments and configurable blobs #532

dimitri-yatsenko · 2018-12-03T05:24:37Z

Nans are interpreted as NULL for floating-point attributes at insert Interpret None as NULL in inserts #467
implement configurable subfolders in external storage subfolders for external storage #475
implement file attachment datatype File attachments #480
implement configurable blobs Configurable blobs #497
add dj.get_schema_names()

…into attachments

…bs and attachments

coveralls · 2018-12-03T05:27:38Z

Coverage decreased (-0.8%) to 90.025% when pulling 2c04d74 on dimitri-yatsenko:attachments into ebfbcd4 on datajoint:master.

coveralls · 2018-12-03T05:27:38Z

Coverage decreased (-0.8%) to 90.025% when pulling 2c04d74 on dimitri-yatsenko:attachments into ebfbcd4 on datajoint:master.

coveralls · 2018-12-03T05:27:38Z

Coverage decreased (-0.5%) to 90.34% when pulling afeadb1 on dimitri-yatsenko:attachments into 2d21899 on datajoint:dev.

…into attachments

…everal previous releases)

…into attachments

…ajoint-python into attachments Add the download_path attribute in fetch

Fixed 3.4 compatibility

…ajoint-python into attachments

…into attachments

austin-hilberg · 2019-02-07T19:42:42Z

datajoint/__init__.py


 __author__ = "Dimitri Yatsenko, Edgar Y. Walker, and Fabian Sinz at Baylor College of Medicine"
-__date__ = "Nov 15, 2018"
+__date__ = "January 14, 2018"


Do you mean 2019?

austin-hilberg · 2019-02-07T19:49:27Z

datajoint/declare.py

-        if store_name[0] != 'external':
-            raise DataJointError('External store types must be specified as "external" or "external-<name>"')
+        if store_name[0] not in ('external', 'blob', 'attach'):
+            raise DataJointError('Invalid attribute type in:\n%s' % line)


The previous error message gave a more explicit indication of what to do to fix the problem. I find the new one less informative.

austin-hilberg · 2019-02-07T20:04:18Z

datajoint/erd.py

-            warnings.warn('ERD functionality depends on matplotlib and pygraphviz. Please install both of these '
-                          'libraries to enable the ERD feature.')
+            warnings.warn('ERD functionality depends on matplotlib, networkx, and pygraphviz. '
+                          'Please install both of these libraries to enable the ERD feature.')


There are now three libraries mentioned.

austin-hilberg · 2019-02-07T20:38:20Z

tests/test_legacy_external_class.py

+
+
+def test_heading():
+    heading = modu.Simple().heading


You removed the () in line 8 of test_external_class.py above. Should you do so here as well?

This does not really matter. Accessing attributes of classes directly still triggers instantiation of an object, so a more efficient way is to instantiate. The omission of parentheses was provided for user convenience.

austin-hilberg · 2019-02-07T20:39:40Z

tests/test_legacy_external_class.py

+
+def test_insert_and_fetch():
+    original_list = [1, 3, 8]
+    modu.Simple().insert1(dict(simple=1, item=original_list))


Same as above; should the () be removed throughout this method?

This is inside legacy tests. We support both syntaxes.

austin-hilberg · 2019-02-07T20:40:24Z

tests/test_legacy_external_class.py

+
+
+def test_populate():
+    image = modu.Image()


Can we remove these () throughout this and the next two methods?

Yes, we could but, when using the same table multiple times, it's more efficient to instantiate once.

This particular snippet is inside test_legacy_external_class, which tests old functionality, so it's a good place to test whether the old syntax still works.

eywalker · 2019-02-08T04:09:24Z

datajoint/version.py

@@ -1 +1 @@
-__version__ = "0.11.1"
+__version__ = "0.12.0"


Please change this to "0.12.dev"

eywalker

Approve for Dev release

dimitri-yatsenko added 14 commits December 2, 2018 16:53

move dj.config into settings.py

5918b2c

Merge branch 'dev' into attachments

bc85f21

Merge branch 'master' of https://github.com/datajoint/datajoint-python …

92553b0

…into attachments

Merge branch 'dev' into attachments

c127693

Merge branch 'dev' into attachments

1b188da

Merge branch 'dev' into attachments

13573c5

add properties for heading attributes for supporting configurable blo…

f3dd5b3

…bs and attachments

rename property is_supported to unsupported for heading attributes

046291a

load configurable fields

1f5fe9d

implement declaration of configurable attributes: blob- and attach.

3350516

prepare for saving attachments

acc07fb

add attach.py for saving and loading attachments

c11bbbf

implement inserting attachments

390b0b7

implement fetch of attachments and configurable blobs

2c04d74

fix issue datajoint#467

43e9c76

dimitri-yatsenko requested a review from eywalker December 3, 2018 18:38

dimitri-yatsenko added 11 commits December 3, 2018 13:33

further cleanup of __init__.py

51336ff

Use DEFAULT instead of NULL when the insert value is None.

cee588e

slight refactor of Table.insert

c04f974

fix for error introduced in previous commit

9de4782

Merge branch 'master' of https://github.com/datajoint/datajoint-python …

0c35af1

…into attachments

Merge branch 'master' of https://github.com/datajoint/datajoint-python …

1588303

…into attachments

implement external file folding

f265674

Merge branch 'master' into attachments

bcebce5

remote the keys property from fetch (a warning was displayed in s…

50f17ce

…everal previous releases)

add dj.get_schema_names()

f141aa7

stylistic improvements

6310c7d

dimitri-yatsenko added 6 commits January 14, 2019 22:44

fix Python 3.4 compatibility

346f47f

fix Python 3.4 compatibility

b2087aa

Merge branch 'master' of https://github.com/datajoint/datajoint-python …

332cfd6

…into attachments

improve error message

0c491e2

improve error message

7e51e4f

Merge branch 'attachments' of https://github.com/dimitri-yatsenko/dat…

0434fc8

…ajoint-python into attachments Add the download_path attribute in fetch

dimitri-yatsenko added 4 commits January 16, 2019 12:47

bugfix in S3 store

484e926

bugfix in S3 store

1a83fe6

Merge branch 'attachments' of https://github.com/dimitri-yatsenko/dat…

4c8b6eb

…ajoint-python into attachments

Merge branch 'master' of https://github.com/datajoint/datajoint-python …

bf66d64

…into attachments

dimitri-yatsenko mentioned this pull request Jan 24, 2019

is datajoint data cached locally? int-brain-lab/IBL-pipeline#12

Closed

dimitri-yatsenko added 2 commits February 4, 2019 10:58

Merge branch 'master' into attachments

8f4e8f9

implement external storage cleanup with subfolding

0826d94

austin-hilberg reviewed Feb 7, 2019

View reviewed changes

fix error message and release date

fb14029

dimitri-yatsenko changed the base branch from master to dev February 7, 2019 21:40

improve warning messages

90cf697

eywalker suggested changes Feb 8, 2019

View reviewed changes

datajoint/version.py Outdated

@@ -1 +1 @@

__version__ = "0.11.1"

__version__ = "0.12.0"

Copy link

Contributor

eywalker Feb 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change this to "0.12.dev"

change version to 0.12.dev

afeadb1

eywalker approved these changes Feb 8, 2019

View reviewed changes

eywalker merged commit 26fc8e1 into datajoint:dev Feb 8, 2019

dimitri-yatsenko mentioned this pull request Mar 19, 2019

Interpret None as NULL in inserts #467

Closed

This was referenced Oct 8, 2019

File attachments #480

Closed

Configurable blobs #497

Closed

Attachments and configurable blobs #532

Attachments and configurable blobs #532

Uh oh!

Conversation

dimitri-yatsenko commented Dec 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Dec 3, 2018

Uh oh!

coveralls commented Dec 3, 2018

Uh oh!

coveralls commented Dec 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eywalker left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dimitri-yatsenko commented Dec 3, 2018 •

edited

Loading

coveralls commented Dec 3, 2018 •

edited

Loading