Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
67e41ef
Fixed #367
dimitri-yatsenko Oct 12, 2017
f89bf0e
Merge branch 'master' of https://github.com/datajoint/datajoint-python
dimitri-yatsenko Oct 12, 2017
159cb51
fixed dependencies
dimitri-yatsenko Oct 12, 2017
bb88117
implemented the Union operator
dimitri-yatsenko Oct 13, 2017
9703126
bugfix in populate
dimitri-yatsenko Oct 13, 2017
4fa1abd
fixed a bug in server-side insert with missing attributes
dimitri-yatsenko Oct 13, 2017
d6b9f2a
work on cascading delete
dimitri-yatsenko Oct 19, 2017
c35ed7a
Merge branch 'master' of https://github.com/dimitri-yatsenko/datajoin…
dimitri-yatsenko Oct 19, 2017
7299058
fixed #375 -- added the max_calls argument to populate
dimitri-yatsenko Oct 19, 2017
3baf487
simplified insert from select
dimitri-yatsenko Oct 20, 2017
4969c36
consolidating the use of hashes
dimitri-yatsenko Oct 20, 2017
ce774be
added module `external` for handling external storage
dimitri-yatsenko Oct 20, 2017
ad650b0
minor fix in external, work in progress
dimitri-yatsenko Oct 20, 2017
860d04b
minor
dimitri-yatsenko Oct 20, 2017
2144b0f
changed how external storage is configured
dimitri-yatsenko Oct 20, 2017
cfc1e21
minor bug
dimitri-yatsenko Oct 20, 2017
0195cfc
minor formatting
dimitri-yatsenko Oct 20, 2017
d821bcf
removed the JobManager class -- it was never used
dimitri-yatsenko Oct 20, 2017
183b324
bugfix from previous commit
dimitri-yatsenko Oct 20, 2017
6164eb5
renamed the display_progress argument in populate()
dimitri-yatsenko Oct 20, 2017
96f0d20
Merge branch 'master' of https://github.com/dimitri-yatsenko/datajoin…
dimitri-yatsenko Oct 20, 2017
b37175b
rolled back unintended changes in delete
dimitri-yatsenko Oct 20, 2017
9688e5d
fixed typo JobsTable -> JobTable
dimitri-yatsenko Oct 20, 2017
bcc0048
fixed bugs introduced in recent commits
dimitri-yatsenko Oct 20, 2017
28208c2
Merge branch 'master' of https://github.com/datajoint/datajoint-python
dimitri-yatsenko Oct 20, 2017
04b1829
fixed test_requirements
dimitri-yatsenko Oct 20, 2017
1c8e071
fixed indentation
dimitri-yatsenko Oct 20, 2017
bb11c1b
minor
dimitri-yatsenko Oct 20, 2017
9fc36b4
minor
dimitri-yatsenko Oct 20, 2017
9fca90f
minor code refactor in schema.py
dimitri-yatsenko Oct 22, 2017
7069dad
improved the error message in autopopulate
dimitri-yatsenko Oct 22, 2017
3bc28c5
correction to previous commit
dimitri-yatsenko Oct 22, 2017
a3d1a53
Merge branch 'master' of https://github.com/datajoint/datajoint-python
dimitri-yatsenko Oct 24, 2017
d9cca7b
Merge branch 'master' of https://github.com/dimitri-yatsenko/datajoin…
dimitri-yatsenko Oct 24, 2017
1b37ee9
typo
dimitri-yatsenko Oct 24, 2017
cf3cf0c
made `make` and acceptable name for the populate callback (issue #387)
dimitri-yatsenko Oct 25, 2017
85b6587
small bugfix for rare cases with multiple inheritance
dimitri-yatsenko Oct 27, 2017
f936073
minor fix
dimitri-yatsenko Oct 27, 2017
2b622c1
Merge branch 'master' of https://github.com/dimitri-yatsenko/datajoin…
dimitri-yatsenko Oct 27, 2017
5270d80
minor fixes
dimitri-yatsenko Oct 29, 2017
85502b0
Merge branch 'master' of https://github.com/dimitri-yatsenko/datajoin…
dimitri-yatsenko Oct 30, 2017
bdba20d
undid an unintended change in delete
dimitri-yatsenko Oct 30, 2017
919efba
implemented declaration of external fields
dimitri-yatsenko Oct 30, 2017
e30dfb2
added tests for external storage
dimitri-yatsenko Oct 30, 2017
259950d
minor cleanup
dimitri-yatsenko Oct 30, 2017
3a7f416
minor cleanup
dimitri-yatsenko Oct 30, 2017
3594c67
added external storage tests
dimitri-yatsenko Nov 5, 2017
7eb114c
Merge branch 'master' of https://github.com/datajoint/datajoint-python
dimitri-yatsenko Nov 5, 2017
4d1af79
Completed basic implementation of external storage.
dimitri-yatsenko Nov 5, 2017
4a58a9a
ERD does not show dependencies on external storage
dimitri-yatsenko Nov 5, 2017
1ca0b09
again, the ERD no longer includes references to ~external
dimitri-yatsenko Nov 5, 2017
add950f
fixed #328: the jobs table now records the error stack
dimitri-yatsenko Nov 5, 2017
90c021e
fixes for #328
dimitri-yatsenko Nov 5, 2017
782a9a5
fixed #388 -- a more elegant way to skip duplicates in insert
dimitri-yatsenko Nov 5, 2017
383595d
followup to previous commit
dimitri-yatsenko Nov 5, 2017
794dc47
made insert from query more consistent with insert from variables
dimitri-yatsenko Nov 5, 2017
5ab3381
fixed issue #381 -- better error messages for syntax errors in declar…
dimitri-yatsenko Nov 5, 2017
4b2671e
typo from previous commit
dimitri-yatsenko Nov 6, 2017
9a0b902
set the strict mode at connection time
dimitri-yatsenko Nov 6, 2017
7302571
set sql_mode in connection
dimitri-yatsenko Nov 6, 2017
569881a
updated the sql_mode
dimitri-yatsenko Nov 6, 2017
86c2480
added tests for union and for external storage. Other minor fixes bas…
dimitri-yatsenko Nov 13, 2017
6f7c6bd
improved documentation and error messages for fetch and fetch1. Fixe…
dimitri-yatsenko Nov 14, 2017
8018a9d
added tests for external storage
dimitri-yatsenko Nov 15, 2017
7627569
changed the shape of the computed nodes in the ERD to elipse to avoid…
dimitri-yatsenko Nov 15, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
made make and acceptable name for the populate callback (issue #387)
  • Loading branch information
dimitri-yatsenko committed Oct 25, 2017
commit cf3cf0ca3279f438c3d368b7740f7d9e1f9f85fc
18 changes: 11 additions & 7 deletions datajoint/autopopulate.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ class AutoPopulate:
"""
AutoPopulate is a mixin class that adds the method populate() to a Relation class.
Auto-populated relations must inherit from both Relation and AutoPopulate,
must define the property `key_source`, and must define the callback method _make_tuples.
must define the property `key_source`, and must define the callback method `make`.
"""
_key_source = None

@property
def key_source(self):
"""
:return: the relation whose primary key values are passed, sequentially, to the
`_make_tuples` method when populate() is called.The default value is the
``make`` method when populate() is called.The default value is the
join of the parent relations. Users may override to change the granularity
or the scope of populate() calls.
"""
Expand All @@ -42,13 +42,15 @@ def key_source(self):
self._key_source *= FreeRelation(self.connection, parents.pop(0)).proj()
return self._key_source

def _make_tuples(self, key):

def make(self, key):
"""
Derived classes must implement method _make_tuples that fetches data from tables that are
Derived classes must implement method `make` that fetches data from tables that are
above them in the dependency hierarchy, restricting by the given key, computes dependent
attributes, and inserts the new tuples into self.
"""
raise NotImplementedError('Subclasses of AutoPopulate must implement the method "_make_tuples"')
raise NotImplementedError('Subclasses of AutoPopulate must implement the method `make`')


@property
def target(self):
Expand All @@ -68,7 +70,7 @@ def _job_key(self, key):
def populate(self, *restrictions, suppress_errors=False, reserve_jobs=False,
order="original", limit=None, max_calls=None, display_progress=False):
"""
rel.populate() calls rel._make_tuples(key) for every primary key in self.key_source
rel.populate() calls rel.make(key) for every primary key in self.key_source
for which there is not already a tuple in rel.

:param restrictions: a list of restrictions each restrict (rel.key_source - target.proj())
Expand Down Expand Up @@ -120,6 +122,8 @@ def handler(signum, frame):
call_count = count()
logger.info('Found %d keys to populate' % len(keys))

make = self._make_tuples if hasattr(self, '_make_tuples') else self.make
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should throw deprecation warning when we find _make_tuples

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concluded that we are holding off deprecation of _make_tuples


for key in (tqdm(keys) if display_progress else keys):
if max_calls is not None and call_count >= max_calls:
break
Expand All @@ -133,7 +137,7 @@ def handler(signum, frame):
logger.info('Populating: ' + str(key))
next(call_count)
try:
self._make_tuples(dict(key))
make(dict(key))
except (KeyboardInterrupt, SystemExit, Exception) as error:
try:
self.connection.cancel_transaction()
Expand Down
2 changes: 1 addition & 1 deletion datajoint/version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.8.2"
__version__ = "0.9.0"