-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Eventhubs new EventProcessor #6550
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
e62825f
Shared connection (sync) draft
84ae397
Shared connection (sync) draft 2
6e6c827
Shared connection (sync) test update
a86146e
Shared connection
c5a23c8
Fix an issue
b83264d
add retry exponential delay and timeout to exception handling
4f17781
put module method before class def
f0d98d1
fixed Client.get_properties error
ed7d414
new eph (draft)
ee228b4
new eph (draft2)
1d65719
remove in memory partition manager
4895385
EventProcessor draft 3
6415ac2
small format change
a685f87
Fix logging
6388f4a
Add EventProcessor example
1cd6275
Merge branch 'eventhubs_preview2' into eventhubs_eph
6394f19
use decorator to implement retry logic and update some tests (#6544)
yunhaoling 56fdd1e
Update livetest (#6547)
yunhaoling 10f0be6
Remove legacy code and update livetest (#6549)
yunhaoling fbb66bd
make sync longrunning multi-threaded
00ff723
small changes on async long running test
0f5180c
reset retry_count for iterator
6e0c238
Don't return early when open a ReceiveClient or SendClient
0efc95f
type annotation change
90fbafb
Update kwargs and remove unused import
yunhaoling e06bad8
Misc changes from EventProcessor PR review
dd1d7ae
raise asyncio.CancelledError out instead of supressing it.
9d18dd9
Merge branch 'eventhubs_dev' into eventhubs_eph
a31ee67
Update livetest and small fixed (#6594)
yunhaoling 19a5539
Merge branch 'eventhubs_dev' into eventhubs_eph
997dacf
Fix feedback from PR (1)
d688090
Revert "Merge branch 'eventhubs_dev' into eventhubs_eph"
2399dcb
Fix feedback from PR (2)
5ad0255
Update code according to the review (#6623)
yunhaoling 5679065
Fix feedback from PR (3)
35245a1
Merge branch 'eventhubs_dev' into eventhubs_eph
83d0ec2
small bug fixing
a2195ba
Remove old EPH
f8acc8b
Update decorator implementation (#6642)
yunhaoling 6fe4533
Remove old EPH pytest
598245d
Revert "Revert "Merge branch 'eventhubs_dev' into eventhubs_eph""
97dfce5
Update sample codes and docstring (#6643)
yunhaoling 64c5c7d
Check tablename to prevent sql injection
13014dd
PR review update
3e82e90
Removed old EPH stuffs.
0f670d8
Small fix (#6650)
yunhaoling 2a8b34c
Merge branch 'eventhubs_dev' into eventhubs_eph
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
EventProcessor draft 3
- Loading branch information
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
19 changes: 0 additions & 19 deletions
19
sdk/eventhub/azure-eventhubs/azure/eph/checkpoint_manager.py
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
127 changes: 0 additions & 127 deletions
127
sdk/eventhub/azure-eventhubs/azure/eph/event_processor.py
This file was deleted.
Oops, something went wrong.
32 changes: 0 additions & 32 deletions
32
sdk/eventhub/azure-eventhubs/azure/eph/partition_processor.py
This file was deleted.
Oops, something went wrong.
5 changes: 5 additions & 0 deletions
5
...hub/azure-eventhubs/azure/eph/__init__.py → ...azure/eventhub/eventprocessor/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
31 changes: 31 additions & 0 deletions
31
sdk/eventhub/azure-eventhubs/azure/eventhub/eventprocessor/checkpoint_manager.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # -------------------------------------------------------------------------------------------- | ||
| # Copyright (c) Microsoft Corporation. All rights reserved. | ||
| # Licensed under the MIT License. See License.txt in the project root for license information. | ||
| # ----------------------------------------------------------------------------------- | ||
|
|
||
|
|
||
| from .partition_manager import PartitionManager | ||
|
|
||
|
|
||
| class CheckpointManager(object): | ||
| """Every PartitionProcessor has a CheckpointManager to save the partition's checkpoint. | ||
|
|
||
| """ | ||
| def __init__(self, partition_id, eventhub_name, consumer_group_name, instance_id, partition_manager: PartitionManager): | ||
| self._partition_id = partition_id | ||
| self._eventhub_name = eventhub_name | ||
| self._consumer_group_name = consumer_group_name | ||
| self._instance_id = instance_id | ||
| self._partition_manager = partition_manager | ||
|
|
||
| async def update_checkpoint(self, | ||
| offset, sequence_number): | ||
| """Users call this method in PartitionProcessor.process_events() to save checkpoints | ||
|
|
||
| :param offset: offset of the processed EventData | ||
| :param sequence_number: sequence_number of the processed EventData | ||
| :return: None | ||
| """ | ||
| await self._partition_manager.\ | ||
| update_checkpoint(self._eventhub_name, self._consumer_group_name, self._partition_id, self._instance_id, | ||
| offset, sequence_number) | ||
13 changes: 13 additions & 0 deletions
13
sdk/eventhub/azure-eventhubs/azure/eventhub/eventprocessor/close_reason.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # -------------------------------------------------------------------------------------------- | ||
YijunXieMS marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # Copyright (c) Microsoft Corporation. All rights reserved. | ||
| # Licensed under the MIT License. See License.txt in the project root for license information. | ||
| # ----------------------------------------------------------------------------------- | ||
|
|
||
| from enum import Enum | ||
|
|
||
|
|
||
| class CloseReason(Enum): | ||
| SHUTDOWN = 0 # user call EventProcessor.stop() | ||
| LEASE_LOST = 1 # lose the ownership of a partition. | ||
| EVENTHUB_EXCEPTION = 2 # Exception happens during receiving events | ||
| USER_EXCEPTION = 3 # user's code in EventProcessor.process_events() raises an exception | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.