Skip to content

[stable9.1] Backport of filecache repair step OLD #29407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tomneedham wants to merge 15 commits into from
Closed

[stable9.1] Backport of filecache repair step OLD #29407

tomneedham wants to merge 15 commits into from

Conversation

@tomneedham
Copy link
Contributor

@tomneedham tomneedham commented Nov 1, 2017 •
edited
Loading

Description

Backport of #28253 to stable9.1. Attempts to repair oc_filecache and retain metadata whilst doing so.

Motivation and Context

Underlying filecache corruption caused by a bug in 9.

How Has This Been Tested?

Manually using occ scan commands.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@tomneedham tomneedham added this to the development milestone Nov 1, 2017
@tomneedham tomneedham self-assigned this Nov 1, 2017
@tomneedham tomneedham requested a review from PVince81 November 1, 2017 11:54
@tomneedham
Copy link
Contributor Author

tomneedham commented Nov 1, 2017

Backported OC\Migration\ConsoleOutput so that the repair step does not require modification

PVince81
PVince81 approved these changes Nov 6, 2017
View reviewed changes
Copy link
Contributor

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks good 👍

If manually tested, this is good to go.

@PVince81
Copy link
Contributor

PVince81 commented Nov 6, 2017

Weird JS setup failures:

21:36:32 /var/lib/jenkins/workspace/owncloud-core_core_PR-29407-NELWZQ3VUO4EKPETQA55Q6XOF2ONH2ZG24LGJMF4PI3EMLL57OCQ/build/lib/node_modules/karma/node_modules/socket.io/lib/store.js:35
21:36:32 Store.prototype.__proto__ = EventEmitter.prototype;
21:36:32                                          ^
21:36:32 
21:36:32 TypeError: Cannot read property 'prototype' of undefined
21:36:32     at Object.<anonymous> (/var/lib/jenkins/workspace/owncloud-core_core_PR-29407-NELWZQ3VUO4EKPETQA55Q6XOF2ONH2ZG24LGJMF4PI3EMLL57OCQ/build/lib/node_modules/karma/node_modules/socket.io/lib/store.js:35:42)

This was referenced Nov 6, 2017
Closed
Closed
@PVince81
Copy link
Contributor

PVince81 commented Nov 6, 2017

tried doing a separate PR to reset the test env but the failure is the same: #29474.

maybe we need the older node JS for these versions of the node packages...

@PVince81
Copy link
Contributor

PVince81 commented Nov 6, 2017

I have node v6.11.4 and tests pass locally

@PVince81
Copy link
Contributor

PVince81 commented Nov 6, 2017

Requires #29476 for the JS tests to pass. Might need to resubmit PR to get a clean env.

@PVince81
Copy link
Contributor

PVince81 commented Nov 7, 2017

Please rebase and track Jenkins directly. If JS test fails within the first minutes, then resubmit PR.

Vincent Petry and others added 15 commits November 7, 2017 10:04
Add repair step to repair mismatch filecache paths
ebaa762 
Whenever a parent-child relationship describes a specific path but the
entry's actual path is different, this new repair step will adjust it.

In the event where another entry already exists under that name, the
latter is deleted and replaced by the above entry. This should help
preserve the metadata associated with the original entry.
Repair detached filecache entries through occ files:scan
0980191
Use DI and add more tests for storage filter
f79837d
Run phase 2 of repair on gathered affected storages from phase 1
d6688a2 
When fixing failed cross-storage moves in the file cache and using the
storage id filter, we filter by target storage for phase 1. However, we
also need to fix the source storages in phase 2. To do so, a list of
affected source storages is now gathered in phase 1 to be run on phase
2.
occ files:scan --all --repair repairs all storages at once
de8295c 
This instead of iterating over all storages which is way less efficient
due to the 1-N nature of potential failed cross-storage moves that we
are repairing.

If singleuser mode is enabled and "--all --repair" is passed, all
storages will be repaired in bulk (no repair filter). If not, it will
fall back to iterating over each storage which is slower.
@tomneedham
Improve output when repairing. Fix non-repair file:scan
ad2c23c
@tomneedham
Add parallel non-corrupt storages to filecache repair step to check f…
1ac999e 
…or isolation
@tomneedham
Test without parallel storages
25edcf7
@tomneedham
Fix lastinsertid for postgres wehn reusing fileid
1a16772
@tomneedham
Try oracle tests without throwing exception for stray transactions
38d8ce0
Oracle hates empty strings
3f3340e
Workaround for Oracle trigger for fileid
238728d
@tomneedham
Cleanup testcase and comments
bf3e4bf
@tomneedham
Modify backported repair step for 9.1
6f5f26a
@tomneedham
Modify backported repair step for 9.1
e81442e
@PVince81
Copy link
Contributor

PVince81 commented Nov 7, 2017

@tomneedham I went forward and rebased this. Let's see...

@PVince81
Copy link
Contributor

PVince81 commented Nov 7, 2017

Jenkins crashed again on the JS test. Going to resent this PR to get a clean env.

@PVince81 PVince81 closed this Nov 7, 2017
@PVince81 PVince81 changed the title [stable9.1] Backport of filecache repair step [stable9.1] Backport of filecache repair step OLD Nov 7, 2017
@PVince81 PVince81 mentioned this pull request Nov 7, 2017
Closed
@PVince81
Copy link
Contributor

PVince81 commented Nov 7, 2017 •
edited
Loading

Resent here #29486 #29487

@PVince81 PVince81 deleted the stable9.1-repair branch November 7, 2017 09:09
@lock
Copy link

lock bot commented Aug 2, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Aug 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@PVince81 PVince81 PVince81 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@tomneedham tomneedham

Labels

3 - To Review status/STALE

Projects

None yet

Milestone

10.0.5

Development

Successfully merging this pull request may close these issues.

4 participants

@tomneedham @PVince81 @ownclouders