Skip to content

feat(backup): Add restore indices and restore backup tasks #2779

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shirshanka merged 7 commits into from
Jun 30, 2021
Merged

feat(backup): Add restore indices and restore backup tasks #2779

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a1254c3
Added restore indices and restore backup tasks
Jun 28, 2021
465a2e2
Merge remote-tracking branch 'origin/master' into dl--add-backup-upgrade
Jun 29, 2021
537f07d
Fix to John's comments
Jun 29, 2021
8e70c47
Fix checkstyle
Jun 29, 2021
9cb3a87
Throw exception
Jun 29, 2021
c4d3f78
Removed'
Jun 29, 2021
3e8131e
Fix
Jun 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Throw exception
  • Loading branch information
Dexter Lee committed Jun 29, 2021
commit 9cb3a87464ea8d9f8782abdab9dacf6963c91051
5 changes: 3 additions & 2 deletions ...e/src/main/java/com/linkedin/datahub/upgrade/restorebackup/backupreader/BackupReader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package com.linkedin.datahub.upgrade.restorebackup.backupreader;

import com.linkedin.datahub.upgrade.UpgradeContext;
import java.util.Optional;
import javax.annotation.Nonnull;


/**
Expand All @@ -11,5 +11,6 @@
public interface BackupReader {
String getName();

Optional<EbeanAspectBackupIterator> getBackupIterator(UpgradeContext context);
@Nonnull
EbeanAspectBackupIterator getBackupIterator(UpgradeContext context);
Copy link
Collaborator

@jjoyce0510 jjoyce0510 Jun 29, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish this interface didn't have to know about UpgradeContext... does it?

Copy link
Contributor Author

@dexter-mh-lee dexter-mh-lee Jun 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was mainly because of the logging. If we can just log.info, then we don't need to pass in the context.

Copy link
Collaborator

@jjoyce0510 jjoyce0510 Jun 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forgot about this -- we should be able to just use the logger as well

}
12 changes: 7 additions & 5 deletions ...main/java/com/linkedin/datahub/upgrade/restorebackup/backupreader/LocalParquetReader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import com.linkedin.datahub.upgrade.UpgradeContext;
import java.io.IOException;
import java.util.Optional;
import javax.annotation.Nonnull;
import lombok.extern.slf4j.Slf4j;
import org.apache.avro.generic.GenericRecord;
import org.apache.hadoop.fs.Path;
Expand All @@ -28,20 +29,21 @@ public String getName() {
return "LOCAL_PARQUET";
}

@Nonnull
@Override
public Optional<EbeanAspectBackupIterator> getBackupIterator(UpgradeContext context) {
public EbeanAspectBackupIterator getBackupIterator(UpgradeContext context) {
Optional<String> path = context.parsedArgs().get("BACKUP_FILE_PATH");
if (!path.isPresent()) {
context.report().addLine("BACKUP_FILE_PATH must be set to run RestoreBackup through local parquet file");
return Optional.empty();
throw new IllegalArgumentException(
"BACKUP_FILE_PATH must be set to run RestoreBackup through local parquet file");
}

try {
ParquetReader<GenericRecord> reader = AvroParquetReader.<GenericRecord>builder(new Path(path.get())).build();
return Optional.of(new ParquetEbeanAspectBackupIterator(reader));
return new ParquetEbeanAspectBackupIterator(reader);
} catch (IOException e) {
context.report().addLine(String.format("Failed to build ParquetReader: %s", e));
return Optional.empty();
throw new RuntimeException(String.format("Failed to build ParquetReader: %s", e));
}
}
}