Skip to content

Conversation

@DerDreschner
Copy link
Collaborator

@DerDreschner DerDreschner commented Nov 25, 2024

To weight the usage of IPs in the data model, the user id is being added multiple times to the dataset. The exact number is being determined here:

$multiplier = (int)log((int)$addr->getSeen(), 2);

The problem is that this isn't always being taken into account when generating the training data. In case the dataset contains no IP with exactly one user id in the array, the array is malformed by containing only user ids (16 columns in size) in this variable:

$shuffledNegatives = $this->negativeSampleGenerator->generateShuffledFromPositiveSamples($collectedData->getTrainingPositives(), $numShuffledNegative);

The reason is the code in the getUniqueIPsPerUser function. It returns only IPs with exactly one user id in the second dimension, while not checking if the user id was already added to the IP during the previous data transformation.

The Rubix ML library then throws an InvalidArgumentException due to the unequal number of columns when trying to merge the malformed array with the other datasets here:

$allSamples = $collectedData->getTrainingPositives()->merge($randomNegatives)->merge($shuffledNegatives);

This PR will fix the issue and adds an unit test to prevent it during future refactoring steps.

Closes #860
Closes #933

@github-actions
Copy link

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

@solonovamax
Copy link

Is there anything major blocking this from being merged?

I've tested out these changes for myself, and it seems to fix #860.

Copy link
Member

@ChristophWurst ChristophWurst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thanks for the descriptive PR and the test coverage 🙏

@ChristophWurst ChristophWurst force-pushed the fix/unique-datasets-only branch from 9690f4f to 0b71fb1 Compare December 16, 2024 06:42
@ChristophWurst ChristophWurst enabled auto-merge (squash) December 16, 2024 06:42
@ChristophWurst ChristophWurst added bug Something isn't working 4. to release labels Dec 16, 2024
@DerDreschner
Copy link
Collaborator Author

@ChristophWurst : You're welcome! I'm looking forward to the release of my changes.

@ChristophWurst
Copy link
Member

Dear @DerDreschner, apologies for the delay. It seems CI hasn't run so my attempt for automerge stalled. Let's try a branch update.

@ChristophWurst ChristophWurst force-pushed the fix/unique-datasets-only branch from 0b71fb1 to d4c332d Compare January 7, 2025 07:44
@ChristophWurst ChristophWurst merged commit 995e370 into nextcloud:master Jan 7, 2025
29 checks passed
@joshtrichards
Copy link
Member

/backport to stable30

@backportbot
Copy link

backportbot bot commented Jan 7, 2025

The backport to stable30 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable30
git pull origin stable30

# Create the new backport branch
git checkout -b backport/958/stable30

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick d1e2b79f cbe0fad3 2d894667 d4c332d3

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/958/stable30

Error: Failed to cherry pick commits: error: no cherry-pick or revert in progress
fatal: cherry-pick failed


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

@joshtrichards
Copy link
Member

/backport to stable29

@backportbot
Copy link

backportbot bot commented Jan 7, 2025

The backport to stable29 failed. Please do this backport manually.

# Switch to the target branch and update it
git checkout stable29
git pull origin stable29

# Create the new backport branch
git checkout -b backport/958/stable29

# Cherry pick the change from the commit sha1 of the change against the default branch
# This might cause conflicts, resolve them
git cherry-pick d1e2b79f cbe0fad3 2d894667 d4c332d3

# Push the cherry pick commit to the remote repository and open a pull request
git push origin backport/958/stable29

Error: Failed to cherry pick commits: error: no cherry-pick or revert in progress
fatal: cherry-pick failed


Learn more about backports at https://docs.nextcloud.com/server/stable/go.php?to=developer-backports.

@DerDreschner
Copy link
Collaborator Author

@ChristophWurst / @joshtrichards : Looks something got very wrong. Backporting didn't work at all and I can't see my changes in the stable30 nor stable29 branch / 30.0.5rc1 tag.

@ChristophWurst
Copy link
Member

Apologies for the delay, @DerDreschner. The bot failed because of the squashed merge. It probably tried to find the original commit hashes.
You can find the backport at #969. Please give it a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

InvalidArgumentException daily during IPv6 training Datasets must have the same number of columns

4 participants