[Follow up] Improve performance when rendering sample waveforms

[khoidauminh]

This PR adds additional changes on top of #7366:

• Attempts to resolve the garbage rendering issue when viewing large samples.

Details

In the old code, `m_paintPixmap` in `SampleClipView` is allocated to the size of the whole clip, and painted at (0, 0) no matter what. On large samples, when the width exceeds 32768 (QPixmap's resolution limit), garbage data will be shown.

In this PR, m_paintPixmap is allocated to the paintEvent's size (but with the clip's height), and painted onto the clip at the paintEvent's X coordinate. To make sure the m_paintPixmap doesn't get drawn on the wrong coordinate (due to something overlaying the clip and changing the paintEvent region), we paint on the coordinate obtained from the last full redraw.

This change allows some fields of VisualizeParameters to be omitted (drawRect).

• Fixes lagging in Automation Editor by specifying the viewportRect.

• Fixes crashes:

  • Crash upon opening Audio File Player's GUI #7723
  • Crash upon AudioFileProcessor generating a thumbnail of the waveform (commit 786088b) #7789

• Simplifies and enhances the implementation.

Should resolve the sample clips part of #3378

[sakertooth]

I also noticed there's some unfinished business I have with the draw and generation code possibly. I want to follow up with that too in this PR if that's okay with you.

[khoidauminh]

Ye sure @sakertooth

Btw there are some changes regarding the VisualizeParameters so check that out first

[bratpeki]

Glad to see this!

@khoidauminh, could you load any samples with a low pitch (kicks, toms, basses, etc) and zoom in on the waveform? I tried this in OFP and it generates a fill on the part of the waveform below 0!

Must've missed checking short samples in my testing, as I was so focused on long ones! 🤣

[khoidauminh]

@bratpeki I think I know what the issue is. Originally the Peak struct min and max default values are +infinity and -infinity respectively

But in a commit it was changed to +infinity and 0 (::max() and ::min() in source code), so max gets stuck at 0 when the waveform sits below 0, making a fill

Simply change the min() back to -max() and it's all good (currently away from laptop so can't do yet)

[sakertooth]

@bratpeki I think I know what the issue is. Originally the Peak struct min and max default values are +infinity and -infinity respectively

But in a commit it was changed to +infinity and 0 (::max() and ::min() in source code), so max gets stuck at 0 when the waveform sits below 0, making a fill

Simply change the min() back to -max() and it's all good (currently away from laptop so can't do yet)

std::numeric_limits<float>::min() gives back 0? It shouldn't.

[sakertooth]

We might want to use std::numeric_limits<float>::infinity() instead @khoidauminh, it should work better than ::max and ::min.

[bratpeki]

After that fix has been applied, is this ready for testing? I view it as an extension of 7366, so it only makes sense to take a look at it as well.

[sakertooth]

Hi @bratpeki and @khoidauminh, I pushed a change that uses std::numeric_limits<float>::infinity() and -std::numeric_limits<float>::infinity() for the initial minimum and maximum values for Peak's respectively (another one incoming to simplify type conversions in the draw code). I'm considering to make other changes such as referencing the original sample data instead of unnecessarily copying all the sample data to a new Thumbnail, as well as figuring out how to parallelize the generation, which both should speed up generation time.

If you need my assistance with anything though let me know.

Feel free to test the new commits @bratpeki.
Also, thank you @khoidauminh for simplifying out one of those three rectangles in VisualizeParameters 👍

[khoidauminh]

Is it just me or the MaxSampleThumbnailCacheSize being 32 might be a little too small? Will LMMS keep regenerating thumbnails over and over again if there are more than 32 samples, especially when they're big?

[bratpeki]

Feel free to test the new commits @bratpeki.

Alright!

[sakertooth]

Is it just me or the MaxSampleThumbnailCacheSize being 32 might be a little too small? Will LMMS keep regenerating thumbnails over and over again if there are more than 32 samples, especially when they're big?

You also have to consider that caches are generally small to avoid keeping unnecessary, unwanted data, keeping it "hot" basically. Too big of a size and you run the risk of increasing the memory usage, even when in some cases evicting a thumbnail to store a new fresh one is better than keeping the old one around. That being said, I'm more than happy to help come up with a better size that will better fit the average workload and system.

If we have more than 32 thumbnails, there are two things that can happen if another thumbnail is requested to be generated: one is a cache miss if the thumbnail is not cached, the other a cache hit if it is. The cache will only keep regenerating new thumbnails if every request for a new thumbnail after meeting that size limit wasn't cached.

Sidebar: We might also want to consider renaming ThumbnailCache to Thumbnails or remove the alias altogether to avoid confusion.

[sakertooth]

Ideally, this idea will be extended to all resources, like sample resources and thumbnail resources, and maybe more (essentially anything that is costly to regenerate and store a lot of all at once is a resource).

[khoidauminh]

@sakertooth Looking at the code, from my understanding, when the cache gets full, the unused thumbnails will be destroyed, but the least still used thumbnails will be detached instead. Existing clips will hold on to this detached cache until all of them are destroyed. So I'm guessing it's not as bad as I'm concerned. All good then

[sakertooth]

@sakertooth Looking at the code, from my understanding, when the cache gets full, the unused thumbnails will be destroyed, but the least still used thumbnails will be detached instead. Existing clips will hold on to this detached cache until all of them are destroyed. So I'm guessing it's not as bad as I'm concerned. All good then

I guess it should be using an LRU eviction policy actually. I wont stop you from making changes, but there are plans to overhaul resource management in the future anyways as mentioned. Maybe since its really needed I might start work on it soon.

[bratpeki]

Before I test, I'd like to know what to test for!

Attempts to resolve the garbage rendering issue when viewing large samples.

Could you elaborate on this? Are you reffering to the black space I was showing in my screenshots 7636?

[khoidauminh]

@bratpeki Yes, on my Linux build it shows up as garbage like in #3378. If you're testing on Windows that's probably why it shows up as black. (There's a collapsed detail section in the desc on why it happens, if you haven't checked it out)

[bratpeki]

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

• Any sample, no matter the length or zoom level, now displays properly
• The part of the waveform between -1.0 and 0.0 isn't filled anymore

[khoidauminh]

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

* Any sample, no matter the length or zoom level, now displays properly

* The part of the waveform between `-1.0` and `0.0` isn't filled anymore

@bratpeki Yes, they have been tested. I have tried the following: zooming in, scrolling, overlaying another clip on the sample and zooming, overlaying a window and zooming, moving a sample (with and without an overlay). They should be fixed. But do test it anyway in order to catch more potential hidden bugs.

There's a behavior when the viewportRect doesn't contain the start of the clip (when it's outside of view or overlayed), the border will draw anyway and the sample filename will move around it until the clip start appears again.

Screencast_20250214_071652.mp4

In the view you'll also see that the sample gets shifted by the border size. I will need to do something about it.

EDIT: Ok I have no idea what happened but when I reopened lmms to work on the issue, it's gone (?) lol what Oh no it's still there, but with a trigger condition

[sakertooth]

Hello, a user was facing an issue with a "blurry" waveform. I believe this is the antialiasing I added to make the waveform look smoother.

In this follow up, we might consider either removing it, or making it an option you can toggle.

I did add the antialiasing on a whim I suppose. I thought it looked better, especially when zooming in the AFP, but we might make it a toggle. How do you all feel about the antialiasing?

[khoidauminh]

How do you all feel about the antialiasing?

I personally don't have a problem with it. But since the thumbnail we select to render is always larger than the clip size, I'm sure we won't run into a "blocky" appearance issue when we turn it off. So it's safe to remove it

[bratpeki]

I, too, am pretty indifferent, at least for now. I'd have to load in a lot of samples at different zoom levels to check which works best, so in the meantime it might be best to make it a toggleable option.

[bratpeki]

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

* Any sample, no matter the length or zoom level, now displays properly

* The part of the waveform between `-1.0` and `0.0` isn't filled anymore

@bratpeki Yes, they have been tested. I have tried the following: zooming in, scrolling, overlaying another clip on the sample and zooming, overlaying a window and zooming, moving a sample (with and without an overlay). They should be fixed. But do test it anyway in order to catch more potential hidden bugs.

There's a behavior when the viewportRect doesn't contain the start of the clip (when it's outside of view or overlayed), the border will draw anyway and the sample filename will move around it until the clip start appears again.

Screencast_20250214_071652.mp4

In the view you'll also see that the sample gets shifted by the border size. I will need to do something about it.

EDIT: Ok I have no idea what happened but when I reopened lmms to work on the issue, it's gone (?) lol what Oh no it's still there, but with a trigger condition

Cool, then, I'll test the things I listed!

As for the clip overlap issue, I don't know, that's too Qt-technical for me to think about.

(Side note, but) I don't even think you can push a clip of your choosing to the front.

If it can be solved so the text is always at the start of the sample, independent of clips that overlap with it, that's great! If not, we can name that as an issue for a future PR to fix in the final report of this PR.

[sakertooth]

Hi @khoidauminh, I was able to reproduce the bug where LMMS would crash in the AUR builds. The code was reading past the end of finerThumbnail after scaling i by finerThumbnailScaleFactor. This is undefined behavior, so it makes sense why it seemed like a very strange bug. I was only able to reproduce it by copying the exact compilation flags used by makepkg on my system onto the local CMake build.

I fixed this by clamping beginIndex and endIndex to the range [0, finerThumbnail->width() - 1].

[bratpeki]

Sorry for not being that active on this, my backlog has become much more extensive. I found something interesting in this PR, and there's a similar behavior in my very recent nightly. @sakertooth and @khoidauminh, could you try and recreate this bug:

2025-03-24.23-56-01.mp4

The height of the sample track doesn't matter, it happens all across the board.

The steps, if it helps to list them, are:

1. Load a sample into a sample track
2. Hover over it with a piano roll
3. Maximize and then close the piano roll
4. Open it back up, un-maximize it, and move it
5. Part of the sample track doesn't draw until you click the song editor

[bratpeki]

Drawing seems to be perfectly fine! I'll take a few days to try and recreate the automation editor bug and let you know on if I succeed. I'll also, OFC, try to find some more bugs. If all is well, I'd still think this is a good PR that should be reviewed and merged, and we can open a bug in the tracker to handle that one erroneous problem. @sakertooth thoughts?

[sakertooth]

If all is well, I'd still think this is a good PR that should be reviewed and merged, and we can open a bug in the tracker to handle that one erroneous problem. @sakertooth thoughts?

I'm hoping it won't come to that. Lets maybe give it a few more days as well as input from @khoidauminh. I'll also try to review this again and see what the problem is.

[khoidauminh]

Seems like the bug is an artifact from partial rendering. When the focus moves from the song editor to another window, it triggers a repaint in both windows. Because the new window is overlaying the clip, the viewportRect will be different, and so the clip will only render the visible regions, moving the overlay away exposes the garbage area.

This issue also happens if the overlay is on the left, this shifts the X position of the viewportRect.

I just pushed my attempt to resolve it:

Trackview's width is more consistent for most of the time, bounded by the Song Editor's window size and won't be disrupted by overlays.

I made viewPortRect's width twice the trackview's width and shifted the whole thing one half to the left, so that if the overlay is on the left and cause the X position of the paint region to shift, it will not enter the visible region.

We do lose a bit of performance because this makes the code render more than what is visible, but it should be negligible

[bratpeki]

2025-03-25.10-30-51.mp4

Another probably "out-of-PR-scope" thing, since this is improving performance of waveforms, and not fixing waveform-related bugs.

Should I open an issue for this?

Note to self: This also happens when you drag the editor below the visible space of LMMS and then back up.

[khoidauminh]

Since we're already dealing with the garbage rendering in this PR, we should try to squash as many related bugs before merging

Hopefully the commit will fix it

[0xf0xx0]

fixes it on my machine

[bratpeki]

Daily-driving this PR today. If I spot no issues, let's do a final review and merge this, it's been open for too long! 🤣

[bratpeki]

I can't recreate the bug mentioned in #7695 (comment) at all! @khoidauminh, did you use some stock samples? I could try those and try to get the conditions to be the exact same as in the video.

[khoidauminh]

@bratpeki I also can't reproduce this either. I didn't use stock samples, but I'm not sure if the sample would be related. One variable I think could be causing the bug is

lmms/src/gui/editors/AutomationEditor.cpp

Line 1211 in f6eafdc

int startPos = xCoordOfTick(0);

Replacing its value with 0 makes the sample stop scrolling

[bratpeki]

I'm all for merging this and investigating if that bug is spotted again.

[bratpeki]

@khoidauminh PR related?

2025-04-04.12-36-08.mp4

[khoidauminh]

@bratpeki Are you testing on the latest commits of this PR? I'm seeing that you're using master with the new mute/solo Icons

[bratpeki]

Totally right, my bad! 😭

[bratpeki]

@sakertooth has this been code-reviewed?

[sakertooth]

@sakertooth has this been code-reviewed?

Yeah, I've looked at it. The changes seem fine. Other potential improvements came to mind (like saving peak data to disk instead of holding all of it in memory, then streaming it in as needed), but this is fine for now. @khoidauminh what do you think?

[khoidauminh]

@sakertooth Everything looks fine by now. Although, will the sample streaming PR affect the the thumbnails in any way?

Btw, I've been wondering if we can store the peak data as signed 16bit integers instead of float. It could cut down the thumbnail memory usage by a half. I've also thought of going signed 8bits even (potentially saving 75% of mem), but the pixelation would show up on the automation editor quickly

I have a few more ideas on the saving peak data to disk as well (where to save them, compression(?), persistent across reboots?, etc), but those are for another PR if we agree to go further.

[sakertooth]

Everything looks fine by now. Although, will the sample streaming PR affect the the thumbnails in any way?

For now, sample streaming in that PR is entirely implemented in its own PlayHandle and is for audio file playback. Streaming peak data will need to use a different approach (probably using something like std::ifstream should suffice).

[bratpeki]

Can we merge? 🙏

[khoidauminh]

Can't find any more bugs for now, so let's merge this

[sakertooth]

Doing one last review before merge.