How to implement JS infine scroll - asynchronous

Question

I would like to display a table which have several thousand rows with complex formatting (color, font, border, etc. done on the ASP.Net Core server).
Initially, I generated an html copy of all the data (stored in a SQL Server database), but realised it wasn't optimal since the generated html data accounted for more than 50MB.
No, I only generate about 200 rows; 100 visible and 50 hidden above and below (cache). I would like to freely scroll the tablen, but when there are only 25 hidden rows above or below, fetch new rows from the controller which are then prepend or append to the table. Basically, I want to give enough room so I can can populate the table when I'm scrolling through the "hidden" (cache) rows.
Everything seems to work well, but I believe I need to use a web-worker to run the function in a background thread which add new rows to the table independently of the table being scrolled.
Below is a excerpt of the code :
I use a debunce function to only catch the lastest position of the mouse scroll.
The scroll function basically only checks whether there are enough hidden rows (cache) above or below the table. If it reaches the threshold, it either prepends (scroll upwards) or appends (scroll downwards) rows obtained from the controller.
The main issue is that I can't scroll the table when the new rows are being fetch as the page freezes. It only takes about 1 to 2 seconds to populate to new (scrollable) rows but it isn't smooth.
Could anyone help me improve the code? (general ideas) I also read that there are already existing libraries but can't really get my head around them..
$('#fields-table > tbody').on('wheel', _.debounce(async function (event) {
await scroll(); // Probably change it to a web-worker or promise?
}
async function scroll() {
var threshold = 200; // Corresponds to approximatively 50 rows (above and below).
var above = $('#fields-table').scrollTop();
var below = $('#fields-table > tbody').height() - $('#fields-table').height() - above;
// Gets the scroll delta based on the table heights.
var delta = 0
if (above < threshold) delta = above - threshold; // Scrolls upwards.
if (below < threshold) delta = threshold - below; // Scrolls downwards.
await addCacheRows(delta); // Prepends (delta < 0) or appends (delta > 0) or appends rows obtained via the fetch API.
}

Answer 1

Your problem is unlikely to resolve with a web worker. Without seeing more code I cannot tell for sure, but I suspect your code to generate new rows is not sufficiently efficient. Remember:
Use DocumentFragment to create the HTML, do not immediately append it to the main DOM tree row by row. Appending elements to a document triggers some recalculations.
Unless this is a LOT of data or requires lots of work serverside, you can immediately start preloading next/previous rows. Keep the promise object and only await it once you need them, that's the simplest way to go around it
Use passive scroll event listener - Firefox even shows a console warning whenever you do not do that
There is no way generating 200 rows of table data should take seconds. Since you use JQuery anyway (really, in 2022?), note that there are plugins for this. I don't remember what I used, but it worked perfect and scrolled smooth with much more data than what you have.

Answer 2

Thank you for your help. I realise it won't be as straightforward as I initally thought (I made some tests with WPF virtualization as well).
Regarding the time it takes to generate the extra rows, I believe it mostly comes from the server. Sure, I can probably load new rows independently of the threshold.
I've never heard about DocumentFragment, but that something I should definitely consider.