View data in pages

[dufferzafar]

For huge files (GBs) can we add the functionality to view data in "pages"?

So when we launch gtabview, only first 1000 or so lines are read and on pressing Ctrl + N
the next 1000 lines are read.

This is a quick fix to allow dealing with huge files without chugging up all memory.

[wavexx]

On Sat, Oct 21 2017, Shadab Zafar wrote: For huge files (GBs) can we add the functionality to view data in "pages"? So when we launch gtabview, only first 1000 or so lines are read and on pressing Ctrl + N the next 1000 lines are read.

It works already like that for compatible data sources. Any blaze URI will load data on-demand.

[dufferzafar]

So, how do I load data from a local CSV file? Example? And how does the on demand feature handle dynamic loading? Keypress or something else?

…

On 22-Oct-2017 1:55 AM, "wavexx" ***@***.***> wrote: On Sat, Oct 21 2017, Shadab Zafar wrote: > For huge files (GBs) can we add the functionality to view data in "pages"? > > So when we launch gtabview, only first 1000 or so lines are read and on pressing Ctrl + N > the next 1000 lines are read. It works already like that for compatible data sources. Any blaze URI will load data on-demand. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYeKJEr40F5Xy5tOweByoilr0TRgIaTks5sulNSgaJpZM4QBrHW> .

[wavexx]

On Sat, Oct 21 2017, Shadab Zafar wrote: So, how do I load data from a local CSV file? Example?

Normally 'csv://file' will trigger the blaze CSV loader (any URI will pass through blaze). Although I just tried now with Blaze from the master branch, it seems that the blaze API has changed, breaking the current code. I'll have a look at that.

And how does the on demand feature handle dynamic loading? Keypress or something else?

Blaze has an internal chunking mechanism which depends on the data source. For CSV it should be chunks of 100mb. gtabview will otherwise loads instantly. Internally it lazy-loads chunks of 16k in an LRU cache. This is triggered according to the current view position automatically. However, CSV is *not* a good format for quick viewing. Last time I checked, the Blaze CSV backend was slow (slower than the plain CSV parser for files <100mb).

[wavexx]

So now that Blaze seems dead, this issue becomes relevant again.

I'm probably going to throw in a segmented CSV loader, however I'd love to find a true successor to "blaze" that provides this features for us instead.