Performance: Reducing HDD usage. Cache fileparts in memory before assembling.

If you got suggestions for new features or feature changes in NewsLeecher, go ahead and let santa know.
Post Reply

Implement filepart-download and assemble in memory?

Yes
25
93%
No need. I am happy with the HDD performance with R&E.
2
7%
 
Total votes: 27

wolf5
Posts: 16
Joined: Mon Apr 09, 2007 11:15 am

Performance: Reducing HDD usage. Cache fileparts in memory before assembling.

Post by wolf5 »

Before R&E was introduced, there was no real problem with HDD performance and NL.

Now that R&E has become - in my eyes - a very important part of NL, the structure on how things are downloaded and assembled should be taken a look at.

Today:
If I understand correctly, NL downloads each filepart and stores it to disk. Then when all parts have been stored to disk, it reads them and assembles them into the final file, and cleaning up the temp parts afterwards.

What I wish for:
Add an option for NL to "eat" a user defined amount of memory to temp store and assemble files from in memory.
Default this option will be off, but if a user has a lot of mem available, he can ie say that 100Mb is to be used to assemble files. Use the mem as FIFO storage. Whatever gets pushed out at the bottom is stored to disk.
Or if a files total size is larger than the memory available, just store to disk as default.

What you optimally get: Just 1 write to disk per file. Today we have write, read, write and then delete operations for one file.

When the fileparts are in memory, R&E can do its verifying part as well. Reducing disk usage even more. Verifying is a LARGE HDD performance crusher when NL is downloading at the same time.

And if the user sets extracting to happen to another disk the HDD vil be as performing as it was before R&E was introduced.

One write per file(every x seconds as the file is being assembled in mem). One read per file for extract. Since the target is another disk, all the better performance.

Did this make sense?

pzam
Posts: 179
Joined: Fri Jul 18, 2008 8:34 pm

Post by pzam »

I am useing this structure now it seems to work a little better.

/user/newsleecherRnE/newsleecherdowload/




put the rar incoming in subdir under extract so it moves them up 1 when done


so far it seems to work ok. only tested a few days
also set the cleanup time to more often and auto if your computer can handle it.

Some computers will slow down if the memory controler/hard drive controler or ICU chips get to warm on come boards these are far from the fans and will get hot under lots of i/o. It can be worth getting a little fan and putting in near them but not touching them as heat will dry oil from fan bearings.

madshi
Posts: 4
Joined: Mon Aug 11, 2008 7:41 am

Post by madshi »

I generally support this wish - but with a twist!

Some binaries have part sizes of 100MB and more. And NL sometimes works on two parts at the same time. So that would mean a RAM consumption increase of 200MB. Currently NL consumes just 25MB. That increase in RAM consumption is not acceptable to me.

I suggest a compromise:

Instead of storing each part on harddisk and then as an additional step assembling the final file, NL could create the final file from the get go: NL should just reserve the final file in its correct size (with "undefined" content) and then with each read article update the file contents. This approach would allow NL to save the final file assembling and part deletion steps without any increase in system resources (such as RAM).

wolf5
Posts: 16
Joined: Mon Apr 09, 2007 11:15 am

Post by wolf5 »

Yup. That is a good idea.

Even if not all the parts are available, the par fix would still find good blocks from whatever got stored.

But still. I got newsleecher running on my homeserver. Lots of free mem not being used. So why not let newsleecher leech all of my mem and make it perform better on my hdd.

madshi
Posts: 4
Joined: Mon Aug 11, 2008 7:41 am

Post by madshi »

wolf5 wrote:But still. I got newsleecher running on my homeserver. Lots of free mem not being used. So why not let newsleecher leech all of my mem and make it perform better on my hdd.
Ok, but it would have to be an option then. While my suggestion could be realized as default behaviour cause it doesn't (seem to) have any disadvantages.

Tig
Posts: 31
Joined: Tue Feb 13, 2007 12:08 pm

Post by Tig »

You could always create a ram drive and set that as your temporary working drive in advanced->folders

flanders
Posts: 36
Joined: Mon Feb 27, 2006 9:54 pm
Location: France

Post by flanders »

The RAM disk will not work as the files will not be dumped to the local HD when the RAM disk is full.

Wolf I see your point, but there is one thing happening in between, which is that files are verified after being downloaded to the disk.
I like the idea of using the RAM as a cache, although a computer reboot will loose all the data (how often does it happen...). Nevetheless it would be a good thing to download and verify files in the RAM instead of hogging the HD with downloads, verification and extraction processes.

BlueSteel
Posts: 17
Joined: Thu Jan 15, 2009 3:52 pm

Post by BlueSteel »

I support this idea, since I ran into HDD performance problems after upgrading my internet speed recently.

Downloading with 12 bots at around 1MB/s each currently causes a high amount of HDD thrashing which collides with RnE operations on the same drive. I like to keep all NL operations on a separate drive, so it does not slow down my work on the main HDD, but RnE + downloading is currently clashing. The download queue is often forcibly paused to let HDD operations keep up.

An adjustable memory buffer would be a very nice addition to NL, since it should stop NL from pausing the downloads all the time while RnE is active. I have plenty of RAM leftover, so a 100-500MB buffer would be no real problem for me. As long as it is optional/adjustable, everyone should be happy.

mickblu
Posts: 15
Joined: Mon Nov 08, 2004 7:23 am

Post by mickblu »

I support the assembly in RAM as I too have been experiencing major slowdowns lately. As many modern machines have a surplus (?) of memory (I have 4GB with plans to upgrade to 8GB -- hey, RAM's cheap these days) on my Quadcore 3GHz machine, I would like either a "use all the PC can spare mode" or perhaps a "Specify how much RAM" mode with an option for "All you can eat" :-0) feature.

I have been a longtime user of NL and love how it's evolving. Although I (probably like many others) don't use some of the new features I'm sure there are some folks out there who are thrilled at those additional functions.

On an aside, I am appalled at the number of downloads that are infected with Trojans or other Malware. It seems the newsgroups have become targets for these people who have nothing better to do than try and harm other's PCs. What I don't understand is what possible pleasure they can get from these actions because they can't see the results of their efforts. Grrr....

Mystical666
Posts: 4
Joined: Thu Aug 21, 2008 11:50 am

Post by Mystical666 »

Personally i think this would be a great addition, personally i use 4 GB of Ram on my downloading machine and would happily donate 1 GB or more to reduce the strain on the HDD, any word yet on if this could or will be added or considered?

teh_l33ch

Post by teh_l33ch »

I use a ramdisk of 256MB to store the temp files of Newsleecher.
After all the parts of a file are downloaded. The file gets stored on the real harddisk.

Ramdisk plus 9.0 can restore the content of a ramdisk after a system restart.

pzam
Posts: 179
Joined: Fri Jul 18, 2008 8:34 pm

Post by pzam »

if you hard drive is getting pounded a lot check in the settings and make sure that this is unchecked under download settings.

move small articles to top of que.




I found that makes newsleecher repair and extract start trying to process every file you have dowloading all at once.

if you uncheck it. then it will just download and process 1 file at a time and you drives will run lots cooler and quieter.

flanders
Posts: 36
Joined: Mon Feb 27, 2006 9:54 pm
Location: France

Post by flanders »

pzam wrote:move small articles to top of que.
or "PAR2 Quick Download" function...

Post Reply