Beta 14: Adding a lot of files to download queue

Found a bug in NewsLeecher? Post a bug report here. Please describe how to reproduce the bug.
• Be sure to read the bug reporting guidelines before posting.
Post Reply
User avatar
Core_F
Posts: 156
Joined: Sun Nov 16, 2003 6:51 pm
Location: Germany
Contact:

Beta 14: Adding a lot of files to download queue

Post by Core_F » Wed Mar 19, 2008 9:58 am

Hey there,

just another little thing. It would be great, if Newsleecher would still be able to show some life, if I add quite a lot of files.
I just added about 300k files to download queue and Newsleecher is not responding. Guess I have to wait a bit till it does something again.
··· Logic Is The Faith Of The Future ···

Sasquatch
Posts: 1319
Joined: Sat Jun 03, 2006 10:41 am
Location: Netherlands

Post by Sasquatch » Sat Mar 22, 2008 11:50 pm

When adding files, it needs to do several things. First add them to the queue, select the folder where to save it to, create a backup of the queue every minute (you might add them just on that minute), look for the groups the files are located and several other checks. Remember that if you select a lot of files in the explorer, that hangs too. If you open 100 small text files, the whole system would hang. This is nothing but normal behaviour. 300.000 files are a lot. Would you be able to browse a book of 300.000 pages in a few seconds (giving you can hold it)?
Fill in your bugs so NL will get better.

User avatar
Core_F
Posts: 156
Joined: Sun Nov 16, 2003 6:51 pm
Location: Germany
Contact:

Post by Core_F » Mon Mar 24, 2008 2:42 pm

Oh I don't have a problem that it takes a while... I have a problem with the application "not responding" while the files are added to the queue. A little "adding files... please wait" window would be enough.
Or even better would be, if it doesn't stop till all files are added, it could start downloading already, after the first file has been added...
··· Logic Is The Faith Of The Future ···

Sasquatch
Posts: 1319
Joined: Sat Jun 03, 2006 10:41 am
Location: Netherlands

Post by Sasquatch » Tue Mar 25, 2008 12:15 am

NL was never designed for adding tons of files at the same time, so no 'please wait' is programmed. That the application is not responding is something that comes with a program that's very busy. You should be glad your not on Vista, or Vista would kick the program and kill the task. Or are you on Vista and is this the reason you post this? Keep in mind that when a program is working on one task, it can't simply start a second one at the same time. Just like you can't do the dishes and the laundry at the same time.
Fill in your bugs so NL will get better.

User avatar
Core_F
Posts: 156
Joined: Sun Nov 16, 2003 6:51 pm
Location: Germany
Contact:

Post by Core_F » Wed Apr 02, 2008 8:00 am

I am not on Vista.

And I know, why it is not responding. But that is just not a thing a modern application should do. To say it simple: You can run the "adding files" method in an extra thread, this way the main application can still run and show i.e. information about the status of the adding process.

Btw I just tested a bit. Newsleecher added like 2000 files in less than a second. But now I have added like 5000 files and it won't respond anymore for a few minutes. So I guess there is another problem, too.
··· Logic Is The Faith Of The Future ···

gigabitshmigabit
Posts: 35
Joined: Thu Apr 20, 2006 2:06 am

Post by gigabitshmigabit » Mon May 12, 2008 3:24 am

NL freezes (cursor hourglass) for me anytime I add 50-100 files or more - for approx 20secs. Running on a Q6600 with 8GB Ram and program+download directory are on a raid 0 array with 4 drives so this is surprising. I download .nzb files to a directory and have NL set to monitor this directory & import file downloads to the queue. Not sure if this method has something to do with it..
"hmm....hard drives....im going to need more hard drives...."
---
yep.im.teh.one.who.beat.u.up.and.stole.ur.l33t.top.site.axx.now.ur.gf.takes.my.cox.in.teh.ar.se

tvgeloof
Posts: 19
Joined: Sun May 11, 2008 9:07 am

Basicly this problem makes NL incompatible with Vista

Post by tvgeloof » Mon May 12, 2008 10:55 am

I frequently add 5000 to 10.000 files from the article tab to the queue in one go. (Ctrl-A Ctrl-D)
I find the blocking extremely annoying as well. :-?:
Thankfully I run NL on XP,not Vista. :):
However: Vista is here to stay and Vista will sometimes kill applications that do this.
Basicly: Newsleecher currently is NOT Vista compatible because of this. :cry:

I can understand the NL is very busy at this point so it takes a while, but what I can't understand is that adding to the queue doesn't scale in a linear way.
About the only thing I can think of is that the new items are INSERTED in the queue at their proper position, in stead of being ADDED at the end with a sort after wards.
Inserts scale exponentially with larger queues, adds scale linear.

Making this multithreading is possibly quite complicated in the current codebase so it might not be feasible to make that change now. I would strongly suggest it must be considered for the upcoming 4.0 version.

On the other hand: NL already contains some "Please Wait..." dialogs. (Like when saving the program state or catching up with the diskcache). It is probably not so difficult to re-use this code to display a progress indicator at this point. (Maybe only show if # of articles to be added > 1000 or so.)
Be nice..... We were all newbies once.......

User avatar
Core_F
Posts: 156
Joined: Sun Nov 16, 2003 6:51 pm
Location: Germany
Contact:

Post by Core_F » Thu Dec 25, 2008 8:00 pm

I have to give this a little push.

3.95 Beta 3, just want to add about 88k files... and it freezes without responding again.
··· Logic Is The Faith Of The Future ···

User avatar
Bilar Crais
Posts: 45
Joined: Tue Jan 25, 2005 4:46 pm

Post by Bilar Crais » Sun Jan 18, 2009 1:27 am

Ditto on this one.

It's driving me crazy. Download a couple hundred thousand headers, and a few thousand posts to the queue, and then it just freezes. Another thing that makes it freeze is when some other app on the computer is taking a good deal of CPU time. The only thing to do is start Newsleecher over again and download the headers all over again.

On a somewhat hopeful note, I just installed Version 3.5 final, and it seems immune to this problem.

Edit: For that matter, I just installed version 3.8 final and it can handle literally anything I can throw at it...hundreds of thousands of posts...doesn't even hiccup.

azz0r
Posts: 27
Joined: Sat Sep 18, 2004 1:25 pm

Post by azz0r » Wed Jan 28, 2009 6:46 pm

I to would love some dialogue box, I can't stand how it freezes and vista won't let me minimise it until its done adding them.

RiZK
Posts: 7
Joined: Mon Feb 16, 2009 10:47 am

Post by RiZK » Mon Feb 16, 2009 11:02 am

It would possibly help if a user could manually adjust the cash newleecher uses....

It's not uncommon for a user to add 5000 files from a certain newsgroup.

Mine doesn't crash but has slight hick-ups, it's as if the download stutters....

Anyone know how this can be fixed?

Post Reply