Updating headers in very large groups fails

Found a bug in NewsLeecher? Post a bug report here. Please describe how to reproduce the bug.
• Be sure to read the bug reporting guidelines before posting.
Post Reply
sandstones
Posts: 15
Joined: Wed Jun 07, 2006 6:01 pm

Updating headers in very large groups fails

Post by sandstones »

This has been happening for over 6 months. Updating very large groups such as a.b.boneless runs to completion, but when saving the group, NL fails. Seems like it started about when a.b.boneless got over 100M headers (for 90 days)

In 3.9 final NL would report a crash when saving the group. In 3.95 beta 3, there is no error, but the progress bar stops midway through the save, and then all the new headers just downloaded for the group are not saved.

I've tried removing the newsgroup and starting over; the first update seems to go well, but any subsequent update has the problem again.

There is plenty of free disk space (>250GB) and free memory (>2GB) so I don't think that is the issue. Its not hard to duplicate - just takes a while; download all headers in a.b.boneless from a server with 90+ days retention, and when it finishes (in a few days) try to update. Just can't get those new headers.

SynthaxError
Posts: 5
Joined: Wed Oct 13, 2004 4:48 pm

Post by SynthaxError »

I have the same problem :cry:

User avatar
Quaraxkad
Posts: 310
Joined: Tue Aug 29, 2006 4:34 am

Post by Quaraxkad »

This is a way too common problem that, as far as I know, has still not been so much as commented on by the developer. I did a lot of testing and found that it's not the number of articles in a group that matters, it's the number of posted files (no, they are not the same thing). More explanation here: viewtopic.php?p=84555&highlight=#84555

I have since switched my OS to XP x64, and so far, the problem has not resursfaced. I assume this is because I have all 4GB of my memory available instead of the ~3.5GB I was limited to in 32-bit XP.

Matthias Vance
Posts: 2
Joined: Mon Jan 19, 2009 9:49 am

Post by Matthias Vance »

I am having the same problem, and I didn't even knew why it was failing because I disabled any popup dialog boxes. Let's hope this will FINALLY be fixed in a NL4 beta..

Regards,

Matthias

sngbrdb
Posts: 8
Joined: Fri Mar 16, 2007 7:27 am

Process space in 32-bit

Post by sngbrdb »

Nice to know that it 64-bit will fix this... unfortunately, 64-bit isn't an option for me just yet :cry:

This is a huge problem for me; newsleecher is pretty much useless until it is fixed.

fyi, the problem on 32-bit isn't that you don't have all 4 Gigs of RAM (you do, but your video card, etc., has to use some of the address space). On a 32-bit system, processes only get 2 Gigs of virtual memory addresses (so the problem is actually worse).

What's odd to me is that

(a) a 32-bit process running on a 64-bit OS still can only address 2 Gigs of memory (it has to be compiled for 64-bit to be able to use more), and

(b) Newsleecher is only using ~700 megs of memory when I get this bug.

If I knew more about the problem than a bunch of letters, I could speculate more about the cause, but whatever the cause, the developer team needs to address this... it's a huge issue

User avatar
Spiril
Site Admin
Posts: 4278
Joined: Fri Nov 07, 2003 3:11 am

Post by Spiril »

Pls try the NewsLeecher v4 preview and see if that helps:
viewtopic.php?t=18067

It handles the larger groups much better than the NewsLeecher v3.9x series.
bug fixed. no idea how. hate it when that happens. trying to break it again now. will. not. be. defeated.

User avatar
Quaraxkad
Posts: 310
Joined: Tue Aug 29, 2006 4:34 am

Re: Process space in 32-bit

Post by Quaraxkad »

sngbrdb wrote:Nice to know that it 64-bit will fix this... unfortunately, 64-bit isn't an option for me just yet :cry:

fyi, the problem on 32-bit isn't that you don't have all 4 Gigs of RAM (you do, but your video card, etc., has to use some of the address space). On a 32-bit system, processes only get 2 Gigs of virtual memory addresses (so the problem is actually worse).
I am aware of the reasons behind the memory issues in 32bit vs 64bit, but the reason doesn't change that newsleecher isn't able to use it :wink: . The end result is that running processes are limited to around 3.5GB. Still, the ammount of RAM is not likely to be the problem anyway.

Currently, I only have 2GB of RAM installed. Newsleecher was working perfectly in XP x64. I had other un-related issues with x64 that forced me to to go back to 32-bit Windows two days ago. I just installed Newsleecher with the same 2GB of RAM and I'm getting the Out of Memory errors again.

Spiril wrote: Pls try the NewsLeecher v4 preview and see if that helps:
viewtopic.php?t=18067

It handles the larger groups much better than the NewsLeecher v3.9x series.
Based on what I'm reading in the preview thread it sounds like it's still very buggy and I wouldn't want to use it daily just yet. Does anybody have at least some idea what's causing this problem yet? If it's still going to be a little while before V4 is stable enough for daily use, I'd like to see a new version of 3.9 to fix this. You must know by now that a lot of people have this problem. Had I not learned that using XP x64 solved this problem for me I would have already gave up on NewsLeecher until v4.

Post Reply