3.95b1: Fails to create .nc1 for large group (62+ million headers)

Found a bug in NewsLeecher? Post a bug report here. Please describe how to reproduce the bug.
• Be sure to read the bug reporting guidelines before posting.
Post Reply
mho
Posts: 6
Joined: Sat Mar 05, 2005 3:18 am

3.95b1: Fails to create .nc1 for large group (62+ million headers)

Post by mho »

Hi,

When downloading headers for a group with 62.7 million headers, 3.95b1 fails to create .nc1 file.

Steps taken:

* Purge group to eliminate possibility of currupt cache.
* Select group and start update.
* Wait.................................
* Exit newsleecher.

When exiting, newsleecher fiickers an error box (I suspect it is an "Out of memory" one but it goes away before it's possible to read...) and exits without creating an .nc1 file.

Newsleecher was using less that 400MB memory on a 4GB machine (max virtual memory around 10GB), so it's not likely to be reaching an OS (hard) limit - I don't know if there are any soft limits in windows.

If I dial back on number of days to keep headers so it will only have to save about 55 million headers, it manages to create .nc1 and all seems well on next start. (Resulting .nc1 size 224MB, .nc2 is 2.9GB)

OS: XP sp3
NSP: giganews

- mho

User avatar
Quaraxkad
Posts: 310
Joined: Tue Aug 29, 2006 4:34 am

Post by Quaraxkad »

I've been having this problem also. I see a lot of threads about it and most go without even a single reply from anyone, and none of them have had any answers or solutions.

I've done a little testing and found out a few things. NewsLeecher is able to download and save alt.binaries.hdtv, which has about 100 million articles. But since each posted file in that group uses more articles than normal, it only comes out to a little over 300,000 posts. On the other hand, the files in alt.binaries.multimedia are smaller, with about 1.5 million posts but only a little over 60 million articles. This group downloads completely, then returns an "out of memory" error when unloading the group from memory, presumably when trying to create the .nc1 file. As the original poster stated, the .nc2 file is created, but the .nc1 is not. After the error message, reloading the group shows only the articles that were there already before attemping to update. The number of articles in a group seems to be irrelevant to this problem, probably because of the way NewsLeecher saves the header cache (as individual files/posts instead of articles?).

I have 4GB of memory (only about 3.5GB usable in 32bit XP Pro). I have tried raising my virtual memory to another 4GB. I have tried with and without Giganews Accelerator. I have tried deleting the cache files for this group and then restarting NewsLeecher. I have enabled "Extreme debug logging", which resulted in a 24GB nl_log.txt file. This log file showed NO entries about the bug or any indication that saving the group cache to disk had failed.

This is a pretty crippling problem and I can't imagine why it's going without even a single response from the developer. A lot of us pay for Giganews because we want/need the high retention, so limiting the number of headers downloaded is not an option. If we can't get that from NewsLeecher, it becomes a pretty useless peice of software. Many years ago I used GrabIt and switched to NewsLeecher because it had a similar problem (and it STILL does).

If this problem can't be addressed within NewsLeecher, can an external tool be developed to fix the cache files for these problem groups? Can an nc1 file be "re-built" using the data that's correctly saved in the nc2?

sngbrdb
Posts: 8
Joined: Fri Mar 16, 2007 7:27 am

the silence of the developers is amazing

Post by sngbrdb »

Yes, this is HUGE. Why haven't the developers responded?? I'm in the same boat as you, Newsleecher is USELESS until this is resolved.

I'm considering asking for a full refund if there is not at least a response to this somewhere.

thenormal
Posts: 42
Joined: Thu Nov 25, 2004 12:39 am

Re: the silence of the developers is amazing

Post by thenormal »

sngbrdb wrote:Yes, this is HUGE. Why haven't the developers responded?? I'm in the same boat as you, Newsleecher is USELESS until this is resolved.

I'm considering asking for a full refund if there is not at least a response to this somewhere.
same problem here. very annoying. also, i can't update sounds.mp3.complete_cd (which is not THAT huge). every time i get that bug and every time the articles are not saved, hence i have to start downloading all the headers again. So far, i haven't been able to save new headers since the last update. real real bad....

Post Reply