1.1 BILLION headers to retrieve

Found a bug in NewsLeecher? Post a bug report here. Please describe how to reproduce the bug.
• Be sure to read the bug reporting guidelines before posting.
Post Reply
Hallie Miles
Posts: 10
Joined: Sun Jan 28, 2007 3:36 pm

1.1 BILLION headers to retrieve

Post by Hallie Miles »

I'm running NewsLeecher v5.0 b15 on Windows 7 Pro 64bit.

I've had problems before of NL losing track of a database and needing to redownload 120,000,000 headers. At around 4 hours. But 15 minutes ago I clicked on a Virtual group and one of the included groups now says

"1,133,566,621 turbo compressed headers".

At 45 KB/s.

For 190:39:00.


That's right at 8 days, running 24 hours a day.

Any idea how to fix this? Needless to say, I'm not waiting around for 8 days to download over a billion headers...

User avatar
Smudge
Site Admin
Posts: 10034
Joined: Tue Aug 17, 2004 1:42 am

Post by Smudge »

Do you really need a billion headers? I doubt you are going to browse thru them looking for stuff to download.

If you just want some, right click and choose to get the last 100k or 1 million. That should catch you up on the group with some backlog for you to search thru. Next time you update the headers on the group, it will only grab the new headers and not the billion old headers.
Please be aware of and use the following pages...
Services Status Page : SuperSearch and Usenet Access server status, retention, server load, indexing time, etc.
Support Request Form : Use this if you have a problem with billing or account status. The forum is only for NewsLeecher application issues.

daleidat
Posts: 88
Joined: Tue Jul 27, 2004 5:03 pm
Location: Kent, UK

Download in groups

Post by daleidat »

If you goto a group with a lot of headers, dowload them at a 1000000 or 2000000 at a time, because at lot of the headers in the 1 billion headers or so, will only be spam, which at the end of the day, will only have been a waste of your time. Or the easiest alternative, is to use Supersearch to find what you are looking for.

User avatar
thany
Posts: 503
Joined: Sun Sep 26, 2004 6:03 pm
Location: Netherlands
Contact:

Post by thany »

Turn OFF compressed headers. It's a little counter-intuitive, but compressed headers are actually slower.

@daleidat
Supersearch cannot be used to "see whats out", so to speak. You have to know what to look for. Headering is for those who don't know what to look for, i.e. browsing around.
Running Newsleecher 7.0 on Windows 10 x64, because 4.0 stopped working

User avatar
Bags
Posts: 397
Joined: Thu Jul 14, 2005 11:53 pm
Location: Outer Space

Post by Bags »

Yeah ,

Stick to 1mill or so .. Supersearch?

Billions is ludicrous amount!

User avatar
thany
Posts: 503
Joined: Sun Sep 26, 2004 6:03 pm
Location: Netherlands
Contact:

Post by thany »

I'm actually having this problem now. Newsleecher, or the server, or something somewhere in a magical place, has decided that every so often, one of my groups needs to be fed up with 1.9 BILLION headers (or milliard for those outside US/UK ;))

It is not me deciding to grab all these headers. I just to an update, expecting only the latest headers, but then it starts to download many times more than I expected.

There must be a solution for this.

Only happens on one server, not on another. So it might be server-related, but don't forgat that the application makes the request for a certain number of headers. So who knows maybe NL is doing something wrong (and has been since the latest maintenance release, which is never).
Running Newsleecher 7.0 on Windows 10 x64, because 4.0 stopped working

User avatar
Bags
Posts: 397
Joined: Thu Jul 14, 2005 11:53 pm
Location: Outer Space

Post by Bags »

lol

This is like downloading the whole of the WWW : )

Wonder if server databases are being corrupted by DMCA take-downs?!

User avatar
Smudge
Site Admin
Posts: 10034
Joined: Tue Aug 17, 2004 1:42 am

Post by Smudge »

We would need to see the log entries when this happens. It will show the header pointer NewsLeecher has from the last update and what the news server responds with.

Also remember that the pointers are based on the news server entry and not the group. This means that if you have multiple news servers configured in NewsLeecher, each one will have its own tracking pointer even if it is for the same group. If you switch off one server and enable another, it will be requesting a different range of headers.
Please be aware of and use the following pages...
Services Status Page : SuperSearch and Usenet Access server status, retention, server load, indexing time, etc.
Support Request Form : Use this if you have a problem with billing or account status. The forum is only for NewsLeecher application issues.

Post Reply