header compression?

Forum to report beta release bugs and discuss the latest beta releases with other users.
• If reporting a beta release bug, be sure read the bug reporting guidelines first.
Forum rules
IMPORTANT : Be sure to read the NewsLeecher forums policy before posting.
Post Reply
tommo123
Posts: 123
Joined: Wed Oct 19, 2005 4:43 pm

header compression?

Post by tommo123 »

any chance of that being implemented at some point? right now i'm downloading 44 million headers and it's taking a while :o

doppler
Posts: 152
Joined: Fri Apr 01, 2005 4:19 pm

Re: header compression?

Post by doppler »

tommo123 wrote:any chance of that being implemented at some point? right now i'm downloading 44 million headers and it's taking a while :o
It has to occur at both ends. It's also one of the things being talked
about. Don't hold your breath. This change will take some deep thought.

tommo123
Posts: 123
Joined: Wed Oct 19, 2005 4:43 pm

Post by tommo123 »

i'm with astraweb and afaik they do support it.

a program like Giganews accelerator would be useful on astrawebs side but at this point, i think it's a feature that really should be supported in news clients as it makes a HUGE difference for people that download headers

bbarker
Posts: 43
Joined: Fri May 29, 2009 6:18 am

Post by bbarker »

I'd like to add a vote for Astraweb's header compression.

User avatar
the doctor
Posts: 250
Joined: Wed Apr 22, 2009 12:40 am
Location: DeLand, Florida USA
Contact:

Server has to support and your side needs to be setup also

Post by the doctor »

Newsleacher already supports header compression - does your newsgroup server support it?

Server has to support and your computer also needs to be setup (running a compression program or service). Newshosting.com uses Stunnel to implement header compression. I am running Stunnel as a service.

Just downloaded 4 million headers in only 11 min vs. uncompressed download time of 2hrs.

I have 2 servers setup - 1 is my provider newsgroup server (uncompressed) and the second is newshosting (using header compression)

tommo123
Posts: 123
Joined: Wed Oct 19, 2005 4:43 pm

Post by tommo123 »

ummm, what you're saying doc is that it **DOESNT** support it.

you're running 3rd party software like giganews accelerator or something. that's not newsleecher. compute?

newsleecher needs to add support for XZVER then it'll support header compression. until then, it doesn't. simple plain basic fact.

i want my client to support this, not to have to install some random 3rd party software. we should all be able to simply click an option for header compression and as long as the server supports it, bingo. tis done.

User avatar
the doctor
Posts: 250
Joined: Wed Apr 22, 2009 12:40 am
Location: DeLand, Florida USA
Contact:

Post by the doctor »

I hope we can keep this as a friendly discussion...

I was just pointing out that you can use header compression with Newsleecher - it is not built into NL but NL does support (work with) header compression.

I use Newsleecher and along with the header compression my newsgroup service provides - I am able to download headers at a significantly decreased amount of time.

It was very simple to implement and it works without my doing anything except to tell NL to get the headers.

As you state in your reply "as long as the server supports it, bingo. tis done."

The good news is - just found this quote in the wishlist forum:
Spiril wrote:Ah, that header compression command is long overdue. Nice to see NNTP server developers are finally implementing it.

I'll make NewsLeecher support it in one of the upcoming releases.

tommo123
Posts: 123
Joined: Wed Oct 19, 2005 4:43 pm

Post by tommo123 »

but NL doesnt do anything, it's the other software that does all the work. as for the wish list, that's been there for at least a year iirc.

NL isn't a free program, and should have support for a feature like this built in. i have no idea what's happening with NL lately. instead of improvements we get basic features removed.

My NL folder is now over a gig (literally!) due to not being able to remove anything ffs!

i'm trying Usenet Explorer at the moment. looking promising

User avatar
the doctor
Posts: 250
Joined: Wed Apr 22, 2009 12:40 am
Location: DeLand, Florida USA
Contact:

Post by the doctor »

I don't understand "not being able to remove anything". All you have to do is:

1. Close NL then restart NL (just to make sure no groups are active)

2. open the Manage tab and

3. hilite all your groups or select what ever groups you want to empty

4. right click (on a selected group) - menu popup - choose Selected: Purge Group

In my case the NL user folder was reduced to slightly more than 3Mb

This procedure will reset the selected groups to 0 headers by purging the cache

You can also do this by finding the cacheV4 folder in your NL user files and delete the contents of this folder or just the cache
files you want to empty.

In my Windows 7 it is in:
C:\Users\michael\AppData\Roaming\NewsLeecher
(In XP Users = documents & settings and AppData = Application Data)

TO see these folders you have to have all files showing in folder options - no hide boxes checked

tommo123
Posts: 123
Joined: Wed Oct 19, 2005 4:43 pm

Post by tommo123 »

purging the group afaik, gets rid of all info - meaning next time i update it, i download all headers again

User avatar
the doctor
Posts: 250
Joined: Wed Apr 22, 2009 12:40 am
Location: DeLand, Florida USA
Contact:

Post by the doctor »

Even the most active groups do not post more than about 1,000,000 headers each day. For intance a.b.boneless only had 900,000 headers since yesterday. Even w/o compression this only takes 5 or 6 min to load at my speed limited to 2MB/s (and much faster at full speed - don't know your connection speed).

You can still limit how many headers you get.

tommo123
Posts: 123
Joined: Wed Oct 19, 2005 4:43 pm

Post by tommo123 »

i sub to 8-9 groups and if you purge them, it starts to download from scratch again. = 10s of millions for some.

i know yuo can tell it to limit how far back it goes to update (limit to 10,000 or whatever), but considering some groups vary massively it's not worth the effort.

i;m just dissapointed in the way NL has gone.

User avatar
Smudge
Site Admin
Posts: 10034
Joined: Tue Aug 17, 2004 1:42 am

Post by Smudge »

Removing the data from the cache files on the hard drive is being worked on in the current alpha test versions.
Please be aware of and use the following pages...
Services Status Page : SuperSearch and Usenet Access server status, retention, server load, indexing time, etc.
Support Request Form : Use this if you have a problem with billing or account status. The forum is only for NewsLeecher application issues.

sheepshagger
Posts: 457
Joined: Thu Aug 16, 2007 4:00 pm

Post by sheepshagger »

the doctor wrote:Even the most active groups do not post more than about 1,000,000 headers each day. For intance a.b.boneless only had 900,000 headers since yesterday. Even w/o compression this only takes 5 or 6 min to load at my speed limited to 2MB/s (and much faster at full speed - don't know your connection speed).

You can still limit how many headers you get.
depends on your server retention but boneless will have more than 1 mill posts per day on good servers

Species1571
Posts: 12
Joined: Fri Jul 10, 2009 10:47 pm

Post by Species1571 »

Have to agree, this should have been implemented a long time ago, as from what I read it is a very simple code to implement. I feel your pain downloading headers, I am also on astraweb and downloading 365 days of headers from alt.binaries.multimedia and some others took about 10 hours, had to do it across several days.

Post Reply