Page 1 of 1

header compression?

Posted: Tue Aug 11, 2009 2:33 pm
by tommo123
any chance of that being implemented at some point? right now i'm downloading 44 million headers and it's taking a while :o

Re: header compression?

Posted: Wed Aug 12, 2009 11:04 pm
by doppler
tommo123 wrote:any chance of that being implemented at some point? right now i'm downloading 44 million headers and it's taking a while :o
It has to occur at both ends. It's also one of the things being talked
about. Don't hold your breath. This change will take some deep thought.

Posted: Wed Aug 12, 2009 11:10 pm
by tommo123
i'm with astraweb and afaik they do support it.

a program like Giganews accelerator would be useful on astrawebs side but at this point, i think it's a feature that really should be supported in news clients as it makes a HUGE difference for people that download headers

Posted: Sun Aug 16, 2009 4:44 am
by bbarker
I'd like to add a vote for Astraweb's header compression.

Server has to support and your side needs to be setup also

Posted: Sun Aug 16, 2009 11:13 am
by the doctor
Newsleacher already supports header compression - does your newsgroup server support it?

Server has to support and your computer also needs to be setup (running a compression program or service). Newshosting.com uses Stunnel to implement header compression. I am running Stunnel as a service.

Just downloaded 4 million headers in only 11 min vs. uncompressed download time of 2hrs.

I have 2 servers setup - 1 is my provider newsgroup server (uncompressed) and the second is newshosting (using header compression)

Posted: Sun Aug 16, 2009 11:47 am
by tommo123
ummm, what you're saying doc is that it **DOESNT** support it.

you're running 3rd party software like giganews accelerator or something. that's not newsleecher. compute?

newsleecher needs to add support for XZVER then it'll support header compression. until then, it doesn't. simple plain basic fact.

i want my client to support this, not to have to install some random 3rd party software. we should all be able to simply click an option for header compression and as long as the server supports it, bingo. tis done.

Posted: Sun Aug 16, 2009 10:19 pm
by the doctor
I hope we can keep this as a friendly discussion...

I was just pointing out that you can use header compression with Newsleecher - it is not built into NL but NL does support (work with) header compression.

I use Newsleecher and along with the header compression my newsgroup service provides - I am able to download headers at a significantly decreased amount of time.

It was very simple to implement and it works without my doing anything except to tell NL to get the headers.

As you state in your reply "as long as the server supports it, bingo. tis done."

The good news is - just found this quote in the wishlist forum:
Spiril wrote:Ah, that header compression command is long overdue. Nice to see NNTP server developers are finally implementing it.

I'll make NewsLeecher support it in one of the upcoming releases.

Posted: Sun Aug 16, 2009 11:36 pm
by tommo123
but NL doesnt do anything, it's the other software that does all the work. as for the wish list, that's been there for at least a year iirc.

NL isn't a free program, and should have support for a feature like this built in. i have no idea what's happening with NL lately. instead of improvements we get basic features removed.

My NL folder is now over a gig (literally!) due to not being able to remove anything ffs!

i'm trying Usenet Explorer at the moment. looking promising

Posted: Mon Aug 17, 2009 3:36 pm
by the doctor
I don't understand "not being able to remove anything". All you have to do is:

1. Close NL then restart NL (just to make sure no groups are active)

2. open the Manage tab and

3. hilite all your groups or select what ever groups you want to empty

4. right click (on a selected group) - menu popup - choose Selected: Purge Group

In my case the NL user folder was reduced to slightly more than 3Mb

This procedure will reset the selected groups to 0 headers by purging the cache

You can also do this by finding the cacheV4 folder in your NL user files and delete the contents of this folder or just the cache
files you want to empty.

In my Windows 7 it is in:
C:\Users\michael\AppData\Roaming\NewsLeecher
(In XP Users = documents & settings and AppData = Application Data)

TO see these folders you have to have all files showing in folder options - no hide boxes checked

Posted: Tue Aug 18, 2009 9:00 am
by tommo123
purging the group afaik, gets rid of all info - meaning next time i update it, i download all headers again

Posted: Tue Aug 18, 2009 2:42 pm
by the doctor
Even the most active groups do not post more than about 1,000,000 headers each day. For intance a.b.boneless only had 900,000 headers since yesterday. Even w/o compression this only takes 5 or 6 min to load at my speed limited to 2MB/s (and much faster at full speed - don't know your connection speed).

You can still limit how many headers you get.

Posted: Tue Aug 18, 2009 5:23 pm
by tommo123
i sub to 8-9 groups and if you purge them, it starts to download from scratch again. = 10s of millions for some.

i know yuo can tell it to limit how far back it goes to update (limit to 10,000 or whatever), but considering some groups vary massively it's not worth the effort.

i;m just dissapointed in the way NL has gone.

Posted: Tue Aug 18, 2009 8:29 pm
by Smudge
Removing the data from the cache files on the hard drive is being worked on in the current alpha test versions.

Posted: Tue Aug 25, 2009 10:30 am
by sheepshagger
the doctor wrote:Even the most active groups do not post more than about 1,000,000 headers each day. For intance a.b.boneless only had 900,000 headers since yesterday. Even w/o compression this only takes 5 or 6 min to load at my speed limited to 2MB/s (and much faster at full speed - don't know your connection speed).

You can still limit how many headers you get.
depends on your server retention but boneless will have more than 1 mill posts per day on good servers

Posted: Sat Sep 05, 2009 9:43 pm
by Species1571
Have to agree, this should have been implemented a long time ago, as from what I read it is a very simple code to implement. I feel your pain downloading headers, I am also on astraweb and downloading 365 days of headers from alt.binaries.multimedia and some others took about 10 hours, had to do it across several days.