Incomplete downloaded files

Found a bug in NewsLeecher? Post a bug report here. Please describe how to reproduce the bug.
• Be sure to read the bug reporting guidelines before posting.
Post Reply
User avatar
salami
Forum Moderator
Posts: 838
Joined: Thu Nov 13, 2003 10:10 pm
Location: Basel, Switzerland
Contact:

Incomplete downloaded files

Post by salami »

hi there

since quite some time, NL is having a nasty bug: files are sometime missing a few bytes after downloading. this behaviour seems to show only in multiserver configurations (or at least more often) and also seems to vary from beta to beta.

i know we can fix the files with PAR2, but it still is one of the worst possible bugs. it has been reported over and over again in endless threads, so i decided to sticky this thread to point some more attention to it.

if you can provide useful information on when and which circumstances it happens, please reply to this thread, maybe this helps to get Spiril to fix it :-)

DaPhatty Edit - Also, if you are a RoadRunner ISP user, the problem is most likely RoadRunner's fault. They have not implemented decent Message ID support and that is why you are having problems. Check the News Provider Compatibility thread in the Tech Support Forum for details.

Hirogen
Posts: 15
Joined: Sat Feb 21, 2004 6:24 pm

Post by Hirogen »

What can I say it happens all the time, when I download all headers, when I do a server search and when using NZB files, it varies from 25 to 90 % files incompleet or corrupt.
Setup:
ISP 3 bots priority 1
news.usenetserver 6 bots priority 1
test.usenetserver 6 bots priority 1
When I download these files with either Xnews or Newsbin all files are just fine
Tried new clean install even an OS install but it keeps occuring with all my downloads will try out Beta 20 later today to see if that one fixes it.
One more thing with Beta 9 I didn't had that problem it started with beta 12

Lips
Forum Moderator
Posts: 3804
Joined: Thu Mar 18, 2004 6:57 pm

Post by Lips »

Hirogen wrote:What can I say it happens all the time, when I download all headers, when I do a server search and when using NZB files, it varies from 25 to 90 % files incompleet or corrupt.
Setup:
ISP 3 bots priority 1
news.usenetserver 6 bots priority 1
test.usenetserver 6 bots priority 1
When I download these files with either Xnews or Newsbin all files are just fine
Tried new clean install even an OS install but it keeps occuring with all my downloads will try out Beta 20 later today to see if that one fixes it.
One more thing with Beta 9 I didn't had that problem it started with beta 12
1. Ok. A clean install (which version of NL?) in a newly installed OS - correct?
So, the cache is brand new as well as the registry. Correct?

2. When you downloaded with Xnews and Newsbin, was article number or message-id used? NL beta 9 retrieved articles by article number, not message-id. Subsequent versions only use message-id. Just trying to narrow this down to (or eliminate) message-id problems with UNS or problems with UNS & NL. I've heard about message-id and sync problems (between front-end and back-end servers) in the past, but didn't know if the new owners had gotten the problems sorted out. Salami recently mentioned message-id problems with UNS with "older" articles. Also, it seems to me that most of the posts about the problem involves UNS servers (no statistics on this, just my often unreliable memory).

3. Also, try (if you haven't already) activating the "Send Group" command option. Since b15 doesn't work for you (correct?) I don't really expect this to solve all, but just see if it helps. Hey, it's easy, and might as well try.

4. Is it possible for you to try downloading from servers other than UNS?

Salami, good idea starting this thread to keep posts for this problem organized. The posts were getting too spread out. Difficult (at least for me) to follow.

0ne more thing. I don't use UNS or any of the unlimited accounts (with the exception of my ISP). I hadn't seen this "incomplete files" problem in a long time. But, I ran into this problem just after updating b19 to b20. Immediately after 2 groups were "cleaned (more that 50% garbage)", files started downloading as incomplete. The log showed "(incomplete file)". Thanks for that log entry Spiril. Definitely a help. The files had a status of "downloaded" and were no longer in the queue. Seemed like a cache problem to me. Resetting the 2 groups, which cleaned the cache and re-downloaded headers, solved the problem immediately. Just to make sure it wasn't a "cleaning" problem in b20, I cleaned the same groups, downloaded files, but could not duplicate the problem.

--
Lips

User avatar
salami
Forum Moderator
Posts: 838
Joined: Thu Nov 13, 2003 10:10 pm
Location: Basel, Switzerland
Contact:

Post by salami »

i don't think this problem is directly related to the providers. other clients do not show such effect, even with message-ID retrival. also the fact that you can download the same file again without a problems clearly indicates it must be the fault of NL.

User avatar
Palanca
Posts: 19
Joined: Tue Apr 06, 2004 11:43 am

Me too!

Post by Palanca »

Happened the same at me with some binaries or images.
- Binaries lost some bytes enough to give me a bad crc during unrar or unzip

- Images got a different crc. (checked with the csv file attached)

Single server
Giganews with 2 bots
Before B19 never had this problem (neither with giganews in 5 years subscription).

Have fun

Eraserhead
Posts: 101
Joined: Fri Dec 05, 2003 1:02 pm

Post by Eraserhead »

I've had this problem the first time with b20. all parts are present, everything seems to be downloaded but the progress bar for the file is at ~99% and the status is queued...
I checked the parts and noticed this: the last part was renamed by Newsleecher from xyz.com to xyz.coó
I guess that's why NL couldn't complete the download.
I did several checks with different files and always the last segment had a wrong ending.

Strangely this only happens when I use 4 different Newsservers (I just signed up for Usenetserver as well) - when I reduce the number of different server to 2 I don't get this problem.
Last edited by Eraserhead on Sat Jan 07, 2006 12:22 am, edited 1 time in total.

User avatar
Spiril
Site Admin
Posts: 4278
Joined: Fri Nov 07, 2003 3:11 am

Post by Spiril »

Hirogen wrote:What can I say it happens all the time, when I download all headers, when I do a server search and when using NZB files, it varies from 25 to 90 % files incompleet or corrupt.
Hi Hirogen

It also happens with .NZB files?

Next time it happens, pls email me an .NZB file that works with newsbin but fails with NL.

Also, could you try to run with a server setup without your ISP server and see if the corruption still occurs? That will probably make it easier for me to track the bug, as I also run with the news./test.usenetserver.com a server setup.

Cheers

User avatar
Spiril
Site Admin
Posts: 4278
Joined: Fri Nov 07, 2003 3:11 am

Post by Spiril »

Eraserhead wrote:I've had this problem the first time with b20. all parts are present, everything seems to be downloaded but the progress bar for the file is at ~99% and the status is queued...
I checked the parts and noticed this: the last part was renamed by Newsleecher from xyz.com to xyz.coó
I guess that's why NL couldn't complete the download.
I did several checks with different files and always the last segment had a wrong ending.
Interesting.

Do you fetch group headers from all 4 servers?

Does the corruption also happen after the first time you've fetched headers from a group? I mean, before the headers have been saved and reloaded from the disk cache database?

Eraserhead
Posts: 101
Joined: Fri Dec 05, 2003 1:02 pm

Post by Eraserhead »

Spiril wrote:Interesting.

Do you fetch group headers from all 4 servers?

Does the corruption also happen after the first time you've fetched headers from a group? I mean, before the headers have been saved and reloaded from the disk cache database?
I download headers from 2 servers.
It happened both after fetching first time and also after restarting/reloading.

I'll do some further tests and report back.

Lips
Forum Moderator
Posts: 3804
Joined: Thu Mar 18, 2004 6:57 pm

Post by Lips »

salami wrote:i don't think this problem is directly related to the providers. other clients do not show such effect, even with message-ID retrival.
You may be right. Just trying to see if a problem exists between NL and another provider or eliminate the possiblity. That's one thing I'm trying to bring out here when asking the OP about Xnews and Newsbin and whether message-id or article number was used. So, you can confirm that other clients consistantly work with message-id on the same failed NL downloads? If UNS has been having some problems that effect access by message-id then there is reasons to at least explore the possibility that this could be (or not) part of the problem.
salami wrote:also the fact that you can download the same file again without a problems clearly indicates it must be the fault of NL.
Maybe. That's what I'm trying to find out. So far, I don't see anything clear about this problem. I guess I've missed something. So, whenever this problem occurs, the files can immediately by downloaded on the next try using NL? Is this is true for everyone with the problem? I didn't think this was the case.
As I've previously stated, I do not normally see this problem. I'm sure I'm not the only one. The only time I've recently seen this happen it was due to a problem with the cache. Has this been ruled out for others with the problem? Is this only a problem with a very large cache? There can be so many variables.
I'm asking questions in the hope that the details of this problem will be brought out and organized in this thread and make it easier for the problem to be diagnosed.
That's enough for now. Haven't even had any coffee yet.

--
Lips

Hirogen
Posts: 15
Joined: Sat Feb 21, 2004 6:24 pm

Post by Hirogen »

Lips wrote:1. Ok. A clean install (which version of NL?) in a newly installed OS - correct?
So, the cache is brand new as well as the registry. Correct?

Correct

2. When you downloaded with Xnews and Newsbin, was article number or message-id used? NL beta 9 retrieved articles by article number, not message-id. Subsequent versions only use message-id. Just trying to narrow this down to (or eliminate) message-id problems with UNS or problems with UNS & NL. I've heard about message-id and sync problems (between front-end and back-end servers) in the past, but didn't know if the new owners had gotten the problems sorted out. Salami recently mentioned message-id problems with UNS with "older" articles. Also, it seems to me that most of the posts about the problem involves UNS servers (no statistics on this, just my often unreliable memory).


Both msg id and article number as I said tried the same posts with all 3 readers NL was the only one giving corrupts

3. Also, try (if you haven't already) activating the "Send Group" command option. Since b15 doesn't work for you (correct?) I don't really expect this to solve all, but just see if it helps. Hey, it's easy, and might as well try.

Correct I had this problem since Beta 12 I tried the "send group command" with a few files last night and still got corrupts

4. Is it possible for you to try downloading from servers other than UNS?

I had Astranews and Easynews before and I have my ISP all the same problem

--
Lips

Hirogen
Posts: 15
Joined: Sat Feb 21, 2004 6:24 pm

Post by Hirogen »

Spiril wrote:Hi Hirogen

It also happens with .NZB files?

Next time it happens, pls email me an .NZB file that works with newsbin but fails with NL.

Also, could you try to run with a server setup without your ISP server and see if the corruption still occurs? That will probably make it easier for me to track the bug, as I also run with the news./test.usenetserver.com a server setup.

Cheers
Yes it happens with NZB files, with server search and when retreiving headers the old fashioned way, I will send you an email with the NZB I will use for testing now expect it soon.

About the server setup I tried every server on it's own but I wil do just that again to see if it is of any help for you.

And Spiril if you can fix this bug I'm ready to register NL atm I'm using NB again because of all those corrupt files NL is giving me but I love NL and would love to switch to NL.

--
Thx

Lips
Forum Moderator
Posts: 3804
Joined: Thu Mar 18, 2004 6:57 pm

Post by Lips »

Hirogen wrote:
Lips wrote:1. Ok. A clean install (which version of NL?) in a newly installed OS - correct?
So, the cache is brand new as well as the registry. Correct?

Correct

2. When you downloaded with Xnews and Newsbin, was article number or message-id used? NL beta 9 retrieved articles by article number, not message-id. Subsequent versions only use message-id. Just trying to narrow this down to (or eliminate) message-id problems with UNS or problems with UNS & NL. I've heard about message-id and sync problems (between front-end and back-end servers) in the past, but didn't know if the new owners had gotten the problems sorted out. Salami recently mentioned message-id problems with UNS with "older" articles. Also, it seems to me that most of the posts about the problem involves UNS servers (no statistics on this, just my often unreliable memory).


Both msg id and article number as I said tried the same posts with all 3 readers NL was the only one giving corrupts

3. Also, try (if you haven't already) activating the "Send Group" command option. Since b15 doesn't work for you (correct?) I don't really expect this to solve all, but just see if it helps. Hey, it's easy, and might as well try.

Correct I had this problem since Beta 12 I tried the "send group command" with a few files last night and still got corrupts

4. Is it possible for you to try downloading from servers other than UNS?

I had Astranews and Easynews before and I have my ISP all the same problem
Thanks, Hirogen for the details.

--
Lips

User avatar
Spiril
Site Admin
Posts: 4278
Joined: Fri Nov 07, 2003 3:11 am

Post by Spiril »

Hirogen wrote:Yes it happens with NZB files, with server search and when retreiving headers the old fashioned way, I will send you an email with the NZB I will use for testing now expect it soon.
Hi Hirogen. Did you send the NZB yet?

Hirogen
Posts: 15
Joined: Sat Feb 21, 2004 6:24 pm

Post by Hirogen »

No not yet I yesterday upgraded to B20 and have tried 2 NZB's and they worked just fine I thought I upgraded a day before but I guess I didn't so I will keep trying some more, what i did notice with B20 that when I do a server search the last file of a batch I download misses 1 segment so if I do a post in 3 batches downen then I will need to redownload 3 files if I can download it 1 time I will need to redownload 1 file and always the last of the queue, but this should be easy to prevent by downloading the whole post and a extra par file and not start downloading somewhere in the middel of the post, but I wil try out some more NZB's but I need to find something of interest to download first.

Post Reply