Incomplete downloaded files

[salami]

hi there

since quite some time, NL is having a nasty bug: files are sometime missing a few bytes after downloading. this behaviour seems to show only in multiserver configurations (or at least more often) and also seems to vary from beta to beta.

i know we can fix the files with PAR2, but it still is one of the worst possible bugs. it has been reported over and over again in endless threads, so i decided to sticky this thread to point some more attention to it.

if you can provide useful information on when and which circumstances it happens, please reply to this thread, maybe this helps to get Spiril to fix it

DaPhatty Edit - Also, if you are a RoadRunner ISP user, the problem is most likely RoadRunner's fault. They have not implemented decent Message ID support and that is why you are having problems. Check the News Provider Compatibility thread in the Tech Support Forum for details.

[Hirogen]

What can I say it happens all the time, when I download all headers, when I do a server search and when using NZB files, it varies from 25 to 90 % files incompleet or corrupt.
Setup:
ISP 3 bots priority 1
news.usenetserver 6 bots priority 1
test.usenetserver 6 bots priority 1
When I download these files with either Xnews or Newsbin all files are just fine
Tried new clean install even an OS install but it keeps occuring with all my downloads will try out Beta 20 later today to see if that one fixes it.
One more thing with Beta 9 I didn't had that problem it started with beta 12

[Lips]

What can I say it happens all the time, when I download all headers, when I do a server search and when using NZB files, it varies from 25 to 90 % files incompleet or corrupt.
Setup:
ISP 3 bots priority 1
news.usenetserver 6 bots priority 1
test.usenetserver 6 bots priority 1
When I download these files with either Xnews or Newsbin all files are just fine
Tried new clean install even an OS install but it keeps occuring with all my downloads will try out Beta 20 later today to see if that one fixes it.
One more thing with Beta 9 I didn't had that problem it started with beta 12

1. Ok. A clean install (which version of NL?) in a newly installed OS - correct?
So, the cache is brand new as well as the registry. Correct?

2. When you downloaded with Xnews and Newsbin, was article number or message-id used? NL beta 9 retrieved articles by article number, not message-id. Subsequent versions only use message-id. Just trying to narrow this down to (or eliminate) message-id problems with UNS or problems with UNS & NL. I've heard about message-id and sync problems (between front-end and back-end servers) in the past, but didn't know if the new owners had gotten the problems sorted out. Salami recently mentioned message-id problems with UNS with "older" articles. Also, it seems to me that most of the posts about the problem involves UNS servers (no statistics on this, just my often unreliable memory).

3. Also, try (if you haven't already) activating the "Send Group" command option. Since b15 doesn't work for you (correct?) I don't really expect this to solve all, but just see if it helps. Hey, it's easy, and might as well try.

4. Is it possible for you to try downloading from servers other than UNS?

Salami, good idea starting this thread to keep posts for this problem organized. The posts were getting too spread out. Difficult (at least for me) to follow.

0ne more thing. I don't use UNS or any of the unlimited accounts (with the exception of my ISP). I hadn't seen this "incomplete files" problem in a long time. But, I ran into this problem just after updating b19 to b20. Immediately after 2 groups were "cleaned (more that 50% garbage)", files started downloading as incomplete. The log showed "(incomplete file)". Thanks for that log entry Spiril. Definitely a help. The files had a status of "downloaded" and were no longer in the queue. Seemed like a cache problem to me. Resetting the 2 groups, which cleaned the cache and re-downloaded headers, solved the problem immediately. Just to make sure it wasn't a "cleaning" problem in b20, I cleaned the same groups, downloaded files, but could not duplicate the problem.

--
Lips

[salami]

i don't think this problem is directly related to the providers. other clients do not show such effect, even with message-ID retrival. also the fact that you can download the same file again without a problems clearly indicates it must be the fault of NL.

[Palanca]

Happened the same at me with some binaries or images.
- Binaries lost some bytes enough to give me a bad crc during unrar or unzip

- Images got a different crc. (checked with the csv file attached)

Single server
Giganews with 2 bots
Before B19 never had this problem (neither with giganews in 5 years subscription).

Have fun

[Eraserhead]

I've had this problem the first time with b20. all parts are present, everything seems to be downloaded but the progress bar for the file is at ~99% and the status is queued...
I checked the parts and noticed this: the last part was renamed by Newsleecher from xyz.com to xyz.coó
I guess that's why NL couldn't complete the download.
I did several checks with different files and always the last segment had a wrong ending.

Strangely this only happens when I use 4 different Newsservers (I just signed up for Usenetserver as well) - when I reduce the number of different server to 2 I don't get this problem.

[Spiril]

What can I say it happens all the time, when I download all headers, when I do a server search and when using NZB files, it varies from 25 to 90 % files incompleet or corrupt.

Hi Hirogen

It also happens with .NZB files?

Next time it happens, pls email me an .NZB file that works with newsbin but fails with NL.

Also, could you try to run with a server setup without your ISP server and see if the corruption still occurs? That will probably make it easier for me to track the bug, as I also run with the news./test.usenetserver.com a server setup.

Cheers

[Spiril]

I've had this problem the first time with b20. all parts are present, everything seems to be downloaded but the progress bar for the file is at ~99% and the status is queued...
I checked the parts and noticed this: the last part was renamed by Newsleecher from xyz.com to xyz.coó
I guess that's why NL couldn't complete the download.
I did several checks with different files and always the last segment had a wrong ending.

Interesting.

Do you fetch group headers from all 4 servers?

Does the corruption also happen after the first time you've fetched headers from a group? I mean, before the headers have been saved and reloaded from the disk cache database?

[Eraserhead]

Interesting.

Do you fetch group headers from all 4 servers?

Does the corruption also happen after the first time you've fetched headers from a group? I mean, before the headers have been saved and reloaded from the disk cache database?

I download headers from 2 servers.
It happened both after fetching first time and also after restarting/reloading.

I'll do some further tests and report back.

[Lips]

i don't think this problem is directly related to the providers. other clients do not show such effect, even with message-ID retrival.

You may be right. Just trying to see if a problem exists between NL and another provider or eliminate the possiblity. That's one thing I'm trying to bring out here when asking the OP about Xnews and Newsbin and whether message-id or article number was used. So, you can confirm that other clients consistantly work with message-id on the same failed NL downloads? If UNS has been having some problems that effect access by message-id then there is reasons to at least explore the possibility that this could be (or not) part of the problem.

also the fact that you can download the same file again without a problems clearly indicates it must be the fault of NL.

Maybe. That's what I'm trying to find out. So far, I don't see anything clear about this problem. I guess I've missed something. So, whenever this problem occurs, the files can immediately by downloaded on the next try using NL? Is this is true for everyone with the problem? I didn't think this was the case.
As I've previously stated, I do not normally see this problem. I'm sure I'm not the only one. The only time I've recently seen this happen it was due to a problem with the cache. Has this been ruled out for others with the problem? Is this only a problem with a very large cache? There can be so many variables.
I'm asking questions in the hope that the details of this problem will be brought out and organized in this thread and make it easier for the problem to be diagnosed.
That's enough for now. Haven't even had any coffee yet.

--
Lips

[Hirogen]

1. Ok. A clean install (which version of NL?) in a newly installed OS - correct?
So, the cache is brand new as well as the registry. Correct?

Correct

2. When you downloaded with Xnews and Newsbin, was article number or message-id used? NL beta 9 retrieved articles by article number, not message-id. Subsequent versions only use message-id. Just trying to narrow this down to (or eliminate) message-id problems with UNS or problems with UNS & NL. I've heard about message-id and sync problems (between front-end and back-end servers) in the past, but didn't know if the new owners had gotten the problems sorted out. Salami recently mentioned message-id problems with UNS with "older" articles. Also, it seems to me that most of the posts about the problem involves UNS servers (no statistics on this, just my often unreliable memory).


Both msg id and article number as I said tried the same posts with all 3 readers NL was the only one giving corrupts

3. Also, try (if you haven't already) activating the "Send Group" command option. Since b15 doesn't work for you (correct?) I don't really expect this to solve all, but just see if it helps. Hey, it's easy, and might as well try.

Correct I had this problem since Beta 12 I tried the "send group command" with a few files last night and still got corrupts

4. Is it possible for you to try downloading from servers other than UNS?

I had Astranews and Easynews before and I have my ISP all the same problem

--
Lips

[Hirogen]

Hi Hirogen

It also happens with .NZB files?

Next time it happens, pls email me an .NZB file that works with newsbin but fails with NL.

Also, could you try to run with a server setup without your ISP server and see if the corruption still occurs? That will probably make it easier for me to track the bug, as I also run with the news./test.usenetserver.com a server setup.

Cheers

Yes it happens with NZB files, with server search and when retreiving headers the old fashioned way, I will send you an email with the NZB I will use for testing now expect it soon.

About the server setup I tried every server on it's own but I wil do just that again to see if it is of any help for you.

And Spiril if you can fix this bug I'm ready to register NL atm I'm using NB again because of all those corrupt files NL is giving me but I love NL and would love to switch to NL.

--
Thx

[Lips]

1. Ok. A clean install (which version of NL?) in a newly installed OS - correct?
So, the cache is brand new as well as the registry. Correct?

Correct

2. When you downloaded with Xnews and Newsbin, was article number or message-id used? NL beta 9 retrieved articles by article number, not message-id. Subsequent versions only use message-id. Just trying to narrow this down to (or eliminate) message-id problems with UNS or problems with UNS & NL. I've heard about message-id and sync problems (between front-end and back-end servers) in the past, but didn't know if the new owners had gotten the problems sorted out. Salami recently mentioned message-id problems with UNS with "older" articles. Also, it seems to me that most of the posts about the problem involves UNS servers (no statistics on this, just my often unreliable memory).


Both msg id and article number as I said tried the same posts with all 3 readers NL was the only one giving corrupts

3. Also, try (if you haven't already) activating the "Send Group" command option. Since b15 doesn't work for you (correct?) I don't really expect this to solve all, but just see if it helps. Hey, it's easy, and might as well try.

Correct I had this problem since Beta 12 I tried the "send group command" with a few files last night and still got corrupts

4. Is it possible for you to try downloading from servers other than UNS?

I had Astranews and Easynews before and I have my ISP all the same problem

Thanks, Hirogen for the details.

--
Lips

[Spiril]

Yes it happens with NZB files, with server search and when retreiving headers the old fashioned way, I will send you an email with the NZB I will use for testing now expect it soon.

Hi Hirogen. Did you send the NZB yet?

[Hirogen]

No not yet I yesterday upgraded to B20 and have tried 2 NZB's and they worked just fine I thought I upgraded a day before but I guess I didn't so I will keep trying some more, what i did notice with B20 that when I do a server search the last file of a batch I download misses 1 segment so if I do a post in 3 batches downen then I will need to redownload 3 files if I can download it 1 time I will need to redownload 1 file and always the last of the queue, but this should be easy to prevent by downloading the whole post and a extra par file and not start downloading somewhere in the middel of the post, but I wil try out some more NZB's but I need to find something of interest to download first.