Incomplete downloaded files

Found a bug in NewsLeecher? Post a bug report here. Please describe how to reproduce the bug.
• Be sure to read the bug reporting guidelines before posting.
dwazegek
Forum Moderator
Posts: 3006
Joined: Tue Sep 28, 2004 8:36 am

Re: Combining files with missing parts

Post by dwazegek »

doveman wrote:I have used PAR's before, but I don't think they can be used in this situation where the existing parts have already been combined. I'll be happy to be corrected on this point. :)
well i'll correct you then ;)
you can use pars (or more likely, par2's) after newsleecher has joined the articles, in fact, i never do it any other way.

doveman
Posts: 9
Joined: Tue Jan 11, 2005 12:06 pm

Post by doveman »

you can use pars (or more likely, par2's) after newsleecher has joined the articles, in fact, i never do it any other way.
I tried to simulate what would happen if I had a corrupted rar file (IE that Newsleecher had created when one or more parts of the article were missing), and then tried to fix it using PAR2 files.

1. Used Yenc32 (v1.0.6.206) to Encode a 3MB rar file into 6 parts (512Kb each)

2. Used QuickPar (v0.8.0.0) to make 4 recovery blocks from the 6 Yenc parts

3. Used Yenc32 to Decode the Yenc parts back into a rar file, but deliberately only select 5 of the 6 parts

4. Tried to open the resulting corrupted rar file, WinRAR gives 'unexpected end of archive error'

5. Used QuickPar to load the PAR2 file made in step 2. Tells me it has found 4 blocks and needs another 8.

6. Click Add and select the corrupted rar file made in step 3. QuickPar still says it needs 8 blocks.

So as far as I can see, what you say you do, doesn't actually work.

dwazegek
Forum Moderator
Posts: 3006
Joined: Tue Sep 28, 2004 8:36 am

Post by dwazegek »

if you don't check the files after they've been joined, how do you do normal do your parring and unrarring?
do you add all the decoded articles into quickpar seperately?
if so you should be getting a hell of lot more incompletes than necessary.

if a poster is posting with 5000 lines (640.000 bytes) and using a par2 block size of 1.280.000 (so 2 articles is 1 block), and you add all the decoded (still split) articles into quickpar it will read 0 blocks, as no single article contains a complete block.
now if you join the articles, then quickpar will be able to read all the blocks.
just tested this myself. i created a rar archive and usend yenc32 as well (same version, the quickpar version i used was 0.91, but that shouldn't matter) to make articles, then decoded the articles one by one. created a par2 archive with blocks that were twice the size of the article size and checked the decoded files. only 1 article contained a block (last article which was a bit smaller than the rest). when i joined the seperate articles all blocks were available...
as a seperate test i decoded the articles again but didn't use 2 articles, quickpar then said i was missing 2 blocks (retested it but now i removed 2 articles that would be in the same block e.g. article 3 and 4, quickpas now gave 1 block missing)

if you're really adding decoded (unjoined) articles into quickpar, it should fail horribly in most cases, it should only really work when the article size and the block size is exactly equal (for efficiency reasons this is the best setting to post with, but most posters don't know this, or just don't bother)

i'm trying to understand what could have gone wrong in your test, but i really can't figure it out (only thing i can imagine, is that instead of decoding 5/6 of the articles, you decoded a few less, but even then the numbers don't really fit). was the rar file exactly 3mb (3.145.728 bytes)? what par2 block size did you use? and when quickpar said it had 4 blocks and needed 8, where did those 4 come from, were they the repairblocks, or did they come from the decoded rar file?

tojada
Posts: 7
Joined: Sat Jan 22, 2005 4:00 pm

Incomplete multiparts

Post by tojada »

:shock: I don't know if this bug was already report, don't want to scan through the hundreds of pages.
The problem I have is the following:
I use two servers for downloading multipart messages, the first one is very fast but has very often incomplete articles (~90%). The second one is slow but is complete. I download all headers with server no.2 but give priority 1 for downloading to server no.1. This usually should work quite fine with NL and I haven't been able to get this kind of operation running automatically with other newsreaders. Compliments to the NL team !!
Now the BUG:
Sometimes I loose connection with server no.2 during dl of a single part, that means this single part is not complete, neither in the raw nor in the bin directory. The file is existing in both directories, but has zero length. However, in the properties window of the arcticle it shows download complete. And there is no way to tell NL, that it's not. Consequently when it comes to merge the multipart attachment, the whole thing is incomplete.
The mistake in NL is that the single parts are marked as complete as soon as the dl starts. There is no check if the individual parts are complete before merging.
Thanks for listening.
tojada

doveman
Posts: 9
Joined: Tue Jan 11, 2005 12:06 pm

Incomplete Multiparts

Post by doveman »

I've sort of forgotten exactly what I did when I tried to fix the incomplete download, that NL had joined and save, with PAR2's, but I might as well tell you what my testing has shown.
i'm trying to understand what could have gone wrong in your test, but i really can't figure it out (only thing i can imagine, is that instead of decoding 5/6 of the articles, you decoded a few less, but even then the numbers don't really fit). was the rar file exactly 3mb (3.145.728 bytes)? what par2 block size did you use? and when quickpar said it had 4 blocks and needed 8, where did those 4 come from, were they the repairblocks, or did they come from the decoded rar file?
Looking back at the example I gave, I think the reason it didn't work is because I should have made the PAR2 files (step 2) from the original RAR file, and not the 6 yEnc encoded parts (doh!). So I did another test.

Test 2 (Works)
--------------
1. Used yEnc32 to to encode Test.Rar (3,046KB or 2.97MB) into 5 parts (set to 640,000 bytes, but makes them 688,128 bytes or 650KB - Why is this?)
2. Used QuickPAR to create 2 recovery blocks from Test.rar (Recovery Data Size=2,561,456). Settings: Source Block Count=3, Block=1,280,000 bytes (Options: Preferred Block Size For Yenc = 5000 lines, Block size=640,000 bytes, Article Size=670,000 bytes)

3. Deleted Test.rar. Used QuickPar to load the PAR2 files made in step 2. Shows 2 recovery blocks available, 0 data blocks, needs 1 block.

4. Joining 2 or more yEnc parts to make Test.rar should enable QuickPAR to repair. It does BUT there's a BUG in QuickPAR - Clicking Add and selecting this Test.rar doesn't always work, but starting again by clicking on Open and selecting the PAR2 file allows QuickPAR to repair (the Test.rar is in the same directory as the PAR files, and QuickPAR sees it). This bug might be fixed in a newer version of QuickPAR than the one I'm using (v0.8.0.0).

NOTE: Only some yEnc decoded 2-part combinations work for QuickPAR (1+2-yes, 1+3-no, 1+4-no, 1+5-yes, 2+3-no, 2+4-no, 2+5-yes, 3+4-yes, 3+5-yes, 4+5-yes. So 5 and any other part work, or 1+2 or 3+4.)

So if NewsLeecher finds, for example, only parts 1+3, and downloads and saves them, yEnc decoded and combined, I wouldn't be able to use the resulting file with PARs to fix it. I guess this is just the way PAR's work, in that 1+2, 3+4, or 5 all constitute 1 block, but non-adjacent blocks like 1+3 don't
if you don't check the files after they've been joined, how do you do normal do your parring and unrarring?
do you add all the decoded articles into quickpar seperately?
if so you should be getting a hell of lot more incompletes than necessary.
I normally use Agent, in which I have to manually save the available parts to a file, and then decode this using yEnc32. So I AM loading a joined and yEnc decoded incomplete file into QuickPAR. Not sure why I couldn't get this to work with NewsLeecher the other day. Might be because of the bug mentioned above.
just tested this myself. i created a rar archive and usend yenc32 as well (same version, the quickpar version i used was 0.91, but that shouldn't matter) to make articles, then decoded the articles one by one.
What do you mean by decoded the articles one by one? I used yEnc32 to encode test.rar (2.97MB or 3,046KB), using 'Limit Size of Encoded Files to'=512KB which created 6 files (test.001.ntx - test.006.ntx (each 533KB except 006=505KB))

I then used yEnc32 to individually decode each of these 6 parts (IE selected test.001.ntx, decoded that, then selected test.002.ntx, decoded that, etc.) Decoding the first ntx file created a file (test.rar) which was 512KB and decoding each subsequent part increased this file by 512KB, until the 6th part which resulted in test.rar being 3,046KB file which opened without error in WinRAR (despite Yenc32 stating 'File test.rar is incomplete')

Then I tried again, but this time I renamed, and moved to a sub-directory, the test.rar after decoding each part. Again this created an inital test.rar of 512KB, which increased by 512KB each time, (so after renaming and moving I had test01.rar - test06.rar each of which was 512KB larger than the last, which seems wierd seeing as how each time I was only decoding one part), so decoding test.006.ntx resulted in a test.rar of 3,046KB. However trying to open this gave an error 'The archive is either in unknown format or damaged'. (NB. you can get the same result by only selecting and decoding test.006.ntx, there's no need to go through the whole process.)

dwazegek
Forum Moderator
Posts: 3006
Joined: Tue Sep 28, 2004 8:36 am

Re: Incomplete Multiparts

Post by dwazegek »

doveman wrote:1. Used yEnc32 to to encode Test.Rar (3,046KB or 2.97MB) into 5 parts (set to 640,000 bytes, but makes them 688,128 bytes or 650KB - Why is this?)
it's because of a overhead in the encoding process, all encoding formats have a certian overhead (uuencode, base64 and binhex have about 40% overhead (which means the poster has to post 40% extra, and the leechers have to download 40% extra, which is never a good thing), yenc and bn have about 3%)
doveman wrote:So if NewsLeecher finds, for example, only parts 1+3, and downloads and saves them, yEnc decoded and combined, I wouldn't be able to use the resulting file with PARs to fix it. I guess this is just the way PAR's work, in that 1+2, 3+4, or 5 all constitute 1 block, but non-adjacent blocks like 1+3 don't
yep, this is also the reason why posters *should* always use par2 blocks that have the same size as the articles they're posting with. and ideally when making rars, the rarsize should be a multiple of the par2 block size (otherwize the last block will require (e.g.) a 640kB par2 block to repair a 100 kB block of data)
doveman wrote:What do you mean by decoded the articles one by one?
i did the same as what you did, i decoded one single article, then decoded the next one and the next one till they were all decoded seperately.
doveman wrote:so after renaming and moving I had test01.rar - test06.rar each of which was 512KB larger than the last, which seems wierd seeing as how each time I was only decoding one part
i noticed the same thing, i guess yenc32 is reserving disk space, so if other articles become available it can just stick them in the already existing file.

doveman
Posts: 9
Joined: Tue Jan 11, 2005 12:06 pm

Post by doveman »

Well at least I know more about PAR's than I did before, thanks to you.

Cheers

Joez
Posts: 4
Joined: Wed Feb 16, 2005 11:02 pm

Post by Joez »

I'm having the same problem with files slightly incomplete after being downloaded. I just got a new computer, installed XP Pro with all the updates, Zone Alarm (free), a few other programs like AIM, Adobe Acrobat, Winrar, Firefox, and the latest beta version of NewsLeecher.

Is there a summary of this thread, or any known workarounds?

sir_fresh
Posts: 19
Joined: Sun Jul 11, 2004 12:21 pm

Post by sir_fresh »

I'm having the same problem with files slightly incomplete after being downloaded.
Me too. On a download of 4 gig (about 90 rar files) I always have one or two files slightly incomplete. Very annoying cause when I delete them and redownload they come through complete. Is there a solution so they come through complete the first time instead of having to redownload the incompletes (or fix them)?
AJAXXXXXXXX

tojada
Posts: 7
Joined: Sat Jan 22, 2005 4:00 pm

Post by tojada »

I think the major problem of NL is its lack of robustness.

In order to find out the reason for those many incomplete multipart downloads I have monitored the status of the cache folder and the tempory files (the raw and binary parts) during some download processes.
It appears that NL at the start of download does a perfect stock taking of what parts are available on which server, but it never controls the d/l process itself. If for any reason a part gets lost, is corrupted, has zero length, etc. (and there are many reasons and minor bugs, which can lead to such a situation), NL continues w/o noticing such things. (see also my posting from January)

I would expect that it checks the status of the cache folder each time it downloads a new part/article, and does a final check before doing the decoding.

I also do not understand why the raw and the bin parts need to be stored in the cache, the raw parts should be sufficient. On the other hand, probably to save disk space, the cache files of incomplete downloads are automatically deleted when NL is closed. This prevents from completing a download at a later opportunity. Anyway, in the current version this is not possible at all, because once a download has been stopped, e.g. by unchecking the box at the left of the download or active group window, and you want to restart it, NL starts all over again, ignoring the fact that parts are already available in the cache; a consequence of the improper download control.

Allthough a lot of functions have been copied from other progies like Grabit and Newsbin, their implementation in NL is still incomplete.

nyt
Posts: 7
Joined: Sat Sep 11, 2004 12:56 am

Post by nyt »

Agreed, if the bug sits after the download part (i.e during rejoin) no amount of logging will show the issue, even if one segment ends up missing in the final file. Please, enable some debugging in this area so we can have this issue fixed without having to redownload the file each time. A simple file size check could enforce a segment validation and redownload the broken one? That would be a workaround for the time being.

ViDi0t
Posts: 56
Joined: Sat Nov 15, 2003 7:10 am

Post by ViDi0t »

Has this problem been fixed in the new versions of NL?

I have been having random icompletes forever with NL and I am still holding off buying it until this is fixed. It still amazes me that with all these new final and beta releases that this one issue has never been fixed.

User avatar
Noir
Posts: 75
Joined: Sun Jul 18, 2004 6:41 pm

Post by Noir »

Nope the bug is still here.

User avatar
Spiril
Site Admin
Posts: 4278
Joined: Fri Nov 07, 2003 3:11 am

Post by Spiril »

I'll look into this prob some more within the next couple of days. hopefully it will be fixed for the 2.1 final release...

Just to sum-up, please write the easiest way you are able to re-produce the bug.

Cheers
Spiril

User avatar
GekkeKoe
Posts: 201
Joined: Sat Mar 19, 2005 1:29 am
Location: Reg'd NL User

Post by GekkeKoe »

tojada wrote:I think the major problem of NL is its lack of robustness.

In order to find out the reason for those many incomplete multipart downloads I have monitored the status of the cache folder and the tempory files (the raw and binary parts) during some download processes.
It appears that NL at the start of download does a perfect stock taking of what parts are available on which server, but it never controls the d/l process itself. If for any reason a part gets lost, is corrupted, has zero length, etc. (and there are many reasons and minor bugs, which can lead to such a situation), NL continues w/o noticing such things. (see also my posting from January)

I would expect that it checks the status of the cache folder each time it downloads a new part/article, and does a final check before doing the decoding.

(snip)
I think that pretty much sums it up. For my part, can't really reproduce at will having NL grab complete files that end up incomplete, it just happens.. Re-downloading will almost always result in an OK file. IMHO (without knowing intricate facts of NL's internals) there should be (more) (error) checking/handling done - while & after downloading and - before and during decoding.

Cheers,
GekkeKoe
NARF!

Post Reply