Hans (BX2ABT) reported that a large file could not be downloaded from FalconSat-3 to his QTH. I tried a fix but it does not seem to have worked fully. I believe it fixes the direct issue that Hans reported, where the hole list is too long, but there was a second issue, where we send a hole that is longer than the maximum 16 bits allowed. I tried to fix that too, but now we get a crash whenever you try to download a file that is longer than 16 bits, i.e. 65k.
I tried to solve the issue by truncating the length of the holes when they are stored. If we try to store a hole that is longer than 16 bits then I capped it at 65k. But that breaks the model of the file that we are storing locally. We no longer have an accurate picture of the holes in the file.
To prove this I tested locally with a short file length 1348 bytes. I set the maximim length of a hole to 512 bytes for this test. Here is the header for the file we are using:
We receive the first Broadcast frame:
FILE> From:PFS3-11 to QST-1 Ctrl: 3 Type: UI Res PID: bb FLG: 2 FILE: 344 TYPE: 0 OFF: f4 CRC: 2403
It is at offset f4, which is 244 bytes from the start of the file. If I tell the ground station to request the rest of the file at this point it sends this file request.
FILE REQ: FLG: 12 FILE: 344 BLK_SIZE: f4 Hole 1: 0 244 Hole 2: 488 24
That has a hole list with two holes. The first hole is from byte 0, the start of the file, to byte 244. Then we have a hole from byte 488 to byte 512, because the hole is only 24 bytes long. This is not what we want and it reflects the actual holes we have stored on disk. The ground station knows the file has data from byte 244 to 488, that part is correct, but it thinks the file now ends at byte 512. When we receive a packet with data for the file past byte 512 the hole list does not update correctly and eventually we get a crash.
I think the simplest solution is to keep the hole list on disk with the holes extending to the end of the file, as we have done before. When we send the holes to the spacecraft we truncate them at that point. That may mean a bit more back and forth for large files, but a single hole can request 65k, so it is unlikely to be a common issue.
The second approach would be to break large holes into smaller ones as they are sent. We would need to make sure the length of the hole list is not exceeded. This could be a later optimization.
I suppose there is a third, most complex, option. We could split the holes and then store them that way on disk. It is perhaps the most elegant but would require changing and re-testing the hole creation algorithm. Not something I want to do for an edge case.
I tested this with the test file. Once the first fragment is recieved, the hole request frame is now like this:
FILE REQ: FLG: 12 FILE: 344 BLK_SIZE: f4 Hole 1: 0 244 Hole 2: 488 512
It's a subtle change, but now the second hole is from 488 to byte 1000, extending the maximum 512 bytes allowed in my test. The actual hole on disk extends from 488 to 0xFFFFFF, or 16 megabytes, because we do not know where the end of the file is yet. Once I remove the limit used for this test the requested hole will extend out 65k.
I loaded another fragment of the file at offset 0x2dc, which is byte 732:
FILE> From:PFS3-11 to QST-1 Ctrl: 3 Type: UI Res PID: bb FLG: 2 FILE: 344 TYPE: 0 OFF: 2dc CRC: 43cb
If I request the file again, the ground station sends this request:
FILE REQ: FLG: 12 FILE: 344 BLK_SIZE: f4 Hole 1: 0 244 Hole 2: 488 244 Hole 3: 976 512 ...
Which looks good! Now we have 3 holes, but the second hole has been updated to run from 488 to 488 + 244 = 732. Just as we expect. If we load all of the fragments of the file then the holes close out as expected. So it seems to work.
This is in Version 0.38.8 right now, and you can download it from here.. If you get a chance to test this version let me know.
73
Chris
g0kla/ac2Cz
Enter Comments Here:
Copyright 2001-2021 Chris Thompson
Send me an email