PacSat files with too many holes?

2021-Feb-27

Hans (BX2ABT) reported that a large file could not be downloaded from FalconSat-3 to his QTH. He was requesting file 4abd (size: 196834 bytes). So large is relative to large in the early history of computing :) He hypothesized in this github issue that:
"... the number of holes being requested to be filled is so large that that FS3 cannot read the incoming data stream because it is too long ... The result is that FS3 doesn't respond to these download requests at all and subsequently the file cannot be finished downloading."

I was aware of this issue when writing the code it seems, because I added a comment that says in all caps "THIS WILL BLOW UP IF THE HOLES LIST IS TOO LONG", but then I never got around to implementing any code to handle it. Oops....

File Holes not as nasty as Directory Holes

I've previously dealt with the difficulties of directory holes, which are best explained in the PACSAT BroardCast Protocol Update 2. But file holes are slightly different and potentially easier to understand. They are described in the PACSAT Broadcast protocol where it explains how to request the mising parts of a file. In this case a Request Frame is sent to the spacecraft. The requests fits wholly inside a single UI frame and has a PID (the AX25 protocol indicator) of 0xBB. The data section of the UI frame contains a short header with some flags and the file id, followed by a list of holes in the file. The request header has this structure:

struct REQUEST_HEADER {
          char flags;
          unsigned long file_id;
          int block_size;
          };

The flags are used to indicate if this is a request to start transmitting a file for the first time or if it is a hole list. There are full details in Section A.3 "File Retransmission Request Format", which is almost the final section of the PACSAT Broadcast protocol.

The holes themselves are each 5 bytes long with a 3 byte file offset and a 2 byte length. It is simply a list of file offsets and the corresponding number of bytes that are missing starting at that address offset. The algorithm to calculate the holes is actually the same algorithm for calculating the holes in internet packets and so I followed the algorithm for IP Packet re-assembly RFC 815 to implement this is my code.

The issue with File 4ABD

I think Hans is likely right given I did not implement a check on the length of the holes list. But I think it is a rare situation. It probably (I hope) has not impacted too many people. The issue is not purely based on file size. If you receive one block in the middle of a large file then you still only have 2 holes. One hole for all the data before the block you received and one hole for all the data after that to the end of the file. So you need to have received a lot of fragments spread across the file to get a lot of holes. But clearly it is possible.

Fixing the issue

The protocols are a bit ambiguous, but I understand that packet length is limited to 255 bytes on the spacecraft. I had previously calculated that the maximum data length in the UI frame was 244 bytes and that 48 holes could fit, as each hole is 5 bytes. But I don't know how I calculated it. So I re-calculated that to be 45 holes as follows:

  If we assume 255 bytes is the maximum length of the packet then:

  - 3 bytes are consumed by the the KISS format
  - UI frame has 16 byte header with a flag at each end and 2 byte CRC - 20 total consumed bytes
  - So we are left with 255 -3 - 20 = 232 for data.
  - We then have the File Request header which is 7 bytes (for the flags, fileId, block_size) leaving 225
  => So 225 / 5 = 45

With that calculated I have now set the limit at 45 holes and now check that we are sending less than that. That is in version 0.38.6 and later.

But is hole size more likely the issue?

There could be another related issue though. Each hole is 5 bytes long, consisting of the file offset (3 bytes) and the length of the hole (2 bytes). So built into the protocol we can't support files longer than 24 bits - which is fine as that is 16MB - or holes that are longer than 16 bits, which is only 65k. This is an issue.

So it is entirely possible that the ground station can get a block or two in the middle of the 196,834 byte 4ABD file and the holes were then longer than 65k. That means we need to limit the size of the holes that the ground station requests too. Given the file chunks transmitted down have a maximum size of 244 bytes, I don't think it matters if we have one hole that is 100k long covering most of the file or that there are two holes each of 50k long. It is going to come down in a lot of small pieces either way.

Instead of splitting large holes in half I decided just to truncate large holes to 65k. This means that the packet re-assembly algorithm can remain unchanged. That code was complex to test and is absolutely core to the downloading of files. So I prefer not to modify it. I think truncating the hole lengths will work fine as we will still gradually cover all of the holes in the file.

This is in Version 0.38.6 right now, and you can download it from here if you want to test it.. If you get a chance to test this version let me know.

PacSat files with too many holes?

2021-Feb-27

File Holes not as nasty as Directory Holes

The issue with File 4ABD

Fixing the issue

But is hole size more likely the issue?

Comments on this post

Hardware Projects

Software Projects

Other Projects

Workbench Notes

References