This document describes a custom upload/download protocol that was designed for use with the ACE-128/64 system and is freely available to anyone who wants it (well, when I finish with the Release #14 of ACE). While this protocol non-standard, it blows the doors off of all other protocols available for Commodore computers, even though it uses a simple "stop-and-wait" acknowledgement scheme. There are two reasons for its speed: the fast device drivers available with ACE, and its large packet size, up to about 18K (although this could be significantly larger is ACE's memory usage were reorganized).
The name of the protocol is "Craig's File eXchange Protocol", or just "FX" for short. It is "file exchange" rather than "upload" or "download" because you will use the same activation of the program to both upload and download all of the files you name.
FX is an external program from the terminal program, so (for now) to activate FX, you have to exit from the terminal program and enter the FX command line, exchange the files, and then re-enter the terminal program from the command line.
When you run FX, you will activate the Server program first on your Unix host and then exit the terminal program and run the Client program on your Commodore. You run the command "fx" on both the client and server machines, which may be a little confusing (but I think you'll get used to it), and name the files that you want to have transferred as arguments to the command on the machine that you want to transfer the files FROM. The usage of the "fx" command is as follows:
fx [-dlvV7] [-m maximums] [-f argfile] [[-b] binfile ...] [-t textfile ...]
-d = debug mode
-l = write to log file ("fx.log")
-v = verbose log/debug mode
-V = extremely verbose log/debug mode
-7 = use seven-bit encoding
-m = set maximum packet sizes; maximums = ulbin/ultxt/dlbin/dltxt (bytes)
-f = take arguments one-per-line from given argfile
-b = binary files prefix
-t = text files prefix
-help = help
well, for the server, anyway. The client program doesn't have the more
exotic options. The "-d", "-l", "-v", and "-V" options are available only
on the Server program, and are for debugging purposes only.The "-7" option tells the protocol to use only 7-bit data. I.e., it tells it to not use the 8th bit position in the data is transmitted. This is useful if you are forced into the humiliation of only being able to use a 7-bit channel to your Unix host. You need only need to give this option on either the client or the host command line and the other side will be informed. It may be useful to create an alias for this command with all of your options set to what you want them to be.
The protocol has the capacity to use different packet sizes for four types of file-transfer situations: uploading binary data, uploading text, downloading binary data, and downloading text. These are useful distinctions, since your host may or may not be able to handle the larger packet sizes without losing bytes (your Commodore, of course, can handle the larger packet sizes with no problems).
In determining which packet size to use for a file transfer (where the type of transfer is known), the protocol finds that largest packet size that both the client and the server can handle and then take the minimum of these two values. The defaults for the client are all the same: the maximum amount of program-area memory that it can use, about 18K. For the server program, I have programmed in default maximum uploading packet sizes of 1K and maximum downloading packet sizes of 64K-1. You can change these defaults in the C program easily by changing some "#define"s.
The "-m" option allows you to manually set the default packet sizes for a transfer. The argument following the "-m" flag should have four numbers with slashes between them, which give the maximum ulbin/ultxt/dlbin/dltxt packet sizes, respectively. Note that the packet sizes only include the size of the user data encoded into packets and not the control or quoting information (below).
The "-f" option on the server allows you to read arguments from a file rather than the command line. This is useful if want to generate and edit the list of files to download before you run the FX command. It's also useful if you don't want other users to see the names of the files that you are downloading. The name of the file comes in the first argument following the "-f" flag and the arguments are put into this file one-per-line. You can put in "-" options in addition to filenames if you wish (like "-t" and "-b"). This option is not supported on the client program.
Finally come the "-b", "-t", and filename arguments. The "-b" argument tells FX that all of the following filenames (until the next "-t" option) are binary files and the "-t" argument says that the following filenames are all of text files. You can use as many "-b" and "-t" arguments as you want. If you don't use any, then all of the files you name will be assumed to be binary files.
For each filename you give on a command line, that file will be transferred from that machine to the other machine. On both Unix and ACE, you can use wildcards in your filenames, of course, to transfer groups of files.
The client program controls the file exchange, and it uploads all of its files first and then asks the server if the server has any files to be downloaded. When the exchange is completed, both the client and server FX programs will exit and you will find yourself back on the command lines in both environments. Re-enter the terminal program to continue with your online session. If something goes very wrong during a transfer or if you decide that you don't really want to transfer any files after activating the server program, you can type three Ctrl-X's to abort the server. This is the same as for the X-modem protocol.
We want the protocol to understand whether a file is text or binary data and to translate them appropriately during downloading. We want the protocol to be aware of filenames, dates, permissions, and we do not want our file contents to get mangled like they do with X-modem (it pads them with Ctrl-Z's, since it was designed for CP/M), and we want it to translate to/from PETSCII if the file is text. We will require that the user tell us whether the file is binary or text (although we may be able to statistically determine this from snooping through the file), and we will use a "canonical form" for encoding the text data during transfer. A convenient canonical form to use is Unix-ASCII (ASCII-LF).
We want our protocol to be simultaneously simple and fast. To make it simple, we will use a stop-and-wait acknowledgement scheme. This means that after each packet is uploaded or downloaded, the transfer will pause and wait for the receiving host to acknowledge that the packet has been transferred correctly, and only then will the protocol continue to transfer more data.
In fact, this scheme fits well with the Commodore hardware, since it is not possible to send or receive serial data while doing disk I/O (in the general case), so we would have to stop listening anyway; the protocol makes it so that there will be no bytes that we end up ignoring while doing I/O.
To make the protocol be fast even though we are using a stop-and-wait acknowledgement scheme, we will use the largest data-packet sizes that we possibly can. In the (current) ACE environment, this means about 18K. This will maximize the amount of time of transferring data over the modem between pauses to do I/O. If the I/O is to the ACE ramdisk, then the length of this pause will be very short and we will achieve a very high link utilization. (The ACE ramdisk can process an 18K read/write request in about 20 milliseconds on a Fast-mode C128 using an REU --- RAMDOS in the same environment would require about 9 _seconds_ (450x slower)).
To allow for future use with other platforms, we will make the protocol define the packet sizes using 32-bit fields. There isn't much data overhead, and this allows us to change implementations to be able to transfer entire files in one large packet. Also, the size of an individual packet should be flexible: be from one to N bytes. This eliminates the X-modem padding problem and the Y-modem crufty hack of using the small packet size when less than 1K of user data remains to be transferred.
We also want our data to be well protected against corruption. Detecting transmission errors efficiently on Commodore computers is already a well solved problem: we will use a table-driven CRC-32 algorithm, the same one that ZMODEM, PKZIP, and CRC32 use. To hide the computation costs of the CRC even more (the cost is very low anyway), we will compute it WHILE sending or receiving packets. Oh, actually, I guess that I forgot to mention an a-prior design decision: we will be using a packet-oriented approach for transferring data (described below); packetization offers so many advantages that this decision is really a no-brainer.
Also, to make the process interaction as straightforward as possible, we want to use the Client/Server programming paradigm. This paradigm combines well with the stop-and-wait acknowledgement scheme to produce a Remote Procedure Call (RPC) type of interaction between the machines. For those not familiar with this Interprocess Communication (IPC) scheme, you can read a couple issues of C= hacking ago where I talked about it for use with a multitasking operation system. RPC is a very useful, powerful, simple, and widely applicable IPC scheme.
To recover from packet corruption, we will be using a timeout+retransmission scheme, and to be consistent with the RPC scheme, the client will do all timeouts and retransmissions. This means that after sending a request RPC packet out, if we don't receive the reply within a certain period of time, we will timeout and send the request again. Or, to be more precise, since we will be working with large packet sizes, we will timeout if we don't receive any bytes from the server for a certain period of time, say 5 seconds, while we are expecting more bytes from him.
The way that corrupted packets are dealt with is very simple: they are ignored. The server could possibly send back a negative acknowledgement, but we won't try that for now.
In order to make retransmissions work out correctly, we will be using sequence numbers and internal-state variables inside of the server to insure that requests aren't carried out more than once. We need these mechanisms because when an RPC fails, we won't know if we got no response because the original request was lost and the operations wasn't carried out, or whether the request was received and carried out but the reply message was lost.
For example, if we request that packet #123 be downloaded and the server carries out that request but the reply message is lost, then the client will time out and retransmit the request. The server remembers the last request number that the client sent it (123 here), so if the client asks for packet #123 again, the server will simply retransmit the reply that it gave last time. If, on the other hand, the client were to request packet #124 (or simply "not 123"), then the server reads the next chunk of data from the file and sends it as the reply. Our protocol will use an 8-bit sequence number even though it only needs a 1-bit sequence number (since eight bits will allow for the future expansion of having multiple requests being processed concurrently: asynchronous RPC).
We also want to be able to both upload and download as conveniently as possible. To me, this means doing both operations by calling only one command (as described in the previous section). This arrangement also allows for the future expansion of uploading and downloading files simultaneously (the protocol as designed places no restrictions on this possibility).
We also want to make use of an eight-bit clean link between the Unix host and your Commodore, but this may not always be possible. Sometimes you may have only a 7-bit connection, and even if you do have an 8-bit connection, there may still be some software-flow-control problems with intermediate devices between your Commodore and your Unix host. So, we want our protocol to not make use of the X-on and X-off characters, and to use only 7-bit characters if it cannot use eight. The way to achieve this is called "escaping", "quoting" or "byte stuffing", and will be discussed in the next section. It turns out that supporting 7-bit characters is pretty simple and the mechanism is required by other aspects of the packetization.
There, that should take care of most of the major design decisions.
Packets also integrate well with many IPC schemes, including Remote Procedure Calls. In fact, you end up emulating a packet-oriented scheme even if you are using RPC over a stream-oriented transport system. Packets also take into account the limited buffering capacity of both end systems and intermediate systems, and allow for the convenient implementation of flow control (even if said flow control consists of simply dropping packets on the floor). Packets are very useful things indeed! And just think that back in the early 1970s packets were dismissed as being infeasible and unusable.
Each packet used in the FX system has four parts to it: the start character, the user data (payload), the error-check characters, and the end character. Graphically, a packet has the following format:
+------------------------+-----------+--------------+----------------------+ | Start-of-packet Char | Payload | ErrorCheck | End-of-packet Char | +------------------------+-----------+--------------+----------------------+The payload can be arbitrarily long, up to whatever limit the two computers involved in the transfer can handle.
The error check is a 32-bit (4-byte) Cyclic-Redundancy-Check value that occupies the last four bytes before the End-of-packet character. The implementation, which is based on a table-lookup method, is so efficient that it is as fast as a simple add-up checksum, except much more reliable. Using this error check, there will be approximately a one-in-4,000,000,000 chance that a packet with an error in it will be accepted has being error-free. These are pretty good odds for our purposes. The CRC is calculated exclusively on the raw payload data.
The following special characters used by packets are defined:
NAME HEX DEC Control Meaning --------- ---- --- ------- -------- CHR_START 0x01 1 Ctrl-A Packet-start indicator CHR_END 0x19 25 Ctrl-Y Packet-end indicator CHR_ESC 0x05 5 Ctrl-E Escape character for next code CHR_ABORT 0x18 24 Ctrl-X Abort transfer if repeated three times CHR_XON 0x11 17 Ctrl-Q Software flow-start: avoided CHR_XOFF 0x13 19 Ctrl-S Software flow-stop: avoided CHR_QUOTE8 0x14 20 Ctrl-T Quote-8 the next 7-bit sequenceCHR_START is used to signify the start of a new packet. This character is not allowed to be used anywhere else for any other purpose.
CHR_END is used to signify the end of the current packet, and cannot be used anywhere else. The reason for using special characters to mark the beginning and the ending of a packet is to allow for easy error recovery after a communication failure. All you do is search for the next CHR_START character after you toss away a garbled packet and you're back in business. I am unaware of any reasonable alternatives to framing packets with a CHR_START character. Using a CHR_END special character is a convenience.
CHR_ESC is used to "escape" the next character. Since there are special character codes that cannot be used in any other way than their intended function (including CHR_START and CHR_ESC itself), this character is needed. The character following the CHR_ESC character must be between "@" and "_" (0x40 and 0x5f) in the ASCII chart, or be the character "?" (0x3f). The character following the CHR_ESC is then "and"ed with the value 0x1f to mask off the "letter" bits and turn it into a control character in the range of 0x00 to 0x1f (the same range as the special control characters) and the "escape sequence" is treated as a single character of user data. If the character following the CHR_ESC is a "?", then a code of 0x7f is interpreted instead. Using a character following the escape that is different from the character being represented allows for greater resiliance of the protocol in the presence of bits being garbled or bytes being dropped. All special characters in a packet except for the starting and ending characters are escaped as described above.
CHR_ABORT can be typed by the user into a terminal program at any time to shut down the server.
CHR_XON and CHR_XOFF can cause problems with intermediate devices on some systems, so the FX protocol does not use these character codes at all; it purposely avoids them and uses escape sequences (CHR_ESC) for them instead.
CHR_QUOTE8 is used to re-generate 8-bit data over a 7-bit link. Kermit uses this same technique. When this character is encountered in the receive stream, the next character is extracted and is "or"ed with a value of 0x80 to give it a "1" in the high-bit position. The CHR_QUOTE8 character can also be followed by a CHR_ESC code, which is interpreted as above and then "or"ed with the 0x80 value.
One of the disadvantages of using this scheme is that each byte in the range of 0x80 and 0xff takes at least two bytes to transmit and some of them three. If fact, for many binary files it may be faster to uuencode the file and transfer the resulting text, since uucode has a static encoding overhead of 33% whereas this quoting scheme has an expected overhead of 50% (plus the CHR_ESC overhead). Of course, this feature is intended to be used as a last resort if you cannot get an 8-bit connection.
So there you have it. Every message sent between the client and the server is encapsulated in a packet as specified above. Packetization allows for convenient error detection and recovery and works well with our interprocess communication scheme.
One implementation note about the packetization has to do with buffering. On the Unix host, it is advantageous to encode a packet into a memory buffer and then send out that buffer in a single "write" operation. This less operating- system overhead (which may or may not be significant) but more importantly, it means that the packet will be sent between intermediate communication devices as efficiently as possible. On my local Unix system, I connect to a terminal server and to my Unix host through that. Performing single-byte writes on the Unix host means that the bytes are sent in individual Ethernet packets between the Unix host and the terminal server, and encounter more overhead and communication delays. When I changed the program to send the FX packet in a single operation, a significant performance gain was realized.
For receiving data on the Unix host, there isn't much you can do other than reading one byte at a time, since the receiver doesn't know when a packet is going to end. However, the same problem is not encountered here that was encountered with sending data because data that is received by the Unix host but not "read" by the user program are buffered and collected, smoothing out the system overhead, which is insignificant compared to the modem speed. The Unix program used the "stdin" and "stdout" file streams for receiving and transmitting data, and sets the tty driver to turn off all line-editing features to get at the raw bytes.
On the Commodore end, it is advantageous to read data from the modem driver in chunks, since the system overhead is significant compared to the modem speed. These are small computers that we are driving to the max, you know. Data is read from the modem in chunks of up to 255 bytes (whatever is available at the time) and processed a byte at a time from the read buffer. The CRC is calculated during processing, to avoid doing this on the critical path. The CRC calculation is performed as an operation by itself since the overhead is very small on fast processors. The character-set translation for text files will be performed on the critical path (on the Commodore) since it is more convenient to do it at a higher layer in the IPC scheme. The packet-handling software is logically at a distinct layer that doesn't have to worry about higher layers. The next layer up is logically the RPC layer and then the file-transfer layer.
There are eight request/ack interactions that are defined for the protocol: two for connection management, three for uploading files, and three for downloading files. The client is in charge of the file-exchange session and of the error handling.
OFF SIZ DESC
--- --- -----
0 1 code: REQ_CONNECT ('C')
1 1 protocol version := 0x01
2 1 transmit byte size: '7' or '8' bits
3 - SIZE
This is what gets put into the the "payload" portion of the packet. All of
the messages used in the protocol have an ASCII letter in the first byte that
identifies what the message type is. Each request has an uppercase letter and
each acknowledgement has the corresponding lowercase letter.The connection-request message is fairly simple: it includes the protocol version number and the number of bits wide that the client thinks that the communication channel is. The version number is currently always 0x01 and is included for cross-compatibility with future versions of the protocol. The channel width is encoded into either a '7' or an '8' ASCII character. The client will think that the channel width is seven bits only if you tell it this on the command line.
When the server receives the connection request, it replies with the following message:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_CONNECT ('c')
1 1 protocol version := 0x01
2 1 transmit byte size: '7' or '8' bits
3 1 recommended request byte size: '7' or '8' bits
4 4 server maximum text-upload data size: H/M/M/L word
8 4 server maximum binary-upload data size: H/M/M/L word
12 4 server maximum text-download data size: H/M/M/L word
16 4 server maximum binary-download data size: H/M/M/L word
20 - SIZE
The "protocol version" is what the server is using, currently always 0x01.
The "transmit byte size" is the size that the user has specified on the
command line that activated the server, and the "recommended request byte
size" is a '7' if either the "transmit byte size" of the either the client or
server is seven bits, or '8' otherwise. This is what should be used for the
all subsequent messages that are exchanged.The server's reply also includes the maximum packet sizes that it can handle for uploading and downloading binary and text files. The client then takes the "min" of the server's maximum packet sizes and its own, and uses the resulting maximum packet sizes for the rest of the file exchange session. The maximum packet sizes in the server's reply are all 32-bit unsigned integers that are stored from most-significant to least-significant bytes (big endian order). I picked big-endian order because that is the order used most commonly in inter-machine protocols.
The reason that the client doesn't have to inform the server of the client's maximum packet sizes in its connection message is that the maximum packet size to use is included with each request to get the next packet of a download file. It is sufficient that the client knows the full max-packet information. Really, the "transmit byte size" field isn't needed in the server reply message either, but I wanted the packet-size fields to be size-aligned.
After all of the file exchanging is completed, the client sends the following message to terminate the connection and return the server back to its command- line mode:
OFF SIZ DESC
--- --- -----
0 1 code: REQ_DISCONNECT ('Q')
1 - SIZE
When the server receives this request, it replies with:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_DISCONNECT ('q')
1 - SIZE
And then exits like it should. Note that once the server exits, it cannot
accept any more packets, since they would be sent to whatever command shell
you use on your Unix system, and wouldn't do anything useful, so if the client
sends the disconnect message but doesn't receive any reply, it will time out
and tell the user that it couldn't disconnect cleanly from the server. This
should be a rare occurrence. Anyway, what the user would do then is re-enter
his terminal program and send Ctrl-X's at the server until it exits like it
should have.This arrangement allows us to avoid the famous(?) "two armies" problem that is inherent in disconnecting two connected processes: there is no "clean" way to do it. What systems like Z-Modem and Berkeley Sockets do is to have the server wait for a period of time that is longer than N times the timeout period of the client so that if there is a retransmission of the disconnection request, it likely that it will be received and processed correctly by the server. This is the reason (presumably) that Z-Modem does an annoying pause of 15 seconds or so after you finish transferring files. I think that my solution is much nicer, since the server can exit immediately (even though my server delays for 1 second, just so that your shell prompt will be cleanly in your modem's ARQ buffer when you re-enter your terminal program, if you have a hardware-flow-control modem).
When the client decides that it wants to upload a file, it first informs the server about this by sending the following message:
OFF SIZ DESC
--- --- -----
0 1 code: REQ_UPLOAD_OPEN ('U')
1 1 data type: 't'=text file, 'b'=binary file: 'd'=directory
2 4 estimated file size: H/M/M/L word
6 2 permissions ("-----sgr:wxrwxrwx"), like Unix, H:L
8 12 modified date: BCD format:
20 n filename, null-terminated
20+n - SIZE
The "data type" field tells whether a text or binary file will be uploaded.
There is a provision for "uploading" a directory entry (as part of uploading
and downloading entire directory hierarchies), but support for this is not
implemented yet. Also, it makes no difference to a Unix system whether a file
contains text or binary data, but it may make a difference to other operating
systems (like Mess-DOS). The "estimated file size" field isn't really used
either, but it allows the server to make intelligent decisions about
pre-allocating space, buffering, etc., if it needed to. However, it is
currently not filled in by the client, since file-size information is
difficult to extract from Commodore-DOS. The file size is an unsigned 32-bit
quantity.The permissions field is currently not supported by the server, but it is intended to allow file permissions to be preserved when passing files from one system to another. The interpretation of the 16 bits of this field is like it is with the Unix operating system: "rwx" bits for the owner, group, and other, and execute-as-owner, execute-as-group bits. The owner-id and group-id fields aren't included since they are generally not portable across systems, and even if they were, we usually want to receive files as our own owner-id and our own group-id.
The "modification date" field is not currently filled in either, since this information is even harder to come across with Commodore-DOS, but when it is, it will have a 12-byte BCD format. The "YY:YY:MM:DD:hh:mm:ss" sub-fields should be easy enough to figure out, and the "tt:t" fields contain thousandths of seconds. The "w" field contains the day of the week, coded as 0-6 for Sunday to Saturday, and 7 for "unknown". The "GG:gg" fields contain the number of hours and minutes that your time zone is off from GMT. If the number is negative (in the western hemisphere), then the regular positive number of hours will be used, execept that the 0x80 bit of the hours byte will be set. Finally, the "aa" sub-field is used to encode the accuracy of the timestamp. The way that it is interpreted is that the time value is accurate to plus/minus 2^aa milliseconds. For example, if my clock were accurate to within one second, then this field would be set to 10 in BCD (2^10 == 1024ms). A value of 99 means "unknown" (or that the clock could be off by many billions of billions of years).
I decided to go all out in defining the date field so that it will be useful in the future when "world consciousness" will be much more important than it is today.
And last but certainly not least, the filename is encoded in ASCII with a trailing zero byte.
Upon receiving this request, the server will attempt to create a file according to your specifications, and will send back a reply of the form:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_UPLOAD_OPEN ('u')
1 1 error code: 'y'=successful, 'n'=open unsuccessful
2 - SIZE
The "error code" field tells whether the open operation was successful or
not. If it was, then the client can continue with uploading its file; if not,
then that file cannot be uploaded (and that the upload channel doesn't need to
be closed). It's up to the client whether to go on to the next file, abort,
or ask the user for help. The client will currently report an error to the
user and then go onto the next file. Of course, it's likely that whatever
caused the error in creating the current file will also cause an error in
creating subsequent files (insufficient access permissions on the current
directory, disk full, etc.). The server will overwrite any existing file
with the same name (since asking permission, etc., would require extra
mechanism, and would probably be a nuisance anyway).If the upload channel is opened successfully, then the packets of upload data should be sent to the server one at a time, until all of the data is uploaded. The client sends the following message to the server to upload a packet of data:
OFF SIZ DESC
--- --- -----
0 1 code; REQ_UPLOAD_PACKET ('R')
1 1 upload sequence number
2 4 data length: H/M/M/L word
6 n data
6+n - SIZE
The "upload sequence number", which was described before, is used to make sure
that retransmissions of packets are detected and handled properly, so that
each packet of data only appears in the file once. The "data length" field
tells the number of user data bytes that follow in the packet, and then the
actual user data bytes appear. The "data length" field is actually redundant,
but I figured that it would make programming a little easier, and allows
additional error checking. Normally, each upload-data packet will contain
the maximum-packet-size number of bytes of user data (according to whether
text or binary data is being uploaded), except for the last packet, which
will contain the number of data bytes that are left in the file. However,
each packet is allowed to contain anywhere from 1 to the maximum-packet-
size number of bytes: whatever the client wishes to use. Variable-sized
packets are a Good Thing (TM, Pat. Pend.). You will note that the data-
size values are also what will be used for the "read" and "write" system
calls on the client and server, respectively. I/O will be done in big,
efficient chunks.Upon receiving each upload packet, the server replies with the following acknowledgement message:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_UPLOAD_PACKET ('r')
1 1 upload sequence number
2 - SIZE
I don't think that the "sequence number" field is actually necessary here, but
it is included to allow for future expansion and to provide redundancy for
protocol-error checking.When the client has uploaded all of the packets of the file currently being uploaded, it then sends the following message:
OFF SIZ DESC
--- --- -----
0 1 code: REQ_UPLOAD_CLOSE ('V')
1 - SIZE
This will close the upload channel and will finish writing the uploaded file
to the Unix file system. The server will then respond with the following
message to acknowledge the request:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_UPLOAD_CLOSE ('v')
1 4 number of bytes uploaded: H/M/M/L word
5 - SIZE
The "number of bytes" field is actually redundant, but is used for additional
error checking.
To open the download channel, the client sends the following request to the server:
OFF SIZ DESC
--- --- -----
0 1 code: REQ_DOWNLOAD_OPEN ('D')
1 - SIZE
To which the server replies with:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_DOWNLOAD_OPEN ('d')
1 1 data type: '0'=no more files (eom),'t'=text,'b'=bin,'e'=err,'d'=dir
2 4 estimated file size: H/M/M/L word
6 2 permissions ("-----sgr:wxrwxrwx"), like Unix, H:L
8 12 modified date: BCD format:
20 n filename, null-terminated
20+n - SIZE
The file information is the same as for opening an upload file, except that
there are more possible return conditions, and all of the "meta data" fields
are actually filled in by the Unix host (since this information is actually
conveniently available via the "stat" system call).If the server replies with a '0' "data type" code, then this means that the server has no more files to offer for downloading. The filenames to download are taken one at a time, from left to right, from the command line that was used to start the server. When the server runs out, then the downloading session is complete and the client disconnects (since the client uploads its files first).
Alternatively, the server could reply with a 'e' code, which means that it could not open the next filename given on its command line. An error return is generated so that the client can inform the user that the file could not be downloaded. This will normally result from the user giving a bad filename on the command line. The client will continue the downloading process by closing the download channel (below) asking for the next file by re-opening the download channel. The download channel needs to be closed on this condition since otherwise there would be no way of distinguishing retransmissions from new requests at the server.
Finally, the server can reply with a 't' or 'b' code ('d' for directories is not currently implemented) indicating that the file was correctly opened and is either text or binary (as specified on the server's command line). Of the meta information about the file, only the filename and file size are currently used: the file is named according to the given name, translated to PETSCII and truncated to 16 characters, and the file size is reported to the user so that he can monitor downloading progress. I am not sure what to do yet about name collisions on the Commodore end: either ask the user whether to overwrite the file, automatically overwrite the file anyway, or automatically give the file a slightly different name and download normally. I think that for the time being, I will just overwrite the existing file. This will mean that you'll want to be extra careful in putting the filenames onto the correct command line (the client's or the server's), although there won't be a problem if the file doesn't exist on the machine whose command line you put the name on.
When the file handling is all squared away and the download channel is opened, the client then sucks packets out of the file until the end of the file is reached. The packets are sucked out with the following request:
OFF SIZ DESC
--- --- -----
0 1 code: REQ_DOWNLOAD_PACKET ('S')
1 1 download sequence number
2 4 maximum acceptable data length: H/M/M/L word
6 - SIZE
The "download sequence number" is used to distinguish retransmissions from
requests for new packets, and the client tells the server the "maximum
acceptable data length" for the reply packet. Although the max-packet
information is actually static during the connection, I included it here in
every "read" request since I didn't really want the server to keep that
particular bit of "state" internally.The server replies to the download-packet request with the following message:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_DOWNLOAD_PACKET ('s')
1 1 download sequence number
2 4 data length: H/M/M/L word, 0==EOF
6 n data
6+n - SIZE
This is the only "large" message that the server can produce. It includes the
sequence number, the number of bytes that are actually included, and the user
data. The number of data bytes in the packet is allowed to be smaller than
the number of bytes requested, but this is normally only the case for the last
packet of the file.To indicate that the end of file has been reached and that no more user data is available, the server will return a download packet with zero bytes of user data in it. Upon receiving this, the client will close the download channel with the following message:
OFF SIZ DESC
--- --- -----
0 1 code: REQ_DOWNLOAD_CLOSE ('E')
1 - SIZE
And the server will reply with:
OFF SIZ DESC
--- --- -----
0 1 code: ACK_DOWNLOAD_CLOSE ('e')
1 4 number of file bytes downloaded: H/M/M/L word
5 - SIZE
The "number of file bytes downloaded" field is redundant but included for
additional error checking. After closing a file, the client will then ask
for the next file, or will disconnect if the last file to download was just
closed.
As pointed out way above, there are two possible reasons for a retransmission being needed: either the request packet was corrupted and dropped, or the reply packet was corrupted and dropped. In the format case, the request wasn't processed by the server, but in the latter case, it was. Since we don't want the server to perform an file operation twice (this is really what the six file-transfer client operations really boil down to from the server's perspective), the server must keep four pieces of internal state: the last upload sequence number, the last download sequence number, whether the upload file is open, and whether the download file is open.
If an upload-open request is received and the file to be uploaded is not open, the the request must be a new one and the server processes it and sends back a reply like normal. If an upload-open request is receive and the upload file IS currently open, then it must be the case that the current request is a retransmission, so all theat the server needs to do is to give a positive reply without performing any internal file operations. The same holds true for the download-open call and for both of the close calls (except that the operation has already been processed if the file is CLOSED).
For the packet-upload and packet-download requests, sequence numbers are used to detect duplicates. You will note that these sequence numbers are distinct from one another, and, in fact, that the entire upload and download file- transfer channels are distinct and independent from each another. This is to allow for the future possibility of simultaneous file uploading and downloading. In fact, if stream numbers (file descriptors) were added to the open/read/write/close requests, then we could have us a full-blown remote-host over-the-phone interactive file server. But anywho, sequence numbers start from 0x00 for the first packet transferred and increment modulo 256 from there.
Note that for high-speed data-compression modems (like I have) that already include error detection and recovery at a level hidden from the user, the FX protocol will work particularly well: there will never be an error, never be a timeout delay, and never be a retransmission. And, really, the CRC-32 error computation and checking is pretty much a zero cost. But, if something does go wrong, outside of the modem-to-modem connection, the FX protocol is right there to pick up the pieces and carry on.
Here is my performance testing so far, using my USR Sportster modem over a 14.4-kbps phone connection, with a 38.4-kbps link to my modem from my C128, to my usual Unix host:
Using FX to/from the ACE ramdisk, REU:
Download 156,260 bytes of ~text: time= 54.1 sec, rate=2888 cps. Download 151,267 bytes of tabular text: time= 45.9 sec, rate=3296 cps. Download 141,299 bytes of JPEG image: time= 92.5 sec, rate=1528 cps. Upload 156,260 bytes of ~text: time= 57.4 sec, rate=2722 cps. Upload 151,267 bytes of tabular text: time= 45.3 sec, rate=3339 cps. Upload 141,299 bytes of JPEG image: time= 95.0 sec, rate=1487 cps.Using FX to/from my CMD Hard Drive:
Download 156,260 bytes of ~text: time= 83.4 sec, rate=1874 cps. Download 151,267 bytes of tabular text: time= 75.4 sec, rate=2006 cps. Download 141,299 bytes of JPEG image: time=118.2 sec, rate=1195 cps. Upload 156,260 bytes of ~text: time= 77.9 sec, rate=2006 cps. Upload 151,267 bytes of tabular text: time= 66.2 sec, rate=2285 cps. Upload 141,299 bytes of JPEG image: time=114.2 sec, rate=1237 cps.Using DesTerm-128 v2.00 to/from my CMD Hard Drive, Y-Modem:
Download 156,260 bytes of ~text: time=189.5 sec, rate= 824 cps. Download 151,267 bytes of tabular text: time=180.4 sec, rate= 839 cps. Download 141,299 bytes of JPEG image: time=199.9 sec, rate= 707 cps. Upload 156,260 bytes of ~text: time=255.1 sec, rate= 611 cps. Upload 151,267 bytes of tabular text: time=238.6 sec, rate= 634 cps. Upload 141,299 bytes of JPEG image: time=233.0 sec, rate= 606 cps.Using NovaTerm-64 v9.5 to my CMD Hard Drive, Z-Modem, C64 mode:
Download 156,260 bytes of ~text: time=245.8 sec, rate= 636 cps. Download 151,267 bytes of tabular text: time=230.0 sec, rate= 658 cps. Download 141,299 bytes of JPEG image: time=262.6 sec, rate= 538 cps.(There is no Z-Modem uploading support)
So there you have it: my simple protocol blows the others away. QED.