When I found the Crystal Castles Source code I thought it would be cool to able to re-assemble the source so modifications could be made to the game from the original source. I didn't take long to realize that the assembler used by Atari had a lot of differences from assemblers I was familiar with. I started Googling some of the unusual pseudo ops in the source to see if I could identify an assembler that was compatible with this code. I found manuals for similar assemblers but nothing was a perfect match.
The best resource I found was the VAX MACRO and Instruction Set Reference Manual:
https://www.ece.lsu.edu/ee4720/doc/vax.pdf
It is not surprising that this assembler is close, Atari did a lot of their arcade development on DEC VAX mainframes. This became my go-to document for writing my assembler. If there was anything that was not clear from the source, but was answered in this document, I used what was in this document.
Addressing Modes
One of the strangest differences between this assembler and other common 6502 assemblers is the syntax to define addressing modes. All the modes can be defined with prefixes to the operand, so for example, for absolute X you would do LDA AX,$1000 instead of LDA $1000,X. There are even multiple ways to define the same mode, at it appears different programmers used different ones. Here are the prefixes for the addressing modes:
# or I, - Indirect
ZX, - Explicit zero page X
ZY, - Explicit zero page Y
Z, - Explicit Zero page
NX, - Indirect X
NY, - Indirect Y
AX, - Absolute X
AY, - Absolute Y
A, - Absolute
X, - Absolute or Zero Page X determined by the assembler
Y, - Absolute or Zero Page Y determined by the assembler
There are also two suffixes that can be used, (X) and (Y). You would think that these would be for indirect modes, but they are not, they are for absolute X and Y. I also found "(X" in the code. I assume this is an anomaly of the assembler where it doesn't parse the entire addressing mode identifier.
Radix
The Atari assembler provides two different ways of controlling the radix (base) of numbers. First, the radix can be specified using a prefix on the number ^H for Hex and ^B for Binary. There is probably a prefix for octal but the Crystal Castles code doesn't use it. The second way of specifying radix is to use the .RADIX pseudo op which specifies the default radix to use when parsing numbers. For example if this is in the source:
.RADIX 16
it will set the default radix to hex so that if a number doesn't have an explicit radix prefix it will be interpreted as hex. To avoid confusion with symbols, a hex number must start with a number if it doesn't have a prefix, so for example A5 would have to be written as 0A5. In my version of the assembler I always treat the parameter for the radix pseudo op as a decimal.
.REPT/.ENDR
The .REPT pseudo op looks pretty straight forward but there turned out to be some tricky parts to it. This pseudo-op repeats a block of code a specified number of times during assembly. It is passed one parameter which is the number of times to repeat it.
The first interesting thing I found about it is that the Crystal Castles sources often uses it with a repeat count of zero. The basically works like a block comment, preventing the block of code from being assembled at all.
The other interesting thing I found in the code was the .REPT blocks were sometimes ended using .ENDM, which ends a macro definition, instead of .ENDR. My theory on this is that if a repeat block is started inside a macro definition then it would make no sense for it to extend beyond the end of the definition, so it's possible that the assembler explicitly ends open repeat blocks at the end of a macro definition. This was probably an un-intended side effect of the way the assembler was written.
I will cover some more differences in a latter post.
