The LLM that couldn't find column 72

In HLASM (IBM High Level Assembler) there is a very funky feature that doesn't really exist in modern programming languages and that is the fixed width of the program itself. Because IBM had to maintain backward compatibility with the dawn of computing (including punch-cards) in HLASM you have to identify, with an X, if the line continues to the next line. Failing to do this and running over column 72 causes what is called a "continuation error" and it is the bane of all HLASM programmers.

Especially me because I've never worked properly on a mainframe except recently on one the OCaml team gave me access to but that wasn't a HLASM job so I constantly make that error. When I was working on Z390 which I'm a contributor to I asked an LLM to help me with a continuation error happening in the SEARCH macro PR I did. It was like looking at a half blind drunkard try and do a binary search. It generated about 12 odd bash commands, 2 whole python scripts and about 30 minutes of work (I was running it while the kettle was on) and it still didn't find where column 72 is. Because it doesn't know WHERE column 72 is or WHAT a column is or WHAT the number 72 is.

It's a text generator giving the next plausible word. Its fantastic on what it's trained on (see all the python and typescript projects in existence) and is really bad at what its not (HLASM which is largely proprietary, historical languages, lower level programming and compilers which is increasingly being cannibalised by LLVM and MLIR). Also LLMs cannot "see" like a human can, I noticed the continuation error in about 10 seconds after I put my mug down (earl grey tea if you're curious) but the LLM could not. This is because an LLM sees a block of characters (sometimes called tokens I believe) which is why for the longest time no LLM could correctly answer "how many R's are in the word Strawberry", it had to be hardcoded on.

So largely I still have to do what I have to do. LLMs have structural limitations to what they can do, especially in the space I work in.

Where I do and don't use them, project by project, is recorded in bits and bobs, alongside the raw output it came from.