Incidentally, I've had the same thing too with Lisps on both o-series and smaller Claude models - always a mismatched paren or two.
clord 6 hours ago [-]
There is something deep in this observation. When I reflect on how I write code, sometimes it’s backwards. Sometimes I start with the data and work back through to the outer functions, unnesting as I go. Sometimes I start with the final return and work back to the inputs. I notice sometimes LLMs should work this way, but can’t. So they end up rewriting from the start.
Makes me wonder if future llms will be composing nonlinear things and be able to work in non-token-order spaces temporarily, or will have a way to map their output back to linear token order. I know nonlinear thinking is common while writing code though. current llms might be hiding a deficit by having a large and perfect context window.
hnuser123456 6 hours ago [-]
Yes, there are already diffusion language models, which start with paragraphs of gibberish and evolve them into a refined response as a whole unit.
altruios 5 hours ago [-]
Right, but that smoothly(ish) resolves all at the same time. That might be sufficient, but it isn't actually replicating the thought process described above. That non-linear thinking is different than diffuse thinking. Resolving in a web around a foundation seems like it would be useful for coding (and other structured thinking, in general).
hansvm 5 hours ago [-]
With enough resolution and appropriately chosen transformation steps, it is equivalent. E.g., the diffusion could focus on one region and then later focus on another, and it's allowed to undo the effort it did in one region. Nothing architecturally prohibits that solution style from emerging.
altruios 4 hours ago [-]
The choice of transformation steps to facilitate this specific diffuse approach seems like a non-trivial problem. It doesn't follow such an organic solution would emerge at all, now, does it?
UltraSane 5 hours ago [-]
I think long term LLMs should directly generate Abstract Syntax Trees. But this is hard now because all the training data is text code.
undfined 3 hours ago [-]
There's a fair amount of experimental work happening trying different parsing and resolution procedures such that the training data reflects an AST and or predicts nodes in an AST as an in-filling capability.
catfacts 19 minutes ago [-]
Do you know if any such experimental work is using a special tokenizer for example in Lisp a special token for left or right parenthesis?
lelanthran 5 hours ago [-]
> Sometimes I start with the final return and work back to the inputs.
Shouldn't be hard to train a coding LLM to do this too by doubling the training time: train the LLM both forwards and backwards across the training data.
jdiff 4 hours ago [-]
GP is talking about the nonlinear way that software engineers think, reason, and write down code. Simply doing the same thing but backwards provides no benefit.
FeepingCreature 6 hours ago [-]
Another example of this is Claude placing unnecessary imports when writing Python, because it's hedge-importing modules that it suspects it might need later.
cenamus 5 hours ago [-]
Is it hedging or did the training data just have lots of unecessary imports?
haiku2077 5 hours ago [-]
Especially in Python, where it can be hard to tell if something is being imported purely for side effects.
0cf8612b2e1e 5 hours ago [-]
That does happen, but not frequently in the common libraries that are going to be in public training data.
Is there a top 100 package that does something funny on import?
semi-extrinsic 1 hours ago [-]
Dunno if it counts as funny, but the following code only works if you keep the matplotlib import:
import matplotlib
import PIL
im = PIL.Image.open("foo.png")
kstrauser 3 hours ago [-]
I'd be surprised. That kind of thing was en vogue for a little while in the early 2000s before cooler heads prevailed, but now people will understandably shout at you for changing behavior in someone else's code.
My guess is that nearly all packages that did this sort of thing were left behind in the 2-to-3 migration, which a lot of us used as the excuse for a clean break.
But I agree that observable side effects are generally pretty rare. And apparently, both libraries are not even in the top 100 packages, depending on how you count. It looks like those spots are all taken by libraries used in uncached, wasteful CI workflows: https://hugovk.github.io/top-pypi-packages/
kstrauser 7 minutes ago [-]
Oof. That doesn't count in my opinion. The conflict is unfortunate, but it's not because either package is trying to modify other code. That is, the error is a side effect of how loading multiple libs into the interpret works. In theory, at least, you could fix those bugs without modifying the packages' behavior at all.
But still a bummer, to be sure. It's easy enough for me to say it doesn't count when I haven't been affected by it.
voxl 2 hours ago [-]
You already know the answer. Claude is not an intelligent sentient thing.
trjordan 7 hours ago [-]
Seems like it could easily be training data set size as well.
I'd love to see some quantification of errors in q/kdb+ (or hebrew) vs. languages of similar size that are left-to-right.
fer 6 hours ago [-]
>Seems like it could easily be training data set size as well.
I'm convinced that's the case. On any major LLM I can carpet bomb Java/Python boilerplate without issue. For Rust, at least last time I checked, it comes up with non-existing traits, more frequent hallucinations and general struggle to use the context effectively. In agent mode it turns into a first fight with the compiler, often ending in credit destroying loops.
And don't get me started when using it for Nix...
So not surprised about something with orders of magnitude smaller public corpus.
dotancohen 6 hours ago [-]
I realized this too, and it led me to the conclusion that LLMs really can't program. I did some experiments to find what a programming language would look like, instead of e.g. python, if it were designed to be written and edited by an LLM. It turns out that it's extremely verbose, especially in variable names, function names, class names, etc. Actually, it turned out that classes were very redundant. But the real insight was that LLMs are great at naming things, and performing small operations on the little things they named. They're really not good at any logic that they can't copy paste from something they found on the web.
weird-eye-issue 6 hours ago [-]
> I did some experiments to find what a programming language would look like, instead of e.g. python, if it were designed to be written and edited by an LLM.
Did your experiment consist of asking an LLM to design a programming language for itself?
dotancohen 6 hours ago [-]
Yes. ChatGPT 4 and Claude 3.7. They led me to similar conclusions, but they produced very different syntax, which led me to believe that they were not just regurgitating from a common source.
QuercusMax 34 minutes ago [-]
Is there a reason you believe the models can accurately predict this sort of thing?
short_sells_poo 4 hours ago [-]
Is this really a surprise? I'd hazard a guess that the ability to program and beyond that - to create new programming languages - requires more than just probabilistic text prediction. LLMs work for programming languages where they have enough existing corpus to basically ape a programmer having seen similar enough text. A real programmer can take the concepts of one programming language and express them in another, without having to have digested gigabytes of raw text.
There may be emergent abilities that arise in these models purely due to how much information they contain, but I'm unconvinced that their architecture allows them to crystallize actual understanding. E.g. I'm sceptical that there'd be an area in the LLM weights that encodes the logic behind arithmetic and gives rise to the model actually modelling arithmetic as opposed to just probabilistically saying that the text `1+1=` tended to be followed by the letter `2`.
ChadNauseam 16 minutes ago [-]
In my experience, claude works well at writing rust, and gemini is terrible. gemini writes rust as if it's a C++ programmer who has spent one day learning the basics of rust.
mfro 6 hours ago [-]
Yep. I had similar issues asking Gemini for help with F#, I assume lack of training data is the cause.
dlahoda 6 hours ago [-]
i tried gemini, openai, copilot, claude on reasonably big rust project.
claude worked well to fix use, clippy, renames, refactorings, ci. i used highest cost claude with custom context per crate.
never was able to get it write new code well.
for nix, i is nice template engine to start or search. did not tried big nix changes.
gizmo686 7 hours ago [-]
Hebrew is still written sequentially in Unicode. The right-to-left aspect there is simply about how the characters get displayed. On mixed documents, there is U+200E and U+200F to change the text direction mid stream.
From the perspective of a LLM learning from Unicode, this would appear as a delimeter that needs to be inserted on language direction boundaries; but everything else should work the same.
Timwi 6 hours ago [-]
I know I'm being pedantic, but I just want to point out that even U+200E/U+200F are generally not needed. If you put a Hebrew word in the middle of an English sentence, it displays correctly all by itself. This is due to the Unicode bidirectional algorithm, which defines a super sensible default behavior. You only need the RTL control characters in weird circumstances, perhaps ones involving punctuation marks or unusual uses of special characters.
cubefox 6 hours ago [-]
> Hebrew is still written sequentially
Everything is written sequentially in the sense that the character that is written first can only be followed by the character that is written next. In this sense writing non-sequentially is logically impossible.
dotancohen 6 hours ago [-]
An older Hebrew encoding actually encoded the last character first, then the penultimate character, then the character preceding that, etc.
Exercise to the reader to guess how line breaks, text wrapping, and search algorithms worked.
6 hours ago [-]
goatlover 6 hours ago [-]
Multiple characters can be written at once, they can also be done in reverse or out of order.
cubefox 6 hours ago [-]
No no, the second character you write must always be temporally preceded by the character you wrote first. Otherwise the second wouldn't have been the second, but the first, and moreover, the first would have been the second, which it wasn't.
dotancohen 6 hours ago [-]
I encourage you to find some place that still uses a Hebrew typewriter. When they have to type numbers, they'll type the number in backwards. And an old Hebrew encoding also encoded characters in reverse order.
short_sells_poo 4 hours ago [-]
I think parent just means that "backwards" is a relative term. Your backwards is someone else's "forward". For someone who is used to reading Hebrew, they would be used to reading right to left and this would seem completely natural, no?
Basically, the numbers 1234 and 4321 are identical assuming one is written left to right and the other is right to left. Then it's just a convention which way you are used to reading.
I know nothing of Old (or New) Hebrew unfortunately so I may be completely off base.
Someone 6 hours ago [-]
You could write multiple characters simultaneously. CRTs sort-of did that, for example, starting characters with ascenders before those without and finishing the characters without descenders before those with descenders.
So, in the word “gif”, they would start writing the “f” first and finish writing the “i” first (just before writing the last part of the “f”. For “if”, writing the “f” would start before writing the “i” started and finish after writing the “i” finished.
In traditional printing “writing” can happen simultaneously for an entire page, but colour printing can make things more complex.
44za12 3 hours ago [-]
That’s what I thought. Lack of training data might be a reason.
17 minutes ago [-]
briandw 6 hours ago [-]
This is something that diffusion based models would capable of. For example diffusion-coder https://arxiv.org/abs/2506.20639
Could be trained on right to left, but it doesn't seem like they did.
electroly 6 hours ago [-]
I always thought APL was written in the wrong direction. It writes like a concatenative language that's backwards--you tack things onto the front. NumPy fixes it by making the verbs all dotted function calls, effectively mirroring the order. e.g. in APL you write "10 10 ⍴ ⍳100" but in NumPy you write "np.arange(1, 101).reshape(10, 10)". Even if you don't know either language, you can tell that the APL version is the reverse of the Python version.
My hot take is that Iverson was simply wrong about this. He couldn't be expected to predict code completion and then LLMs both wanting later tokens to depend on earlier tokens. SQL messed it up, too, with "from" not coming first. If APL were developed today, I think left-to-right evaluation would have been preferred. The popularity of dotted function calls in various languages makes it reasonably clear that people like tacking things onto the end and seeing a "pipeline" form from left to right.
beagle3 6 hours ago [-]
APL was designed as a notation for math; if you pronounce it properly, it makes more sense than numpy:
The 10 by 10 reshaping of counting to 100
isoprophlex 3 hours ago [-]
Numpy: Counting to 100, then reshaped to 10 x 10. Doesn't really seem all that different to me.
beagle3 3 hours ago [-]
It’s not very different, but it’s the numpy way is not the math way: when you talk math, you say “the exponent of the absolute value of the cosine of x”, like in APL, not “take x, get its cosine, then take the absolute value, and then get its exponent”
In fact, for many things, you so the math way in numpy as well. But in other things, the dot/object-oriented way is preferred.
APL is just consistent, terse, mathematical notation.
fwip 5 hours ago [-]
With complicated formulas, it often makes more sense and can give more guidance by first talking about the last operations to be applied. This seems to match the LLM structure, by starting by describing what we want, and then filling in the more specialized holes as we get to them. "Top-down" design vs "bottom-up".
Your insight about APL being reverse-concatenative is very cool.
catfacts 3 hours ago [-]
Cognitive load in LLMs: When LLMs are faced with syntactic complexity (Lisp/J parentheses/RL-NOP), distractors (cat facts), or unfamiliar paradigms (right-to-left evaluation), the model’s performance degrades because its "attention bandwidth" is split or overwhelmed. This mirrors human cognitive overload.
My question: is there a way to reduce cognitive load in LLMs?, one solution seems to be process the input and output format so that the LLM can use a more common format. I don't know if there is a more general solution.
Isn't the whole idea of Lisp that there is _no_ syntactic complexity? Lisp programs are roughly a serialized AST.
pornel 39 minutes ago [-]
LLMs use tokens, with 1d positions and rich complex fuzzy meanings, as their native "syntax", so for them LISP is alien and hard to process.
That's like reading binary for humans. 1s and 0s may be the simplest possible representation of information, but not the one your wet neural network recognizes.
jdiff 2 hours ago [-]
Agreed, Gleam as a language has very few, generalized syntactic constructs compared to most procedural languages. There's enough of a signal in the data to be able to answer queries about the language; but when writing, LLMs universally trip over themselves. The signal from other nearby languages is too strong and it ends up trying to do early returns, if statements, even loops on occasion.
2 hours ago [-]
umanwizard 2 hours ago [-]
I'm not disputing that LLMs are bad for Lisp code, I'm just saying I don't think "syntactic complexity" is a correct explanation for why that is.
catfacts 21 minutes ago [-]
Yes, the concept of "syntactic complexity" applied to LLMs can be very different of what we think and I think it depends of the tokenizer. Perhaps LLMs could be fine-tuned by using a grammar for computer languages and special tokens for this grammar in order to reduce syntactic complexity. For example in Lisp, a right or left parenthesis could be tokenized in a special way (indicating left-lisp-parenthesis or right-lisp-parenthesis), that way the LLM could learn faster and reduce syntactic errors.
catfacts 2 hours ago [-]
I usually use deepseek (gratis) for code, and when using defun and let it usually lacks one (or more) closing parenthesis. So the way to mark the end is not well understood by this LLM, or perhaps that the height of the AST is usually bigger than in python.
sitkack 4 hours ago [-]
Ordering issues can be overcome by allowing the model to think in one direction and then reverse the output once it has created it.
vessenes 8 hours ago [-]
Interesting. Upshot - right to left eval means you generally must start at the end, or at least hold an expression in working memory - LLMs - not so good at this.
I wonder if diffusion models would be better at this; most start out as sequential token generators and then get finetuned.
Another quirk inserting random whitespace when generating code. Seem to be tokens for different lengths of whitespace
aghilmort 5 hours ago [-]
most mainstream models are decoders vs. encoders-decoders, diffusers, etc. and lack reversible causal reasoning, which of course can be counter-intuitive since it doesn’t feel that way when models can regenerate prior content
some hacks for time / position/ space flipping the models:
- test spate of diffusion models emerging. pro is faster, con is smaller context, ymmv is if trained on that language &/or context large enough to ICL lang booster info
- exploit known LTL tricks that may work there’s bunch of these
- e.g., tell model to gen drafts in some sort RPN variant of lang, if tests tell it to simulate creating such a fork of this and then gen clean standard form at end
- have it be explicit about leapfrogging recall and reasoning, eg be excessively verbose with comments can regex strip later
- have it build a stack / combo of the RPN & COT & bootstrapping its own ICL
- exploit causal markers - think tags that can splinter time - this can really boost any of the above methods - eg give each instance of things disjoint time tags, A1 vs K37 for numbered instances of things that share a given space - like a time GUID
- use orthogonal groups of such tags to splinter time and space recall and reasoning in model, to include seemingly naive things like pass 1 etc
- our recent arXiv paper on HDRAM / hypertokens pushes causal markers to classic-quantum holographic extreme and was built for this, next version will be more accessible
- the motivators are simple - models fork on prefix-free modulo embedding noise, so the more you make prefix-free, the better the performance, there’s some massive caveats on how to do this perfectly which is exactly our precise work - think 2x to 10x gain on model and similar on reasoning, again ymmv as we update preprint, post second paper that makes baseline better, prep git release etc to make it tons easier to get better recall and exploit same to get better reasoning by making it possible for any model to do the equivalent of arbitrary RPN
- our future state is exactly this a prompt compiler for exactly this use case - explainable time-independent computation in any model
knome 4 hours ago [-]
don't plan on it staying that way. I used to toss wads of my own forth-like language into LLMs to see what kinds of horrible failure modes the latest model would have in parsing and generating such code.
at first they were hilariously bad, then just bad, then kind of okay, and now anthropic's claude4opus reads and writes it just fine.
sitkack 4 hours ago [-]
How much incontext documentation for your language are you giving it, or does it just figure it out?
knome 11 minutes ago [-]
it varied. with the earlier models, generally more, trying to see if some apparition of mechanical understanding would eventually click into place.
IIRC, none of the gpt3 models did well with forth-like syntax. gpt4 generally did okay with it but could still get itself confused. claude4opus doesn't seem to have any trouble with it at all, and is happy to pick up the structures contextually, without explicit documentation of any sort.
another of my languages uses some parse transforming 'syntactic operators' that earlier models could never quite fully 'get', even with explanation. likely because at least one of them has no similar operator in popular languages. claude4opus, however, seems to infer them decently enough, and a single transform example is sufficient for it to generalize that understanding to the rest of the code it sees.
so far, claude has proved to be quite an impressive set of weights.
i000 3 hours ago [-]
R has right assigment `1 -> x` LLMs seem to enjoy it a bit too much.
tantalor 5 hours ago [-]
Languages that are difficult for LLM to read & write are also difficult for the general public. These languages have always had poor uptake and never reach critical mass, or are eventually replaced by better languages.
Language designers would be smart to recognize this fact and favor making their languages more LLM friendly. This should also make them more human friendly.
jdiff 3 hours ago [-]
I don't find this to be true. There are languages that are difficult to wrap your head around initially, but that turn out to be delightfully productive with a few adjustments to the mental model. Adjustments that LLMs don't have the training data for.
That says nothing about the language at all, actually. Just that it's small and easily confused for something more idiomatic to a newbie.
DaiPlusPlus 1 hours ago [-]
> Adjustments that LLMs don't have the training data for.
Methinks if you want job-security in a post—LLM-zero-shot-app-generator world, get into Lisp or Haskell; People that know only Node+React from YouTube learn-2-code tutorials are going to be slaughtered.
I just had an idea: an app/GUI/backend framework for Lisp or Haskell (with an S-expression lib) where everything is structurally inverted so it must be manually run through foldr - behold: an LLM-resistant (if not LLM-proof?) dev environment!
markerz 4 hours ago [-]
I actually think Ruby on Rails is incredibly difficult for LLMs to write because of how many implicit "global state" things occur. I'm always surprised how productive people are with it, but people are productive with it for sure.
short_sells_poo 4 hours ago [-]
That's because global state is very convenient early on. Everything is in one place and accessible. It's convenient to prototype things this way. This is very similar to doing scientific research (and why often research code is an ugly boondoggle).
Most techies (generalizing here) start with a reasonably clear spec that needs to be implemented and they can focus on how to architect the code.
Research - whether science, finance or design - is much more iterative and freeform. Your objective is often very fuzzy. You might have a vague idea what you want, but having to think about code structure is annoying and orthogonal to your actual purpose.
This is why languages like Ruby work well for certain purposes. They allow the person to prototype extremely rapidly and iterate on the idea. It will eventually reach a breaking point where global state starts being an impediment, but an experienced dev will have started refactoring stuff earlier than that as various parts of the implementation becomes stable.
Tainnor 4 hours ago [-]
This argument in favour of mediocrity and catering to the lowest common denominator is one of the key reasons why I dislike people who want to shove LLMs into everything (including art).
draw_down 4 hours ago [-]
[dead]
cess11 7 hours ago [-]
"Claude is aware of that, but it struggled to write correct code based on those rules"
It's actually not, and unless they in some way run a rule engine on top of their LLM SaaS stuff it seems far fetched to believe it adheres to rule sets in any way.
Local models confuse Python, Elixir, PHP and Bash when I've tried to use them for coding. They seem more stable for JS, but sometimes they slip out of that too.
Seems pretty contrived and desperate to invent transpilers from quasi-Python to other languages to try and find a software development use for LLM SaaS. Warnings about Lisp macros and other code rewrite tools ought to apply here as well. Plus, of course, the loss of 'notation as a tool of thought'.
strangescript 7 hours ago [-]
If your model is getting confused by python, its a bad model. Python is routinely the best language for all major models.
cess11 6 hours ago [-]
I don't know what counts as a major model. Relevant to this, I've dabbled with Gemma, Qwen, Mistral, Llama, Granite and Phi models, mostly 3-14b varieties but also some larger ones on CPU on a machine that has 64 GB RAM.
wild_egg 6 hours ago [-]
I think the issue there is those smaller versions of those models. I regularly use Gemma3 and Qwen3 for programming without issue but in the 27b-32b range. Going smaller than that generally yields garbage.
yujzgzc 7 hours ago [-]
Humans can't either? I think if this convention had been more usable form of programming, we'd know by now
anonzzzies 6 hours ago [-]
Once you get used to it, traditional ways look tedious and annoying to me. I think the power is in 'once you get used to it'. That will keep out most people. See python llm implementations vs k ones as a novice and you will see verbose unreadable stuff vs line noise. When you learn the math you see verbose code where the verbose code adds nothing at all vs exactly what you would write if you could.
IshKebab 6 hours ago [-]
Tedious and annoying for one-off commands maybe. It's like regex. Pretty compelling if you're writing a one-off pattern, you get immediate feedback and then you throw it away.
But it's not a good idea to use regexes in code that you're going to use long term. It's justifiable for simple regexes, and many people go against this advice, but really for anything remotely complex regexes become totally unreadable and extremely bug prone. Complex regexes are a huge code smell and array languages are pretty much one enormous regex.
Timwi 5 hours ago [-]
What would you propose as an alternative to regexes that provides the same functionality without the unreadable syntax?
I wrote something like that in C# once [0] but I'm not getting the impression that there's a lot of demand for that kind of thing.
Parsing expression grammars (pegs) are usually IME more maintainable long term, partially just because of much more testable and composable they are
afiori 4 hours ago [-]
I suspect but am not sure that PEGs cannot do negative nor positive lookbehind, but it is not a very used feature.
girvo 2 hours ago [-]
Yeah that’s true IIRC but it’s rarely been a problem for my usage!
fwip 5 hours ago [-]
Chiming in - I've found the Pattern type in Unison[0] to be very nice to use. When you're just using the built-in patterns, it is similar to verbose regex. The real power of them is that it's easy to define, name, and re-use sub-patterns. I think it's similar to parser combinators in this way, like nom from rust[1].
I mean, I got used to RPN and think that's the utmost bestest way to write. Objectively it's not as usable. Learnability is a part of usability
kragen 6 hours ago [-]
It might be a question of familiarity rather than objective usability. I'm writing this comment in Latin letters rather than Cyrillic or Hebrew because I find Latin letters much more usable than Cyrillic or Hebrew. But that's because I've been surrounded by Latin letters since I was born, and have only occasionally encountered Cyrillic or Hebrew.
I think it's obvious that Cyrillic isn't any less usable than the Latin alphabet in any objective sense. In fact, I'm using English orthography, which has all kinds of unnecessary usability problems which aren't present in any Cyrillic orthography that I know of. But familiarity is a much stronger factor; even today I can barely sound out words in Russian or Ukrainian, while English text printed in Latin letters is clearer to me than speech.
On theoretical grounds, I suspect that the APL syntax Gabi is calling RL-NOP is less usable for left-to-right readers than at least LR-NOP and maybe even conventional Please Brutally Execute My Dear Aunt Sally operator precedence. But familiarity is such a strong force that this hypothesis is very difficult to test.
The theoretical grounds are that, when reading left to right, a reader must maintain a stack of pending operators and values in their mind, unless they are saved by parentheses. (The Iverson quote disagrees with this, but I think Iverson was wrong.) Maintaining mental stacks is difficult and error-prone; this is the reason for the Tim Peters proverb, "Flat is better than nested."
I suspect that operator precedence might be superior for two reasons:
1. It more often avoids parentheses, which are extra symbols to recognize and correctly pair up in your mind.
2. The meaning of high-precedence subexpressions like `x×b` are almost context-independent—although an exponentiation operator or something like a C struct field selector could still follow `b` and change its meaning, following multiplications, divisions, additions, subtractions, or comparisons will not, and preceding additions, subtractions, or comparisons also will not. I conjecture that this facilitates subconscious pattern recognition.
But the familiarity factor enormously outweighs these theoretical considerations for me.
adrian_b 4 hours ago [-]
> " I suspect that the APL syntax ... is less usable for left-to-right readers"
On the contrary, I find it much more usable for left-to-right readers, because it allows a "top-down" reading of the expressions, instead of a "bottom-up" reading.
When trying to understand an unfamiliar program, for debugging or maintenance, you normally do not want to waste time by reading completely all expressions, which provide irrelevant computation details.
You typically search where some variables are modified and how and why. For this it is frequently enough to look only at the last operations that have been performed before storing a modified value into a variable.
With the Iverson notation, the last operations are always conveniently grouped at the left side of a text line. Thus you read from left to right only as much as necessary to find what you need, then you can skip the rest of the line.
With the school notation, the required information is not grouped at one end of the line, so reading becomes slower.
The opposite of the Iverson notation, which was used in some stack-oriented languages, also groups the information, but in a way that is less usable for left-to-right users.
From natural languages, left-to-right readers expect that a sentence starts with its topic (at the left side), i.e. the most important part, e.g. the last assignment, like in the Iverson notation, instead of ending with its topic, like in the opposite notation.
> "a reader must maintain a stack of pending operators and values in their mind"
I believe that few readers, if any, do this.
The normal case when reading is that you do not want to reproduce in your mind what the computer does, but only to find the information flows between program variables. For this, it is enough to read partial expressions, as explained above.
In the very rare case when you wanted to make a mental calculation identical to that of the computer, you would normally read the expression from right to left.
When writing, the Iverson notation is usually more convenient than the school notation, while writing normally, from left to right. The reason is that for most computations the natural way to find the expression that must be computed is to go backwards, from the desired result towards the available data.
kragen 3 hours ago [-]
Hmm, could be. I'll have to think about that.
adrian_b 5 hours ago [-]
The popularity of a convention has no relationship with its usability.
Everybody learns in school the traditional convention for writing mathematical expressions.
It appears that for most people it is difficult or impossible to unlearn later such a convention, even if they encounter a superior convention.
On the other hand, I am among those fewer for which this is not true, so when I have first read the book "A Programming Language" of K. Iverson, on which the later APL language and its successors have been based, I have immediately recognized that the Iverson convention is much better than the school convention, and I have no trouble in using it.
When reading a program written with the Iverson convention, you still read from left to right, but you typically do not read until the end of the line, but only as much of the left part as necessary to understand the purpose of the line. (Because the right operand of any operator is everything that follows it until the end of the line, and the details of that computation may be irrelevant. With school notation, when searching where a variable has been modified and how, you must jump between the beginning of the line and the end of the line, to find the last operations that have generated the stored value, when reading and understanding the complete expression would be a waste of time.)
The original motivation of the Iverson convention, which remains very important, was to give a useful meaning for a sequence of identical non-commutative operators, e.g. subtraction and division. This is particularly desirable when the operators are used in vector reductions.
(With school notation, a0 - a1 - a2 - ... - an is seldom a useful expression, but with the Iverson convention it becomes alternate sum, which is needed very frequently. Similarly for division.)
maest 7 hours ago [-]
I think there is a reason for this, but maybe not a good one.
1. Function application should be left to right, e.g. `sqrt 4`
2. Precedence order should be very simple. In k, everything has the same precedence order (with the exceptions of brackets)
1 + 2 forces you to have this right to left convention, annoyingly.
Fwiw, I think 2 is great and I would rather give up 1 than 2. However, writing function application as `my_fun arg` is a very strong convention.
layer8 6 hours ago [-]
Even if you aren’t used to it, you’d be able to reason yourself through it, knowing how the language works, and would be aware that you need to reason through it. And it isn’t that LLMs don’t know that the language works that way, if you ask them about it. It also isn’t that they aren’t able to reason through it, if you ask them to do so. It’s that they lack awareness when to switch modes, lack the ability to have their knowledge interrupt their “intuitive” output and instead start reasoning about how to proceed.
leprechaun1066 5 hours ago [-]
It's not because of the left of right evaluation. If the difference was that simple, most humans, let alone LLMs, wouldn't struggle with picking up q when they come from the common languages.
Usually when someone solves problems with q, they don't use the way one would for Python/Java/C/C++/C#/etc.
This is probably a poor example, if I asked someone to write a function to create an nxn identity matrix for a given number the non-q solution would probably involve some kind of nested loop that checks if i==j and assigns 1, otherwise assigns 0.
In q you'd still check equivalence, but instead of looping, you generate a list of numbers as long as the given dimension and then compare each item of the list to itself:
{x=/:x:til x}3
An LLM that's been so heavily trained on an imperative style will likely struggle to solve similar (and often more complex) problems in a standard q manner.
wat10000 5 hours ago [-]
A human can deal with right-to-left evaluation by moving the cursor around to write in that direction. An LLM can’t do that on its own. A human given an editor that can only append would struggle too.
anticensor 3 hours ago [-]
Idea: feed the language model the parse tree instead of the textual sequence.
wat10000 57 minutes ago [-]
Might help. You could also allow it to output edits instead of just a sequence. Probably have to train it on edits to make that work well, and the training data might be tricky to obtain.
grej 6 hours ago [-]
This is, in part, one of the reasons why I am interested in the emerging diffusion based text generation models.
roschdal 6 hours ago [-]
I can write code right-to-left, I simply choose to not do it.
rob_c 7 hours ago [-]
Same reason the same models don't fundamentally understand all languages. They're not trained to. Frankly the design changes to get this to work in training is minimal but this isn't the way English works so expect most of the corporate LLM to struggle because that's where the interest and money is.
Give it time until we have true globally multi lingual models for superior context awareness.
strangescript 7 hours ago [-]
A byte tokenized model is naturally 100% multi-lingual in all languages in its data set. There just isn't a lot of reason for teams to spend the extra training time to build that sort of model.
careful_ai 6 hours ago [-]
[dead]
AustinSerb 6 hours ago [-]
[dead]
ape4 6 hours ago [-]
I read the other day here that the new Apple AI can write out-of-order. Maybe it can do this.
Makes me wonder if future llms will be composing nonlinear things and be able to work in non-token-order spaces temporarily, or will have a way to map their output back to linear token order. I know nonlinear thinking is common while writing code though. current llms might be hiding a deficit by having a large and perfect context window.
Shouldn't be hard to train a coding LLM to do this too by doubling the training time: train the LLM both forwards and backwards across the training data.
Is there a top 100 package that does something funny on import?
My guess is that nearly all packages that did this sort of thing were left behind in the 2-to-3 migration, which a lot of us used as the excuse for a clean break.
But I agree that observable side effects are generally pretty rare. And apparently, both libraries are not even in the top 100 packages, depending on how you count. It looks like those spots are all taken by libraries used in uncached, wasteful CI workflows: https://hugovk.github.io/top-pypi-packages/
But still a bummer, to be sure. It's easy enough for me to say it doesn't count when I haven't been affected by it.
I'd love to see some quantification of errors in q/kdb+ (or hebrew) vs. languages of similar size that are left-to-right.
I'm convinced that's the case. On any major LLM I can carpet bomb Java/Python boilerplate without issue. For Rust, at least last time I checked, it comes up with non-existing traits, more frequent hallucinations and general struggle to use the context effectively. In agent mode it turns into a first fight with the compiler, often ending in credit destroying loops.
And don't get me started when using it for Nix...
So not surprised about something with orders of magnitude smaller public corpus.
Did your experiment consist of asking an LLM to design a programming language for itself?
There may be emergent abilities that arise in these models purely due to how much information they contain, but I'm unconvinced that their architecture allows them to crystallize actual understanding. E.g. I'm sceptical that there'd be an area in the LLM weights that encodes the logic behind arithmetic and gives rise to the model actually modelling arithmetic as opposed to just probabilistically saying that the text `1+1=` tended to be followed by the letter `2`.
for nix, i is nice template engine to start or search. did not tried big nix changes.
From the perspective of a LLM learning from Unicode, this would appear as a delimeter that needs to be inserted on language direction boundaries; but everything else should work the same.
Everything is written sequentially in the sense that the character that is written first can only be followed by the character that is written next. In this sense writing non-sequentially is logically impossible.
Exercise to the reader to guess how line breaks, text wrapping, and search algorithms worked.
Basically, the numbers 1234 and 4321 are identical assuming one is written left to right and the other is right to left. Then it's just a convention which way you are used to reading.
I know nothing of Old (or New) Hebrew unfortunately so I may be completely off base.
So, in the word “gif”, they would start writing the “f” first and finish writing the “i” first (just before writing the last part of the “f”. For “if”, writing the “f” would start before writing the “i” started and finish after writing the “i” finished.
In traditional printing “writing” can happen simultaneously for an entire page, but colour printing can make things more complex.
My hot take is that Iverson was simply wrong about this. He couldn't be expected to predict code completion and then LLMs both wanting later tokens to depend on earlier tokens. SQL messed it up, too, with "from" not coming first. If APL were developed today, I think left-to-right evaluation would have been preferred. The popularity of dotted function calls in various languages makes it reasonably clear that people like tacking things onto the end and seeing a "pipeline" form from left to right.
The 10 by 10 reshaping of counting to 100
In fact, for many things, you so the math way in numpy as well. But in other things, the dot/object-oriented way is preferred.
APL is just consistent, terse, mathematical notation.
Your insight about APL being reverse-concatenative is very cool.
My question: is there a way to reduce cognitive load in LLMs?, one solution seems to be process the input and output format so that the LLM can use a more common format. I don't know if there is a more general solution.
Edit: Cat attack https://the-decoder.com/cat-attack-on-reasoning-model-shows-...
That's like reading binary for humans. 1s and 0s may be the simplest possible representation of information, but not the one your wet neural network recognizes.
I wonder if diffusion models would be better at this; most start out as sequential token generators and then get finetuned.
some hacks for time / position/ space flipping the models:
- test spate of diffusion models emerging. pro is faster, con is smaller context, ymmv is if trained on that language &/or context large enough to ICL lang booster info
- exploit known LTL tricks that may work there’s bunch of these
- e.g., tell model to gen drafts in some sort RPN variant of lang, if tests tell it to simulate creating such a fork of this and then gen clean standard form at end
- have it be explicit about leapfrogging recall and reasoning, eg be excessively verbose with comments can regex strip later
- have it build a stack / combo of the RPN & COT & bootstrapping its own ICL
- exploit causal markers - think tags that can splinter time - this can really boost any of the above methods - eg give each instance of things disjoint time tags, A1 vs K37 for numbered instances of things that share a given space - like a time GUID
- use orthogonal groups of such tags to splinter time and space recall and reasoning in model, to include seemingly naive things like pass 1 etc
- our recent arXiv paper on HDRAM / hypertokens pushes causal markers to classic-quantum holographic extreme and was built for this, next version will be more accessible
- the motivators are simple - models fork on prefix-free modulo embedding noise, so the more you make prefix-free, the better the performance, there’s some massive caveats on how to do this perfectly which is exactly our precise work - think 2x to 10x gain on model and similar on reasoning, again ymmv as we update preprint, post second paper that makes baseline better, prep git release etc to make it tons easier to get better recall and exploit same to get better reasoning by making it possible for any model to do the equivalent of arbitrary RPN
- our future state is exactly this a prompt compiler for exactly this use case - explainable time-independent computation in any model
at first they were hilariously bad, then just bad, then kind of okay, and now anthropic's claude4opus reads and writes it just fine.
IIRC, none of the gpt3 models did well with forth-like syntax. gpt4 generally did okay with it but could still get itself confused. claude4opus doesn't seem to have any trouble with it at all, and is happy to pick up the structures contextually, without explicit documentation of any sort.
another of my languages uses some parse transforming 'syntactic operators' that earlier models could never quite fully 'get', even with explanation. likely because at least one of them has no similar operator in popular languages. claude4opus, however, seems to infer them decently enough, and a single transform example is sufficient for it to generalize that understanding to the rest of the code it sees.
so far, claude has proved to be quite an impressive set of weights.
Language designers would be smart to recognize this fact and favor making their languages more LLM friendly. This should also make them more human friendly.
That says nothing about the language at all, actually. Just that it's small and easily confused for something more idiomatic to a newbie.
Methinks if you want job-security in a post—LLM-zero-shot-app-generator world, get into Lisp or Haskell; People that know only Node+React from YouTube learn-2-code tutorials are going to be slaughtered.
I just had an idea: an app/GUI/backend framework for Lisp or Haskell (with an S-expression lib) where everything is structurally inverted so it must be manually run through foldr - behold: an LLM-resistant (if not LLM-proof?) dev environment!
Most techies (generalizing here) start with a reasonably clear spec that needs to be implemented and they can focus on how to architect the code.
Research - whether science, finance or design - is much more iterative and freeform. Your objective is often very fuzzy. You might have a vague idea what you want, but having to think about code structure is annoying and orthogonal to your actual purpose.
This is why languages like Ruby work well for certain purposes. They allow the person to prototype extremely rapidly and iterate on the idea. It will eventually reach a breaking point where global state starts being an impediment, but an experienced dev will have started refactoring stuff earlier than that as various parts of the implementation becomes stable.
It's actually not, and unless they in some way run a rule engine on top of their LLM SaaS stuff it seems far fetched to believe it adheres to rule sets in any way.
Local models confuse Python, Elixir, PHP and Bash when I've tried to use them for coding. They seem more stable for JS, but sometimes they slip out of that too.
Seems pretty contrived and desperate to invent transpilers from quasi-Python to other languages to try and find a software development use for LLM SaaS. Warnings about Lisp macros and other code rewrite tools ought to apply here as well. Plus, of course, the loss of 'notation as a tool of thought'.
But it's not a good idea to use regexes in code that you're going to use long term. It's justifiable for simple regexes, and many people go against this advice, but really for anything remotely complex regexes become totally unreadable and extremely bug prone. Complex regexes are a huge code smell and array languages are pretty much one enormous regex.
I wrote something like that in C# once [0] but I'm not getting the impression that there's a lot of demand for that kind of thing.
[0] https://github.com/Timwi/Generex
[0] https://share.unison-lang.org/@unison/website/code/main/late...
[1] https://docs.rs/nom/latest/nom/
I think it's obvious that Cyrillic isn't any less usable than the Latin alphabet in any objective sense. In fact, I'm using English orthography, which has all kinds of unnecessary usability problems which aren't present in any Cyrillic orthography that I know of. But familiarity is a much stronger factor; even today I can barely sound out words in Russian or Ukrainian, while English text printed in Latin letters is clearer to me than speech.
On theoretical grounds, I suspect that the APL syntax Gabi is calling RL-NOP is less usable for left-to-right readers than at least LR-NOP and maybe even conventional Please Brutally Execute My Dear Aunt Sally operator precedence. But familiarity is such a strong force that this hypothesis is very difficult to test.
The theoretical grounds are that, when reading left to right, a reader must maintain a stack of pending operators and values in their mind, unless they are saved by parentheses. (The Iverson quote disagrees with this, but I think Iverson was wrong.) Maintaining mental stacks is difficult and error-prone; this is the reason for the Tim Peters proverb, "Flat is better than nested."
I suspect that operator precedence might be superior for two reasons:
1. It more often avoids parentheses, which are extra symbols to recognize and correctly pair up in your mind.
2. The meaning of high-precedence subexpressions like `x×b` are almost context-independent—although an exponentiation operator or something like a C struct field selector could still follow `b` and change its meaning, following multiplications, divisions, additions, subtractions, or comparisons will not, and preceding additions, subtractions, or comparisons also will not. I conjecture that this facilitates subconscious pattern recognition.
But the familiarity factor enormously outweighs these theoretical considerations for me.
On the contrary, I find it much more usable for left-to-right readers, because it allows a "top-down" reading of the expressions, instead of a "bottom-up" reading.
When trying to understand an unfamiliar program, for debugging or maintenance, you normally do not want to waste time by reading completely all expressions, which provide irrelevant computation details.
You typically search where some variables are modified and how and why. For this it is frequently enough to look only at the last operations that have been performed before storing a modified value into a variable.
With the Iverson notation, the last operations are always conveniently grouped at the left side of a text line. Thus you read from left to right only as much as necessary to find what you need, then you can skip the rest of the line.
With the school notation, the required information is not grouped at one end of the line, so reading becomes slower.
The opposite of the Iverson notation, which was used in some stack-oriented languages, also groups the information, but in a way that is less usable for left-to-right users.
From natural languages, left-to-right readers expect that a sentence starts with its topic (at the left side), i.e. the most important part, e.g. the last assignment, like in the Iverson notation, instead of ending with its topic, like in the opposite notation.
> "a reader must maintain a stack of pending operators and values in their mind"
I believe that few readers, if any, do this.
The normal case when reading is that you do not want to reproduce in your mind what the computer does, but only to find the information flows between program variables. For this, it is enough to read partial expressions, as explained above.
In the very rare case when you wanted to make a mental calculation identical to that of the computer, you would normally read the expression from right to left.
When writing, the Iverson notation is usually more convenient than the school notation, while writing normally, from left to right. The reason is that for most computations the natural way to find the expression that must be computed is to go backwards, from the desired result towards the available data.
Everybody learns in school the traditional convention for writing mathematical expressions.
It appears that for most people it is difficult or impossible to unlearn later such a convention, even if they encounter a superior convention.
On the other hand, I am among those fewer for which this is not true, so when I have first read the book "A Programming Language" of K. Iverson, on which the later APL language and its successors have been based, I have immediately recognized that the Iverson convention is much better than the school convention, and I have no trouble in using it.
When reading a program written with the Iverson convention, you still read from left to right, but you typically do not read until the end of the line, but only as much of the left part as necessary to understand the purpose of the line. (Because the right operand of any operator is everything that follows it until the end of the line, and the details of that computation may be irrelevant. With school notation, when searching where a variable has been modified and how, you must jump between the beginning of the line and the end of the line, to find the last operations that have generated the stored value, when reading and understanding the complete expression would be a waste of time.)
The original motivation of the Iverson convention, which remains very important, was to give a useful meaning for a sequence of identical non-commutative operators, e.g. subtraction and division. This is particularly desirable when the operators are used in vector reductions.
(With school notation, a0 - a1 - a2 - ... - an is seldom a useful expression, but with the Iverson convention it becomes alternate sum, which is needed very frequently. Similarly for division.)
1. Function application should be left to right, e.g. `sqrt 4`
2. Precedence order should be very simple. In k, everything has the same precedence order (with the exceptions of brackets)
1 + 2 forces you to have this right to left convention, annoyingly.
Fwiw, I think 2 is great and I would rather give up 1 than 2. However, writing function application as `my_fun arg` is a very strong convention.
Usually when someone solves problems with q, they don't use the way one would for Python/Java/C/C++/C#/etc.
This is probably a poor example, if I asked someone to write a function to create an nxn identity matrix for a given number the non-q solution would probably involve some kind of nested loop that checks if i==j and assigns 1, otherwise assigns 0.
In q you'd still check equivalence, but instead of looping, you generate a list of numbers as long as the given dimension and then compare each item of the list to itself:
An LLM that's been so heavily trained on an imperative style will likely struggle to solve similar (and often more complex) problems in a standard q manner.Give it time until we have true globally multi lingual models for superior context awareness.