I’ve been using Q (built on K) for Advent of Code. It’s an amazing language that has made me reconsider what a language and programming should be. I come from a FP background and have always considered abstraction the heart of powerful and expressive programming.
In Q, abstraction is wholly unnecessary. Libraries are unnecessary. I can fit the entire language in my head and all code in the language is built out of the fundamental building blocks of it. There’s no layer after layer of abstraction, just a small base of operators and functions.
One might be tempted to compare it to other small languages, like Scheme. But for Scheme it’s intended that you build higher and higher levels of abstractions. With Q it’s both small and compact, and one rarely if ever feels the need to actually build a library of utility functions or anything really: it renders unnecessary the false prophet of code reuse.
I got interested in Q because some of my friends at other financial institutions were using it for HFT and research on historical tick data. I found it surprising at the time I that their software stacks could be so simple, essentially just a single small executable that entirely fit in the CPU cache that managed both the runtime and their entire database. At the time I thought this was impossible, but now I realize how possible and eminently reasonable this was. Instead of Kafka and Redis and SQL Dbs and Kubernetes and god knows what else, you just have Q. No DevOps bullshit, no containers, no complex shit to manage.
In the past 3 weeks I’ve had to rethink my philosophy of what software needs to be and have concluded that most of us are doing things very very wrong: we’ve created this unnecessary world of complexity that is tantamount to masturbation.
Now let’s talk about the language itself. It may look complex, but it’s actually quite simple. The only real data types are lists, dictionaries, symbols, and numbers. And surprisingly, that’s enough. When programming in it I have no desire for classes or objects or trees or whatever, in fact, just using lists and dictionaries can usually solve the problem in a better way.
Also, while Q is interpreted, it’s so goddamn fast. I’ve done some benchmarks against C and Q is about as fast, sometimes even faster. The optimizations the interpreter does are just insane. Not even joking, a single line of Q can accomplish somewhere between 100 to 1000 lines of what the equivalent C code would look like.
Another plus is that editor tooling is unnecessary, I have no need for autocomplete or inline docs because I can literally remember the entire thing. And since Q is so compact, I can view an entire program on a single screen, no scrolling, no jumping between files.
I’ve also noticed that I can remember the verbatim definition of my code. I can almost perfectly remember my exact code for each Advent of Code solution.
Anyways, good luck on Klong, I’m going to check it out. I don’t like J and wish there was a good open source K/Q like language. I don’t actually like K because the overloads on function arity creates a lot of complexity (Q does away with this), so Klong might be right up my alley!
For anyone who hasn’t tried an array based language, I highly recommend it. Q has expanded my mind more than I thought possible at this point in my career. My last really mind expanded experience in programming was when I picked up Scheme and Haskell in high school. And now after using Q, those languages aren’t that great anyways!
> Instead of Kafka and Redis and SQL Dbs and Kubernetes and god knows what else, you just have Q. No DevOps bullshit, no containers, no complex shit to manage.
What if you need to scale beyond what one machine can handle? Or if you need redundancy and failover? Transactional guarantees? Or need to talk to external systems?
Its easy to build a single-executable no-DB no-container no-devops application in most languages if you don’t care about the finer details of building a high scale resilient system that can survive a machine going down or has to talk to third party external systems.
IPC is pretty easy in Q. I'm not saying you could be at Google scale with one machine, I'm just saying you could get rid of your entire stack and just use a bunch of networked machines running Q. Though, for maybe the majority of businesses, a single beefy server running Q would be sufficient. The performance of the language and integrated database is pretty exceptional.
k is incredibly good for distributed computing. It's not as difficult as you're thinking: Arthur started out (as far as languages he published: he implemented a few before) with A and A+ at Morgan Stanley, and Morgan Stanley up until a few years ago actively used both. Distribution was inherent in k from the outset.
Also, your comment on "noDB" is unfounded. k and q are exclusively distributed with kdb+, which is probably the fastest SQL database on the planet.
Interesting experience report. I've seen some comments on hn on people starting out like you do, but realizing after a year that they need to work really hard to pickup old code. Do you imagine to have an easy time picking up your advent of code programs next December?
Another question - are there debuggers for these languages? Can you set breakpoints?
How do you interact with the www? Can you get some soap/json data and write a simple web service (even just a json endpoint)?
Or do you mostly see it useful to export data in some tsv/csv format, filter it through k, and write it back out?
Interesting experience report. I've seen some comments on hn on people starting out like you do, but realizing after a year that they need to work really hard to pickup old code. Do you imagine to have an easy time picking up your advent of code programs next December?
If you don't understand something in k or APL after writing it, you probably wrote it wrong to begin with.
A cool example of how much APL lends itself to this is co-dfns, which the author rewrote multiple times:
In particular, the desire to make the code as "disposable" as possible has lead to me using strictly combinatory/points-free style programming in the core compiler. It makes it very easy to change things because it's easy to see the whole picture and easier for me to delete any given piece of code. Many "fixes" in my code are single character APL fixes, or maybe two character APL fixes.
This whole discussion got started because of the amount of change that has happened with this code base (hundreds of thousands of lines). You can read more about that in the other historical threads referenced by dang. I've rewritten the compiler at least five times.
>Do you imagine to have an easy time picking up your advent of code programs next December?
Without comoments, it might be a little tough. I've gotten into the habit of commenting each line.
>Another question - are there debuggers for these languages? Can you set breakpoints?
There's an integrated debugger that supports breakpoints. I really haven't used it at all since the REPL works quite well, but I could imagine it being useful in production. Especially since you can debug and fix your program while it's running.
>How do you interact with the www? Can you get some soap/json data and write a simple web service (even just a json endpoint)?
There's included functions for parsing and generating html and json. I haven't used them though.
>Or do you mostly see it useful to export data in some tsv/csv format, filter it through k, and write it back out?
I would imagine that live Q programs work by leveraging the real-time streaming functionality and just operating directly on the stream or in-memory database. For offline analysis the ability to load and store data in binary is probably useful. You can also read and write the entire working image in order to transfer state across machines or time.
J is really nice, though different than Q (more directly a dialect of APL than a derivative).
Though if you want an open source k, ngn/k was just released under the AGPL a few days ago.
I'd recommend starting with J, though, because there are a lot more resources for it, there's a mobile app for it (that's really nice), and it's the last thing Iverson worked on.
IIRC, Iverson commented somewhere that K's (Arthur's) practicality wins over APL/J's (Iverson's) purity in some places; specifically,
K/Q does scan/reduce from left to right, which is easier for most people to reason about and use, even though it makes "-/1 2 3 4 5" different from "1-2-3-4-5" and thus less "pure"/"consistent" in some sense.
J's trains and forks (and APL's simpler products) are much more confusing for people than K/Q's juxtaposition and projection.
Can't remember or find Iverson's comments now. He might have made more, or I might be misremembering/misattributing. But I agree with these and think K is a better start than J (though not sure between APL and K).
It’s not as easy to get information as other languages, but the mailing list is super helpful and all the questions I’ve asked have been replied to by paid Kx (the people that make Q) employees:
Not the person above, but I'd recommend learning APL first. Arthur Whitney (the author of k and q) got started programming by his father and Ken Iverson (the inventor of APL) being friends, and Iverson giving him live demos of APL. Iverson has a lot of amazing resources for learning J (APL's successor) and APL, and learning it will teach you how to be more productive in q, k, J, APL and more!
This is the arthurese micro APL implementation, which inspired the J implementation - although IIRC the design for J was already well on its way at the time (and indeed, Iverson did not consider J a new language, but rather an improved APL dialect, so you could say the design goes back to 1956...)
It likely also formed a draft for A/A+ which Arthur did at Morgan Stanley (open sourced at aplusdev.org, but unmaintained and rotting theese days) - which themselves led to the creation of K.
In Q, abstraction is wholly unnecessary. Libraries are unnecessary. I can fit the entire language in my head and all code in the language is built out of the fundamental building blocks of it. There’s no layer after layer of abstraction, just a small base of operators and functions.
One might be tempted to compare it to other small languages, like Scheme. But for Scheme it’s intended that you build higher and higher levels of abstractions. With Q it’s both small and compact, and one rarely if ever feels the need to actually build a library of utility functions or anything really: it renders unnecessary the false prophet of code reuse.
I got interested in Q because some of my friends at other financial institutions were using it for HFT and research on historical tick data. I found it surprising at the time I that their software stacks could be so simple, essentially just a single small executable that entirely fit in the CPU cache that managed both the runtime and their entire database. At the time I thought this was impossible, but now I realize how possible and eminently reasonable this was. Instead of Kafka and Redis and SQL Dbs and Kubernetes and god knows what else, you just have Q. No DevOps bullshit, no containers, no complex shit to manage.
In the past 3 weeks I’ve had to rethink my philosophy of what software needs to be and have concluded that most of us are doing things very very wrong: we’ve created this unnecessary world of complexity that is tantamount to masturbation.
Now let’s talk about the language itself. It may look complex, but it’s actually quite simple. The only real data types are lists, dictionaries, symbols, and numbers. And surprisingly, that’s enough. When programming in it I have no desire for classes or objects or trees or whatever, in fact, just using lists and dictionaries can usually solve the problem in a better way.
Also, while Q is interpreted, it’s so goddamn fast. I’ve done some benchmarks against C and Q is about as fast, sometimes even faster. The optimizations the interpreter does are just insane. Not even joking, a single line of Q can accomplish somewhere between 100 to 1000 lines of what the equivalent C code would look like.
Another plus is that editor tooling is unnecessary, I have no need for autocomplete or inline docs because I can literally remember the entire thing. And since Q is so compact, I can view an entire program on a single screen, no scrolling, no jumping between files.
I’ve also noticed that I can remember the verbatim definition of my code. I can almost perfectly remember my exact code for each Advent of Code solution.
Anyways, good luck on Klong, I’m going to check it out. I don’t like J and wish there was a good open source K/Q like language. I don’t actually like K because the overloads on function arity creates a lot of complexity (Q does away with this), so Klong might be right up my alley!
For anyone who hasn’t tried an array based language, I highly recommend it. Q has expanded my mind more than I thought possible at this point in my career. My last really mind expanded experience in programming was when I picked up Scheme and Haskell in high school. And now after using Q, those languages aren’t that great anyways!