A Lispy Interlude
Table of Contents
TL;DR: Lisp is a fascinating family of languages. Claims of enlightenment are overblowm, but it seems to approach an upper limit in language design if expressiveness and metaprogramming are primary goals. It exposed me to lambda calculus and broadened my programming purview beyond the algol descendants which was all I'd known previously. I still use common lisp and clojure on a regular basis and programming in lisp remains a pleasure.
Intro
(Obligatory xkcd quote)
These are your father's parentheses, elegant weapons for a more civilized age - xkcd
I came to Lisp quite by accident. By chance I chose to try out Emacs (cargo cult style because Casey Muratori and Jonathan Blow use Emacs & I wanted to use a different system than programmers around me, i.e. Vim).
After the initial leap of faith, I stayed due to the lindy effect and the purported ability to completely extend/ customize Emacs. (Who doesn't hate learning arbitrary pieces of software/ being dependent on mutable toolchains?) As such, while trying to come to terms with the steep learning curve and starting to slowly customize my Emacs, I was exposed to Emacs Lisp.
This, combined with a burgeoning interest in functional programming (FP) at the time (I was motivated to pickup another language to try out a different paradigm in general) led me to look into Lisp. (My lisp of choice, Common lisp, turned out to be paradigm agnostic)
Of the standard set of recommended languages to learn FP, Lisp sounded compelling if only from the way people talked about it:
Lisp is worth learning for the profound enlightenment experience you will have when you finally get it; that experience will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot." -Eric Raymond
After considering which Lisp (Lisp is a family of languages) to jump into, I settled on Common Lisp (CL), see Note on Common Lisp.
Well, I assumed I had a pretty strong foundation in programming. I had a few languages under my belt and been exposed to a variety of languages and programming subdisciplines like graphics for many years.
To my surprise and frustration, I couldn't understand the syntax at all. (A very common first reaction to Lisp; such reactions have even given rise to the joke that lisp stands for lots of irritating superfluous parentheses ) In a very basic way, I couldn't even declare or assign simple variables or understand how to return something from a function. There seemed to be an overwhelming amount of specialized verbiage/ specialized syntax (Lisp actually has almost no syntax like the lambda caluclus at its core) and anytime I messed around in a REPL I got compilation errors that I couldn't follow
It felt like I was touching C++ or OpenGL again for the first time. Some convoluted, black box that didn't seem worth the effort. I was very close to calling it quits, but the army of intelligent people talking about englightenment kept me going.
I understand now, after quite a bit of fumbling (save yourself the effort)
Why?
REPL & image based development
The REPL (read-eval-print-loop) is excellent for incremental / exploratory development. It's a highly ergonomic style of coding and lends itself to rapid idea iteration / maintaining flow.
Instead of write-compile-debug cycle, you can incrementally build interactively. You can compile functions, redefine classes, etc. all while the program is running via hot reloads/ changing the internal state of the image. (all compiled code, data, sybol table, environment state)
You don't have to restart a process and then re-create objects. Compile the bit that you want or is not working and get instant feedback. If something isn't working or you don't understand something, just hop over to your REPL and fix it or prove it to yourself. You can inspect trees of live values, or rewind the stack to undo an exception.
Macros
Lisp is called the programmable programming language. It's fundamentally different from other languages because it runs in the same context it's written in, i.e. it's written in the same form as running Lisp code which is the same as Lisp data (the so called s-expression). Lisp's syntax, the hated parens, are a necessary result of this. (its source code is already in a form that is directly interpretable as an AST, the s-expression is the AST, this correspondence is what enables homoiconicity, i.e. code as data / data as code duality)
"you express programs directly in the parse trees that get built behind the scenes when other languages are parsed, and these trees are made of lists, which are Lisp data structures."
It's ironic that Lisp's syntax is such a sticking point as it's actually a lack of syntax. Everything is an expression, every expression gets evaluated, if you don't want it evaluated, you put a quote on it.
Lisp is powerful because all Lisp programs are also Lisp data -– everything that can be run can be written (and read) in an s-expression. This makes it possible to write Lisp code that reliably generates and transforms source code through the metaprogramming facility of macros: programs that write programs, that write programs… s-expressions all the way down.
The whole language is there all the time. There is no real distinction between read-time, compile-time, and runtime. You can compile or run code while reading, read or run code while compiling, and read or compile code at runtime. Running code at read-time lets users reprogram Lisp's syntax; running code at compile-time is the basis of macros; compiling at runtime is the basis of Lisp's use as an extension language in programs like Emacs; and reading at runtime enables programs to communicate using s-expressions, an idea recently reinvented as XML. - Paul Graham
This aspect is what people are talking about when they use hyperbolic terms like "enlightenment", or about the necessity of Lisp, that Lisp isn't quite an invention, but rather is closer to a discovery.
Lisp isn't a language, it's a building material."
- Alan Kay
"Lisp's core occupies some kind of local optimum in the space of programming languages"
- John McCarthy
"Lisp is a minimal fixed point amongst programming languages. It's not an invention, but a discovery. That's why it won't just go away."
- Brian Beckman
Part of what makes Lisp distinctive is that it is designed to evolve. As new abstractions become popular (object-oriented programming, for example), it always turns out to be easy to implement them in Lisp. Like DNA, such a language does not go out of style.
- Pual Graham
See these two articles for extended explaination (Short cut enlightenment! Life's too short) https://dl.acm.org/doi/10.1145/3386330 https://stopa.io/post/265 https://www.defmacro.org/ramblings/Lisp.html
It would seem that Lisp is an expressiveness limit that a language can approach. Whether through s-expressions or not Lisp transcends language features, by having a single language feature that lets you define language features
Lisp's strange syntax (lack of syntax), and its concomitant homoiconicity (expressing the language in its own data structures) creates this limiting behavior. One could argue that if you add this facility to a language, you can no longer claim to have invented a new language, but only a new dialect of Lisp. Analogous maybe, but more funadamental, to how C is often described as a portable assembly language. If you change the abstraction layer at that low level, it's more or less a syntactic isomorphism, you haven't really gained anything. A reason C is ubiquitous and hasn't really been replaced.
More fundamental, I think, because just like the Lambda Calculus it's based on, it exists in the abstract. Lisp (see Paul Graham) was originally intended as a theoretical exercise to "define a more convenient alternative to the Turing Machine". "Lisp was a piece of theory that unexpectedly got turned into a programming language."
Practically speaking macros allow:
- Domain specific languages (bottom-up design in the words of Paul Graham) Lisp makes no assumption how to orient to this. Through its macro system, Lisp does not assume what syntax, features or functions will be necessary for the problem domain. Like list comprehensions from Python or Haskell, write a macro No need to wait for syntax extensions (e.g. waiting for Oracle to add for each semantics to Java) the language is naturally scultped to the problem.
design patterns, higher levels of abstraction
Patterns mean "I have run out of language." - Rich Hickey
compile time computing / metaprogramming/ code generation & transformation Other languages have metaprogramming techniques, but they're neither as reliable nor powerful as Lisp (C and C++ examples here; indeed, if they were as good, they would actually have to be a Lisp!) "To add object-orientation to C, Bjarne Stroustrup had to write a whole front-end called cfront which processed C++ into a mess of C that could be compiled. Lispers could do the same thing with macros (CLOS), without having to do the kind of parsing cfront had to do. Creating an OO extension to Lisp using something like message-passing is a single chapter textbook exercise using macros." (link)
"Common Lisp macros are to C++ templates what poetry is to IRS tax forms."
- Christian Schafmeister
Other's opinions on why bother with Lisp
Problems
Lisp remains an unpopular language (relatively speaking). It will never be a good career move to invest in this language as far as I can tell. I cannot understand why, save for a black swan event, Python is so popular and Common Lisp (or any industrial strength Lisp) is so fringe. (See some old speed comparisons, among other things, from Peter Norvig)
Why isn't Lisp more popular?
- The curse of Lisp
- No advocating group (Benevolent Dictator For Life, Oracle like company, etc.)
- Worse is better
- Linguistic imperialism (ALGOL descendence) Why is English the lingua franca of our time? (Why is the by-word for such a thing called lingua franca)
- The Mathew principle (Libraries, learning resources): The ecosystem is worse than the dominating languages => Vicious cycle, not enough resources (libraries, learning resources etc.) exist because not enough people are using the language; no one wants to use the language because there aren't enough resources. Everything is give and take however – an advantage of this situation is that the resources that do exist are very high quality. The people who are active in the commuinity aren't freshly minted devs from a coding bootcamped looking for their senior position in JS.
- Not everyone can be Jedis
Outro
Lisp has assisted a number of our most gifted fellow humans in thinking previously impossible thoughts.
- Edsger Dijkstra
As talked about in more detail in the already referenced essay Revenge of the Nerds by Paul Graham, "Over time, the default language, embodied in a succession of popular languages, has gradually evolved toward Lisp" Most modern languages (or the predominately used subsets of those languages) look more like Lisp than not. Python, despite its Algol inherited syntax is more Lisp-like than its actual ancestor. (Even more on the nose, as discussed in the above essay, modern Fortran is more Lisp like than Fortan like; Fortran and Lisp representing the big evolutionary bifurcation in programming languages)
Lisp hasn't so much died out as melded with modern languages (I think this is the more common evolutionary trajectory rather than extinction. eurasian humans are 1-4% Neanderthal supposedly, the original Blub languages (Fortran) being the Neanderthals of course). Still, macros, as discussed toward the beginning, are a structural dilineation. Many languages have higher order functions, garbage collection, recursion and so on (see "Revenge of the Nerds"), but they can't cross that threshold without becoming and isomorphism to lisp, another dialect.
As such, there are still unique rewards to studying Lisp (philosophical and technical). There are also associated costs to be paid for using uncommon/ non-"standard" technologes, emphatically, there is no practical or commerical application to learning Lisp. Hoever, for me the juice was worth the sequeeze:
- A different perspective / breaking out of the ALGOL world (C, C++, Java, C# and Python personally) Beyond syntax, macro/ metaprogramming is so different than the paradigms I had seen before.
- A better ability to code using recursion as an iterative technique.
(As long as a compiler is tail call optimized (See this excellent explaination if TCO is confusing to you), I think recursion should be used more often; it often fits the problem better, is smaller, less mental/ complexity management and is almost always more elegant)
- Deeper understanding of functional programming/ closure oriented programming (Lisp is not strictly functional. But there is a reason why s-expressions resemble Lambda calculus so much and a lot of resources are written in a fucntional style)
- Life long Emacs
These points won't spin the world the other way, but they have made me a better programmer and this was the goal all along.
Good luck, have fun!
Note on Common Lisp
My main reasons for choosing CL between the dominant Lisps available (i.e. Common Lisp, Clojure, Scheme and Emacs Lisp) as it was the easiest to set up & seemed to have most resources (libraries, learning materials etc.). I think Scheme would have been better to start with in hindsight.
CL is a language explicitly designed around being able to write macros (Lisp-1 vs. Lisp-2), and this is an initial complexity hurdle coming from the Algol/C/Java world. CL is also an industrial grade languae with a lot of batteries included. The CL standard library is built up using macros and macros can superficially violate syntax rules (or the lack of syntax rules). When you're told while learning that everything is a function return (s-expression)/ basically lambda calculus, it seems initially inexplicable to see standard library functions in CL.
Clojure carries some FP complexities (laziness, immutable data etc) that adds to the cognitive overhead of strictly trying to learn a lisp
Thus, Scheme seems like the optimal first lisp. The syntax carry-over will be fairly seamless after the initial hurdle of s-expressions, then the additional, dialect specific complexities can be added incrementally
Resources
Setup:
Emacs + SLIME + SBCL Emacs is a text editor that is itself a Lisp system (Lisp intepreter that just happens to emphasize text editing) This makes the editor as customizable and programmable as any other Lisp system; and most editor extensions are in fact Emacs Lisp programs that get loaded into a running editor to add their functionalit
SLIME is a Lisp development tool that comes in two parts: one is an Emacs Lisp program, the other is a Common Lisp program. The two halves communicate via a protocol called SWANK and provide lots of helpful features that make Lisp development easier (function signature completions for exmaple) (SLIME: Emacs addon, a client that sends commands from Emacs to a Common Lisp language server (SWANK); SWANK executes client commands, running on your chosen Common Lisp e.g. SBCL)
(Clojure has an equivalent environment in the form of CIDER)
Reading
My recommended learning resources
- Programming Algorithms in Lisp (probably start with this, crash course at beginning)
- Practical Common Lisp (then this)
- ANSI Common Lisp (then this, feel free to swim around)
- Common Lisp Recipes
- On Lisp (Recommended to start understanding macros)
- Let over Lambda (Pretty intense, "On Lisp" is a prerequisite)
Collected learning resources
- The Structure and Interpretation of Computer Programs
- Paradigms of Artificial Intelligence
- The Schemer Series (Little, Seasoned, Reasoned)
- Essentials of Programming Languages
- Concrete Abstractions
Lisp compilers and interpreters
- Lisp in Small Pieces
- Lisp from Nothing
Misc
Vsevold Dyomkin (Author of "Programming Algorithms in Lisp" above)
Great collection of articles on macros https://malisper.me/
Compile time computing https://medium.com/@MartinCracauer/a-gentle-introduction-to-compile-time-computing-part-1-d4d96099cea0
The Roots of Lisp https://www.paulgraham.com/rootsofLisp.html
Lisp in 99 lines of C https://github.com/Robert-van-Engelen/tinyLisp
Erik Naggum: man, myth, legend.
Videos
Exercism recap of Lisp and various Lisp dialects Baggers Bagger's Lispy OpenGL – CEPL (These videos, specifically "Little bits of Lisp" were what initially got me over the setup/ starting hurdle. Thank you baggers)
Collected Opinions of others
https://beautifulracket.com/appendix/why-racket-why-lisp.html https://www.reddit.com/r/lisp/comments/maagi2/main_reasons_for_a_programmer_to_try_their_hand/
Lisp, Smalltalk & the power of Symmetry
Giga-post from reddit
https://groups.google.com/g/comp.lang.Lisp/c/oSslA8mJdho?pli=1 Google for Lisp at the JPL, https://news.ycombinator.com/item?id=2212211 https://thenewstack.io/nasa-programmer-remembers-debugging-Lisp-in-deep-space/ A Lisp implementation running on a spacecraft where a bug was debugged live on the craft using a remote repl guerilla Lisp opus. https://groups.google.com/g/comp.lang.Lisp/c/HULKDUj_mBA/m/-UKK60tFz4YJ Characterize your problem in an abstract syntax that you make up as you go and suddenly you end up with a powerful compiler Google cbaggers and cepl Live coding open GL graphics programming. He even wrote his slide presentation software in it so that he can live code in front of you while giving presentations https://youtu.be/PqwuIfl-G1w Live coding with music I've heard that in Lisp environments of old you were able to live code a GUI. So imagine while you're developing an application it gets complicated enough that you load a database and make changes to the data and click all these menus and now you want to debug some function. With Lisp you can load the whole state of the application get to that menu and experiment with and debug the dialog you're clicking on. You can keep the application state alive while iterating on your dialog without having to close the program and open it again and set up the state each time you want to make a change and test. You can even get live feedback from stakeholders as you demonstrate your app. This extends from the idea that you have a terminal that allows you to make changes to a running program. Imagine that you have a code path that turns out to be begging for memoization. You can iterate and test live in the app with A copy of production data, and when you're satisfied with your memoization implementation you can swap the functions definition with yours that calls the original whenever it needs a new value while the program is running and suddenly your change is live. Is this a great way to develop software at scale? Maybe not, there is a trade-off to be made when it is easy to make changes in a rebel that you forget to capture, it can be a little confusing to back your way out and save the changes you tested. Getting around this is just a matter of learning good practices. Try doing this in any other language. When you consider that it took languages like C++ and Java years and years and years to add things like foreach loops, not only was it out of the box in Lisp to begin with but it becomes a simple macro. You can build object oriented language support using macros alone, and while it might not necessarily be fast, look what happened when someone added object oriented programming support for C. It became the abortion you see today. Along the same lines, since common list has a standard and the standard hasn't been updated since it was released, you don't have to worry about the Python 2 to Python 3 problem, you don't have to wait for a new standard to come out to support some new functionality, you don't have to deal with the various versions of C++, code that was written 20 years ago will just work. https://atlas.engineer/technical-article/why-Lisp.org C++ template meta programming is truly contrived and dizzying. It is a powerful tool that is very complex. And it basically constitutes a whole new syntax for compile time optimization and programming. Meanwhile in Lisp the language of compile time computing is the same as runtime computing. It's the same language, it just runs at a different time. Once you're familiar with programming and Lisp you are then familiar with how to program code that runs a compile time, yes they're slightly more involved in understanding the mental model however the learning curve is significantly less steep. How many different languages do you need in order to write a C++ program? Make/cmake/catkin/colcon/Conan/ninja just to get it to build in link, The language itself, The m4 macro pre-processor, and the cluster that is template metaprogramming. In Common Lisp, it is Lisp all the way down. Try writing a binary serialization framework in C++ without resorting to code generation in Python. I think this was only finally possible in C++ 17 if that. This is why I love this language Edit: I forgot to mention syntax. In JavaScript do you frequently use every closing brace and semicolon when ending a function that has a lambda callback. A long time ago I asked Ron Garret, the author of the JPL article, about his experience in more depth. To paraphrase he said I don't consider myself a very strong programmer, so with Lisp I was able to do great things with reduced cognitive load. 2nd Edit: oh! And typing! If you turn on optimization support, SBCL will give you very relevant hints about what it can and cannot optimize for you and how you can go about rewriting things and specifying types so that it can do the optimization for you. Every Common Lisp implementation supports generating disassembly for a given function so you can see the effects of your changes. You can put types where you need them, when you need them, and don't have to worry about potentially getting stuck later down the road because you had to decide about types before you knew more about your domain problems. Oh, and multiple inheritance with multiple dispatch. Design and object hierarchy that allows you to represent the sound that is made between two objects that are struck together. Does one object take the other in a "clap" method? If you have a drum, a table, and a door on one side, and a drumstick, a drinking glass, and your knuckles on the other, try to write something that allows you to represent the sound between all of the combinations of these. Maybe you can work something out like that, but then what happens when you want to combine the sound of three different objects?.... With multiple dispatch you can design the objects independently and have a method that dispatches on the types of both inputs. Write one generic function, clap, that takes two arguments. The method that is called is dispatched on the types of both arguments. So you have (clap drum hand) and (clap knuckle door). None of these classes needs to know anything about the others. And frankly don't necessarily need to have a unified interface among them either, each method combination can use the specific implementations of those objects,. The objects don't need to be related in any way shape or form although they often would be because of the nature of the domain. If you suddenly have a new object somebody wrote from some other library that isn't related to your class hierarchy, as long as it gives you the information necessary to determine how to resolve the clap, just write that specific method. Can't do that in C++ Edit 3:. When you've got the running image the way you want, you can dump the image to disk. This saves the data you've been messing with, functions and variables etc. Os resource handles go away of course. But if you have a lot of compile time code that makes startup take a while, your can do this work once and save the image. When you want too use it again you load it up and everything is the way you left it (with exceptions). Going beyond this, your can change the function the is called at image startup. Instead of being the repl, it can be for own "main" function. Suddenly, you have a self contained shippable binary. Can't do that in Python Edit 4: when you can bring all of the above to bear as a configuration language for your editor (Emacs) you get wonders like magit mode for git and org mode etc. I mean come on! Every editor can be extended in some way. But the fact that Emacs is basically itself an (Emacs) Lisp program means that it reads Lisp code as both configuration state as well as functionality. You even get a repl, where you can modify the state of the editor by evaluating Lisp code. And guess what? You can DEBUG Emacs as you use it! Add breakpoints on a specific function on exit, or error, and you land in a back trace you can step through, navigate to relevant code, etc,. WHILE USING THE APP. That's the power of Lisp baby Edit X: I was trying to process a large amount of data on a server and the link was slow. I ended up pushing ccl to the server and fired up swank over ssh. I connected slime to the ssh tunnel and coded up my processing remotely. I read small bits of the data stream in, experimented with how I wanted to process it. Once I realized I needed to pull in some Lisp libraries from quickLisp, I issued Quickload on my machine, configured swank to start, dumped image, shipped it, and then connected over ssh. All the dependencies are in the image. I got my reports, and just for fun, dumped the image so that the main report loop ran when executed, and viola, my coworkers could rerun my program themselves. Do this in any other language. In Python the furthest you get are virtual envs to match specific dependencies between systems, but forget about the rest This experience is unparalleled as far as I know. I'm happy to be proven wrong. Edit: and SIMD just landed on SBCL! https://www.sbcl.org/manual/index.html#sb_002dsimd Edit: added links Edit: How can I forget DART https://en.wikipedia.org/wiki/Dynamic_Analysis_and_Replanning_Tool 10 weeks from 0 to working prototype to revamp Army logistics leading up to Operation Desert Storm. I remember reading somewhere that it can be argued that the operation was delayed until the working prototype was successfully demonstrated, due to the fact that there had been disastrous logistics issues driving the need for the tool. The development process alone looks like an early version of agile done right, and also within 4 years the cost savings to the DoD had already more than paid back the investment in AI made over the prior 30 years.
Gary Hollis via Quora
I started learning C++ to do physics research. While learning C++, I explored computer science in general and came across lots of praise and advice to try functional programming and Common Lisp specifically. I read Paul Graham’s articles, got one of his books, and started excitedly exploring Common Lisp. What I found was that the learning curve of Common Lisp for me as beginner programmer was so steep that I quit studying it fairly quickly and went back to focusing on C++ to get work done. After a few years of getting skilled at C++, I became interested in functional programming again and stumbled on Haskell, which I excitedly studied and used to solve some interesting mathematical problems and play with competitive programming challenges. I then got the idea to use Haskell for physics research, since it had some clear advantages and the potential for easy parallelization, and my physics research was full of embarrassingly parallel problems that would be easy to write as reduction-map-filter combinations. However, at the time I was exploring Haskell, my benchmarks put Haskell a few factors too slow for practical use in analyzing data. No doubt an expert could have massaged Haskell into running in an acceptable time, but from the stats I saw at the time it would have cost at least 2x run time performance, and when it already took a weekend to get back results, I did not like the prospect of a 4-day wait time to get results back just so I could use a cool new language. So yet again I was back to C++, and I used it until I stumbled on the old Lisp book I bought years ago. I had the idea to explore Common Lisp again, having reached a higher level of programming ability, and this time it stuck. I was using Lisp to solve problems and even get below the 2x performance gap with C++, which incentivized more exploration. Ultimately I built a data analysis framework that leveraged Lisp’s metaprogramming and code-as-data features to a good degree, leading to a kind of Make-like programming environment where targets can be defined separately and then merged into parallel computing tasks. Once I ran into a final performance wall, I added a module to this Make-like system that would generate, compile, and run C++ code for those combinations of tasks that needed to run on a compute cluster with high efficiency. This system included automatic job submission, download, and resubmit in the case of errors with the compute cluster job management system, so that to me, it didn’t matter that C++ code was generated and sent remotely to execute; it might as well have been executed locally as some kind of Lisp operation. To conclude, I highly recommend Common Lisp as a language. It provides abilities that no other language has yet to fully emulate, but it also has a very steep learning curve compared to other options. I don’t recommend committing to scaling that learning curve for a beginner for the same reason I don’t recommend a new hiker trying to climb Everest. Take a look at it, and explore the base of the summit as much as you like when you’re not training to climb progressively higher hills and take longer hikes. You’ll have a better feeling about Everest once you’ve trained adequately for it.