Best OS my opinion

I have tried quite a few os/s on the AP and honestly the best preforming is Peppermint. Low memory requirements, very responsive and best of all fits easily on the emmc with everything you need at install. Now if I can get the darned hdmi audio to work it'll be perfect. So if you haven't tried it yet you'll be pleasantly suprised.

Lubuntu 18.04 (LTS) works well.
TBH anything that uses lxde is snappy, xfce would be my next choice (tried both with Stretch). Anything Gnome or KDE based is going to feel slow. MATE might vary from person to person.
Tho I haven't tried it, I'm happy with Lubuntu running Linuxcnc on my API, Puppy Linux might be worth a try. Salix usually works ok on low spec boxes.

I'm going for Lubuntu 20.04 server.
No need for a gui for my project.

If you don't need a GUI, why Lubuntu and not plain Ubuntu?

Because I'm most at home with Lubuntu.
Lubuntu has proven itself to be just as reliable as Ubuntu.
Every time you go with a smaller distro, there always will be performance or compatibility issues.
That being said, peppermint is a derivative of Lubuntu, which is a derivative of Ubuntu, which is a derivative of Debian.

I did end up with the full desktop, but doing 'sudo init 3' will unload the desktop, in case I need 150 more MB of RAM.

Aside from Peppermint using X11 as the dekstop interface, underneath they're pretty identical.

Edit: I just tried installing the 32bit version, for some tests, and it's not installing on the A-Pi!
Trying out the 64 bit version now...