...if you aren't a data center selling VPSes for your clients, or if your workload isn't tailored for VMs.

Everyone uses Docker nowadays, but Proxmox doesn't natively support Docker. But you do have three different solutions on how to run Docker in it.

• Installing a Linux OS in a VM and running Docker in it. This is the recommended solution by Proxmox.
• Running Docker in a LXC container. This is not recommended, but it does work. However, if you are using ZFS, you need to install fuse-overlayfs in the container, and some people have reported that this solution can cause the Proxmox host to lock up due to deadlocks.
• Running Docker on the Proxmox host itself. This is super not recommended since you should keep the hypervisor layer "clean".

I always thought that this was a super wtf move, "why wouldn't they support Docker? Everyone uses it nowadays!"

And recently, after using Proxmox since 2017... I understood why.

Keep in mind that I'm not a sysadmin that wants to manage a fleet of VMs, I'm a developer that wants to run my applications. I started using Proxmox because mmolitor87 on /r/admincraft recommended me to use Proxmox because "it makes your life sooo much easier since you can containerize your Minecraft servers!"

So in the beginning, Proxmox was pretty nice! I could containerize my Minecraft's servers into different LXC containers: BungeeCord, Lobby and Survival, sweet!

But tech stacks change, and after so many years using Proxmox, my workload has since shifted from mostly LXC containers to Docker containers, rarely touching VMs, making my Proxmox experience go from "fun!" to hell on earth, and only recently I found out that maybe Proxmox wasn't meant for me.

Proxmox's target demographic isn't "people that want to run Docker containers", Proxmox is tailored for people with mostly VM workloads (example: you are selling VPS on your data center) that are running their own data center (so they have separate uplinks for private network X public network). This is what Proxmox (the company) focuses on.

...and then we have home labbers, which it is technically your own smol data center, heh.

In the data center demographic, it makes sense why you wouldn't want to run Docker on your hypervisor: Are you going to run untrusted client code in your hypervisor? Are you crazy??

But home labbers may want to run Docker, but alas, Proxmox target audience isn't home labbers.

When you look at Proxmox, you think "wow such nice features!!" but...

• High Availability: Only useful if you have shared storage, and if you are renting servers from OVH/Hetzner/etc where you don't have a dedicated private network for your servers, then it is not worth it for you. Besides, if you are using Docker you can use Docker Swarm or Kubernetes for high availability.
• Live Migration: Only useful if you have shared storage AND you are using VMs.
• Ceph: I never got it to work, it always fails to install and, the only time I was able to get it to work, the health status was "degraded" from the get go.
• Backups: While backups with Proxmox Backup Server are super cool, sometimes I have issues where some of my LXC containers always fails to get backed up. Couldn't find a solution for this.

  root@saiyan-dreamer:/home# vzdump 40001
  INFO: starting new backup job: vzdump 40001
  INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp2634212_40001 for temporary files
  INFO: Starting Backup of VM 40001 (lxc)
  INFO: Backup started at 2022-11-21 10:55:51
  INFO: status = running
  INFO: CT Name: YASMR
  INFO: including mount point rootfs ('/') in backup
  INFO: mode failure - some volumes do not support snapshots
  INFO: trying 'suspend' mode instead
  INFO: backup mode: suspend
  INFO: ionice priority: 7
  INFO: CT Name: YASMR
  INFO: including mount point rootfs ('/') in backup
  INFO: starting first sync /proc/10745/root/ to /var/tmp/vzdumptmp2634212_40001
  ERROR: Backup of VM 40001 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/10745/root//./ /var/tmp/vzdumptmp2634212_40001' failed: exit code 23
  INFO: Failed at 2022-11-21 10:56:50
  INFO: Backup job finished with errors
  

• Docker in LXC:
  • Do you like using non-recommended methods of running Docker container within a LXC container??? Even tho multiple people have already reported that using ZFS + Docker + fuse-overlayfs can cause Proxmox to lock up??? I never had the issue (as far as I know) but it is kinda "meh" having multiple LXCs containers with the sole purpose of running Docker within them, would be better to cut off the middle man (LXC containers) altogether.
• Docker in VMs:
  • Are you going to run a VM for each container? Then you are throwing a lot of resources away since you are virtualizing an entire OS just to run Docker.
  • Are you going to run a single VM for all containers? Then you lose the flexibility you have in Proxmox for backups and live migration, so you need to fallback to the manual way of doing this... But if you are going to do it the manual way, why not just run Docker on bare metal?
  • Of course, VMs does provide better security for your Docker containers, which is something that the Proxmox team hammers about in the forums every time someone talks about running Docker... but if you think about it, they hammer this point because Proxmox was targeted more for people selling VPSes, or big enterprise business that are already virtualizing everything, so it makes sense beacuse "Are you going to run untrusted client code..."... okay you get the point.
  • But for me, or someone that wants to run their own trusted applications in a server? What's the point of having "more security" and a more cumbersome setup, if I already trust my own applications?

I was having too many issues with it, the most recent one is that any DDoS attack on my dedi server that saturated the entire bandwidth caused corosync to simply stop working and use 200% CPU on every node.

And it gets even worse! Every node stopped being able to communicate between each other until I restarted all dedicated servers. Yes, even if the dedicated server didn't have anything to do with the attack, it got the good old "no quorum???".

root@doge-reborn:~# pct start 20000
cluster not ready - no quorum?

Why don't you quorum with some bitches then /s

I got so fed up with this that I've just decided to remove the nodes from the cluster, so I don't need to restart all dedis just to fix this issue.

After that last issue, I made the decision to migrate my dedicated servers to Ubuntu Server. Most of my workloads in Proxmox were already Docker containers within LXC containers anyway, and for anything stateful I can use LXC containers in LXD, or even QEMU VMs in LXD.

If you are planning to use Proxmox, first analyze what workloads are you going to run:

• Are you a Docker fanatic that runs everything in Docker?
  • Install Ubuntu Server, Debian, insert your favorite linux distro here and Docker, yay!
• Are you going to mostly run Docker, but you also want to containerize entire OSes?
  • Install Ubuntu Server, Debian, insert your favorite linux distro here, Docker and LXD. If you want something more "cult", you can even use systemd-nspawn. You can even migrate your LXC containers to LXD, there are some guides about it on the internet.
• Are you going to mostly run Docker, but you also want to containerize entire OSes and run a few VMs?
  • Install Ubuntu Server, Debian, insert your favorite linux distro here, Docker and LXD. Heck, if you prefer you can use QEMU directly too.
• Are you going to run mostly VMs?
  • Go with Proxmox!

Nothing in Proxmox has a special magic sauce that makes it tick, LXC containers and QEMU are available on every Linux distro, so if you aren't used any of the advanced Proxmox features that requires Proxmox, just go with a normal Linux distro.

---

---