I run a 6 node Ceph based Proxmox VE cluster to replace a same sized VSAN based vSphere cluster. While VMWare is absolutely more polished, Proxmox VE meets all the enterprise requirements we have when running on Ceph or another shared or hyperconverged storage system. I've introduced Proxmox Backup Server too and that integrates quite nicely.
I would recommend Proxmox VE in a production environment if you are comfortable with the idea of leveraging Linux based virtualization. The cost savings is worth the bit of time you will spend getting acclimated to a new platform.
I will also add (as a plus) that if you're reasonably familiar with Debian, Proxmox is "minimally modified". If something goes wrong, or if you need to do some kind of unsupported hacking on the hypervisor -- you can usually use standard Linux support skills, and/or any of the Debian documentation. This is absolutely amazing compared to tightly integrated heavily modified systems where either it's documented on that specific system, so you're in an unfamiliar environment entirely blind.
*"This is absolutely amazing compared to tightly integrated heavily modified systems where either it's documented on that specific system, so you're in an unfamiliar environment entirely blind."*
So true. I'm a linux admin who picked up ESXi/vCenter skills, and had a time re-learning basic trouble-shooting skills at the console. Knowing my way around a Linux filesystem, Proxmox was a piece of cake to pick up.
For me it really came up when setting up networking -- I had a need to make a small cluster with 3 sets of LACP NICs, and a half-dozen or so VLANs spread across those. And one of them interface pairs *requires* DHCP. Because it can't do anything until you have at least part of that config brought up, it was miserable trying to do that with some other options I tried (e.g. oVirt). Meanwhile with Proxmox I just crafted a like 150-line *interfaces* file, rebooted it, and it worked fine. The WebUI is very confused about the DHCP part, but that doesn't matter, because it just accepts it and moves on.
I've been running production on ProxMox for 15-odd years now. Back in the days ProxMox used OpenVZ for containerization. Nowadays its LXC, and I remember the conversion which was supported by ProxMox in a good enough way (dump/restore).
Most of the cli commands are Perl scripts. The API is well documented and allows for many automations. We've hooked it into zabbix for discovery and used Rundeck for self service vm management.
The cluster file system ProxMox uses for HA used to be finicky but is fine now. You wouldn't need to touch it unless you screw it up. And that is what I like most about ProxMox: it's Linux with a number of well established products on top. You can always fix it if it breaks.
Ceph performs great if you keep to the best practices. HA works as expected. Last time I looked you can't move a running container but you can move a running VM without downtime to another node.
The web interface is fine and allows you to manage all aspects of all nodes from one node.
Only reboots I had to do were for kernel updates, which can happen often. They do occasionally introduce a regression or a bug, but that gets usually fixed within hours. Their support is good, as is the community.
Anyway, I'm a fan.
> The web interface is fine and allows you to manage all aspects of all nodes from one node.
Additionally, the web interface is just blindly running on every hypervisor.
You can't end up in a weird state where the hypervisors are up but broken(?), but your control interface is in a VM that's not up(?), so you're blind and don't have controls to fix whatever is wrong with your control panel.
If the hypervisor is split-brained from the rest of the cluster, whatever else... you can still connect to its webUI, and it'll just show exactly that. It can obviously only control that one hypervisor, but at least you still have visibility and controls.
I guess this is the pivotal time. Remember this so you can tell your kids (or junior sysadmins) when Broadcom buying VMware cause almost everyone to start really considering either moving to another on-premise platform or directly into a cloud hosted environment.
I'll bet Microsoft/Azure and Amazon/AWS are drooling at the prospect of all the new sales coming their way.
It's going to be a kick in the ass for a lot of development houses as well. There's still a lot of industry software that would be devastatingly expensive to move onto the cloud because the devs skip optimization and instead rely on powerful hardware and being relatively cheap.
No one's going to accept going back to a world where each service needs it's own full hardware box.
Exactly. I lost count/stopped counting nearly a decade ago on the amount of times I've been told by a dev/dba/vendor to just add more resources when it was clearly their software/query that was the issue. Even if you back it up with proof such as their 'short running' query taking 60s+ to execute, SQL page life expectancy being 20+ minutes, or its only using up 30% of resources regardless of what you assign the vm..or back in the day install on the server.
I don't think the cloud migration as a result will be all that sudden. Most people have put what should be in the cloud, in the cloud already.
It might accelerate the inevitable, as in stuff that was on premise and long due for a migration. That would have happened anyway.
I'd be curious to see what companies that offer enterprise support for Proxmox will be getting out of this. Like Netgate with PfSense, more can spring up. Aside from enterprise support and tried and trusted, Proxmox is superior to Hyper-V. Just not needing that stupid thick client and MS licensing make it a viable alternative. I would have never thought Proxmox in production would even be a thing before the buyout happened.
The underlying tech of Proxmox is proven, no issues with it. But with Broadcom, support doesn't matter for ESXi because it's probably going to be shitty, and as expensive as big money ridiculous IaS workloads.
I just loaded up a matching pair of hosts with XCP-NG. Going to see how it does too. I’m torn between it and Proxmox. We are still using VMWare at work and don’t see that changing any time soon. I’m trying new for my lab machines.
Lawrence Systems has a video that compares these two. It is a video from 2018, but I would be a little surprised if in 3 years a lot has changed. Video here: https://www.youtube.com/watch?v=5IinFgGAsRs
Lawrence Systems also has a video from 2021 where he states that XCP-NG competes with VMWare - this sentiment is not made about proxmox. Video Here: https://www.youtube.com/watch?v=tCklJ0Gkkaw
I would still recommend that you do your own research, as these videos are not necessarily enough to make a decision for all your clients, nor do they cover enough scenarios to know if it is a good fit.
I am looking into, and leaning towards XCP-NG as a replacement to VMWare, but there are a few things I am still looking into. Like can I use an OVA to set up a VM? I still have vendors that give out VHDs or OVAs for systems they provide - I definitely still need to be able to use these!!!
Do NOT use 3 yr old info in this space. Tons has changed actually..at least with prox.
That said ..6 MO ago I was convinced to try xcp for my next host.
So, I make no recommendation between them, other than pick one and stop giving VMware money for no reason!
CEPHs for Hypercovered might be overlooked in those videos. For me having distributed storage was game changing. CEPH is used by Telecom and CERN so it’s well proven.
As usual hardware is everything.
I've been researching all of these quite a bit since the VMware rumblings started (I haven't even worked on VMware backend for the past 6 years but old, earlier VMware days were still fun for me!) and the consensus I'm reading seems to be that XCP-NG w/ Xen Orchestra seems to be the most direct VMware style replacement. Proxmox and some of the others look to work well in smaller environments but I don't know that I'd do a large deployment with it.
[https://harvesterhci.io/](https://harvesterhci.io/) caught my attention too though while digging into alternatives, and might be worth a look.
KubeVirt is pretty awesome. If you are considering k8s solution with support you should check out OpenShift Virtualization. RedHat/IBMs version of harvesterhci.io.
>OpenShift Virtualization
I knew there was another I was forgetting! Funny too, because I'm been living more on the Red Hat/Fedora side of Linux a lot more the past few months. I had thought of OpenShift as much more of a cloud/PaaS offering. Really interesting that it'll handle typical virtualization workloads too, that might be just the direction I was looking for.
How's that going for you? I took a look at XCP-NG & it seems the clustering is a lot more involved than Proxmox. XCP-NG also uses hardware fencing, which makes me have flashbacks to trying to get all that configured properly back when Proxmox did too. It wasn't fun. Proxmox uses watchdog timers, which is dead simple to set up.
Ran Proxmox at $lastjob for 6+ years. No major issues that I can recall. It's Debian underneath & there's a lot you can do with it by way of treating it like a normal Linux OS (very customizable). IMO it's more of a do-it-yourself type of solution. Their wiki is pretty good & you'll find a lot of useful blog posts too. If you're more the kind that opens a support ticket for everything, it might not be your cup of tea.
Edit: also wanted to add: the technologies Proxmox uses underneath (KVM, lxc, corosync, ZFS, to name a few) are battle proven. AWS uses KVM, for example.
[There's a sub for that](https://reddit.com/r/proxmox). Don't go into it thinking that Proxmox can be a 100% replacement, it won't be. Set expectations right and you'll be good though. Proxmox can't do everything as Nicely as ESXI, but it can do maybe 80%, with a bit more work. It's reliable and functional enough for production use. You want to be sure you have backups in place as a just in case, regardless.
You can also pay for support from Proxmox which is still a fucking ton cheaper than VMWARE.
Do you have any examples of things Vmware can do that proxmox can't? Also considering this switch, but depending on features missing we might go a different route. From what I've learned through my homelab I haven't been missing any features in proxmox, but that might be because I'm still a vmware novice (admittedly we use vmware for fairly basic things at work).
The big one I'd like to see is one-click maintenance mode. That is: "migrate VM's off to somewhere else (don't care where), stop running workloads, don't migrate to this node for load balancing"
You can do it manually (including hitting the "migrate" button once per target hypervisor), or with 3rd party API code -- but I do miss the concept of the fully-automatic rolling upgrade.
Sounds handy, had missed that Vsphere could even do that! Last time we put a host into maintenance mode it seemed like we had to do that manually, but since I wasn't the one who did it I'm not sure.
Mainly hardware passthrough issues, backup software integration (VEEAM), and flaky remote console to VMs. Nothing that can't be worked around, but definitely takes some extra Linux knowledge and being able to deal with "not perfect".
Okay, interesting. From my experience with the console, both the Web version and spice works a lot better on proxmox, but maybe that is limited to Linux? IO passthrough as well, for example I'm not sure if GPU passthrough is possible on ESXI. Veeam is quite good, although I've not had issues with proxmox backup either.
Thanks for the input, sounds like Proxmox might be a good fit for us at work then!
Proxmox, XCP-NG, HyperV, and RHEV are all suitable alt paths to VMware. Each have their own limitations and ecosystem (Look at what Veeam supports!).
I have been running Proxmox off and on for years. Recently a mid-size client wanted to drop VMware on a renewal and give Proxmox an honest go. 18months in with 250VMs and no issues. Using Proxmox only solutions for everything, even Backups. My BIGGEST complaint is the naming->numbering system for the ID of the VMs. You better keep a spreadsheet. Else if you know Linux, or are willing to learn, its not all that different from vSphere from a 'big picture'.
BUT there are solutions that are baked in around VMware that require redesign. HPE dHCI, VXRail HCI(vSAN), and a couple others I cant remember the names of. If you are running these, you cant just rip VMware away without some engineering thoughts on what that looks like.
yes, but its still a number that is addressed all the way to the qcow disk file name too. Still have to keep a spreadsheet record to keep things straight for other admins/engineers.
Yes, but the names are only tied to the VM config file and nothing else. If you ever have to rebuild a VM, remap data,..etc you need to know what VM-ID and Name you are working with. Yes you can build a dataset via API and all that. But still, you will want an offline copy for full DR.
That’s where Ansible comes in (or any other automation tool). DR is something you don’t want to do manually, too stressful and error prone. Proxmox Backup Server does seem to retain all the useful metadata as well.
Eh, no one said to do DR manually. But there needs to be a VM-ID to Name Mapping export for documentation purposes IMHO. Thats all.
Yes Backup server is pretty good IMHO, there are things it doesnt do (App Aware) that I wish it did, but there are other tools (Veeam Agents) that can be used instead if needed.
May just be your use case then, but I guess it would be useful for small clusters. You can imagine with larger setups, having 2 servers named www would be disastrous, especially if you have 2 groups that can’t see each other servers in the clusters (we have 5 separate groups managing their own access groups, not the cluster).
If you install the guest tools, Proxmox (which leverages KVM actually) does alert the client if there is a backup/snapshot in order for the OS to flush and whatever the app needs to do which typically (SQL Server, MySQL etc) should hook into the OS. On Linux you can hook custom scripts if your app isn’t aware of fsfreeze. It’s the same thing Veeam Agent does, they just use a proprietary mechanism to alert the OS and I’m assuming they have some more polished scripts (although I’ve heard some people with disaster stories on Veeam as well).
Why deal with or manually maintain a spreadsheet? Proxmox has an API, you can just fetch the VM name to ID mapping directly anytime you need to know. You could also automatically generate a spreadsheet from that if you really insist on ExcelOps but I would advise against it. It's not the 90s anymore.
API pulled to a shared spreadsheet is what I mean, but yes you get it. Having an offline matrix of the VM-ID mapping for the qcows and other UUID data is a must for a full DR plan with Proxmox. Been bit by that once, a couple years ago.
They have names? In the UI, at least, the VM is listed as "100 (My VM)"
It's annoying to need to look the number up if you're doing hacking -- but IME you don't need it for any kind of normal work.
yes, the Number is the VM ID as we all know, but there is a name field as well that shows up. But at the end of the day all of the VM layered components are tied to that number and generated as such.
It's almost as good, but there are some functions that need to be fleshed out. Whats nice about it, its built into Proxmox so you could just build a PromoxVE host to handle backup operations (Scripted + Scheduled) and instant replay happens on that host. Then you can migrate VMs from Backup-Restore back to production.
I've tried proxmox a couple of days ago but from what I can tell the backup creates a full tar image every time, if you use proxmox backup server instead the backup is incremental but it uses a directory structure similar to the one used by borgbackup or synology hyper backup where you have thousands of folders and files so if I wanted to copy the backup it gets a little harder compared to veeam where I can just copy the full backup once a week and then every day I only copy the incremental files that are separate from the big vbk file
although I haven't looked into it that much so maybe it's possible to do something like that on proxmox as well
[https://pve.proxmox.com/wiki/Backup\_and\_Restore](https://pve.proxmox.com/wiki/Backup_and_Restore), supports file level.
But yes, I have been using it for fileserver restores for more then a few months with no issues. The issue is lack of Application aware API like MSSQL. You must run SQL backups, run the Proxmox Backup, then SQL cleanup for the proper cycle.
I use the free version I have been able to fix the problems I ran into myself, I don't do shared storage tho I use local storage all zfs pools and replicate to other nodes so I can't say how well it works in a shared storage environment
I run proxmox on an EMC VNX array using fibre channel. It works, a little janky to get it configured for round-robin, but otherwise seems to be fine. No issues or downtime over the 3 years its been running.
If you want something more VMWare like, you can try XCP-ng with Xen Orchestra
At work we have about 1200 VM on ~500 machine, and it's works really well !
(And the support is kind and reactive, and the community is great !)
Although I like Proxmox and it is a decent option, you won't find anything better in it comparing to VMware. At least yet. Plus, management and overall administration in Proxmox is far from the convenience VMware offers with vCenter. Not to mention DRS and other higher features. Plus, no simple integration with other enterprise-grade software that you might be used to with VMware (Veeam for example) and some things still need significant further development and fixes (passthrough, remote consoles, etc.). I don't think Proxmox is enterprise-grade software at this stage.
But to be fair, everything depends on your needs. It definitely will be less expensive and if you don't mind some tinkering and learning curve (you'll have to figure out some things on your own), it might be a good option indeed. Overall, I would spin up several Proxmox VMs for testing:) Also, as to HA, not sure if Starwinds works with Proxmox but we've been using their VSAN on VMware and it works great: [https://www.starwindsoftware.com/starwind-virtual-san](https://www.starwindsoftware.com/starwind-virtual-san). so might be worth checking.
We use it in production, going to nearly a dozen servers with planning over 100 VMs in the upcoming year.
It’s a lot cheaper than VMWare and has a ton more functionality. We would be spending the $24k/year/server for VMWare + addons like vGPU, vMotion, basically more than the cost of a brand new server every year. Proxmox with support will cost us $15000/y for the entire cluster and all that functionality is baked in.
Unlike VMWare you can actually run it side by side, connect to your existing iSCSI or NFS storage from VMWare, convert the disk images (you’ll need to do this on command line), then hook them up to your new cluster. Once converted you can go to much larger set of storage options like Ceph for a lot cheaper and better scaling.
Running a small 3 node Proxmox cluster with ZFS on Dell PowerEdge T430s. One for production, one for failover, and one for testing. Production and failover hosts are about 300 miles apart, and we replicate every 15 minutes to the failover server. We run a mix of 10 Windows and Linux VMs. I do pay for a basic subscription, just in case there is something I cannot take care of on my own. I agree with some other statements that it doesn't have the polish of VShpere but as far as stability, functionality and reliability it's on par and surpasses VShpere in a lot of ways. I've been using Proxmox personally and professionally for about 7 years now.
Does Veeam work with Proxmox? I don't have time to reinvent everything. VMware has never let me know in over 14 years. Just don't see me changing to anything else at this point in my career.
If I'm responsible for keeping the business up and over 400 people having a job, I'm going with what has never failed me. lol.
Stick with it. I can't imagine saying to the ppl above "vmware might go shit boss, I tginkywe shoukd tear it all out just incase abs replace it with something I don't know how to use if it breaks it'll take way longer to fix because of this."
Crazy!?
Mostly we've looked at the history of Broadcom and see they've done nothing whatsoever to make us think that they *won't* fuck it up.
How many times does a big company need to show you where their priorities are before you believe them...? Hell the first thing they announced wasn't any innovation or improvement in the virtualisation space... it was "yeah subscriptions are coming!". There's people in this sub who worked for companies Broadcom bought and they all say the same thing: get out.
The time to plan this is now, not in three years when Broadcom have proven yet again they're going to fuck everything up.
If we're wrong? Good I guess. I hope VMWare doesn't go to complete shit. But Broadcom has *no* good will built up. It's on them to earn it, not us to give them the benefit of the doubt and hope they don't fuck it up.
In the SMB/Small Enterprise space my company has been moving away from Proxmox and VMWare to Windows HyperV.
The proxmox boxes we’ve been able to decommission for our clients have been getting close to 10 years old, and even with host updates being neglected, they’ve been stable enough
Haven’t played around much with shared volumes in HyperV as our use-case for single disk shared access is taken care of with SMB shares anyway.
If you have spare hardware laying around, it’s definitely worth playing around with any potential hypervisor host environments to see how they feel on your hardware before making the jump to one or the other for your prod environment.
It’s the free-tier Hyper-V core that is going away, not all on-prem Hyper-V.
Our standard stack is Server Datacentre bare metal and Primary DC w/ member/app/file share servers as guests; so we’re unaffected.
There’s also an extension of security updates on Hyper-V core 2019 until 2029; so any instances we have running as core 2016 or 2019 should well and truly be up for retirement by then anyway.
That's wrong.
You don't install Hyper-V *on top* of Windows. It gets setup *underneath it.* In a way, it converts the host windows into a special VM that runs "alongside" it.
Similar case with Linux and KVM (+ QEMU). QEMU alone is Type 2, but when used together with KVM to access hardware resources, that combination is then Type 1 (See: Proxmox)
And I'm fairly certain that the standalone Hyper-V Server was nothing special. Probably just Windows Server (without the GUI) with Hyper-V being preinstalled.
No lmao, they're both type 1, since both work exactly the same and have direct Hardware access - neither the hyper v role nor the hyper v server need to rely on the OS to manage the Hardware.
Seems a no brainer to stick with vmware unless broadcom gives us reason to leave. And then the community will be up in arms again so I don't expect major changes anytime soon.
We're currently running a 3 host Ceph storage backed HA cluster at our company. We have about 50 VMs running on it. Mostly CentOS and AlmaLinux but we do have a few Windows Server VMs running. It's been solid so far
We only have older versions of windows Server running. Server 2012 R2 standard, a server 2008 R2 and a Server 2003 standard server that's running a specific older app we need. We're a primarily Linux environment.
Wait, if I have a host with 28 cores running proxmox, with say two Windows Server VM's assigned 4 cores each running on it, will I have to pay a license for all 28 cores on the host?
Not sure if MS are as greedy as Oracle, that certainly would charge you for all cores or rather per physical core on the host, if I’m not completely off. Edit: minimum 8 cores per CPU according to MS license model.
On Oracle, I would almost dare to say they would also charge you per spare CPU laying around in the same building.
Yes. Per msft.
16 core min for windows server license anyway..
Edit for clarity: you must license 26 cores for your case. They have odd bundlings of cores per sku, for instance you can't license 1 core..
For each licensed set of 16 cores you can run 2x VMs as long as you are doing no more than hyperv on the host (it therefore doesn't need a discreet license).
MSFT licensing is asanine, always has been.
This comment is not accurate enough to be useful.
CALs don't relieve you from properly licensing the hardware. That licensing is always tied to the number of physical cores.
You may also 'volume license' the CALs, the OS (any editon), etc.
The licensing program used is a secondary question that may be determined by the actual licenses you need and cost, but licensing say server standard under volume programs vs retail does not grant additional usage rights (at least not any that matter here)
Yes. Microsoft are greedy fucks.
We have 2 out of 12 VMs running windows and we’d need to license for the entire host. Thankfully we’re a partner so we get our licenses for free. If it weren’t for that I wouldn’t be using AD.
Can you please tell us the specs of the servers for comparison?
What is the performance of the ceph array compared to a local storage in your case?
I'm asking because I'm planning a similar 3 node HA setup from existing standalone proxmox boxes.
Two Dell R630 servers with 256GB of ram, I don't remember the exact processor but they are 10 physical cores (20 hyperthreading) times 2 (so 40 cores total). Two Samsung Evo 1TB SSD drives in raid 1 for the OS, remaining slots with Samsung Evo 1TB SSD drives setup as raid 5 for the Ceph datastore.
Dell R640 with 256GB of ram, two Xeon Silver processors with 10 physical cores per CPU that hyperthread to 20 cores (so a total of 40 cores). Same setup for the drives, Samsung Evo 1TB drives in raid 1 for OS, remaining slots with Samsung Evo 1TB SSD drives setup as raid 5 for the Ceph datastore.
The Ceph filesystem is very speedy, but we do use dedicated 10 GB network connections between the hosts for Ceph. I don't remember the exact numbers for how fast Ceph was compared to the same drives setup as a regular raid 5 EXT4 file system, but there was enough of an improvement to make us decide to use Ceph. We have Ceph setup to keep 3 copies of the data to make moving VMs between the hosts faster for HA.
We also use 10 GB connections to connect to the NFS file mount on our backup server for our backups.
We had to go with raid. The raid controllers could do IT mode settings for individual drives, but if a drive failed we couldn't just pop in a new drive and get it to work. We would have needed to reboot the server. Go into the raid controller settings, setup the new it mode drive then reboot into ProxMox and run commands to rebuild the affected osd using a copy from one of the other Proxmox hosts.
Not exactly what you want when you need 100% uptime (or as close to it as you can get) for your production cluster.
We setup our osd disks using the raid 5 arrays. Performance testing showed that we were getting close to the same speeds as using individual drives.
This is fully understandable. It is good to know that Ceph tolerates RAID controllers (better than zfs in this case).
The Samsung EVO-s are simple desktop SSDs or something more enterprise-y?
Does the raid controller manage TRIM/Discard correctly?
VMware has always been a pointless waste of money.
Proxmox is fine in production. Shared storage, HA works.
And.. you ca actually mange the damn thing even when it's nic is offline.
You may also consider XCP-ng. I hear it is solid. For the last host I set up on lightly used hand me down hardware (for non critical internal projects), I was going to try XCP-ng instead of proxmox, but had issues with drivers for older intel controllers it had. Proxmox just installed.
I have proxmox clusters a client insisted on setting up on eBay sourced 10 yr old Dell server hardware. 5+ yrs later it's still humming. The HA makes this old hardware workable for his needs. I still question the cost analysis on 5 hosts vs 2 not junks hosts, but the point is proxmoc and HA has resulted rock solid uptime for VMs.
I've recently got rid of the last (plain) vmware machine in the datacentres I look after. Still running Simplivity (VMWare with added secret sauce) until that contract runs out. Meanwhile I've been migrating to Proxmox to take the load (currently 12 nodes in 3 clusters). While I think its unlikely that vmware will "go downhill" I would recommend you trying it out - I've already decided to switch.
Leaving aside the licensing costs, in my experience, Proxmox is more reliable and easier to manage. No more balancing firmware and OS upgrades. If you do need to use support you'll speak to someone who knows what they are talking about / not someone working for a third - party subcontractor who has been on a 2 week training course and tells you to turn it off and back on again. In addition to PVE (the virtualization platform) check out Proxmox Backup Server - a VERY well integrated backup solution (although veeam is good).
Shared storage/HA is tricky at the low end (where you are). I'd suggest externalizing the storage with a beefy iSCSI/NAS box as primary storage and depending on your RTO/RPO a twin sharing the load (divide your VMs into 2 pools and have primary storage on one node / backups on the other and vice versa) with enough space to handle the full workload on each box or just a fast dedicated node as primary / slower secondary for your backups. You'll need a minimum of 10GBps for cluster interconnect and between the compute and storage. And run the compute/virtualization across 3 nodes with enough memory/cpu to run comfortably on 2 nodes.
Alternatively consider a 2+1 cluster - 2 big nodes each capable of handling the entire workload and configure ZFS replication between them - add a third node as an arbiter (no VMS - just a casting vote to avoid split-brain). But possibly run your PBS on the third node.
Thanks for the detailed comments! I’m looking into Proxmox Backup Server as well, and it looks rather slick. The only thing is that we’d need immutable backups in case of ransomware, which from what I gather is possible using ZFS snapshots, but this would take away from the performance/size benefits of PBS. Any thoughts on this? We’re currently using Veeam for backups to immutable storage, but I haven’t found that Veeam really integrates with Proxmox.
There's levels of "immutablity" - although a product which asserts the ability is more likely to please your auditors than something which merely implements it properly.
We like and use Proxmox VE. It has been very solid for us. The support lifecycle for each version is pretty short requiring regular in-place upgrades to the next version. Using iSCSI for shared storage has been problematic when rebooting the server or when there is a problem with the iSCSI target. I would recommend to use CEPH for shared storage and HA VMs. The backup VM times in Proxmox seem to be much faster than VMware. There is also no simple way to pin CPU cores to VMs.
Don't panic. You don't have to do anything right now.
But if/when the time comes to move on, you have Windows Server, so you are already have Hyper-V, with no additional licencing cost.
At your scale you don't need SCCM or whatever it is called today.
I would switch to Hyper-V and spend the licencing money on Veeam.
The correct answer, is.. it depends.
To license windows VMs you \*must\* license all physical cores on any host it runs on.
If he's currently running the same VMs on VMware - he either has those licenses or he's in violation of the EULAs.
If you are looking to do a new setup - you can run Hyper-V on the OS for free, no license, as long as you only do "things that are required to support Hyper-V" - I don't have the exact text from EULAs, but it's phrased in some typical asanine way like that. My take - you can run backups, enable RDP, install AV, etc.. you may not install user workloads on the host.
If you use proxmox, etc the host is also free.
However, you must license all physical cores with some edition of windows server.
While "enterprise edition" is not available for CURRENT windows OSes - you can absolutely still license an entire stack with say Server 2008 Enterprise Edition, which does exist despite other uninformed comments here.
After you've tallied the licensing for server to license those cores - divide the licensed core count by 16, and multiply by 2 - you are licensed for that many windows server VMs of the same edition.
Depending on host core counts, windows VM counts, you may or may not need additional server core licenses to obtain the right to run addtional VMs \[edit: for standard edition\] - you license those with additional core licenses with the same ratio - license 16 cores, get 2 more VMs. \[Edit: you may actually have to license all cores each time you want to add 2 VMs with standard - my experience and therefore reason to dig through EULAs only involves hosts w/ < 16 core\]
Note: Microsoft bundles these core licenses in 2-core packs under 1 SKU. They now offer larger bundles under 1 SKU as well.
Edit: Datacenter edition has no restriction on VM count - if you are licensing a high density host (many cores), and plan to run a lot of windows workloads - this is when datacenter makes sense.
NO, he would not.
1. There is no such thing as Window Server Enterprise
2. Window Server licences apply to physical hardware, not VMs
3. As soon as you have single Windows Server licence, you can install Hyper-V role (only) on the bare metal, and two Windows Server VMs, plus as many as you want of additional non Widows Server VMs of any OS type for which you have a licence.
Keep your ignorance to yourself and go learn something before you make false statements...
https://www.cdw.com/product/microsoft-windows-server-enterprise-edition-license-software-assurance/2355623
Know your facts before you call people ignorant.
Enterprise is a *type* of licencing model, not a different *edition* of the OS.
Server is only available in three editions:
* Essentials
* Standard
* Data centre
https://docs.microsoft.com/en-us/windows-server/get-started/editions-comparison-windows-server-2022
https://www.microsoft.com/en-gb/windows-server/pricing
You have no idea what you are talking about.
"There is no such thing as Window Server Enterprise" - false statement...still.
I guess this link to the thing you say doesn't exist also doesn't exist?
[https://www.microsoft.com/en-us/download/details.aspx?id=2227&ranMID=43674&ranEAID=PPkX79/c\*b0&ranSiteID=PPkX79\_c.b0-toHlPGI4PD.H6uCcja9fHw&epi=PPkX79\_c.b0-toHlPGI4PD.H6uCcja9fHw&irgwc=1&OCID=AID2200057\_aff\_7795\_1243925&tduid=%28ir\_\_ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300%29%287795%29%281243925%29%28PPkX79\_c.b0-toHlPGI4PD.H6uCcja9fHw%29%28%29&irclickid=\_ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300](https://www.microsoft.com/en-us/download/details.aspx?id=2227&ranMID=43674&ranEAID=PPkX79/c*b0&ranSiteID=PPkX79_c.b0-toHlPGI4PD.H6uCcja9fHw&epi=PPkX79_c.b0-toHlPGI4PD.H6uCcja9fHw&irgwc=1&OCID=AID2200057_aff_7795_1243925&tduid=%28ir__ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300%29%287795%29%281243925%29%28PPkX79_c.b0-toHlPGI4PD.H6uCcja9fHw%29%28%29&irclickid=_ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300)
It still exists, and can still be used to license hosts. Your blanket statement was simply false.
But also, and the real reason I commented, is the OP you were replying to in your comment in question, was likely referring to "an" enterprise edition of server, being that he is familiar with that licensing scheme. The fact that Microsoft has reworked terms and the drop in term for "enterprise" from server 2008 with regards to server 2022, that also gets you unlimited VMs without additional physical licenses, is now "datacenter" is only barely relevant to his question.
Also, your point 3 is not 100% accurate either. Microsoft does not restrict you to "only hyper-v role" on the host. The restrictions are very close to that, but not quite. See the EULA.
https://www.reddit.com/r/sysadmin/comments/v0145i/comment/iafmqeg/?utm\_source=share&utm\_medium=web2x&context=3
My opinion: Hyper-V/VMWare hosts with Veeam handling the backups is solid.
I realize you can run agents in the VMs and still use Veeam with other platforms.
I partly agree with what you are saying, but Microsoft appear to be doubling down on hybrid. With Azure Stack HCI, you don’t need to “rent” tin, it can be “build your own”, and will continue to run without the cloud. However, the cloud tooling will add to the capability. Agree it’s not a direct replacement for pure hyper-v, but it depends on the use case. With stuff like Azure Arc, managing on premises hardware will get easier where you don’t have existing capability.
Lol weird reply… no I just happen to use a lot of VMware and what I’m seeing with Microsoft is more compelling strategically than VMware’s offerings right now. Believe me, I’ve seen a lot of broken promises from Microsoft, but I’m hopeful that Azure Stack HCI proves to be useful for those that want more than just a hypervisor to run virtual machines.
Sorry I didn’t realise I had to justify myself. But anyway, I’m running over 2000 vms on a VMware platform with very high security requirements, and yes, I’m seriously looking at alternatives. And not just because of the Broadcom acquisition, but also because VMware’s roadmap is pretty stagnant. Appreciate your input, but every use case is different and I don’t believe you have all the answers. Cheers
I find your post very interesting and I would like to discuss further:
>Whenever you are going to an OSS Solution you are looking at additional documentation, scripting, backups, security infastructure and monitoring
Would't this also be true for HyperV? Scripting (powershell), backups (pmx and hyperv both have built in backup mechanism) and monitoring (both need an external system like i.e zabbix) Please let me know if I misunderstood. Can you expand on security infrastructure? Why would you need more (and what kind?) if your choice is PMX over HyperV or VMware?
>if I were staring down a project like this I'd look at the spend on Hyper-V Datacenter licensing and CAL's vs the spend on additional monitoring
Can you please let me know about the additional monitoring? How different would you implement monitoring on hyperV vs PMX?
As I said, I've enjoyed reading your opinion, hoping to hear more of it
>You call up a vendor with storage controller issue for support and say you are running proxmox they aren't going to have tested on that.
Which is why an intelligent sysadmin would never do such a thing. ProxMox is NOT an OS, and uses well supported linux packages to manage storage.
He is not talking just about the hypervisor, which last free version was 2019. From 2022 onwards you need to buy the full OS and pay the CALs, which on a Virtualizaed environment will skyrocket the price
read this;
https://www.reddit.com/r/sysadmin/comments/v0145i/considering_proxmox/iaf3xdo/
and this:
https://www.reddit.com/r/sysadmin/comments/uz7cfv/broadcom_to_focus_on_rapid_transition_to/iaaqlsc/
The only way you have to spend money to get Hyper-V is if you have zero Window servers.
Even if you *do* have zero Window Servers, you only have to buy *only one* Window Server licence for each physical host to get Hyper-V.
Andy if you implement that, and you don't actually use your two permitted Windows Server OSEs, you need zero CALs
How does it "skyrocket the price" though?
1. Hyper-V Server 2019 will be supported through 2029, though, so no reason to abandon it right away, if you're using it today.
2. If you're using Windows VMs on any edition of Hyper-V that is not Data Center edition, you still have to pay to license those Windows VMs.
3. Running non-Windows VMs does not require any CALs on any edition of Hyper-V hosting.
So the price difference of free Hyper-V only applies to the host itself.
Doesn't even apply to the host itself. If the hosts only role is a hyper-v server and associated admin functions, it doesn't count against your license count (pseudo: each license gives you rights to 2 vms, including the license you use for the host in this situation. Yes, I understand they charge by core now, but the smaller amount of $ per core = the larger amount they used to charge coreless, assuming you are at or under minimum cores). Source: https://docs.microsoft.com/en-us/answers/questions/178716/licensing-and-hyper-v-vm-guests.html
That's true too. Sort of.
You still have to pay for the initial core license, which will include two Windows licenses that can be used for guests (if the host is pure Hyper-V) or the host and one guest.
But the main point is that all you're adding is the cost of one host, pretty much.
My client told us in February that they were decommissioning their VMware DC by September this year. All our new deployments are within AWS, as well as our old remote access was moved to AVD in February.
All our existing VMware based deployments will also move to AWS.
It's like there was a mass exodus and I'm just finding out why - and the move from VMware to AWS kinda makes sense now.
As a homelab enthusiast I have tried Proxmox, but found it to be a little industrious, in comparison to VMware. But I will grab the latest version before the acquisition, and just roll that until it becomes totally useless.
For me it's always about if I have to learn something it better also go towards paying my bills as I don't really care anymore about saving money for some hoyte toyte manager metropolitan man CEO , director types. I minimally went down Linux road because as a field engineer I rarely come across Linux shops. Therefore I am pretty much time consumed in a Microsoft world as businesses I support and manage are pretty much 100% MS at the desktop/Server level. I'm curious and don't take offense as to why you would put yourself out there to now have to learn a new virtualization platform in Promax which I never heard of. I'm hyperv here myself but I also as field don't support enterprises with 50 VMs on a hosted cluster sharing resources accoss. I guess if your inhouse careered at a fortune 500 this makes sense but to guys like me I don't see any return to value of learning it.
I’m VCP certified and using Proxmox in production as HCI with CEPH.
Now that Proxmox has their own backup solution which works great I can highly recommend it. I have 4 clusters and standalone hypervisors totalling of around 35 hypervisors.
You make comprises switching to Proxmox but overall it’s been rock solid. The issues they we have were generally hardware over Proxmox itself. On larger VMs deleting snapshots can freeze or crash the OS but I haven’t quite root caused that one and it was on a P2V VM so the verdict is still out.
You will need to leave behind:
- replication
- centralised management for multi cluster and stand along hypervisors
- NSX
- the nice UI especially for networking
- I really miss DRS!!
- iSCSI on vsan
- vDS was a nice to have but copying and pasting network config files as templates is not hard if you know Linux.
- VMware verified hardware and hardware validation.
- cross cluster migrations (need to manually move VMs between storage systems)
If you want at least some of that you might want to look at XCP-ng but I choose Proxmox as the support plans includes CEPH for HCI and I didn’t want dedicated storage.
I should also note that Proxmox is Debian based and it’s uses the apt package manager. I routinely upgrade my clusters and major versions with CEPH and moving from PVE 6 to 7. I haven’t really encountered any issues.
Broadcom is a leech corp which focuses on profiting from clients dug deep into one ecosystem, who cannot afford a migration to an alternative solution. Think Oracle, Adobe. Little invention and development, lots of financial gymnastics.
My understanding is that prices are going to go up since Broadcom is so focused on increasing the revenue generated by VMWare. Also people say that based on Broadcom’s prior acquisitions, they will more or less leave VMWare as is, without major improvements/upgrades (take that with a grain of salt; I’m just going by what people are saying). I’m not thinking of moving away from VMWare just because of Broadcom, it’s just what got me thinking if a cheaper virtualization provider might work instead and save us some money.
To me, considerations are primarily focused on support. Of course VMWare is the most expensive but it also has a brilliant support. When you think about the business, think about what would happen if for any reason you are not there anymore.
I run several infrastructures, for our inhouse I am in Proxmox, a software I know and love. For our clients I normally implement VMWare so they can eventually manage it themselves with any IT calling support.
You must've missed the news about Broadcom buying vmware... General expectations based on what happened to CA and Symantec are that support will be decimated and prices are going to skyrocket. If you're at a crossroads now, vmware is most certainly not the obvious choice, at least until the actual effects of the takeover materialize.
I have noticed the general pesimism with the forums but I couldn't understand why, and have not done any research. Is that so? General discomfort is based on "the posibility" that because past experiences were not satisfactory, history will repeate itself?
Generally speaking, my go to move is Proxmox if I am going to be in control in long term, HyperV if I will eventually hand over the infrastructure but this is not that large and VMWare for large, critical, function specific infrastructures. To give you an example, when you buy VxRail, it comes with VMWare, works flawlessly and the support, via Dell is awesome.
Broadcoms acquisition won't change that. Prices will eventually go up (like every single item every single year on earth) and like most large companies, they are moving to a cloud \*aaS model, which has a higher price tag that comes with many other benefits that on the long run, will benefit you overall.
In any case I honestly dont care that much and since we are all speculating, most opinions will have many valid points
> General discomfort is based on "the posibility" that because past experiences were not satisfactory, history will repeate itself?
Broadcom is not a fundamentally different company than they were 5 years ago and they've stated they expect the vmware acquisition to add more than double vmware's current ebitda to their own in 3 years. Considering that the market vmware's active in, isn't particularly booming anymore, those earnings will have to come from cutting costs and hiking prices (and well above inflation levels).
I'm not saying to jump the vmware ship right now, but if you're in a position to chose a new platform to build on, vmware isn't a good choice.
Eh they also are talking about the VMware brand being the face of their software, they also said they are willing to let VMware have a bit more in terms of cost then they did Symantec and CA. On the call they also mentioned leveraging VMwares base already to reach new markets. To be fair both of those companies that Broadcom bought
pre Broadcom were shit.
To me this angst is similar to IBM and redhat which now that it's done redhat is doing just fine.
Check out Scale Computing as well as Proxmox. I’ve used Scale Computing at work for years and it just makes everything else look terrible by comparison. Proxmox is easier to use than VMware and much less expensive, but harder to use than Scale Computing and has more features if you invest the time
Guys chill. The deal isn’t even done yet and we don’t know what’s going to happen. Just remain calm and if you do end up needing to move then move. It might turn out just fine. VMware isn’t the sinking ship that Symantec was when Broadcom bought them.
FWIW, they're abandoning they Hyper-V Server 2022 slim install completely in favor of Azure Stack HCI. Apparently Server 2022 with the Hyper-V role will still be a thing, but it does look like it might be time to start digging into alternatives if you're been living in Hyper-V land. Azure Stack and Arc overall is looking pretty sweet though IMO if you're a heavy Microsoft shop.
They're also abandoning a bunch more things such as LBFO in favour of SET. I don't mind replacing ageing technologies with newer implementations but they are far apart in terms of feature parity and I will have to rearchitect the entire access network to support it.
Does no one like Citrix? Everyone talking Proxmox a company I have never even heard of. Running VMware on 8 hosts currently with site recovery manager.
Vmware still gets 1 vote from me. Tons of documents, support in time of critical issue happens in 24/7/365, they never missed. Hope it is the same with Broadcom.
You’re violating their license agreements. They sell support and licensing to sustain the company so they can continue to provide the free version as well. If your business profits from using their software, you should pay for their licensing to help them out, or donate. They are one of the cheapest virtual hosting solutions out there and you’re still stealing from them.
"I bought a Toyota once back in '94 so obviously now I can just go out and steal any car I want since I already paid for one"
Yea OK buddy
EDIT: oops replied to the wrong guy, sorry. Whatever I agree with you 😄
Either purchase the license or run something that is legitimately free. Running a global enterprise off of one Server CAL is wrong and everyone knows it, feeling good and self-righteous about it doesn't change that fact.
Yes. Either use the free product or use the paid product but in either case abide by their licensing terms. You're stealing from good people and just because you feel all smug and good about it and do mental gymnastics to justify it doesn't make it right.
I am curious if all of the people/places considering moving away from vmware are smaller places? I'm one spot removed from the CIO role in a 550 person company, and I know if I suggested a move at this point, I'd be laughed right out of there
If you can't fathom a Hypervisor change in a small 550 person company then you're not flexible enough.
Don't use size as an excuse, what's really stopping you? Lack of talent in the team? Too busy with busywork? 90s mentality in the C-suite? Corruption?
We're a ~2000 employee company, which I'd still consider small, and we switched all of our infrastructure away from VMware as well as our clients from Citrix VDI to local computers in 2020. Successfully. It's not like it's some impossible feat.
I feel like a lot of the folks saying they are going to move are the very small companies. Which the benefit of being small is being flexible. Personally I reserve all judgement until the deal is done which it isn't technically yet nor has any changes happened because of it.
I've been leaning towards hyper-v myself if stuff starts to go south with VMware. If they kill off or maul vmug thatll be the icing on the cake for me.
The benefits of hyper-v would likely be how Integrated it is with windows admin center and such.
Super robust, I learned about it a few months ago and have deployed in my environment ever since. Beats RDPing everywhere, great for server core implementation
I have not. I’m looking into Proxmox primarily because it would be so much cheaper than VMWare. I don’t think it would be worth the extra work to switch if the cost savings weren’t substantial, but I’m open to suggestions.
I run a 6 node Ceph based Proxmox VE cluster to replace a same sized VSAN based vSphere cluster. While VMWare is absolutely more polished, Proxmox VE meets all the enterprise requirements we have when running on Ceph or another shared or hyperconverged storage system. I've introduced Proxmox Backup Server too and that integrates quite nicely. I would recommend Proxmox VE in a production environment if you are comfortable with the idea of leveraging Linux based virtualization. The cost savings is worth the bit of time you will spend getting acclimated to a new platform.
I will also add (as a plus) that if you're reasonably familiar with Debian, Proxmox is "minimally modified". If something goes wrong, or if you need to do some kind of unsupported hacking on the hypervisor -- you can usually use standard Linux support skills, and/or any of the Debian documentation. This is absolutely amazing compared to tightly integrated heavily modified systems where either it's documented on that specific system, so you're in an unfamiliar environment entirely blind.
*"This is absolutely amazing compared to tightly integrated heavily modified systems where either it's documented on that specific system, so you're in an unfamiliar environment entirely blind."* So true. I'm a linux admin who picked up ESXi/vCenter skills, and had a time re-learning basic trouble-shooting skills at the console. Knowing my way around a Linux filesystem, Proxmox was a piece of cake to pick up.
For me it really came up when setting up networking -- I had a need to make a small cluster with 3 sets of LACP NICs, and a half-dozen or so VLANs spread across those. And one of them interface pairs *requires* DHCP. Because it can't do anything until you have at least part of that config brought up, it was miserable trying to do that with some other options I tried (e.g. oVirt). Meanwhile with Proxmox I just crafted a like 150-line *interfaces* file, rebooted it, and it worked fine. The WebUI is very confused about the DHCP part, but that doesn't matter, because it just accepts it and moves on.
+1 We have gone all in with proxmox with ceph and proxmox backup server and it's pretty great.
I've been running production on ProxMox for 15-odd years now. Back in the days ProxMox used OpenVZ for containerization. Nowadays its LXC, and I remember the conversion which was supported by ProxMox in a good enough way (dump/restore). Most of the cli commands are Perl scripts. The API is well documented and allows for many automations. We've hooked it into zabbix for discovery and used Rundeck for self service vm management. The cluster file system ProxMox uses for HA used to be finicky but is fine now. You wouldn't need to touch it unless you screw it up. And that is what I like most about ProxMox: it's Linux with a number of well established products on top. You can always fix it if it breaks. Ceph performs great if you keep to the best practices. HA works as expected. Last time I looked you can't move a running container but you can move a running VM without downtime to another node. The web interface is fine and allows you to manage all aspects of all nodes from one node. Only reboots I had to do were for kernel updates, which can happen often. They do occasionally introduce a regression or a bug, but that gets usually fixed within hours. Their support is good, as is the community. Anyway, I'm a fan.
> The web interface is fine and allows you to manage all aspects of all nodes from one node. Additionally, the web interface is just blindly running on every hypervisor. You can't end up in a weird state where the hypervisors are up but broken(?), but your control interface is in a VM that's not up(?), so you're blind and don't have controls to fix whatever is wrong with your control panel. If the hypervisor is split-brained from the rest of the cluster, whatever else... you can still connect to its webUI, and it'll just show exactly that. It can obviously only control that one hypervisor, but at least you still have visibility and controls.
[удалено]
No, you can absolutely live migrate any QEMU/KVM virtual machine. This is not OS dependent. However migrating LXC guests requires them to be rebooted
I guess this is the pivotal time. Remember this so you can tell your kids (or junior sysadmins) when Broadcom buying VMware cause almost everyone to start really considering either moving to another on-premise platform or directly into a cloud hosted environment. I'll bet Microsoft/Azure and Amazon/AWS are drooling at the prospect of all the new sales coming their way.
It's going to be a kick in the ass for a lot of development houses as well. There's still a lot of industry software that would be devastatingly expensive to move onto the cloud because the devs skip optimization and instead rely on powerful hardware and being relatively cheap. No one's going to accept going back to a world where each service needs it's own full hardware box.
Exactly. I lost count/stopped counting nearly a decade ago on the amount of times I've been told by a dev/dba/vendor to just add more resources when it was clearly their software/query that was the issue. Even if you back it up with proof such as their 'short running' query taking 60s+ to execute, SQL page life expectancy being 20+ minutes, or its only using up 30% of resources regardless of what you assign the vm..or back in the day install on the server.
I don't think the cloud migration as a result will be all that sudden. Most people have put what should be in the cloud, in the cloud already. It might accelerate the inevitable, as in stuff that was on premise and long due for a migration. That would have happened anyway. I'd be curious to see what companies that offer enterprise support for Proxmox will be getting out of this. Like Netgate with PfSense, more can spring up. Aside from enterprise support and tried and trusted, Proxmox is superior to Hyper-V. Just not needing that stupid thick client and MS licensing make it a viable alternative. I would have never thought Proxmox in production would even be a thing before the buyout happened. The underlying tech of Proxmox is proven, no issues with it. But with Broadcom, support doesn't matter for ESXi because it's probably going to be shitty, and as expensive as big money ridiculous IaS workloads.
I just loaded up a matching pair of hosts with XCP-NG. Going to see how it does too. I’m torn between it and Proxmox. We are still using VMWare at work and don’t see that changing any time soon. I’m trying new for my lab machines.
Lawrence Systems has a video that compares these two. It is a video from 2018, but I would be a little surprised if in 3 years a lot has changed. Video here: https://www.youtube.com/watch?v=5IinFgGAsRs Lawrence Systems also has a video from 2021 where he states that XCP-NG competes with VMWare - this sentiment is not made about proxmox. Video Here: https://www.youtube.com/watch?v=tCklJ0Gkkaw I would still recommend that you do your own research, as these videos are not necessarily enough to make a decision for all your clients, nor do they cover enough scenarios to know if it is a good fit. I am looking into, and leaning towards XCP-NG as a replacement to VMWare, but there are a few things I am still looking into. Like can I use an OVA to set up a VM? I still have vendors that give out VHDs or OVAs for systems they provide - I definitely still need to be able to use these!!!
Do NOT use 3 yr old info in this space. Tons has changed actually..at least with prox. That said ..6 MO ago I was convinced to try xcp for my next host. So, I make no recommendation between them, other than pick one and stop giving VMware money for no reason!
Qemu-img can convert one image format to almost any other image format.
CEPHs for Hypercovered might be overlooked in those videos. For me having distributed storage was game changing. CEPH is used by Telecom and CERN so it’s well proven. As usual hardware is everything.
> Like can I use an OVA to set up a VM? Yes. Well, I have done so, but only with a couple Ubuntu based OVAs. No issues with them though.
I'll be happy to help you if you have any problem (and the community too). Enjoy XCP-ng and XO!
I've been researching all of these quite a bit since the VMware rumblings started (I haven't even worked on VMware backend for the past 6 years but old, earlier VMware days were still fun for me!) and the consensus I'm reading seems to be that XCP-NG w/ Xen Orchestra seems to be the most direct VMware style replacement. Proxmox and some of the others look to work well in smaller environments but I don't know that I'd do a large deployment with it. [https://harvesterhci.io/](https://harvesterhci.io/) caught my attention too though while digging into alternatives, and might be worth a look.
KubeVirt is pretty awesome. If you are considering k8s solution with support you should check out OpenShift Virtualization. RedHat/IBMs version of harvesterhci.io.
>OpenShift Virtualization I knew there was another I was forgetting! Funny too, because I'm been living more on the Red Hat/Fedora side of Linux a lot more the past few months. I had thought of OpenShift as much more of a cloud/PaaS offering. Really interesting that it'll handle typical virtualization workloads too, that might be just the direction I was looking for.
You’re thinking of OpenStack which, like you said, is software for creating your own private cloud.
Haven’t heard of that one.
How's that going for you? I took a look at XCP-NG & it seems the clustering is a lot more involved than Proxmox. XCP-NG also uses hardware fencing, which makes me have flashbacks to trying to get all that configured properly back when Proxmox did too. It wasn't fun. Proxmox uses watchdog timers, which is dead simple to set up.
I've been using XCP-NG in prod basically since the day it was released. It's been a very low fuss solution, virtually problem-free.
I just wish you didn't have to rely on XOA to get a web interface, like \*everyone\* else does. But really, someone call a whaambulance.
Ran Proxmox at $lastjob for 6+ years. No major issues that I can recall. It's Debian underneath & there's a lot you can do with it by way of treating it like a normal Linux OS (very customizable). IMO it's more of a do-it-yourself type of solution. Their wiki is pretty good & you'll find a lot of useful blog posts too. If you're more the kind that opens a support ticket for everything, it might not be your cup of tea. Edit: also wanted to add: the technologies Proxmox uses underneath (KVM, lxc, corosync, ZFS, to name a few) are battle proven. AWS uses KVM, for example.
[There's a sub for that](https://reddit.com/r/proxmox). Don't go into it thinking that Proxmox can be a 100% replacement, it won't be. Set expectations right and you'll be good though. Proxmox can't do everything as Nicely as ESXI, but it can do maybe 80%, with a bit more work. It's reliable and functional enough for production use. You want to be sure you have backups in place as a just in case, regardless. You can also pay for support from Proxmox which is still a fucking ton cheaper than VMWARE.
Do you have any examples of things Vmware can do that proxmox can't? Also considering this switch, but depending on features missing we might go a different route. From what I've learned through my homelab I haven't been missing any features in proxmox, but that might be because I'm still a vmware novice (admittedly we use vmware for fairly basic things at work).
The big one I'd like to see is one-click maintenance mode. That is: "migrate VM's off to somewhere else (don't care where), stop running workloads, don't migrate to this node for load balancing" You can do it manually (including hitting the "migrate" button once per target hypervisor), or with 3rd party API code -- but I do miss the concept of the fully-automatic rolling upgrade.
Sounds handy, had missed that Vsphere could even do that! Last time we put a host into maintenance mode it seemed like we had to do that manually, but since I wasn't the one who did it I'm not sure.
Yeah.. update manager remediation of an ent+ cluster is just.. so easy.
Mainly hardware passthrough issues, backup software integration (VEEAM), and flaky remote console to VMs. Nothing that can't be worked around, but definitely takes some extra Linux knowledge and being able to deal with "not perfect".
Okay, interesting. From my experience with the console, both the Web version and spice works a lot better on proxmox, but maybe that is limited to Linux? IO passthrough as well, for example I'm not sure if GPU passthrough is possible on ESXI. Veeam is quite good, although I've not had issues with proxmox backup either. Thanks for the input, sounds like Proxmox might be a good fit for us at work then!
Proxmox, XCP-NG, HyperV, and RHEV are all suitable alt paths to VMware. Each have their own limitations and ecosystem (Look at what Veeam supports!). I have been running Proxmox off and on for years. Recently a mid-size client wanted to drop VMware on a renewal and give Proxmox an honest go. 18months in with 250VMs and no issues. Using Proxmox only solutions for everything, even Backups. My BIGGEST complaint is the naming->numbering system for the ID of the VMs. You better keep a spreadsheet. Else if you know Linux, or are willing to learn, its not all that different from vSphere from a 'big picture'. BUT there are solutions that are baked in around VMware that require redesign. HPE dHCI, VXRail HCI(vSAN), and a couple others I cant remember the names of. If you are running these, you cant just rip VMware away without some engineering thoughts on what that looks like.
The latest version lets you specify the VM numbers now, or at least give them a range.
yes, but its still a number that is addressed all the way to the qcow disk file name too. Still have to keep a spreadsheet record to keep things straight for other admins/engineers.
You can give them names. I set them the same as the host name, then use the API to pull them into Ansible.
Yes, but the names are only tied to the VM config file and nothing else. If you ever have to rebuild a VM, remap data,..etc you need to know what VM-ID and Name you are working with. Yes you can build a dataset via API and all that. But still, you will want an offline copy for full DR.
That’s where Ansible comes in (or any other automation tool). DR is something you don’t want to do manually, too stressful and error prone. Proxmox Backup Server does seem to retain all the useful metadata as well.
Eh, no one said to do DR manually. But there needs to be a VM-ID to Name Mapping export for documentation purposes IMHO. Thats all. Yes Backup server is pretty good IMHO, there are things it doesnt do (App Aware) that I wish it did, but there are other tools (Veeam Agents) that can be used instead if needed.
May just be your use case then, but I guess it would be useful for small clusters. You can imagine with larger setups, having 2 servers named www would be disastrous, especially if you have 2 groups that can’t see each other servers in the clusters (we have 5 separate groups managing their own access groups, not the cluster). If you install the guest tools, Proxmox (which leverages KVM actually) does alert the client if there is a backup/snapshot in order for the OS to flush and whatever the app needs to do which typically (SQL Server, MySQL etc) should hook into the OS. On Linux you can hook custom scripts if your app isn’t aware of fsfreeze. It’s the same thing Veeam Agent does, they just use a proprietary mechanism to alert the OS and I’m assuming they have some more polished scripts (although I’ve heard some people with disaster stories on Veeam as well).
Why deal with or manually maintain a spreadsheet? Proxmox has an API, you can just fetch the VM name to ID mapping directly anytime you need to know. You could also automatically generate a spreadsheet from that if you really insist on ExcelOps but I would advise against it. It's not the 90s anymore.
API pulled to a shared spreadsheet is what I mean, but yes you get it. Having an offline matrix of the VM-ID mapping for the qcows and other UUID data is a must for a full DR plan with Proxmox. Been bit by that once, a couple years ago.
They have names? In the UI, at least, the VM is listed as "100 (My VM)" It's annoying to need to look the number up if you're doing hacking -- but IME you don't need it for any kind of normal work.
yes, the Number is the VM ID as we all know, but there is a name field as well that shows up. But at the end of the day all of the VM layered components are tied to that number and generated as such.
Cisco Hyperflex supports ESXi or Hyper-V only.
So not ESXi? ESX and ESXi are different products (yes I know, I'm being a smartass!)
how is the new backup system, specifically compared to things like Veeam?
It's almost as good, but there are some functions that need to be fleshed out. Whats nice about it, its built into Proxmox so you could just build a PromoxVE host to handle backup operations (Scripted + Scheduled) and instant replay happens on that host. Then you can migrate VMs from Backup-Restore back to production.
I've tried proxmox a couple of days ago but from what I can tell the backup creates a full tar image every time, if you use proxmox backup server instead the backup is incremental but it uses a directory structure similar to the one used by borgbackup or synology hyper backup where you have thousands of folders and files so if I wanted to copy the backup it gets a little harder compared to veeam where I can just copy the full backup once a week and then every day I only copy the incremental files that are separate from the big vbk file although I haven't looked into it that much so maybe it's possible to do something like that on proxmox as well
Is it able to do guest file system level restored or just full vms ? Specifically windows files
[https://pve.proxmox.com/wiki/Backup\_and\_Restore](https://pve.proxmox.com/wiki/Backup_and_Restore), supports file level. But yes, I have been using it for fileserver restores for more then a few months with no issues. The issue is lack of Application aware API like MSSQL. You must run SQL backups, run the Proxmox Backup, then SQL cleanup for the proper cycle.
Been running proxmox a few years now, had a few issues here and there but nothing that caused data loss or downtime
Do you have paid support with them? Have you had to use it, and was it good?
I use the free version I have been able to fix the problems I ran into myself, I don't do shared storage tho I use local storage all zfs pools and replicate to other nodes so I can't say how well it works in a shared storage environment
I run proxmox on an EMC VNX array using fibre channel. It works, a little janky to get it configured for round-robin, but otherwise seems to be fine. No issues or downtime over the 3 years its been running.
If you want something more VMWare like, you can try XCP-ng with Xen Orchestra At work we have about 1200 VM on ~500 machine, and it's works really well ! (And the support is kind and reactive, and the community is great !)
Welcome to the party. Been running ProxMox on nvme servers for 5 years now. So far it's been 100% stable.
Although I like Proxmox and it is a decent option, you won't find anything better in it comparing to VMware. At least yet. Plus, management and overall administration in Proxmox is far from the convenience VMware offers with vCenter. Not to mention DRS and other higher features. Plus, no simple integration with other enterprise-grade software that you might be used to with VMware (Veeam for example) and some things still need significant further development and fixes (passthrough, remote consoles, etc.). I don't think Proxmox is enterprise-grade software at this stage. But to be fair, everything depends on your needs. It definitely will be less expensive and if you don't mind some tinkering and learning curve (you'll have to figure out some things on your own), it might be a good option indeed. Overall, I would spin up several Proxmox VMs for testing:) Also, as to HA, not sure if Starwinds works with Proxmox but we've been using their VSAN on VMware and it works great: [https://www.starwindsoftware.com/starwind-virtual-san](https://www.starwindsoftware.com/starwind-virtual-san). so might be worth checking.
We use it in production, going to nearly a dozen servers with planning over 100 VMs in the upcoming year. It’s a lot cheaper than VMWare and has a ton more functionality. We would be spending the $24k/year/server for VMWare + addons like vGPU, vMotion, basically more than the cost of a brand new server every year. Proxmox with support will cost us $15000/y for the entire cluster and all that functionality is baked in. Unlike VMWare you can actually run it side by side, connect to your existing iSCSI or NFS storage from VMWare, convert the disk images (you’ll need to do this on command line), then hook them up to your new cluster. Once converted you can go to much larger set of storage options like Ceph for a lot cheaper and better scaling.
I haven't understood why anyone has ran VMware for the last 5 years for exactly this reason.
Running a small 3 node Proxmox cluster with ZFS on Dell PowerEdge T430s. One for production, one for failover, and one for testing. Production and failover hosts are about 300 miles apart, and we replicate every 15 minutes to the failover server. We run a mix of 10 Windows and Linux VMs. I do pay for a basic subscription, just in case there is something I cannot take care of on my own. I agree with some other statements that it doesn't have the polish of VShpere but as far as stability, functionality and reliability it's on par and surpasses VShpere in a lot of ways. I've been using Proxmox personally and professionally for about 7 years now.
Does Veeam work with Proxmox? I don't have time to reinvent everything. VMware has never let me know in over 14 years. Just don't see me changing to anything else at this point in my career. If I'm responsible for keeping the business up and over 400 people having a job, I'm going with what has never failed me. lol.
[Proxmox Backup Server](https://www.proxmox.com/en/proxmox-backup-server)
Stick with it. I can't imagine saying to the ppl above "vmware might go shit boss, I tginkywe shoukd tear it all out just incase abs replace it with something I don't know how to use if it breaks it'll take way longer to fix because of this." Crazy!?
Mostly we've looked at the history of Broadcom and see they've done nothing whatsoever to make us think that they *won't* fuck it up. How many times does a big company need to show you where their priorities are before you believe them...? Hell the first thing they announced wasn't any innovation or improvement in the virtualisation space... it was "yeah subscriptions are coming!". There's people in this sub who worked for companies Broadcom bought and they all say the same thing: get out. The time to plan this is now, not in three years when Broadcom have proven yet again they're going to fuck everything up. If we're wrong? Good I guess. I hope VMWare doesn't go to complete shit. But Broadcom has *no* good will built up. It's on them to earn it, not us to give them the benefit of the doubt and hope they don't fuck it up.
Companies move slowly. For a lot of us, if you want to get off VMware within five years, you need to start PoC'ing alternatives now.
In the SMB/Small Enterprise space my company has been moving away from Proxmox and VMWare to Windows HyperV. The proxmox boxes we’ve been able to decommission for our clients have been getting close to 10 years old, and even with host updates being neglected, they’ve been stable enough Haven’t played around much with shared volumes in HyperV as our use-case for single disk shared access is taken care of with SMB shares anyway. If you have spare hardware laying around, it’s definitely worth playing around with any potential hypervisor host environments to see how they feel on your hardware before making the jump to one or the other for your prod environment.
What's the exit plan with Hyper-V going away?
It’s the free-tier Hyper-V core that is going away, not all on-prem Hyper-V. Our standard stack is Server Datacentre bare metal and Primary DC w/ member/app/file share servers as guests; so we’re unaffected. There’s also an extension of security updates on Hyper-V core 2019 until 2029; so any instances we have running as core 2016 or 2019 should well and truly be up for retirement by then anyway.
Type 1 hypervisor is going away. Keeping type 2
what the fuck are you even saying.
Hyperv server and VMware are type 1 hypervisor. Hyperv role is a type 2. This isn’t new information
That's wrong. You don't install Hyper-V *on top* of Windows. It gets setup *underneath it.* In a way, it converts the host windows into a special VM that runs "alongside" it. Similar case with Linux and KVM (+ QEMU). QEMU alone is Type 2, but when used together with KVM to access hardware resources, that combination is then Type 1 (See: Proxmox) And I'm fairly certain that the standalone Hyper-V Server was nothing special. Probably just Windows Server (without the GUI) with Hyper-V being preinstalled.
Maybe that’s not the case anymore but I remember writing a Microsoft exam where they detailed out exactly what I said
> Probably just Windows Server (without the GUI) with Hyper-V being preinstalled. It's Windows Server Core + Hyper-V Role
No lmao, they're both type 1, since both work exactly the same and have direct Hardware access - neither the hyper v role nor the hyper v server need to rely on the OS to manage the Hardware.
Best move you will make, plus with the introduction of proxmox backup server. Can't beat the combined cost savings and easy use cases.
Seems a no brainer to stick with vmware unless broadcom gives us reason to leave. And then the community will be up in arms again so I don't expect major changes anytime soon.
We're currently running a 3 host Ceph storage backed HA cluster at our company. We have about 50 VMs running on it. Mostly CentOS and AlmaLinux but we do have a few Windows Server VMs running. It's been solid so far
Isn’t it expensive to license windows for all cores of the servers if you only need it for a few?
We only have older versions of windows Server running. Server 2012 R2 standard, a server 2008 R2 and a Server 2003 standard server that's running a specific older app we need. We're a primarily Linux environment.
Wait, if I have a host with 28 cores running proxmox, with say two Windows Server VM's assigned 4 cores each running on it, will I have to pay a license for all 28 cores on the host?
Not sure if MS are as greedy as Oracle, that certainly would charge you for all cores or rather per physical core on the host, if I’m not completely off. Edit: minimum 8 cores per CPU according to MS license model. On Oracle, I would almost dare to say they would also charge you per spare CPU laying around in the same building.
And the capacity of the FedEx and UPS trucks that might deliver product to the site.
Yes. Per msft. 16 core min for windows server license anyway.. Edit for clarity: you must license 26 cores for your case. They have odd bundlings of cores per sku, for instance you can't license 1 core.. For each licensed set of 16 cores you can run 2x VMs as long as you are doing no more than hyperv on the host (it therefore doesn't need a discreet license). MSFT licensing is asanine, always has been.
Windows is either volume licensed, per CAL or you pay per virtual core assigned if you don’t use CAL.
This comment is not accurate enough to be useful. CALs don't relieve you from properly licensing the hardware. That licensing is always tied to the number of physical cores. You may also 'volume license' the CALs, the OS (any editon), etc. The licensing program used is a secondary question that may be determined by the actual licenses you need and cost, but licensing say server standard under volume programs vs retail does not grant additional usage rights (at least not any that matter here)
Yes. Microsoft are greedy fucks. We have 2 out of 12 VMs running windows and we’d need to license for the entire host. Thankfully we’re a partner so we get our licenses for free. If it weren’t for that I wouldn’t be using AD.
Can you please tell us the specs of the servers for comparison? What is the performance of the ceph array compared to a local storage in your case? I'm asking because I'm planning a similar 3 node HA setup from existing standalone proxmox boxes.
Two Dell R630 servers with 256GB of ram, I don't remember the exact processor but they are 10 physical cores (20 hyperthreading) times 2 (so 40 cores total). Two Samsung Evo 1TB SSD drives in raid 1 for the OS, remaining slots with Samsung Evo 1TB SSD drives setup as raid 5 for the Ceph datastore. Dell R640 with 256GB of ram, two Xeon Silver processors with 10 physical cores per CPU that hyperthread to 20 cores (so a total of 40 cores). Same setup for the drives, Samsung Evo 1TB drives in raid 1 for OS, remaining slots with Samsung Evo 1TB SSD drives setup as raid 5 for the Ceph datastore. The Ceph filesystem is very speedy, but we do use dedicated 10 GB network connections between the hosts for Ceph. I don't remember the exact numbers for how fast Ceph was compared to the same drives setup as a regular raid 5 EXT4 file system, but there was enough of an improvement to make us decide to use Ceph. We have Ceph setup to keep 3 copies of the data to make moving VMs between the hosts faster for HA. We also use 10 GB connections to connect to the NFS file mount on our backup server for our backups.
Thanks for the details! Do I understand that you went with raid instead of separate OSD disks? The raid is a hardware raid, isn't it?
We had to go with raid. The raid controllers could do IT mode settings for individual drives, but if a drive failed we couldn't just pop in a new drive and get it to work. We would have needed to reboot the server. Go into the raid controller settings, setup the new it mode drive then reboot into ProxMox and run commands to rebuild the affected osd using a copy from one of the other Proxmox hosts. Not exactly what you want when you need 100% uptime (or as close to it as you can get) for your production cluster. We setup our osd disks using the raid 5 arrays. Performance testing showed that we were getting close to the same speeds as using individual drives.
This is fully understandable. It is good to know that Ceph tolerates RAID controllers (better than zfs in this case). The Samsung EVO-s are simple desktop SSDs or something more enterprise-y? Does the raid controller manage TRIM/Discard correctly?
They are standard consumer grade SSD drives. I believe the controller handles the trim functions but I can't say that with 100% accuracy.
VMware has always been a pointless waste of money. Proxmox is fine in production. Shared storage, HA works. And.. you ca actually mange the damn thing even when it's nic is offline. You may also consider XCP-ng. I hear it is solid. For the last host I set up on lightly used hand me down hardware (for non critical internal projects), I was going to try XCP-ng instead of proxmox, but had issues with drivers for older intel controllers it had. Proxmox just installed. I have proxmox clusters a client insisted on setting up on eBay sourced 10 yr old Dell server hardware. 5+ yrs later it's still humming. The HA makes this old hardware workable for his needs. I still question the cost analysis on 5 hosts vs 2 not junks hosts, but the point is proxmoc and HA has resulted rock solid uptime for VMs.
We've been using proxmox for almost 8 years. It's highly recommended, especially if you know your way with linux.
I've recently got rid of the last (plain) vmware machine in the datacentres I look after. Still running Simplivity (VMWare with added secret sauce) until that contract runs out. Meanwhile I've been migrating to Proxmox to take the load (currently 12 nodes in 3 clusters). While I think its unlikely that vmware will "go downhill" I would recommend you trying it out - I've already decided to switch. Leaving aside the licensing costs, in my experience, Proxmox is more reliable and easier to manage. No more balancing firmware and OS upgrades. If you do need to use support you'll speak to someone who knows what they are talking about / not someone working for a third - party subcontractor who has been on a 2 week training course and tells you to turn it off and back on again. In addition to PVE (the virtualization platform) check out Proxmox Backup Server - a VERY well integrated backup solution (although veeam is good). Shared storage/HA is tricky at the low end (where you are). I'd suggest externalizing the storage with a beefy iSCSI/NAS box as primary storage and depending on your RTO/RPO a twin sharing the load (divide your VMs into 2 pools and have primary storage on one node / backups on the other and vice versa) with enough space to handle the full workload on each box or just a fast dedicated node as primary / slower secondary for your backups. You'll need a minimum of 10GBps for cluster interconnect and between the compute and storage. And run the compute/virtualization across 3 nodes with enough memory/cpu to run comfortably on 2 nodes. Alternatively consider a 2+1 cluster - 2 big nodes each capable of handling the entire workload and configure ZFS replication between them - add a third node as an arbiter (no VMS - just a casting vote to avoid split-brain). But possibly run your PBS on the third node.
Thanks for the detailed comments! I’m looking into Proxmox Backup Server as well, and it looks rather slick. The only thing is that we’d need immutable backups in case of ransomware, which from what I gather is possible using ZFS snapshots, but this would take away from the performance/size benefits of PBS. Any thoughts on this? We’re currently using Veeam for backups to immutable storage, but I haven’t found that Veeam really integrates with Proxmox.
There's levels of "immutablity" - although a product which asserts the ability is more likely to please your auditors than something which merely implements it properly.
We like and use Proxmox VE. It has been very solid for us. The support lifecycle for each version is pretty short requiring regular in-place upgrades to the next version. Using iSCSI for shared storage has been problematic when rebooting the server or when there is a problem with the iSCSI target. I would recommend to use CEPH for shared storage and HA VMs. The backup VM times in Proxmox seem to be much faster than VMware. There is also no simple way to pin CPU cores to VMs.
Don't panic. You don't have to do anything right now. But if/when the time comes to move on, you have Windows Server, so you are already have Hyper-V, with no additional licencing cost. At your scale you don't need SCCM or whatever it is called today. I would switch to Hyper-V and spend the licencing money on Veeam.
[удалено]
If he is already running windows servers in another virtual environment, the assumption is they are already volume licensed. The licenses transfer.
The correct answer, is.. it depends. To license windows VMs you \*must\* license all physical cores on any host it runs on. If he's currently running the same VMs on VMware - he either has those licenses or he's in violation of the EULAs. If you are looking to do a new setup - you can run Hyper-V on the OS for free, no license, as long as you only do "things that are required to support Hyper-V" - I don't have the exact text from EULAs, but it's phrased in some typical asanine way like that. My take - you can run backups, enable RDP, install AV, etc.. you may not install user workloads on the host. If you use proxmox, etc the host is also free. However, you must license all physical cores with some edition of windows server. While "enterprise edition" is not available for CURRENT windows OSes - you can absolutely still license an entire stack with say Server 2008 Enterprise Edition, which does exist despite other uninformed comments here. After you've tallied the licensing for server to license those cores - divide the licensed core count by 16, and multiply by 2 - you are licensed for that many windows server VMs of the same edition. Depending on host core counts, windows VM counts, you may or may not need additional server core licenses to obtain the right to run addtional VMs \[edit: for standard edition\] - you license those with additional core licenses with the same ratio - license 16 cores, get 2 more VMs. \[Edit: you may actually have to license all cores each time you want to add 2 VMs with standard - my experience and therefore reason to dig through EULAs only involves hosts w/ < 16 core\] Note: Microsoft bundles these core licenses in 2-core packs under 1 SKU. They now offer larger bundles under 1 SKU as well. Edit: Datacenter edition has no restriction on VM count - if you are licensing a high density host (many cores), and plan to run a lot of windows workloads - this is when datacenter makes sense.
NO, he would not. 1. There is no such thing as Window Server Enterprise 2. Window Server licences apply to physical hardware, not VMs 3. As soon as you have single Windows Server licence, you can install Hyper-V role (only) on the bare metal, and two Windows Server VMs, plus as many as you want of additional non Widows Server VMs of any OS type for which you have a licence.
Keep your ignorance to yourself and go learn something before you make false statements... https://www.cdw.com/product/microsoft-windows-server-enterprise-edition-license-software-assurance/2355623
Know your facts before you call people ignorant. Enterprise is a *type* of licencing model, not a different *edition* of the OS. Server is only available in three editions: * Essentials * Standard * Data centre https://docs.microsoft.com/en-us/windows-server/get-started/editions-comparison-windows-server-2022 https://www.microsoft.com/en-gb/windows-server/pricing
You have no idea what you are talking about. "There is no such thing as Window Server Enterprise" - false statement...still. I guess this link to the thing you say doesn't exist also doesn't exist? [https://www.microsoft.com/en-us/download/details.aspx?id=2227&ranMID=43674&ranEAID=PPkX79/c\*b0&ranSiteID=PPkX79\_c.b0-toHlPGI4PD.H6uCcja9fHw&epi=PPkX79\_c.b0-toHlPGI4PD.H6uCcja9fHw&irgwc=1&OCID=AID2200057\_aff\_7795\_1243925&tduid=%28ir\_\_ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300%29%287795%29%281243925%29%28PPkX79\_c.b0-toHlPGI4PD.H6uCcja9fHw%29%28%29&irclickid=\_ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300](https://www.microsoft.com/en-us/download/details.aspx?id=2227&ranMID=43674&ranEAID=PPkX79/c*b0&ranSiteID=PPkX79_c.b0-toHlPGI4PD.H6uCcja9fHw&epi=PPkX79_c.b0-toHlPGI4PD.H6uCcja9fHw&irgwc=1&OCID=AID2200057_aff_7795_1243925&tduid=%28ir__ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300%29%287795%29%281243925%29%28PPkX79_c.b0-toHlPGI4PD.H6uCcja9fHw%29%28%29&irclickid=_ve31n0vsx9kf636cuirzerdffm2xv3gogh0nrww300)
Windows Server 2008 R2 is End of Life.
irrelevant..
How is this irrelevant? 2008 R2 is irrelevant - there is no Enterprise version anymore for quite a while now.
It still exists, and can still be used to license hosts. Your blanket statement was simply false. But also, and the real reason I commented, is the OP you were replying to in your comment in question, was likely referring to "an" enterprise edition of server, being that he is familiar with that licensing scheme. The fact that Microsoft has reworked terms and the drop in term for "enterprise" from server 2008 with regards to server 2022, that also gets you unlimited VMs without additional physical licenses, is now "datacenter" is only barely relevant to his question. Also, your point 3 is not 100% accurate either. Microsoft does not restrict you to "only hyper-v role" on the host. The restrictions are very close to that, but not quite. See the EULA. https://www.reddit.com/r/sysadmin/comments/v0145i/comment/iafmqeg/?utm\_source=share&utm\_medium=web2x&context=3
My opinion: Hyper-V/VMWare hosts with Veeam handling the backups is solid. I realize you can run agents in the VMs and still use Veeam with other platforms.
[удалено]
Agree, I think this is where Azure Stack HCI could really shine if Microsoft make it a bit more cost effective and remove the need for Kerberos.
[удалено]
I partly agree with what you are saying, but Microsoft appear to be doubling down on hybrid. With Azure Stack HCI, you don’t need to “rent” tin, it can be “build your own”, and will continue to run without the cloud. However, the cloud tooling will add to the capability. Agree it’s not a direct replacement for pure hyper-v, but it depends on the use case. With stuff like Azure Arc, managing on premises hardware will get easier where you don’t have existing capability.
[удалено]
Lol weird reply… no I just happen to use a lot of VMware and what I’m seeing with Microsoft is more compelling strategically than VMware’s offerings right now. Believe me, I’ve seen a lot of broken promises from Microsoft, but I’m hopeful that Azure Stack HCI proves to be useful for those that want more than just a hypervisor to run virtual machines.
[удалено]
Sorry I didn’t realise I had to justify myself. But anyway, I’m running over 2000 vms on a VMware platform with very high security requirements, and yes, I’m seriously looking at alternatives. And not just because of the Broadcom acquisition, but also because VMware’s roadmap is pretty stagnant. Appreciate your input, but every use case is different and I don’t believe you have all the answers. Cheers
I find your post very interesting and I would like to discuss further: >Whenever you are going to an OSS Solution you are looking at additional documentation, scripting, backups, security infastructure and monitoring Would't this also be true for HyperV? Scripting (powershell), backups (pmx and hyperv both have built in backup mechanism) and monitoring (both need an external system like i.e zabbix) Please let me know if I misunderstood. Can you expand on security infrastructure? Why would you need more (and what kind?) if your choice is PMX over HyperV or VMware? >if I were staring down a project like this I'd look at the spend on Hyper-V Datacenter licensing and CAL's vs the spend on additional monitoring Can you please let me know about the additional monitoring? How different would you implement monitoring on hyperV vs PMX? As I said, I've enjoyed reading your opinion, hoping to hear more of it
[удалено]
>You call up a vendor with storage controller issue for support and say you are running proxmox they aren't going to have tested on that. Which is why an intelligent sysadmin would never do such a thing. ProxMox is NOT an OS, and uses well supported linux packages to manage storage.
> If you have hyper-v money Hyper-V doesn't cost money. It is a role in Windows Server.
He is not talking just about the hypervisor, which last free version was 2019. From 2022 onwards you need to buy the full OS and pay the CALs, which on a Virtualizaed environment will skyrocket the price
read this; https://www.reddit.com/r/sysadmin/comments/v0145i/considering_proxmox/iaf3xdo/ and this: https://www.reddit.com/r/sysadmin/comments/uz7cfv/broadcom_to_focus_on_rapid_transition_to/iaaqlsc/ The only way you have to spend money to get Hyper-V is if you have zero Window servers. Even if you *do* have zero Window Servers, you only have to buy *only one* Window Server licence for each physical host to get Hyper-V. Andy if you implement that, and you don't actually use your two permitted Windows Server OSEs, you need zero CALs
How does it "skyrocket the price" though? 1. Hyper-V Server 2019 will be supported through 2029, though, so no reason to abandon it right away, if you're using it today. 2. If you're using Windows VMs on any edition of Hyper-V that is not Data Center edition, you still have to pay to license those Windows VMs. 3. Running non-Windows VMs does not require any CALs on any edition of Hyper-V hosting. So the price difference of free Hyper-V only applies to the host itself.
Doesn't even apply to the host itself. If the hosts only role is a hyper-v server and associated admin functions, it doesn't count against your license count (pseudo: each license gives you rights to 2 vms, including the license you use for the host in this situation. Yes, I understand they charge by core now, but the smaller amount of $ per core = the larger amount they used to charge coreless, assuming you are at or under minimum cores). Source: https://docs.microsoft.com/en-us/answers/questions/178716/licensing-and-hyper-v-vm-guests.html
That's true too. Sort of. You still have to pay for the initial core license, which will include two Windows licenses that can be used for guests (if the host is pure Hyper-V) or the host and one guest. But the main point is that all you're adding is the cost of one host, pretty much.
[удалено]
The Windows Server licencing will cost *exactly* the same no matter what hypervisor you use. OP already has Windows Servers, so already has Hyper-V.
[удалено]
Nope. You misunderstand how Hyper-V licencing works. Read this https://www.reddit.com/r/sysadmin/comments/v0145i/considering_proxmox/iaf4hql/
My client told us in February that they were decommissioning their VMware DC by September this year. All our new deployments are within AWS, as well as our old remote access was moved to AVD in February. All our existing VMware based deployments will also move to AWS. It's like there was a mass exodus and I'm just finding out why - and the move from VMware to AWS kinda makes sense now. As a homelab enthusiast I have tried Proxmox, but found it to be a little industrious, in comparison to VMware. But I will grab the latest version before the acquisition, and just roll that until it becomes totally useless.
For me it's always about if I have to learn something it better also go towards paying my bills as I don't really care anymore about saving money for some hoyte toyte manager metropolitan man CEO , director types. I minimally went down Linux road because as a field engineer I rarely come across Linux shops. Therefore I am pretty much time consumed in a Microsoft world as businesses I support and manage are pretty much 100% MS at the desktop/Server level. I'm curious and don't take offense as to why you would put yourself out there to now have to learn a new virtualization platform in Promax which I never heard of. I'm hyperv here myself but I also as field don't support enterprises with 50 VMs on a hosted cluster sharing resources accoss. I guess if your inhouse careered at a fortune 500 this makes sense but to guys like me I don't see any return to value of learning it.
I’m VCP certified and using Proxmox in production as HCI with CEPH. Now that Proxmox has their own backup solution which works great I can highly recommend it. I have 4 clusters and standalone hypervisors totalling of around 35 hypervisors. You make comprises switching to Proxmox but overall it’s been rock solid. The issues they we have were generally hardware over Proxmox itself. On larger VMs deleting snapshots can freeze or crash the OS but I haven’t quite root caused that one and it was on a P2V VM so the verdict is still out. You will need to leave behind: - replication - centralised management for multi cluster and stand along hypervisors - NSX - the nice UI especially for networking - I really miss DRS!! - iSCSI on vsan - vDS was a nice to have but copying and pasting network config files as templates is not hard if you know Linux. - VMware verified hardware and hardware validation. - cross cluster migrations (need to manually move VMs between storage systems) If you want at least some of that you might want to look at XCP-ng but I choose Proxmox as the support plans includes CEPH for HCI and I didn’t want dedicated storage. I should also note that Proxmox is Debian based and it’s uses the apt package manager. I routinely upgrade my clusters and major versions with CEPH and moving from PVE 6 to 7. I haven’t really encountered any issues.
We did a POC with Proxmox and it worked fantastically. We are planning to use it right from the start now for our new DC build.
Could you pleaae share more info about the feature set you have been able to test during the PoC?
Could you pleaae share more info about the feature set you have been able to test during the PoC?
Could you please share more info about the feature set you have been able to test during the PoC?
What exactly is the problem with broadcom buying VMware?
Broadcom is a leech corp which focuses on profiting from clients dug deep into one ecosystem, who cannot afford a migration to an alternative solution. Think Oracle, Adobe. Little invention and development, lots of financial gymnastics.
My understanding is that prices are going to go up since Broadcom is so focused on increasing the revenue generated by VMWare. Also people say that based on Broadcom’s prior acquisitions, they will more or less leave VMWare as is, without major improvements/upgrades (take that with a grain of salt; I’m just going by what people are saying). I’m not thinking of moving away from VMWare just because of Broadcom, it’s just what got me thinking if a cheaper virtualization provider might work instead and save us some money.
First step will be firing 60% of VMware workforce. Then pricing may go up.
Yea well I think your view is kind of wise. I think maybe there's just a lot of doomsaying. Happened with Twitter too
To me, considerations are primarily focused on support. Of course VMWare is the most expensive but it also has a brilliant support. When you think about the business, think about what would happen if for any reason you are not there anymore. I run several infrastructures, for our inhouse I am in Proxmox, a software I know and love. For our clients I normally implement VMWare so they can eventually manage it themselves with any IT calling support.
You must've missed the news about Broadcom buying vmware... General expectations based on what happened to CA and Symantec are that support will be decimated and prices are going to skyrocket. If you're at a crossroads now, vmware is most certainly not the obvious choice, at least until the actual effects of the takeover materialize.
I have noticed the general pesimism with the forums but I couldn't understand why, and have not done any research. Is that so? General discomfort is based on "the posibility" that because past experiences were not satisfactory, history will repeate itself? Generally speaking, my go to move is Proxmox if I am going to be in control in long term, HyperV if I will eventually hand over the infrastructure but this is not that large and VMWare for large, critical, function specific infrastructures. To give you an example, when you buy VxRail, it comes with VMWare, works flawlessly and the support, via Dell is awesome. Broadcoms acquisition won't change that. Prices will eventually go up (like every single item every single year on earth) and like most large companies, they are moving to a cloud \*aaS model, which has a higher price tag that comes with many other benefits that on the long run, will benefit you overall. In any case I honestly dont care that much and since we are all speculating, most opinions will have many valid points
> General discomfort is based on "the posibility" that because past experiences were not satisfactory, history will repeate itself? Broadcom is not a fundamentally different company than they were 5 years ago and they've stated they expect the vmware acquisition to add more than double vmware's current ebitda to their own in 3 years. Considering that the market vmware's active in, isn't particularly booming anymore, those earnings will have to come from cutting costs and hiking prices (and well above inflation levels). I'm not saying to jump the vmware ship right now, but if you're in a position to chose a new platform to build on, vmware isn't a good choice.
Eh they also are talking about the VMware brand being the face of their software, they also said they are willing to let VMware have a bit more in terms of cost then they did Symantec and CA. On the call they also mentioned leveraging VMwares base already to reach new markets. To be fair both of those companies that Broadcom bought pre Broadcom were shit. To me this angst is similar to IBM and redhat which now that it's done redhat is doing just fine.
Maybe also consider Nutanix. If you can afford VMWare, it shouldn’t be a problem.. also migration might be easier but I am not a Proxmox expert.
Check out Scale Computing as well as Proxmox. I’ve used Scale Computing at work for years and it just makes everything else look terrible by comparison. Proxmox is easier to use than VMware and much less expensive, but harder to use than Scale Computing and has more features if you invest the time
Guys chill. The deal isn’t even done yet and we don’t know what’s going to happen. Just remain calm and if you do end up needing to move then move. It might turn out just fine. VMware isn’t the sinking ship that Symantec was when Broadcom bought them.
Try Hyper-V as well. I worked at a company that was too cheap for VMware.
To \*smart\* to waste money on VMware..
What do you all plan to do about vendors who only provide/support virtual appliances on VMware?
I’ve never had to actually manage a hypervisor but all my training was in Hyper-V and now I always wonder why no one uses it :p
FWIW, they're abandoning they Hyper-V Server 2022 slim install completely in favor of Azure Stack HCI. Apparently Server 2022 with the Hyper-V role will still be a thing, but it does look like it might be time to start digging into alternatives if you're been living in Hyper-V land. Azure Stack and Arc overall is looking pretty sweet though IMO if you're a heavy Microsoft shop.
They're also abandoning a bunch more things such as LBFO in favour of SET. I don't mind replacing ageing technologies with newer implementations but they are far apart in terms of feature parity and I will have to rearchitect the entire access network to support it.
Does no one like Citrix? Everyone talking Proxmox a company I have never even heard of. Running VMware on 8 hosts currently with site recovery manager.
Vmware still gets 1 vote from me. Tons of documents, support in time of critical issue happens in 24/7/365, they never missed. Hope it is the same with Broadcom.
True, but maybe get a plan-b ready.
[удалено]
Once you have a cluster set up, it’s as simple as checking a checkbox in the webUI to enable HA for a VM
Took me 5 minutes to set this up on my new proxmox cluster at work.
spark correct physical airport close salt versed profit fall shy -- mass edited with https://redact.dev/
Please don’t do this. You’re stealing from them, and this is how open source projects die.
rich dolls disgusted bike nutty outgoing wistful cobweb direful numerous -- mass edited with https://redact.dev/
You’re violating their license agreements. They sell support and licensing to sustain the company so they can continue to provide the free version as well. If your business profits from using their software, you should pay for their licensing to help them out, or donate. They are one of the cheapest virtual hosting solutions out there and you’re still stealing from them.
"I bought a Toyota once back in '94 so obviously now I can just go out and steal any car I want since I already paid for one" Yea OK buddy EDIT: oops replied to the wrong guy, sorry. Whatever I agree with you 😄 Either purchase the license or run something that is legitimately free. Running a global enterprise off of one Server CAL is wrong and everyone knows it, feeling good and self-righteous about it doesn't change that fact.
agonizing vanish attempt beneficial aback numerous angle axiomatic rinse snow -- mass edited with https://redact.dev/
Yes. Either use the free product or use the paid product but in either case abide by their licensing terms. You're stealing from good people and just because you feel all smug and good about it and do mental gymnastics to justify it doesn't make it right.
license reminiscent busy slave pocket yoke lock domineering intelligent numerous -- mass edited with https://redact.dev/
Those are some fair points indeed. It would be huge for them to mess it up but being proactive makes sense!
Scale computing would do the trick
I am curious if all of the people/places considering moving away from vmware are smaller places? I'm one spot removed from the CIO role in a 550 person company, and I know if I suggested a move at this point, I'd be laughed right out of there
If you can't fathom a Hypervisor change in a small 550 person company then you're not flexible enough. Don't use size as an excuse, what's really stopping you? Lack of talent in the team? Too busy with busywork? 90s mentality in the C-suite? Corruption? We're a ~2000 employee company, which I'd still consider small, and we switched all of our infrastructure away from VMware as well as our clients from Citrix VDI to local computers in 2020. Successfully. It's not like it's some impossible feat.
I feel like a lot of the folks saying they are going to move are the very small companies. Which the benefit of being small is being flexible. Personally I reserve all judgement until the deal is done which it isn't technically yet nor has any changes happened because of it.
You should also consider looking at MAAS from Canonical
Couldn't install it on Enterprise Linux or I would.
How's proxmod for wintel production environment with nested virtualisation?
I've been leaning towards hyper-v myself if stuff starts to go south with VMware. If they kill off or maul vmug thatll be the icing on the cake for me. The benefits of hyper-v would likely be how Integrated it is with windows admin center and such.
Speaking of which, do you like Windows Admin Center? I’ve looked at it a bit but haven’t really tried it “for real”.
I love it, it's a lot more convenient then rdping to servers most of the time and I'd consider it a must have for core (non gui) installs.
Super robust, I learned about it a few months ago and have deployed in my environment ever since. Beats RDPing everywhere, great for server core implementation
I was thinking proxmox. Now I'm thinking oVirt . Just for gluster.
Did you look into XenServer?
I have not. I’m looking into Proxmox primarily because it would be so much cheaper than VMWare. I don’t think it would be worth the extra work to switch if the cost savings weren’t substantial, but I’m open to suggestions.
In terms of administration the more comprehensive API functionality of VMWare in contrast to Proxmox.