r/sysadmin • u/Fatboy40 • Nov 05 '24
Windows 2022 Servers Unexpectedly Upgrading to 2025, Aaaargh! Question
Arriving at work this morning, an "SME" sized business in the UK, something seemed a little off. Further investigation showed that all of our Windows 2022 Servers had either upgraded themselves to 2025 overnight or were about to do so. This obviously came as a shock as we're not at the point to do so for many reasons and the required licensing would not be present.
We manage the updating of clients and servers using the product Heimdal, so I would be surprised if this instigated the update, so our number one concern is why the update occured and how to prevent it.
Is 2025 being pushed out as a simple Windows update to our servers, just like "Patch Tuesday" events, have we missed something we should have set or are we just unlucky?
Is this happening to anyone else?
Edit: A user in a reply has provided some great info, regarding KB5044284, below. Microsoft appear to class this as a "Security Update", however our patch management tool Heimdal classes it internally as an "Upgrade" and also states "Update Name: Windows Server 2025". So, potentially this KB may be miss-classified by Microsoft and / or third-party patch management tools, but it requires further investigation.
Edit 2: Our servers were on the 21H2 build.
Edit 3: Regarding this potential problem your milage may vary depending upon what systems / tools you use to patch / update your Windows servers. Some may potentially not honour the "Classification" from Windows Update, and are applying their own specific classifications, so the 2025 update could potentially get installed even if you don't want it to be.
Edit 4: Be aware that the update to Windows Server 2025 may potential be classified as an "Optional Update" in your RMM, so if you have chosen to also install these then this could also be a route for it to be installed.
Edit 5: Someone from Heimdal has kindly replied on this matter...
... so I thought I'd link to their reply so it's not lost in other comments. So, it appears that Microsoft have screwed up here, and will have cost me and my team a few days of effort to recover. I very much doubt that they'll take any responsibility but I'll go through our primary VAR to see if they can raise this with their Microsoft contacts.
Edit 6: This has made The Register now...
... so is getting some coverage in other media.
It's not been a great week at work, too much time lost on this, and the outcome is that in some instances backups have come into play however Windows Server 2025 licensing will have to be purchased for others. Our primary VAR is not yet selling WS 2025 licensing so the only way to get new 2025 keys is by purchasing 2022 licensing with SA :(
127
71
u/UseMstr_DropDatabase DO IT! YOU WON'T! YOU WON'T! Nov 05 '24
Does it remain activated after the upgrade?
67
u/Fatboy40 Nov 05 '24
Nope :(
264
u/CluelessPentester Nov 05 '24
Sorry, but this is kinda hilarious.
"Oh, here, let us upgrade your server to the newest version automatically! Oopsie, it looks like you don't have a license. Get fucked!"
How can a company be so out of touch with the real world
67
Nov 05 '24
[deleted]
35
u/joeytwobastards Nov 05 '24
They only ever cared about what shareholders wanted.
18
u/bassgoonist AWS Admin Nov 05 '24
that's basically the definition of a publicly traded company existing in capitalism
8
→ More replies6
36
u/ourlastchancefortea Nov 05 '24
That's why Microsoft, like any responsible company, beta tests their updates. They simply do it in production. YOUR production, not theirs. They aren't stupid.
→ More replies21
u/ApprehensiveBowl5091 Nov 05 '24
exactly what i've been saying for 20 years.
Every other release of windows is basicly a beta test that we as consumers even pay for, then a year or two after they release a functional OS on the same premise/principle as the "beta"Examples: Windows 2000/ME = It's a wonder I decided to make IT a career.
Windows XP = Good stuff
Windows Vista = Good lord...
Windows 7 = Good stuff
Windows 8 = ⛥ K̷͎̖̄̎Ǹ̷̹͎̠̌͌͑͘Ḛ̵͛̃͋̌͂E̶͔̰̜̓Ë̶͈͓L̵̯͑ ̸̥̬͕̹́͋B̴̺͖̞̙͐͊̅Ẻ̸̟̠̳̰̒͜F̴̣̪̫̔̋́̚͝Ŏ̵̢͖ͅŘ̸̘̀̋̍̊E̸̗̓̓̊̕ ̶̡̳͉̈́̂̄̕͝M̸͔̗̙͉͑Ȩ̶̗͓̺̺̀ ̶̛͈̎̍͘͝P̴̨̜̺̥͎͂͆Ẹ̵̛̜̗̳̐̓̓̄A̵̞̣͑S̵̙̦͆̇Á̴͓̒̋N̸̻̺̂̐Ţ̵͍̖͛̑͘S̵̹̩̘̮̃͋͌̃!̶͕͈̬̲͊̎̋ ⛥
Windows 10 = Back on track
Windows 11 = lEtS tRy SoMeThInG nEw!?!
Consumers: Are you asking or telling windows 11?
Windows 11 = I have no fecken clue boi!37
u/baw3000 Nov 05 '24
Windows 2000 was great, possibly even peak Microsoft. Windows ME was a shitshow.
7
u/MeanE Nov 05 '24
I used W2K until some small programs stopped supporting it even before it was EOL. Sad day.
2
u/BlackV Nov 06 '24
ditto, once things started moving to direct x, er.. 8? 10?, then I had to move cause me games stopped working
hmmm actually Am I thinking of 2000 to 7?
→ More replies3
u/chaoslord Jack of All Trades Nov 05 '24
Friends I knew at the time working on ME called it "the dark time"
12
u/renegadecanuck Nov 05 '24
The alternating thing does require you to blend 8/8.1 together, and ignore the initial launch of Windows 10.
Windows 10 was a big improvement over 8 and 8.1, but it was still a bit of a tire fire at first. There's a reason so many people held on to Windows 7 until it was ripped away from them (and there's still an entire subreddit of people using it, in affront to all that is secure and righteous).
7
u/Old-Olive-4233 Nov 05 '24
XP was also pretty awful until at least SP1 and at the time I'm pretty sure I disliked it until SP2.
→ More replies→ More replies2
→ More replies4
u/autogyrophilia Nov 05 '24
This feels right but is wrong.
Windows ME was an attempt to modernize 95 with NT components, keeping the system on MS-DOS to try to keep it light. It didn't work well.
2000 (NT 5.0) did. Not without it's issues because it's Windows software.
Windows XP was most of NT 5.0 released to the general public. Built upon 2000, as 2003
Windows Vista (NT 6.0) was poorly handled but it was always going to be painful as it was a huge overhaul with many changes that allow windows graphical session to be pretty secure ( the graphical session, we are still dealing with NTML1, nevermind 3rd party apps...) we are talking features such as the protected screen, running the graphics in user mode and not in kernel mode... As well as improving the support for the modern graphics Put this in perspective. It's what the Unix world is trying to do with Wayland and you see how that is going.
All other versions of Windows build on NT 6.0, with a disappointing lack of additions versus changes. With some of these changes being baffling resulting in Windows 8 in particular
→ More replies→ More replies3
u/BloodyIron DevSecOps Manager Nov 05 '24
Because we as a collective industry do not push back enough on application vendors demanding they offer support for alternatives like Linux.
We need to ring the bell loudly that this is not okay and that we need app vendors to do better.
→ More replies18
u/lordcochise Nov 05 '24 edited Nov 05 '24
I can totally believe MS wants to deliver server upgrade paths as they do on clients, but if it's not a free update for 2022 installations GOOD GOD who approved this without any kind of licensing warning
EDIT: at least on Server 2022 21H2 LTSC there is indeed a warning
→ More replies12
u/skipITjob IT Manager Nov 05 '24
Activated and licensed are two different things. It's the license Microsoft cares about...
21
u/Remarkable_Cook_5100 Nov 05 '24
In this case it is neither activated or licensed after the 2025 upgrade.
5
u/Hi_Kate Nov 05 '24
Unless you use licencing channel which lets you upgrade, like with SA or SPLA. Then it is licenced, but not activated.
4
62
u/Andrei_Hinodache Nov 05 '24 edited Nov 06 '24
Hi u/Fatboy40
Andrei from Heimdal here, man, I'm really sorry for the havoc that was created with this update, our team (thanks for raising this with them - I have a feeling you were the first to bring it up to our Customer Success team) managed to pinpoint this and blocked this update across all server policies to avoid any further upgrades from 2022 to 2025
I also notice another point in the chat where you're asking how to apply a granular approach to updates - if you'd like, we can set a call up tomorrow and we can look at this one together.
Here's the official com. that just went out a while ago:
On 5th Nov 12.16UTC, Heimdal was notified by a customer about unexpected upgrades related to Windows Server 2025 in their environment. Due to the limited initial footprint, identifying the root cause took some time. By 18:05 UTC, we traced the issue to the Windows Update API, where Microsoft had mistakenly labelled the Windows Server 2025 upgrade as KB5044284.
Our Analysis and Fix:
Our team discovered this discrepancy in our patching repository, as the GUID for the Windows Server 2025 upgrade does not match the usual entries for KB5044284 associated with Windows 11. This appears to be an error on Microsoft's side, affecting both the speed of release and the classification of the update. After cross-checking with Microsoft’s KB repository, we confirmed that the KB number indeed references Windows 11, not Windows Server 2025.
To prevent further unintended upgrades, we have immediately blocked KB5044284 across all server group policies.
If you would like to address this patch on your servers, we recommend manually removing it.
22
u/Fatboy40 Nov 05 '24
If you would like to address this patch on your servers, we recommend manually removing it or reaching out to our support team for assistance.
Hi Andrei,
The real problem here is that from what I can see, and I know this is not Heimdal's fault, is that there's no way to "rollback" the upgrade to Windows Server 2025 unless you know otherwise?
We've now a selection of 2019 servers that we either need to bare metal restore, try to rebuild, or purchase 2025 licensing that we have not budgeted for.
So, do we now assume that Microsoft must be held liable for this mistake, and somehow hope that they provide a method to get back to Windows Server 2019? (which I'm assuming is not possible, and I've no doubt that they'll not own up to it and cover customers for the required 2025 server and CAL licensing).
Thank you.
22
u/Andrei_Hinodache Nov 06 '24
You're spot on with your analysis - I hope our Founder doesn't kill me for quoting him, but "it's like upgrading a tesla OS and saying, now to drive your car, insert your credit card."
We're doing all that we can internally to see if anything can be done - even the roll-back is a b..... since it's a new version of the OS...
6
u/Narrow_Ruin Nov 07 '24
That sounds like a free upgrade customer satisfaction situation to me. To stick with that car analogy, there are all kinds of small issues that car companies fix under customer satisfaction that are not serious enough to be a recall, but fixing the problem for free helps keep a customer coming back. I am not saying this because I want some free upgrades, my employer already pays for on-going upgrades in an EA. I am saying that because it is the right thing to do.
→ More replies3
5
u/bdam55 Nov 07 '24
FWIW, this was not Microsoft's fault. They published the update properly: https://www.reddit.com/r/sysadmin/comments/1gl6jsw/comment/lvyps27
I think you are also misunderstanding how KBs related to updates and the fact that there's ... unfortunately ... no actual source of truth for any of it.
This was a Feature Update released to the Windows Update channel (not the Update Catalog) that is properly classified as an Upgrade (Feature Update). As much crap as MS deserves for screwing up updates, this is one of the rare times where they are not to blame.
→ More replies2
u/Lando_uk Nov 06 '24
I'm confused by your analysis, how did the KB5044284, which is an standard update for Win11/Server 24H2, even manage to get approved and installed on Server 2019 and 2022 clients?
If you ran KB5044284 on a Server 2022 manually, surely it would stop, saying its the wrong OS. None of this makes any sense to me.
2
u/Clear_Key5135 Nov 06 '24
KB5044284 is for the October CU for all os's on the current production branch of windows.
3
u/Lando_uk Nov 06 '24
No it isn't. The Oct CU for Server 2019 is KB5044277 and the Oct CU for Server 2022 is KB5044281.
→ More replies2
u/nont0xicentity Nov 06 '24
It happened outside of Heimdal so it is not limited to them and their analysis may be correct. Say you have KB5044285 meant to be able to upgrade 2019/2022 to 2025. But for some reason, MS labeled it as KB5044284 everywhere and made KB5044284 applicable to 2019 and 2022. Now you have a patch showing under KB5044284 that was never supposed to but since the installer is actually KB5044285, it can be installed on 2019/2022. For a simple explanation, download Teams, and rename it to OneDrive, it will install Teams because that is what is under the hood. If you check the catalog it has 3 entries, one being for server OS and from what I understand, that was never supposed to be there. The other 2 entries are for Win11 24H2 and lasted updated 10/8, whereas the server one was last updated 10/31, which is unusual. If you look at the KB, it only list Windows 11 under the Applies To section.
→ More replies
88
u/brink668 Nov 05 '24 edited Nov 05 '24
Yes 2022 can be upgraded to 2025 via Windows Update just like workstations now
This video talks about it a little I randomly watched and learned yesterday too.
https://www.youtube.com/live/j470Tp4b6es?si=SU4-Acabnu2MqMcA (toward end /winget section)
https://www.youtube.com/live/LCcug9HHnIQ?si=dQ-x8XrDPpuSLSEn
Edit: another video
Edit2: your only option is likely is restore from backup and set settings to prevent auto inplace upgrade. Server inplace upgrade does not support rollback to previous version
20
u/Fatboy40 Nov 05 '24
Thank you.
So you'd be leaning more towards Windows Update having instigated the in-place upgrade that the third-party tool? (or I suppose the third-party tool may have just instantly pushed it out).
It looks like we need to understand where the logs are for Windows Update and why the update was triggered so soon with 2025 being only available for a few days.
→ More replies4
17
→ More replies7
u/zz9plural Nov 05 '24
WTF? Even my DCs are offering inplace upgrades to 2025. Are inplace upgrades of DCs supported now?
22
u/Justsomedudeonthenet Jack of All Trades Nov 05 '24
It's been supported for a long time. Few recommend it since it's trivially easy to spin up a new DC, but it's supported.
→ More replies9
u/NoSelf5869 Nov 05 '24
In my understanding, in-place upgrade of DC's has been supported, but not recommended, for long time.
8
u/PkRavix Nov 05 '24
In particular you should not in-place upgrade to 2025, the new 32k mode is only supported on new installs. 2025 can run in 8k compatability mode until all your DCs are 2025.
→ More replies→ More replies4
u/brink668 Nov 05 '24
Yes in-place upgrades have been around but via Windows Update for Server that is new.
60
u/cloudAhead Nov 05 '24
I manually checked Windows update and was not unexpectedly upgraded to 2025. There is a separate section in the UI to upgrade to 2025 if you choose to do so. The experience is similar to what Microsoft did client side with Windows 11.
My guess is that OP may have auto approved all packages, or a similar option, in their patching tool.
40
u/Fatboy40 Nov 05 '24
It looks like you've made a pretty accurate guess :(
12
u/RandomLukerX Nov 05 '24
Can you clarify for my sanity, this was caused by a third party patch management tool in your environment?
→ More replies17
u/Fatboy40 Nov 05 '24
The simple answer is "yes", however it's a little more nuanced that that in that KB5044284 is a Security Update from Microsoft but our RMM tool classed it as an OS Update.
It seems that for others their RMM may also be potentially miss-classifying it, and even some Microsoft tools cannot be trusted 100% to not install the upgrade to 2025.
6
u/cloudAhead Nov 05 '24
KB5044284 is an OS update - a servicing stack update, but not an upgrade to 2025. I wouldn't be surprised if it delivered the code to offer the in place upgrade, though.
2
u/SonicDart Nov 07 '24
Does anyone know if the same issue could happen in other patch management systems? We're using SCCM for the bulk of our windows servers
4
u/soccer362001 Nov 05 '24
We got a notice from an RMM we are trialing that we should block it because it was causing 2022 to update to 2025. This is likely a global issue.
6
u/zz9plural Nov 05 '24
Yes, same here. Looks like Heimdal is at least partly at fault for OPs problem. The exact reason for the miss-classification remains to be determined.
2
u/YnysYBarri Nov 06 '24
What's worrying me more than the "who's fault is it anyway?" is this delightful piece of advice from Heimdal:
Sorry, what century are we in? We no longer play the "my server has an uptime of 2.3 squilion years!" game. You don't encourage disabling automatica updates, you encourage managing them in a controlled fashion.
→ More replies→ More replies3
u/My1xT Nov 05 '24
Even then this shouldn't just be a 1 click thing as unlike with win11, ws2025 iirc ISNT a free upgrade
27
u/ColXanders Nov 05 '24
Ah crap this has happened to us too. Using Heimdal as well. Just waking up to this reality...
16
u/Fatboy40 Nov 05 '24
I feel a little less crap now knowing that I'm not on my own, good luck with the remediation.
Looking on one server, under "Windows Update > Update History > Uninstall updates", there is an Uninstall option available for KB5044284. So, once an incremental backup of the server has completed I'm going to attempt the Uninstall and keep my fingers crossed that it can roll itself back (there a Windows.old folder on the C drive / volume so fingers crossed).
6
u/ColXanders Nov 05 '24
Please post back how it goes. I'm in the US and just getting notice of this so we are in discovery mode. Any additional info would be helpful. I have our MSSP involved which has a direct relationship with Heimdal and will post any updates I get here as well.
→ More replies5
22
u/KernicPanel Nov 05 '24
This would be a disaster if it happened to rds servers or brokers as the windows version needs to match.
53
u/small_horse Nov 05 '24
Yep, our RMM tool is set to hold any new updates for review, this morning got 40~ packages all nicely named "Server 2025" - jesus mary and joseph Microsoft what are you THINKING?!
19
u/ourlastchancefortea Nov 05 '24
THINKING
Office 2025 Dictionary: Unknown word, do you want to add it?
8
u/what-the-puck Nov 05 '24
Wouldn't that be a good thing? That your RMM clearly identified and labeled and held them?
7
u/small_horse Nov 05 '24
yes it (for once) actually did its job properly! it was more that MS are deciding to issue an update package to entirely change the underlying OS, which seems really dumb
→ More replies4
u/what-the-puck Nov 05 '24
I suppose, it's nothing new though.
Since the Internet on average has been able to "handle" service packs or OS updates, they've been moving over the wire.
Windows 8.0 to 8.1, 8.1 to 10, various major updates to versions of 10, 10 to 11... Those were all update available through Windows Update.
And likewise on the Server side (2012 -> R2 -> 2016 -> 2019 -> 2022). Those could be done in-place as well through downloads that happen while Windows is up and running (and restarting) via files downloaded over the Internet.
2
u/spetcnaz Nov 06 '24
The issue isn't between inplace vs wipe upgrade. The issue is that a server OS, now has the same, relatively easy way of getting upgraded in place while in production. That's an absolute insanity. Server isn't a desktop, it can break so many things.
No version of the server before had this toes to auto updates, and that was good.
17
u/Lughnasadh32 Nov 05 '24
After reading this post, I checked the servers at an NPO that I manage. Both are 2022 (21H2) and both have the upgrade to 2025 option. My main question here is....is there a cost? If so, I am not a fan of this 'marketing tactic'. Someone with less experience could click download and install and then they would be on the hook for whatever the licensing costs at that point.
16
u/Jeeper08JK Nov 05 '24
11
u/Lughnasadh32 Nov 05 '24
TY - I can see this biting people in the butt. Most people don't read these warnings. They will install the update then wonder why the server stopped working 180 days later.
10
u/Fatboy40 Nov 05 '24
My main question here is....is there a cost?
100% there is, in Windows Server licensing for the CPU cores and also CAL's.
→ More replies3
u/sweetrobna Nov 05 '24
Normally for a non profit purchasing through techsoup or azure for non profits windows server licenses/CALs have software assurance. Your 2019/2022 cals work for server 2025 at no additional cost.
→ More replies
9
u/PhantomWang Nov 05 '24
I'm also worried about this because our servers are managed by Azure Update Manager and I noticed this evening they're starting to show Server 2025 as a pending update. Luckily it appears the current classification for it is "Unsupported" so I don't believe it will automatically install, but at this point I have to actively monitor it because I can't trust Microsoft.
7
u/Electrical_Arm7411 Nov 05 '24
Make sure you exclude the KB ID in each of your maintenance configurations in Azure Update Manager.
→ More replies
53
u/spetcnaz Nov 05 '24
Wowww who's bright idea at Microsoft was this?
Who wants servers to migrate to a new version, basically an in-place upgrade.
Microsoft should give serious heads up for such things.
36
u/dustojnikhummer Nov 05 '24
Even ignoring compatibility, what about licensing??
26
Nov 05 '24
"Oh shucks, guess you'll have to pay us more money, this is so sad"
I'm sure they really care.
→ More replies6
u/babywhiz Sr. Sysadmin Nov 05 '24
Go buy one now, sucka!
12
u/dustojnikhummer Nov 05 '24
One? Server itself is one thing but you need a whole new set of CALs.
→ More replies6
u/lordcochise Nov 05 '24
Have done in-place upgrades since the 2003 days, mostly they've gone pretty ok (albeit on a very specific schedule and we have pretty vanilla setups). But it's sounding like those that have tried this have broken activation, also not sure if the default optional feature / update AD blocks would catch this or not...
5
u/spetcnaz Nov 05 '24
Yeah, there is a huge difference between a planned n place upgrade, and getting one through auto update.
→ More replies→ More replies9
u/andrea_ci The IT Guy Nov 05 '24 edited Nov 05 '24
in-place upgrades are ok in the last two versions.
not optimal, but they work
5
u/spetcnaz Nov 05 '24
Until they don't.
That's not the point, the point is so many things can go wrong, this is absolutely insane.
→ More replies
10
u/Lando_uk Nov 05 '24
ok, so this is a Heimdal issue and not a general WU issue everyone should be aware of?
9
u/nont0xicentity Nov 05 '24
No, you should be aware because other tools sees it as varying things, some as Security Updates, some as Feature Updates, and other classifications. In Ninja, it is showing up as a Feature Update on our 2019 and 2022. If someone had Feature Updates auto approved, it would upgrade. I had globally blocked it because it is also the same KB that upgrades Windows 11 to 24H2 and we're staying away from that for a while.
→ More replies2
u/ChrisDnz82 Nov 05 '24
Even as a Feature Update it will still catch a lot out who will think its just going up another version of 2022 and not actually 22 to 25. This happened to so many people with Win 10 to Win 11 when MSFT recently made that upgrade exactly the same as the normal FU
→ More replies2
u/Lando_uk Nov 05 '24
Correct me if I’m wrong but server OSs stay on the same version for their lifespan, there aren’t two different versions of 2022 for example ?
→ More replies→ More replies3
u/VinzentValentyn Nov 05 '24
It shows as available for server OS 2019 and up.
Whether it installs or not is down to your policy. It's not a Heimdal issue
7
u/Jeeper08JK Nov 05 '24
19
u/Remarkable_Cook_5100 Nov 05 '24
If you click the Download and Install you get this, which indicates it is not a FREE upgrade!!
4
→ More replies2
u/lordcochise Nov 05 '24 edited Nov 05 '24
AH, ok so at least there IS a warning then; lol though this method of upgrade leaves you no uninstall / removal method (though not a big deal if you're already virtualizing, have good backups / snapshots, etc)
4
u/Randalldeflagg Nov 05 '24
fun fact, if you use an RMM tool, you dont get this popup warning, it just happens. And then you are screwed when you find out it upgraded your SQL servers and you can't get an outage to take those DB offline to restore the OS to 2022 and then restore those DBs back to production.
→ More replies4
5
u/YellowOnline Sr. Sysadmin Nov 05 '24
I have no issue with in-place upgrades at all, but you should of course consciously choose to do it, not only because of compatibility, but also because of CALs. I'm fine with my 2022 DCs becoming 2025, but I only have 2022 CALs. Or did MS change how CALs work?
11
u/Remarkable_Cook_5100 Nov 05 '24
Honestly, if Microsoft was simply giving everyone a free upgrade from 2019/2022 to 2025 with CAL and RDP license upgrades, that would be fine with me. But they are not, so this option should not even exist.
→ More replies
12
u/mb194dc Nov 05 '24
I've seen this happen with Office, but not Server itself, though on 2019 not 2022.
22
u/Vicus_92 Nov 05 '24 edited Nov 05 '24
Fuck me, it's a server not a desktop. Who thought this was a good idea!?
Guess I know what I'm reviewing tomorrow.
Edit: For anyone scrolling through comments, I did some testing this morning and using N-Able NSight RMM or native Windows patching, I'm not seeing this behaviour on server 2022 21H2 servers.
The option is present in native Windows update UI, but nothing being forced.
As the OP and other comments suggest, this seems to be a Heimdal issue. That said, be careful and review your patch management mechanisms!
12
u/longlivemsdos Nov 05 '24
yep I think MS forgot that since around WS2016 (or 19 can't remember which) with xbox services and Edge auto opening on 'news' tab instead of protected.
3
u/MBILC Acr/Infra/Virt/Apps/Cyb/ Figure it out guy Nov 05 '24
Don't forget the Coupon's option in Edge, cause servers need that too...
12
u/TrueStoriesIpromise Nov 05 '24
In WSUS/SCCM, KB5044284 shows as 0 required/0 installed for 24H2.
Seems like Heimdal is the problem, not Microsoft.
→ More replies1
5
u/UltraEngine60 Nov 05 '24
As soon as they figure out patching at a decent cadence, and now hotpatching, they start treating major OS updates the same as hotfixes. One step forward two steps back. I can handle major OS upgrades myself Microsoft, back the fuck off.
4
u/MBILC Acr/Infra/Virt/Apps/Cyb/ Figure it out guy Nov 05 '24
Just to test, brand new clean install of Server 2022 - Not yet activated, used an MSDN ISO image:
en-us_windows_server_2022_updated_sep_2024_x64_dvd_cab4e960
First check for Windows Updates:
14
u/ConfectionCommon3518 Nov 05 '24
Why do I sense this is the idea of the MS marketing dept to show massive uptake figures?
Servers are quite often delicate creatures playing home to licensing services and other stuff that may take one look at the server and knowing things have changed just decide to not play taking down the entire production line and then the fun starts both at the practical level and the point where they start waking up the lawyers.
→ More replies
4
u/tehcheez Nov 05 '24
So we didn't update to 2025, but I can confirm the 4 2022 VMs I have updated this morning (not automatically, that's just the update schedule we have) and now have an option under Windows Update to update to 2025. Have never seen that until today.
→ More replies2
u/spittlbm Nov 05 '24
Confirmed. Just did a manual "check for new updates" abd the upgrade option appeared.
3
u/RestartRebootRetire Nov 05 '24
Here's what I see on my Server 2022 Standard (10.0.20348) server that I manually update.
4
4
u/FutureSafeMSSP Nov 06 '24
Here is the Heimdal CPO reply explaining how the misclassification in the Microsoft API caused the curfuffle.
5
u/DeltaSierra426 Nov 11 '24
Folks, quit blaming MS for once. I know it's too easy to do (their own fault, lol). The only aspect that you can blame them is for enabling in-place upgrades to Server 2025. That's why this is happening and Heimdal hasn't been honest and forthcoming about this: that they didn't program the necessary changes to properly handle this change.
Well, Microsoft also could have written about this more rather than just stashing it in a video:
If it was a Microsoft problem, why did most RMM solutions not have this problem?
And yes, if sysadmins were actually testing updates before pushing to larger production swaths, this would have been caught on one host instead of tens or more servers. You guys are leaving too much to autopilot (no, I don't mean MS's solution) and not enough manually checking down. Patch management automation is a great thing, but it still takes some care and thoughtfulness -- this is the Windows ecoysystem, after all!
→ More replies
6
u/SnooDucks5078 Nov 05 '24
wow, thanks for the heads up! I just noticed it appear as an optional install on my 2022 domain controllers! Better check SConfig set to manual.
7
u/Weird_Lawfulness_298 Nov 05 '24
I looked at a 2022 server and one of the options it had in Windows update was to download and install Server 2025.
→ More replies8
u/TkachukMitts Nov 05 '24
Also seeing this on 2019 servers.
16
Nov 05 '24
[deleted]
6
u/TkachukMitts Nov 05 '24
Well to be fair the CRTs must be so dim at this point that it would be hard to see.
4
→ More replies3
u/Weird_Lawfulness_298 Nov 05 '24
Yeah, I just checked and it was on 2019 servers.
2
u/neko_whippet Nov 05 '24
Where was it im checking on some 2022 and some 2019 and I dont see an upgrade option to 2025
3
u/severnd Nov 05 '24
we've got this garbage pending on 75 servers! mix of 2022 and 2019! block that KB if you can, but probably too late if you're updating to meet security compliance rules.
3
3
3
u/konikpk Nov 06 '24
So as i read its Heimdal problem. We have MECM + WSUS and no servers updating.
KB5044284 is not required in any of 2022 servers.
3
u/terrybradford Nov 06 '24
We don't have that issue when rocking 2003 server - someone before who I dismissed as an idiot clearly saw this coming 👏
8
u/greenstarthree Nov 05 '24
I knew I made the right decision to stick with WSUS for server patching for now and not go with 3rd party solutions.
Might be the only opportunity I get to say that.
→ More replies
5
10
u/tuntaalam Nov 05 '24
If all else fails, call Microsoft and ask them to explain the behaviour of their shitty os.
33
u/Absolute_Bob Nov 05 '24
7
u/KingStannisForever Nov 05 '24
Isn't that Ubisfot logo there?
Anyway, Microsoft doesn't know what Microsoft is doing
6
Nov 05 '24
[deleted]
2
u/pdp10 Daemons worry when the wizard is near. Nov 05 '24
They know you're not just gonna up and leave them for Linux.
Those who could leave for Linux easily, decamped years ago. Many of those enterprises left, can't leave easily.
The same for IBM mainframes -- you don't keep paying for those if you have decent options. But they did it to themselves. Past them decided it was a problem for future them.
2
u/BloodyIron DevSecOps Manager Nov 05 '24
Actually there's plenty of AD environments (on-prem) that actually are eligible for migration to Samba AD (running on Linux), as the functionality said environments care about is fully served by Samba AD. Yes, not all scenarios are covered by Samba AD, but most are. (I know because this is something my company offers by the way)
So while there are those who have migrated Windows->Linux already in part or whole, there's plenty of opportunity left for more of that!
10
u/DattiHD Nov 05 '24
Even if they bother to explain "he behaviour of their shitty os" it won't change the f****ed up situation for affected admins.
6
9
u/Dependent_Price_1306 Nov 05 '24
Why? It won't be in the script of the moron on the other end of the phone.
4
2
u/InfamousStrategy9539 Nov 05 '24
Is the Heimdal dashboard showing the update in the assets for the servers? When did they update? Ours is set to update them on Fridays, but just checked our DC and it hasn’t been updated.
3
u/Fatboy40 Nov 05 '24
The "GP" (why on Earth did they call it that, for me GP = Group Policy in Active Directory) was set so that OS Updates occurred on a Tuesday and Thursday, so overnight today it started to push it out.
→ More replies
2
u/lordcochise Nov 05 '24 edited Nov 05 '24
Wasn't seeing this ANYWHERE in WSUS but checking online for updates on Server 2022 VMs does make this appear as an optional update not unlike Windows 10/11 client-side major build updates; On one hand I'm not surprised they eventually went this route for what used to be 'R2' versions (though Server 2019 -> 2022 -> 2025 could be more of an R3?); at the same time, everyone seems to be saying this isn't a 'free' update and requires a 2025 license or upgrade rights? HOO BOY there's gonna be plenty of admins pissed at M$ if that ends up being the case. GOOD GOD I'm glad I saw this post before I checked this stuff today
Currently all our hypervisors / VMs are Server 2022 (21H2 LTSC) and I have yet to see a WSUS update normally requiring approval that matches this; is it possible that what's really meant as an optional inline upgrade for the non-LTSC server builds got released wrong? Would make sense for those on active / enterprise licensing to have this path but PROBABLY NOT the rest of us if it breaks activation....
EDIT: on LTSC it's only appearing in the 'optional features' area of Settings -> Windows Update and it does require you to affirm that (1) it's a 1-way upgrade regardless of consequences and (2) you'd better have 2025 license(s) handy
2
u/Gummyrabbit Nov 05 '24 edited Nov 05 '24
I have a test server running 21H2 and I downloaded KB5044284 (which also downloads KB5043080). I can't even install it on 21H2. I get "Installer encountered an error: 0xca00a005". So I'm not sure how your patch tool is managing to get it installed. If I check for updates on 24H2 (Server 2025), I see KB5044284 and KB5043080 available and I'm able to install them. So maybe your patch system is upgrading your 21H2 to 24H2 and THEN you get KB5044284 and KB5043080 as available.
2
u/damnedbrit Nov 05 '24
Testing on W2K22 and I see that there is the option under Windows Update in the GUI below pending normal updates and below the Install Now an area that says the next version is here and a "Download and install" link. Running:
Install-WindowsUpdate -WindowsUpdate
does not offer the upgrade to W2K25. It does look like from the descriptions elsewhere in this thread that it's a Heimdal setting that is enabled to 'upgrade to Windows 11' that is being misused to upgrade to W2k25 as well.
→ More replies
2
2
u/CptCptLuxx Nov 05 '24
Just make a gpo (windows update for business), target version 21H2 and the update is no longer offered to any Server
→ More replies2
u/ITStril Nov 05 '24
I can confirm: GPO with target version 1809 for Windows 2019 and 21H2 for Windows 2022 seems to suppress the upgrade notification
2
u/cpupro Nov 05 '24
Unintended, unintentional, free upgrade to the latest OS.
Absolutely NOTHING bad could happen...
Right?
LOL
2
u/mankycrack Nov 05 '24
NinjaOne put a yellow banner across the top of their portal today warning about this. I blocked the update on Monday because I was getting bad vibes over the weekend
2
u/moonwolf3533 Nov 05 '24
We have a separate section in the UI to upgrade our server. This should never be an option unless they are giving the upgrade away for free and even then it shouldn't be there.
2
u/Vexser Nov 06 '24
Maybe it's time to block MS at the company DNS. Only let trusted/secured hosts contact them. Otherwise you might turn up to a room full of bricks.
2
u/AdWerd1981 Nov 06 '24
Had the option in Windows Updates on a 2022 VM yesterday, but today that option has vanished. I'll check my other VMs to see if it's the same, but it feels like M$ pulled the feature update part.
2
u/raffey_goode Nov 06 '24
if we are using SCCM and WSUS is there any action we need to take?
2
u/RCTID1975 IT Manager Nov 06 '24
Just don't blindly auto approve any patches like good policy dictates and you're fine.
2
u/External_Gain2380 Nov 07 '24
It's reasons like these where I have blocked all URLs to Download Windows Updates. This way nothing network wide can check for download or install updates. WSUS can deploy them.
2
u/bushmaster2000 Nov 08 '24
So if they force the update then I expect cal licenses to be upgrades as well automatically free instead of having to pay to upgrade them unexpected
2
u/Comfortable_Swim_380 Linux Admin Nov 10 '24 edited Nov 10 '24
So help me. Im actually impressed at this level of screwing up this time. I've been weening my new and existing customers off windows just because of issues like this.
→ More replies
2
u/Comfortable_Swim_380 Linux Admin Nov 10 '24
God help the sysadmin people if the dc decides to do this. New plan will be "enjoy your new server 2025 install."
2
u/KoalaOfTheApocalypse End User Support Nov 11 '24
In a reverse circumstance, I tried to install KB5044284. First I specified with pswindowsupdate and it couldn't find it. Next I manually downloaded the KB from update catalog and it failed to install. I was trying to upgrade server standard 2022. I had to end up using the .iso, which was it's own adventure.
4
5
2
u/Celikooo Sysadmin Nov 05 '24
According to WSUS, KB5044284 is only available for 2025 servers. It is declared as a Security Update.
It is most likely not upgrading the OS from 2022->2025
Furthermore, the OP apparently configured Heimdal in a way to install all updates (including optional updates pulled from Microsoft), which most probably caused the servers to update to 2025.
However, the Windows Update GUI displays a button to download and install the in-place upgrade to 2025, mainly when contacting the Microsoft Update Servers directly.
7
u/Fatboy40 Nov 05 '24
According to WSUS, KB5044284 is only available for 2025 servers. It is declared as a Security Update. It is most likely not upgrading the OS from 2022->2025
Nope, it 100% installed KB5044284 this morning, it's all in logs etc., and our RMM tool classifies it as an Operating System Update and installed it onto two 2019 servers + it errored on a third so thank God for that.
2
1
1
u/Mysterious_Manner_97 Nov 05 '24
Looks like this is a screw up perhaps due to kb5044281 having the exact same name? Outside of a comma.. wondering if ppl are using txt based approval rules?
1
u/ChrisDnz82 Nov 05 '24
would anyone care to share their patch logs/windowsupdate logs? or provide the patch guid of the patch they think did it. I would like to check our patch db (I work for N-able) to see if we can help figure out more
→ More replies
515
u/TNTGav IT Systems Director Nov 05 '24
We are tracking this elsewhere - the running *theory* at the moment is https://www.catalog.update.microsoft.com/Search.aspx?q=KB5044284 this, published as a security update, is actually an update to 2025. Not validated yet.