Over years of managing servers and infrastructure, we've learned something that seems obvious, but is often only fully internalized after a scare:
There are jobs that, even if they can technically be done during business hours, are simply not worth doing while users are working.
Updating a server, modifying a network configuration, rebooting a firewall, expanding virtual machine resources, or tweaking a backup infrastructure may seem like a simple intervention.
And often it is.
The problem is that in IT, the risk isn't always in what you expect to happen, but in what happens afterwards.
The classic “this will take five minutes”
This is probably one of the most dangerous phrases when working with production systems.
“This will take five minutes.”
You install an update.
You reboot.
And the server takes longer than expected to boot up.
A service decides not to start.
A business application stops connecting to SQL.
An update alters a configuration setting.
The firewall comes back up, but a VPN does not.
Or simply Windows decides that the five-minute reboot is going to take considerably longer.
It doesn't mean anything was done wrong.
It means that when working on complex systems, there is always a variable you don't fully control.
That's why we've gradually evolved the way we work.
If an intervention can impact production, we prefer to perform it when the company no longer depends on those systems to work.
Maintenance doesn't end when the update finishes
This is probably one of the most common mistakes.
Updating a server and verifying that the desktop reappears does not mean the work is done.
After an intervention, you must inspect everything.
- Did it boot correctly?
- Are the services running?
- Is SQL responding?
- Are remote accesses working?
- Does it communicate with other servers?
- Are backups functioning?
- Will users be able to log in tomorrow?
Because a machine can be powered on yet have crucial services stopped.
That is why a maintenance intervention typically has three parts:
Making the change, checking the environment, and having time to react if something did not go as planned.
And that third part is precisely what makes working after hours so important.
We've seen enough to know that Murphy also works in IT
There's a certain irony in this.
You can update twenty servers without a single issue.
And it will be the exact server you needed available at 9:00 AM that decides today it doesn't want to boot properly.
It also happens with hardware.
A drive can run for years without issue and fail right after a reboot.
A service that ran for months might not start up again.
An application might behave differently after installing a Windows update.
It's not an exception.
It's part of the job.
The difference lies between encountering that problem at 11:00 AM with twenty people waiting, or finding it at 11:00 PM with several hours ahead to resolve it.
Security updates cannot be postponed forever
We've also encountered infrastructures where servers went months, or even years, without updates.
Usually the explanation is the same:
“If it works, better not touch it.”
It's understandable.
But it's also dangerous.
Many attacks exploit vulnerabilities for which a patch already exists.
The vendor identified the issue.
Developed the patch.
Published it.
And weeks later, production servers remain exposed because nobody wants to take the risk of rebooting them.
In reality, we are simply swapping a known, controllable risk—performing an update—for a much less controllable one: keeping a vulnerability open.
The solution isn't to stop updating.
The solution is to create maintenance windows where updates can be done safely.
Before touching something important, we want to know how to rollback
Another lesson learned quickly in system administration is that a backup isn't worth much if you've never thought about how you will actually restore that system.
Before certain changes, we check backups.
In virtualized environments, we can also use snapshots or additional recovery mechanisms.
For databases, we take specific dumps.
For network devices, we export configurations.
Not because we expect something to fail.
Precisely because if it ever fails, we want the problem to be reversible.
It's an important mindset shift.
It's not just about knowing how to make the change.
It's about knowing what we'll do if the change doesn't work.
We work at night so others can work in the morning
There's something curious about well-executed IT maintenance.
Usually nobody sees it.
A server updates.
An infrastructure reboots.
Security patches are applied.
A backup is verified.
A firewall is updated.
Services are checked.
And the next morning, users arrive, turn on their computers, and work exactly as they did the day before.
From their perspective, nothing happened.
And that is probably the best sign that the maintenance was done right.
Because a large part of our job is precisely that:
Making important changes without those changes turning into a problem for those who rely on technology.
Not everything has to be done after hours
It wouldn't make sense to take this philosophy to the extreme either.
There are many tasks that can be performed perfectly well during regular working hours:
Monitoring, minor tweaks, user creation, reviewing alerts, routine checks, or work that does not impact service.
The key is assessing risk.
If there's a chance of a reboot, outage, connectivity loss, or impact on multiple users, we prefer to set up an appropriate maintenance window.
Not because we expect something to go wrong.
But because we know that one day something will go wrong.
And when that day comes, we want to have time to fix it.
Experience changes how you manage systems
At first, it's easy to focus solely on completing a task.
Over time, you start thinking about everything surrounding that task.
- What depends on that server.
- Who is currently working.
- What services could be affected.
- Where the backups are stored.
- How long recovery would take.
- What happens if the machine doesn't come back online.
That shift in mindset is probably one of the biggest differences between simply maintaining systems and administering critical infrastructure.
Technology is never entirely predictable.
Our job is to try and make it seem that way to the users.