[server] In production, Do you stop most processes before upgrading the packages?

samedi 23 juillet 2016

Hello:

When I need to upgrade the packages installed in my servers, I always schedule a maintenance period during which most services are turned off (I usually reboot the machines after that). This means that I've to notify my clients in advance, so that they notify their own users. It's a very tiring task. Specially when I've to upgrade the packages of a host machine that runs several virtual machines, because I need to count with the approbation of everybody. But that's not always possible and, very often, I've to postpone the upgrades for another time. In addition to that, when I reboot the machines, I've to verify that all the existing daemons have started properly, among other checks.

I know that you can upgrade the packages while the applications and daemons are running. Because the old versions of the files are kept in disk (and maybe in memory) while they're in use. Indeed, that's what happens in any typical GNU/Linux system although you stop the most processes. Because always there are some essential processes that must be running, which also are subject to upgrades. Otherwise, a GNU/Linux system couldn't upgrade itself without rebooting.

But, in my opinion, the least processes are running, the smallest is the possibility of a crash. Specially if there are processes that handle data that can become corrupted, such as database servers. I know that the updates in stable distributions usually contain security patches and bug fixes only, so they don't change the ABIs. But still... I'm a little paranoid. Furthermore, some updates cause that the daemons are restarted automatically, which makes things even worse.

So my question is: What do you do in your production servers? Do you stop as many processes as possible before upgrading, with the corresponding downtime? Or do you upgrade with all them running? If you do the later, Do you choose manually those upgrades that won't cause an automatic restart of the process?

In my case, the only thing that I do to make the upgrade process more agile is downloading the upgrades very often, but without installing them (apt-get -d upgrade). That way, when I do the real upgrade, it will be done faster.