In my normal everyday job, I am tasked with managing and maintaining about 30-40 production CentOS servers. Being a security guy, I maintain a pretty rigorous patching routine. However, because these servers are customer production servers, one very important caveat is that I need to do everything I can to minimize customer downtime.
Normally when I patch a server, my routine is:
yum check-update (check what updates are available)
yum -y update (update everything)
And if the list produced by check-update shows the kernel or kernel-headers packages in the list, I promptly reboot the server. This translates into about 5 minutes of downtime for the customer as the server reboots.
So that got me thinking. Is every kernel update critical or can they easily be delayed? So then I stumbled across this excellent plug-in for yum.
yum-changelog-1.1.10-9.el5.centos
Name : yum-changelog
Arch : noarch
Version: 1.1.10
Release: 9.el5.centos
Size : 12 k
Repo : installed
Summary: Yum plugin for viewing package changelogs before/after updating
Description:
This plugin adds a command line option to allow viewing package changelog
deltas before or after updating packages.
Perfect! That will allow me to see exactly what is changing with each new version of the kernel. So I install that with:
yum install yum-changelog
Now we can use yum to show us the change log for certain packages. So, if I want to see the change log for the kernel related package, I could run something like:
yum update kernel kernel-headers --changelog
This will produce output similiar to:
Changes in packages about to be updated:
kernel-headers - 2.6.18-92.1.22.el5.x86_64
* Wed Dec 17 06:00:00 2008 Karanbir Singh [2.6.18-92.1.22.el5.centos]
- Roll in CentOS Branding
* Sat Dec 6 06:00:00 2008 Jiri Pirko [2.6.18-92.1.22.el5]
- [misc] hugepages: ia64 stack overflow and corrupt memory (Larry Woodman ) [474347 472802]
- [misc] allow hugepage allocation to use most of memory (Larry Woodman ) [474760 438889]
Ah, ha. As I suspected. Two memory related bugfixes and CentOS branding. Because we are currently not expirencing any memory related issues, this patch does NOT rate as critical and warrent immediate customer downtime. This can be delayed.
So now I can apply the other patches and exclude the kernel upgrades with:
yum update --exclude=kernel,kernel-headers
Now, I have a script that runs nightly on all my CentOS servers. This script gathers nightly statistics, logs entries, etc from my servers and emails it to me. This is pretty much jsut a CentOS port of my old Gentoo Update Script, with some CentOS speficic changes and additional features. The other thing it does, is generate a list (via yum check-update) of all the updates required. So the question now is, now can I get this interactive command to run via an automated script? The easiest way I could come up with is:
echo n | yum update kernel kernel-headers --changelog
Probably not the cleanest way, but does the job very well.
REFERENCE
http://monkey-house-org.blogspot.com/2009/02/centos-patching.html
Normally when I patch a server, my routine is:
yum check-update (check what updates are available)
yum -y update (update everything)
And if the list produced by check-update shows the kernel or kernel-headers packages in the list, I promptly reboot the server. This translates into about 5 minutes of downtime for the customer as the server reboots.
So that got me thinking. Is every kernel update critical or can they easily be delayed? So then I stumbled across this excellent plug-in for yum.
yum-changelog-1.1.10-9.el5.centos
Name : yum-changelog
Arch : noarch
Version: 1.1.10
Release: 9.el5.centos
Size : 12 k
Repo : installed
Summary: Yum plugin for viewing package changelogs before/after updating
Description:
This plugin adds a command line option to allow viewing package changelog
deltas before or after updating packages.
Perfect! That will allow me to see exactly what is changing with each new version of the kernel. So I install that with:
yum install yum-changelog
Now we can use yum to show us the change log for certain packages. So, if I want to see the change log for the kernel related package, I could run something like:
yum update kernel kernel-headers --changelog
This will produce output similiar to:
Changes in packages about to be updated:
kernel-headers - 2.6.18-92.1.22.el5.x86_64
* Wed Dec 17 06:00:00 2008 Karanbir Singh [2.6.18-92.1.22.el5.centos]
- Roll in CentOS Branding
* Sat Dec 6 06:00:00 2008 Jiri Pirko [2.6.18-92.1.22.el5]
- [misc] hugepages: ia64 stack overflow and corrupt memory (Larry Woodman ) [474347 472802]
- [misc] allow hugepage allocation to use most of memory (Larry Woodman ) [474760 438889]
Ah, ha. As I suspected. Two memory related bugfixes and CentOS branding. Because we are currently not expirencing any memory related issues, this patch does NOT rate as critical and warrent immediate customer downtime. This can be delayed.
So now I can apply the other patches and exclude the kernel upgrades with:
yum update --exclude=kernel,kernel-headers
Now, I have a script that runs nightly on all my CentOS servers. This script gathers nightly statistics, logs entries, etc from my servers and emails it to me. This is pretty much jsut a CentOS port of my old Gentoo Update Script, with some CentOS speficic changes and additional features. The other thing it does, is generate a list (via yum check-update) of all the updates required. So the question now is, now can I get this interactive command to run via an automated script? The easiest way I could come up with is:
echo n | yum update kernel kernel-headers --changelog
Probably not the cleanest way, but does the job very well.
REFERENCE
http://monkey-house-org.blogspot.com/2009/02/centos-patching.html
