Wednesday, March 8, 2017

systemctrl seems to ignore my.cnf no matter where it lives. selinux is in the way!

I recently faced a daunting problem.  The systems group turned over a new host to me, mysql was installed but not running, my configuration file was present, and all seemed perfect.

I double checked /etc/my.cnf and tweaked a couple of settings and went to initialize the db and get it running.     mysqld --initialize-insecure
worked perfect, my new ibdata and ib_logfile files got created, the log was clean, and I thought whoopee, let's just start this up.
    systemctl start mysqld 
was my next command.

Whoa!.  What's going on here.  All the log messages are going to /var/log/messages and mysql is not starting but is cycling up and down up and down.  Pretty soon /var/log/messages was full and my bad hair day began.

Well there must be a renegade my.cnf file somewhere.  Checking mysqld to see where it thought the files might live I ran
    mysqld --help --verbose 
and found the lines
    Default options are read from the following files in the given order:
    /etc/my.cnf /etc/mysql/my.cnf /usr/etc/my.cnf ~/.my.cnf
and so dutifully went off to see if one of those other files existed and was overwriting my options.  

No such luck.  The sound you hear is my hair being pulled out.  After 72 hours, reinstalling mysql, running initialize multiple time, making my.cnf small with just a few options I was no closer to a solution.  

But then (clue music) I remembered having something like this a long time ago.  selinux.  Checking 
/etc/selinux/config I found
    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    #     enforcing - SELinux security policy is enforced.
    #     permissive - SELinux prints warnings instead of enforcing.
    #     disabled - No SELinux policy is loaded.
    SELINUX=disabled
    # SELINUXTYPE= can take one of three two values:
    #     targeted - Targeted processes are protected,
    #     minimum - Modification of targeted policy. Only selected processes are     protected.
    #     mls - Multi Level Security protection.
    SELINUXTYPE=targeted
which is exactly what I would expect EXCEPT that the time stamp was more recent than the latest uptime.   Sure enough, the os team had fixed selinux like I asked, but they forgot to reboot the host.  selinux was getting in my way and until it was resolved mysql would never start.

Now, after a reboot, all was well,  mysql reads my my.cnf and starts up just fine.  

Remember, after disabling selinux you must reboot.  I surely won't forget to check this again. 

No comments:

Post a Comment