"Error: Failed while fetching Server version! Could be due to unauthorized access." when you run mysql_upgrade

Recently I met this error "Error: Failed while fetching Server version! Could be due to unauthorized access." when I have upgrade mysql to a new version and run "mysql_upgrade", which version doesn't matter here.

# mysql_upgrade -uroot -p                     
Enter password:
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Error: Failed while fetching Server version! Could be due to unauthorized access.
FATAL ERROR: Upgrade failed

How to fix the error? Let's start troubleshooting.

The first clue is "Failed while fetching Server version", so we skip version check and see how.

# mysql_upgrade --skip-version-check -uroot -p

Enter password:

Looking for 'mysql' as: mysql

Looking for 'mysqlcheck' as: mysqlcheck

Running 'mysqlcheck with default connection arguments

mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) when trying to connect

FATAL ERROR: Error during call to mysql_check for fixing the db/tables names.

OK, the problem is much clearer,  it's can not connect through socket, so we check the socket file.

# ll /var/lib/mysql/mysql.sock

ls: cannot access /var/lib/mysql/mysql.sock: No such file or directory

The socket file doesn't exist in /var/lib/mysql. 

Let's check mysql process.

# ps axu |grep mysql

root      6098  0.0  0.0  11440  1536 ?        S    17:24   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/data/mysql/ --pid-file=/data/mysql/mysql.pid

mysql     6931  3.0  7.7 20851736 2544580 ?    Sl   17:24   7:21 /usr/sbin/mysqld --basedir=/usr --datadir=/data/mysql/ --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/data/mysql/mysql-error.log --open-files-limit=65535 --pid-file=/data/mysql/mysql.pid --socket=/data/mysql/mysql.sock --port=3306

See? The socket file is under /data/mysql. This is because in my.cnf

datadir   =   /data/mysql

So we create a soft link /data/mysql to /var/lib/mysql

#ln -s /data/mysql /var/lib/mysql

Problem solved.