openstack高可用环境搭建（一）：非高可用环境的搭建

1 方案设计

四节点基本信息：

10.192.44.148

10.192.44.149

10.192.44.150

10.192.44.151

每台设备1个128G的ssd系统盘，4个2T的数据盘

usr:root

pwd:9b648

1.1 网络方案

目前先采用单网卡方案，即每台设备使用一个网卡。IP地址即采用目前的地址。

后续将管理网络、存储网络、存储管理网络、虚拟机网络、外部网络分开。目前采用单网卡方式。

IP地址列表：

HostnameIP(eth0)IP1(备用IP)

隧道IP(eth0:1)openstack roleCeph mon roleCeph osd配置Vip

node110.192.44.148(eth0)172.16.2.148(eth3)

Controller1+network1Mon0Osd0~osd34Core 16G

node210.192.44.149(eth0)172.16.2.149(eth1)

Controller2+network2Mon1Osd4~osd74Core 16G

node310.192.44.150(eth0)172.16.2.150(eth1)

Compute1Mon2Osd8~osd114Core 16G

node410.192.44.151(eth1)172.16.2.151(eth2)

Compute2Mon3Osd12~osd158Core 16G

注意，后来已调整为：

因为150、151升级了libvirt，然后150每次重启后都不通

所以改为先安装(控制节点+网络节点：148)+(计算节点：149)，后面高可用再把150、151安装上：

HostnameIP(eth0)IP1(备用IP)

隧道IP(eth0:1)openstack roleCeph mon roleCeph osd配置Vip

10.192.44.148(eth0)172.16.2.148(eth3)eth0:1controller1+network1Mon0Osd0~osd34Core 16G

10.192.44.149(eth0)172.16.2.149(eth1)eth0:1compute1Mon1Osd4~osd74Core 16G

第二个网口的IP： 172.16.2.148 149150 151

[root@compute1 network-scripts]# catifcfg-eth1

DEVICE=eth1

ONBOOT=yes

STARTMODE=onboot

MTU=1500

BOOTPROTO=static

IPADDR=172.16.2.150

NETMASK=255.255.255.0

GATEWAY=10.192.44.254

使用两个节点作为(控制+网络)的复用节点

使用两个节点作为计算节点

原因：

控制节点运行服务非常多，不适合全部用来复用为计算节点跑虚拟机

计算节点需要资源多(CPU、内存)，所以最大的资源的那台作为计算节点。

1.2 存储方案

目前系统盘为SSD：128G，存储盘为SATA：2T

系统盘还有空间，将剩余空间作为ceph的journal空间，剩余大概有90G，设置为hda5分区。

HostnameCeph monCeph journalCeph osd

Node1Mon0/dev/hda5Osd.0~osd.3: sdb1/sdc1/sdd1/sde1

Node2Mon1/dev/hda5Osd.4~osd.7: sdb1/sdc1/sdd1/sde1

Node3Mon2/dev/hda5Osd.8~osd.11: sdb1/sdc1/sdd1/sde1

Node4Mon3/dev/hda5Osd.12~osd.15: sdb1/sdc1/sdd1/sde1

Rbd pools:

ServiceRbd poolPg nums

GlanceImages128

CinderVolumes128

NovaVms128

备注：先给磁盘分区，否则安装时给sdx分区，每个磁盘全部分成sdx1，ceph-deploy 分出来的sdx1只有5G大小。

1.3 运行服务

Openstack role

ControllerHttpd，rabbitmq，mysql;keystone, glance, neutron-server, nova-api & scheduler, cinder-api & scheduler

NetworkNeutron agents：l3-agent, openvswitch-agent, dhcp-agent

ComputeNova-compute，neutron-openvswitch，cinder-volume

1.4 其他备注

(1)暂时不安装ceilometer，比较耗资源，且当前不对ceilometer做高可用

(2)暂时不安装swift对象存储

(3)资源非常有限，验证时可能只能开2台虚拟机

1.5 特别注意

1.每走一步，验证一下创建镜像、云硬盘、网络、虚拟机功能，避免错误积累导致重装

2.修改的各种配置注意保存到git上：

https://git.hikvision.com.cn/projects/FSDMDEPTHCLOUD/repos/hcloud_install_centos/browse/project_beijing

3.不明确的问题一定要现在虚拟机上验证

4.Horizon还是先安装2个

2基本环境

2.1yum还是rpm

yum源：

配置内核和系统不升级：

/etc/yum.conf：

keepcache=1

exclude=kernel*

exclude=centos-release*

删除原来的

# rm yum.repos.d/ -rf

替换现在的

然后更新源：

# yum clean all

# yum makecache

# yum update –y

#yum upgrade –y

坚决不能执行yum update和yum upgrade

后续改进：此处后续做成自动化脚本

安装方案：

1.在虚拟机上使用yum安装一遍all-in-one，把缓存的rpm包保存下来

2.实体机上使用rpm包安装，实现持续集成，持续集成终究要做rpm包来安装，不如现在一次搞好，实现脚本化

[root@node1 etc]# vi yum.conf

[main]

cachedir=/var/cache/yum

keepcache=1

(1)Ceph的使用yum 安装完全没问题

(2)Openstack使用all-in-one先安装一个节点来检查环境

2.2/etc/hostname的设置

[root@localhost ~]# cat /etc/hostname

node1

[root@localhost ~]# cat /etc/hostname

node2

[root@localhost ~]# cat /etc/hostname

node3

[root@localhost ~]# cat /etc/hostname

node4

后续改进：此处集成到自动化脚本

2.3 /etc/hosts设置

[root@localhost ~]# vi /etc/hosts

127.0.0.1localhost localhost.localdomain localhost4 localhost4.localdomain4

::1localhost localhost.localdomain localhost6 localhost6.localdomain6

10.1.14.235mirrors.hikvision.com.cn

10.192.44.148 node1

10.192.44.149 node2

10.192.44.150 node3

10.192.44.151 node4

后续改进：此处集成到自动化脚本

2.4关闭防火墙

systemctl stop firewalld.service

systemctl disable firewalld.service

2.5 各种命令汇总

systemctl stop firewalld.service

systemctl disable firewalld.service

yum install ceph -y

yum install ceph-deploy -y

yum install yum-plugin-priorities -y

yum install snappy leveldb gdiskpython-argparse gperftools-libs -y

#ceph-deploy new lxp-node1 lxp-node2lxp-node3

# ceph-deploy install lxp-node1 lxp-node2lxp-node3

#ceph-deploy--overwrite-conf mon create-initial

ceph-deploy mon create lxp-node1 lxp-node2lxp-node3

ceph-deploy gatherkeys lxp-node1 lxp-node2lxp-node3

/etc/init.d/ceph-a start osd

systemctl enable haproxy

systemctl start haproxy

systemctl enable keepalived

systemctl start keepalived

# systemctl enable rabbitmq-server.service

# systemctl start rabbitmq-server.service

# rabbitmqctl add_user guest guest

chown rabbitmq:rabbitmq .erlang.cookie

rabbitmqctl stop_app

rabbitmqctljoin_cluster rabbit@lxp-node1

rabbitmqctlstart_app

rabbitmqctl cluster_status

rabbitmqctl set_policy ha-all'^(?!amq\.).*' '{"ha-mode": "all"}'

yuminstall MySQL-python mariadb-galera-server galera xtrabackup socat

# systemctl enable mariadb.service

# systemctl restart mariadb.service

yuminstall openstack-keystone httpd mod_wsgi python-openstackclient memcachedpython-memcached

systemctlenable memcached.service

systemctlstart memcached.service

#yum install python-pip

#pip install eventlet

mkdir-p /var/www/cgi-bin/

将node1的keystone打包过来解压

chown-R keystone:keystone /var/www/cgi-bin/keystone

chmod755 /var/www/cgi-bin/keystone/ -R

重启httpd:

# systemctlenable httpd.service

#systemctl start httpd.service

[root@lxp-node1~]# export OS_TOKEN=c5a16fa8158c4208b5764c00554bde49

[root@lxp-node1~]# export OS_URL=http://192.168.129.130:35357/v2.0

#systemctlenable openstack-glance-api.service openstack-glance-registry.service

#systemctl start openstack-glance-api.serviceopenstack-glance-registry.service

systemctlrestart openstack-glance-api.service openstack-glance-registry.service

systemctlrestart openstack-glance-api.service openstack-glance-registry.service

MariaDB [(none)]> GRANT ALL PRIVILEGESON glance.* TO 'glance'@'localhost' IDENTIFIED BY '6fbbf50542084b7c';

MariaDB [(none)]> GRANT ALL PRIVILEGESON glance.* TO 'glance'@'%' IDENTIFIED BY '6fbbf50542084b7c';

MariaDB [(none)]> FLUSH PRIVILEGES;

ceph osd tree

/etc/init.d/ceph-a start osd

# ceph-deploy --overwrite-conf osd preparelxp-node1:/data/osd4.lxp-node1:/dev/sdb2lxp-node2:/data/osd5.lxp-node2:/dev/sdb2lxp-node3:/data/osd6.lxp-node3:/dev/sdb2

# ceph-deploy --overwrite-conf osd activatelxp-node1:/data/osd4.lxp-node1:/dev/sdb2lxp-node2:/data/osd5.lxp-node2:/dev/sdb2lxp-node3:/data/osd6.lxp-node3:/dev/sdb2

# ceph osd lspools

# ceph pg stat

ceph osd pool create image 32

# ceph osd lspools

yum installopenstack-dashboardhttpd mod_wsgi memcached pythonmemcached

# systemctlrestarthttpd.service

yuminstall openstack-nova-api openstack-nova-cert openstack-nova-conductor openstack-nova-consoleopenstack-nova-novncproxy penstack-nova-scheduler python-novaclient

# systemctl enable openstack-nova-api.service openstack-nova-cert.service openstack-nova-consoleauth.serviceopenstack-nova-scheduler.service openstack-nova-conductor.serviceopenstack-nova-novncproxy.service

# systemctl start openstack-nova-api.service openstack-nova-cert.serviceopenstack-nova-consoleauth.service openstack-nova-scheduler.serviceopenstack-nova-conductor.service openstack-nova-novncproxy.service

GRANTALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'b7cf13724ff948d7';

GRANTALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'b7cf13724ff948d7';

FLUSH PRIVILEGES;

# systemctl restart openstack-nova-api.service openstack-nova-cert.serviceopenstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.serviceopenstack-nova-novncproxy.service

yuminstall openstack-cinder python-cinderclient python-oslo-db

# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service

# systemctl start openstack-cinder-api.serviceopenstack-cinder-scheduler.service

# systemctl restartopenstack-cinder-api.service openstack-cinder-scheduler.service

GRANT ALL PRIVILEGES ON cinder.* TO'cinder'@'localhost' IDENTIFIED BY 'afdfc435eb0b4372';

GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%'IDENTIFIED BY 'afdfc435eb0b4372';

FLUSH PRIVILEGES;

yuminstall openstack-neutron openstack-neutron-ml2 python-neutronclient

MariaDB [(none)]> GRANT ALL PRIVILEGESON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY '11be293368c044cb';

MariaDB [(none)]> GRANT ALL PRIVILEGESON neutron.* TO 'neutron'@'%' IDENTIFIED BY '11be293368c044cb';

MariaDB [(none)]> FLUSH PRIVILEGES;

# systemctl enable neutron-server.service

# systemctl restart neutron-server.service

# systemctl enable openvswitch.service

# systemctl restart openvswitch.service

# systemctl enable neutron-openvswitch-agent.serviceneutron-l3-agent.service neutron-dhcp-agent.serviceneutron-metadata-agent.service neutron-ovs-cleanup.service

# systemctl restart neutron-openvswitch-agent.serviceneutron-l3-agent.service neutron-dhcp-agent.serviceneutron-metadata-agent.service

systemctlrestart openstack-nova-api.service openstack-nova-scheduler.serviceopenstack-nova-conductor.service

# systemctl enable openstack-cinder-volume.service target.service

# systemctl start openstack-cinder-volume.service target.service

2.6 遇到的差异记录

yum -y update

升级所有包，改变软件设置和系统设置,系统版本内核都升级

yum -y upgrade

升级所有包，不改变软件设置和系统设置，系统版本升级，内核不改变

貌似系统起不来了

2.7 特别注意

配置一项就要确认一下虚拟机、vnc是否还正常

如果系统起不来需要重装，那么立即沟通

在北京重装的过程中，自己做尽可能多的验证

3. Ceph安装

3.1 磁盘分区：SSD(/dev/hda5)作为Ceph journal盘

原因：Journal使用SSD盘，会对Ceph性能有一定的提升。

设置/dev/hda5自动挂载到/data/目录

(1)将hda5做成ext4系统：

# mkfs.ext4 /dev/hda5

(2)创建目录/data

# mkdir /data

(3)在/etc/fstab中添加：

/dev/hda5/data ext4defaults,async,noatime,nodiratime,data=writeback,barrier=0 0 0

(4)重启验证是否被挂载

[root@node1 ~]# mount |grep hda5

/dev/hda5 on /data type ext4(rw,noatime,nodiratime,nobarrier,data=writeback)

OK，挂载成功

SSD(/dev/hda5)开机自动挂载为本地目录作为journal盘的方法：

[root@lxp-node1 osd.lxp-node1]# ceph-deployosd prepare --help

usage: ceph-deploy osd prepareHOST:DISK[:JOURNAL] [HOST:DISK[:JOURNAL] ...]

3.2 ceph、ceph-deploy安装

Ceph的包目前使用rpm安装没有任何问题：

yum install ceph -y

yum install ceph-deploy -y

yum install yum-plugin-priorities -y

yum install snappy leveldb gdiskpython-argparse gperftools-libs -y

重启看是否还可以重启成功。Ceph目前只需要这些包

可以重启成功，先不处理ceph，没有风险，先处理openstack。先验证安装方案!

3.3 ceph mon安装

放到第7章

3.4 ceph osd安装

放到第7章

4. 使用packstack进行openstack基本环境安装(4节点)--【此路不通】

先安装一个all-in-one的，检查一下环境是否有冲突

在10.192.44.148使用packstack安装openstack的all-in-one环境

验证安装方案的可行性

# yum install openstack-packstack

# yum install screen

# packstack--gen-answer-file=hcloud.txt

关闭如下选项：

CONFIG_PROVISION_DEMO=n

CONFIG_CEILOMETER_INSTALL=n

CONFIG_SWIFT_INSTALL=n

CONFIG_NAGIOS_INSTALL=n

安装：

# screen packstack --answer-file=hcloud.txt

出现问题：

10.192.44.151_mariadb.pp: [ ERROR ]

Applying Puppet manifests [ ERROR ]

4.1 解决数据库冲突问题

ERROR : Error appeared during Puppet run:10.192.44.151_mariadb.pp

Error: Execution of '/usr/bin/rpm -emariadb-server-5.5.35-3.el7.x86_64' returned 1: error: Failed dependencies:

You will find full trace in log/var/tmp/packstack/20160524-195517-yG5qIz/manifests/10.192.44.151_mariadb.pp.log

数据库安装错误，依赖问题，

删除原来的mysql包，packstack会下载galera版本的mariadb：

[root@localhost ~]# rpm -aq |grep maria

mariadb-devel-5.5.35-3.el7.x86_64

mariadb-5.5.35-3.el7.x86_64

mariadb-test-5.5.35-3.el7.x86_64

mariadb-libs-5.5.35-3.el7.x86_64

mariadb-embedded-5.5.35-3.el7.x86_64

mariadb-embedded-devel-5.5.35-3.el7.x86_64

mariadb-server-5.5.35-3.el7.x86_64

# rpm -e --nodeps mariadb-devel mariadbmariadb-test mariadb-libs mariadb-embedded mariadb-embedded-develmariadb-server

重新安装openstack，在node1上：

删除之后还是有问题

手动安装试试：

把galera也删除：

# rpm -e --nodeps mariadb-galera-commonmariadb-galera-server galera

# rpm -e --nodeps mariadb-libs mariadb

[root@localhost ~]# rpm -aq |grepmaria

[root@localhost ~]# rpm -aq |grep galera

手动验证：

# yum install mariadb mariadb-serverMySQL-python

问题：

Error: mariadb-galera-server conflicts with1:mariadb-server-5.5.44-2.el7.centos.x86_64

解决：

OK，可以安装

再次用packstack，看是否还会出错，如果出错，选择mariadb_install=n是否可行

如果不行，修改packstack是否可行

手动安装一直无法启动，有如下和sql相关的报错：

yum installmariadb-galera-server galera

MySQL-python-1.2.3-11.el7.x86_64 hasmissing requires of libmysqlclient.so.18()(64bit)

MySQL-python-1.2.3-11.el7.x86_64 hasmissing requires of libmysqlclient.so.18(libmysqlclient_18)(64bit)

perl-DBD-MySQL-4.023-5.el7.x86_64 hasmissing requires of libmysqlclient.so.18()(64bit)

perl-DBD-MySQL-4.023-5.el7.x86_64 hasmissing requires of libmysqlclient.so.18(libmysqlclient_18)(64bit)

Installing : 1:mariadb-libs-5.5.44-2.el7.centos.x86_641/5

/sbin/ldconfig: /lib64/libosipparser2.so.3is not a symbolic link

/sbin/ldconfig: /lib64/libeXosip2.so.4 isnot a symbolic link

/sbin/ldconfig: /lib64/libosip2.so.3 is nota symbolic link

/sbin/ldconfig: /lib64/libosipparser2.so.3is not a symbolic link

/sbin/ldconfig: /lib64/libeXosip2.so.4 isnot a symbolic link

/sbin/ldconfig: /lib64/libosip2.so.3 is nota symbolic link

解决：/sbin/ldconfig: /lib64/libosipparser2.so.3 is not a symbolic link

(1)

[root@localhost lib64]# rmlibosipparser2.so.3

[root@localhost lib64]# ln -slibosipparser2.so libosipparser2.so.3

[root@localhost lib64]# ls libosipparser2.*-l

-rw-r--r-- 1 root root 707666 Apr 1 13:52 libosipparser2.a

-rw-r--r-- 1 root root 857 Apr1 13:52 libosipparser2.la

-rw-r--r-- 1 root root 380223 Apr 1 13:52 libosipparser2.so

lrwxrwxrwx 1 root root 17 May 24 21:06 libosipparser2.so.3 ->libosipparser2.so

解决：

/sbin/ldconfig: /lib64/libeXosip2.so.4 isnot a symbolic link

(2)

[root@localhost lib64]# rm libeXosip2.so.4

[root@localhost lib64]# ln -s libeXosip2.solibeXosip2.so.4

[root@localhost lib64]# ls libeXosip2.so*-l

-rw-r--r-- 1 root root 818385 Apr 1 13:52 libeXosip2.so

lrwxrwxrwx 1 root root 13 May 24 21:08 libeXosip2.so.4 ->libeXosip2.so

(3)/sbin/ldconfig: /lib64/libosip2.so.3 is not a symbolic link

解决：

[root@localhost lib64]# rm libosip2.so.3

[root@localhost lib64]# ln -s libosip2.solibosip2.so.3

重新安装：

yum installmariadb-galera-server galera

没有再报这些错误!!!!!

还有其他依赖错误：

MySQL-python-1.2.3-11.el7.x86_64 has missingrequires of libmysqlclient.so.18()(64bit)

MySQL-python-1.2.3-11.el7.x86_64 hasmissing requires of libmysqlclient.so.18(libmysqlclient_18)(64bit)

10:libcacard-1.5.3-60.el7.x86_64 hasmissing requires of libgfapi.so.0()(64bit)

10:libcacard-1.5.3-60.el7.x86_64 hasmissing requires of libgfrpc.so.0()(64bit)

10:libcacard-1.5.3-60.el7.x86_64 hasmissing requires of libgfxdr.so.0()(64bit)

libvirt-daemon-driver-storage-1.1.1-29.el7.x86_64has missing requires of libgfapi.so.0()(64bit)

libvirt-daemon-driver-storage-1.1.1-29.el7.x86_64has missing requires of libgfrpc.so.0()(64bit)

libvirt-daemon-driver-storage-1.1.1-29.el7.x86_64has missing requires of libgfxdr.so.0()(64bit)

libvirt-daemon-driver-storage-1.1.1-29.el7.x86_64has missing requires of libglusterfs.so.0()(64bit)

perl-DBD-MySQL-4.023-5.el7.x86_64 hasmissing requires of libmysqlclient.so.18()(64bit)

perl-DBD-MySQL-4.023-5.el7.x86_64 hasmissing requires of libmysqlclient.so.18(libmysqlclient_18)(64bit)

2:postfix-2.10.1-6.el7.x86_64 has missingrequires of libmysqlclient.so.18()(64bit)

2:postfix-2.10.1-6.el7.x86_64 has missingrequires of libmysqlclient.so.18(libmysqlclient_18)(64bit)

10:qemu-img-1.5.3-60.el7.x86_64 has missingrequires of libgfapi.so.0()(64bit)

10:qemu-img-1.5.3-60.el7.x86_64 has missingrequires of libgfrpc.so.0()(64bit)

10:qemu-img-1.5.3-60.el7.x86_64 has missingrequires of libgfxdr.so.0()(64bit)

10:qemu-kvm-1.5.3-60.el7.x86_64 has missingrequires of libgfapi.so.0()(64bit)

10:qemu-kvm-1.5.3-60.el7.x86_64 has missingrequires of libgfrpc.so.0()(64bit)

10:qemu-kvm-1.5.3-60.el7.x86_64 has missingrequires of libgfxdr.so.0()(64bit)

10:qemu-kvm-common-1.5.3-60.el7.x86_64 hasmissing requires of libgfapi.so.0()(64bit)

10:qemu-kvm-common-1.5.3-60.el7.x86_64 hasmissing requires of libgfrpc.so.0()(64bit)

10:qemu-kvm-common-1.5.3-60.el7.x86_64 hasmissing requires of libgfxdr.so.0()(64bit)

先试一下可否启动mysql

还是启动失败

将这些库从OK的环境整理过来：

解决：

MySQL-python-1.2.3-11.el7.x86_64 hasmissing requires of libmysqlclient.so.18()(64bit)

MySQL-python-1.2.3-11.el7.x86_64 hasmissing requires of libmysqlclient.so.18(libmysqlclient_18)(64bit)

还是启动失败：

手动启动试试：

/usr/bin/mysqld_safe --basedir=/usr

解决办法：

删除ib_logfile0 ib_logfile1 文件：

#cd /var/lib/mysql/

#rm ib_logfile0 ib_logfile1

重启mysql服务

还是有错：

mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

解决：

touch /var/run/mariadb/mariadb.pi

# chown mysql:mysql/var/run/mariadb/mariadb.pid

# chmod 0660 /var/run/mariadb/mariadb.pid

再次启动：

# systemctl enable mariadb.service

# systemctl restart mariadb.service

查看/var/log/mariadb/mariadb.log，报如下错误：

160525 8:34:16 [Note] Plugin 'FEEDBACK' is disabled.

160525 8:34:16 [Note] Server socket created on IP:'0.0.0.0'.

1605258:34:16 [ERROR] Fatal error: Can't open and lock privilege tables: Table'mysql.host' doesn't exist

16052508:34:16 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended

执行mysql_install_db

Mariadb.log中报如下错误：

1605258:37:30 [Note] WSREP: Read nil XID from storage engines, skippingposition init

1605258:37:30 [Note] WSREP: wsrep_load(): loading provider library 'none'

1605258:37:30 [ERROR] mysqld: Incorrect information in file:'./mysql/tables_priv.frm'

ERROR: 1033Incorrect information in file: './mysql/tables_priv.frm'

1605258:37:30 [ERROR] Aborting

重启：

1605258:39:42 [ERROR] mysqld: Can't find file: './mysql/host.frm' (errno: 13)

1605258:39:42 [ERROR] Fatal error: Can't open and lock privilege tables: Can'tfind file: './mysql/host.frm' (errno: 13)

可能是权限问题：

http://181054867.iteye.com/blog/614656

/var/lib/mysql

[root@localhost mysql]# pwd

/var/lib/mysql

[root@localhost mysql]# chmod 770 mysql/ -R

还是报找不到

改成777：

chmod 777mysql/ -R

继续重启

报错：

1605258:45:20 [ERROR] mysqld: Incorrect information in file:'./mysql/proxies_priv.frm'

1605258:45:20 [ERROR] Fatal error: Can't open and lock privilege tables:Incorrect information in file: './mysql/proxies_priv.frm'

[root@localhost mysql]# rm proxies*

重启后重启成功

但是启动日志会有很多错误：

tail -f mariadb.log

1605258:48:38 [ERROR] mysqld: Incorrect information in file: './mysql/tables_priv.frm'

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'THREAD_ID' atposition 0 to have type int(11), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'EVENT_NAME' atposition 2, found 'END_EVENT_ID'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'EVENT_NAME' atposition 2 to have type varchar(128), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'SOURCE' at position3, found 'EVENT_NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'SOURCE' at position 3to have type varchar(64), found type varchar(128).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'TIMER_START' atposition 4, found 'SOURCE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'TIMER_START' atposition 4 to have type bigint(20), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'TIMER_END' atposition 5, found 'TIMER_START'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'TIMER_WAIT' atposition 6, found 'TIMER_END'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'SPINS' at position 7,found 'TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_current:expected column 'SPINS' at position 7 to have type int(10), found typebigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OBJECT_SCHEMA' atposition 8, found 'SPINS'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OBJECT_SCHEMA' atposition 8 to have type varchar(64), found type int(10) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OBJECT_NAME' atposition 9, found 'OBJECT_SCHEMA'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OBJECT_NAME' atposition 9 to have type varchar(512), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OBJECT_TYPE' atposition 10, found 'OBJECT_NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OBJECT_TYPE' atposition 10 to have type varchar(64), found type varchar(512).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column'OBJECT_INSTANCE_BEGIN' at position 11, found 'INDEX_NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column'OBJECT_INSTANCE_BEGIN' at position 11 to have type bigint(20), found typevarchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'NESTING_EVENT_ID' atposition 12, found 'OBJECT_TYPE'.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_current:expected column 'NESTING_EVENT_ID' at position 12 to have type bigint(20),found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OPERATION' at position13, found 'OBJECT_INSTANCE_BEGIN'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'OPERATION' atposition 13 to have type varchar(16), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'NUMBER_OF_BYTES' atposition 14, found 'NESTING_EVENT_ID'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'FLAGS' at position15, found 'NESTING_EVENT_TYPE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_current: expected column 'FLAGS' at position 15to have type int(10), found type enum('STATEMENT','STAGE','WAIT').

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'THREAD_ID' atposition 0 to have type int(11), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_history:expected column 'EVENT_NAME' at position 2, found 'END_EVENT_ID'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'EVENT_NAME' atposition 2 to have type varchar(128), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'SOURCE' at position3, found 'EVENT_NAME'.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_history:expected column 'SOURCE' at position 3 to have type varchar(64), found typevarchar(128).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'TIMER_START' at position4, found 'SOURCE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'TIMER_START' atposition 4 to have type bigint(20), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'TIMER_END' atposition 5, found 'TIMER_START'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'TIMER_WAIT' at position6, found 'TIMER_END'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'SPINS' at position 7,found 'TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_history:expected column 'SPINS' at position 7 to have type int(10), found typebigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OBJECT_SCHEMA' atposition 8, found 'SPINS'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OBJECT_SCHEMA' atposition 8 to have type varchar(64), found type int(10) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OBJECT_NAME' atposition 9, found 'OBJECT_SCHEMA'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OBJECT_NAME' atposition 9 to have type varchar(512), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OBJECT_TYPE' atposition 10, found 'OBJECT_NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OBJECT_TYPE' atposition 10 to have type varchar(64), found type varchar(512).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column'OBJECT_INSTANCE_BEGIN' at position 11, found 'INDEX_NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column'OBJECT_INSTANCE_BEGIN' at position 11 to have type bigint(20), found typevarchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'NESTING_EVENT_ID' atposition 12, found 'OBJECT_TYPE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'NESTING_EVENT_ID' atposition 12 to have type bigint(20), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OPERATION' atposition 13, found 'OBJECT_INSTANCE_BEGIN'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'OPERATION' atposition 13 to have type varchar(16), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'NUMBER_OF_BYTES' atposition 14, found 'NESTING_EVENT_ID'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'FLAGS' at position15, found 'NESTING_EVENT_TYPE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history: expected column 'FLAGS' at position 15to have type int(10), found type enum('STATEMENT','STAGE','WAIT').

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'THREAD_ID' atposition 0 to have type int(11), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_history_long:expected column 'EVENT_NAME' at position 2, found 'END_EVENT_ID'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'EVENT_NAME' atposition 2 to have type varchar(128), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'SOURCE' atposition 3, found 'EVENT_NAME'.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_history_long:expected column 'SOURCE' at position 3 to have type varchar(64), found typevarchar(128).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'TIMER_START' atposition 4, found 'SOURCE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'TIMER_START' atposition 4 to have type bigint(20), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'TIMER_END' atposition 5, found 'TIMER_START'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'TIMER_WAIT' atposition 6, found 'TIMER_END'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'SPINS' atposition 7, found 'TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'SPINS' atposition 7 to have type int(10), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OBJECT_SCHEMA'at position 8, found 'SPINS'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OBJECT_SCHEMA'at position 8 to have type varchar(64), found type int(10) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OBJECT_NAME' atposition 9, found 'OBJECT_SCHEMA'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OBJECT_NAME' atposition 9 to have type varchar(512), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OBJECT_TYPE' atposition 10, found 'OBJECT_NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OBJECT_TYPE' atposition 10 to have type varchar(64), found type varchar(512).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column'OBJECT_INSTANCE_BEGIN' at position 11, found 'INDEX_NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OBJECT_INSTANCE_BEGIN'at position 11 to have type bigint(20), found type varchar(64).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column'NESTING_EVENT_ID' at position 12, found 'OBJECT_TYPE'.

160525 8:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column'NESTING_EVENT_ID' at position 12 to have type bigint(20), found typevarchar(64).

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.events_waits_history_long:expected column 'OPERATION' at position 13, found 'OBJECT_INSTANCE_BEGIN'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'OPERATION' atposition 13 to have type varchar(16), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'NUMBER_OF_BYTES'at position 14, found 'NESTING_EVENT_ID'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'FLAGS' atposition 15, found 'NESTING_EVENT_TYPE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_history_long: expected column 'FLAGS' atposition 15 to have type int(10), found type enum('STATEMENT','STAGE','WAIT').

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.threads: expected column 'THREAD_ID' at position 0 to havetype int(11), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.threads: expected column 'PROCESSLIST_ID' at position 1,found 'NAME'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.threads: expected column 'PROCESSLIST_ID' at position 1 tohave type int(11), found type varchar(128).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.threads: expected column 'NAME' at position 2, found 'TYPE'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.threads: expected column 'NAME' at position 2 to have typevarchar(128), found type varchar(10).

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.events_waits_summary_by_thread_by_event_name: expected column'THREAD_ID' at position 0 to have type int(11), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.file_summary_by_event_name: expected column 'COUNT_READ' atposition 1, found 'COUNT_STAR'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.file_summary_by_event_name: expected column 'COUNT_WRITE' atposition 2, found 'SUM_TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.file_summary_by_event_name: expected column'SUM_NUMBER_OF_BYTES_READ' at position 3, found 'MIN_TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.file_summary_by_event_name: expected column'SUM_NUMBER_OF_BYTES_WRITE' at position 4, found 'AVG_TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.file_summary_by_instance: expected column 'COUNT_READ' atposition 2, found 'OBJECT_INSTANCE_BEGIN'.

1605258:48:38 [ERROR] Incorrect definition of table performance_schema.file_summary_by_instance:expected column 'COUNT_WRITE' at position 3, found 'COUNT_STAR'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.file_summary_by_instance: expected column'SUM_NUMBER_OF_BYTES_READ' at position 4, found 'SUM_TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.file_summary_by_instance: expected column'SUM_NUMBER_OF_BYTES_WRITE' at position 5, found 'MIN_TIMER_WAIT'.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.mutex_instances: expected column 'LOCKED_BY_THREAD_ID' atposition 2 to have type int(11), found type bigint(20) unsigned.

1605258:48:38 [ERROR] Incorrect definition of tableperformance_schema.rwlock_instances: expected column'WRITE_LOCKED_BY_THREAD_ID' at position 2 to have type int(11), found typebigint(20) unsigned.

1605258:48:38 [ERROR] mysqld: Incorrect information in file:'./mysql/event.frm'

1605258:48:38 [ERROR] Cannot open mysql.event

1605258:48:38 [ERROR] Event Scheduler: An error occurred when initializingsystem tables. Disabling the Event Scheduler.

1605258:48:38 [Note] WSREP: Read nil XID from storage engines, skippingposition init

将表信息全部删除：

其他尝试：

升级数据库表失败：

# /usr/bin/mysql_upgrade -u root

终极解决办法：

rpm 删除包之后，手动清理一下mysql文件：

[root@localhost var]# find ./ -name mysql

./lib/mysql

./lib/mysql/mysql

[root@localhost var]# rm ./lib/mysql/ -rf

[root@localhost usr]# find ./ -name mysql |xargs rm –rf

再次安装试试：

yum install mariadb-galera-server galera

启动：

# systemctl enable mariadb.service

# systemctl start mariadb.service

OK，终于解决了

需要手动清理干净!!!

将/etc/my.cnf和/etc/my.cnf.d也清理掉

4.2 解决后数据库问题重新使用packstack自动化安装，检查环境差异

清理后重新用packstack重装，否则带着手动安装的还是会报错

依旧会报错，手动使用yum逐个组件安装