redash迁移

工作中,需要把一台自有实体机上的redash迁移到公司容器中。目标容器系统为centos6。迁移过程如下:

redash基础环境搭建

redis
  • 安装redis
cd /opt/xxx/apps
[xxx@set-xxx-xxx02 apps]$ mkdir redis
wget http://download.redis.io/releases/redis-5.0.5.tar.gz
tar -zxvf redis-5.0.5.tar.gz
cd redis-5.0.5
make
  • 启动redis
cd redis-5.0.5/src
./redis-server
  • 测试redis启动ok
cd redis-5.0.5/src
./redis-cli
127.0.0.1:6379> ping
PONG
PostgreSQL

以下操作为root权限

  • 安装PostgreSQL
sudo yum install -y postgresql96-server
  • 启动PostgreSQL
pg_ctl -D /opt/xxx/pgdata/ start

若启动报错:

[xxx@set-xxx-xxx02 ~]$ < 2019-07-22 11:33:20.555 CST > FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Permission denied

解决方法:给other用户加写权限

sudo chmod a+w /var/run/postgresql

然后再次启动。
进入psql报错:

psql
psql: FATAL:  database "xxx" does not exist

解决方法: https://stackoverflow.com/questions/17633422/psql-fatal-database-user-does-not-exist

[xxx@set-xxx-xxx02 ~]$ createdb xxx
[xxx@set-xxx-xxx02 ~]$ psql
psql (9.6.14)
Type "help" for help.

xxx=#

修改密码:

xxx=# \password
Enter new password:
Enter it again:
xxx=#
  • 可执行文件加入到xxx用户的环境变量,修改/home/xxx/.bash_profile
export PATH=/usr/pgsql-9.6/bin:$PATH
source /home/xxx/.bash_profile
nodejs
  • 安装nodejs
cd /opt/xxx/apps
wget https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.xz
tar -xvf node-v10.9.0-linux-x64.tar.xz
mv node-v10.9.0-linux-x64 nodejs
ln -s /opt/xxx/apps/nodejs/bin/npm /usr/local/bin/
ln -s /opt/xxx/apps/nodejs/bin/node /usr/local/bin/
Python2.7

以下操作为root权限

yum install -y gcc openssl-devel bzip2-devel
cd /home/xxx/
wget https://www.python.org/ftp/python/2.7.16/Python-2.7.16.tgz
tar xzf Python-2.7.16.tgz
cd Python-2.7.16
./configure --enable-optimizations
make altinstall
  • 安装pip
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python2.7 get-pip.py
  • 可执行文件加入到xxx用户的环境变量,修改/home/xxx/.bash_profile
export PATH=/usr/local/bin:/usr/pgsql-9.6/bin:$PATH
source /home/xxx/.bash_profile
  • 安装Python隔离环境
pip install virtualenv
virtualenv -p /usr/local/bin/python2.7 current
原redash代码和数据备份,并copy到新机器上
  • 数据备份
pg_dump -U postgres -d postgres -f postgres.sql
pg_dump -U postgres -d qa_mgmt -f qa_mgmt.sql
pg_dump -U postgres -d redash -f redash.sql
redash
  • 安装python包
source /home/xxx/current/bin/activate
cd /home/xxx/redash-7.0.0
pip install -r requirements.txt -r requirements_dev.txt -r requirements_all_ds.tx
  • 安装nodejs包
cd /home/xxx/redash-7.0.0/client
npm install
npm run build
  • 还远原数据库数据到新机器上的postgresql中
psql -U xxx -d redash -f redash.sql
psql -U xxx -d qa_mgmt -f qa_mgmt.sql
psql -U xxx -d postgres -f postgres.sql
  • 配置.env
REDASH_DATABASE_URL="postgresql://xxx:xxxxxx@localhost/redash"
Supervisor

以下为root权限

  • 安装Supervisor
pip install supervisor
  • 创建配置文件
echo_supervisord_conf > /etc/supervisord.conf
  • 修改配置文件
    修改以下几处:
[unix_http_server]
file=/var/run/supervisor.sock   ; the path to the socket file
[supervisord]
logfile=/var/log/supervisord.log ; main log file; default $CWD/supervisord.log
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket
[include]
files = /etc/supervisor.d/*.conf
  • 加写权限
chmod 777 /var/run/supervisor.sock
chmod 777 /var/log/supervisord.log
  • 进程配置
mkdir -p /etc/supervisor.d
touch redash.conf

文件内容从原机器的.conf中copy出,主要改以下几个地方:目录替换为新目录,用户替换为xxx

[group:redash]
programs=redash_server,redash_celery,redash_celery_scheduled

[program:redash_server]
#environment=PYTHONPATH=/home/xxx/current/bin:%(ENV_PYTHONPATH)s,PATH=/home/xxx/current/bin:%(ENV_PATH)s
command=/home/xxx/redash-7.0.0/bin/run gunicorn -b 0.0.0.0:5000 --name redash -w 4 --max-requests 1000 redash.wsgi:app
directory=/home/xxx/redash-7.0.0
process_name=redash_server
user=xxx
numprocs=1
autostart=true
autorestart=true
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 20
stdout_logfile = /home/xxx/redash-7.0.0/logs/redash_server_stdout.log

# There are two queue types here: one for ad-hoc queries, and one for the refresh of scheduled queries
# (note that "scheduled_queries" appears only in the queue list of "redash_celery_scheduled").
# The default concurrency level for each is 2 (-c2), you can increase based on your machine's resources.

[program:redash_celery]
#environment=PYTHONPATH=/home/xxx/current/bin:%(ENV_PYTHONPATH)s,PATH=/home/xxx/current/bin:%(ENV_PATH)s
command=/home/xxx/redash-7.0.0/bin/run celery worker --app=redash.worker --beat -c2 -Qqueries,celery --maxtasksperchild=10 -Ofair
directory=/home/xxx/redash-7.0.0
process_name=redash_celery
user=xxx
numprocs=1
autostart=true
autorestart=true
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 20
stdout_logfile = /home/xxx/redash-7.0.0/logs/redash_celery_stdout.log

[program:redash_celery_scheduled]
#environment=PYTHONPATH=/home/xxx/current/bin:%(ENV_PYTHONPATH)s,PATH=/home/xxx/current/bin:%(ENV_PATH)s
command=/home/xxx/redash-7.0.0/bin/run celery worker --app=redash.worker -c2 -Qscheduled_queries --maxtasksperchild=10 -Ofair
directory=/home/xxx/redash-7.0.0
process_name=redash_celery_scheduled
user=xxx
numprocs=1
autostart=true
autorestart=true
redirect_stderr = true
stdout_logfile_maxbytes = 20MB
stdout_logfile_backups = 20
stdout_logfile = /home/xxx/redash-7.0.0/logs/redash_celery_scheduled_stdout.log
启动redash
supervisord -c /etc/supervisord.conf
遇到的问题
  • redash启动调试服务 ./bin/run python manage.py run --host=0.0.0.0报错
[2019-07-22 16:06:08,618][PID:5136][DEBUG][redash.query_runner] TreasureData query runner enabled but not supported, not registering. Either disable or install missing dependencies.
Traceback (most recent call last):
  File "manage.py", line 6, in <module>
    from redash.cli import manager
  File "/home/xxx/redash-7.0.0/redash/__init__.py", line 80, in <module>
    import_query_runners(settings.QUERY_RUNNERS)
  File "/home/xxx/redash-7.0.0/redash/query_runner/__init__.py", line 271, in import_query_runners
    __import__(runner_import)
  File "/home/xxx/redash-7.0.0/redash/query_runner/sqlite.py", line 2, in <module>
    import sqlite3
  File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 28, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

解决方法:

https://stackoverflow.com/questions/1210664/no-module-named-sqlite3
(1)yum install -y sqlite-devel
(2)重新编译安装Python2.7
  • python sasl包安装失败
yum install -y cyrus-sasl-devel