基因家族扩张与收缩分析及物种进化树构建(上)

一、数据准备

首先,选取不同物种的Protein数据集:Arabidopsis_thaliana.fa;Citrus_grandis.fa;Dimocarpus_longan.fa;Durio_zibethinus.fa;Prunus_persica.fa; Vitis_vinifera.fa;Citrus_clementina.fa;Citrus_sinensis.fa;Diospyros_oleifera.fa;Malus_domestica.fa;Oryza_sativa.fa;Pyrus_communis.fa
然后进行数据处理,去冗余,只保留最长转录本,去除可变剪切:
python3 removeRedundantProteins.py -i input.fa -o output.fa
removeRedundantProteins.py

import sys,getopt
def usage():
    print('usage:python3 removeRedundantProteins.py -i <in_fasta> -o <out_fasta> <-h>')
    return
def removeRedundant(in_file,out_file):
    gene_dic = {}
    flag = ''
    with open (in_file) as in_fasta:
        for line in in_fasta:
            if '>' in line:
                line1 = line.strip('>\n')
                line2 = line1.split('.')
                li = line2[0]
                flag = li
                try:
                    gene_dic[li]
                except KeyError:
                    gene_dic[li] = [line]
                else:
                    gene_dic[li].append(line)
            else:
                gene_dic[flag][-1] += line
    with open (out_file,'w') as out_fasta:
        for k,v in gene_dic.items():
            if len(v) == 1:
                out_fasta.write(gene_dic[k][0])
            else:
                trans_max = ''
                for trans in gene_dic[k]:
                    a = len(list(trans))
                    b = len(list(trans_max))
                    if a > b:
                        trans_max = trans
                out_fasta.write(trans_max)
def main(argv):
    try:
        opts, args = getopt.getopt(argv,'hi:o:')
    except getopt.GetoptError:
        usage()
        sys.exit()
    for opt, arg in opts:
        if opt == '-h':
            usage()
            sys.exit()
        elif opt == '-i':
            in_fasta_name = arg
        elif opt == '-o':
            outfile_name = arg
    try:
        removeRedundant(in_fasta_name,outfile_name)
    except UnboundLocalError:
        usage()
    return
if __name__ == '__main__':
    main(sys.argv[1:])

将处理好的数据置于一个文件夹中“Dataset”

二、使用OrthoFinder查找直系同源基因

OrthoFinder这个软件,之前有一篇文章已经介绍过了,这里就不在赘述,这个软件安装十分友好,直接conda安装即可;
nohup orthofinder -f Dataset -M msa -S diamond -T iqtree -t 24 -a 24 2> orthofinder.log &
orthofinder参数详情:
-t 并行序列搜索线程数(默认= 16)
-a 并行分析线程数(默认值= 1)
-M 基因树推断方法。可选:dendroblast和msa(默认= dendroblast)
-S 序列搜索程序(默认= blast)选项:blast,mmseqs,,blast_gz,diamond(推荐使用diamond,比对速度很给力)
-A 多序列联配方式,需要添加参数-M msa时才有效;(默认= mafft)可选择:muscle,mafft
-T 建树方法,需要添加参数-M msa时才有效,(默认 = fasttree)可选:iqtree,raxml-ng,fasttree,raxml
-s <文件> 可指定特定的根物种树
-I 设定MCL的通胀参数(默认 = 1.5)
-x Info用于以othoXML格式输出结果
-p <dir>将临时pickle文件写入到<dir>
-l 只执行单向序列搜索
-n 名称以附加到结果目录
-h 打印帮助文本
如果只需要查找直系同源基因,只需接“-f” 参数即可;此步也可建树,采用默认的建树方法fasttree,为无根树。
nohup orthofinder -f Dataset &
如果添加-M msa -T iqtree设定制定参数,可按照设定的参数使用最大似然法构建有根的物种进化树,构建的树为STAG树。
nohup orthofinder -f Dataset -M msa -S diamond -T iqtree -t 24 -a 24 2> orthofinder.log &

关于构建系统进化树,有很多种做法,常见的有利用物种全部的蛋白序列,构建STAG物种树;也有使用单拷贝直系同源基因构建的物种进化树,关于这一点,OrthoFinder查找同源基因,可以输出直系单拷贝同源基因的序列结果,后续也可使用其他构树软件及算法进行进化树构建。关于建树方法,则有距离矩阵法、最大简约法、最大似然法以及贝叶斯;当然目前主流采用的基本为最大似然法和贝叶斯,其中贝叶斯算法计算量巨大,耗时最久,其构建的树也认为最为“逼真”,但文章中使用较多的还是最大似然法,其耗时也需蛮久。

三、OrthoFinder输出结果解读

OrthoFinder输出的结果会在OrthoFinder文件夹下面的以日期命名的文件夹中,如:~/OrthoFinder/Results_May08

orthofinder输出结果

其中Orthogroups文件夹下面为查找的同源基因分组结果:
Orthogroups.GeneCount.tsv:每一行为直系同源基因组对应的基因数目;
Orthogroups_SingleCopyOrthologues.txt
Orthogroups.tsv:每一行为直系同源基因组对应的基因;
Orthogroups.txt:类似Orthogroups.tsv,输出格式为OrthoMCL;
Orthogroups_UnassignedGenes.tsv:物种特异性基因
Single_Copy_Orthologue_Sequences里为单拷贝直系同源基因
Comparative_Genomics_Statistics的相关结果文件:
Orthogroups_SpeciesOverlaps.csv:不同物种间的同源基因的交集;
SingleCopyOrthogroups.txt:单拷贝基因组的编号;
Statistics_Overall.csv:总体统计;
Statistics_PerSpecies.csv:分物种统计;
Gene_Trees: 每个直系同源基因基因组里的基因树;
SpeciesTree_rooted.txt: 从所有包含STAG支持的直系同源组推断的STAG物种树;
SpeciesTree_rooted_node_labels.txt: 同上,只不过多了一个标签信息,用于解释基因重复数据;

其中,我们可以用Orthogroups.GeneCount.tsv来作为CAFE的输入文件,分析基因家族的扩张与收缩;使用SpeciesTree_rooted.txt作为推断的物种树,并使用r8s,从中提取超度量树(ultrametric tree)即时间树;

四、用推断的物种树提取超度量树(ultrametric tree)

1、软件准备(r8s)

###Install:
wget http://ceiba.biosci.arizona.edu/r8s/r8s1.81.tar.gz 
tar -xzvf  r8s1.81.tar.gz 
cd r8s1.81/src
make -f Makefile.linux

2、准备r8s输入文件r8s_ctl_file.txt

python cafetutorial_prep_r8s.py -i SpeciesTree_rooted.txt -o r8s_ctl_file.txt -s 6650255 -p 'Oryza_sativa,Arabidopsis_thaliana' -c '152'
参数:
-i path_tree_file: path to .txt file containing tree in NEWICK format
-s n_sites: number of sites in alignment that was used to infer species tree
-p list_of_spp_tuples: list of tuples (each tuple being two species IDs whose mrca's age we are constraining; e.g., [('ENSG00','ENSPTR'),('ENSFCA','ENSECA')]
-c list_of_spp_cal_points: list of flats, one for each tuple in list_of_spp_tuples (e.g., [6.4,80])
-s 即用于推断物种树的比对序列碱基数目;
-p 已知物种树中的一对物种;
-c 已知一对物种的分化年限:
可在timetree网站查询:为152 mya

Pairwise Divergence Time For Oryza sativa & Arabidopsis thaliana

cafetutorial_prep_r8s.py

__author__ = "Fabio H. K. Mendes"
import os
import argparse
def r8s_prep(path_tree_file, sites_n, list_of_spp_tuples, list_of_spp_cal_points, path_output_file):
    tree_str = str()
    with open(path_tree_file, "r") as tree_file:
        tree_str = tree_file.readline().rstrip()
    fixage_lines = list()
    with open(path_output_file, "w") as output_file:
        output_file.write(
            "#NEXUS\nbegin trees;\ntree nj_tree = [&R] " + tree_str + "\nEnd;\nbegin rates;\nblformat nsites=" + \
            sites_n + " lengths=persite ultrametric=no;\ncollapse;\n")
        for idx, pair in enumerate(list_of_spp_tuples):
            node_name = pair[0][-3:] + pair[1][-3:]
            output_file.write(
                "mrca " + node_name + " " + pair[0] + " " + pair[1] + ";\n")
            fixage_lines.append("fixage taxon=" + node_name + " age=" + list_of_cal_points[idx] + ";\n")
        for line in fixage_lines:
            output_file.write(line)
        output_file.write(
            "divtime method=pl algorithm=tn cvStart=0 cvInc=0.5 cvNum=8 crossv=yes;\ndescribe plot=chronogram;\ndescribe plot=tree_description;\nend;"
            )
if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__, prog="mcl2rawcafe.py")
    parser.add_argument("-i", "--input-file", action="store", dest="input_file", required=True, type=str, help="full path to .txt file containing tree in NEWICK format")
    parser.add_argument("-o", "--output-file", action="store", dest="output_file", required=True, type=str, help="full path to file to be written (r8s input file)")
    parser.add_argument("-s", "--sites-n", action="store", dest="sites_n", required=True, type=str, help="number of sites in alignment used to estimate species tree")
    parser.add_argument("-p", "--pairs-species", action="store", dest="spp_pairs", required=True, type=str, help="")
    parser.add_argument("-c", "--calibration-points", action="store", dest="cal_points", required=True, type=str, help="")
    args = parser.parse_args()
    list_of_spp_tuples = list()
    for pair in args.spp_pairs.split(" "):
        list_of_spp_tuples.append(tuple(pair.split(",")))
    list_of_cal_points = args.cal_points.split(",")
    print "\nRunning cafetutorial_clade_and_size_filter.py as a standalone...\n"
    r8s_prep(args.input_file, args.sites_n, list_of_spp_tuples, list_of_cal_points, args.output_file)

3、运行r8s并提取超度量树

/home/Tools/r8s1.81/src/r8s -b -f r8s_ctl_file.txt > r8s_tmp.txt
#提取超度量树
$  tail -n 1 r8s_tmp.txt | cut -c 16- > twelve_spp_r8s_ultrametric.txt

五、基因家族扩张与收缩分析

1、软件安装(CAFE)

conda install cafe

2、cafe输入数据处理

awk -F"\t" '{print "Desc""\t"$1"\t"$2"\t"$3"\t"$4"\t"$5"\t"$6"\t"$7"\t"$8"\t"$9"\t"$10"\t"$11"\t"$12"\t"$13}' Orthogroups.GeneCount.tsv > cafe-input.data 
sed -e '1 s/(null)/Desc/' -e '1 s/Orthogroup/Family/' -i cafe-input.data
python cafetutorial_clade_and_size_filter.py -i cafe-input.data -o filtered_cafe_input.txt -s 2> filtered.log 

cafetutorial_clade_and_size_filter.py

__author__ = "Fabio H. K. Mendes"
import os
import argparse
def clade_filter(mcl_dump, clade_str):
    lines_to_keep_list = list()
    spp_idx_dict = dict()
    clades_list = list()
    if clade_str: # if clade filter was specified
        clades_list = clade_str.split(" ")
    with open(mcl_dump, "r") as input_file:
        for line_n, line in enumerate(input_file):
            line = line.rstrip()
            tokens = line.split("\t")
            spp_info = tokens[2:]
            if line.startswith("Desc"):
                spp_idx_dict = dict((sp, idx) for idx,sp in enumerate(spp_info))
                continue
            if clades_list:                
                clades_ok_list = list()
                for clade in clades_list:
                    spp_list = clade.split(",")
                    clade_count = sum(1 for sp in spp_list if int(spp_info[spp_idx_dict[sp]]) >= 1)
                    if clade_count >= 2:
                        clades_ok_list.append(1)
                if sum(clades_ok_list) == len(clades_list):
                    lines_to_keep_list.append(line_n)               
            # just keeping lines where >=2 species (among all of them) have gene copies
            clade_count = sum(1 for sp_count in spp_info if int(sp_count) >= 1)
            if clade_count >= 2:
                lines_to_keep_list.append(line_n)              
    return set(lines_to_keep_list)
def size_filter(mcl_dump, lines_to_keep_set):
    lines_to_remove_set = set()
    size_cutoff = 100
    fam_size = int()
    with open(mcl_dump, "r") as input_file:
        for line_n, line in enumerate(input_file):
            line = line.rstrip()
            if line.startswith("Desc"):
                continue
            elif line_n not in lines_to_keep_set and len(lines_to_keep_set) > 0:
                continue
            tokens = line.split("\t")
            spp_info = tokens[2:]
            for gene_count in spp_info:
                if int(gene_count) >= size_cutoff:
                    lines_to_separate_set.add(line_n)                 
    lines_to_keep_set -= lines_to_separate_set
    return lines_to_keep_set, lines_to_separate_set
def filter_print(mcl_dump, lines_to_keep_set, lines_to_separate_set, output_file_name):
    if len(lines_to_keep_set) == 0 and len(lines_to_separate_set) == 0:
        exit("No filtering was done! Exiting...\n")
    with open(output_file_name, "w") as output_file:
        with open("large_"+output_file_name, "w") as output_file2:
            with open(mcl_dump, "r") as input_file:
                for line_n, line in enumerate(input_file):
                    line = line.rstrip() + "\n"
                    if line_n == 0:
                        output_file.write(line)
                        output_file2.write(line) 
                    elif line_n in lines_to_keep_set and len(lines_to_keep_set) >= 1:
                        output_file.write(line)
                    elif line_n not in lines_to_separate_set and len(lines_to_keep_set) == 0:
                        output_file.write(line)
                    # has to be if, not elif
                    if line_n in lines_to_separate_set:
                        output_file2.write(line)
        # cleaning up in case size filtering was not done
        if len(lines_to_separate_set) == 0:
            os.unlink("large_"+output_file_name)
if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__, prog="cafetutorial_clade_and_size_filter.py")
    parser.add_argument("-i", "--input-file", action="store", dest="input_file", required=True, type=str, help="full path to mcl's output dump file")
    parser.add_argument("-o", "--output-file", action="store", dest="output_file", required=True, type=str, help="full path to file to be written")
    parser.add_argument("-cl", "--clade-filter", action="store", dest="clade_str", default=None, required=False, type=str, help="list of clades (separated by white spaces) comprised of species identifiers (separated by comma) that must have at least two species with gene copies for a given gene family")
    parser.add_argument("-s", "--size-filter", action="store_true", dest="size_filter", required=False, help="option to perform size filtering")
    args = parser.parse_args()
    lines_to_keep_set, lines_to_separate_set = set(), set()
    # applying size filter (if no groups are specified, just the lines where just 1 species has gene counts are removed
    lines_to_keep_set = clade_filter(args.input_file, args.clade_str)
    if args.size_filter:
        lines_to_keep_set, lines_to_separate = size_filter(args.input_file, lines_to_keep_set)
    filter_print(args.input_file, lines_to_keep_set, lines_to_separate_set, args.output_file) # .add(0) to get header back

3、配置cafe文件

vim cafetutorial_run.sh

cafetutorial_run.sh

tree即为r8s提取的超度量树;

4、运行cafe

mkdir -p reports
cafe cafetutorial_run.sh

5、结果统计

python cafetutorial_report_analysis.py -i reports/report_run.cafe -o reports/summary_run
summary_run_node.txt:统计每个节点中扩张,收缩的基因家族数目;
summary_run_fams.txt:具体发生变化的基因家族

六、CAFE_fig可视化

python3 /home/Tools/CAFE_fig/CAFE_fig.py resultfile.cafe -pb 0.05 -pf 0.05 --dump test/ -g svg --count_all_expansions
输出svg格式的文件,可导入AI编辑美化;

CAFE_fig运行报错:(module 'ete3' has no attribute 'TreeStyle')
报错解决:
vim /home/Tools/CAFE_fig/CAFE_fig.py

CAFE_fig.py

修改脚本,添加from PyQt5 import QtGui

程序还在运行,后续贴出结果图。


参考

OrthoFinder
timetree
http://www.chenlianfu.com/?tag=genomecomparison
https://www.jianshu.com/p/146093c91e2b
r8s
【OrthoFinder】
Emms, D.M., Kelly, S. OrthoFinder: solving fundamental biases in whole genome comparisons dramatically improves orthogroup inference accuracy. Genome Biol 16, 157 (2015) (https://doi.org/10.1186/s13059-015-0721-2)
Emms, D.M., Kelly, S. OrthoFinder: phylogenetic orthology inference for comparative genomics. Genome Biol 20, 238 (2019) https://doi.org/10.1186/s13059-019-1832-y)
【CAFE v.4.2.1】
Han, M. V., Thomas, G. W. C., Lugo-Martinez, J., and Hahn, M. W. Estimating gene gain and loss rates in the presence of error in genome assembly and annotation using CAFE 3. Molecular Biology and Evolution 30, 8 (2013)
【iqtree v. 1.6.12】
Lam-Tung Nguyen, Heiko A. Schmidt, Arndt von Haeseler, and Bui Quang Minh (2015) IQ-TREE: A fast and effective stochastic algorithm for estimating maximum likelihood phylogenies. Mol Biol Evol, 32:268-274. https://doi.org/10.1093/molbev/msu300
【modelFinder】
Subha Kalyaanamoorthy, Bui Quang Minh, Thomas KF Wong, Arndt von Haeseler, and Lars S Jermiin (2017) ModelFinder: Fast model selection for accurate phylogenetic estimates. Nature Methods, 14:587–589. https://doi.org/10.1038/nmeth.4285
【R8s v. 1.81】
Sanderson M J. R8s: inferring absolute rates of molecular evolution and divergence times in the absence of a molecular clock. Bioinformatics, 2003, 19(2): 301-302.
【STAG tree】
Emms D.M. & Kelly S. STAG: Species Tree Inference from All Genes (2018), bioRxiv https://doi.org/10.1101/267914


补充材料

直系同源低拷贝核基因(orthologous low-copy nuclear genes, LCN):在进化过程中,新基因通常来自事先存在的基因,新基因的功能从先前基因的功能进化而来。新基因的原材料来自基因组区域的重复,这种重复可包括一个或多个基因。作为物种形成的伴随事件而被重复,并继续保持相同功能的基因,称为直系同源基因(orthologous)。新的基因功能可由在单个物种的基因组中发生的重复引起的。在一个基因组内部的重复导致旁系同源基因(paralogous gene)。
最大似然法(maximum likelihood method):使用概率模型,寻找能够以较高概率产生观察数据的系统发生树。
外群的选择:大多数的种系发生重建方法会产生无根树,但是观察树的拓扑结构无法识别树根应在哪一分支上。实际上,对于要证实哪一个分类单元的分支先于其他的分类单元,树根必须确定。在无根树中设定一个根,最简单的方法是在数据集中增加一个外群(outgroup)。外群是一种分类操作单元,且有外部信息表明外群在所有分类群之前就已分化。研究演化历史,一般选择比目标序列具有较早进化历史的序列作为外类群。
Bootstrap support: bootstrap是统计学上一种非参数统计方法,通过有放回的随机抽样,构建分类回归树。Jackknife与bootstrap类似,只是每次抽样时会去除几个样本,像小刀一样切去一部分。所谓bootstrap法就是从整个序列的碱基(氨基酸)中任意选取一半,剩下的一半序列随机补齐组成一个新的序列。这样,一个序列就可以变成许多序列,一个序列组也就可以变成许多个序列组。根据某种算法(距离矩阵法、最大简约法、最大似然法),每个多序列组都可以生成一个进化树。将生成的许多进化树进行比较,按照多数规则(majority-rule)就会得到一个最“逼真”的进化树。