{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Example : k times repetition with the list of k input files \n", "\n", "DeepBiome package takes microbiome abundance data as input and uses the phylogenetic taxonomy to guide the decision of the optimal number of layers and neurons in the deep learning architecture.\n", "\n", "To use DeepBiome, you can experiment (1) __k times repetition__ or (2) __k fold cross-validation__.\n", "For each experiment, we asuume that the dataset is given by\n", "- __A list of k input files for k times repetition.__\n", "- __One input file for k fold cross-validation.__\n", "\n", "This notebook contains an example of (1) __k times repetition__ for the deep neural netowrk using deepbiome." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Load library\n", "\n", "First, we load the DeepBiome package. The DeepBiome package is built on the tensorflow and keras library" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] } ], "source": [ "import os\n", "import logging\n", "import json\n", "from pkg_resources import resource_filename\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "pd.set_option('display.float_format', lambda x: '%.03f' % x)\n", "np.set_printoptions(formatter={'float_kind':lambda x: '%.03f' % x})\n", "\n", "from deepbiome import deepbiome" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Prepare the dataset\n", "\n", "In this example, we assume that we have __a list of k input files for k times repetition.__\n", "\n", "DeepBiome needs 4 data files as follows:\n", "1. __the tree information__\n", "1. __the lists of the input files__ (each file has all sample's information for one repetition)\n", "1. __the list of the names of input files__ \n", "1. __y__\n", "\n", "In addition, we can set __the training index for each repetition__. If we set the index file, DeepBiome builds the training set for each repetition based on each column of the index file. If not, DeepBiome will generate the index file locally.\n", "\n", " \n", "Each data should have the csv format as follow:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of the tree information\n", "\n", "First we need a file about the phylogenetic tree information. This tree information file should have the format below:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
GenusFamilyOrderClassPhylum
0StreptococcusStreptococcaceaeLactobacillalesBacilliFirmicutes
1TropherymaCellulomonadaceaeActinomycetalesActinobacteriaActinobacteria
2VeillonellaVeillonellaceaeSelenomonadalesNegativicutesFirmicutes
3ActinomycesActinomycetaceaeActinomycetalesActinobacteriaActinobacteria
4FlavobacteriumFlavobacteriaceaeFlavobacterialesFlavobacteriaBacteroidetes
5PrevotellaPrevotellaceaeBacteroidalesBacteroidiaBacteroidetes
6PorphyromonasPorphyromonadaceaeBacteroidalesBacteroidiaBacteroidetes
7ParvimonasClostridiales_Incertae_Sedis_XIClostridialesClostridiaFirmicutes
8FusobacteriumFusobacteriaceaeFusobacterialesFusobacteriaFusobacteria
9PropionibacteriumPropionibacteriaceaeActinomycetalesActinobacteriaActinobacteria
10GemellaBacillales_Incertae_Sedis_XIBacillalesBacilliFirmicutes
11RothiaMicrococcaceaeActinomycetalesActinobacteriaActinobacteria
12GranulicatellaCarnobacteriaceaeLactobacillalesBacilliFirmicutes
13NeisseriaNeisseriaceaeNeisserialesBetaproteobacteriaProteobacteria
14LactobacillusLactobacillaceaeLactobacillalesBacilliFirmicutes
15MegasphaeraVeillonellaceaeSelenomonadalesNegativicutesFirmicutes
16CatonellaLachnospiraceaeClostridialesClostridiaFirmicutes
17AtopobiumCoriobacteriaceaeCoriobacterialesActinobacteriaActinobacteria
18CampylobacterCampylobacteraceaeCampylobacteralesEpsilonproteobacteriaProteobacteria
19CapnocytophagaFlavobacteriaceaeFlavobacterialesFlavobacteriaBacteroidetes
20SolobacteriumErysipelotrichaceaeErysipelotrichalesErysipelotrichiaFirmicutes
21MoryellaLachnospiraceaeClostridialesClostridiaFirmicutes
22TM7_genera_incertae_sedisTM7_genera_incertae_sedisTM7_genera_incertae_sedisTM7_genera_incertae_sedisTM7
23StaphylococcusStaphylococcaceaeBacillalesBacilliFirmicutes
24FilifactorPeptostreptococcaceaeClostridialesClostridiaFirmicutes
25OribacteriumLachnospiraceaeClostridialesClostridiaFirmicutes
26BurkholderiaBurkholderiaceaeBurkholderialesBetaproteobacteriaProteobacteria
27SneathiaLeptotrichiaceaeFusobacterialesFusobacteriaFusobacteria
28TreponemaSpirochaetaceaeSpirochaetalesSpirochaetesSpirochaetes
29MoraxellaMoraxellaceaePseudomonadalesGammaproteobacteriaProteobacteria
30HaemophilusPasteurellaceaePasteurellalesGammaproteobacteriaProteobacteria
31SelenomonasVeillonellaceaeSelenomonadalesNegativicutesFirmicutes
32CorynebacteriumCorynebacteriaceaeActinomycetalesActinobacteriaActinobacteria
33RhizobiumRhizobiaceaeRhizobialesAlphaproteobacteriaProteobacteria
34BradyrhizobiumBradyrhizobiaceaeRhizobialesAlphaproteobacteriaProteobacteria
35MethylobacteriumMethylobacteriaceaeRhizobialesAlphaproteobacteriaProteobacteria
36OD1_genera_incertae_sedisOD1_genera_incertae_sedisOD1_genera_incertae_sedisOD1_genera_incertae_sedisOD1
37FinegoldiaClostridiales_Incertae_Sedis_XIClostridialesClostridiaFirmicutes
38MicrobacteriumMicrobacteriaceaeActinomycetalesActinobacteriaActinobacteria
39SphingomonasSphingomonadaceaeSphingomonadalesAlphaproteobacteriaProteobacteria
40ChryseobacteriumFlavobacteriaceaeFlavobacterialesFlavobacteriaBacteroidetes
41BacteroidesBacteroidaceaeBacteroidalesBacteroidiaBacteroidetes
42BdellovibrioBdellovibrionaceaeBdellovibrionalesDeltaproteobacteriaProteobacteria
43StreptophytaChloroplastChloroplastChloroplastCyanobacteria_Chloroplast
44Lachnospiracea_incertae_sedisLachnospiraceaeClostridialesClostridiaFirmicutes
45ParacoccusRhodobacteraceaeRhodobacteralesAlphaproteobacteriaProteobacteria
46FastidiosipilaRuminococcaceaeClostridialesClostridiaFirmicutes
47PseudonocardiaPseudonocardiaceaeActinomycetalesActinobacteriaActinobacteria
\n", "
" ], "text/plain": [ " Genus Family \\\n", "0 Streptococcus Streptococcaceae \n", "1 Tropheryma Cellulomonadaceae \n", "2 Veillonella Veillonellaceae \n", "3 Actinomyces Actinomycetaceae \n", "4 Flavobacterium Flavobacteriaceae \n", "5 Prevotella Prevotellaceae \n", "6 Porphyromonas Porphyromonadaceae \n", "7 Parvimonas Clostridiales_Incertae_Sedis_XI \n", "8 Fusobacterium Fusobacteriaceae \n", "9 Propionibacterium Propionibacteriaceae \n", "10 Gemella Bacillales_Incertae_Sedis_XI \n", "11 Rothia Micrococcaceae \n", "12 Granulicatella Carnobacteriaceae \n", "13 Neisseria Neisseriaceae \n", "14 Lactobacillus Lactobacillaceae \n", "15 Megasphaera Veillonellaceae \n", "16 Catonella Lachnospiraceae \n", "17 Atopobium Coriobacteriaceae \n", "18 Campylobacter Campylobacteraceae \n", "19 Capnocytophaga Flavobacteriaceae \n", "20 Solobacterium Erysipelotrichaceae \n", "21 Moryella Lachnospiraceae \n", "22 TM7_genera_incertae_sedis TM7_genera_incertae_sedis \n", "23 Staphylococcus Staphylococcaceae \n", "24 Filifactor Peptostreptococcaceae \n", "25 Oribacterium Lachnospiraceae \n", "26 Burkholderia Burkholderiaceae \n", "27 Sneathia Leptotrichiaceae \n", "28 Treponema Spirochaetaceae \n", "29 Moraxella Moraxellaceae \n", "30 Haemophilus Pasteurellaceae \n", "31 Selenomonas Veillonellaceae \n", "32 Corynebacterium Corynebacteriaceae \n", "33 Rhizobium Rhizobiaceae \n", "34 Bradyrhizobium Bradyrhizobiaceae \n", "35 Methylobacterium Methylobacteriaceae \n", "36 OD1_genera_incertae_sedis OD1_genera_incertae_sedis \n", "37 Finegoldia Clostridiales_Incertae_Sedis_XI \n", "38 Microbacterium Microbacteriaceae \n", "39 Sphingomonas Sphingomonadaceae \n", "40 Chryseobacterium Flavobacteriaceae \n", "41 Bacteroides Bacteroidaceae \n", "42 Bdellovibrio Bdellovibrionaceae \n", "43 Streptophyta Chloroplast \n", "44 Lachnospiracea_incertae_sedis Lachnospiraceae \n", "45 Paracoccus Rhodobacteraceae \n", "46 Fastidiosipila Ruminococcaceae \n", "47 Pseudonocardia Pseudonocardiaceae \n", "\n", " Order Class \\\n", "0 Lactobacillales Bacilli \n", "1 Actinomycetales Actinobacteria \n", "2 Selenomonadales Negativicutes \n", "3 Actinomycetales Actinobacteria \n", "4 Flavobacteriales Flavobacteria \n", "5 Bacteroidales Bacteroidia \n", "6 Bacteroidales Bacteroidia \n", "7 Clostridiales Clostridia \n", "8 Fusobacteriales Fusobacteria \n", "9 Actinomycetales Actinobacteria \n", "10 Bacillales Bacilli \n", "11 Actinomycetales Actinobacteria \n", "12 Lactobacillales Bacilli \n", "13 Neisseriales Betaproteobacteria \n", "14 Lactobacillales Bacilli \n", "15 Selenomonadales Negativicutes \n", "16 Clostridiales Clostridia \n", "17 Coriobacteriales Actinobacteria \n", "18 Campylobacterales Epsilonproteobacteria \n", "19 Flavobacteriales Flavobacteria \n", "20 Erysipelotrichales Erysipelotrichia \n", "21 Clostridiales Clostridia \n", "22 TM7_genera_incertae_sedis TM7_genera_incertae_sedis \n", "23 Bacillales Bacilli \n", "24 Clostridiales Clostridia \n", "25 Clostridiales Clostridia \n", "26 Burkholderiales Betaproteobacteria \n", "27 Fusobacteriales Fusobacteria \n", "28 Spirochaetales Spirochaetes \n", "29 Pseudomonadales Gammaproteobacteria \n", "30 Pasteurellales Gammaproteobacteria \n", "31 Selenomonadales Negativicutes \n", "32 Actinomycetales Actinobacteria \n", "33 Rhizobiales Alphaproteobacteria \n", "34 Rhizobiales Alphaproteobacteria \n", "35 Rhizobiales Alphaproteobacteria \n", "36 OD1_genera_incertae_sedis OD1_genera_incertae_sedis \n", "37 Clostridiales Clostridia \n", "38 Actinomycetales Actinobacteria \n", "39 Sphingomonadales Alphaproteobacteria \n", "40 Flavobacteriales Flavobacteria \n", "41 Bacteroidales Bacteroidia \n", "42 Bdellovibrionales Deltaproteobacteria \n", "43 Chloroplast Chloroplast \n", "44 Clostridiales Clostridia \n", "45 Rhodobacterales Alphaproteobacteria \n", "46 Clostridiales Clostridia \n", "47 Actinomycetales Actinobacteria \n", "\n", " Phylum \n", "0 Firmicutes \n", "1 Actinobacteria \n", "2 Firmicutes \n", "3 Actinobacteria \n", "4 Bacteroidetes \n", "5 Bacteroidetes \n", "6 Bacteroidetes \n", "7 Firmicutes \n", "8 Fusobacteria \n", "9 Actinobacteria \n", "10 Firmicutes \n", "11 Actinobacteria \n", "12 Firmicutes \n", "13 Proteobacteria \n", "14 Firmicutes \n", "15 Firmicutes \n", "16 Firmicutes \n", "17 Actinobacteria \n", "18 Proteobacteria \n", "19 Bacteroidetes \n", "20 Firmicutes \n", "21 Firmicutes \n", "22 TM7 \n", "23 Firmicutes \n", "24 Firmicutes \n", "25 Firmicutes \n", "26 Proteobacteria \n", "27 Fusobacteria \n", "28 Spirochaetes \n", "29 Proteobacteria \n", "30 Proteobacteria \n", "31 Firmicutes \n", "32 Actinobacteria \n", "33 Proteobacteria \n", "34 Proteobacteria \n", "35 Proteobacteria \n", "36 OD1 \n", "37 Firmicutes \n", "38 Actinobacteria \n", "39 Proteobacteria \n", "40 Bacteroidetes \n", "41 Bacteroidetes \n", "42 Proteobacteria \n", "43 Cyanobacteria_Chloroplast \n", "44 Firmicutes \n", "45 Proteobacteria \n", "46 Firmicutes \n", "47 Actinobacteria " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tree_information = pd.read_csv(resource_filename('deepbiome', 'tests/data/genus48_dic.csv'))\n", "tree_information" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This file has `.csv` format below:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Genus,Family,Order,Class,Phylum\n", "Streptococcus,Streptococcaceae,Lactobacillales,Bacilli,Firmicutes\n", "Tropheryma,Cellulomonadaceae,Actinomycetales,Actinobacteria,Actinobacteria\n", "Veillonella,Veillonellaceae,Selenomonadales,Negativicutes,Firmicutes\n", "Actinomyces,Actinomycetaceae,Actinomycetales,Actinobacteria,Actinobacteria\n", "Flavobacterium,Flavobacteriaceae,Flavobacteriales,Flavobacteria,Bacteroidetes\n", "Prevotella,Prevotellaceae,Bacteroidales,Bacteroidia,Bacteroidetes\n", "Porphyromonas,Porphyromonadaceae,Bacteroidales,Bacteroidia,Bacteroidetes\n", "Parvimonas,Clostridiales_Incertae_Sedis_XI,Clostridiales,Clostridia,Firmicutes\n", "Fusobacterium,Fusobacteriaceae,Fusobacteriales,Fusobacteria,Fusobacteria\n", "Propionibacterium,Propionibacteriaceae,Actinomycetales,Actinobacteria,Actinobacteria\n", "Gemella,Bacillales_Incertae_Sedis_XI,Bacillales,Bacilli,Firmicutes\n", "Rothia,Micrococcaceae,Actinomycetales,Actinobacteria,Actinobacteria\n", "Granulicatella,Carnobacteriaceae,Lactobacillales,Bacilli,Firmicutes\n", "Neisseria,Neisseriaceae,Neisseriales,Betaproteobacteria,Proteobacteria\n", "Lactobacillus,Lactobacillaceae,Lactobacillales,Bacilli,Firmicutes\n", "Megasphaera,Veillonellaceae,Selenomonadales,Negativicutes,Firmicutes\n", "Catonella,Lachnospiraceae,Clostridiales,Clostridia,Firmicutes\n", "Atopobium,Coriobacteriaceae,Coriobacteriales,Actinobacteria,Actinobacteria\n", "Campylobacter,Campylobacteraceae,Campylobacterales,Epsilonproteobacteria,Proteobacteria\n", "Capnocytophaga,Flavobacteriaceae,Flavobacteriales,Flavobacteria,Bacteroidetes\n", "Solobacterium,Erysipelotrichaceae,Erysipelotrichales,Erysipelotrichia,Firmicutes\n", "Moryella,Lachnospiraceae,Clostridiales,Clostridia,Firmicutes\n", "TM7_genera_incertae_sedis,TM7_genera_incertae_sedis,TM7_genera_incertae_sedis,TM7_genera_incertae_sedis,TM7\n", "Staphylococcus,Staphylococcaceae,Bacillales,Bacilli,Firmicutes\n", "Filifactor,Peptostreptococcaceae,Clostridiales,Clostridia,Firmicutes\n", "Oribacterium,Lachnospiraceae,Clostridiales,Clostridia,Firmicutes\n", "Burkholderia,Burkholderiaceae,Burkholderiales,Betaproteobacteria,Proteobacteria\n", "Sneathia,Leptotrichiaceae,Fusobacteriales,Fusobacteria,Fusobacteria\n", "Treponema,Spirochaetaceae,Spirochaetales,Spirochaetes,Spirochaetes\n", "Moraxella,Moraxellaceae,Pseudomonadales,Gammaproteobacteria,Proteobacteria\n", "Haemophilus,Pasteurellaceae,Pasteurellales,Gammaproteobacteria,Proteobacteria\n", "Selenomonas,Veillonellaceae,Selenomonadales,Negativicutes,Firmicutes\n", "Corynebacterium,Corynebacteriaceae,Actinomycetales,Actinobacteria,Actinobacteria\n", "Rhizobium,Rhizobiaceae,Rhizobiales,Alphaproteobacteria,Proteobacteria\n", "Bradyrhizobium,Bradyrhizobiaceae,Rhizobiales,Alphaproteobacteria,Proteobacteria\n", "Methylobacterium,Methylobacteriaceae,Rhizobiales,Alphaproteobacteria,Proteobacteria\n", "OD1_genera_incertae_sedis,OD1_genera_incertae_sedis,OD1_genera_incertae_sedis,OD1_genera_incertae_sedis,OD1\n", "Finegoldia,Clostridiales_Incertae_Sedis_XI,Clostridiales,Clostridia,Firmicutes\n", "Microbacterium,Microbacteriaceae,Actinomycetales,Actinobacteria,Actinobacteria\n", "Sphingomonas,Sphingomonadaceae,Sphingomonadales,Alphaproteobacteria,Proteobacteria\n", "Chryseobacterium,Flavobacteriaceae,Flavobacteriales,Flavobacteria,Bacteroidetes\n", "Bacteroides,Bacteroidaceae,Bacteroidales,Bacteroidia,Bacteroidetes\n", "Bdellovibrio,Bdellovibrionaceae,Bdellovibrionales,Deltaproteobacteria,Proteobacteria\n", "Streptophyta,Chloroplast,Chloroplast,Chloroplast,Cyanobacteria_Chloroplast\n", "Lachnospiracea_incertae_sedis,Lachnospiraceae,Clostridiales,Clostridia,Firmicutes\n", "Paracoccus,Rhodobacteraceae,Rhodobacterales,Alphaproteobacteria,Proteobacteria\n", "Fastidiosipila,Ruminococcaceae,Clostridiales,Clostridia,Firmicutes\n", "Pseudonocardia,Pseudonocardiaceae,Actinomycetales,Actinobacteria,Actinobacteria\n", "\n" ] } ], "source": [ "with open(resource_filename('deepbiome', 'tests/data/genus48_dic.csv')) as f:\n", " print(f.read())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of the list of the name of input files\n", "\n", "In this example. we assume that input is given by the lists of files. Each file has all sample's information for one repeatition.\n", "If we want to use the list of the input files, we need to make a list of the names of each input file. Below is an example file for `k=3` repetition." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0
0gcount_0001.csv
1gcount_0002.csv
2gcount_0003.csv
\n", "
" ], "text/plain": [ " 0\n", "0 gcount_0001.csv\n", "1 gcount_0002.csv\n", "2 gcount_0003.csv" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "list_of_input_files = pd.read_csv(resource_filename('deepbiome', 'tests/data/gcount_list.csv'))\n", "list_of_input_files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of the lists of the input files\n", "\n", "Below is an example of each input file. This example has 1000 samples as rows, and the abandunces of each microbiomes as columns. Below is an example file for `k=3` repetition. This example is `gcount_0001.csv` for the first repetition in the list of the names of input files above. This file has the 1000 samples' microbiome abandunce. __The order of the microbiome should be same as the order of the microbiome in the Genus level in the tree information above.__" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StreptococcusTropherymaVeillonellaActinomycesFlavobacteriumPrevotellaPorphyromonasParvimonasFusobacteriumPropionibacterium...MicrobacteriumSphingomonasChryseobacteriumBacteroidesBdellovibrioStreptophytaLachnospiracea_incertae_sedisParacoccusFastidiosipilaPseudonocardia
084108135055322403621165...08700000002133
114450157301278828569154...0120000003638
2125908056500108800740...02813900003445
398203275940960811990...1571046000003507
411620130969163151516741623...09000060003945
\n", "

5 rows × 48 columns

\n", "
" ], "text/plain": [ " Streptococcus Tropheryma Veillonella Actinomyces Flavobacterium \\\n", "0 841 0 813 505 5 \n", "1 1445 0 1 573 0 \n", "2 1259 0 805 650 0 \n", "3 982 0 327 594 0 \n", "4 1162 0 130 969 163 \n", "\n", " Prevotella Porphyromonas Parvimonas Fusobacterium Propionibacterium \\\n", "0 3224 0 362 11 65 \n", "1 1278 82 85 69 154 \n", "2 1088 0 0 74 0 \n", "3 960 81 19 9 0 \n", "4 1515 167 4 162 3 \n", "\n", " ... Microbacterium Sphingomonas Chryseobacterium Bacteroides \\\n", "0 ... 0 87 0 0 \n", "1 ... 0 1 2 0 \n", "2 ... 0 2 8 1 \n", "3 ... 157 1 0 4 \n", "4 ... 0 9 0 0 \n", "\n", " Bdellovibrio Streptophyta Lachnospiracea_incertae_sedis Paracoccus \\\n", "0 0 0 0 0 \n", "1 0 0 0 0 \n", "2 39 0 0 0 \n", "3 60 0 0 0 \n", "4 0 0 60 0 \n", "\n", " Fastidiosipila Pseudonocardia \n", "0 0 2133 \n", "1 0 3638 \n", "2 0 3445 \n", "3 0 3507 \n", "4 0 3945 \n", "\n", "[5 rows x 48 columns]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x_1 = pd.read_csv(resource_filename('deepbiome', 'tests/data/count/%s' % list_of_input_files.iloc[0,0]))\n", "x_1.head()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StreptococcusTropherymaVeillonellaActinomycesFlavobacteriumPrevotellaPorphyromonasParvimonasFusobacteriumPropionibacterium...MicrobacteriumSphingomonasChryseobacteriumBacteroidesBdellovibrioStreptophytaLachnospiracea_incertae_sedisParacoccusFastidiosipilaPseudonocardia
995140143052609232501270...0070000004470
996265561067409527613158125...0200000002826
99733507125967718144167...024600600006527
99864969966122705082305500...0000060004402
999125800111902348250137176...0200000002585
\n", "

5 rows × 48 columns

\n", "
" ], "text/plain": [ " Streptococcus Tropheryma Veillonella Actinomyces Flavobacterium \\\n", "995 1401 4 30 526 0 \n", "996 2655 6 106 74 0 \n", "997 335 0 71 259 67 \n", "998 649 69 966 1227 0 \n", "999 1258 0 0 1119 0 \n", "\n", " Prevotella Porphyromonas Parvimonas Fusobacterium Propionibacterium \\\n", "995 923 25 0 127 0 \n", "996 952 76 13 158 125 \n", "997 718 1 4 4 167 \n", "998 508 2 30 550 0 \n", "999 2348 25 0 137 176 \n", "\n", " ... Microbacterium Sphingomonas Chryseobacterium Bacteroides \\\n", "995 ... 0 0 7 0 \n", "996 ... 0 2 0 0 \n", "997 ... 0 246 0 0 \n", "998 ... 0 0 0 0 \n", "999 ... 0 2 0 0 \n", "\n", " Bdellovibrio Streptophyta Lachnospiracea_incertae_sedis Paracoccus \\\n", "995 0 0 0 0 \n", "996 0 0 0 0 \n", "997 6 0 0 0 \n", "998 0 6 0 0 \n", "999 0 0 0 0 \n", "\n", " Fastidiosipila Pseudonocardia \n", "995 0 4470 \n", "996 0 2826 \n", "997 0 6527 \n", "998 0 4402 \n", "999 0 2585 \n", "\n", "[5 rows x 48 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x_1.tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This file has .csv format below:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\"Streptococcus\",\"Tropheryma\",\"Veillonella\",\"Actinomyces\",\"Flavobacterium\",\"Prevotella\",\"Porphyromonas\",\"Parvimonas\",\"Fusobacterium\",\"Propionibacterium\",\"Gemella\",\"Rothia\",\"Granulicatella\",\"Neisseria\",\"Lactobacillus\",\"Megasphaera\",\"Catonella\",\"Atopobium\",\"Campylobacter\",\"Capnocytophaga\",\"Solobacterium\",\"Moryella\",\"TM7_genera_incertae_sedis\",\"Staphylococcus\",\"Filifactor\",\"Oribacterium\",\"Burkholderia\",\"Sneathia\",\"Treponema\",\"Moraxella\",\"Haemophilus\",\"Selenomonas\",\"Corynebacterium\",\"Rhizobium\",\"Bradyrhizobium\",\"Methylobacterium\",\"OD1_genera_incertae_sedis\",\"Finegoldia\",\"Microbacterium\",\"Sphingomonas\",\"Chryseobacterium\",\"Bacteroides\",\"Bdellovibrio\",\"Streptophyta\",\"Lachnospiracea_incertae_sedis\",\"Paracoccus\",\"Fastidiosipila\",\"Pseudonocardia\"\n", "\n", "841,0,813,505,5,3224,0,362,11,65,156,1,55,0,1,20,382,1,333,24,80,43,309,2,3,4,0,1,32,0,2,4,382,0,0,96,23,0,0,87,0,0,0,0,0,0,0,2133\n", "\n", "1445,0,1,573,0,1278,82,85,69,154,436,3,0,61,440,0,394,83,33,123,0,49,414,0,0,37,0,0,42,0,0,384,27,0,0,0,146,0,0,1,2,0,0,0,0,0,0,3638\n", "\n", "1259,0,805,650,0,1088,0,0,74,0,155,228,430,765,0,0,11,102,68,90,77,83,322,10,0,7,0,122,76,0,1,25,0,0,0,44,13,0,0,2,8,1,39,0,0,0,0,3445\n", "\n", "982,0,327,594,0,960,81,19,9,0,45,457,1049,0,3,450,19,170,388,147,0,0,41,63,0,1,0,0,121,0,0,1,0,0,0,0,344,0,157,1,0,4,60,0,0,0,0,3507\n", "\n", "1162,0,130,969,163,1515,167,4,162,3,12,0,48,73,93,259,52,0,201,85,14,14,434,2,0,0,0,0,187,0,0,188,45,0,0,0,4,0,0,9,0,0,0,0,60,0,0,3945\n", "\n", "1956,37,41,661,47,1555,374,7,142,19,61,226,0,30,52,0,6,480,142,148,9,575,12,0,0,0,0,3,168,0,56,50,0,0,0,98,989,0,0,12,0,0,0,0,0,0,0,2044\n", "\n", "1037,14,83,1595,132,305,103,174,1195,0,410,224,1,320,26,0,476,0,7,37,46,61,20,0,0,0,0,0,226,0,239,8,1,0,0,0,0,188,0,20,4,0,4,0,0,0,0,3044\n", "\n", "641,0,172,179,0,1312,84,9,81,376,128,223,160,0,532,155,89,355,1,282,0,0,25,0,0,43,0,9,311,0,0,0,0,0,0,0,845,0,0,8,0,0,0,0,0,0,0,3980\n", "\n", "852,146,504,99,2,376,116,152,67,0,120,3,23,2,34,0,127,75,240,60,42,0,9,0,15,0,62,0,13,0,197,187,396,0,0,20,51,0,0,3,0,0,0,0,0,0,0,6007\n", "\n" ] } ], "source": [ "with open(resource_filename('deepbiome', 'tests/data/count/%s' % list_of_input_files.iloc[0,0])) as f:\n", " x_csv = f.readlines()\n", " _ = [print(l) for l in x_csv[:10]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of the Y (regression)\n", "\n", "This is an example of the output file for regression problem. One column contains outputs of the samples for one repetition. \n", "Below example file has 1000 samples in rows, `k=3` repetitions in columns." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
x1x2x3
04.9975.4925.474
15.0041.5004.640
25.4854.1875.491
35.4904.8631.500
41.5005.4815.490
\n", "
" ], "text/plain": [ " x1 x2 x3\n", "0 4.997 5.492 5.474\n", "1 5.004 1.500 4.640\n", "2 5.485 4.187 5.491\n", "3 5.490 4.863 1.500\n", "4 1.500 5.481 5.490" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y = pd.read_csv(resource_filename('deepbiome', 'tests/data/regression_y.csv'))\n", "y.head()" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
x1x2x3
9952.6105.4913.319
9965.4893.7405.489
9973.4984.2505.488
9985.4861.9175.415
9995.3205.4831.500
\n", "
" ], "text/plain": [ " x1 x2 x3\n", "995 2.610 5.491 3.319\n", "996 5.489 3.740 5.489\n", "997 3.498 4.250 5.488\n", "998 5.486 1.917 5.415\n", "999 5.320 5.483 1.500" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For one repetition, the DeepBiome will use the one column." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "0 4.997\n", "1 5.004\n", "2 5.485\n", "3 5.490\n", "4 1.500\n", "Name: x1, dtype: float64" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.iloc[:,0].head()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "995 2.610\n", "996 5.489\n", "997 3.498\n", "998 5.486\n", "999 5.320\n", "Name: x1, dtype: float64" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.iloc[:,0].tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Example of the Y (classification)\n", "\n", "This is an example of the output file for classification problem. Below example file has 1000 samples in rows, 3 repetitions in columns." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
V1V2V3
0101
1111
2010
3011
4110
\n", "
" ], "text/plain": [ " V1 V2 V3\n", "0 1 0 1\n", "1 1 1 1\n", "2 0 1 0\n", "3 0 1 1\n", "4 1 1 0" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y = pd.read_csv(resource_filename('deepbiome', 'tests/data/classification_y.csv'))\n", "y.head()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
V1V2V3
995101
996010
997110
998011
999111
\n", "
" ], "text/plain": [ " V1 V2 V3\n", "995 1 0 1\n", "996 0 1 0\n", "997 1 1 0\n", "998 0 1 1\n", "999 1 1 1" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For one repetition, the DeepBiome will use the one column." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "0 1\n", "1 1\n", "2 0\n", "3 0\n", "4 1\n", "Name: V1, dtype: int64" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.iloc[:,0].head()" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "995 1\n", "996 0\n", "997 1\n", "998 0\n", "999 1\n", "Name: V1, dtype: int64" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y.iloc[:,0].tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exmple of the training index file for repetition\n", "\n", "For each repetition, we have to set the training and test set. If the index file is given, DeepBiome sets the training set and test set based on the index file. Below is the example of the index file. Each column has the training indexs for each repetition. The DeepBiome will only use the samples in this index set for training." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
V1V2V3
049069062
1498968123
2389999335
351139843
459283204
\n", "
" ], "text/plain": [ " V1 V2 V3\n", "0 490 690 62\n", "1 498 968 123\n", "2 389 999 335\n", "3 51 139 843\n", "4 592 83 204" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idxs = pd.read_csv(resource_filename('deepbiome', 'tests/data/regression_idx.csv'), dtype=np.int)\n", "idxs.head()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
V1V2V3
745599824997
746720633821
74780268661
74857032750
749440589607
\n", "
" ], "text/plain": [ " V1 V2 V3\n", "745 599 824 997\n", "746 720 633 821\n", "747 80 268 661\n", "748 570 32 750\n", "749 440 589 607" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idxs.tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Below is the index set for 1st repetition. From 1000 samples above, it uses 750 samples for training." ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "0 490\n", "1 498\n", "2 389\n", "3 51\n", "4 592\n", "Name: V1, dtype: int64" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idxs.iloc[:,0].head()" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "745 599\n", "746 720\n", "747 80\n", "748 570\n", "749 440\n", "Name: V1, dtype: int64" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "idxs.iloc[:,0].tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Prepare the configuration\n", "\n", "For detailed configuration, we can build the configuration information for the network training by:\n", "1. the python dictionary format\n", "1. the configufation file (.cfg).\n", "\n", "In this notebook, we show the python dictionary format configuration.\n", "\n", "Please check the detailed information about each option in the [documantation](https://young-won.github.io/deepbiome/prerequisites.html#configuration)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### For preparing the configuration about the network information (`network_info`)\n", "\n", "To give the information about the training process, we provide a dictionary of configurations to the `netowrk_info` field.\n", "Your configuration for the network training should include the information about:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "network_info = {\n", " 'architecture_info': {\n", " 'batch_normalization': 'False',\n", " 'drop_out': '0',\n", " 'weight_initial': 'glorot_uniform',\n", " 'weight_l1_penalty':'0.',\n", " 'weight_decay': 'phylogenetic_tree',\n", " },\n", " 'model_info': {\n", " 'lr': '0.01',\n", " 'decay': '0.001',\n", " 'loss': 'binary_crossentropy',\n", " 'metrics': 'binary_accuracy, sensitivity, specificity, gmeasure, auc',\n", " 'taxa_selection_metrics': 'sensitivity, specificity, gmeasure, accuracy',\n", " 'network_class': 'DeepBiomeNetwork',\n", " 'optimizer': 'adam',\n", " 'reader_class': 'MicroBiomeClassificationReader',\n", " 'normalizer': 'normalize_minmax',\n", " },\n", " 'training_info': {\n", " 'batch_size': '50', \n", " 'epochs': '10',\n", " 'callbacks': 'ModelCheckpoint',\n", " 'monitor': 'val_loss',\n", " 'mode' : 'min',\n", " 'min_delta': '1e-7',\n", " },\n", " 'validation_info': {\n", " 'batch_size': 'None', \n", " 'validation_size': '0.2'\n", " },\n", " 'test_info': {\n", " 'batch_size': 'None'\n", " }\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### For preparing the configuration about the path information (`path_info`)\n", "\n", "To give the information about the path of dataset, paths for saving the trained weights and the evaluation results, we provide a dictionary of configurations to the `path_info` field.\n", "Your configuration for the network training should include the information about:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": true }, "outputs": [], "source": [ "path_info = {\n", " 'data_info': {\n", " 'count_list_path': resource_filename('deepbiome', 'tests/data/gcount_list.csv'),\n", " 'count_path': resource_filename('deepbiome', 'tests/data/count'),\n", " 'data_path': resource_filename('deepbiome', 'tests/data'),\n", " 'idx_path': resource_filename('deepbiome', 'tests/data/classification_idx.csv'),\n", " 'tree_info_path': resource_filename('deepbiome', 'tests/data/genus48_dic.csv'),\n", " 'x_path': '',\n", " 'y_path': 'classification_y.csv'\n", " },\n", " 'model_info': {\n", " 'evaluation': 'eval.npy',\n", " 'history': 'hist.json',\n", " 'model_dir': './example_result/',\n", " 'weight': 'weight.h5'\n", " }\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. DeepBiome Training\n", "\n", "Now we can train the DeepBiome network based on the configurations." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For logging, we use the python logging library." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": true }, "outputs": [], "source": [ "logging.basicConfig(format = '[%(name)-8s|%(levelname)s|%(filename)s:%(lineno)s] %(message)s',\n", " level=logging.DEBUG)\n", "log = logging.getLogger()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The deeobiome_train function provide the test evaluation, train evaluation and the deepbiome network instance.\n", "\n", "If we set `number_of_fold`, then DeepBiome performs cross-validation based on that value. If not, DeepBiome package performs cross-validation based on the index file. If both `number_of_fold` option and the index file are missing, then the library performs leave-one-out-cross-validation (LOOCV)." ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|deepbiome.py:115] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:153] -------1 simulation start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:164] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:165] Build network for 1 simulation\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/resource_variable_ops.py:432: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Colocations handled automatically by placer.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[tensorflow|WARNING|deprecation.py:328] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/resource_variable_ops.py:432: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Colocations handled automatically by placer.\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|deepbiome.py:176] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:177] 1 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:137] Training start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:2862: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.cast instead.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[tensorflow|WARNING|deprecation.py:328] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:2862: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.cast instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 600 samples, validate on 150 samples\n", "Epoch 1/10\n", "600/600 [==============================] - 1s 1ms/step - loss: 0.6743 - binary_accuracy: 0.6517 - sensitivity: 0.9212 - specificity: 0.0833 - gmeasure: 0.0194 - auc: 0.4725 - val_loss: 0.6409 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5159\n", "Epoch 2/10\n", "600/600 [==============================] - 0s 223us/step - loss: 0.6310 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.7404 - val_loss: 0.6117 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.7595\n", "Epoch 3/10\n", "600/600 [==============================] - 0s 199us/step - loss: 0.6206 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8101 - val_loss: 0.6110 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.7792\n", "Epoch 4/10\n", "600/600 [==============================] - 0s 225us/step - loss: 0.6214 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8168 - val_loss: 0.6108 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.7948\n", "Epoch 5/10\n", "600/600 [==============================] - 0s 217us/step - loss: 0.6204 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8246 - val_loss: 0.6112 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8087\n", "Epoch 6/10\n", "600/600 [==============================] - 0s 237us/step - loss: 0.6205 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8379 - val_loss: 0.6113 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8203\n", "Epoch 7/10\n", "600/600 [==============================] - 0s 238us/step - loss: 0.6220 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8410 - val_loss: 0.6123 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8255\n", "Epoch 8/10\n", "600/600 [==============================] - 0s 205us/step - loss: 0.6196 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8492 - val_loss: 0.6103 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8313\n", "Epoch 9/10\n", "600/600 [==============================] - 0s 219us/step - loss: 0.6202 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8526 - val_loss: 0.6098 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8407\n", "Epoch 10/10\n", "600/600 [==============================] - 0s 195us/step - loss: 0.6204 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8572 - val_loss: 0.6092 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8457\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_0.h5 \n", "[root |INFO|build_network.py:147] Training end with time 4.45297384262085!\n", "[root |INFO|build_network.py:83] Saved trained model weight at ./example_result/weight_0.h5 \n", "[root |DEBUG|deepbiome.py:185] Save weight at ./example_result/weight_0.h5\n", "[root |DEBUG|deepbiome.py:188] Save history at ./example_result/hist_0.json\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "750/750 [==============================] - 0s 7us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.011869668960571289!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6164737343788147, 0.690666675567627, 1.0, 0.0, 0.0, 0.8536480069160461]\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 18us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.01761341094970703!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6278528571128845, 0.6759999990463257, 1.0, 0.0, 0.0, 0.8606910705566406]\n", "[root |INFO|deepbiome.py:199] Compute time : 5.686180830001831\n", "[root |INFO|deepbiome.py:200] 1 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:153] -------2 simulation start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:164] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:165] Build network for 2 simulation\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|deepbiome.py:176] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:177] 2 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:137] Training start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 600 samples, validate on 150 samples\n", "Epoch 1/10\n", "600/600 [==============================] - 1s 1ms/step - loss: 0.6452 - binary_accuracy: 0.7067 - sensitivity: 0.9363 - specificity: 0.0677 - gmeasure: 0.0364 - auc: 0.4412 - val_loss: 0.6043 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5880\n", "Epoch 2/10\n", "600/600 [==============================] - 0s 613us/step - loss: 0.5917 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5369 - val_loss: 0.5893 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.6307\n", "Epoch 3/10\n", "600/600 [==============================] - 0s 289us/step - loss: 0.5877 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.6219 - val_loss: 0.5875 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.6821\n", "Epoch 4/10\n", "600/600 [==============================] - 0s 621us/step - loss: 0.5869 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.6838 - val_loss: 0.5867 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.7095\n", "Epoch 5/10\n", "600/600 [==============================] - 0s 335us/step - loss: 0.5855 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.7137 - val_loss: 0.5867 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.7346\n", "Epoch 6/10\n", "600/600 [==============================] - 0s 597us/step - loss: 0.5860 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.7500 - val_loss: 0.5866 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.7710\n", "Epoch 7/10\n", "600/600 [==============================] - 0s 223us/step - loss: 0.5852 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.7697 - val_loss: 0.5867 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.7906\n", "Epoch 8/10\n", "600/600 [==============================] - 0s 452us/step - loss: 0.5855 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8008 - val_loss: 0.5869 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8195\n", "Epoch 9/10\n", "600/600 [==============================] - 0s 471us/step - loss: 0.5878 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8200 - val_loss: 0.5871 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8315\n", "Epoch 10/10\n", "600/600 [==============================] - 0s 390us/step - loss: 0.5874 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.8272 - val_loss: 0.5868 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.8467\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_1.h5 \n", "[root |INFO|build_network.py:147] Training end with time 4.857975244522095!\n", "[root |INFO|build_network.py:83] Saved trained model weight at ./example_result/weight_1.h5 \n", "[root |DEBUG|deepbiome.py:185] Save weight at ./example_result/weight_1.h5\n", "[root |DEBUG|deepbiome.py:188] Save history at ./example_result/hist_1.json\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "750/750 [==============================] - 0s 18us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.028972148895263672!\n", "[root |INFO|build_network.py:179] Evaluation: [0.5853651762008667, 0.7279999852180481, 1.0, 0.0, 0.0, 0.7630988359451294]\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 48us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.02693629264831543!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6156458258628845, 0.6959999799728394, 1.0, 0.0, 0.0, 0.782667875289917]\n", "[root |INFO|deepbiome.py:199] Compute time : 5.397497892379761\n", "[root |INFO|deepbiome.py:200] 2 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:153] -------3 simulation start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:164] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:165] Build network for 3 simulation\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|deepbiome.py:176] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:177] 3 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:137] Training start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 600 samples, validate on 150 samples\n", "Epoch 1/10\n", "600/600 [==============================] - 1s 1ms/step - loss: 0.6592 - binary_accuracy: 0.6867 - sensitivity: 0.9821 - specificity: 0.0189 - gmeasure: 0.0352 - auc: 0.5518 - val_loss: 0.6501 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5313\n", "Epoch 2/10\n", "600/600 [==============================] - 0s 203us/step - loss: 0.6245 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5300 - val_loss: 0.6511 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5359\n", "Epoch 3/10\n", "600/600 [==============================] - 0s 186us/step - loss: 0.6232 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5332 - val_loss: 0.6529 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5422\n", "Epoch 4/10\n", "600/600 [==============================] - 0s 211us/step - loss: 0.6226 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5512 - val_loss: 0.6460 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5510\n", "Epoch 5/10\n", "600/600 [==============================] - 0s 183us/step - loss: 0.6222 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5571 - val_loss: 0.6485 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5556\n", "Epoch 6/10\n", "600/600 [==============================] - 0s 179us/step - loss: 0.6216 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5725 - val_loss: 0.6475 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5648\n", "Epoch 7/10\n", "600/600 [==============================] - 0s 224us/step - loss: 0.6206 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5685 - val_loss: 0.6468 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5722\n", "Epoch 8/10\n", "600/600 [==============================] - 0s 214us/step - loss: 0.6201 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5792 - val_loss: 0.6481 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5773\n", "Epoch 9/10\n", "600/600 [==============================] - 0s 239us/step - loss: 0.6203 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5836 - val_loss: 0.6475 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5862\n", "Epoch 10/10\n", "600/600 [==============================] - 0s 171us/step - loss: 0.6205 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - auc: 0.5945 - val_loss: 0.6487 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00 - val_auc: 0.5951\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_2.h5 \n", "[root |INFO|build_network.py:147] Training end with time 3.4070072174072266!\n", "[root |INFO|build_network.py:83] Saved trained model weight at ./example_result/weight_2.h5 \n", "[root |DEBUG|deepbiome.py:185] Save weight at ./example_result/weight_2.h5\n", "[root |DEBUG|deepbiome.py:188] Save history at ./example_result/hist_2.json\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "750/750 [==============================] - 0s 9us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.016149044036865234!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6257709264755249, 0.6813333630561829, 1.0, 0.0, 0.0, 0.5530750155448914]\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 24us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.014859676361083984!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6038214564323425, 0.7120000123977661, 1.0, 0.0, 0.0, 0.47752809524536133]\n", "[root |INFO|deepbiome.py:199] Compute time : 4.102193593978882\n", "[root |INFO|deepbiome.py:200] 3 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:211] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:212] Train Evaluation : ['loss' 'binary_accuracy' 'sensitivity' 'specificity' 'gmeasure' 'auc']\n", "[root |INFO|deepbiome.py:213] mean : [0.609 0.700 1.000 0.000 0.000 0.723]\n", "[root |INFO|deepbiome.py:214] std : [0.017 0.020 0.000 0.000 0.000 0.126]\n", "[root |INFO|deepbiome.py:215] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:216] Test Evaluation : ['loss' 'binary_accuracy' 'sensitivity' 'specificity' 'gmeasure' 'auc']\n", "[root |INFO|deepbiome.py:217] mean : [0.616 0.695 1.000 0.000 0.000 0.707]\n", "[root |INFO|deepbiome.py:218] std : [0.010 0.015 0.000 0.000 0.000 0.165]\n", "[root |INFO|deepbiome.py:219] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:230] Total Computing Ended\n", "[root |INFO|deepbiome.py:231] -----------------------------------------------------------------\n" ] } ], "source": [ "test_evaluation, train_evaluation, network = deepbiome.deepbiome_train(log, network_info, path_info, number_of_fold=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `deepbiome_train` saves the trained model weights, evaluation results and history based on the path information from the configuration." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets check the history files." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xt8VPWd//HXZyaTC7kQLuGWBDIq\nyEUEwhS1SBSt/tS2XoEVdVvtr/Jbt/bm1t/abh9t190+HnZ//lzbLo/+qt3atauyKIWyFaW7ruu1\nKgERhCgiFwkXCQFCLuQyk8/vj3OSTEKSCcmcnFw+z8djHjNzzplzPhnxvOd8v+d8j6gqxhhjTHcC\nfhdgjDFm4LOwMMYYk5CFhTHGmIQsLIwxxiRkYWGMMSYhCwtjjDEJWVgYY4xJyMLCGGNMQhYWxhhj\nEkrxu4BkGTt2rBYVFfldhjHGDCqbN28+pqp5iZYbMmFRVFREaWmp32UYY8ygIiL7e7KcNUMZY4xJ\nyMLCGGNMQhYWxhhjEhoyfRbGmKGjqamJ8vJy6uvr/S5lyEhPT6egoIBQKNSrz1tYGGMGnPLycrKz\nsykqKkJE/C5n0FNVKisrKS8vJxwO92od1gxljBlw6uvrGTNmjAVFkogIY8aM6dORmoWFMWZAsqBI\nrr5+nxYWdcfhv38Ch9/zuxJjjBmwLCwCQXjlISj7g9+VGGMGiMWLF7Nx48Z20x599FHuueeeLj+T\nlZUFwKFDh1iyZEmny1x++eUJLx5+9NFHqaura31/3XXXcfLkyZ6W7hkLi/SRMGke7HvN70qMMQPE\n8uXLWbVqVbtpq1atYvny5Qk/O2nSJJ577rleb7tjWGzYsIHc3Nxery9ZLCwAwiVQvgkaa/2uxBgz\nACxZsoTnn3+exsZGAPbt28ehQ4eYN28eV155JcXFxcyePZvf//73Z3x23759XHDBBQCcPn2aW2+9\nlRkzZnDTTTdx+vTp1uXuueceIpEIs2bN4oc//CEAP/vZzzh06BCLFy9m8eLFgDOU0bFjxwB45JFH\nuOCCC7jgggt49NFHW7c3Y8YM7r77bmbNmsXVV1/dbjvJYqfOghMWr/8jfPInOO9zfldjjInzt/++\ng52HTiV1nTMn5fDDL87qcv7o0aNZsGABL7zwAjfccAOrVq1i2bJlZGRksHbtWnJycjh27BgXX3wx\n119/fZedx7/4xS8YMWIEZWVlbNu2jeLi4tZ5P/7xjxk9ejSxWIwrr7ySbdu28Y1vfINHHnmEl19+\nmbFjx7Zb1+bNm3niiSd4++23UVUuuugiLrvsMkaNGsVHH33EM888w+OPP86yZctYs2YNd9xxR3K+\nLJcdWQAUXgyBEOx91e9KjDEDRHxTVEsTlKryve99jwsvvJDPfe5zHDx4kE8//bTLdbz66qutO+0L\nL7yQCy+8sHXe6tWrKS4uZt68eezYsYOdO3d2W8/rr7/OTTfdRGZmJllZWdx888289prTfB4Oh5k7\ndy4A8+fPZ9++fX350ztlRxYAqSOg4DMWFsYMQN0dAXjphhtu4Nvf/jZbtmyhrq6O+fPn85vf/IaK\nigo2b95MKBSiqKioV9cu7N27l4cffphNmzYxatQo7rzzzj5dA5GWltb6OhgMetIMZUcWLcIlzumz\np0/4XYkxZgDIyspi8eLFfOUrX2nt2K6qqmLcuHGEQiFefvll9u/vfnTvkpISnn76aQDef/99tm3b\nBsCpU6fIzMxk5MiRfPrpp7zwwgutn8nOzqa6uvqMdS1atIh169ZRV1dHbW0ta9euZdGiRcn6cxOy\nsGgRLgFthv1v+l2JMWaAWL58Oe+9915rWNx+++2UlpYye/ZsnnzySaZPn97t5++55x5qamqYMWMG\nP/jBD5g/fz4Ac+bMYd68eUyfPp3bbruNhQsXtn5mxYoVXHPNNa0d3C2Ki4u58847WbBgARdddBFf\n/epXmTdvXpL/4q6JqvbbxrwUiUS0Tzc/ijbAQ1Ng/pfh2p8krzBjzFkrKytjxowZfpcx5HT2vYrI\nZlWNJPqsHVm0SEmDyRdbv4UxxnTC07AQkWtE5EMR2S0iD3SxzDIR2SkiO0TkaXfaYhHZGveoF5Eb\nvawVcJqiju6EmgrPN2WMMYOJZ2dDiUgQWAlcBZQDm0RkvarujFtmKvBdYKGqnhCRcQCq+jIw111m\nNLAb+KNXtbYKX+Y873sNLrjZ880ZY8xg4eWRxQJgt6ruUdVGYBVwQ4dl7gZWquoJAFU92sl6lgAv\nqGpdJ/OSa+IcSMuxpihjjOnAy7DIBw7EvS93p8WbBkwTkTdE5C0RuaaT9dwKPNPZBkRkhYiUikhp\nRUUSmo6CKTBloYWFMcZ04HcHdwowFbgcWA48LiKtI2aJyERgNrCxsw+r6mOqGlHVSF5eXnIqCpfA\n8Y+hqjw56zPGmCHAy7A4CBTGvS9wp8UrB9arapOq7gV24YRHi2XAWlVt8rDO9sLuRS57bRRaY4ar\nyspK5s6dy9y5c5kwYQL5+fmt71sGF0zkrrvu4sMPP+x2mZUrV/LUU08lo2TPeTncxyZgqoiEcULi\nVuC2DsuswzmieEJExuI0S+2Jm78cpwO8/4ybBRmjnaaouYmHIzbGDD1jxoxh69atAPzoRz8iKyuL\n73znO+2WUVVUlUCg89/cTzzxRMLtfO1rX+t7sf3EsyMLVY0C9+I0IZUBq1V1h4g8KCLXu4ttBCpF\nZCfwMnC/qlYCiEgRzpHJK17V2KlAwDm62PsqDJELFo0xybF7925mzpzJ7bffzqxZszh8+DArVqxo\nHWr8wQcfbF320ksvZevWrUSjUXJzc3nggQeYM2cOl1xyCUePOufyfP/7328davzSSy/lgQceYMGC\nBZx//vm8+aYzmkRtbS233HILM2fOZMmSJUQikdYg60+eDiSoqhuADR2m/SDutQL3uY+On93HmR3i\n/SNcAjt/D8f3wJhzfSnBGON64QE4sj2565wwG659qFcf/eCDD3jyySeJRJyLnh966CFGjx5NNBpl\n8eLFLFmyhJkzZ7b7TFVVFZdddhkPPfQQ9913H7/+9a954IEzLz1TVd555x3Wr1/Pgw8+yIsvvsjP\nf/5zJkyYwJo1a3jvvffaDXPen/zu4B6YWq63sLOijDEdnHvuua1BAfDMM89QXFxMcXExZWVlnQ41\nnpGRwbXXXgt0P4T4zTfffMYyr7/+OrfeeivgjCk1a5Y/o/DaEOWdGXMeZE90wiJyl9/VGDO89fII\nwCuZmZmtrz/66CN++tOf8s4775Cbm8sdd9zR6VDjqampra+DwSDRaLTTdbcMNd7dMn6xI4vOiDhN\nUftes34LY0yXTp06RXZ2Njk5ORw+fJiNGzs9y79PFi5cyOrVqwHYvn17wpskecWOLLoSLoFt/wYV\nH8A4G/3SGHOm4uJiZs6cyfTp05kyZUq7ocaT5etf/zpf+tKXmDlzZutj5MiRSd9OIjZEeVdOfgKP\nzoZr/wEu+l/JW68xJiEborxNNBolGo2Snp7ORx99xNVXX81HH31ESsrZ/9bvyxDldmTRldzJMKrI\n6bewsDDG+KSmpoYrr7ySaDSKqvLLX/6yV0HRVxYW3SlaBGXroTkGgaDf1RhjhqHc3Fw2b97sdxnW\nwd2t8GVQXwVHtvldiTHDzlBpIh8o+vp9Wlh0p3WcKLvewpj+lJ6eTmVlpQVGkqgqlZWVpKen93od\n1gzVnewJMPZ8JywWftPvaowZNgoKCigvLycptx4wgBPABQUFvf68hUUi4RLY+jTEmiAY8rsaY4aF\nUChEOBz2uwwTx5qhEgmXQFMtHNzidyXGGOMbC4tEii4FxPotjDHDmoVFIiNGOyNU7u3fkdKNMWYg\nsbDoiXAJHHgHmk77XYkxxvjCwqInwpdBrMEJDGOMGYYsLHpiyiUgQeu3MMYMWxYWPZGWDfnFFhbG\nmGHLwqKnwiVwcDM0VPtdiTHG9DsLi54Kl4DGYP+f/K7EGGP6nYVFTxVeBMFUO4XWGDMseRoWInKN\niHwoIrtF5IEullkmIjtFZIeIPB03fbKI/FFEytz5RV7WmlAowwmMfa/5WoYxxvjBs7AQkSCwErgW\nmAksF5GZHZaZCnwXWKiqs4Bvxc1+Evg/qjoDWAAc9arWHguXwOFtUHfc70qMMaZfeXlksQDYrap7\nVLURWAXc0GGZu4GVqnoCQFWPArihkqKq/+FOr1HVOg9r7ZlwCaCw/w2/KzHGmH7lZVjkAwfi3pe7\n0+JNA6aJyBsi8paIXBM3/aSI/E5E3hWR/+MeqfhrUjGEMu0UWmPMsON3B3cKMBW4HFgOPC4iue70\nRcB3gM8A5wB3dvywiKwQkVIRKe2Xce9TUp0L9CwsjDHDjJdhcRAojHtf4E6LVw6sV9UmVd0L7MIJ\nj3Jgq9uEFQXWAcUdN6Cqj6lqRFUjeXl5nvwRZyhaBBUfQPWn/bM9Y4wZALwMi03AVBEJi0gqcCuw\nvsMy63COKhCRsTjNT3vcz+aKSEsCXAHs9LDWnguXOM92VpQxZhjxLCzcI4J7gY1AGbBaVXeIyIMi\ncr272EagUkR2Ai8D96tqparGcJqgXhKR7YAAj3tV61mZOAfSRtr1FsaYYcXT26qq6gZgQ4dpP4h7\nrcB97qPjZ/8DuNDL+nolEHRuiGT9FsaYYcTvDu7BKVwCJ/bByU/8rsQYY/qFhUVvtPRb7LV+C2PM\n8GBh0RvjZsCIsdYUZYwZNiwsekPEObrY+yqo+l2NMcZ4zsKit8IlUH0IKj/2uxJjjPGchUVvtfZb\n2Cm0xpihz8Kit0afAzn51m9hjBkWLCx6q6XfYt9r0NzsdzXGGOMpC4u+CJdAXSUcHRgjkRhjjFcs\nLPqiaJHzbE1RxpghzsKiL3ILnb4LCwtjzBBnYdFX4RLnznmxqN+VGGOMZyws+ipcAg2n4Mh7fldi\njDGesbDoK+u3MMYMAxYWfZU1DsbNtLAwxgxpFhbJEC6B/X+CaKPflRhjjCcsLJKhaBFET8PBUr8r\nMcYYT1hYJEPRQkCsKcoYM2RZWCRDxijn3twWFsaYIcrCIlnCJXDgHWis87sSY4xJOguLZAlfBs1N\ncOAtvysxxpik8zQsROQaEflQRHaLyANdLLNMRHaKyA4ReTpuekxEtrqP9V7WmRSTL4ZAit2X2xgz\nJKV4tWIRCQIrgauAcmCTiKxX1Z1xy0wFvgssVNUTIjIubhWnVXWuV/UlXVoW5Ees38IYMyR5eWSx\nANitqntUtRFYBdzQYZm7gZWqegJAVY96WI/3wiVwaAvUV/ldiTHGJJWXYZEPHIh7X+5OizcNmCYi\nb4jIWyJyTdy8dBEpdaff6GGdyRMuAW12LtAzxpghxO8O7hRgKnA5sBx4XERy3XlTVDUC3AY8KiLn\ndvywiKxwA6W0oqKiv2ruWsFnICXdmqKMMUOOl2FxECiMe1/gTotXDqxX1SZV3QvswgkPVPWg+7wH\n+G9gXscNqOpjqhpR1UheXl7y/4KzFUqHwgUWFsaYIcfLsNgETBWRsIikArcCHc9qWodzVIGIjMVp\nltojIqNEJC1u+kJgcNy7NFwCn26H2kq/KzHGmKTxLCxUNQrcC2wEyoDVqrpDRB4UkevdxTYClSKy\nE3gZuF9VK4EZQKmIvOdOfyj+LKoBLXyZ87zPTqE1xgwdoqp+15AUkUhES0sHwEB+sSb4SRFc+Gfw\nhUf8rsYYY7olIpvd/uFu+d3BPfQEQzDls9ZvYYwZUiwsvBAugcqP4NRhvysxxpiksLDwQrjEebZ+\nC2PMEGFh4YXxsyE9F/a+4nclxhiTFBYWXggEILzI+i2MMUOGhYVXwpfByU/gxD6/KzHGmD6zsPBK\n0SLn2Y4ujDFDgIWFV/LOh8xxFhbGmCHBwsIrIs5ZUXtfhSFy4aMxZviysPBSuARqPoVju/yuxBhj\n+sTCwkst11tYU5QxZpDrUViIyLlxo8BeLiLfiLvvhOnKqCIYOdnCwhgz6PX0yGINEBOR84DHcO5T\n8bRnVQ0VLf0W+16D5ma/qzHGmF7raVg0u0OO3wT8XFXvByZ6V9YQEi6B0yfg0/f9rsQYY3qtp2HR\nJCLLgS8Df3CnhbwpaYgJ2/UWxpjBr6dhcRdwCfBjVd0rImHgt96VNYTkTIIxUy0sjDGDWkpPFnLv\nUvcNABEZBWSr6k+8LGxICS+CbaudGyMF7YDMGDP49PRsqP8WkRwRGQ1sAR4XEbsNXE+FS6CxBg5t\n9bsSY4zplZ42Q41U1VPAzcCTqnoR8DnvyhpiWseJsiHLjTGDU0/DIkVEJgLLaOvgNj2VORbGX2D9\nFsaYQaunYfEgsBH4WFU3icg5wEfelTUEhUvgwNsQbfC7EmOMOWs9CgtVfVZVL1TVe9z3e1T1Fm9L\n6z+Hq05T0xD1diPhEojWQ/kmb7djjDEe6GkHd4GIrBWRo+5jjYgU9OBz14jIhyKyW0Qe6GKZZSKy\nU0R2iMjTHebliEi5iPxTz/6cs7e/spaFD/0Xa7eUe7UJx5TPggSsKcoYMyj1tBnqCWA9MMl9/Ls7\nrUsiEgRWAtcCM4HlIjKzwzJTge8CC1V1FvCtDqv5O8DTveuUMZlMn5DD6lKPwyJ9JEyaZ2FhjBmU\nehoWear6hKpG3cdvgLwEn1kA7HabrBqBVcANHZa5G1ipqicAVPVoywwRmQ+MB/7Ywxp7bWmkgO0H\nq9h56JS3GwqXOM1QjbXebscYY5Ksp2FRKSJ3iEjQfdwBVCb4TD5wIO59uTst3jRgmoi8ISJvicg1\nACISAP4v8J3uNiAiK0SkVERKKyoqevinnOnGufmkBgM8u/lA4oX7IlwCzVH45E/ebscYY5Ksp2Hx\nFZzTZo8Ah4ElwJ1J2H4KMBW4HFiOc7FfLvCXwAZV7bZtSFUfU9WIqkby8hId6HRtVGYqV80cz7p3\nD9IQjfV6PQkVXgyBkDVFGWMGnZ6eDbVfVa9X1TxVHaeqNwKJzoY6iDOUeYsCd1q8cmC9qjap6l5g\nF054XALcKyL7gIeBL4nIQz2ptbeWRgo4UdfES2VHEy/cW6kjoOAzFhbGmEGnL3fKuy/B/E3AVBEJ\ni0gqcCtOJ3m8dThHFYjIWJxmqT2qeruqTlbVIpymqCdVtdOzqZJl0dQ8JuSk82xpPzRFHX7PGbbc\nGGMGib6EhXQ3073/xb04F/OVAatVdYeIPCgi17uLbcTpD9kJvAzcr6qJ+kI8EQwIS+YX8MquCo5U\n1Xu3oXAJaDPsf9O7bRhjTJL1JSw04QKqG1R1mqqeq6o/dqf9QFXXu69VVe9T1ZmqOltVV3Wyjt+o\n6r19qLPHlswvoFlhjZfXXBREICXDmqKMMYNKt2EhItUicqqTRzXO9RZDStHYTC4Kj+bZ0gOoJszC\n3klJg8kXw97XvFm/McZ4oNuwUNVsVc3p5JGtqj26F8ZgszRSyL7KOjbt87BPIVwCR3dATe9P9zXG\nmP7Ul2aoIem62RPITA2y2suO7vBlzvM+O7owxgwOFhYdjEhN4YtzJvH8tsPeDS44cQ6k5Vi/hTFm\n0LCw6MTSSCGnm2I8v+2QNxsIpsCUhRYWxphBw8KiE8WTczk3L5NnvRxcMLwIjn8MVR4PYGiMMUlg\nYdEJEWFZpJDS/Sf4uKLGm42ES5xnOyvKGDMIWFh04abifIIB8e7oYtwsyBhtTVHGmEHBwqIL47LT\nWXz+ONZsKScaa07+BgIBpylq76vg1TUdxhiTJBYW3VgaKaCiuoFXdnl0PUS4BE6Vw/E93qzfGGOS\nxMKiG1dMH8fYrFTvrrmw6y2MMYOEhUU3QsEAN83L56WyoxyraUj+BsacB9kTrd/CGDPgWVgksDRS\nSLRZWfdux1txJIGI0xRl/RbGmAHOwiKBaeOzmVuYy2qvBhcMl0BtBVR8kPx1G2NMklhY9MCySCG7\nPq1hW3lV8lfeer2FNUUZYwYuC4se+MKciaSHAt50dOdOhtwpFhbGmAHNwqIHctJDXHfBRNZvPcTp\nxljyNxAucc6IavZg3cYYkwQWFj20JFJAdUOUjTuOJH/l4cugvgqObEv+uo0xJgksLHro4vAYCkdn\neNMUFV7kPFtTlDFmgLKw6KFAQFg6v5A3P67kwPG65K48ewKMPd/CwhgzYFlYnIVb5hcgAs9u9mBw\nwXAJ7P8TxJqSv25jjOkjT8NCRK4RkQ9FZLeIPNDFMstEZKeI7BCRp91pU0Rki4hsdaf/hZd19lR+\nbgaXnjeWNZvLaW5O8jUX51wGTbXw5s+Tu15jjEkCz8JCRILASuBaYCawXERmdlhmKvBdYKGqzgK+\n5c46DFyiqnOBi4AHRGSSV7WejWWRQg6ePM2bH1cmd8XTroVZN8FLfwsv/Z1d0W2MGVC8PLJYAOxW\n1T2q2gisAm7osMzdwEpVPQGgqkfd50ZVbRmMKc3jOs/KVTPHMzIjlPyO7mAK3PLPUPwleO1h2HA/\nNHswNLoxxvSClzvhfCB+j1ruTos3DZgmIm+IyFsick3LDBEpFJFt7jp+oqoe3RD77KSHgtw4dxIv\n7jhCVV2S+xcCQfjiz+CSe2HT47DuL6wPwxgzIPj9iz0FmApcDiwHHheRXABVPaCqFwLnAV8WkfEd\nPywiK0SkVERKKyo8uudEJ5ZGCmmMNrP+PY8GF7z67+GK78O2f4PVX4Km+uRvxxhjzoKXYXEQKIx7\nX+BOi1cOrFfVJlXdC+zCCY9W7hHF+8CijhtQ1cdUNaKqkby8vKQW351Zk3KYMTGH1V7dclUESu6H\n6x6GDzfA00uhodqbbRljTA94GRabgKkiEhaRVOBWYH2HZdbhHFUgImNxmqX2iEiBiGS400cBlwIf\neljrWRERlkUK2H6wip2HTnm3oQV3w02/hH1vwJM3Qt1x77ZljDHd8CwsVDUK3AtsBMqA1aq6Q0Qe\nFJHr3cU2ApUishN4GbhfVSuBGcDbIvIe8ArwsKpu96rW3rhxbj6pwQDPbvboLnot5twKf/ZbZyiQ\n33weqj0YbsQYYxIQT+7R4INIJKKlpaX9us2vPbWFNz8+xtvf+xypKR53/+z5b3jmNsjKgy/9HkYV\nebs9Y8ywICKbVTWSaDm/O7gHtaWRAk7UNfFS2afeb+ycy+HL6+H0Sfj1NXDUbpZkjOk/FhZ9sGhq\nHhNy0r0ZXLAzBRG4awNoMzxxLRzc0j/bNcYMexYWfRAMCEvmF/DKrgqOVPXT6a3jZ8FdL0BaFvzL\n9bDv9f7ZrjFmWLOw6KMl8wtoVlizxaPTaDsz5lz4ykbImQT/egvs2th/2zbGDEsWFn1UNDaTBeHR\nPFt6gH49WSBnknOEkTcdVt0G25/rv20bY4YdC4skWBYpZF9lHZv2nejfDWeOgS//OxReBGu+CqW/\n7t/tG2OGDQuLJLhu9gQyU4P919EdLz0H7lgDU6+GP3wbXn+0/2swxgx5FhZJMCI1hS/OmcSG7Yep\naYj2fwGhDLj1KbjgFvjPH8J//siGODfGJJWFRZIsjRRS1xhjw7bD/hQQDMHNj8P8u+D1f4Tn77Mh\nzo0xSWNhkSTFk3M5Ny/Tn6aoFoEgfOEfYeG3nP6LtStsiHNjTFJYWCSJM7hgIaX7T/BxRY2fhcBV\nfwtX/hC2Pwv/9uc2xLkxps8sLJLopuJ8ggHhWa+GLj8bi+6Dz/9f2PUiPLXEhjg3xvSJhUUSjctO\nZ/H5eazZUk40NgD6Cz7zVacfY/+bztXeNsS5MaaXLCySbGmkkIrqBl7Z1X937uvWhUudM6U+3eGM\nJ3XKpw54Y8ygZmGRZFdMH8fYrNSB0RTV4vxrnWsxqsrh1/8Dju/1uyJjzCBjYZFkoWCAm+bl859l\nn1JZ0+B3OW3Ci5whzhtOuUOcl/ldkTFmELGw8MDSSCHRZmXtux1vOe6z/PnOeFLgNEmVb/a3HmPM\noGFh4YFp47OZW5jL6v4eXLAnxs2Ar7wI6SPhyeth76t+V2SMGQRS/C5gqFoaKeBv1r7PtvIq5hTm\n+l1Oe6PDcNeL8Nub4F+XwLJ/cfo1jGkRi0LVJ3B8j9PHdXyP8zh1EMacBwWfcR4TLoRQut/Vmn5g\nYeGRL86ZxIP/vpPVpQcGXlgA5Ex07rr31BJYdTvc9P/gwmV+V2X6U7QBTuxvC4L4R9UBaI4b5yw0\nAkafA9kToLwUdqx1pgdCMGF2W3gURJz7w4v48icZ71hYeCQnPcR1syeyfushvv/5mWSkBv0u6Uwj\nRsOXfg/PLIffrYD6Klhwt3fbizY413qcPhH36PC+df5J5/axmWMhaxxkjoOsPMjMi3s9znmfkupd\nzYNdYy2c2NdJIOx1zo4jrpk0LccJhEnznEEpR5/jHIWOPgeyxrcPgOojTmiUb3Ke3/0tvPNLZ96I\nsU5oFEScAJlU7IyObAY1T8NCRK4BfgoEgV+p6kOdLLMM+BHOv9r3VPU2EZkL/ALIAWLAj1X137ys\n1QtLIwWsffcgG3cc4cZ5+X6X07m0bLj9OXj2TtjwHedsqUV/1f1nztjpd7XDd3f6LfOb6rpeZyAE\nGaOcAMsYBbmFIAGoOersjGoroLGLYVTSR7oB4oZHS7icETTjIHVEr7+qAau+qn1TUfzrmiPtlx0x\nxtn5T/msGwbnwCg3EEaM7vkRQfYEmPEF5wFOs9XRnU54HNzsPO960V1YnL6y/PltRyB55ztjmZlB\nQ7zqgBWRILALuAooBzYBy1V1Z9wyU4HVwBWqekJExqnqURGZBqiqfiQik4DNwAxVPdnV9iKRiJaW\nlnryt/RWc7Ny2cMvM3n0CJ766sV+l9O9WBOs+0vYvhrm3AbZ4zs5Cjhx9jv9jFGQMRoyctvet5vn\nzk/NTLyjaqyD2qNQU+E+H3VCpLai7XXNUWdefVXn6whltgVHu3DJax80WXnOL20vmlNU3Udzgkfc\nMs1RqD7ceZNRXWX79WdNaAuC0eG2o4NRYee/Q385fcINjtK2o5B693/h1GzIL247+siPON+56Xci\nsllVI4mW8/LIYgGwW1X3uAWtAm4AdsYtczewUlVPAKjqUfd5V8sCqnpIRI4CeUCXYTEQBQLC0vmF\nPPIfuzhwvI7C0QP4V20wBDf90tl5v/PLTn7pT4aJc7vZ6buve7LT763UEZBa5LSJJxJtiAuSDuHS\nEiiVH8Mnb7k7205+NAXTnAAzryJdAAAQPElEQVQJZXS9E0+0k+/s0dm2zorAyAInBGZ8sf3Rwagi\nSMvq4/qTJGMUnPc55wHO91L5sdt0tQkOljo369KYM39UUduRR37E6QsZKE2Mqs6/qYZq5+i7scZ5\n3VjnBHnrIxb3uqnD+06WiXW2TGfvm7qfP24G3PIrT78CL8MiH4gfr7scuKjDMtMAROQNnKaqH6nq\ni/ELiMgCIBX42LtSvXPL/AL+8T938dzmcr591TS/y+leIADX/QNc/XcQTB3cnZQpac4OdWRB4mVj\nUScwOguUmgqINTpNYmc8pIvpPZ1/luvIdo8YcqcMzjOQRGDsec5j7nJnWmMdHN7aFiD7XndGSwYn\nrCfOaes4L/iM89/zbP5dxqLQWO3u5KuhoaZth99Q3bbTb50W976xpm25hur2Hf59/i4CEEiJewSd\nH2jt3necn+L8qAukOD9g4uf35N95H/ndwZ0CTAUuBwqAV0Vkdktzk4hMBH4LfFlVzxiZT0RWACsA\nJk+e3F81n5X83AwuPW8sz20u55tXTiUQGAQ74JQ0vyvoX8EUp9kte7zflQw/qSOc/pMpn22bVnWw\nLTzKS6H0n+Gtlc68rPFOaIyb6YR4oh1+9HTP6ghlOv13aVnuczaMmNL2unVeTtv71CznSLrdTr2T\nHX2ww3wJOj/MBhkvw+IgUBj3vsCdFq8ceFtVm4C9IrILJzw2iUgO8DzwN6r6VmcbUNXHgMfA6bNI\ncv1JszRSyDeeeZc3P67k0qlj/S7HmIFtZL7zmHWj8z7WBJ++H3f21Sb44A/OL/H0HGen3bITz5oA\nY+J2+Gkt87vb4Wc5O3TTLS+/oU3AVBEJ44TErcBtHZZZBywHnhCRsTjNUntEJBVYCzypqs95WGO/\nuHrmeHLSU1hdesDCwpizFQw5p/NOmtd2ancsajv4fubZsZCqRoF7gY1AGbBaVXeIyIMicr272Eag\nUkR2Ai8D96tqJbAMKAHuFJGt7mOuV7V6LT0U5MZ5+by44whVdXabU2P6zIKi33l26mx/G4inzsZ7\n/2AVX/j56/zdDbP480uK/C7HGGOAnp86O/h6WQapWZNymDExh2c3D6D7XBhjTA9ZWPQTEWFZpIBt\n5VWUHT7ldznGGHNWLCz60Y1z80kNBgbWXfSMMaYHLCz60ajMVK6aOZ6175bTGD3jshFjjBmwLCz6\n2ZJIASfqmnip7FO/SzHGmB6zsOhnJVPzmJCTzurSA4kXNsaYAcLCop8FA8It8/N5ZVcFR6rq/S7H\nGGN6xMLCB0vnF9KssGaLdXQbYwYHCwsfFI3NZEF4NM9tLmeoXBRpjBnaLCx8sixSyN5jtZTuP+F3\nKcYYk5CFhU+umz2BzNQgqzdZR7cxZuCzsPDJiNQUvjhnEs9vP0xNQxJvqmKMMR6wsPDR0kgBdY0x\nNmw77HcpxhjTLQsLHxVPHsU5eZl2zYUxZsCzsPCRM7hgIaX7T/BxRY3f5RhjTJcsLHx287x8ggGx\nwQWNMQOahYXPxuWks/j8PH63pZxozAYXNMYMTBYWA8DSSCFHqxt49aMKv0sx5gzNzcrx2kZ2H61m\n8/7jHK2ut4tJhyG7ke0AcMX0cYzNSmX1pnKumD7e73LMEKeq1DbGqKxp4FhNI8drG6msaaCytpFj\nNQ1U1jRSWes8H6tp5ERdI7Hm9uEwJjOV6ROzmT4hh/MnZDNjQg5Tx2eRHgr69FcZr1lYDAChYICb\n5uXzxBv7+P667YzMCJGbkcrIjBAjR4Sc9y3PGamkhwKIiN9lmwGkIRpzd/ptO/zjtY0cc3f6LWHQ\nMr+hi/upZKWlMCYrlTGZqRSOHsG8ybmMyUxzpmWlkZ2Wwr7KWj48Uk3ZkWqeens/9U3OugIC4bGZ\nTJ+Yw4wJTpBMn5hNfm6G/XsdAiwsBog7Lp7Cq7uOsWH7EU7WNdLczVF+ajDQFiIZofahkpHKyIwU\nckd0CJuMEDkZIUJBa3k8W6pKrFmJqdLcDM3a8lppVog1qzPNfW5uhpj7Xt1lndfEraf9Z+M/H3O3\n0dzc9tlmVarqmtwAcHf+NY2tRwPV9Z1f2JkaDLg7+lTGZKZx3rgsxmalMSbT2fmPyUplrBsGozNT\nz/rIINasfHK8jg8On6LsSDUfHD7F9vIqno+7dig7LaX1KCT+aCQrzXY/g4kMlbbHSCSipaWlfpeR\nFKpKTUOUk3VNVJ1ue7S8P3m6kVNx71vmnTrdRHWCq8Gz0lKcEMlof8QSHzZZ6SmEAkJKMEBKQEgJ\nCsGAkBIIkBIUZ1qH18GgEAq4y7mfCwaEUDBAQEjqL0tVpSHaTENTM6ebYtQ3xVqf65ua3ecY9dEY\npxubW+c3NMWojzZzurFlvvO6IRpzpkWdz7dMq3fX37EJxk8BgdGZzo5/dKYTAmfs/N1gGJOVSlZa\nii+/6msaonx4pJoPjpzig8Ntz/H/PgtHZzB9gnsUMjGH6ROymTImk2DAjkL6k4hsVtVIouU8jXYR\nuQb4KRAEfqWqD3WyzDLgR4AC76nqbe70F4GLgddV9Qte1jnQiAjZ6SGy00MUnuVno7FmTtVHOVnX\n6AZLU7tgaQsYZ/7HFTWcrHOW8/JWr6EuAscJlE7mBQM0q1Lf1ExDxzCIxujtb5z0UICMUJD0UJCM\nUJC0ULB1Wk5GiPRQgPS4+emhgBt4To0BEQJC62vnGQIBIShCoHU67rLS9bIiBAIQdOeJ+xwUQdxt\nxG/TCffUQbEzzUpLYf6UUcyfMqp1mqpy8OTp1vAoO1LNh0eqeans09Yj6fRQgPPHtz8KmT4hm1GZ\nqT79JaaFZ0cWIhIEdgFXAeXAJmC5qu6MW2YqsBq4QlVPiMg4VT3qzrsSGAH8r56ExVA6svBLfVOM\nqtNNVNc3EW1WojEl2qzEmptpijnNIU2xZvfZeR9tbibaMq85fl5zu3VEY93Ma33f7K7TmQe4O+64\nHXxqkPSUAGmtO/P289M6hEHL/PRQkLQU6+sZiOqbYuw+WkPZ4VN84B6NlB2u5nhtY+syE3LS24XH\neeOyyEkPMSItyIhU57+1/bftnYFwZLEA2K2qe9yCVgE3ADvjlrkbWKmqJwBagsJ9/ZKIXO5hfaaD\nll/U43PS/S7FDCPpoSAX5I/kgvyRrdNUlYqahnZNWGVHqnlj9x6aYmf+wBWBjFCQEakpjEgNtj4y\n01LICLnPqUEyU4NkpKaQ2bqMu3xa/OdS3OWc14PhSK4/eBkW+UD8oEflwEUdlpkGICJv4DRV/UhV\nX+zpBkRkBbACYPLkyX0q1hgzcIgI47LTGZedTsm0vNbpTbFm9lTUsvdYDTUNMU43RqltjFHXGKOu\nIUpdk/vsTqttiFJR3eC+b5t+NtJDgQ4hlNIWKmlBstJS2h7pznN2egqZaW2vs9JCZKWnMCIUJDBI\nw8fv0xFSgKnA5UAB8KqIzFbVkz35sKo+BjwGTjOUV0UaYwaGUDDA+ROyOX9Cdq/X0dys1Edj1DY4\nJzbUtoZIh+eGWLv3tY1Rd3knpA6dPE1tY5TahijV9dEuT0eOJwKZqe2DJXHQOI/M+PfpKaSl9O81\nLV6GxUFo1z9b4E6LVw68rapNwF4R2YUTHps8rMsYM4wFAuIeHSR399cUa24NjpoG91EfpbrBCZSW\n1zX1UWoamtxlYtTUN3G0ur7dsj05AS81GHCObNJTmFOQyz/dVpzUv6cjL8NiEzBVRMI4IXErcFuH\nZdYBy4EnRGQsTrPUHg9rMsYYT4SCAXJHpJI7om9nbqkqp5tiHcKlLXxaXle7oVPbEGPiSO/7GT0L\nC1WNisi9wEac/ohfq+oOEXkQKFXV9e68q0VkJxAD7lfVSgAReQ2YDmSJSDnwP1V1o1f1GmPMQCDS\nduQzzu9i4thFecYYM4z19NRZG/vBGGNMQhYWxhhjErKwMMYYk5CFhTHGmIQsLIwxxiRkYWGMMSYh\nCwtjjDEJDZnrLESkAtjfh1WMBY4lqZzBzr6L9uz7aM++jzZD4buYoqp5iRYaMmHRVyJS2pMLU4YD\n+y7as++jPfs+2gyn78KaoYwxxiRkYWGMMSYhC4s2j/ldwABi30V79n20Z99Hm2HzXVifhTHGmITs\nyMIYY0xCwz4sROQaEflQRHaLyAN+1+MnESkUkZdFZKeI7BCRb/pdk99EJCgi74rIH/yuxW8ikisi\nz4nIByJSJiKX+F2Tn0Tk2+7/J++LyDMi4v0diHw0rMNCRILASuBaYCawXERm+luVr6LAX6nqTOBi\n4GvD/PsA+CZQ5ncRA8RPgRdVdTowh2H8vYhIPvANIKKqF+Dc4O1Wf6vy1rAOC2ABsFtV96hqI7AK\nuMHnmnyjqodVdYv7uhpnZ5Dvb1X+EZEC4PPAr/yuxW8iMhIoAf4ZQFUbVfWkv1X5LgXIEJEUYARw\nyOd6PDXcwyIfOBD3vpxhvHOMJyJFwDzgbX8r8dWjwP8Gmv0uZAAIAxXAE26z3K9EJNPvovyiqgeB\nh4FPgMNAlar+0d+qvDXcw8J0QkSygDXAt1T1lN/1+EFEvgAcVdXNftcyQKQAxcAvVHUeUAsM2z4+\nERmF0woRBiYBmSJyh79VeWu4h8VBoDDufYE7bdgSkRBOUDylqr/zux4fLQSuF5F9OM2TV4jIv/pb\nkq/KgXJVbTnSfA4nPIarzwF7VbVCVZuA3wGf9bkmTw33sNgETBWRsIik4nRQrfe5Jt+IiOC0SZep\n6iN+1+MnVf2uqhaoahHOv4v/UtUh/cuxO6p6BDggIue7k64EdvpYkt8+AS4WkRHu/zdXMsQ7/FP8\nLsBPqhoVkXuBjThnM/xaVXf4XJafFgJ/DmwXka3utO+p6gYfazIDx9eBp9wfVnuAu3yuxzeq+raI\nPAdswTmL8F2G+NXcdgW3McaYhIZ7M5QxxpgesLAwxhiTkIWFMcaYhCwsjDHGJGRhYYwxJiELC2MS\nEJGYiGyNeyTtymURKRKR95O1PmO8MqyvszCmh06r6ly/izDGT3ZkYUwvicg+EfkHEdkuIu+IyHnu\n9CIR+S8R2SYiL4nIZHf6eBFZKyLvuY+W4SGCIvK4e2+EP4pIhrv8N9x7i2wTkVU+/ZnGABYWxvRE\nRodmqD+Lm1elqrOBf8IZpRbg58C/qOqFwFPAz9zpPwNeUdU5OOMqtYwWMBVYqaqzgJPALe70B4B5\n7nr+wqs/zpiesCu4jUlARGpUNauT6fuAK1R1jzsA4xFVHSMix4CJqtrkTj+sqmNFpAIoUNWGuHUU\nAf+hqlPd938NhFT170XkRaAGWAesU9Uaj/9UY7pkRxbG9I128fpsNMS9jtHWl/h5nDs5FgOb3Jvs\nGOMLCwtj+ubP4p7/5L5+k7ZbbN4OvOa+fgm4B1rv7T2yq5WKSAAoVNWXgb8GRgJnHN0Y01/sl4ox\niWXEjcILzn2oW06fHSUi23CODpa7076Oc0e5+3HuLtcyOus3gcdE5H/iHEHcg3OXtc4EgX91A0WA\nn9ltTI2frM/CmF5y+ywiqnrM71qM8Zo1QxljjEnIjiyMMcYkZEcWxhhjErKwMMYYk5CFhTHGmIQs\nLIwxxiRkYWGMMSYhCwtjjDEJ/X9KXOLqM7SgugAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "with open('./%s/hist_0.json' % path_info['model_info']['model_dir'], 'r') as f:\n", " history = json.load(f)\n", " \n", "plt.plot(history['val_loss'], label='Validation')\n", "plt.plot(history['loss'], label='Training')\n", "plt.legend()\n", "plt.xlabel('Epochs')\n", "plt.ylabel('Loss')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Test evauation and train evauation is the numpy array of the shape (number of repetitions, number of evaluation measures)." ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.628, 0.676, 1.000, 0.000, 0.000, 0.861],\n", " [0.616, 0.696, 1.000, 0.000, 0.000, 0.783],\n", " [0.604, 0.712, 1.000, 0.000, 0.000, 0.478]])" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_evaluation" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.616, 0.691, 1.000, 0.000, 0.000, 0.854],\n", " [0.585, 0.728, 1.000, 0.000, 0.000, 0.763],\n", " [0.626, 0.681, 1.000, 0.000, 0.000, 0.553]])" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_evaluation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Load the pre-trained network for training\n", "\n", "If you have a pre-trianed model, you warm_start next training using the pre-trained weights by setting the `warm_start` option in `training_info` to `True`. The file path of the pre-trained weights passed in the `warm_start_model` option. Below is the example:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true }, "outputs": [], "source": [ "warm_start_network_info = {\n", " 'architecture_info': {\n", " 'batch_normalization': 'False',\n", " 'drop_out': '0',\n", " 'weight_initial': 'glorot_uniform',\n", " 'weight_l1_penalty':'0.',\n", " 'weight_decay': 'phylogenetic_tree',\n", " },\n", " 'model_info': {\n", " 'decay': '0.001',\n", " 'loss': 'binary_crossentropy',\n", " 'lr': '0.01',\n", " 'metrics': 'binary_accuracy, sensitivity, specificity, gmeasure',\n", " 'network_class': 'DeepBiomeNetwork',\n", " 'normalizer': 'normalize_minmax',\n", " 'optimizer': 'adam',\n", " 'reader_class': 'MicroBiomeClassificationReader',\n", " 'taxa_selection_metrics': 'sensitivity, specificity, gmeasure, accuracy',\n", " },\n", " 'training_info': {\n", " 'warm_start':'True',\n", " 'warm_start_model':'./example_result/weight.h5',\n", " 'batch_size': '50',\n", " 'epochs': '10',\n", " 'callbacks': 'ModelCheckpoint',\n", " 'monitor': 'val_loss',\n", " 'mode' : 'min',\n", " 'min_delta': '1e-7',\n", " },\n", " 'validation_info': {\n", " 'batch_size': 'None', \n", " 'validation_size': '0.2'\n", " },\n", " 'test_info': {\n", " 'batch_size': 'None'\n", " }\n", "}" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|deepbiome.py:115] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:153] -------1 simulation start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:164] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:165] Build network for 1 simulation\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_0.h5 \n", "[root |INFO|deepbiome.py:176] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:177] 1 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:137] Training start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 600 samples, validate on 150 samples\n", "Epoch 1/10\n", "600/600 [==============================] - 0s 797us/step - loss: 0.6207 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6080 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 2/10\n", "600/600 [==============================] - 0s 123us/step - loss: 0.6181 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6089 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 3/10\n", "600/600 [==============================] - 0s 107us/step - loss: 0.6151 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6033 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 4/10\n", "600/600 [==============================] - 0s 118us/step - loss: 0.6123 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5989 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 5/10\n", "600/600 [==============================] - 0s 126us/step - loss: 0.6060 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5929 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 6/10\n", "600/600 [==============================] - 0s 149us/step - loss: 0.5992 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5807 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 7/10\n", "600/600 [==============================] - 0s 144us/step - loss: 0.5838 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5654 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 8/10\n", "600/600 [==============================] - 0s 165us/step - loss: 0.5646 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5402 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 9/10\n", "600/600 [==============================] - 0s 158us/step - loss: 0.5396 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5113 - val_binary_accuracy: 0.7000 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 10/10\n", "600/600 [==============================] - 0s 136us/step - loss: 0.5159 - binary_accuracy: 0.7067 - sensitivity: 0.9975 - specificity: 0.0588 - gmeasure: 0.0983 - val_loss: 0.4866 - val_binary_accuracy: 0.7467 - val_sensitivity: 0.9447 - val_specificity: 0.2790 - val_gmeasure: 0.5111\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_0.h5 \n", "[root |INFO|build_network.py:147] Training end with time 2.830467700958252!\n", "[root |INFO|build_network.py:83] Saved trained model weight at ./example_result/weight_0.h5 \n", "[root |DEBUG|deepbiome.py:185] Save weight at ./example_result/weight_0.h5\n", "[root |DEBUG|deepbiome.py:188] Save history at ./example_result/hist_0.json\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "750/750 [==============================] - 0s 3us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.009613752365112305!\n", "[root |INFO|build_network.py:179] Evaluation: [0.4993871748447418, 0.7493333220481873, 0.9517374634742737, 0.29741379618644714, 0.5320336818695068]\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 8us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.008378982543945312!\n", "[root |INFO|build_network.py:179] Evaluation: [0.5000216364860535, 0.7480000257492065, 0.9585798978805542, 0.3086419701576233, 0.543928325176239]\n", "[root |INFO|deepbiome.py:199] Compute time : 3.409234046936035\n", "[root |INFO|deepbiome.py:200] 1 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:153] -------2 simulation start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:164] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:165] Build network for 2 simulation\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_1.h5 \n", "[root |INFO|deepbiome.py:176] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:177] 2 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:137] Training start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 600 samples, validate on 150 samples\n", "Epoch 1/10\n", "600/600 [==============================] - 0s 830us/step - loss: 0.5858 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5866 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 2/10\n", "600/600 [==============================] - 0s 159us/step - loss: 0.5853 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5865 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 3/10\n", "600/600 [==============================] - 0s 148us/step - loss: 0.5857 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5867 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 4/10\n", "600/600 [==============================] - 0s 126us/step - loss: 0.5862 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5869 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 5/10\n", "600/600 [==============================] - 0s 124us/step - loss: 0.5852 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5865 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 6/10\n", "600/600 [==============================] - 0s 108us/step - loss: 0.5853 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5864 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 7/10\n", "600/600 [==============================] - 0s 126us/step - loss: 0.5851 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5863 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 8/10\n", "600/600 [==============================] - 0s 117us/step - loss: 0.5853 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5862 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 9/10\n", "600/600 [==============================] - 0s 158us/step - loss: 0.5854 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5862 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 10/10\n", "600/600 [==============================] - 0s 112us/step - loss: 0.5848 - binary_accuracy: 0.7283 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.5860 - val_binary_accuracy: 0.7267 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_1.h5 \n", "[root |INFO|build_network.py:147] Training end with time 2.74385929107666!\n", "[root |INFO|build_network.py:83] Saved trained model weight at ./example_result/weight_1.h5 \n", "[root |DEBUG|deepbiome.py:185] Save weight at ./example_result/weight_1.h5\n", "[root |DEBUG|deepbiome.py:188] Save history at ./example_result/hist_1.json\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "750/750 [==============================] - 0s 5us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.01107335090637207!\n", "[root |INFO|build_network.py:179] Evaluation: [0.5847012400627136, 0.7279999852180481, 1.0, 0.0, 0.0]\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 10us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.009724140167236328!\n", "[root |INFO|build_network.py:179] Evaluation: [0.617382287979126, 0.6959999799728394, 1.0, 0.0, 0.0]\n", "[root |INFO|deepbiome.py:199] Compute time : 3.649474859237671\n", "[root |INFO|deepbiome.py:200] 2 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:153] -------3 simulation start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:164] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:165] Build network for 3 simulation\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_2.h5 \n", "[root |INFO|deepbiome.py:176] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:177] 3 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:137] Training start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Train on 600 samples, validate on 150 samples\n", "Epoch 1/10\n", "600/600 [==============================] - 1s 1ms/step - loss: 0.6220 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6461 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 2/10\n", "600/600 [==============================] - 0s 451us/step - loss: 0.6220 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6533 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 3/10\n", "600/600 [==============================] - 0s 377us/step - loss: 0.6229 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6460 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 4/10\n", "600/600 [==============================] - 0s 134us/step - loss: 0.6208 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6482 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 5/10\n", "600/600 [==============================] - 0s 388us/step - loss: 0.6204 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6482 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 6/10\n", "600/600 [==============================] - 0s 454us/step - loss: 0.6198 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6467 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 7/10\n", "600/600 [==============================] - 0s 149us/step - loss: 0.6202 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6470 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 8/10\n", "600/600 [==============================] - 0s 127us/step - loss: 0.6199 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6466 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 9/10\n", "600/600 [==============================] - 0s 295us/step - loss: 0.6196 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6454 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n", "Epoch 10/10\n", "600/600 [==============================] - 0s 529us/step - loss: 0.6201 - binary_accuracy: 0.6883 - sensitivity: 1.0000 - specificity: 0.0000e+00 - gmeasure: 0.0000e+00 - val_loss: 0.6478 - val_binary_accuracy: 0.6533 - val_sensitivity: 1.0000 - val_specificity: 0.0000e+00 - val_gmeasure: 0.0000e+00\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_2.h5 \n", "[root |INFO|build_network.py:147] Training end with time 4.314647912979126!\n", "[root |INFO|build_network.py:83] Saved trained model weight at ./example_result/weight_2.h5 \n", "[root |DEBUG|deepbiome.py:185] Save weight at ./example_result/weight_2.h5\n", "[root |DEBUG|deepbiome.py:188] Save history at ./example_result/hist_2.json\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "750/750 [==============================] - 0s 3us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.009068727493286133!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6245588660240173, 0.6813333630561829, 1.0, 0.0, 0.0]\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 11us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.008772611618041992!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6026607751846313, 0.7120000123977661, 1.0, 0.0, 0.0]\n", "[root |INFO|deepbiome.py:199] Compute time : 4.939546346664429\n", "[root |INFO|deepbiome.py:200] 3 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:211] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:212] Train Evaluation : ['loss' 'binary_accuracy' 'sensitivity' 'specificity' 'gmeasure']\n", "[root |INFO|deepbiome.py:213] mean : [0.570 0.720 0.984 0.099 0.177]\n", "[root |INFO|deepbiome.py:214] std : [0.052 0.028 0.023 0.140 0.251]\n", "[root |INFO|deepbiome.py:215] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:216] Test Evaluation : ['loss' 'binary_accuracy' 'sensitivity' 'specificity' 'gmeasure']\n", "[root |INFO|deepbiome.py:217] mean : [0.573 0.719 0.986 0.103 0.181]\n", "[root |INFO|deepbiome.py:218] std : [0.052 0.022 0.020 0.145 0.256]\n", "[root |INFO|deepbiome.py:219] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:230] Total Computing Ended\n", "[root |INFO|deepbiome.py:231] -----------------------------------------------------------------\n" ] } ], "source": [ "test_evaluation, train_evaluation, network = deepbiome.deepbiome_train(log, warm_start_network_info, path_info, \n", " number_of_fold=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's check the history plot again." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VNXdx/HPL5N9hSysCSRAWGVN\nDMqiIiqgAi6IIqCgiNIq1bZWan2s2vrUqvXBrVZkEXABhCJqRUSloqJAQEB2wh7CEgIkBELW8/xx\nb0LAQCJk5k6S3/v1ui9m7tw782PEfHPOufccMcaglFJKnY+P0wUopZTyfhoWSimlKqVhoZRSqlIa\nFkoppSqlYaGUUqpSGhZKKaUqpWGhlFKqUhoWSimlKqVhoZRSqlK+ThdQXaKjo018fLzTZSilVI2y\natWqw8aYmMqOqzVhER8fT2pqqtNlKKVUjSIiu6tynHZDKaWUqpSGhVJKqUppWCillKpUrRmzUErV\nHoWFhaSnp3Pq1CmnS6k1AgMDiY2Nxc/P74LO17BQSnmd9PR0wsLCiI+PR0ScLqfGM8aQlZVFeno6\nCQkJF/Qe2g2llPI6p06dIioqSoOimogIUVFRF9VSc2tYiEh/EdkiImkiMuEcxwwVkY0iskFE3rP3\ndRGR7+1960TkdnfWqZTyPhoU1etiv0+3dUOJiAt4HbgWSAdWishHxpiN5Y5JBP4I9DTGHBWRBvZL\nJ4G7jDHbRKQJsEpEFhljjlV7ocVF8NUzEB4LEU0hvClExEJwFOg/VqWUAtzbskgB0owxO4wxBcAs\nYPBZx9wHvG6MOQpgjDlk/7nVGLPNfpwBHAIqvcPwgpzIhO//CQsfhVl3wqQr4YWW8GwjeKUrvH0j\nzH8AvvwLpE6FrZ/DwQ2Qdwx0/XKlaqU+ffqwaNGiM/ZNnDiRcePGnfOc0NBQADIyMhgyZEiFx1x1\n1VWV3jw8ceJETp48Wfb8+uuv59ix6v89+Zdy5wB3U2BvuefpQPezjmkNICLfAS7gKWPMZ+UPEJEU\nwB/YfvYHiMhYYCxAs2bNLqzK8MbwxCErNHLSIXsf5OyD7HT7z32wcykc3w+m5Mxz/UPtlki5FknZ\nc7ul4h9yYXUppRwzbNgwZs2aRb9+/cr2zZo1i+eff77Sc5s0acLcuXMv+LMnTpzIiBEjCA4OBuDT\nTz+94PeqTk5fDeULJAJXAbHAUhHpWNrdJCKNgZnA3cac/ZMajDGTgEkAycnJF/5rvo8PhDW0tqZJ\nFR9TXAS5B+wwqSBUDqyHE4d+fl5gvbNC5OxQaQq+ARdculKq+g0ZMoQnnniCgoIC/P392bVrFxkZ\nGXTt2pW+ffty9OhRCgsL+etf/8rgwWd2mOzatYsbb7yR9evXk5eXx+jRo1m7di1t27YlLy+v7Lhx\n48axcuVK8vLyGDJkCE8//TSvvPIKGRkZ9OnTh+joaJYsWVI2lVF0dDQvvfQSU6dOBWDMmDE8/PDD\n7Nq1iwEDBtCrVy+WLVtG06ZNWbBgAUFBQdX6nbgzLPYBceWex9r7yksHlhtjCoGdIrIVKzxWikg4\n8B/gT8aYH9xYZ9W4fK0f8hGx/LyBZCvKh5yM0y2SM0JlH6SvgLyjPz8vJOZ0iNRrDvXj7a051GsG\nftX7H12pmuTpjzewMSOnWt+zfZNw/jywwzlfj4yMJCUlhYULFzJ48GBmzZrF0KFDCQoKYv78+YSH\nh3P48GEuu+wyBg0adM7B4zfeeIPg4GA2bdrEunXr6NatW9lrzz77LJGRkRQXF9O3b1/WrVvH+PHj\neemll1iyZAnR0dFnvNeqVauYNm0ay5cvxxhD9+7dufLKK6lfvz7btm3j/fff56233mLo0KHMmzeP\nESNGVM+XZXNnWKwEEkUkASsk7gDuPOuYD4FhwDQRicbqltohIv7AfGCGMebC23Oe5hsAkQnWdi4F\nJ6xAKd/NVRoqh7dB2hdQdNblbWGNfx4ipY9DG1ktI6VUtSrtiioNiylTpmCM4fHHH2fp0qX4+Piw\nb98+Dh48SKNGjSp8j6VLlzJ+/HgAOnXqRKdOncpemzNnDpMmTaKoqIj9+/ezcePGM14/27fffsvN\nN99MSIjVtX3LLbfwzTffMGjQIBISEujSpQsASUlJ7Nq1q5q+hdPcFhbGmCIReRBYhDUeMdUYs0FE\nngFSjTEf2a9dJyIbgWLgUWNMloiMAK4AokRklP2Wo4wxa9xVr8f4h0B0orVVxBjIPQhHd8PRXXDM\n/vPoLtj1LaybDZTrcXMFWK2P8gFSPlgCw93791HKzc7XAnCnwYMH88gjj7B69WpOnjxJUlISb7/9\nNpmZmaxatQo/Pz/i4+Mv6N6FnTt38uKLL7Jy5Urq16/PqFGjLuoeiICA013ZLpfrjO6u6uLWMQtj\nzKfAp2fte7LcYwP81t7KH/MO8I47a/NaIhDWyNqaVdDdVZQPx/bCsV12iJQLk70rIT/7zOOD6lcc\nIvWbQ0QcuC7s1n+larvQ0FD69OnDPffcw7BhwwDIzs6mQYMG+Pn5sWTJEnbvPv/s3ldccQXvvfce\nV199NevXr2fdunUA5OTkEBISQkREBAcPHmThwoVcddVVAISFhXH8+PGfdUP17t2bUaNGMWHCBIwx\nzJ8/n5kzZ1b/X/wcnB7gVr+UbwBEt7K2iuQdPTNESlsm+9fCpk+gpPD0seJz1jhJc6ifYD2vFwch\nDbSLS9Vpw4YN4+abb2bWrFkADB8+nIEDB9KxY0eSk5Np27btec8fN24co0ePpl27drRr146kJOsC\nms6dO9O1a1fatm1LXFwcPXv2LDtn7Nix9O/fnyZNmrBkyZKy/d26dWPUqFGkpKQA1gB3165d3dLl\nVBExteRegeTkZKOLH1WipNgaLzmje6tcy+Tsq7lc/tbAe704iGhm/xlrtUjqxemVXMptNm3aRLt2\n7Zwuo9ap6HsVkVXGmOTKztWWRV3i47J+yNeLA3r//PWCE3BsjxUg2Xut7Zj95/Yv4fgBzhgvQSC0\n4Vkh0uzMQAmM8NBfTinlThoW6jT/EGjQztoqUpR/+t6S0hApDZT9a2Hzf6C44MxzAiLsrq44K0DK\nHtuhEtpQu7qUqgE0LFTV+QZAZAtrq0hJiXUn/NmtktJw2fM9nDprAP6Mrq640y2SiFiIbmPdYa+U\ncpyGhao+5e+Ejz1HF+ipHCs8svdaXV5lj/fC9iXWtCrlu7pi2kHLq62teQ/wD/bIX0UpdSYNC+VZ\ngeEQ2B4atq/49aICu6trL2Ssge1fwcrJ8MPr1j0lzS+Hln2t8GjYQWcGVspDNCyUd/H1P30XfMIV\n0HM8FJyEPcuslkfal7D4f6wttKHd6ugLLa6CUPdMTKyU0rBQNYF/MLS6xtr6PWtd/rv9K2vbugjW\nvm8d16gTtLJbHXGXWcGj1AXIysqib9++ABw4cACXy0VMjPXLyIoVK/D3r/zf1ujRo5kwYQJt2rQ5\n5zGvv/469erVY/jw4dVTuBvpfRYOM8ZwqrCEkwVFFBtDTGiArhD2S5QUW1dibf/SannsXQ4lReAX\nAvG9TodHVCvtsqpBvOk+i6eeeorQ0FB+//vfn7HfGIMxBp8adDWf3mfhRiUlhrzCYk4WFHPK/tN6\nXkRe2eNyr53xegl5hdZxpeeVPi7/XuU1DA8gJSGKlIRIuidEktggVMPjfHxc0LSbtV3xKOQfh53f\n2C2PL2GbvYBNRDNo2ccKjhZXWtOgKPULpaWlMWjQILp27cqPP/7I4sWLefrpp1m9ejV5eXncfvvt\nPPmkNaNRr169eO2117jkkkuIjo7mgQceYOHChQQHB7NgwQIaNGjAE088QXR0NA8//DC9evWiV69e\nfPXVV2RnZzNt2jR69OjBiRMnuOuuu9i0aRPt27dn165dTJ48uWziQE+p82Fx7GQBv5m15qwf8qdD\nIL/oZ8toVCrQz4dgf1+C/FwE+bsI9ncR5OciMsSf4PouAv2sfcH+vuUeuyguMfy45xjLd2bx8doM\nACJD/Lk0vj4pCVF0T4ikXeNwXD4aHucUEAZtr7c2gCM7T3dZbZgPq6db05w0TTo9UN40yZqCXnmn\nhRPgwE/V+56NOsKA5y7o1M2bNzNjxgySk61fxp977jkiIyMpKiqiT58+DBkyhPbtz7yAIzs7myuv\nvJLnnnuO3/72t0ydOpUJEyb87L2NMaxYsYKPPvqIZ555hs8++4xXX32VRo0aMW/ePNauXXvGNOee\nVOf/D/HxEY6dLCDI30V0qD/B/sEE2T/cg/1dZ/wwtx77nrG//LFB/i4CfV34XMQP89E9rX8we4/k\n8cPOLFbsPMKKnUdYtOEgAGEBviSXhkeLSDo2jcDPVXOawR4XmQCR98Kl90JxIexbZQVH2pew9Hn4\n+jnrxsEWV5wOj/rNna5aebGWLVuWBQXA+++/z5QpUygqKiIjI4ONGzf+LCyCgoIYMGAAYE0h/s03\n31T43rfcckvZMaVzPn377bc89thjgDWnVIcOzszCW+fDIjzQjwUP9nK6jDOICM2igmkWFczQZGv9\nqP3ZeazYeYTldngs2bIZgCA/F92a1yMl3gqPLnH1CPRzOVm+93L5QbPLrK3P43DyCOz82g6Pr2DT\nx9ZxkS1Pj3XE94aAUGfrrususAXgLqXrSQBs27aNl19+mRUrVlCvXj1GjBhR4VTj5QfEXS4XRUVF\nFb536VTj5zvGKXU+LGqKxhFBDO7SlMFdmgJwODefleXCY+KXWzFfgL/Lh85xEfaYRxTdmtcnNED/\nM1coOBI63GxtxliLT23/0gqPH9+BFZPAxw6YNgOgdX+Iaul01cqL5OTkEBYWRnh4OPv372fRokX0\n79+/Wj+jZ8+ezJkzh969e/PTTz+xcePGan3/qtKfIjVUdGgAAzo2ZkBHazqM7JOFpO4+Utb6+NfX\nO3h9yXZcPsIlTcLp3iKKlPhILo2PJCJY17D4GRGIaW1tl42z5sHa84MVHls/h0WPW1tUIrTpD60H\nQFx3Heuo47p160b79u1p27YtzZs3P2Oq8ery0EMPcdddd9G+ffuyLSLC8xN06qWztdSJ/CJW7zlq\nhceOI6zZe4yC4hJEoG2jcLonRJKSYIVHTJhOM16po7usezq2LLRWLCwphMB6kHit1eJodQ0E1XO6\nylrDmy6ddVpRURFFRUUEBgaybds2rrvuOrZt24av7y//RUUvnVU/ExLgS+/EGHonWjcSnSosZu3e\nY2Utj9kr9/L2sl0AtIgJobt9tVVKQiRN6gU5WLmXqh8P3e+3tvzjVlfVls+sS3N/+gDEZc1d1bq/\n1WWl3VWqmuTm5tK3b1+KioowxvDmm29eUFBcLG1Z1FGFxSWs35ddFh4rdx3h+ClrQK1pvSAuaRpO\nhyYRdGhi/dkwXG8WrFBJsXWF1ZaFsPUzOGT3J0clQut+VnDEXabdVb+Qtizc42JaFhoWCoDiEsPm\nAzks33GE1XuOsjEjhx2HT5S9HhXiT/sm5QMknPiokIu6TLhWOrrb6q7aandXFRec1V3VV28IrIJN\nmzbRtm1b/QWlGhlj2Lx5s3eGhYj0B14GXMBkY8zProETkaHAU1jzUq81xtxp778beMI+7K/GmOnn\n+ywNi+qXm1/Epv05bNiXzYaMHDZk5LDt0HEKi61/MyH+Lto1Di9rfbRvEk7rhmH4++p9H4DdXbXE\nanFsXQQnD5frrupnDZKfay31Om7nzp2EhYURFRWlgVENjDFkZWVx/PhxEhISznjN8bAQERewFbgW\nSAdWAsOMMRvLHZMIzAGuNsYcFZEGxphDIhIJpALJWCGyCkgyxhw91+dpWHhGQVEJWw8eZ2NGDhsy\nrBDZtD+HEwXWtCV+LiGxQVhZ66ND0wjaNQ7Xy3dLSqzuqq0LrbGOQxus/VGtrBZH6/7WJbouvVIN\noLCwkPT09ArvWVAXJjAwkNjYWPz8zvw35g1hcTnwlDGmn/38jwDGmL+VO+Z5YKsxZvJZ5w4DrjLG\n3G8/fxP4rzHm/XN9noaFc0pKDLuyTpS1PjZkZLMxI4esE9YSqyIQHxVid2Od7sqKDq3DV2Ed21Pu\n6qpv7O6qCGhld1clXqPdVcojvOFqqKbA3nLP04HuZx3TGkBEvsPqqnrKGPPZOc5t6r5S1cXw8RFa\nxITSIiaUgZ2bAFaz92BOflnrY0NGNmv3HuM/6/aXndcwPOCMMZAOTSKIrR9UN7od6jWDlPusLf84\n7Pjv6aur1s+1uquaXW7f09EfohOdrljVcU73DfgCicBVQCywVEQ6VvVkERkLjAVo1qyZO+pTF0hE\naBQRSKOIQPq2a1i2P/tkIRv2Z9vdWFaI/HfLIUrsBm54oG/ZQHrnuHpc3iKq9t8HEhAG7QZaW0kJ\nZKw+fXXV509YW2RLayGobnfrVOvKEU53Q/0LWG6MmWY//xKYALRCu6HqjLyCYjYfyCnrxtqYkc3m\nA8fLZvxt0zCMHq2i6NkympQWkYQH1qF+/WN7rdD46QNrrY4WfWDQq1AvzunKVC3hDWMWvlgD3H2B\nfVgD3HcaYzaUO6Y/1qD33SISDfwIdOH0oHbpXLyrsQa4j5zr8zQsapei4hI2ZOTw3fbDLEvLYuWu\nI+QXleDyETo2jaCnHR7dmtevGxMnGgOpU+Hz/7GmWO/3V21lqGrheFjYRVwPTMQaj5hqjHlWRJ4B\nUo0xH4nVOf0PoD9QDDxrjJlln3sP8Lj9Vs+Wtj7ORcOidjtVWMzqPUf5fnsW36UdZm16NsUlBn9f\nH5Kb16dnq2h6tIyiY9MIfGvzlO1Hd8GCB61B8ZZ9YdArEBHrdFWqBvOKsPAkDYu65fipQlbsPMJ3\naVks236YzQeOA9Z6H91bRNGjZRQ9W0XTumEtXGmwpARSp8DiP1srBfZ7FrqO1FaGuiAaFqpOOZyb\nz/fbreD4Li2LPUdOAtbsvFZwRNGjZTRxkcEOV1qNjuyEjx6yWhmtroGBr0CEXjSofhkNC1Wn7T1y\n0uqy2n6YZduzyDyeD0BcZBA9W0bTw+62qvH3epSUwMrJ8MWfrbU3+v8vdBmurQxVZRoWStmMMaQd\nyuW7tMN8tz2LH3ZklU2a2LZRGD1aWsHRvUUkYTX1SqsjO6yxjN3fQeJ1MPBlCG/idFWqBtCwUOoc\niopLWJ+Rw7IKrrTqFBthtzyi6Nashl1pVVJire73xVPg8reWI+08TFsZ6rw0LJSqotIrrZbZg+Wl\nV1oF+PqQHF+fgZ2aMCQptuZcZZW1HRb8GvZ8D4n97FZGY6erUl5Kw0KpC1T+Squl2zJJO5RLi+gQ\nfnddG67v2KhmXF1VUgLL/wVfPgO+/tD/79D5Dm1lqJ/RsFCqGhhj+GLTIV5YtJmtB3Pp2DSCx/q3\npVditNOlVU3WdvjwV7D3B2tK9IETIayR01UpL6JhoVQ1Ki4xzP9xH/+3eCv7juXRs1UUf+jXls5x\nNWDd7ZLicq2MQBjwPHQaqq0MBWhYKOUW+UXFvPvDHl5bksaREwVc37ERv7uuDS1jQp0urXKHt1mt\njPQV0OZ6uHEihDWs/DxVq2lYKOVGuflFTP5mB28t3cGpohJuS4rlN9ck0jgiyOnSzq+kGH74J3z1\nV6uVcf0L0PE2bWXUYRoWSnnA4dx8Xl+Sxrs/7AGBUT3iGXdlS+qH+Dtd2vkd3gYfjoP0ldD2Rrjx\n/yC0gdNVKQdoWCjlQXuPnGTiF9v494/phPr7cv+VLbinVwLB/k4vGXMeJcXw/Wvw1bPgHwzXvwiX\n3KqtjDpGw0IpB2w5cJwXFm3hi00HiQ4N4Dd9W3H7pc3w9/XiezQyt1hjGftSrQWYbnhJWxl1iIaF\nUg5atfsIf1+4hRW7jtAsMpjfXdeagZ2a4OPjpb+1FxdZrYwlz4J/KNzwInS4RVsZdYCGhVIOM8bw\n362ZPP/ZFjbtz6Fd43D+0L8NV7WO8d4b+w5ttsYyMlZDu0F2KyPG6aqUG2lYKOUlSkoMH6/L4B+f\nb2XPkZOkJETyWP+2JDWv73RpFSsugmWvwH//Zq0PfsM/oMPNTlel3ETDQikvU1BUwuyVe3j5yzQO\n5+ZzbfuGPNqvDa0bhjldWsUObYL5D8D+NdD+Jis0QmrIneuqyjQslPJSJwuKmPrtTt78ege5BUXc\n0jWWR65NJLa+Fy7MVFwE302E/z4HgRF2K+Mmp6tS1UjDQikvd/REAW98vZ23l+0CAyMua86v+7Qk\nyhsXZDq40RrL2L8GLr0P+j8HLi++LFhVmYaFUjVExrE8XvlyG3NS9xLk5+K+K1owpncLQgO87Idx\ncaG1Vsb3r0GLPnDbNAjy0nEXVWUaFkrVMGmHcvnH51tYuP4AUSH+PHh1K+7s3owAXy9bgGn1TPjk\nEagfD3fOhqiWTlekLkJVw8KtdwqJSH8R2SIiaSIyoYLXR4lIpoissbcx5V57XkQ2iMgmEXlFvPZa\nQ6WqR6sGobwxIokFv+5Jm0ZhPP3xRq5+8WvmrUqnuMSLfqnrNhLuWgAns+Ctq2HH105XpDzAbWEh\nIi7gdWAA0B4YJiLtKzh0tjGmi71Nts/tAfQEOgGXAJcCV7qrVqW8See4erx332W8c293IkP8+d0H\naxnw8lKWpR12urTT4nvCfV9Za2O8cwukTnW6IuVm7mxZpABpxpgdxpgCYBYwuIrnGiAQ8AcCAD/g\noFuqVMpL9UqMZsGve/L6nd3ILyrhzsnLeeLDnziRX+R0aZbIBLh3sTV+8ckjsPAx6+opVSu5Myya\nAnvLPU+3953tVhFZJyJzRSQOwBjzPbAE2G9vi4wxm84+UUTGikiqiKRmZmZW/99AKYf5+Ag3dGrM\nooevYEyvBN5dvod+E72olREYbo1bXPZra4Gl94bCqWynq1Ju4PTsZh8D8caYTsBiYDqAiLQC2gGx\nWAFztYj0PvtkY8wkY0yyMSY5JkanJFC1V6CfiydubM/cBy7H3+XjXa0MHxf0/18Y+DLs/BomXwtH\ndjhdlapm7gyLfUBcueex9r4yxpgsY0y+/XQykGQ/vhn4wRiTa4zJBRYCl7uxVqVqhKTmkXz6m95n\ntjK2e0krI2kUjPwQThyyBr53fet0RaoauTMsVgKJIpIgIv7AHcBH5Q8Qkcblng4CSrua9gBXioiv\niPhhDW7/rBtKqbqotJXxwf2X4+fy4c63lvM/H673jlZGQm8Y8yWExMCMwbBqutMVqWritrAwxhQB\nDwKLsH7QzzHGbBCRZ0RkkH3YePvy2LXAeGCUvX8usB34CVgLrDXGfOyuWpWqiZLjI/l0vNXKeGf5\nbvq/7CWtjKiW1sB3wpXw8Xj47I/WQkuqRtOb8pSqBVJ3HeHRuevYefgEIy9rzoQBbQlx+g7w4iL4\n/E/WwHera2HIVGtAXHkVr7gpTynlGaWtjHvLtTK+357lbFEuXxjwd2tNjO1fwZRr4chOZ2tSF0zD\nQqlaIsjfxf/c2J4591+OS4Rhb/3Akwu8YCzj0nth5Hw4fsAe+P7O2XrUBdGwUKqWuTQ+koW/uYJ7\neiYw8wcvaWW0uNK64zs40hr4Xj3T2XrUL6ZhoVQtFOTv4smBXtbKiGoJY76wpgr56EH4/Akd+K5B\nNCyUqsW8rpURVB+Gz7PWxFj2Ksy6E07lOFePqjINC6VqudJWxuyxp1sZf16wnpMFDrUyXL5ww4tw\n/YuwbTFM7QdHdztTi6oyDQul6oiUhNOtjBk/7Kb/xG/4YYeDrYyU+2DEPMjZZw187/7euVpUpTQs\nlKpDyrcyfATumORwK6NlH+uO78AImDEI1rznTB2qUhoWStVBpa2M0T3jnW9lRCdaA9/NLrPW+V78\npA58eyENC6XqqCB/F38e2IHZYy9H7FbGUx9tcKaVERwJI/4NyffAdy/D7BGQn+v5OtQ5aVgoVcdZ\nrYzejOoRz9vLdjnXynD5WXd7D3gBtn5mDXwf2+P5OlSFNCyUUgT7+/LUoA7MHnuZs60MEeg+FoZ/\nAMf2WgPfe1d4tgZVIQ0LpVSZ7i2izmhlDHj5G5Y70cpodQ2MWQz+ofD2DbB2tudrUGfQsFBKnaF8\nK8MYuN2pVkZMG2uKkLjuMH8sfPE0lJR4tgZVRsNCKVWh7i2i+OzhM1sZK3Ye8WwRpQPf3e6Gb1+C\nOSN14NshGhZKqXMqbWXMKmtlfM/7Kzw86Ozrb63v3e9vsOVTmNYfstM9W4PSsFBKVe4yu5VxZesY\n/jT/Jz5bv9+zBYjA5b+CO+fAkV3WwHfmVs/WUMdpWCilqiTY35d/Du9G57h6jJ+1xpnLaxOvtQa+\nTQnMvBmy93m+hjpKw0IpVWXB/r5MvftSmkUGc9/0VDZkZHu+iAbtrDmlTmVbgXHSw+ModZSGhVLq\nF6kf4s+Me1IIDfTl7qkr2ZN10vNFNO4Mw96Ho7vg3dt00NsD3BoWItJfRLaISJqITKjg9VEikiki\na+xtTLnXmonI5yKySUQ2iki8O2tVSlVdk3pBzLw3haKSEkZOXU7m8XzPF5HQG4ZMhYzV1lVSRQWe\nr6EOcVtYiIgLeB0YALQHholI+woOnW2M6WJvk8vtnwG8YIxpB6QAh9xVq1Lql2vVIIypoy7lUE4+\no6at4PipQs8X0e5GGPgKbP8KPnxA78NwI3e2LFKANGPMDmNMATALGFyVE+1Q8TXGLAYwxuQaYxxo\n6yqlzqdbs/r8c0Q3thw4ztgZqzhV6MBssd1GwjVPw/p5sPAPYIzna6gD3BkWTYG95Z6n2/vOdquI\nrBORuSISZ+9rDRwTkX+LyI8i8oLdUlFKeZk+bRrwwm2d+H5HFo/MXkNxiQM/rHs9DD0egpVvwdd/\n9/zn1wFVCgsRaSkiAfbjq0RkvIjUq4bP/xiIN8Z0AhYD0+39vkBv4PfApUALYFQFdY0VkVQRSc3M\nzKyGcpRSF+LmrrE8cUM7Fq4/wJML1mOc+O3+2r9Al+Hw37/Birc8//m1XFVbFvOAYhFpBUwC4oDK\nlrTaZx9XKtbeV8YYk2WMKR0Zmwwk2Y/TgTV2F1YR8CHQ7ewPMMZMMsYkG2OSY2JiqvhXUUq5w5je\nLXjgypa8u3wPE7/Y5vkCRKzxizbXw6ePwk9zPV9DLVbVsCixf2jfDLxqjHkUaFzJOSuBRBFJEBF/\n4A7go/IHiEj59xgEbCp3bj3THo/1AAAZ2klEQVQRKU2Aq4GNVaxVKeWQx/q34bakWF7+chszf9jt\n+QJcvtYVUs17wPwHIO1Lz9dQS1U1LApFZBhwN/CJvc/vfCfY4fIgsAgrBOYYYzaIyDMiMsg+bLyI\nbBCRtcB47K4mY0wxVhfUlyLyEyCAtiuV8nIiwt9u6cg17Rrw5IL1/Gedh6cFAfALsu7BiGlrrbiX\nnur5GmohqUrfon110gPA98aY90UkARhqjPGakaTk5GSTmqr/KJTyBnkFxYycspy16cd4e3QKPVtF\ne76I4wet1fZOHYPRn0GDtp6voQYQkVXGmOTKjqtSy8IYs9EYM94OivpAmDcFhVLKuwT5u5hy96W0\niA5l7IxUfkp3YFqQsIYwcj64/OGdW6yV99QFq+rVUP8VkXARiQRWA2+JyEvuLU0pVZNFBPsx/Z4U\n6gX7M2raCnYePuH5IiITrPUw8nOteaROHPZ8DbVEVccsIowxOcAtwAxjTHfgGveVpZSqDRpFBDLz\n3hQMMHLKcg7lnHKgiEvgzlmQvRfeHQL5xz1fQy1Q1bDwta9cGsrpAW6llKpUi5hQpo26lCMnCrhr\n6gqy8xyYFqR5D7htOuxfB7OGQ5EDc1nVcFUNi2ewrmraboxZKSItAAcupFZK1USd4+rx5sgktmfm\nct/0VGemBWnTHwa/Dju/hn/fByUO1FCDVXWA+wNjTCdjzDj7+Q5jzK3uLU0pVZv0TozhH0O7sHL3\nER56/0eKih2Y9K/LMLjuWdi4AP7zO51H6heo6gB3rIjMF5FD9jZPRGLdXZxSqnYZ1LkJf76xPYs3\nHuRP8x2aFqTHg9DrEVg1DZY86/nPr6F8q3jcNKzpPW6zn4+w913rjqKUUrXXqJ4JZJ0o4NWv0ogO\n8+fRfg7c/9D3z3AyC5a+AMHRcNkDnq+hhqlqWMQYY6aVe/62iDzsjoKUUrXfb69tzeHcAl5fsp3o\n0ABG90zwbAEicMP/WUuyfvYYBEdCp6GeraGGqeoAd5aIjBARl72NABxYrV0pVRuICH+96RL6dWjI\n0x9vZMGafZWfVN1cvnDrFIjvDR+Og22LPV9DDVLVsLgH67LZA8B+YAgVTBmulFJV5fIRXr6jK90T\nIvndnLV8vdWBZQb8AuGO96BhB5g9EvYs93wNNURVr4babYwZZIyJMcY0MMbcBOjVUEqpixLo5+Kt\nu5NJbBjGuHdWsWbvMQeKCIfh8yC8Cbx3GxzUCa4rcjEr5f222qpQStVZ4YF+TB99KVGh/oyetoK0\nQ7meLyI0xppHyi/YmkfqqAPTq3u5iwkLqbYqlFJ1WoPwQGbe0x2Xj3D31BXsz87zfBH1m1uBUZgH\nM2+CXF19s7yLCQu9m0UpVW3io0N4e3QK2XmF3D11BcdOFni+iAbtYPgHcPyA1cI4leP5GrzUecNC\nRI6LSE4F23GgiYdqVErVEZc0jWDSXUnsOnySe6enklfgwJQccSkwdCYc2gjvD4NCByY/9ELnDQtj\nTJgxJryCLcwYU9V7NJRSqsp6tIxm4h1dWL3nKL9+bzWFTkwLkngN3PQv2P0tzLsXios8X4OXuZhu\nKKWUcovrOzbmL4Mv4avNh5gw7ydnpgXpdBsMeB42fwKfPFzn55HS1oFSyiuNuKw5h3PzmfjFNqJD\n/fnj9e08X0T3+60Fk5Y+D8FRcO3Tnq/BS2hYKKW81m/6JpKVW8CbS3cQHRrAfVe08HwRfR6Hk4fh\nu4kQEg09HvJ8DV5Aw0Ip5bVEhKcGdeDIiQKe/XQTkSH+3Jrk4QmvReD6FyHvKHz+hNXC6HKnZ2vw\nAm4dsxCR/iKyRUTSRGRCBa+PEpFMEVljb2POej1cRNJF5DV31qmU8l4uH+Gl2zvTs1UUf5i3jq82\nH/R8ET4uuPlNaHEVLHgQtiz0fA0Oc1tYiIgLeB0YALQHholI+woOnW2M6WJvk8967S/AUnfVqJSq\nGQJ8Xbw5Mpl2jcP41burWbX7iOeL8A2A29+Fxp3hg1Gw6zvP1+Agd7YsUoA0e1W9AmAWMLiqJ4tI\nEtAQ+NxN9SmlapDQAF/eHp1Co/BA7nk7lZ2HT3i+iIBQGD4X6jWD9++AAz95vgaHuDMsmgJ7yz1P\nt/ed7VYRWScic0UkDkBEfIB/AL8/3weIyFgRSRWR1MxMvTVfqdouOjSAGfd0RwQemLmKE/kO3P8Q\nEgUj/g0BYTDzFjic5vkaHOD0fRYfA/HGmE7AYmC6vf9XwKfGmPTznWyMmWSMSTbGJMfExLi5VKWU\nN2gWFcyrw7qy7dBx/jBvnTP3YNSLg5EfAgamD4QjOz1fg4e5Myz2AXHlnsfa+8oYY7KMMfn208lA\nkv34cuBBEdkFvAjcJSLPubFWpVQN0jsxhkf7teU/6/bz1jc7nCkipjXctQCK8mD6IDi2t/JzajB3\nhsVKIFFEEkTEH7gD+Kj8ASLSuNzTQcAmAGPMcGNMM2NMPFZX1AxjzM+uplJK1V0PXNmC6zs24rmF\nm/ku7bAzRTTsYLUwTmXDjEGQs9+ZOjzAbWFhjCkCHgQWYYXAHGPMBhF5RkQG2YeNF5ENIrIWGI+u\nvqeUqiIR4fkhnWkZE8qD760m/ehJZwpp0gVGzIPcQ1Zg1NKpzcWR/j43SE5ONqmpqU6XoZTysB2Z\nuQx+7TuaRwcz94EeBPq5nClk9zJ451aonwCjPoHgSGfq+IVEZJUxJrmy45we4FZKqYvSIiaUl27v\nwvp9Ofxp/npnBrwBmveAYe9DVpq1eFKeA0vEupGGhVKqxru2fUPG901k3up03vnBwSVRW1wFd7xr\nreP97hDIP+5cLdVMw0IpVSs83DeRq9s24OmPN5K6y4E7vEslXgtDp0PGj/DuUChw4OZBN9CwUErV\nCj4+wv/d3oXY+kGMe3c1B3McXOGu7Q1wy1uw9wd7tT0H1hSvZhoWSqlaIyLIjzdHJnMiv4hfvbua\ngiIHVtkrdcktcNMbsHMpzLkLivIrP8eLaVgopWqVNo3CeH5IJ1btPspfPtnobDGd74CBE2Hb5zD3\nHigudLaei6BhoZSqdW7s1ISxV7Rg5g+7+SDV4Turk0bBgBes5Vn/PRZKip2t5wLp4kdKqVrpD/3a\nsCEjmz99uJ42jcLoFFvPuWK6j4WiU7D4f6ypzgf/E3xq1u/qNatapZSqIl+XD68O60ZMaAAPzFxF\nVq7DYwY9x0OfJ2Dt+/DJw1DDbojWsFBK1VqRIf78a0QSh08U8ND7P1JU7OCAN8CVj0Lv38Pq6bDw\nsRoVGBoWSqlarWNsBP97c0eWbc/i+UVbnC4Hrn4CLn8QVrxpdUvVkMDQMQulVK03JCmWdenHmLR0\nBx2bRjCwcxPnihGB6/5qXUq77FXwDYKr/+RcPVWkYaGUqhOeuKE9GzNy+MPcdSQ2DKVto3DnihGB\nAc9bg95Ln7cGva8478KgjtNuKKVUneDv68M/h3cjLNCX+2euIvukw/c8+PjAwJeh0+3w1V9g2WvO\n1lMJDQulVJ3RIDyQN0Z0I+NYHg/P/pGSEofHC3xc1mW07W+Cz/8EK95ytp7z0LBQStUpSc0jeXJg\nB5ZsyWTiF1udLgdcvnDrZGhzA3z6e1g9w+mKKqRhoZSqc0Z0b8ZtSbG88lUaizcedLoccPnBbdOg\n1TXw0XhYO9vpin5Gw0IpVeeICH+56RI6xUbw29lr2J6Z63RJ1iD37e9AfC/48AHYMN/pis6gYaGU\nqpMC/Vy8MSIJP18f7p+5itz8IqdLAr8guHM2xHWHeWNg86dOV1RGw0IpVWc1rRfEa8O6siMzl0c/\nWOvckqzl+YfAnXOgcWf44G7Y9oXTFQFuDgsR6S8iW0QkTUQmVPD6KBHJFJE19jbG3t9FRL4XkQ0i\nsk5EbndnnUqpuqtHq2j+OKAdC9cf4I2vtztdjiUwHEbMg5g2MHs47Pja6YrcFxYi4gJeBwYA7YFh\nItK+gkNnG2O62Ntke99J4C5jTAegPzBRRBycMlIpVZuN6Z3AwM5NeHHRFpZuzXS6HEtQfRi5AOon\nwPt3wO7vHS3HnS2LFCDNGLPDGFMAzAIGV+VEY8xWY8w2+3EGcAiIcVulSqk6TUT4+60dad0wjPGz\nfmTvkZNOl2QJiYK7P4LwpvDubZC+yrFS3BkWTYHyq46k2/vOdqvd1TRXROLOflFEUgB/wEvah0qp\n2ijY35c3RyZRUmK4f+Yq8gq8ZJGi0AZWYIREwTs3w/61jpTh9AD3x0C8MaYTsBiYXv5FEWkMzARG\nG2N+NrewiIwVkVQRSc3M9JKmo1KqxmoeFcLLw7qy6UAOj8//yTsGvAHCm8DdH0NAOMy4CQ56frlY\nd4bFPqB8SyHW3lfGGJNljCldkWQykFT6moiEA/8B/mSM+aGiDzDGTDLGJBtjkmNitJdKKXXx+rRp\nwCPXtGb+j/t4e9kup8s5rV4zuGuBdT/GjEGQ6dm7z90ZFiuBRBFJEBF/4A7go/IH2C2HUoOATfZ+\nf2A+MMMYM9eNNSql1M882KcV17RryLP/2cTyHVlOl3NaVEu4y/4xOmMQHNnhsY92W1gYY4qAB4FF\nWCEwxxizQUSeEZFB9mHj7ctj1wLjgVH2/qHAFcCocpfVdnFXrUopVZ6Pj/DS7Z1pFhnMr99bzYHs\nU06XdFpMayswivJh+iA4tscjHyte0yd3kZKTk01qaqrTZSilapG0Q8cZ/Np3JDYMY/b9lxHg63K6\npNMy1liti6BIGP2pNa5xAURklTEmubLjnB7gVkopr9WqQRj/GNqZNXuP8dRHnh9UPq8mXWDEv+HE\nYXh3KJS49+otXSlPKaXOo/8ljfnVVS3553+30zk2gjtSmjld0mmxyTD8Ayg8aa2N4UYaFkopVYnf\nXdeGn/Zl8+SCDbRpFEbXZvWdLum05pd75GO0G0oppSrh8hFeuaMrDcIDGPfOajKP51d+Ui2jYaGU\nUlVQP8SfN0cmcSyvgAffW01h8c/uE67VNCyUUqqKOjSJ4LlbOrF85xH+9ulmp8vxKB2zUEqpX+Cm\nrk1Zm36Mqd/tpHNcBIO7VDTlXe2jLQullPqFHr++Hd0TIvnD3HXedYe3G2lYKKXUL+Tn8uGNEUnE\nRQYzZnoq6/dlO12S22lYKKXUBYgM8WfmvSmEB/lx99QVbM/Mdbokt9KwUEqpC9Q4Ioh3xnRHBEZO\nXk7GsTynS3IbDQullLoICdEhTL8nheP5RYyYspys3Np5D4aGhVJKXaQOTSKYOupSMo7lcfe0FRw/\nVeh0SdVOw0IpparBpfGRvDE8ic37jzNmeiqnCr1kWdZqomGhlFLVpE/bBvxjaGdW7DpS6+7y1rBQ\nSqlqNLhLU54ZfAlfbDrEH+auo6SkdqwZpHdwK6VUNRt5WXNy8gp5YdEWIoL8+PPA9oiI02VdFA0L\npZRyg19d1ZJjJwt465udRAT58ci1rZ0u6aJoWCillBuICI9f347svEJe/nIb9YL9GN0zwemyLpiG\nhVJKuYmI8L83dyQnr4inP95IRJAft3SLdbqsC+LWAW4R6S8iW0QkTUQmVPD6KBHJFJE19jam3Gt3\ni8g2e7vbnXUqpZS7+Lp8eHlYF3q2iuLRuetYvPGg0yVdELeFhYi4gNeBAUB7YJiItK/g0NnGmC72\nNtk+NxL4M9AdSAH+LCJetI6hUkpVXYCvi0kjk+nYNIJfv7ea77fXvJlq3dmySAHSjDE7jDEFwCxg\ncBXP7QcsNsYcMcYcBRYD/d1Up1JKuV1IgC/TRl1KfFQw981IZV36MadL+kXcGRZNgb3lnqfb+852\nq4isE5G5IhL3C89VSqkao36IPzPv7U69YGum2rRDx50uqcqcvinvYyDeGNMJq/Uw/ZecLCJjRSRV\nRFIzMzPdUqBSSlWnhuGBvDumO74uH0ZOWUH60ZNOl1Ql7gyLfUBcueex9r4yxpgsY0zpFI2TgaSq\nnmufP8kYk2yMSY6Jiam2wpVSyp2aR4Uw454UTuQXMXLKCg7XgJlq3RkWK4FEEUkQEX/gDuCj8geI\nSONyTwcBm+zHi4DrRKS+PbB9nb1PKaVqhXaNw5k2+lL2Z+dx15QV5Hj5TLVuCwtjTBHwINYP+U3A\nHGPMBhF5RkQG2YeNF5ENIrIWGA+Mss89AvwFK3BWAs/Y+5RSqtZIah7JmyOT2XboOGPeTiWvwHtn\nqhVjasckV8nJySY1NdXpMpRS6hf7ZF0GD73/I1e1jmHSXcn4uTw3nCwiq4wxyZUd5/QAt1JK1Xk3\ndmrCszd1ZMmWTH43Z61XzlSr030opZQXuLN7M7LzCvn7Z5uJCPLjmcEdvGqmWg0LpZTyEuOuasmx\nvALe/HoH9YL9+N11bZwuqYyGhVJKeZEJ/duSfbKQV79KIyLIjzG9WzhdEqBhoZRSXkVEePbmjhw/\nVcRf/7OJiCA/bkuOq/xEN9OwUEopL+PyEV66vTM5pwp5bN46woP86NehkaM16dVQSinlhQJ8Xbw5\nMokucfV46L0f+S7tsKP1aFgopZSXCvb3ZdqoFFrEhHDfjFTW7HVuploNC6WU8mIRwX7MuCeF6NAA\nRk1bwbaDzsxUq2GhlFJerkF4IO/c2x1/lw8jpixn7xHPz1SrYaGUUjVAs6hgZt7bnVOFJYycspxD\nx0959PM1LJRSqoZo0yiMaaMv5WBOPndNWUF2nudmqtWwUEqpGqRbs/pMuiuJ7Zm53PP2Sk4WFHnk\nczUslFKqhumdGMPLd3Tlxz1HGffOagqKStz+mRoWSilVA13fsTF/u6UjX2/N5Ldz1rh9plq9g1sp\npWqo2y+1ZqrNzS/G3RPUalgopVQNNvaKlh75HO2GUkopVSkNC6WUUpXSsFBKKVUpDQullFKVcmtY\niEh/EdkiImkiMuE8x90qIkZEku3nfiIyXUR+EpFNIvJHd9aplFLq/NwWFiLiAl4HBgDtgWEi0r6C\n48KA3wDLy+2+DQgwxnQEkoD7RSTeXbUqpZQ6P3e2LFKANGPMDmNMATALGFzBcX8B/g6UnxXLACEi\n4gsEAQVAjhtrVUopdR7uDIumwN5yz9PtfWVEpBsQZ4z5z1nnzgVOAPuBPcCLxpgjbqxVKaXUeTh2\nU56I+AAvAaMqeDkFKAaaAPWBb0TkC2PMjrPeYyww1n6aKyJbLqKkaMDZdQu9h34XZ9Lv40z6fZxW\nG76L5lU5yJ1hsQ+IK/c81t5XKgy4BPivWPepNwI+EpFBwJ3AZ8aYQuCQiHwHJANnhIUxZhIwqTqK\nFZFUY0xydbxXTaffxZn0+ziTfh+n1aXvwp3dUCuBRBFJEBF/4A7go9IXjTHZxphoY0y8MSYe+AEY\nZIxJxep6uhpAREKAy4DNbqxVKaXUebgtLIwxRcCDwCJgEzDHGLNBRJ6xWw/n8zoQKiIbsEJnmjFm\nnbtqVUopdX5uHbMwxnwKfHrWvifPcexV5R7nYl0+60nV0p1VS+h3cSb9Ps6k38dpdea7EGPcOwe6\nUkqpmk+n+1BKKVWpOh8WVZ2SpC4QkTgRWSIiG0Vkg4j8xumanCYiLhH5UUQ+cboWp4lIPRGZKyKb\n7Wl4Lne6JieJyCP2/yfrReR9EQl0uiZ3qtNhUdUpSeqQIuB3xpj2WFeg/bqOfx9gTUWzyekivMTL\nWJe0twU6U4e/FxFpCowHko0xlwAurCs+a606HRZUfUqSOsEYs98Ys9p+fBzrh0HT859Ve4lILHAD\nMNnpWpwmIhHAFcAUAGNMgTHmmLNVOc4XCLKnJQoGMhyux63qelhUOiVJXWVP3NiVMyd4rGsmAn8A\nSpwuxAskAJnANLtbbrJ9D1SdZIzZB7yIdU/YfiDbGPO5s1W5V10PC1UBEQkF5gEPG2Pq5ASOInIj\ncMgYs8rpWryEL9ANeMMY0xVr7rY6O8YnIvWxeiESsKYlChGREc5W5V51PSwqm5KkzhERP6ygeNcY\n82+n63FQT2CQiOzC6p68WkTecbYkR6UD6caY0pbmXKzwqKuuAXYaYzLtaYn+DfRwuCa3quthcd4p\nSeoasSbpmgJsMsa85HQ9TjLG/NEYE2tPRXMH8JUxplb/5ng+xpgDwF4RaWPv6gtsdLAkp+0BLhOR\nYPv/m77U8gF/x2ad9QbGmCIRKZ2SxAVMNcZscLgsJ/UERgI/icgae9/j9p34Sj0EvGv/YrUDGO1w\nPY4xxiwXkbnAaqyrCH+klt/NrXdwK6WUqlRd74ZSSilVBRoWSimlKqVhoZRSqlIaFkoppSqlYaGU\nUqpSGhZKVUJEikVkTbmt2u5cFpF4EVlfXe+nlLvU6fsslKqiPGNMF6eLUMpJ2rJQ6gKJyC4ReV5E\nfhKRFSLSyt4fLyJficg6EflSRJrZ+xuKyHwRWWtvpdNDuETkLXtthM9FJMg+fry9tsg6EZnl0F9T\nKUDDQqmqCDqrG+r2cq9lG2M6Aq9hzVIL8Cow3RjTCXgXeMXe/wrwtTGmM9a8SqWzBSQCrxtjOgDH\ngFvt/ROArvb7POCuv5xSVaF3cCtVCRHJNcaEVrB/F3C1MWaHPQHjAWNMlIgcBhobYwrt/fuNMdEi\nkgnEGmPyy71HPLDYGJNoP38M8DPG/FVEPgNygQ+BD40xuW7+qyp1TtqyUOrimHM8/iXyyz0u5vRY\n4g1YKzl2A1bai+wo5QgNC6Uuzu3l/vzefryM00tsDge+sR9/CYyDsrW9I871piLiA8QZY5YAjwER\nwM9aN0p5iv6molTlgsrNwgvWOtSll8/WF5F1WK2DYfa+h7BWlHsUa3W50tlZfwNMEpF7sVoQ47BW\nWauIC3jHDhQBXtFlTJWTdMxCqQtkj1kkG2MOO12LUu6m3VBKKaUqpS0LpZRSldKWhVJKqUppWCil\nlKqUhoVSSqlKaVgopZSqlIaFUkqpSmlYKKWUqtT/A4y8RdqoXiVHAAAAAElFTkSuQmCC\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "with open('./%s/hist_0.json' % path_info['model_info']['model_dir'], 'r') as f:\n", " history = json.load(f)\n", " \n", "plt.plot(history['val_loss'], label='Validation')\n", "plt.plot(history['loss'], label='Training')\n", "plt.legend()\n", "plt.xlabel('Epochs')\n", "plt.ylabel('Loss')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 6. Load the pre-trained network for testing\n", "\n", "To test the trained model, we can use the `deepbiome_test` function. \n", "\n", "If you use the index file (`idx_path`), this function provides the evaluation using the test index (index set not included in the index file) for each fold. If not, this function provides the evaluation using the whole samples. \n", "\n", "If `number_of_fold` is set to `k`, the function will test the model only with first `k` repetitions.\n", "\n", "We can use the testing metrics different with the training. In the example below, we additionally used `AUC` metric." ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": true }, "outputs": [], "source": [ "test_network_info = {\n", " 'architecture_info': {\n", " 'batch_normalization': 'False',\n", " 'drop_out': '0',\n", " 'weight_initial': 'glorot_uniform',\n", " 'weight_l1_penalty':'0.',\n", " 'weight_decay': 'phylogenetic_tree',\n", " },\n", " 'model_info': {\n", " 'lr': '0.01',\n", " 'decay': '0.001',\n", " 'loss': 'binary_crossentropy',\n", " 'metrics': 'binary_accuracy, sensitivity, specificity, gmeasure, auc',\n", " 'taxa_selection_metrics': 'sensitivity, specificity, gmeasure, accuracy',\n", " 'network_class': 'DeepBiomeNetwork',\n", " 'optimizer': 'adam',\n", " 'reader_class': 'MicroBiomeClassificationReader',\n", " 'normalizer': 'normalize_minmax',\n", " },\n", " 'test_info': {\n", " 'batch_size': 'None'\n", " }\n", "}" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": true }, "outputs": [], "source": [ "test_path_info = {\n", " 'data_info': {\n", " 'count_list_path': resource_filename('deepbiome', 'tests/data/gcount_list.csv'),\n", " 'count_path': resource_filename('deepbiome', 'tests/data/count'),\n", " 'data_path': resource_filename('deepbiome', 'tests/data'),\n", " 'idx_path': resource_filename('deepbiome', 'tests/data/classification_idx.csv'),\n", " 'tree_info_path': resource_filename('deepbiome', 'tests/data/genus48_dic.csv'),\n", " 'x_path': '',\n", " 'y_path': 'classification_y.csv'\n", " },\n", " 'model_info': {\n", " 'evaluation': 'eval.npy',\n", " 'model_dir': './example_result/',\n", " 'weight': 'weight.h5'\n", " }\n", "}" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|deepbiome.py:293] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:325] Test Evaluation : ['loss' 'binary_accuracy' 'sensitivity' 'specificity' 'gmeasure' 'auc']\n", "[root |INFO|deepbiome.py:327] -------1 fold test start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:338] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:339] Build network for 1 fold testing\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_0.h5 \n", "[root |INFO|deepbiome.py:350] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:351] 1 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 352us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.2729969024658203!\n", "[root |INFO|build_network.py:179] Evaluation: [0.5000216364860535, 0.7480000257492065, 0.9585798978805542, 0.3086419701576233, 0.543928325176239, 0.8766162395477295]\n", "[root |INFO|deepbiome.py:356] \n", "[root |INFO|deepbiome.py:357] Compute time : 0.9056384563446045\n", "[root |INFO|deepbiome.py:358] 1 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:327] -------2 fold test start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:338] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:339] Build network for 2 fold testing\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_1.h5 \n", "[root |INFO|deepbiome.py:350] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:351] 2 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 353us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.25853705406188965!\n", "[root |INFO|build_network.py:179] Evaluation: [0.617382287979126, 0.6959999799728394, 1.0, 0.0, 0.0, 0.8632410764694214]\n", "[root |INFO|deepbiome.py:356] \n", "[root |INFO|deepbiome.py:357] Compute time : 0.8716459274291992\n", "[root |INFO|deepbiome.py:358] 2 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:327] -------3 fold test start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:338] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:339] Build network for 3 fold testing\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_2.h5 \n", "[root |INFO|deepbiome.py:350] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:351] 3 fold computing start!----------------------------------\n", "[root |INFO|build_network.py:173] Evaluation start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 381us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:178] Evaluation end with time 0.28519654273986816!\n", "[root |INFO|build_network.py:179] Evaluation: [0.6026607751846313, 0.7120000123977661, 1.0, 0.0, 0.0, 0.5095973610877991]\n", "[root |INFO|deepbiome.py:356] \n", "[root |INFO|deepbiome.py:357] Compute time : 0.8778982162475586\n", "[root |INFO|deepbiome.py:358] 3 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:367] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:368] Test Evaluation : ['loss' 'binary_accuracy' 'sensitivity' 'specificity' 'gmeasure' 'auc']\n", "[root |INFO|deepbiome.py:369] mean : [0.573 0.719 0.986 0.103 0.181 0.750]\n", "[root |INFO|deepbiome.py:370] std : [0.052 0.022 0.020 0.145 0.256 0.170]\n", "[root |INFO|deepbiome.py:371] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:372] Total Computing Ended\n", "[root |INFO|deepbiome.py:373] -----------------------------------------------------------------\n" ] } ], "source": [ "evaluation = deepbiome.deepbiome_test(log, test_network_info, test_path_info, number_of_fold=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This function provides the evaluation result as a numpy array with a shape of (number of repetition, number of evaluation measures)." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " loss binary_accuracy sensitivity specificity gmeasure auc\n", "Mean: 0.573 0.719 0.986 0.103 0.181 0.750\n", "Std : 0.052 0.022 0.020 0.145 0.256 0.170\n" ] } ], "source": [ "print(' %s' % ''.join(['%16s'%'loss']+ ['%16s'%s.strip() for s in network_info['model_info']['metrics'].split(',')]))\n", "print('Mean: %s' % ''.join(['%16.3f'%v for v in np.mean(evaluation, axis=0)]))\n", "print('Std : %s' % ''.join(['%16.3f'%v for v in np.std(evaluation, axis=0)]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 7. Load the pre-trained network for prediction\n", "\n", "If you want to predict using the pre-trained model, you can use the `deepbiome_prediction` function. If `number_of_fold` is setted as `k`, the function will predict only with first `k` repetitions sample's outputs. \n", "\n", "If `change_weight_for_each_fold` is set as `False`, the function will predict the output of every repetition by same weight from the given path. If `change_weight_for_each_fold` is set as `True`, the function will predict the output of by each repetition weight.\n", "\n", "If 'get_y=True', the function will provide a list of tuples (prediction, true output) as a output with the shape of `(n_samples, 2, n_classes)`. If 'get_y=False', the function will provide predictions only. The output will have the shape of `(n_samples, n_classes)`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 7.1 Prediction with fixed weight\n", "\n", "If we want to predict new data from one pre-trained model, we can use the option below. We fixed the weight `weight_0.h5` for predicting the whole samples (without using index file)." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": true }, "outputs": [], "source": [ "prediction_network_info = {\n", " 'architecture_info': {\n", " 'batch_normalization': 'False',\n", " 'drop_out': '0',\n", " 'weight_initial': 'glorot_uniform',\n", " 'weight_l1_penalty':'0.',\n", " 'weight_decay': 'phylogenetic_tree',\n", " },\n", " 'model_info': {\n", " 'decay': '0.001',\n", " 'loss': 'binary_crossentropy',\n", " 'lr': '0.01',\n", " 'metrics': 'binary_accuracy, sensitivity, specificity, gmeasure, auc',\n", " 'network_class': 'DeepBiomeNetwork',\n", " 'normalizer': 'normalize_minmax',\n", " 'optimizer': 'adam',\n", " 'reader_class': 'MicroBiomeClassificationReader',\n", " 'taxa_selection_metrics': 'sensitivity, specificity, gmeasure, accuracy',\n", " },\n", " 'test_info': {\n", " 'batch_size': 'None'\n", " }\n", "}" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": true }, "outputs": [], "source": [ "prediction_path_info = {\n", " 'data_info': {\n", " 'count_list_path': resource_filename('deepbiome', 'tests/data/gcount_list.csv'),\n", " 'count_path': resource_filename('deepbiome', 'tests/data/count'),\n", " 'data_path': resource_filename('deepbiome', 'tests/data'),\n", " 'tree_info_path': resource_filename('deepbiome', 'tests/data/genus48_dic.csv'),\n", " 'x_path': '',\n", " },\n", " 'model_info': {\n", " 'model_dir': './example_result/',\n", " 'weight': 'weight_0.h5'\n", " }\n", "}" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|deepbiome.py:450] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:480] -------1 th repeatition prediction start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:498] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_0.h5 \n", "[root |INFO|deepbiome.py:509] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:193] Prediction start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "1000/1000 [==============================] - 0s 49us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:198] Prediction end with time 0.052338600158691406!\n", "[root |INFO|deepbiome.py:513] Compute time : 0.6934528350830078\n", "[root |INFO|deepbiome.py:514] 1 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:519] Total Computing Ended\n", "[root |INFO|deepbiome.py:520] -----------------------------------------------------------------\n" ] } ], "source": [ "prediction = deepbiome.deepbiome_prediction(log, prediction_network_info, prediction_path_info,\n", " num_classes = 1, number_of_fold=None)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(1, 1000, 1)" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "prediction.shape" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.779],\n", " [0.771],\n", " [0.620],\n", " [0.501],\n", " [0.863],\n", " [0.622],\n", " [0.630],\n", " [0.496],\n", " [0.822],\n", " [0.843]], dtype=float32)" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "prediction[0,:10]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 7.2 Prediction with each fold weight\n", "\n", "If we want to predict the test outputs from each repetitions, we can use the option belows.\n", "\n", "The example below shows how to predict the 5 repetition outputs.\n", "We set `idx_path` for using the index file `classification_idx.csv` to predict only the test set for each repetition." ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": true }, "outputs": [], "source": [ "prediction_network_info = {\n", " 'architecture_info': {\n", " 'batch_normalization': 'False',\n", " 'drop_out': '0',\n", " 'weight_initial': 'glorot_uniform',\n", " 'weight_l1_penalty':'0.',\n", " 'weight_decay': 'phylogenetic_tree',\n", " },\n", " 'model_info': {\n", " 'decay': '0.001',\n", " 'loss': 'binary_crossentropy',\n", " 'lr': '0.01',\n", " 'metrics': 'binary_accuracy, sensitivity, specificity, gmeasure, auc',\n", " 'network_class': 'DeepBiomeNetwork',\n", " 'normalizer': 'normalize_minmax',\n", " 'optimizer': 'adam',\n", " 'reader_class': 'MicroBiomeClassificationReader',\n", " 'taxa_selection_metrics': 'sensitivity, specificity, gmeasure, accuracy',\n", " },\n", " 'test_info': {\n", " 'batch_size': 'None'\n", " }\n", "}" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "collapsed": true }, "outputs": [], "source": [ "prediction_path_info = {\n", " 'data_info': {\n", " 'count_list_path': resource_filename('deepbiome', 'tests/data/gcount_list.csv'),\n", " 'count_path': resource_filename('deepbiome', 'tests/data/count'),\n", " 'data_path': resource_filename('deepbiome', 'tests/data'),\n", " 'idx_path': resource_filename('deepbiome', 'tests/data/classification_idx.csv'),\n", " 'tree_info_path': resource_filename('deepbiome', 'tests/data/genus48_dic.csv'),\n", " 'x_path': '',\n", " 'y_path': 'classification_y.csv'\n", " },\n", " 'model_info': {\n", " 'model_dir': './example_result/',\n", " 'weight': 'weight.h5'\n", " }\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To predict the CV outputs from each fold, we set `change_weight_for_each_fold = True`. Also, we set `get_y=True` to get the paired output of each prediction too." ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|deepbiome.py:450] -----------------------------------------------------------------\n", "[root |INFO|deepbiome.py:480] -------1 th repeatition prediction start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:498] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_0.h5 \n", "[root |INFO|deepbiome.py:509] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:193] Prediction start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 136us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:198] Prediction end with time 0.03711438179016113!\n", "[root |INFO|deepbiome.py:513] Compute time : 0.6130969524383545\n", "[root |INFO|deepbiome.py:514] 1 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:480] -------2 th repeatition prediction start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:498] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_2\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_1.h5 \n", "[root |INFO|deepbiome.py:509] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:193] Prediction start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 271us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:198] Prediction end with time 0.07141661643981934!\n", "[root |INFO|deepbiome.py:513] Compute time : 0.6672861576080322\n", "[root |INFO|deepbiome.py:514] 2 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:480] -------3 th repeatition prediction start!----------------------------------\n", "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|deepbiome.py:498] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_3\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:61] Build Network\n", "[root |INFO|build_network.py:62] Optimizer = adam\n", "[root |INFO|build_network.py:63] Loss = binary_crossentropy\n", "[root |INFO|build_network.py:64] Metrics = binary_accuracy, sensitivity, specificity, gmeasure, auc\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result/weight_2.h5 \n", "[root |INFO|deepbiome.py:509] -----------------------------------------------------------------\n", "[root |INFO|build_network.py:193] Prediction start!\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "250/250 [==============================] - 0s 301us/step\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|build_network.py:198] Prediction end with time 0.07864809036254883!\n", "[root |INFO|deepbiome.py:513] Compute time : 0.7205522060394287\n", "[root |INFO|deepbiome.py:514] 3 fold computing end!---------------------------------------------\n", "[root |INFO|deepbiome.py:519] Total Computing Ended\n", "[root |INFO|deepbiome.py:520] -----------------------------------------------------------------\n" ] } ], "source": [ "prediction = deepbiome.deepbiome_prediction(log, prediction_network_info, prediction_path_info,\n", " num_classes = 1, number_of_fold=None, \n", " change_weight_for_each_fold = True, \n", " get_y=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We gathered the outputs from each fold. " ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": true }, "outputs": [], "source": [ "prediction = np.vstack(prediction)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since we set the option `get_y=True`, the output has the shape of `(n_samples, 2, n_classes)`. With this options, we can get the predictions of test set and the true output of each predictions.\n", "\n", "Now, we can calculate the test performance by the test predictions." ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|:4] Shape of the predict function ourput: (750, 2, 1)\n", "[root |INFO|:5] Shape of the prediction: (750, 1)\n", "[root |INFO|:6] Shape of the true_output for each prediction: (750, 1)\n" ] } ], "source": [ "predict_output = prediction[:,0]\n", "true_output = prediction[:,1]\n", "\n", "log.info('Shape of the predict function ourput: %s' % str(prediction.shape))\n", "log.info('Shape of the prediction: %s' % str(predict_output.shape))\n", "log.info('Shape of the true_output for each prediction: %s' % str(true_output.shape))" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|:1] CV accuracy: 0.719\n" ] } ], "source": [ "log.info('CV accuracy: %6.3f' % np.mean((predict_output >= 0.5) == true_output))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8. Load trained weight matrix\n", "\n", "The `deepbiome_get_trained_weight` function convert the trained weight `*.h5` saved from the `deepbiome_train` to a list of numpy arrays.\n", "In this exampe, the list has numpy array of weights from 6 layers. (`[genus to family, family to order, order to Class, class to phylum, phylum to output]`)" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|build_network.py:521] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:522] Read phylogenetic tree information from /DATA/home/muha/github_repos/deepbiome/deepbiome/tests/data/genus48_dic.csv\n", "[root |INFO|build_network.py:528] Phylogenetic tree level list: ['Genus', 'Family', 'Order', 'Class', 'Phylum']\n", "[root |INFO|build_network.py:529] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:537] Genus: 48\n", "[root |INFO|build_network.py:537] Family: 40\n", "[root |INFO|build_network.py:537] Order: 23\n", "[root |INFO|build_network.py:537] Class: 17\n", "[root |INFO|build_network.py:537] Phylum: 9\n", "[root |INFO|build_network.py:546] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:547] Phylogenetic_tree_dict info: ['Phylum', 'Order', 'Family', 'Class', 'Number', 'Genus']\n", "[root |INFO|build_network.py:548] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:558] Build edge weights between [ Genus, Family]\n", "[root |INFO|build_network.py:558] Build edge weights between [Family, Order]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Order, Class]\n", "[root |INFO|build_network.py:558] Build edge weights between [ Class, Phylum]\n", "[root |INFO|build_network.py:571] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:586] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:587] Build network based on phylogenetic tree information\n", "[root |INFO|build_network.py:588] ------------------------------------------------------------------------------------------\n", "[root |INFO|build_network.py:670] ------------------------------------------------------------------------------------------\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model_1\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input (InputLayer) (None, 48) 0 \n", "_________________________________________________________________\n", "l1_dense (Dense_with_tree) (None, 40) 1960 \n", "_________________________________________________________________\n", "l1_activation (Activation) (None, 40) 0 \n", "_________________________________________________________________\n", "l2_dense (Dense_with_tree) (None, 23) 943 \n", "_________________________________________________________________\n", "l2_activation (Activation) (None, 23) 0 \n", "_________________________________________________________________\n", "l3_dense (Dense_with_tree) (None, 17) 408 \n", "_________________________________________________________________\n", "l3_activation (Activation) (None, 17) 0 \n", "_________________________________________________________________\n", "l4_dense (Dense_with_tree) (None, 9) 162 \n", "_________________________________________________________________\n", "l4_activation (Activation) (None, 9) 0 \n", "_________________________________________________________________\n", "last_dense_h (Dense) (None, 1) 10 \n", "_________________________________________________________________\n", "p_hat (Activation) (None, 1) 0 \n", "=================================================================\n", "Total params: 3,483\n", "Trainable params: 3,483\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|:3] 5\n" ] } ], "source": [ "weight_path = '%s/%s' % (path_info['model_info']['model_dir'], 'weight_0.h5')\n", "trained_weight_list = deepbiome.deepbiome_get_trained_weight(log, network_info, path_info, num_classes=1, weight_path=weight_path)\n", "log.info(len(trained_weight_list))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "First weight between the `genus` and `family` layers has the shape of `(number of genus = 48, number of family = 40)`" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|:1] (48, 40)\n" ] } ], "source": [ "log.info(trained_weight_list[0].shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 9. Taxa selection performance\n", "\n", "If we know the true disease path, we can calculate the taxa selection performance by `deepbiome_taxa_selection_performance` funciton. First, we prepared the true weight list based on the true disease path. For each fold, we prepared 4 weights from the 5 layers (`[genus to family, family to order, order to Class, class to phylum]`). An example of the list of the true weights from each fold is as follow:" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|:2] (5, 4)\n" ] } ], "source": [ "true_tree_weight_list = np.load(resource_filename('deepbiome', 'tests/data/true_weight_list.npy'), allow_pickle=True)\n", "log.info(true_tree_weight_list.shape)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "The first weight between the genus and family layers for first epoch has the shape below:" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|:1] (48, 40)\n" ] } ], "source": [ "log.info(true_tree_weight_list[0][0].shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will calculate the taxa selection performance of the trained weight below:" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['./example_result//weight_0.h5',\n", " './example_result//weight_1.h5',\n", " './example_result//weight_2.h5']" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trained_weight_path_list = ['%s/weight_%d.h5' % (path_info['model_info']['model_dir'], i) for i in range(3)]\n", "trained_weight_path_list" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is the summary of the taxa selection accuracy of trained weights from each fold." ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "[root |INFO|readers.py:58] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:59] Construct Dataset\n", "[root |INFO|readers.py:60] -----------------------------------------------------------------------\n", "[root |INFO|readers.py:61] Load data\n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result//weight_0.h5 \n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result//weight_1.h5 \n", "[root |INFO|build_network.py:87] Load trained model weight at ./example_result//weight_2.h5 \n" ] } ], "source": [ "summary = deepbiome.deepbiome_taxa_selection_performance(log, network_info, path_info, num_classes=1, \n", " true_tree_weight_list=true_tree_weight_list, \n", " trained_weight_path_list = trained_weight_path_list)" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ModelPhyloTreeNo. true taxaNo. total taxaSensitivity_meanSensitivity_stdSpecificity_meanSpecificity_stdGmeasure_meanGmeasure_stdAccuracy_meanAccuracy_std
0./example_result/Genus31480.9460.0550.9830.0090.9640.0260.9830.008
1Family23400.9420.0200.9810.0020.9610.0110.9800.002
2Order9230.9630.0520.9620.0020.9620.0280.9620.004
3Class7170.9520.0670.9250.0100.9380.0320.9260.008
\n", "
" ], "text/plain": [ " Model PhyloTree No. true taxa No. total taxa Sensitivity_mean \\\n", "0 ./example_result/ Genus 31 48 0.946 \n", "1 Family 23 40 0.942 \n", "2 Order 9 23 0.963 \n", "3 Class 7 17 0.952 \n", "\n", " Sensitivity_std Specificity_mean Specificity_std Gmeasure_mean \\\n", "0 0.055 0.983 0.009 0.964 \n", "1 0.020 0.981 0.002 0.961 \n", "2 0.052 0.962 0.002 0.962 \n", "3 0.067 0.925 0.010 0.938 \n", "\n", " Gmeasure_std Accuracy_mean Accuracy_std \n", "0 0.026 0.983 0.008 \n", "1 0.011 0.980 0.002 \n", "2 0.028 0.962 0.004 \n", "3 0.032 0.926 0.008 " ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "summary" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 2 }