Issue with yaml library

[btemperton]

Hi there,

I'm trying to get medaka working using an RTX2080 GTi card that needs cuda 10.0. I can run tensorflow and keras in the medaka virtualenv, so that part of it is working. The conda install uses the wrong libraries for cuda 10 (and fails looking for libcublas.9.0), so I had to install it using from git. For that I also had to install pyyaml using pip. However, now when medaka runs, it throws an error (file attached):

yaml.constructor.ConstructorError: while constructing a Python instance
expected a class, but found <class 'builtin_function_or_method'>
  in "<unicode string>", line 3, column 5:
      - !!python/object/apply:numpy.core ... 

Any advice on how I can fix this?
Thanks,
Ben
medaka.log

[cjw85]

Hi @btemperton,

Firstly the conda issue is largely beyond our control: the conda-forge tensorflow package using google's prebuilt binaries which are linked to CUDA 9 and cuDNN 7.

There's a bit of history with pyaml (yaml/pyyaml#193), that looks to have been resolved today! pyyaml has been neglected from our requirement.txt as a direct dependency. Can you try installing pyyaml == 3.13? That is the version I have here.

From your log it looks like you are using python 3.7, which is not supported (I do not immediately recall why, but it is what we have listed in the package meta data on pypi).

[btemperton]

Ok, after a morning of playing around I finally got it working using the following:

## Remove previous attempt
cd ~/tools
conda remove -n medaka --all
rm -rf medaka

## create a conda environment
conda create -n medaka python=3.6 pyyaml=3.13 samtools=1.3.1 minimap2=2.11
conda activate medaka

##install medaka into it
git clone https://github.com/nanoporetech/medaka.git
cd medaka

#change tensorflow requirements in requirements.txt to tensorflow==1.13.1 and add pyyaml==3.13
make install
. ./venv/bin/activate

[damientully]

I have the same issue when installing with pip

Checking program versions
Program Version Required Pass
bgzip 1.5 1.3.1 True
minimap2 2.15 2.11 True
samtools 1.5 1.3.1 True
tabix 1.5 1.3.1 True
Aligning basecalls to draft
Constructing minimap index.
[M::mm_idx_gen::0.0040.63] collected minimizers
[M::mm_idx_gen::0.0050.72] sorted minimizers
[M::main::0.0060.76] loaded/built the index for 1 target sequence(s)
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
[M::mm_idx_stat::0.0060.80] distinct minimizers: 1730 (99.71% are singletons); average occurrences: 1.003; average spacing: 5.308
[M::main] Version: 2.15-r906-dirty
[M::main] CMD: minimap2 -I 16G -x map-ont --MD -d /home/dct7/medaka/bin/AC347.contigs.fasta.mmi /home/dct7/medaka/bin/AC347.contigs.fasta
[M::main] Real time: 0.008 sec; CPU: 0.005 sec; Peak RSS: 0.004 GB
[samfaipath] build FASTA index...
[M::main::0.0030.88] loaded/built the index for 1 target sequence(s)
[M::mm_mapopt_update::0.0030.83] mid_occ = 3
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 1
[M::mm_idx_stat::0.0040.82] distinct minimizers: 1730 (99.71% are singletons); average occurrences: 1.003; average spacing: 5.308
[M::worker_pipeline::20.9120.74] mapped 11143 sequences
[M::main] Version: 2.15-r906-dirty
[M::main] CMD: minimap2 -x map-ont --MD -t 1 -a /home/dct7/medaka/bin/AC347.contigs.fasta.mmi /n/scratch2/dct7/Run11_guppy_v2.3.5_porechop/BC05.fastq
[M::main] Real time: 20.918 sec; CPU: 15.519 sec; Peak RSS: 0.094 GB
Running medaka consensus
Using TensorFlow backend.
[11:22:50 - Predict] Processing region(s): tig00000002:0-9209
/home/dct7/medaka/lib/python3.6/site-packages/medaka/datastore.py:131: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
return {g: yaml.load(self.fh[g][()]) for g in groups if g in self.fh}
Traceback (most recent call last):
File "/home/dct7/medaka/bin/medaka", line 10, in
sys.exit(main())
File "/home/dct7/medaka/lib/python3.6/site-packages/medaka/medaka.py", line 261, in main
args.func(args)
File "/home/dct7/medaka/lib/python3.6/site-packages/medaka/inference.py", line 526, in predict
meta = ds.meta
File "/home/dct7/medaka/lib/python3.6/site-packages/medaka/datastore.py", line 51, in meta
self._meta = self._load_metadata()
File "/home/dct7/medaka/lib/python3.6/site-packages/medaka/datastore.py", line 131, in _load_metadata
return {g: yaml.load(self.fh[g][()]) for g in groups if g in self.fh}
File "/home/dct7/medaka/lib/python3.6/site-packages/medaka/datastore.py", line 131, in
return {g: yaml.load(self.fh[g][()]) for g in groups if g in self.fh}
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/init.py", line 114, in load
return loader.get_single_data()
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 43, in get_single_data
return self.construct_document(node)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 47, in construct_document
data = self.construct_object(node)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 94, in construct_object
data = constructor(self, tag_suffix, node)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 612, in construct_python_object_apply
args = self.construct_sequence(node, deep=True)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 122, in construct_sequence
for child in node.value]
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 122, in
for child in node.value]
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 99, in construct_object
for dummy in generator:
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 404, in construct_yaml_map
value = self.construct_mapping(node)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 210, in construct_mapping
return super().construct_mapping(node, deep=deep)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 131, in construct_mapping
key = self.construct_object(key_node, deep=deep)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 92, in construct_object
data = constructor(self, node)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 504, in construct_python_tuple
return tuple(self.construct_sequence(node))
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 122, in construct_sequence
for child in node.value]
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 122, in
for child in node.value]
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 94, in construct_object
data = constructor(self, tag_suffix, node)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 624, in construct_python_object_apply
instance = self.make_python_instance(suffix, node, args, kwds, newobj)
File "/home/dct7/medaka/lib/python3.6/site-packages/yaml/constructor.py", line 570, in make_python_instance
node.start_mark)
yaml.constructor.ConstructorError: while constructing a Python instance
expected a class, but found <class 'builtin_function_or_method'>
in "", line 3, column 5:
- !!python/object/apply:numpy.core ...
^
Failed to run medaka consensus.

[cjw85]

We are updating medaka to use pyyaml 5.1, there will be a release shortly.

[cjw85]

v0.6.2 is available from github and pypi. bioconda will follow.

[cjw85]

@btemperton
To my knowledge you shouldn't need to use CUDA 10. We have not tested medaka with tensorflow-gpu==1.13.1 (which is linked to CUDA 10). It is generally possible to install multiple versions of the CUDA libraries.

[btemperton]

The RTX2080 GTI drivers from Nvidia don't support CUDA <10, sadly.

[mbhall88]

Am seeing the following error using the latest biocontainers image (which is built from the bioconda recipe).

Using TensorFlow backend.
[15:45:05 - Predict] Processing region(s): contig_1:0-4420730
Traceback (most recent call last):
  File "/usr/local/bin/medaka", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/medaka/medaka.py", line 350, in main
    args.func(args)
  File "/usr/local/lib/python3.6/site-packages/medaka/inference.py", line 538, in predict
    meta = ds.meta
  File "/usr/local/lib/python3.6/site-packages/medaka/datastore.py", line 85, in meta
    self._meta = self._load_metadata()
  File "/usr/local/lib/python3.6/site-packages/medaka/datastore.py", line 170, in _load_metadata
    return {g: yaml.unsafe_load(self.fh[g][()]) for g in groups if g in self.fh}
  File "/usr/local/lib/python3.6/site-packages/medaka/datastore.py", line 170, in <dictcomp>
    return {g: yaml.unsafe_load(self.fh[g][()]) for g in groups if g in self.fh}
AttributeError: module 'yaml' has no attribute 'unsafe_load'

Example of how I ran medaka

medaka="docker://quay.io/biocontainers/medaka:0.7.1--py36h2b5150b_1"
singularity exec "$medaka" medaka_consensus [options]