How to extract labels from not unnotated waves ?

[gg4u]

I have a db that is annotated with starts and ends of samples in vocalizations,
and the units of samples are probably syllables - not single notes.
(They are bats, not birds).

1. Does pykanto help in the step to label those ?
2. Can you describe a possible approach?

Below some temptative answers, I'd love to hear your perspectives and guidance (I am MSc student):

--

Looking at:

/pykanto/data/raw/storm-petrel/XC46092.xml

It seems pykanto support automatic label annotation: the labels are None, only the intervals are known. For exploratory purpose, I tried :

DATASET_ID = "storm-petrel"
DIRS = pykanto_data(dataset=DATASET_ID)
# ---------
params = Parameters() # Using default parameters for simplicity, which you should't!
dataset = KantoData(DIRS, parameters=params, overwrite_dataset=True)
dataset.data.head(3)

but error was found:

FileNotFoundError: There are no .wav files in directory [abs_path]/pykanto/pykanto/data/segmented/storm-petrel

for some reason, it is not looking at the folder "raw", differently than the example for "GREAT-TIT".

2. How could a possible approach for unsupervised automatic labeling could be ?

Considering that I have data of vocalizations with the following annotations:

• context
• emitter

Some possible approaches could be:
a. to compare the spectrograms of the syllables from the same emitter, and in same context, and label them accordingly. Expected results: classify similar syllables, constraint by emitter and context.

b. to project on UMAP, and manually label those groups of syllables appearing close.
In such an option, I wonder if I could project All of the vocalizations directly, for all emitters and contexts. Expected challenges: let me say with an example, if I say "apple" and "pinapple" when I am "hungry", and you also say "apple" and "pinapple" when you are "hungry", my two utterances may appear closer than yours if the emitter features weights more than context, or "apple" and "apple" may appear closer if the context weigths more than emitte features. How to address the issue and extract the utterance "apple" and "pinapple" as unique "vocabulary" for both ?

REading that pykanto may be somewhat similar to works of vak, can you explain how would it handle this problem and, if so, types of model / Neural Networks adopted to label syllables ?

[nilomr]

I have a db that is annotated with starts and ends of samples in vocalizations,
and the units of samples are probably syllables - not single notes.
(They are bats, not birds). Does pykanto help in the step to label those ?

As described in this article, (p. 3, limitations section) the labelling method included in pykanto (and any dim reduction +clustering method, for that matter) will work well when there are 'real' categorical differences between vocalisation types, i.e. they are easily separable in principle.

If you set song_level = False in Parameters() (the default) and import your dataset including onset and offset times you could then run the next steps (i.e. .segment_into_units(), etc, see docs, and see what things look like on the interactive app. I can't say much more than that since I'm not familiar with your dataset, but my best guess is that bat vocalisations will be fairly continuous. If this is the case, any fine-grained class labels that you assign will be pretty arbitrary and possibly misleading.

How could a possible approach for unsupervised automatic labeling could be ?
Considering that I have data of vocalizations with the following annotations:

• context
• emitter

This is more of a research question / general methods problem, and I'm afraid I don't have the time to think about this just now. Having said that, I would be happy to try to answer more concrete questions that you might have either now or as you work on your project.

Reading that pykanto may be somewhat similar to works of vak, can you explain how would it handle this problem and, if so, types of model / Neural Networks adopted to label syllables?

In brief, pykanto uses UMAP and HDBSCAN (see e.g. Sainburg et al., 2020) to conservatively find clusters that can then be manually reviewed using an interactive app. If you were to annotate a subset of your data in this way, you could then train a classifier to label the rest of the data. This is what NNs like tweetynet can do (vak which you mention was written to benchmark that architecture). Again, whether this approach would work depends on your specific research question and, more importantly, the existence of real categories or differences in the data. For example, you might be able to predict which context a call was produced in, or the ID of the producer, but still not be able to clearly assign vocalisations to discrete acoustic classes if the calls are graded.

[nilomr]

Finally,

FileNotFoundError: There are no .wav files in directory [abs_path]/pykanto/pykanto/data/segmented/storm-petrel

Reason: see the docs for the KantoData class, first argument. You need to have segmented the raw files first before building the dataset, in this case using existing onset/offset information present in the metadata. This vignette shows what that would look like with the storm-petrel dataset: storm-petrel vignette, see also segmenting vocalisations.

[gg4u]

Hi Nilo,

first of all, thanks for you help and availability might I need some debrief in my journey :)

Insightful to keep in mind "continuity" Vs more discrete sounds; I will split this thread and move discussion types questions into other threads.

On aim to replicate tutorials, could you please clarify the following will automatically download datasets for storm-petrel or bengalese finch, as for great tit ?

DATASET_ID = "STORM-PETREL"
DIRS = pykanto_data(dataset=DATASET_ID)
# ---------
params = Parameters() # Using default parameters for simplicity, which you should't!
dataset = KantoData(DIRS, parameters=params, overwrite_dataset=True)
dataset.data.head(3)

My confusion is that I see the folder /raw/STORM-PETREL , so I thought raw data is there, and is not yet segmented.

But the error refers to segmented data.

FileNotFoundError: There are no .wav files in directory /Users/FOLDER/Sites/pykanto/pykanto/data/segmented/storm-petrel

Instead:

DATASET_ID = "GREAT_TIT"
DIRS = pykanto_data(dataset=DATASET_ID)
# ---------
params = Parameters() # Using default parameters for simplicity, which you should't!
dataset = KantoData(DIRS, parameters=params, overwrite_dataset=True)
dataset.data.head(3)

will work.

• How to use the tutorials for preinstalled datasets, storm-petrel or bengalese finch ?

https://nilomr.github.io/pykanto/_build/html/contents/basic-workflow.html

[nilomr]

Hi Luigi,

[...] I see the folder /raw/STORM-PETREL, so I thought raw data is there, and is not yet segmented.

That is accurate. The package includes some sample data, some are already segmented (in this sense: DOCS), some aren't. This is intentional, both for testing purposes and so that users can see how to work with different types of datasets.
The long 'raw' recordings need to be segmented before building a KantoData dataset, see my previous comment:

see the docs for the KantoData class, first argument. You need to have segmented the raw files first before building the dataset, in thThis vignette shows what that would look like with the storm-petrel dataset: storm-petrel vignette, see also segmenting vocalisations.is case using existing onset/offset information present in the metadata.

As to your question "_How to use the tutorials for preinstalled datasets, storm-petrel or bengalese finch?", please see my message above:

This vignette shows what that would look like with the storm-petrel dataset: storm-petrel vignette, see also segmenting vocalisations.

Hope that helps!

[nilomr]

Tasks

• Include a clearer description of the sample datasets in the docs.

[gg4u]

Hi!

Now I figured it out.

The missing steps was in the 3-rd hidden cell:
https://nilomr.github.io/pykanto/_build/html/contents/segmenting-vocalisations.html

Feedback:

If useful, in my opinion you may want to explain that raw data can / must be segmented already in the first tutorial:
https://nilomr.github.io/pykanto/_build/html/contents/basic-workflow.html
and explicitly point reference at the hidden cell :
segment_files_parallel()

It was not clear to me to understand what was going on, simply looking at the classes and parameters.

When running segment_files_parallel() function, I met and solved this error that may affect other Mac users as me:

Error was related to tqdm: 'tqdm' object has no attribute 'disable’ and found comments in this repo:

3fon3fonov/exostriker#80

In the official tqdm repo, they suggest to bypass the issue with a wrapper:
tqdm/tqdm#487 (comment)

However, I was able to solve it by checking out install again with conda:

conda install tqdm

Conda said it was correctly installed, but (mysteriously) the issue was solved.

Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

My 2 cents : I suspect tqdm may have previously installed via pip and may be useful to ensure packages are installed via conda, not pip. But not sure about this...

Thanks again Nilo!

[nilomr]

Thanks, Luigi. Would you mind opening a separate issue about this tqdm error? I work on Linux and it's quite difficult for me to get my hands on a machine running macOS for testing, as you can probably notice! 😅