Hi all, I have been experimenting with using NatureLM-audio/BEANS-Zero for call-type classification on the xeno-canto dataset like is detailed in the paper. I have successfully performed the zero-shot call-type classification, but I am wondering how I should determine the accuracy of the model. For example, if I have a 30-second audio clip that is being classified for call-type, this clip is now split into 3 sub-clips (with the default window of 10 seconds). If two of the sub-clips classify as “song” and the last as “call”, should I consider that the model predicted “song”? Or is there a way to identify where the call is located within the clip to then take the classification of the corresponding sub-clip call-type. On the other hand, should I just designate the sliding window to be as large as the longest audio clip in the dataset to avoid the issue? Or is this automatically handled by either NatureLM-audio or BEANS-Zero?
Hi CarterWebb,
NatureLM-audio unfortunately doesn’t provide “strong” labels in its output, i.e. it can’t tell you when in a 30 sec clip a song or call occurred, just whether it did. So yes, for a sliding window approach is required.
The evaluation of NatureLM on the beans zero dataset was limited to a max 10 sec (cropping). We haven’t yet defined the right metric to compute over sliding windows. You could consider taking a majority vote method to determine the final output (num songs > num call detections = song).