Thursday, April 29, 2010
Monday, April 26, 2010
Milestone I draft
As milestone one is approaching, I have a draft of its description.
Existing systems may be listed here (...)
Musical qualities I want to extract from these elements consist of:
At this stage I need to start thinking about possible responses/transformations of the audio material already, along with coding the gesture recognition.
Another problem is phrase recognition, which will be manual at the beginning (keyboard/pedal input). However I plan to use mixture of pitch and rhythm values to make system split the gestures and recognize phrases automatically.
- finish research on the existing systemsDuring my research I found many projects of various people doing various real-time performance / improvisation related tasks. However, I did not find a ready to use system, that would behave in a way I want it to behave - listen to the live input and respond in a meaningful musical way.
Existing systems may be listed here (...)
- collect information on the gestures to be analyzed by the systemI plan to extract musical information from the audio input using five analysis techniques:
- onset detection (rhythm)
- pitch tracking (pitch)
- amplitude (loudness)
- beat tracking (meter)
- FFT (timbre).
Musical qualities I want to extract from these elements consist of:
- rhythmic speed
- rhythm regularity (this is where beat tracking can help)
- pitch range and character (spread pitches vs. linear structure)
- loudness (RMS)
- discrete pitch and rhythm values (may be usable to model computer response)
- pitch class set (mode/harmonic structure)
- beat detection (it would be great to have some kind of "synchronized" response from time to time).
At this stage I need to start thinking about possible responses/transformations of the audio material already, along with coding the gesture recognition.
Another problem is phrase recognition, which will be manual at the beginning (keyboard/pedal input). However I plan to use mixture of pitch and rhythm values to make system split the gestures and recognize phrases automatically.
- start gesture recognition implementationImplementation hasn't been started yet.
Computer Music Journal articles
Another search in Computer Music Journal archives provided valuable papers on gesture recognition.
A Multimodal System for Gesture Recognition in Interactive Music Performance
Computer Music Journal, 33:4, pp. 69–82, Winter 2009
This paper describes a system, that incorporates not only musical gesture recognition, but also video tracking of the performer, used mainly to provide natural and expressive score-following.
Expressive Concatenative Synthesis by Reusing Samples from Real Performance Recordings
Computer Music Journal, 33:4, pp. 23–42, Winter 2009
In this paper I found systematic classification of certain values used to analyze musical qualities. Here used for re-synthesis purposes, seem to fit the method I want to use in my system for gestures recognition.
Feature Set Patterns in Music
Computer Music Journal, 32:1, pp. 60–70, Spring 2008
This paper provides some background on pattern discovery algorithms. I might not get that far, but who knows.
Above mentioned articles can be obtained at MIT Press website from UW campus.
A Multimodal System for Gesture Recognition in Interactive Music Performance
Computer Music Journal, 33:4, pp. 69–82, Winter 2009
This paper describes a system, that incorporates not only musical gesture recognition, but also video tracking of the performer, used mainly to provide natural and expressive score-following.
Expressive Concatenative Synthesis by Reusing Samples from Real Performance Recordings
Computer Music Journal, 33:4, pp. 23–42, Winter 2009
In this paper I found systematic classification of certain values used to analyze musical qualities. Here used for re-synthesis purposes, seem to fit the method I want to use in my system for gestures recognition.
Feature Set Patterns in Music
Computer Music Journal, 32:1, pp. 60–70, Spring 2008
This paper provides some background on pattern discovery algorithms. I might not get that far, but who knows.
Above mentioned articles can be obtained at MIT Press website from UW campus.
Thursday, April 22, 2010
Gestures and patterns analysis/recognition
Beside researching existing projects for computer-aided improvisation/accompaniment, I was thinking about some concepts regarding how my system will behave getting certain input (sound)m that is how could musical gestures recognition work.
Today in class brainstorming with Juan and Stelios clarified and directed my considerations into a more narrow and more realistic area.
Things, that I need to take care of anyway involve phrase recognition, pitch and loudness analysis, as well as some kind of rhythm structure examination. Phrase recognition would involve onset detection as well as timing information analysis, but may end up being triggered manually (e.g. foot switch). It's not crucial to make it happen automatically at that point. Inside each phrase elements mentioned above (pitch, loudness, rhythm) would of course change... Initially I was thinking about some kind of envelope extraction for their values, but that seems to be too complicated for further manipulation at this point. Thus I want to end up with single "meta" parameter for each of those element, for example: high pitches or low pitches, big pitch deviation or small pitch deviation (more static structure), faster or slower rhythm etc.
Beside thinking about using envelopes for presenting these parameters across given phrase, I was thinking about really fancy idea (I know, fancy ideas have serious potential being useless) of some kind of machine learning system. It would analyze and juxtapose actually played phrase with the one played right before, and with the one that computer played/generated right before. That way system would develop a database of more common and less common "responses" or relationships between phrases. However effective implementation of such mechanism would probably involve neural networks... and thus is beyond the scope of this 10-week project. Beside the problem of machine learning mechanisms there is a question: how would computer interpret those data? Especially that transformation of the input material only is considered at this point (no synthesis). Possible answer is that such data could be used to create actually synthesized and more autonomous system response. But that's an idea to be saved for much later I believe.
Today in class brainstorming with Juan and Stelios clarified and directed my considerations into a more narrow and more realistic area.
Things, that I need to take care of anyway involve phrase recognition, pitch and loudness analysis, as well as some kind of rhythm structure examination. Phrase recognition would involve onset detection as well as timing information analysis, but may end up being triggered manually (e.g. foot switch). It's not crucial to make it happen automatically at that point. Inside each phrase elements mentioned above (pitch, loudness, rhythm) would of course change... Initially I was thinking about some kind of envelope extraction for their values, but that seems to be too complicated for further manipulation at this point. Thus I want to end up with single "meta" parameter for each of those element, for example: high pitches or low pitches, big pitch deviation or small pitch deviation (more static structure), faster or slower rhythm etc.
Beside thinking about using envelopes for presenting these parameters across given phrase, I was thinking about really fancy idea (I know, fancy ideas have serious potential being useless) of some kind of machine learning system. It would analyze and juxtapose actually played phrase with the one played right before, and with the one that computer played/generated right before. That way system would develop a database of more common and less common "responses" or relationships between phrases. However effective implementation of such mechanism would probably involve neural networks... and thus is beyond the scope of this 10-week project. Beside the problem of machine learning mechanisms there is a question: how would computer interpret those data? Especially that transformation of the input material only is considered at this point (no synthesis). Possible answer is that such data could be used to create actually synthesized and more autonomous system response. But that's an idea to be saved for much later I believe.
Monday, April 19, 2010
Mechanisms of perceptual organization
Stelios pointed me to a Masters Thesis about integrating analytical with compositional processes, written by Pandelis Diamantides at the Institute of Sonology, Royal Conservatory in Hague. It provides lots of useful information on musical gestures recognition.
http://sonology.org/NL/thesis-
http://sonology.org/NL/thesis-
Tuesday, April 13, 2010
MIR
I'll probably be only touching Music Information Retrieval area, but it's interesting to see some research that may be potentially helpful for my project, like the winning paper of the ISMIR 2008 Best Student Paper Award:
http://ismir2008.ismir.net/papers/ISMIR2008_261.pdf.
ISMIR stands for The International Society for Music Information Retrieval.
http://ismir2008.ismir.net/papers/ISMIR2008_261.pdf.
ISMIR stands for The International Society for Music Information Retrieval.
Saturday, April 10, 2010
Search results
I found couple projects dealing with interactive performance. So far it seems they may convince me to abandon the idea of creating another live performance tool.
http://www.informatics.sussex.ac.uk/users/nc81/index.html - Nick Collins website with various machine listening software; especially interesting interactive projects
http://www.sfsound.org/matt.html - Matt Ingalls website (clarinetist, composer, improviser, and computer musician)
http://www.informatics.sussex.ac.uk/users/nc81/index.html - Nick Collins website with various machine listening software; especially interesting interactive projects
http://www.sfsound.org/matt.html - Matt Ingalls website (clarinetist, composer, improviser, and computer musician)
Tuesday, April 6, 2010
The first idea about the project
My initial idea for the project was to create a performance tool, that would interact in a meaningful musical way with my own performance. It would be similar to the situation, where members of a chamber group or band interact with each other while they improvise.
As I don't think that I'd manage to both create the system and compose a piece for it, I consider this project to be research-based.
The first research that needs to be done is to find similar projects and to see if any of them already fulfills my expectations and thus allows me to use it, instead of developing my own system.
I'm still considering moving to a composition-based project instead, as I'm not sure if the scope of my proposition isn't too big for a given time frame and if similar tool isn't already developed. Regarding my possible composition I would like to learn and use various FFT techniques, as this is the area I don't know much about yet.
As I don't think that I'd manage to both create the system and compose a piece for it, I consider this project to be research-based.
The first research that needs to be done is to find similar projects and to see if any of them already fulfills my expectations and thus allows me to use it, instead of developing my own system.
I'm still considering moving to a composition-based project instead, as I'm not sure if the scope of my proposition isn't too big for a given time frame and if similar tool isn't already developed. Regarding my possible composition I would like to learn and use various FFT techniques, as this is the area I don't know much about yet.
Subscribe to:
Posts (Atom)