NSYSU Pioneers the Creative Use of Music for Painting
As the piano was played, a small green plant gradually grew on the screen. Professor Kwang-I Ying from the Department of Music and Professor Chi-Cheng Cheng from the Department of Mechanical and Electromechanical Engineering first creatively used music to paint by converting sound parameters (detecting the frequency, amplitude, and strength of a musical instrument) into visual points, lines, and surfaces through an algorithm. A specific sound in a piece of music triggered an animated presentation, thereby combining music with painting. In the future, this type of inter-disciplinary collaboration can be applied in interactive games, medical rehabilitation, and art therapy.
Using music to paint is a creative undertaking that involves the combination of digital technology and interactive art. A research team formed of members from the Department of Music and Department of Mechanical and Electromechanical Engineering at NSYSU composed a piece of music and developed a story, and then used a computer to draw 500 pictures, of which 31 were set to be triggered by a sound string. The mathematical method of fast Fourier transform was employed to recognize the sound string. Professor Chi-Cheng Cheng stated that “so long as a melody is played on the piano, animations such as a growing tree, thunder, and clouds floating are triggered.”
After half a year of research, the aforementioned research team presented its results. One member, Hao Wei Hsu, a fourth-year student in the Department of Mechanical and Electromechanical Engineering, was interested in audio engineering and organized a rock band. In his third year, he enrolled in a course titled “Special Topics in Mechanical and Electromechanical Engineering” coordinated by Professor Chi-Cheng Cheng and learned about the inter-disciplinary research team. He subsequently volunteered to participate in the collaboration. Hsu stated that for him, the greatest challenge of using music to paint was the audio recording quality because timbre varied between different pianos. In addition, microphone height and piano position can influence the quality of audio recording performed using a computer. Hsu stated that a player incorrectly playing a sound could prevent an animation from being triggered.
Professor Kwang-I Ying from the Department of Music asserted that the research team’s efforts constituted an innovative attempt. Regarding the integration of sounds and images, ready-made music coupled with images is typically played in a postproduction setting as opposed to a live setting. Even if an immediate visual effect occurs, an abstract plot rather than a narrative plot is often presented. In such situations, a musical team is responsible for musical composition and performance and a technical team is responsible for detecting and collecting musical elements and designing interactive programs. Each collected sound parameter resembles a brushstroke that represents one element of a painting. At various stages, a piece of music evokes various emotions, produces visual effects, and provides artistic connotations and entertainment.
Using music to paint is a creative undertaking that involves the combination of digital technology and interactive art. A research team formed of members from the Department of Music and Department of Mechanical and Electromechanical Engineering at NSYSU composed a piece of music and developed a story, and then used a computer to draw 500 pictures, of which 31 were set to be triggered by a sound string. The mathematical method of fast Fourier transform was employed to recognize the sound string. Professor Chi-Cheng Cheng stated that “so long as a melody is played on the piano, animations such as a growing tree, thunder, and clouds floating are triggered.”
After half a year of research, the aforementioned research team presented its results. One member, Hao Wei Hsu, a fourth-year student in the Department of Mechanical and Electromechanical Engineering, was interested in audio engineering and organized a rock band. In his third year, he enrolled in a course titled “Special Topics in Mechanical and Electromechanical Engineering” coordinated by Professor Chi-Cheng Cheng and learned about the inter-disciplinary research team. He subsequently volunteered to participate in the collaboration. Hsu stated that for him, the greatest challenge of using music to paint was the audio recording quality because timbre varied between different pianos. In addition, microphone height and piano position can influence the quality of audio recording performed using a computer. Hsu stated that a player incorrectly playing a sound could prevent an animation from being triggered.
Professor Kwang-I Ying from the Department of Music asserted that the research team’s efforts constituted an innovative attempt. Regarding the integration of sounds and images, ready-made music coupled with images is typically played in a postproduction setting as opposed to a live setting. Even if an immediate visual effect occurs, an abstract plot rather than a narrative plot is often presented. In such situations, a musical team is responsible for musical composition and performance and a technical team is responsible for detecting and collecting musical elements and designing interactive programs. Each collected sound parameter resembles a brushstroke that represents one element of a painting. At various stages, a piece of music evokes various emotions, produces visual effects, and provides artistic connotations and entertainment.
Click Num:
Share