Modeling Music and Code Knowledge to Support a Co-creative AI Agent for Education

Abstract: EarSketch is an online environment for learning introductory computing concepts through code-driven, sample-based music production. This paper details the design and implementation of a module to perform code and music analyses on projects on the EarSketch platform. This analysis module combines inputs in the form of symbolic metadata, audio feature analysis, and user code to produce comprehensive models of user projects. The module performs a detailed analysis of the abstract syntax tree of a user's code to model use of computational concepts. It uses music information retrieval (MIR) and symbolic metadata to analyze users' musical design choices. These analyses produce a model containing users' coding and musical decisions, as well as qualities of the algorithmic music created by those decisions. The models produced by this module will support future development of CAI, a Co-creative Artificial Intelligence. CAI is designed to collaborate with learners and promote increased competency and engagement with topics in the EarSketch curriculum. Our module combines code analysis and MIR to further the educational goals of CAI and EarSketch and to explore the application of multimodal analysis tools to education.

Modeling Music and Code Knowledge to Support a Co-creative AI Agent for Education

Jason Smith, Erin Truesdell, Jason Freeman, Brian Magerko, Kristy Boyer, Tom Mcklin

Comments

Similar Papers

Play to Grade: Testing Coding Games as Classifying Markov Decision Process

Allen Nie, Emma Brunskill, Chris Piech

Keywords Abstract Paper

reinforcement learning and planning

Nonintrusive-sensing and reinforcement-learning based adaptive personalized music recommendation

Daocheng Hong, Yang Li, Qiwen Dong

Keywords Abstract Paper

adaptive playlist recommendation, musical preference learning, reinforcement learning, nonintrusive sensing

TangibleCircuits: An Interactive 3D Printed Circuit Education Tool for People with Visual Impairments

Josh Davis, Te-Yen Wu, Bo Shi and Hanyi Lu, Athina Panotopoulou, Emily Whiting, Xing-Dong Yang

Keywords Abstract Paper

tangible user interfaces, universal design, accessibility, circuit prototyping, education tools

An Explicitly Relational Neural Network Architecture

Murray Shanahan, Kyriacos Nikiforou, Antonia Creswell and Christos Kaplanis, David Barrett, Marta Garnelo

Keywords Abstract Paper

Deep Learning - General

Lifelong Learning of Compositional Structures

Jorge Mendez, ERIC EATON

Keywords Abstract Paper

lifelong learning, compositional learning, modular networks, continual learning

On understanding data worker interaction behaviors

Lei Han, Tianwa Chen, Gianluca Demartini and Marta Indulska, Shazia Sadiq

Keywords Abstract Paper

interaction behavior, search pattern, data curation

Giving Voice to Silent Data: Designing with Personal Music Listening History

Jordan Wirfs-Brock, Sarah Mennicken, Jennifer Thom

Keywords Abstract Paper

voice assistants, co-design, participatory design, personal informatics, music, speculative design

Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing

Xilun Chen, Asish Ghoshal, Yashar Mehdad and Luke Zettlemoyer, Sonal Gupta

Keywords Abstract Paper

task-oriented parsing, low-resource adaptation, generalization, virtual assistants

PointAugment: An Auto-Augmentation Framework for Point Cloud Classification

Ruihui Li, Xianzhi Li, Pheng-Ann Heng, Chi-Wing Fu

Keywords Abstract Paper

auto-augmentation framework, point cloud processing, sample-aware, jointly optimizing, classification

What is AI Literacy? Competencies and Design Considerations

Duri Long, Brian Magerko

Keywords Abstract Paper

ai literacy, ai education, ai for k-12, artificial intelligence, machine learning, computing education

Understanding the Relationship between Interactions and Outcomes in Human-in-the-Loop Machine Learning

Yuchen Cui, Pallavi Koppol, Henny Admoni and Scott Niekum, Reid Simmons, Aaron Steinfeld, Tesca Fitzgerald

Keywords Abstract Paper

Humans and AI, General

Solving Compositional Reinforcement Learning Problems via Task Reduction

Yunfei Li, Yilin Wu, Huazhe Xu and Xiaolong Wang, Yi Wu

Keywords Abstract Paper

reinforcement learning, task reduction, compositional task, sparse reward, imitation learning

The Rhythmic Dictator: Does Gamification of Rhythm Dictation Exercises Help?

Matevž Pesek, Lovro Suhadolnik, Peter Šavli, Matija Marolt

Keywords Abstract Paper

Applications, Music training and education, Human-centered MIR, User-centered evaluation

Muspy: a Toolkit for Symbolic Music Generation

Hao-Wen Dong, Ke Chen, Julian McAuley, Taylor Berg-Kirkpatrick

Keywords Abstract Paper

MIR fundamentals and methodology, Symbolic music processing, Applications, Music composition, performance, and production, Domain knowledge, Machine learning/Artificial intelligence for music, Representations of music, Evaluation, datasets, and reproducibility, Evaluation metrics, Reproducibility

Knowledge-Guided Object Discovery with Acquired Deep Impressions

Jinyang Yuan, Bin Li, Xiangyang Xue

Keywords Abstract Paper

Dmelodies: a Music Dataset for Disentanglement Learning

Ashis Pati, Siddharth Kumar Gururani, Alexander Lerch

Keywords Abstract Paper

Domain knowledge, Machine learning/Artificial intelligence for music, Representations of music, Evaluation, datasets, and reproducibility, Novel datasets and use cases, Reproducibility, MIR fundamentals and methodology, Symbolic music processing

RL Unplugged: A Collection of Benchmarks for Offline Reinforcement Learning

CAGLAR Gulcehre, Ziyu Wang, Alexander Novikov and Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Li, Mohammad Norouzi, Matthew Hoffman, Nicolas Heess, Nando de Freitas

Keywords Abstract Paper

Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal, Joel Lehman and Kenneth Stanley, Jeffrey Clune

Keywords Abstract Paper

Transfer, Multitask and Meta-learning

CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraints

Anselm Paulus, Michal Rolinek, Vit Musil and Brandon Amos, Georg Martius

Keywords Abstract Paper

Optimization, Combinatorial Optimization

Human-AI Co-creation in Songwriting

Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex and Monica Dinculescu, Carrie Cai

Keywords Paper

Keywords Paper

Josh Davis, Te-Yen Wu, Bo Shi and
Hanyi Lu, Athina Panotopoulou, Emily Whiting, Xing-Dong Yang

Keywords Paper

Murray Shanahan, Kyriacos Nikiforou, Antonia Creswell and
Christos Kaplanis, David Barrett, Marta Garnelo

Keywords Paper

Keywords Paper

Lei Han, Tianwa Chen, Gianluca Demartini and
Marta Indulska, Shazia Sadiq

Keywords Paper

Keywords Paper

Xilun Chen, Asish Ghoshal, Yashar Mehdad and
Luke Zettlemoyer, Sonal Gupta

Keywords Paper

Keywords Paper

Keywords Paper

Yuchen Cui, Pallavi Koppol, Henny Admoni and
Scott Niekum, Reid Simmons, Aaron Steinfeld, Tesca Fitzgerald

Keywords Paper

Yunfei Li, Yilin Wu, Huazhe Xu and
Xiaolong Wang, Yi Wu

Keywords Paper

Keywords Paper

Keywords Paper

Keywords Paper

Keywords Paper

CAGLAR Gulcehre, Ziyu Wang, Alexander Novikov and
Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Li, Mohammad Norouzi, Matthew Hoffman, Nicolas Heess, Nando de Freitas

Keywords Paper

Felipe Petroski Such, Aditya Rawal, Joel Lehman and
Kenneth Stanley, Jeffrey Clune

Keywords Paper

Anselm Paulus, Michal Rolinek, Vit Musil and
Brandon Amos, Georg Martius

Keywords Paper

Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex and
Monica Dinculescu, Carrie Cai

Keywords Paper

Keywords Paper

Guillaume Klein, Dakun Zhang, Clément Chouteau and
Josep Crego, Jean Senellart

Keywords Paper

Qing Cui, Qitao Shi, Hao Qian and
Caizhi Tang, Xixi Li, Yiming Zhao, Tao Jiang, Longfei Li, Jun Zhou

Keywords Paper

Jon Gauthier, Jennifer Hu, Ethan Wilcox and
Peng Qian, Roger Levy

Keywords Paper

Qian Liu, Shengnan An, Jian-Guang Lou and
Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, Dongmei Zhang

Keywords Paper

Ioannis Petros Samiotis, Sihang Qiu, Andrea Mauri and
Cynthia C. S. Liem, Christoph Lofi, Alessandro Bozzon

Keywords Paper

Shijie Geng, Peng Gao, Moitreya Chatterjee and
Chiori Hori, Jonathan Le Roux, Yongfeng Zhang, Hongsheng Li, Anoop Cherian

Keywords Paper

Giovanni Troiano, Qinyu Chen, Angela Vargas Alba and
Gregorio Robles, Michael Cassidy, Eli Tucker-Raymond, Gillian Puttick, Casper Harteveld

Keywords Paper

Umair Ahmed, Maria Christakis, Aleksandr Efremov and
Nigel Fernandez, Ahana Ghosh, Abhik Roychoudhury, Adish Singla

Keywords Paper

Sanjeev Arora, Simon Du, Sham Kakade and
Yuping Luo, Nikunj Umesh Saunshi

Keywords Paper

Augustus Odena, Kensen Shi, David Bieber and
Rishabh Singh, Charles Sutton, Hanjun Dai

Keywords Paper

Ayush Patwari, Nicholas Kong, Jun Wang and
Ullas Gargi, Michele Covell, Aren Jansen

Keywords Paper

Keywords Paper

Keywords Paper

Keywords Paper

Keywords Paper