Fast and flexible: Human program induction in abstract reasoning tasks

The Abstraction and Reasoning Corpus (ARC) is a collection program induction tasks that was recently proposed by Chollet (2019) as a measure of machine intelligence. Here, we report a preliminary set of results from a behavioral study of humans solving a subset of tasks from ARC (40 out of 1000). We found that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 84% of tasks solved per participant, and with 65% of tasks being solved by more than 80% of participants. Additionally, we find interesting patterns of behavioral consistency and variability across the action sequences to generate their responses, the natural language descriptions used to describe the rule for each task, and the errors people make. Our findings suggest that people can quickly and reliably determine the relevant features and properties of a task to compose a correct solution, despite limited experience in this domain. This dataset offers useful insights for designing AI systems that can solve abstract reasoning tasks such as ARC with the fluidity of human intelligence.