Lars Lien Ankile

51 Vassar St

Cambridge, MA 02139

I’m a visiting researcher at the Improbable AI Lab at MIT CSAIL working on robot learning with Prof. Pulkit Agrawal.

Before this, I completed an M.Eng. of Data Science at Harvard University, where I completed my thesis work in the Improbable AI group, working on sample-efficient imitation learning. I also spent a year in the Data to Actionable Knowledge Lab at Harvard, working with Profs. Weiwei Pan and Finale Doshi-Velez on applying RL and Bayesian inference to model human decision-making for frictionful tasks in healthcare settings. Finally, I also spent a summer and fall interning with Prof. David Parkes and Matheus Ferreira in the EconCS Lab at Harvard working on detecting manipulation in multi-agent settings.

I did my undergrad at the Norwegian University of Science and Technology (NTNU) and did my thesis work on applying Deep Learning to econometric forecasting of complex and multivariate time series, supervised by Prof. Sjur Westgaard.

Research Interests

My research goal is to enable machines to learn in a human-like manner, primarily through observation of and interaction with the environment by developing methods merging imitation and reinforcement learning with robust policy representations. I aim to bridge the current divide between general policies that address simple tasks and more specialized policies tailored for complex tasks.

news

Jul 23, 2024	Our latest work, From Imitation to Refinement is now available on arXiv! In this work, we take the lessons from Juicer, and show the limitations of imitation learning for tasks requiring precise control, and propose a simple-yet-effective way to finetune pre-trained diffusion models for such tasks using residual models.
Jul 23, 2024	The paper led by Allen Ren at Princeton, Diffusion Policy Policy Optimization, is now available on arXiv! Here, we introduce DPPO, an algorithmic framework and set of best practices for diretly fine-tuning diffusion-based policies (as opposed to finetuning a residual model as in ResIP).
Jun 30, 2024	Data-Efficient Imitation Learning for Robotic Assembly got accepted to IROS 2024 October 14-18! Please reach out if you are going and want to chat about imitation learning, robotics, or anything else!
Apr 10, 2024	Our recent work on Data-Efficient Imitation Learning for Robotic Assembly is available to read on arXiv! In this work, we show how one can learn long assemblies (~2500 timesteps) with only <50 demonstrations using diffusion policies and data augmentation strategies.

selected publications

From Imitation to Refinement–Residual RL for Precise Visual Assembly

Lars Ankile, Anthony Simeonov, Idan Shenfeld, and 2 more authors

2024

Abs HTML PDF Code

Behavior cloning (BC) currently stands as a dominant paradigm for learning real-world visual manipulation. However, in tasks that require locally corrective behaviors like multi-part assembly, learning robust policies purely from human demonstrations remains challenging. Reinforcement learning (RL) can mitigate these limitations by allowing policies to acquire locally corrective behaviors through task reward supervision and exploration. This paper explores the use of RL fine-tuning to improve upon BC-trained policies in precise manipulation tasks. We analyze and overcome technical challenges associated with using RL to directly train policy networks that incorporate modern architectural components like diffusion models and action chunking. We propose training residual policies on top of frozen BC-trained diffusion models using standard policy gradient methods and sparse rewards, an approach we call ResiP (Residual for Precise manipulation). Our experimental results demonstrate that this residual learning framework can significantly improve success rates beyond the base BC-trained models in high-precision assembly tasks by learning corrective actions. We also show that by combining ResiP with teacher-student distillation and visual domain randomization, our method can enable learning real-world policies for robotic assembly directly from RGB images. Find videos and code at \urlhttps://residual-assembly.github.io.
Diffusion Policy Policy Optimization

Allen Z Ren, Justin Lidard, Lars L Ankile, and 6 more authors

2024

Abs HTML PDF Code

We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task. Website with code: diffusion-ppo.github.io
Scaling Robot-Learning by Crowdsourcing Simulation Environments

Marcel Torne Villasevil, Arhan Jain, Vidyaaranya Macha, and 5 more authors

In RSS 2024 Workshop: Data Generation for Robotics 2024
JUICER: Data-Efficient Imitation Learning for Robotic Assembly

Lars Ankile, Anthony Simeonov, Idan Shenfeld, and 1 more author

2024

Abs HTML PDF Code

While learning from demonstrations is powerful for acquiring visuomotor policies, high-performance imitation without large demonstration datasets remains challenging for tasks requiring precise, long-horizon manipulation. This paper proposes a pipeline for improving imitation learning performance with a small human demonstration budget. We apply our approach to assembly tasks that require precisely grasping, reorienting, and inserting multiple parts over long horizons and multiple task phases. Our pipeline combines expressive policy architectures and various techniques for dataset expansion and simulation-based data augmentation. These help expand dataset support and supervise the model with locally corrective actions near bottleneck regions requiring high precision. We demonstrate our pipeline on four furniture assembly tasks in simulation, enabling a manipulator to assemble up to five parts over nearly 2500 time steps directly from RGB images, outperforming imitation and data augmentation baselines.
I See You! Robust Measurement of Adversarial Behavior

Lars Ankile, Matheus XV Ferreira, and David Parkes

In Multi-Agent Security Workshop @ NeurIPS’23 2023

Abs PDF Code

We introduce the study of non-manipulable measures of manipulative behavior in multi-agent systems. We do this through a case study of decentralized finance (DeFi) and blockchain systems, which are salient as real-world, rapidly emerging multi-agent systems with financial incentives for malicious behavior, for the participation in algorithmic and AI systems, and in highlighting the need for new methods with which to measure levels of manipulative behavior. We introduce a new surveillance metric for measuring malicious behavior in this context and demonstrate its effectiveness in a natural experiment to the Uniswap DeFi ecosystem.
Discovering User Types: Mapping User Traits by Task-Specific Behaviors in Reinforcement Learning

Lars Lien Ankile, Brian Ham, Kevin Mao, and 4 more authors

In First Workshop on Theory of Mind in Communicating Agents @ ICML’23 2023

Abs PDF Code

When assisting human users in reinforcement learning (RL), we can represent users as RL agents and study key parameters, called user traits, to inform intervention design. We study the relationship between user behaviors (policy classes) and user traits. Given an environment, we introduce an intuitive tool for studying the breakdown of "user types": broad sets of traits that result in the same behavior. We show that seemingly different real-world environments admit the same set of user types and formalize this observation as an equivalence relation defined on environments. By transferring intervention design between environments within the same equivalence class, we can help rapidly personalize interventions.
M.Sc.

Exploration of Forecasting Paradigms and a Generalized Forecasting Framework

Lars Lien Ankile, and Kjartan Krange

2022

Abs HTML PDF Code

First, this paper presents a total ordering of the theoretical lower bound loss of different forecasting paradigms in the following descending order: Model selection, Model combination, Non-parametric univariate models, and Non-parametric multivariate models. Second, we create a generalized forecasting framework to test the above forecasting paradigms ex-ante. We implement the framework by creating a novel datacube consisting of daily stock prices and 100,000 quarterly reports from about 1600 global companies and several daily macro time series, all from 2000 to spring 2022. Lastly, we utilize the framework and show that modern multivariate time series approaches are powerful but domain-dependent. We demonstrate the domain-dependent accuracy by showing convincing results when predicting corporate bankruptcy risk, moderate results when predicting stock price volatility, and lacking results when finally predicting company market capitalization. Given the domain-dependent convincing results and mostly unrealized theoretical lower bound loss of multivariate approaches, we hope to encourage further research on non-parametric, multi-signal approaches that leverage a wider array of available information.