ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
ScarfBench is an open benchmark that evaluates how well AI agents can migrate Java applications across Spring, Jakarta EE, and Quarkus, focusing on behavior preservation and deployment. It measures the success of migrations by requiring applications to build, deploy, and maintain functionality, highlighting the complexity of framework transitions beyond simple code translation. The benchmark reveals that while agents perform well on traditional benchmarks, framework migration remains challenging, with only a small percentage of migrations achieving successful behavior, emphasizing the importance of validation and configuration management in migration processes.
More from Huggingface
LFM2.5-VL-3B for Better and Faster Vision Capabilities for the Edge
Thinking of ACE? We Can Do It with Fewer Tokens
Meta is back with Muse Glimmer: local, agentic, multimodal, and open source
