The Alternative Implementation Problem
What I’ve concluded, based on experience, is that positioning your project as an alternative implementation of something is a losing proposition. It doesn’t matter how smart you are. It doesn’t matter how hard you work. The problem is, when you build an alternative implementation, you’ve made yourself subject to the whims of the canonical implementation. They have control over the direction of the project, and all you can do is try to keep up. In the case of JITted implementations of traditionally interpreted languages, there’s a bit of a weird dynamic, because it’s much faster to implement new features in an interpreter. The implementers of the canonical implementation may see you as competition they are trying to outrun. You may be stuck trying to ice skate uphill.
Almost 4 years ago, with support from Shopify, two dedicated colleagues and I started a project to build YJIT, yet another Ruby JIT. The difference is that we made the key choice to build YJIT not as an alternative implementation, but directly inside CRuby itself. This came with a number of design tradeoffs, but critically, YJIT could be 100% compatible with every CRuby feature from the start. YJIT is now the “official” Ruby JIT, and is deployed at Shopify, Discourse and GitHub among others. If you’ve visited github.com today, or any Shopify store, you’ve interacted with YJIT. We’ve had more success than any other Ruby JIT compiler so far, and compatibility has been key in achieving this.
See also: The Ruby on Rails Podcast.
Previously:
This made me think of Mojo before I even saw the link at the end. I still think that the "superset of Python" goal of Mojo sounds like a fantasy.
The world is littered with also-ran Python implementations with the vast majority of the ecosystem not making any effort to support them, even if it would have been relatively easy. Meanwhile, we all made the jump to Python 3 (which was undoubtably 10x more difficult than migrating to something like PyPy) because Python 3 was where the canonical implementation was going.
Maybe Chris Lattner is brilliant enough to pull it off, but this isn't a situation like Swift and Obj-C where the owner of the canonical framework has made it clear that the future of the platform is Swift.