New Approach to C/Java Native Extensions

[jdantonio]

Now that we've had C/Java native extensions in the latest release for a few months, they have proven to be as big of a pain as I expected. For the next major release I suggest that we move all C and Java native code into the ruby-atomic gem, keep concurrent-ruby pure Ruby, have this gem detect the presence of ruby-atomic when installed, and use the optimized native code when present. Additionally, I suggest we use the current automated build process to build pre-compiled versions of ruby-atomic, but allow for compilation of extensions on install (which we currently do not do for concurrent-ruby).

A few other approaches have been suggested in other threads. Lets use this thread to discuss alternatives and make a decision.

The big problem posed by native extensions is obtaining a consistent build across all platforms. We've been able to manage this for many platforms by creating the pre-compiled builds. Things work as expected on Windows, JRuby, Rubinius, and standard Linux (Ubuntu and CentOS) installations. We have had problems on OS X (Issue #170) and Heroku (Issue #181). In both cases the pure-Ruby code is used even when the target platform supports compilations.

It is important to note that the issues #170 and #181 are the result of an intentional decision. Expecting a gem to compile on installation often causes problems because the necessary build tools may not be installed on the target system. This becomes an unrecoverable error that prevents installation of the gem. Even when the necessary build tools can be easily installed, many DevOps professionals prefer not to. Installing build tools on a production system is considered a security issue by many.

Although many applications will benefit by the performance improvements provided by the native code, many do not need them. Additionally, interpreters other than MRI and JRuby (such as Rubinius) cannot use the extensions anyway. Moving the native code into a different gem and making it optional seems like the most problem-free approach.

[pitr-ch]

I have to admit I have no idea how big of pain it was so I may easily misjudge.

I am not sure I've understood the problem in #181? Heroku is always asking for pure Ruby?

I am assuming that we would like to ideally support:

• precompiled versions
• let gem to compile if there is no precompiled version for the platform and build tools are available
• fallback to pure Ruby if all the above fails

Then we could enable gem compilation and find a way how to fallback gracefully to pure Ruby if compilation fails (that may not be possible based on quick googling).

Second option could be to provide two gems concurrent-ruby (only pure Ruby) and concurrent-ruby-native where the ..-native would provide precompiled versions or it would try to compile on installation. (It would be better to avoid monkey-patching but rather to build a helper object which would select implementations based on what is available on current platform. concurrent-ruby would provide just 'pure' and concurrent-ruby-native would provide others like C and Java for selection. Using one git repository with 2 gemspecs should help with maintainability. That would also allow to ask at runtime what implementation is used.)

[jdantonio]

In #181, Heroku is not installing the pre-compiled gem. It is installing the pure-Ruby version. I haven't investigated yet, but there are two possible reasons that I can think of. 1) When Heroku pulls the gem from Rubygems it reports the "platform" as something Rubygems doesn't recognize so it defaults to the pure-Ruby version. 2) The user is caching the gem packages within the git repo (a very common practice with Rails) and the cached gem is the pure-Ruby one (possibly because development is being done on OS X). Either way, this isn't the behavior we desire. Heroku instanced have the build tools necessary for compiling the extensions so we want to make sure the compiled versions are installed.

I do not know if it is possible to have the base gem support both compile-at-install and a pure-Ruby fallback. The problem is that Rubygems performs the compilation as it installs the gems. The mechanism by which it does this was not designed to be optional. If the necessary build tools do not exist the Rubygems generates an error and fails the gem installation. I've read a few hacks online and experimented with a few of them during my initial exploration with native extensions, but I do not have confidence in them at this point. I will have to do a lot more testing. Even so, a working solution would still be a hack that circumvents the normal operation of Rubygems and we'll need to be careful.

[jdantonio]

I had a conversation today with a developer from MongoDB who gave a tall on C-extensions. To her knowledge there is no way to optionally compile extensions at gem installation, which is what my research has indicated. She explained that at MongoDB they put all their C and Java extensions in separate, optional gems, and recommended that we do the same.

Presuming we take that approach, the remaining question is whether we create a new gem or use the existing ruby-atomic gem. I spoke to @headius briefly yesterday but haven't had a chance to ask him how he feels about us adding more code to ruby-atomic and using it that way.

[chrisseaton]

I don't understand what the benefits of re-using ruby-atomic would be. Surely we just want concurrent-ruby, concurrent-ruby-native and concurrent-ruby-java?

[jdantonio]

@chrisseaton No real advantage, but we have to do something with ruby-atomic. The original plan from @headius was to end-of-life that gem because the code is now in concurrent-ruby. It's been downloaded millions of times, however. What we don't know is whether the users of that gem will want to upgrade to concurrent-ruby (with vastly more code) or continue using the last release of ruby-atomic. If we decide to create a separate gem with the compiled atomic classes we may better serve the current users of ruby-atomic by keeping it alive and usable as a stand-alone gem. That's the only reason.

[chrisseaton]

Could make ruby-atomic a thin layer over concurrent-ruby?

[jdantonio]

I should also note that if we choose to keep ruby-atomic around as a stand-alone gem that integrates with concurrent-ruby, rather than create a new gem that is purely and extension of concurrent-ruby, we'll may end up adding some duplication. Specifically, we'll have to decide how to handle the pure-Ruby versions of the atomic classes.

When I get back home from RubyConf I can spike some of these ideas if that will help.

[jdantonio]

I apologize for causing so much confusion. Let me start over. The question is this:

Now that we have merged all code from ruby-atomic into concurrent-ruby, what do we do about the 600 thousand people who have downloaded the latest version of ruby-atomic?

One option is to end-of-life ruby-atomic and force those users to upgrade to concurrent-ruby. Another option is to devise a solution that will keep ruby-atomic alive but fully integrated into concurrent-ruby.

The first option is best for the concurrent-ruby project but may disenfranchise the current users of ruby-atomic who will now need to install over 4600 LOC that they don't necessarily need. The second option is best for the current users of ruby-atomic but will cause additional complexity for this project. My original suggestion above was a suggestion for the second option.

If we (specifically @headius) decide to end-of-life ruby-atomic (the first option) then the plan for 8.0 should be fairly simple:

• Create one more release of ruby-atomic that includes a deprecation warning (something like this PR
• Move all our native C code into an optional extension gem called concurrent-ruby-ext that only works in conjunction with the main gem
• Keep concurrent-ruby pure-Ruby (plus compiled Java) but have it automatically detect when concurrent-ruby-ext is installed and behave accordingly (similar to what we already do).

If we decide on the second option then we need to devise a plan similar to my original suggestion.

@headius @chrisseaton @mighe @pitr-ch @lucasallan Thoughts?

[jdantonio]

It should also be noted that our pre-compiled Java gem package works flawlessly on all tested operating systems (Ubuntu and CentOS 32 and 64, Windows 32 and 64, Solaris, and OS X). This is to be expected thanks to the consistency of the JVM. Thus we would keep our compiled Java code in concurrent-ruby and only put compiled C code for MRI in concurrent-ruby-ext. (This is a change from what I originally stated but I've thought it through farther. I've updated my previous comment accordingly.)

At each release we would use our automated build process to create the following gem packages:

• concurrent-ruby.gem -- pure Ruby code, all operating systems, all Ruby runtimes except JRuby
• concurrent-ruby-java.gem -- pure Ruby + compiled Java code, all operating systems, JRuby only
• concurrent-ruby-ext.gem -- uncompiled native C code, all operating systems, MRI only
• concurrent-ruby-ext-x64-mingw32.gem -- compiled C code, Windows 64, MRI only
• concurrent-ruby-ext-x86-mingw32.gem -- compiled C code, Windows 32, MRI only
• concurrent-ruby-ext-x86_64-linux.gem -- compiled C code, Linux 64, MRI only
• concurrent-ruby-ext-x86-linux.gem -- compiled C code, Linux 32, MRI only
• concurrent-ruby-ext-x86-solaris-2.11.gem -- compiled C code, Solaris, MRI only

We would also consider a pre-compiled gem for MRI on OS X, assuming I can figure out the bug that caused me to yank concurrent-ruby- x86_64-darwin-13.gem from Rubygems.

[mighe]

atomic-ruby is a very very simple gem, it just provides an optimized and thread safe class and it does it really well. It won't probably require a lot of work in the future, since it is well focused, complete and stable. ( @headius made a great job!)
I think we could keep it alive (your second option) for that reason: at most we could add some kind of warning saying that we suggest to move to concurrent-ruby for a more complete library.

[mattwildig]

On Heroku, what I think is likely happening is that users add concurrent-ruby to their Gemfile and then (if they are on OSX) when they run bundle install Bundler is finding the pure Ruby version and adding that to the Gemfile.lock. Heroku will then try to install the pure Ruby version (Bundler will try to match the version exactly). So the gems don’t need to be cached in git – whatever version is installed on the platform that first runs bundle install will be the version used everywhere.

This is an known issue for people developing on Windows and trying to deploy to Heroku since precompiled Windows gems are more common than for other platforms.