Requiring concurrent in a gemspec results in superclass mismatch on RBX

[sempervictus]

Using rubinius 2.2.10 (2.1.0 bf61ae2e 2014-06-27 JI), currently converting an NLP implementation to use futures and eventually become a dataflow. Works great in IRB, but requiring 'concurrent' in the code causes gem build to raise "Superclass mismatch: Concurrent::Atomic != Rubinius::AtomicReference." Likely some evil gremlin within Rubinius...

Having glanced over the class definition for Concurrent::Atomic, the class heirarchy from RBX' AtomicReference to Concurrent::Atomic seems reasonable, but i figure that posting the issue here can't hurt since you guys seem close to a release.

[jdantonio]

@sempervictus Than you for letting me know. I'm going to work on this today so that I create the next build. Can you please clarify for me, specifically which version number are you specifying in the gemspec?

[jdantonio]

@sempervictus I think I've found the issue, which raises an interesting question. As a Rubinius user I'd love your feedback on this. First, let me explain what I've discovered.

Rather than forcing our users to have complete build environments, we now created pre-compiled binaries for multiple platforms. So I added

s.add_dependency 'concurrent-ruby', '0.7.0.rc2'

to the gemspec file in one of my other gem projects, built the other gem, then installed the other gem under rbx-2.2.10. Rubygems then installed concurrent-ruby-0.7.0.rc2-x86_64-darwin-13 via the dependency in the gemspec (I'm working on a Mac). That particular gem version is one that has pre-compiled binaries. So even though I was running my code under Rubinius (I also wrote a test script) I still had access to the compiled Concurrent::CAtomic, pure-C implementation.

When I look at the code for atomic.rb what I notice is that the if/elsif/else code looks for CAtomic (line 60) before it looks for RbxAtomic (line 69). So even though I am running my code under Rubinius, I'm getting the pure-C implementation.

Which leads me to my question: As a Rubinuis user, would you rather have your code default to CAtomic or RbxAtomic? It would be easy for me to change the order of operations in atomic.rb and detect RbxAtomic first, thus using RbxAtomic by default on Rubinius, even when CAtomic is available.

I don't use Rubinius myself so I may not be the best person to make this judgement call. But I'm happy to make the change if you feel the Rubinius community would prefer the alternative approach.

[sempervictus]

Dependency is added this way already, and i've not seen the C ext work outside MRI yet.
I'd like the ability to use the C ext if available, but it presently doensnt build on RBX. Current use case will benefit from all the locking (or however you want to consider atomic dereferencing flow control) performance we can get.
My setup is Linux(3.15), Ubuntu(ish) x64, in case that matters to the build system.

I've pinged @brixen with this issue as well - chances are he'll politely explain how i'm wrong in what i'm doing somewhere, but barring that how to resolve the actual namespace problem.

What happens if you flush your C-ext and attempt to build a gem depending on this?

[jdantonio]

So there are two issues here: building the gem and choosing which version of Atomic to use. I think the current setup should meet your needs.

Because we push pre-compiled extensions to Rubygems, no one should ever have to build the gem locally. We currently create pre-compiled builds for Linux 32 & 64, Windows 32 & 64, OS X, Solaris, and Java. Anyone installing the gem on any of those platforms gets a pre-compiled build and never has to compile locally. The "base" gem, which is installed when not on the aforementioned platforms, is 100% pure Ruby and has the native extensions turned off. So it won't try to compile at all.

Within the gem itself, the pure-Ruby versions of every class are always available. The pure-Ruby classes are always prefixed with "Ruby" or "Mutex" (examples: RubyThreadPoolExecutor and MutexAtomic). The pure-Ruby versions can always be used explicitly simply by using the full class name. In cases where we also provide a platform-optimized variant, the native code has a different class name (examples: JavaThreadPoolExecutor and CAtomic). If those classes are defined on a given platform they can be used explicitly by using the full class name.

For convenience and cross-platform portability, we always create a platform-independent alias which points to one of the platform-optimized classes (examples: ThreadPoolExecutor and Atomic). When the gem is required in an application there is if/elsif/else logic that dynamically determines which platform-specific class is used as the base. So if you run your code under JRuby, the class Atomic aliases JavaAtomic, but on MRI Atomic will alias MutexAtomic.

As the programmer you always have a choice. If you don't trust C or Java extensions you can always choose to use the pure-Ruby versions. Always.

Given what I found this morning, on Rubinius you will always have at least two options for atomic references: Concurrent::MutexAtomic and Concurrent::RbxAtomic. You may also have the option to use Concurrent::CAtomic if you are running on a platform for which we have provided pre-compiled binaries (like my MacBook Pro). As the gem exists right now, however, on Rubinius the Concurrent::Atomic class will choose to alias CAtomic over RbxAtomic. It would be trivial for me to change that preference order. This change would not, however, limit which classes are made available.

[sempervictus]

Seems odd that source installs wouldnt build the same as the upstream gem - an unexpected inconsistency. Moreover, it appears that this build pattern causes RBX to break, but only when building a gem due to whatever process it goes through to validate the language tree at build time.

When i manually try to build the extensions under Rubinius (ruby extconf.rb in ext/concurrent_ruby_ext) i get 'C optimizations are not supported on this version of Ruby.' This seems to indicate that the behavior you're seeing under RBX with C extensions is actually unsupported (and i get the sense that you know better than it does), and that the class assignment would in fact be broken when depending on the presence of CAtomic (even if not used) since no such CAtomic class actually exists.

[jdantonio]

Why are you manually building the gem? Our intent in providing pre-compiled binaries is to prevent users from having to compile on their own, thus avoiding the problems often associated with gem compilation. What operating system are you using?

The original intent was for C-extensions to not be available on Rubinius. That's why you get the "C optimizations are not supported..." message. I was under the (mistaken) impression that when the gem was installed under Rubinius that only the pure-Ruby classes would be available, and only learned otherwise this morning when researching your issue. Atomic is the only class for which there is a Rubinius-specific version (because Rubinius provides its own atomic reference class), so that is the only class for which this issue exists.

One option would be up update the class aliasing logic so so that the C-native classes, like CAtomic, are only used when on MRI. That would align with the original intent and be more consistent than the behavior we discovered this morning.

[brixen]

I'm barely able follow the discussion here but there are a couple points I'd make:

1. You cannot build C-exts for MRI and then load them in Rubinius. Our C-API compatibility layer is source only. You need to compile the C-exts using Rubinius to load them.
2. You should not need C-exts on Rubinius. If there is some concurrency-related primitive that is missing, let us know and we'll expose it to Ruby from the VM.
3. Consider using the redcard gem to separate Ruby code by implementation.

[jdantonio]

@brixen Thanks for adding your thoughts. Our original intent was in alignment with your points 1 and 2. We have pure-Ruby versions of everything and had only intended to expose the C-native code under MRI. The C-native code being exposed under Rubinius is a bug that was discovered by @sempervictus.

I'm currently working on an update that provides better guards against the unintentional loading of C-extensions. This will guarantee that Rubinius users will only get the pure-Ruby code.

[sempervictus]

Thank you both, and sorry for the hassle.
@jdantonio please feel free to ping me for testing.

[jdantonio]

@sempervictus Apologies are mine. You exposed a very problematic bug, and for that I am thankful. I would love to have your help testing once I've created the PR.

[jdantonio]

@brixen Is it possible to create a platform-specific gem build for Rubinius?

We are currently uploading seven platform-specific builds of our gem, plus a pure-Ruby version. I would like to create a Rubinius-specific build if possible. It appears to me that Rubygems uses RUBY_PLATFORM to decide which gem build to install. JRuby sets RUBY_PLATFORM to "java" which enables the creation of JRuby-specific gem builds. Since Rubinius sets RUBY_PLATFORM to the same value as MRI, Rubinius is downloading the same version as MRI. In our case, this is the version with pre-compiled extension. Is there a trick I'm missing that would allow me to create a Rubinius-specific gem build that is pure-Ruby? If not I'll add guards in the code to prevent loading the C extensions from loading under Rubinius.

[chuckremes]

Use redcard as suggested earlier by @brixen. See here: http://rubygems.org/gems/redcard

That gem is tailor made for adding the kinds of “guards” that you seek.

On Aug 11, 2014, at 4:57 PM, jdantonio notifications@github.com wrote:

@brixen Is it possible to create a platform-specific gem build for Rubinius?

We are currently uploading seven platform-specific builds of our gem, plus a pure-Ruby version. I would like to create a Rubinius-specific build if possible. It appears to me that Rubygems uses RUBY_PLATFORM to decide which gem build to install. JRuby sets RUBY_PLATFORM to "java" which enables the creation of JRuby-specific gem builds. Since Rubinius sets RUBY_PLATFORM to the same value as MRI, Rubinius is downloading the same version as MRI. In our case, this is the version with pre-compiled extension. Is there a trick I'm missing that would allow me to create a Rubinius-specific gem build that is pure-Ruby? If not I'll add guards in the code to prevent loading the C extensions from loading under Rubinius.

—
Reply to this email directly or view it on GitHub.

[jdantonio]

@chuckremes My hope is to avoid adding guards at all. My hope is to create a Rubinius-specific gem the way we create a JRuby-specific gem. I'm just not sure that will be possible.