We just wrapped up RailsConf 2008. A lot has been said about RailsConf elsewhere, so as the program chair I’ll just say “thank you” to the speakers, attendees, co-organizers, sponsors, and everyone else for making it the best conference I’ve ever been to.
A lot of exciting things happened. Among the most exciting to me was the public unveiling of Maglev. I’ve had a version of Maglev on my computer for about a month now, but I haven’t been able to talk about it. Now that it has been announced, most things I know are fair game.
Avi Bryant and I met at the first RubyConf. He’s been a fan and friend of Ruby ever since. He happens to be doing mostly Smalltalk these days, but he’s been following the Ruby and (more recently) Rails communities closely. After that first RubyConf, we reconnected while both attending Dick Gabriel’s pilot MFA in Software Development program at UIUC in early 2003. Avi’s been talking since then about running Ruby on a Smalltalk VM. I’ve even started my own attempt once or twice, based on Avi’s ideas of how it could be done.
Given Avi’s interest in Ruby on Smalltalk and that he also created the mind-bending Seaside web application framework, we invited him to keynote at last year’s RailsConf. It was there, in his message from the future keynote, that he made public his belief that Ruby could benefit from being hosted on a mature, dynamic (Smalltalk) VM.
It was also, as I understand it, from that talk that Gemstone got the idea to run Rails.
The conversations started just after RailsConf last year, but it was only a few short months ago that Avi started work on Maglev. That’s part of what makes it so amazing. He and the engineers at Gemstone have gotten a huge subset of the Ruby language running on the Gemstone VM in just three months.
Well, not really, but it’s fun to say.
There are a lot of things to like about what Maglev promises. By far the most interesting to me is Gemstone’s persistence engine.
What’s coolest about how Gemstone’s persistence engine works within the context of Maglev is that it doesn’t feel like a persistence engine. It’s just objects. Everything’s just objects.
If you think about it, the average Rails application is a styled set of CRUD operations on top of an object graph. That’s right, an object graph. The fact that most Rails applications are backed by a relational database is a somewhat-necessary annoyance.
So in circumstances where you don’t NEED an RDBMS, why would you want to litter your pristine Ruby objects with SQL and other RDBMS-related work-arounds? If it were possible to simply instantiate objects and not have to think about the underlying storage implementation at all, wouldn’t that be better?
That’s what Gemstone (and, therefore, Maglev) gives you.
There were questions during the demo about ActiveRecord compatibility. To me, building in ActiveRecord compatibility is worse than a waste of time. Sure, there are things in ActiveRecord which would be useful regardless of the storage mechanism. Declarative relationships and validations are examples. But, when you’re dealing with objects in memory, the idea of a :conditions clause with SQL in it is absolutely unnecessary and unappealing.
Avi and the Gemstone people knew from their understanding of how the VM works that a Ruby implementation on Gemstone would be really fast. They were right. But as I said, the speed of the implementation is secondary to the persistence engine.
That being said, it’s really really fast. On micro-benchmarks, it’s up to 100 times faster than MRI.
As has already been whined about ad nauseam, micro-benchmarks can be misleading. They test very specific parts of the implementation and don’t always give you a picture of what real-world application performance will be like. So, let’s take this approach with the Maglev benchmarks: it’s fast enough. It’s probably a lot faster than MRI. And, we’ve been using MRI for years for real applications.
Good enough. Probably awesome. But at least good enough.
Now is it going to slow down as they start implementing more of the libraries and language features? That’s the conventional wisdom. I say no.
Smalltalk and Ruby are, as Avi has said numerous times, essentially the same language. So the Gemstone VM has been tailor-made for running dynamic language code with closures, continuations, etc. and has been doing it for years. Imagine the process, then, of converting Ruby to Smalltalk. It’s in most cases a fairly straightforward language to language translation.
Where it isn’t as simple as that, Gemstone have a pretty big advantage: they own the VM and they can change it to fit what Ruby needs.
I expect Maglev to get faster as the team starts to spend time on optimization. So far, they’re in the “make it run” phase of the process. Remember, three months in.
Antonio Cangiano also had a look at Maglev recently and has a bit more to say about the specifics of Maglev performance
So what’s next for Maglev? As the Gemstone team has said, their next goal is to start running and passing the Ruby specs created by the other Ruby implementers. They hadn’t prioritized this leading up to RailsConf, because, RailsConf being a web development conference, they wanted us all to see WEBRick running.
But, hey, they have WEBRick running! Have you looked at the code for WEBRick? That’s a lot of Ruby.
I’m looking forward to a report on how well the Ruby specs run at this moment. I think everyone will be pleasantly surprised. And the specs will give the team a clear goal to work toward.
When you don’t NEED an RDBMS (more often than you think) or you need scalable Ruby servers, I think Maglev is going to be an enticing alternative. It’s not going to kill the other implementations. Each implementation has its place. This isn’t a war, after all.
Macruby, JRuby, and IronRuby are for integration. MRI/YARV and Rubinius will compete for the “standard” use cases. Maglev is for big, distributed object stores inside of fast, scalable servers. Think Rails, Adhearsion, or various coordination services running custom protocols on sockets. Don’t run scripts in Maglev. Don’t write desktop apps in Maglev. Just use Maglev to serve tons of requests against billions of objects.
Maglev is good for the Ruby ecosystem just as MRI/YARV, Macruby, JRuby, and IronRuby are good for the Ruby ecosystem. The fact that we (will) have so many alternatives is a good thing.
Sorry, comments are closed for this article.
June 5th, 2008 at 06:41 PM
Can you comment on what kind of apps you’ve been running on top of MagLev? Just simple Webrick handlers, or are you using any of the existing Ruby web frameworks?
June 5th, 2008 at 07:05 PM
Geoff, I’ve just been playing so far—-mostly with the persistence stuff. More of an informal testing setup than anything else.
June 5th, 2008 at 07:40 PM
I agree you don’t need AR per se. Why would you want domain objects whose fields are defined in some other file or system if it isn’t needed?
But you do need some things AR gives you. “Customer.find_all_by_last_name” is handy. I assume Gemstone AR will map stuff like that to its own query language.
June 5th, 2008 at 07:47 PM
Until something substantial is said about open licensing I’ll remain entirely skeptical. It boggles the mind that such a crucial question is getting ignored.
June 5th, 2008 at 09:45 PM
Jim, right. There are ActiveRecord features which would be useful in this environment. Something ActiveRecord-like which makes no attempt to be compatible with ActiveRecord would be really useful.
Phil, definitely makes sense to be skeptical. If I didn’t know the players and hadn’t personally seen it, I would be too. That being said, I believe what we’re going to see is a very open license for a Ruby implementation sitting on top of a closed technology (Gemstone’s VM). Similar to IronRuby’s model I’d bet. Gemstone has also said that while it wasn’t nailed down yet, the majority of applications will be runnable under a free-as-in-beer VM installation. Gemstone really want to make their money from large clients with large applications and data needs.
June 5th, 2008 at 10:03 PM
BTW Gemstone already has a model with their GLASS product: http://seaside.gemstone.com/ Yeah: GLASS == Gemstone, Linux, Apache, Smalltalk, Seaside
For details how much of it is free (of charge) check the (gem)Stone Soup blog: http://gemstonesoup.wordpress.com/
So yeah: the Gemstone guys are trying to support modern tools, such as Seaside and now (probably) Rails or all of the Ruby web frameworks.
June 5th, 2008 at 10:08 PM
Interesting… I’d think that AR compatibility would be a good way to give MagLev a rocket launch. The value proposition “Run your existing rails apps faster” would be pretty easy to sell I think.
You’re a braver man than I am, prognosticating at this point what the performance of MagLev is going to be relative to the other alternatives. Me, I wouldn’t be surprised if it stays as fast (relatively) as it is now. I’d also be totally unsurprised if some weird corner-case gotcha brought the performance crashing to earth. This is complicated stuff.
But I’m really glad Avi & friends are working on this.
June 5th, 2008 at 11:07 PM
For all of you that can read portuguese (or don’t mind Google Translator :-) I have an extensive review about that as well: http://www.akitaonrails.com/2008/6/4/railsconf-2008-a-controv-rsia-maglev
June 5th, 2008 at 11:21 PM
Wow, that is impressive. I wasn’t able to be at RailsConf this year, and the chatter I heard about MagLev was mostly negative. There was some quick excitement about MagLev, and then alot of cold water thrown on it. I’m glad to hear from someone who’s has their hands on it for a while.
June 6th, 2008 at 12:00 AM
I agree with your view regarding AR. IMHO, if Avi & Gang is to anything in that area, I’d wish it is an adapter for Ambition
June 6th, 2008 at 12:07 AM
Tim, I can see what you’re saying about straight compatibility, but I just don’t feel like that’s where Maglev shines. You might be right though.
choonkeat, I may be missing something, but if everything’s just objects, what would Ambition do?
June 6th, 2008 at 12:43 AM
I agree with the “ActiveRecord to Gemstone” compatibility is harmful sentiment. Part of the advantage of using an OODBMS is that you have a different perspective on how you model your data – you won’t get this advantage if you are sticking to the ActiveRecord API. For example, ActiveRecord maps a Model class to a database table – in an object database you might have separate container classes and model classes. So instead of a “Books” class who’s class methods return “Book” objects, you can have a “Books” class and a “Book” class. When working with a collection of Books, you would then have an instance of the Books class instead of just calling class methods. Or you can have a generic Container class, and store instances of a Book class and a Video class in the same container.
OODBMS and RDBMS also don’t have to be mutually exclusive, either. If you’ve got an existing relational database, but are storing say application-specific data (e.g. user preferences or application settings) that isn’t part of the core relational database, then you can store just that data in an OODBMS. I’ve used Zope to keep changes to objects stored in both the ZODB and in PostgreSQL in the same transaction and it works quite nicely – Zope’s transaction manager implicitly puts transaction boundaries across requests, so it’s not even necessary to pollute your own code with lower level data persistence management.
June 6th, 2008 at 01:03 AM
As someone who spent 5 years with Gemstone/J in production I have some insights into the technology.
While the enhanced Java VM is not the same as the Smalltalk vm, I was told “All the important parts” are shared w/rspt to persistence.
There are no queries. The limited “OQL” like support they had was not very good. So things like Customer.find_all_by_last_name don’t really translate. Instead, you have your customer collection and you call Enumerable#find_all the ruby way.
We used Object traversal heavily. Start from the “Root” object, and start referencing children. Just like you would if you had ALL the objects in memory. Because that’s what it looks like. ALL the objects in memory. You write your domain objects VERY differently.
The database generates garbage, just as dereferenced objects do in memory. So the DB needs to have garbage collection run on it. As I recall, we could only run gc during low periods of load as it was resource intensive. And the GC took longer and longer the more objects there were in the DB. Nightly GCs were moved to Weekly.
Because objects are persistent, “shape changes” are a major deployment consideration. Think ActiveRecord migrations, except on objects and not tables. So you have to run a migration that touches all the “old” objects and turns them “new” objects. This had to be done while the system was offline. There was talk from Gemstone that there was plans to do this “on the fly”.
Upgrades of Major release required migration of all the objects in the system. To do this you had to bring up two instances of the system, old and the new. Then objects were copied over the network between the two system. I still remember pulling all nighters waiting for this complete (ask Bob Walker)
Because your app is effectively running “in” the database, it becomes a giant stored procedure. There was a “clustering” option, but only one machine in the cluster could write to disk. Scaling across machines was a huge concern.
Hopefully some of these concerns were specific to the J product. But some of it is inherent.
The guys at Gemstone are great, and really smart. Great customer support. I wish them all the best.
June 6th, 2008 at 03:03 AM
@Bob Cotton
It’s interesting to hear about your Gemstone experiences. One of the downsides of Gemstone is that between it’s proprietary nature and the fact that it tends to get deployed in industries that aren’t closely involved with the open source there isn’t a lot of information out there about working with the technology.
Another aspect that Ruby folks might re-think when using Ruby w/ MagLev is Routes. If you have an object graph, then mapping URLs onto that object graph is very natural. For example, if you populated an OODB with a few objects, such as:
Then you could automatically form the URL:
This has advantages and disadvantages: you don’t have to write a single line of Routes config and it’s also easier to deduce the URL of an object since there is only one “path” to get to at it, but you can break URLs easier if you restrucutre your object layout. This object traversal technique is what Zope uses, although since it’s possible to use different URL mapping schemes in Zope, there has been some recent discussions about the pros/cons of using a port of Routes to map onto an object graph on the Grok mailing list (http://mail.zope.org/pipermail/grok-dev/2008-May/004838.html).
June 6th, 2008 at 03:46 AM
Actually, IronRuby’s model lets you deploy on a Free stack or a proprietary stack; Maglev seems much more restrictive than that.
Too bad… that rules it out for me.
Gemstone is going to have no trouble making money if they play this smart. Worrying about licensing is about the silliest thing they could do as it alienates the community for short-term benefits.
June 6th, 2008 at 04:35 AM
I missed the maglev talk at RailsConf but I did get a chance to talk to one of the Gemstone founders at one of the parties and it sounded like they were still deciding on how they were going to be licensing this. I personally think it’s great to see an option like this for ruby along with the IronRuby, JRuby, MacRuby, Rubinius efforts.
Can’t wait to see it in action.
June 6th, 2008 at 04:47 AM
@Bob Cotton
your experience sounds a lot like the experience that I’ve had with Prevayler in the Java world as well. A root object, with lots of hierarchy traversal. Pruning that tree became something of a process.
It sounds like they may have moved onto something that’s a little more like the db4o model (which provides compatibility between Java and .NET, as well as a JDBC interface to the store for legacy compatibility)... And slapping it together with Terracotta or Coherence.
@Chad
While I’d love to agree that ideally object databases could replace RDBMSes (and perhaps RDBMSes are user more than they should be) the two things have very different strengths. Reporting engines, and multidimensional analysis and data warehousing – seem to be better fits for RDBMSes. And it seems to me that the database has been in the drivers seat from day one with the Active Record pattern – completely corrupting any idea of a pristine object model; an E-R diagram, and 3rd normal form do not an object model make (either static or dynamic).
In all it will be interesting to see where this goes – but I am a skeptic at heart.
June 6th, 2008 at 04:58 AM
Hi Bob Cotton, long time no hear! You’ve raised some valid points about GemStone/J; I’ll offer some comments.
Offline schema migration could indeed be painful in GemStone/J, though that changed quite a bit with the Facets 3.0 release (Facets is essentially GS/J minus the J2EE app server). Although schema/instance migration is still an offline process in Facets, it’s now fully multithreaded making it much faster. Persistent GC was also made multithreaded – sorry you never go to see that!
The offline nature of Facets schema migration is specific to that product, as Java isn’t nearly as nice about letting you manipulate Class structure (unlike Smalltalk or Ruby) – after all it’s not a dynamic language. The complexities of keeping different versions of the same class around (by putzing with ClassLoaders – yecch!) was not a very appealing solution, so we chose not to go there.
Smalltalk, being a dynamic language, makes it a whole lot easier to handle changes in class structure. GemStone/S allows for multiple versions of any given class, and for both automatic and programmatic instance migration between class versions. Online, without skipping a beat. That’s been in the “S” product for a very long time.
To be clear, MagLev is founded on GemStone Smalltalk; it inherits the ability to do online instance migration without skipping a beat.
BTW, I’ll make sure that the support org sees the compliment – thank you very much :)
It would be great to catch up with you one of these days, drop me an email – you know the address!
Bob W.
June 6th, 2008 at 05:26 AM
you don’t seem too enthusiastic about JRuby (“Macruby, JRuby, and IronRuby are for integration”).
Why is that given it’s currently the fastest of the alternative implementations?
Your enthusiasm seems to be directed towards Rubinius and MagLev….
just curious
June 6th, 2008 at 05:31 AM
Avi and I talked a lot about this in my interview with him about a year ago.
June 6th, 2008 at 06:16 AM
I did some consulting work for one of Gemstone’s competitors a while back. It was an engineering-driven company, with a pretty good suite of technology. About the only competitor they “feared” on a technology basis was Gemstone, because it was also great stuff. Gemstone’s Achilles heel, in the sales process, was always the object database (our product used a standard RDBMS). The sales guys would exploit that to the max, attempting to induce an allergic reaction to the OODB; Bob Cotton’s comments give a good idea of what the sales guys and sales engineers would explain. My client’s sales reps didn’t always get the sale, but if we didn’t, more often than not it went to a big name, rather than Gemstone.
For a variety of reasons, object databases have not taken off. There are some neat technical advantages to them, and some outstanding implementations like Gemstone. But, through combinations of limitations (business and technical), FUD, and luck, they are a distant also-ran to relational databases, at least today. You can argue about whether that’s “right” or not, but it’s hard to argue with the dollars spent on one vs. the other.
My point is, I hope that MagLev will not tie its success to people accepting and leveraging the object database part of the product. Yes, it’s great. There are lots of reasons to use it. And if you use a full implementation of Rails, including all of ActiveRecord, talking to all the standard RDBMS products, to trojan horse the OODBMS into various shops, they will eventually give it a try, and probably end up loving and using it, at least in certain contexts.
But if MagLev just provides most of Rails, with an ActiveRecord adapted to use the OODBMS, it will (I predict) fail in the marketplace, for all the same reasons that OODBMSes are losing to RDBMSes today.
From Bob Walker’s comments, it seems that many of Gemstone’s limitations have either been mitigated, or won’t apply to the Smalltalk-derived VM. DO NOT FOR ONE SECOND THINK THAT WILL MATTER. One of the reasons my client never made it big was because they had great technology, and thought that was enough, or would trump other considerations. But the reality is that lots and lots of purchasing decisions are made by people who will believe competing sales reps explaining 10-year-old limitations, thinking they still apply today. It’s stupid, and the Open Source world has changed things a lot, but it’s still a big factor, at least in getting from the Early Adopters to the Early Majority.
Takeaway (at least for Gemstone): Think “Trojan Horse” in all of your marketing strategies for MagLev. Let people try/buy a faster version of Rails, and sneak the OODB in the back door. The developers will use it, and the suits will eventually accept it.
June 6th, 2008 at 07:19 AM @Bob Walker: <quote>The offline nature of Facets schema migration is specific to that product, as Java isn’t nearly as nice about letting you manipulate Class structure (unlike Smalltalk or Ruby) – after all it’s not a dynamic language.</quote>
Bob, take a look at ASM/cglib stack that gives you ability to manipulate – maybe you will have a chance to migrate objects in memory, while your appliance will still be online. And.. good luck, MagLev! Hope ruby can beat java on speed one day…
June 6th, 2008 at 09:14 AM
hi guys,
in last days i heard often about MagLev and how fast it should be aso, but now i have a problem, in this time i write my diploma thesis, in one of the biggest it projects, in this project is all optimized for speed, so i search for a fast ruby interpreter, and now i hear so much about MagLev, but i can’t test himself, because, i never found a distribution.
can somebody, help me?
if you can, contact me under my email: pscholz@cern.ch
many thanks.
June 6th, 2008 at 12:44 PM
I agree that maglev is good for ruby in general. I however still have problems to accept that a smalltalk guy claims to improve ruby, before I see that even complicated ruby code runs, I think its smalltalk-behind-ruby … but i want ruby, not smalltalk! Anyway, dont take this too serious, I am sure time will show whether they can do it or not ;)
BTW you have a spam problem. I think you should automatically filter these words so that the annoying bots will not make using your html formular harder just look at the above comments
June 6th, 2008 at 01:56 PM
@Porter
“While I’d love to agree that ideally object databases could replace RDBMSes (and perhaps RDBMSes are user more than they should be) the two things have very different strengths. Reporting engines, and multidimensional analysis and data warehousing – seem to be better fits for RDBMSes. And it seems to me that the database has been in the drivers seat from day one with the Active Record pattern – completely corrupting any idea of a pristine object model; an E-R diagram, and 3rd normal form do not an object model make (either static or dynamic).”
I think we completely agree. Specifically that there are some places where RDBMS systems shine and some where they don’t. You’re absolutely right that ActiveRecord is a pattern tied to relational databases. If you’re just using objects, the pattern isn’t relevant anymore.
June 6th, 2008 at 02:00 PM
@phil swenson: I’m very enthusiastic about JRuby, as I have been since the first public release of it (in 2001 I think). It makes Ruby on Java possible, which is life saving for legions of programmers stuck in J2EE environments. I’m talking more about Rubinius and Maglev lately because they are new and each offer something different from the implementations before them. I think all of the Ruby implementations are great and all fill important and interesting gaps.
June 6th, 2008 at 05:23 PM
Part of the problem people have with object databases seems to be a failure to realize that the whole point of an ORM is to recreate the experience of an object database on top of a relational database (especially something like Ambition). The problems of searching and querying objects are as old as computer science itself.
Given the ingenuity of the Ruby community, I expect to see libraries written that would provide a nice interface for generating expressive queries on the objects. We’re all familiar with SQL, but I would like to think that there could be better ways to work that would also be more native to Ruby.
June 6th, 2008 at 05:36 PM
@chad maybe i’m still half-way pass this paradigm… but u mean they’ll hook every enumerable method to somehow do index search? then there’s nothing special to introduce…
June 7th, 2008 at 05:02 AM
When will the general public be able to start experimenting with Maglev?
June 7th, 2008 at 03:44 PM
I hate to sound like a broken record, but this sentence could hardly be less true. “He and the engineers at Gemstone have gotten the full Ruby language running on the Gemstone VM in just three months.”
Until you support things like: eval(“caller”, some_proc.binding), you are not ‘the full Ruby language’.
In case you think I made up that crazy line of code, that is how RSpec works.
June 7th, 2008 at 04:02 PM
@Wilson Good point. I was incorrectly thinking of the language syntax and saying “language”. Will edit.
June 7th, 2008 at 05:27 PM
As with most things shiny and new, there’s the inevitable swell of bright-eyed enthusiasm based on optimistically incorrect assumptions that are not subjected to hard-looking analysis. Object databases have been around for more than three decades, ZODB for quite a long time as well. Image-based systems are not new on the block either.
I’m always extremely skeptical of assertions that these great technologies are not adopted by the widespread masses of people because, well, they’re just too dumb to know what’s good for them.
The other craze-fad-phase in the Ruby space recently centers around concurrency and Erlang. Concurrency in your program structure is a qualitatively different problem to solve. RDBMS and OODBMS are more similar than different, so I would suggest that solving the concurrency problem is more pressing that whether you use an OODB vs ORM. If you’ve watched this talk http://www.infoq.com/presentations/erlang-software-for-a-concurrent-world or read Joe’s book, you might pause to consider the implications of a single, gigantic, share-everything (significant) system.
Putting a rabbit in my hat so you can pull it out of yours is good entertainment, but to stretch the analogy perhaps too far, how much of what you see in a magic show ends up being useful in your daily life? I’m eagerly waiting for the real world to join up with the “zomg so fast it’ll blow your socks off in only three months we run WEBrick”. I won’t believe that Gemstone has even implemented the language syntax (let alone the semantics) until they show me running the RubySpecs language specs (we don’t expect them to pass the core and stdlib specs in just three months; six maybe).
I’m not bagging on MagLev or Gemstone. But ignorance propagates too easily. Closed systems have no long term benefit (has your Vista crashed lately?). And putting your faith in something you don’t understand will surely lead to disappointment.
June 7th, 2008 at 08:44 PM
Here’s a Ruby feature I just learned about when I had to get it working in Rubinius: http://metaclass.org/2008/6/7/calling-in-the-dark
Ruby is a tricky thing to implement. I wish Maglev the best of luck; we got along really well with them at RailsConf. However, I think it is helpful for everyone (including the Maglev team) to keep an eye on the darkest Ruby magic in the distance, like a weather-front of evil.
June 8th, 2008 at 01:12 AM
@Wilson I don’t doubt this kind of thing would be tough to catch (and therfore implement), but it looks like pretty straightforward Ruby metaprogramming to me. What struck you as being dark magicky about it? Blocks being closures, it does what I expect it to do (call the caller() method with the Proc’s binding (which close over the binding when the Proc was created)).
June 8th, 2008 at 07:07 AM
I talked with some ex Gemstone programmers and they mentioned that that one of the major issues with an Object database is that it is difficult to migrate production data. I wonder if Gemstone has done anything lately to make that less of an issue.
June 8th, 2008 at 01:22 PM
@Chad: In this case, “caller” does not return the caller of the Proc whose binding it is evaluated in.