AkitaOnRails.com

So, Rails 5.0.0 Beta 1 has just been released and the main new feature is Action Cable.

It's basically a complete solution on top of vanilla Rails so you can implement WebSocket based applications (the usual real time chats and notifications) with full access to your Rails assets (models, view templates, etc). For small to medium apps, this is a terrific solution that you might want to use instead of having to go to Node.js.

In summary you control Cable Channels that can receive messages sent through a WebSocket client wiring. The new Channel generator takes care of the boilerplate and you just have to fill in the blanks for what kinds of messages you want to send from the client, what you want to broadcast from the server, and to what channels that your clients are subscribed to.

For a more in-depth introduction, DHH himself published a bare bone Action Cable screencast that you should watch just to get a feeling of what the fuzz is all about. If you watched it already and have experience in programming, you may have spotted the problem I mention in the title, so just jump to "The Problem" section below for a TL;DR.

In the end you will end up with a code base like the one I reproduced in my Github repository up until the tag "end_of_dhh". You will have a (very) bare bone single-room real time chat app for you to play with the main components.

Let's just list the main components here. First, you will have the ActionCable server mounted in the "routes.rb" file:

# config/routes.rbRails.application.routes.draw do  root to: 'rooms#show'# Serve websocket cable requests in-process  mount ActionCable.server => '/cable'end

This is the main server component, the channel:

1234567891011121314

# app/channels/room_channel.rbclassRoomChannel< ApplicationCable::Channeldefsubscribed    stream_from "room_channel"enddefunsubscribed# Any cleanup needed when channel is unsubscribedenddefspeak(data)Message.create! content: data['message']endend

Then you have the boilerplace Javascript:

# app/assets/javascripts/cable.coffee#= require action_cable#= require_self#= require_tree ./channels#@App ||= {}App.cable = ActionCable.createConsumer()

And the main client-side Websocket hooks:

12345678910111213141516171819

# app/assets/javascripts/channels/room.coffeeApp.room = App.cable.subscriptions.create "RoomChannel",connected: -># Called when the subscription is ready for use on the serverdisconnected: -># Called when the subscription has been terminated by the serverreceived: (data) ->$('#messages').append data['message']speak: (message) ->@perform'speak', message: message$(document).on "keypress", "[data-behavior~=room_speaker]", (event) ->if event.keyCode is 13App.room.speak event.target.value    event.target.value = ''    event.preventDefault()

The view template is a bare bone HTML just to hook a simple form and div to list the messages:

1234567891011

<!-- app/views/rooms/show.html.erb --><h1>Chat room</h1><divid="messages"><%= render @messages %></div><form><label>Say something:</label><br><inputtype="text"data-behavior="room_speaker"></form>

The Problem

In the "RoomChannel", you have the "speak" method that saves a message to the database. This is already a red flag for a WebSocket action that is supposed to have very short lived, light processing. Saving to the database is to be considered heavyweight, specially under load. If this is processed inside EventMachine's reactor loop, it will block the loop and avoid other concurrent processing to take place until the database releases the lock.

# app/channels/room_channel.rbclassRoomChannel< ApplicationCable::Channel  ...defspeak(data)Message.create! content: data['message']endend

I would say that anything that goes inside the channel should be asynchronous!

To add harm to injury, this is what you have in the "Message" model itself:

123	classMessage< ApplicationRecord after_create_commit { MessageBroadcastJob.perform_later self }end

A model callback (avoid those as the plague!!) to broadcast the received messsage to the subscribed Websocket clients as an ActiveJob that looks like this:

12345678910111213

classMessageBroadcastJob< ApplicationJob  queue_as :defaultdefperform(message)ActionCable.server.broadcast  'room_channel', message: render_message(message)end  privatedefrender_message(message)ApplicationController.renderer.render(partial: 'messages/message', locals: { message: message })endend

It renders the HTML snippet to send back for the Websocket clients to append to their browser DOMs.

DHH even goes on to say "I'd like to show it because this is how most apps will end up."

Indeed, the problem is that most people will just follow this pattern and it's a big trap. So, what's the solution instead?

The Proper Solution

For just the purposes of a simple screencast, let's make a quick fix.

First of all, if at all possible you want your channel code to block as little as possible. Waiting for a blocking operation in the database (writing) is definitely not one of them. The Job is underused, it should be called straight from the channel "speak" method, like this:

12345678

# app/channels/room_channel.rbclassRoomChannel< ApplicationCable::Channel   ...defspeak(data)-    Message.create! content: data['message']+    MessageBroadcastJob.perform_later data['message']endend

Then, we move the model writing to the Job itself:

1234567891011

# app/jobs/message_broadcast_job.rbclassMessageBroadcastJob< ApplicationJob   queue_as :default-  defperform(message)-    ActionCable.server.broadcast  'room_channel', message: render_message(message)+  defperform(data)+    message = Message.create! content: data+    ActionCable.server.broadcast 'room_channel', message: render_message(message)end   ...

And finally, we remove that horrible callback from the model and make it bare-bone again:

123	# app/models/message.rbclassMessage< ApplicationRecordend

This returns quickly, defer processing to a background job and should sustain more concurrency out-of-the-box. The previous, DHH solution, have a built-in bottleneck in the speak method and will choke as soon as the database becomes the bottleneck.

It's by no means a perfect solution yet, but it's less terrible for a very quick demo and the code ends up being simpler as well. You can check out this code in my Github repo commit.

I may be wrong in the conclusion that the channel will block or if this is indeed harmful for the concurrency. I didn't measure both solutions, it's just a gut feeling from older wounds. If you have more insight into the implementation of Action Cable, leave a comment down below.

By the way, be careful before considering migrating your Rails 4.2 app to Rails 5 just yet. Because of the hard coded dependencies on Faye, Eventmachine, Rails 5 right now rules out Unicorn (even Thin seems to be having problem booting up). It also rules out JRuby and MRI on Windows as well because of Eventmachine.

If you want the capabilities of Action Cable without having to migrate, you can use solutions such as "Pusher.com", or if you want your own in-house solution, follow my evolution on the subject with my mini-Pusher clone written in Elixir.

There has been recent discussions about how nice it would be to have something like the awesome Elixir Pipe Operator for Ruby.

If you don't know what the "Pipe Operator" is in Elixir, take the following code:

1	Enum.sum(Enum.filter(Enum.map(1..100_000, &(&1 * 3)), odd?))

It's ugly, we all know that. In an Object Oriented language like Ruby we would do something like this:

1234	(1..100_000). map { \|i\| i * 3 }. select(&:odd?). reduce(&:+)

But Elixir does not have objects, only functions, so how can we code more elegantly? The solution came up in the form of the so called "Pipe Operator" which takes the last returning value and pass it through as the first argument of the next function call, like this:

1234	1..100_000 \|> Stream.map(&(&1 * 3)) \|> Stream.filter(odd?) \|> Enum.sum

So Ruby and Elixir "feels" the same when we are able to "chain" methods. In the Ruby world we don't have the "need" for an operator like that. But it would be nice to have a mechanism that we could use to make our codes more expressive, or more testable, or more readable. For example, what if we would write something like this:

1234	(1..100_000). multiple_each_element_by_three. filter_out_odd_elements. give_the_sum_of_all_elements

Of course, this is a very constrained example with really bad method naming. But if we get Mo Lawson's article I linked above, it becomes more interesting:

keywords  .map { |kw| kw.gsub(/(^[^[:alpha:]]+|[^[:alpha:]]+$)/, '') }  .map { |kw| LegacySpanishCorrector.new.correct(kw) }  .map { |kw| kw.gsub(/[^[:alpha:]\d'_\-\/]/, '') }  .reject { |kw| STOP_WORDS.include?(kw) }

Ruby allows us to chain Enumerable methods one after the other, transforming the initial keywords list into "something" that is very difficult to infer just by looking at this code.

What about this other version?

123456789101112

classKeywordNormalizerdefcall(keywords)Collection.new(keywords)      .strip_outer_punctuation      .fix_spanish      .clean_inner_punctuation      .remove_stop_words      .to_aend# ...end

This is where we gets by the end of his article: much more readable and each new isolated method is unit testable, resulting in more robust code.

The whole idea of this post is to present you to my new little gem "Chainable Methods". The source code is on Github, as usual, so please contribute.

My gem will allow you to write the Lawson's last code like this:

12345678

KeywordNormalizer  .chain_from(keywords)  .strip_outer_punctuation  .fix_spanish  .clean_inner_punctuation  .remove_stop_words  .to_a  .unwrap

You add the chainable_methods to your Gemfile as usual (you know the drill), then you can write Lawson's module like this:

12345678910111213141516

moduleKeywordNormalizer  extend ChainableMethodsdefself.strip_outer_punctuation(array)    array.map { |el| el.gsub(/(^[^[:alpha:]]+|[^[:alpha:]]+$)/, '') }enddefself.fix_spanish(array)    array.map { |el| LegacySpanishCorrector.new.correct(el) }enddefself.clean_inner_punctuation(array)    array.map { |el| el.gsub(/[^[:alpha:]\d'_\-\/]/, '') }enddefself.remove_stop_words(array)    array.reject { |el| STOP_WORDS.include?(el) }endend

And that's it, now you can chain everything like I showed previously. The pattern is:

1) Write a Module with class-level methods that receive at least one argument and extend the 'ChainableMethods' module:

123456789101112131415

moduleMyModule  extend ChainableMethodsdefself.method_a(argument1)# ...enddefself.method_b(argument1, argument2)# ...enddefself.method_c(argument1)# yield a block passing the received argumentendend

2) Wrap an initial state that will be passed to the first method as it's first argument:

1234	some_initial_state = "Hello World"MyModule .chain_from(some_initial_state)# ...

3) Chain as many methods from the module or methods that the returning state recognizes:

123456789

MyModule  .chain_from(some_initial_state)  .upcase  .method_a  .method_b(argument2)  .method_c { |foo| foo }  .split("")  .join(", ")  .unwrap

Notice that we do not need to pass the first argument to the methods within the 'MyModule' module, it will get the result from the last call automatically.

4) Do not forget to call #unwrap as the last call to get the last result from the chain.

An Experiment

And that's it! I isolated this behavior only into modules that explicitly extend the 'ChainableMethods' module instead of automagically enabling it in the BasicObject level as many would initially think because we don't want a global 'method_missing' dangling around unchecked.

This behavior makes use of 'method_missing' so it's not going to be fast in a synthetic benchmark against a direct method call, for obvious reasons. The purpose is not to be fast, just expressive. So keep that in mind.

The use case is: whenever you have some kind of transformation, you will want a chain of unit testable, isolated, functions, and this is how you can get it without too much hassle.

This is an experiment. Because I'm using 'method_missing' there may be side-effects I am not seeing right now, so please let me know in the Github Issues and send feedback if it helped you out in some project.

Pull Requests are most welcome!

Last year I did a mistake: I left an open vulnerability in a client's project. Fortunately it was a short lived project with no reported breaches, but this was such a stupid oversight that I might as well bang my head in a wall because of it.

So, let me explain the situation that might lead to this kind of vulnerability: a server-side API, serving tampered raw data to a client-side consumer that is assuming that the data is safe, and rendering it directly into the front-end.

First of all, for the uninitiated, a XSS or Cross Site Scripting is when you allow user input to be rendered in your site without proper sanitization. For example, a simple comments section. You expect users to just fill in the usual rants, but someone pastes in a javascript and when you render this comment, it executes in all your users browsers. It can go from simple site defacing up to stealing your users private data. A nasty vulnerability.

To avoid this, by default, Rails sanitizes every string that you throw into the view templates. You have to manually declare your string as safe if you want to render its unfiltered raw content. There are many nuances to this as scripts can come from many sources that flag it as being safe when it's not. The official Rails Guide has an entire page for Security best practices and it includes several injection vectors that you need to know.

1234567891011

classPagesController  ...defshow@page = Page.find_by_slug(params[:id])    respond_to do |format|      format.html      format.json { render json: @page.to_json(only: [:title, :slug, :body]) }endend  ...end

Simple one-liner to allow your Page URL to respond to "/pages/some-page.json" and now you can add a simple Javascript/Coffeescript to your page to load this JSON like this:

$ ->$.ajax    url: "/pages/" + $("meta[name=page]").attr("id") + ".json"    success: (data, textStatus, jqXHR) ->      body = data['body']$('.ajax_body').append(body)

And now, all hell breaks loose. If you allowed some outside user to save content with Page#create and you DID NOT manually sanitize the data your client is screwed, because different from Rails View Templates, the #to_json method does not sanitize the JSON it generates and the application is open to XSS attacks.

The main pattern is when you have a vanilla Rails app and you quickly convert it to be used by some fancy SPA (Single Page App) that loads data from your quickly built API endpoints and fail to sanitize the data before appending to the browser's DOM.

Fix 1: Sanitize in the Server-Side rendering

The easiest fix for this particular problem is to remember to sanitize whatever comes out of your app. The Rails View Templates takes care of this automatically and we are spoiled by it. So we forget what needs to be done to manually sanitize a user inputted text:

123456789101112131415

classPagesController< ApplicationController  before_action :set_page, only: [:show, :edit, :update, :destroy]  include ActionView::Helpers::SanitizeHelper  include ActionView::Helpers::JavaScriptHelper  ...defshow    ...    respond_to do |format|      ...      format.json do        json_body = escape_javascript(sanitize(@page.to_json(only: [:title, :slug, :body])))        render json: json_bodyendendend

Now, when you call the JSON URI "http://localhost:3000/pages/1.json", for example, you will receive this sanitized version:

1	{\"title\":\"Hello World\",\"body\":\"\\u003cscript\\u003ealert(\'XSS\')\\u003c/script\\u003e\"}

Instead of the previously tampered raw body:

1	{"title":"Hello World","body":"\u003cscript\u003ealert('XSS')\u003c/script\u003e"}

Fix 2: Always add a redundant XSS check in the Client-Side

Even if you're given guarantees that the API you're consuming always provides safe data, you can't be too sure. Always assume that data that comes from outside of your domain might come compromised. It only takes one breach, one time.

So, in the client-side I believe one of the ways is to use the "xss" module. In a Rails app, the canonical way to add it is to use Rails Assets:

1234	# in your Gemfilesource 'https://rails-assets.org'do gem 'rails-assets-xss'end

12345678

// in your app/assets/javascripts/application.js//= require jquery//= require jquery_ujs//= require turbolinks//= require xss//= require_tree .

And now, in the previously vulnerable "pages.coffee":

$ ->  $.ajax    url: "/pages/" + $("meta[name=page]").attr("id") + ".json"    success: (data, textStatus, jqXHR) ->      body = filterXSS(data['body'])      $('.ajax_body').append(body)

Assume that all data that comes from Ajax endpoints could be tampered, so always filter against XSS, specially if you're going to append the result into your user browser's DOM.

I believe most modern SPA frameworks such as Ember already checks for that, but check the Model documentation of your favorite framework to be sure.

Fix 3: Filter all User Input, regardless. Use Rack-Protection

The usual workflow is for the Rails app to receive raw data from the user, store it in the Model table and when it needs to be rendered, let the View layer make the sanitization. Specially because if you need to parse the user data from the database, you would have to de-sanitize first.

But if you really don't care about the raw user input (you're not making a Content Management System) and you only really care about the plain text, then you should sanitize all user input by default.

One way to do it is to put the sanitization in the Rack Middleware layer directly, so your app only receives filtered data.

12	# in your Gemfilegem 'rack-protection'

Then add this to your "config/application.rb" application block:

1	config.middleware.use Rack::Protection::EscapedParams

And that's it!

Without this middleware, this is what a vanilla form would receive from the user when he posts a javascript into the controller:

Started POST "/pages" for 127.0.0.1 at 2016-02-22 13:15:58 -0300Processing by PagesController#create as HTML  Parameters: {"utf8"=>"✓", "authenticity_token"=>"...", "page"=>{"title"=>"Hello 5", "body"=>"<script>alert('XSS 5')</script>"}, "commit"=>"Create Page"}

You can see the raw javascript that, if gone unchecked through an API, will execute in the user browser after an uncaring Ajax fetches it.

Now, with the Rack Protection middleware, your Rails app will not receive the raw javascript, instead it will come pre-escaped.

Started POST "/pages" for 127.0.0.1 at 2016-02-22 13:16:44 -0300Processing by PagesController#create as HTML  Parameters: {"utf8"=>"✓", "authenticity_token"=>"...", "page"=>{"title"=>"Hello 6", "body"=>"&lt;script&gt;alert(&#x27;XSS 6&#x27;)&lt;&#x2F;script&gt;"}, "commit"=>"Create Page"}

I believe there are clever ways to try to fool the escape process by using some combination of special characters, but this should cover most cases.

What about Brakeman and other security scanners?

If you use Brakeman it will usually warn you if you try to inject user data into a View Template or SQL query without proper sanitization. But because this is a cross-app scenario, the server app will be flagged as "secure", and you will not notice it until too late.

So the recommendation is: do ALL 3 things I listed above.

Always manually sanitize your JSON APIs in the server-side.
Always manually sanitize your Ajax fetches in the client-side.
Always add Rack Protection (see the documentation for other protection other than the EscapedParams and also check the Rack-Attack for further protection).

Those are all easy to add security layers, and this is only one vector of attack, so better to cover all basis.

As a bonus: do not forget to install "Bundler Audit" to check if you're not using vulnerable gems, many XSS and other vulnerabilities come from dependencies that you're not even aware of. So run Brakeman and Bundler Audit regularly against your application as well.

You can never be too safe. Security is not binary, you're vulnerable by default. There is no such as thing as "vulnerability-free" app, no matter how many audits you ran. There are only vulnerabilities that you didn not find yet, but they're there, be assured of that.

This is a checklist. You should follow these instructions if you want some peace of mind after deploying an application to production. If you're experienced, please contribute in the comments section below. But if you're a beginner, you should pay attention.

Where to Deploy? TL;DR: Heroku

Many people will first target the cheap choices. Usually EC2 micro instances or small Digital Ocean droplets. Unless you're aware that you must constantly monitor and upgrade those instances, DON'T do it.

Instead choose a PaaS solution such as Heroku. It's a no-brainer. Software in production is not a one-off thing. For security alone you should not do it by yourself unless you know how to harden a OS distribution. For example, most people doesn't even know what fail2ban is. Don't assume that because you use SSH with keypairs you're secure. You're not.

For example, did you know that we just went through a glibc security crisis? If you don't and you didn't touch your production servers in some time to upgrade all your components, you're open, wide open. I had many clients that hired freelancers to implement their apps and deploy and then stayed without anyone checking. You're all unsafe.

Heroku is no guarantee either, because app libraries get old, vulnerabilities get discovered, and you should upgrade them. Rails has an active security team releasing security patches all the time. If you haven't been upgrading, you're unsafe.

But 24x7 support is not cheap, if you have your own infrastructure or use an IaaS platform such as AWS EC2, you should have 24x7 personnel, this means at least 2 full-time system administrators/devops taking care of things. Or, you use a PaaS that at least keep those basic sanitations in check and a part-time developer to at least keep upgrading your apps.

If you don't, you're opening your business to serious liability. And if you're a developer, do the responsible thing: disclose this fact to your clients and let them make an informed choice. They can choose to stay in the cheap, but at least they know what this actually means.

Which web server? TL;DR: Passenger or Puma

Most Rails apps lack basic libraries that should be installed to make them work correctly.

I've seen people running production apps by just firing up rails server and nothing else! This is running the basic Webrick web server, in production! You must choose between Puma, Unicorn, Passenger. Unicorn actually does not protect your app against slow clients, so avoid it if you can.

If you're in doubt, install Passenger and call it a day.

The most common choice nowadays is using Puma. But if you're running under Heroku or using small instances, keep in mind your RAM usage. A normal Rails application consumes at least 150Mb to 200Mb or more depending on the size of your app. The smallest Heroku dyno has only 512Mb of RAM, that means you should not configure Puma or Unicorn to have more than 2 instances. I'd recommend going for the 2X Dyno (1GB) and have at most 3 instances (which means 3 possible concurrent requests being processed at once).

It's not uncommon for your app to keep increasing RAM usage over time and once every couple of days you see R14 happening, then your dyno reboots and it starts working better for a while until it reaches the ceiling again.

If you don't know why that it's happening, and it's infrequent, you can temporarily use Puma Worker Killer - if you're using Puma, of course - to keep monitoring your instances until they reach a certain lower ceiling and they automatically recycle before your users see the R14 errors.

You may also have heard of Out-of-Band GC. Just be warned that Puma in threaded mode won't work well with it. If you're using Unicorn, by all means do use it.

Measure, Measure, Measure. TL;DR: New Relic

Heroku Metrics

You should always monitor your app. Be it with custom installed software such as Nagios, Munin, Zabbix, Zenoss, Ganglia, etc or at the very least the metrics that Heroku provides you, as in the image above.

If your instances are behaving strangely, you should take a look at the warnings. A frequent set of R14 errors means that your app's memory consumption is growing overtime and blowing the maximum amount of RAM available in your instance (a probable memory leaking). At the very least this means that you should decrease the number of concurrency/workers in your Puma/Unicorn and also increase the number of dynos to compensate if you do have enough traffic to justify it. But this is one metrics-oriented action that you can take because you have the information to make an informed decision.

But you should also be storing and indexing your logs using custom installed software such as Greylog2 or Logstash or using a SaaS alternative such as Papertrail. There are many reasons why an app might be leaking, of why everything seems fine in the metrics but you're having random errors. Always query the logs for answers.

Better yet, install New Relic APM. You will have both the metrics, the automated alerts, the errors measurement and really deep diagnostics to see exactly what line in your source code is guilty for your poor performance or frequent errors or erratic behavior. If you're in doubt, just install it, the basic plan is free anyway.

Some Rails Dependencies

You should NOT be using Rails below version 4.0 by now. 3.2 is still widely used but it will stop being supported soon. Do not use Rails 5.0 right now unless you're experienced enough to know what to do. Anything from 4.0 to 4.2 should be ok, but be aware of the minor releases that fixes security vulnerabilities.

You should NOT be using Ruby below version 2.2 by now. Security upgrades for 2.0 ended this February 2016. 2.1 was not a good stable release, but 2.2 and 2.3 are very good and you should be using them.

So the combination of Ruby 2.3.0 and Rails 4.2.5.2 (by March 2016) is the optimal choice for best stability, security and ecosystem support.

You should have at least the following dependencies installed by default:

Rails 12 Factor gem
Rack-Cache middleware
- Dalli gem with Memcached or Memcachier - even if you're not using fragment caching or AR caching, this is still useful at least for the Rack-Cache gem.
Rack-Attack - read how to properly configure!
Rack-Protection - read how to properly configure and what protections you want to activate, not everything, but at least CSRF and XSS.
Sprockets 3.3+ - make sure you're using the Sprockets version that fixes the slow speeds and makes it reasonably faster

Which Database? TL;DR: use Postgresql

Do NOT choose MongoDB or similar unless you have a very strong reason to and you are confident that you're very good at what you're doing and you will be sticking around to maintain what you deploy. Otherwise stick to Postgresql and the "pg" gem, you won't regret it, nor will your client.

Do use small Redis instances (particularly SaaS options such as Heroku Redis ) with Sidekiq. You can choose to use the lighter Sucker Punch but be aware that it will increase the RAM usage of your instances considerably depending on the size of your queues and the serialized input data for the workers. It's now always trivial how many Sidekiq workers you should have, but this calculator could help.

There are at least 2 big mistakes most people make using Sidekiq:

Having heavy workers hitting your master database. For example a big analytics procedure fetching large sets of data while your main website also have heavy users usage writing data all the time. Definitely add a Follower database and point the Sidekiq workers (read-only) there for analytics, reports and similar workloads.
Fetching large datasets in-memory and blowing your worker dynos. If you do anything similar to "YourModel.all.each { |r| ... }" you're shooting yourself in the foot. Rails has easy ways to fetch small sets in batch for you to process, use them.

Are you using a CDN? TL;DR: just do

This is easy, use AWS CloudFront or Fastly or something else, but USE a CDN. It's a no-brainer.

Otherwise your assets will be served by your already small web dyno instances. And this is a heavy lifting that are better served from a much faster CDN. Your users will perceive orders of magnitude in usability by changing just a couple of configuration lines in your code.

If you implemented your Rails views, templates, stylesheets using the "image_tag", "asset_path" helpers, it's just automatic. Just do it already.

And while we're in the subject of assets, if you happen to have image uploading and you're using the simple way of posting a multipart form directly to your Rails app, you will have trouble because Heroku imposes - a correct - maximum of 30 seconds of timeout. If you have many users uploading many high-resolution and heavy sized images, they will see increasing amounts of timeouts.

Use the Attachinary gem to direct upload via Javascript, from the browser, to the Cloudinary service, completly bypassing your Rails app, which will just receive the ID once the upload finishes. Again, there is a very generous free plan and it's so simple to implement that you must do it right away.

You laid off your developer, what to do? TL;DR: revoke accesses!

This is something that happen a lot: a small company needs to cut costs and lay off developers.

Make sure you revoke accesses. In AWS it's a lot more convoluted if you don't know your way through their IAM authorization tool. Or worse: if they have the private keys (.pem files) to your EC2 instances!! You will have to create new keypairs, deactivate the old EC2 instances and create new ones. A royal PITA.

You must change passwords to your DNS service, Github/Bitbucket organization, remove SSH keys from the .ssh/authorized_keys files in all your servers, etc.

Make sure your company has good legal support and make every collaborator, employee or 3rd party, sign a non-disclosure agreement, non-compete or confidentiality agreements.

You MUST have at least one key person in the company that has control over the keys, passwords and access to every service your company depends on to run. Otherwise don't be surprised when your app goes offline.

And don't fool yourself into thinking that just because an app is online and running it will keep that way for long. Security vulnerabilities are on the wild, it's just a matter of time until some glitch put your service down and you better have someone to put it back online for you if you don't know how to do it.

Big Site? Congratulations, what to do now?

If your business is thriving, blooming, congratulations. You're in the very very small percentage that actually made it.

Soon your brand new app will become that ugly, dreaded Legacy system. Worse, it's probably one of those feared monoliths or even worse, you let a hipster developer create a spaghetti architecture full of unchecked microservices. You're doomed either way.

This situation requires you to be grounded: there is no silver bullet, just business as usual.

You must keep ahead of the game, and the monitoring part I mentioned plays a big part to steer you through the many possible directions you can go. For example, for a short period of time you can increase the plans of your SaaS services such as databases, caches, queues, and also add more parallel web dynos.

While that will hold for a while (your metrics can help you estimate for how long), now you need to come up with a plan to optimize what you have. And no, rewriting everything is that very last thing you want to do, unless you intend to not have a life for the next couple of years.

Trust Paretto: in real life, 80% of the main problems can be solved with 20% of the effort. The main big tip: open New Relic - now loaded with real production data - and check the Top worse performers and start from there. The first one you solve (be it an ugly N+1 query to your database, slow web service integration), will give you a lot of room to breath.

Conclusion

What I just said is nothing but the tip of the iceberg in terms of Rails performance optimization. And I can assure you that there is a lot of room to improve without having to resort to rewriting everything in the next hip language out there. I recommend you buy Nate Berkopec's "The Complete Guide to Rails Performance" to really learn your turf.

This is a very quick compilation that I just dumped out of my mind, but there is much more that we can consider. Shoot questions in the comments section and other important items that I may have overlooked.

Everybody is into microservices. There is no way around it. In the Rails world we are well equipped to satisfy any trending Javascript Framework's crave for API consumption.

In summary, most people are just exposing their contents through simple JSON API endpoints and consuming them from other microservices through simple HTTP GETs. The more microservices they add to the chain, the longer the last endpoint takes to return. There are many techniques to improve this situation, but I want to show just a simple one that can solve many common situations without too much hassle.

First of all, if you're dealing with caching, never try to expire cache entries. The most important thing to learn about caching is how to generate proper cache keys. Do it right and most problems with caching are gone.

Second, if you're using HTTP, try to use everything you can from it, instead of reinventing the wheel.

The "TL;DR" version is: make your APIs return proper ETAGs and handle "If-None-Match" headers properly, return the correct 304 status code instead of full blown 200 with full content body in the responses everytime. And in the consumer end, cache the ETAG with the corresponding response body and use it from cache when you receive 304s. You will save at least expensive rendering time in the consumed end and slow bandwidth from the consumer end. In the end you should be able to at least be 100% faster, or more, with just a few tweaks.

The Example Applications

In a very very contrived example we could have a Rails API controller like this:

12345678

## 1st applicationclassApi::ProductsController< ApplicationControllerdefindex@products = Product.page (params[:page] || 1)    render json: [num_pages: @products.num_pages, products: @products]end  ...end

For the purposes of this contrived post example, we load it up on localhost port 3001. Now, whenever we call "http://localhost:3001/api/products?page=1" the API server dumps something like this in the log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 13:29:34 -0300Processing by Api::ProductsController#index as */*  Parameters: {"page"=>"1"}   (0.3ms)  SELECT COUNT(*) FROM "products"  Product Load (0.9ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0Completed 200 OK in 26ms (Views: 23.0ms | ActiveRecord: 1.2ms)

In summary, it's taking around 26ms to send back a JSON with the first page of products of this application. Not too bad.

Then we can create another Rails API application that consumes this first one. Something also very contrived and stupid like this:

1234567891011

# 2nd applicationclassApi::ProductsController< ApplicationControllerdefindex# never, ever, hard code hostnames like this, use dotenv-rails or secrets.yml    url = "http://localhost:3001/api/products?page=?" % (params[:page] || "1")    json = Net::HTTP.get_response(URI(url)).body    response.headers["Content-Type"] = "application/json"    render plain: jsonend  ...

We load this other app in localhost port 3000 and when we call "http://localhost:3000/api/products?page=1" the server dumps the following log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 13:31:59 -0300Processing by Api::ProductsController#index as HTML  Parameters: {"page"=>"1"}  Rendered text template (0.0ms)Completed 200 OK in 51ms (Views: 7.1ms | ActiveRecord: 0.0ms)

Now, this second application is taking twice the time compared to the first one. We can assume that part of those 51ms are the 26ms of the first application.

The more APIs we add on top of each other, the more time the entire flow will take. 26ms for the first, another 25ms for the second, and so on.

There are many things we could do. But I'd argue that we should start simple: by actually using a bit more of the HTTP protocol.

Sending proper ETAGs and handling "If-None-Match"

In a nutshell, we can tag any HTTP response with an ETAG, an identifier for the content of the response. If the ETAG is the same, we can assume the content hasn't changed. Web browsers receive the ETAGs and send them back if we choose to refresh the content as a "If-None-Match" header. When a web server receives this header it compares against the ETAG of the response and doesn't send back any content, just a "304 Not Modified" HTTP header, which is much, much lighter and faster to transport back.

An ETAG can be as complicated as an entire SHA256 hexdigest of the entire response content or as simple as just the "updated_at" timestamp if this indicates that the record has changed (in a "show" controller action, for example). It must be a digest that represents the content and it must change if the content changes.

Rails has support for ETAGs for a long time in the ActionController::ConditionalGet API.

In our contrived example, the 1st application on port 3001 fetches a page of ActiveRecord objects and send back an array represented in JSON format. If we choose to digest the final content we would have to let ActionView do it's job, but it is by far the most expensive operation so we want to avoid it.

One thing that we could do is just check the "updated_at" fields of all the records and see if they changed. If any one of them changed, we would need to re-render everything and send a new ETAG and a new response body. So the code could be like this:

1234567891011121314151617

classApi::ProductsController< ApplicationController  layout falsedefindex@products = Product.page (params[:page] || 1)if stale?(freshness @products)      render json: [num_pages: @products.num_pages, products: @products]endend  ...  privatedeffreshness(collection)    dates = collection.map(&:updated_at).sort    etag = dates.map(&:to_i).reduce(&:+)    {etag: Digest::MD5.hexdigest(etag.to_s), last_modified: dates.last, public: true}end

Now, when we try to "curl -I http://localhost:3001/api/products\?page\=1" we will see the following headers:

1234567891011121314

HTTP/1.1 200 OK X-Frame-Options: SAMEORIGINX-Xss-Protection: 1; mode=blockX-Content-Type-Options: nosniffEtag: "ccf372c24cd259d0943fa3dc99830b10"Last-Modified: Wed, 23 Mar 2016 16:25:53 GMTContent-Type: application/json; charset=utf-8Cache-Control: publicX-Request-Id: 601f22bc-72a9-4960-97cb-c30a0b56dbf4X-Runtime: 0.053529Server: WEBrick/1.3.1 (Ruby/2.3.0/2015-12-25)Date: Wed, 23 Mar 2016 17:00:13 GMTContent-Length: 0Connection: Keep-Alive

Great! We have an ETAG that uniquely represents this page of products. Now we can go one step further and add the following:

1234567891011121314

# Gemfilegem 'dalli'gem 'rack-cache'# config/environments/[development|production].rb...config.cache_store = :dalli_storeclient = Dalli::Client.newconfig.action_dispatch.rack_cache = {:metastore    => client,:entitystore  => client}config.static_cache_control = "public, max-age=2592000"...

This configuration is assuming that we have Memcached installed and running in the same localhost machine (our development environment), but in production you can follow this good documentation from Heroku.

Now, our 1st application has an internal HTTP cache, with the same role as something more advanced such as Varnish in front of it. It will cache all HTTP 200 responses from the application, together with the ETAGs. Whenever a new call comes for the same URI, it will check the cache first, and if the application sends back the same ETAG, it will send the content back from the cache.

So if we call the above "curl" command multiple times, we will see this from the Rails server log:

12345678

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:05:16 -0300Processing by Api::ProductsController#index as */*  Parameters: {"page"=>"1"}  Product Load (0.8ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0  Couldn't find template for digesting: products/index   (1.0ms)  SELECT COUNT(*) FROM "products"Completed 200 OK in 31ms (Views: 16.3ms | ActiveRecord: 1.8ms)cache: [HEAD /api/products?page=1] miss, store

Notice the last line: it says that it tried to find the returning ETAG in the cache and it "missed", so it "stored" the new content. Now, if we run the came "curl" command again, we will see this:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:05:59 -0300Processing by Api::ProductsController#index as */*  Parameters: {"page"=>"1"}  Product Load (0.5ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0  Couldn't find template for digesting: products/indexCompleted 304 Not Modified in 12ms (ActiveRecord: 0.5ms)cache: [HEAD /api/products?page=1] stale, valid, store

The simple curl command is not sending the "If-None-Match" header, so it expects to receive the full response body. But because we have Rack Cache it is adding the "If-None-Match" ETAG digest from the cache to the request before hitting the Rails app. The Rails app now compares the received "If-None-Match" digest through the "stale?" methods with the ETAG it just computed and because they match, it just send an empty body response with the status code of 304. Rack Cache receives the 304 and fetches the cached JSON from Memcached and changes the HTTP response from the 304 to a normal 200 with the full body, which is what Curl can receive.

Because we are skipping the expensive ActionView rendering, the response time went from the previous 26ms to around 12ms: we are now twice as fast!

Consuming APIs with ETAGs

But we can go one step further. If we change nothing about the 2nd application, it will keep receiving just HTTP 200 with full body responses from Rack Cache of the 1st application. Let's see the code again

1234567891011

# 2nd applicationclassApi::ProductsController< ApplicationControllerdefindex# never, ever, hard code hostnames like this, use dotenv-rails or secrets.yml    url = "http://localhost:3001/api/products?page=?" % (params[:page] || "1")    json = Net::HTTP.get_response(URI(url)).body    response.headers["Content-Type"] = "application/json"    render plain: jsonend  ...

We can do better. How about the following:

123456789101112131415161718192021222324252627282930313233343536373839404142

# 2nd application - upgrade!classApi::ProductsController< ApplicationControllerdefindex    page = params[:page] || "1"    url = "http://localhost:3001/api/products?page=?" % page# 1 - fetch the ETAG for the URL    etag = Rails.cache.fetch(url)# 2 - fetch from external API or fetch from internal cache    json = fetch_with_etag(url, etag)    response.headers["Content-Type"] = "application/json"    render plain: jsonend  ...  privatedeffetch_with_etag(url, etag)    uri = URI(url)    req = Net::HTTP::Get.new(uri)# 3 - add the important If-None-Match header    req['If-None-Match'] = etag if etag    res = Net::HTTP.start(uri.hostname, uri.port) {|http|      http.request(req)    }    etag = res['ETAG']    etag = etag[1..-2] if etag.present?if res.is_a?(Net::HTTPNotModified)# 4 - if got a 304, then we already have the content in the internal cacheRails.cache.read(etag)elsif res.is_a?(Net::HTTPSuccess)# 5 - if we got a 200 it's new content to store in internal cache before returningRails.cache.write(url, etag)Rails.cache.write(etag, res.body)      res.bodyelse"{}"endendend

I know, feels overwhealming, but it's actually quite simple. Let's go over it step-by-step:

First we see if we already have an ETAG associated to the URL we want to fetch (be aware of query parameters!)
Now we call the separated "fetch_with_etag" method
This is all boilerplate "Net::HTTP" setup, but the important piece is that we add the "If-None-Match" header if we found an ETAG for the URL in the cache.
After we make the external call we can have 2 responses. The first being the very very short, body-less, header-only, "304 Not Modified". In this case, it means that we already have the full content in the internal cache and it is still valid, so we use it.
In case we receive the normal HTTP "200" status code, we either didn't send any ETAG or the one we sent was invalidated and a new ETAG and content body was returned, so we must update them in our internal cache before exiting.

Now, the first time we call "curl http://localhost:3000/api/products\?page\=1" for the 2nd application endpoint we will see this log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:14:05 -0300Processing by Api::ProductsController#index as */*  Parameters: {"page"=>"1"}  Rendered text template (0.0ms)Completed 200 OK in 62ms (Views: 5.6ms | ActiveRecord: 0.0ms)

Caches are cold, it is taking the same "around 50ms" like we had before, in this case, it's more like 62ms.

Just to recap, this call to the 2nd application made it call the 1st application API, which shows the following it its log:

123456789

Started GET "/api/products?page=?" for 127.0.0.1 at 2016-03-23 14:14:05 -0300Processing by Api::ProductsController#index as */*  Parameters: {"page"=>"?"}/"32b82ebbd99854ea2ca0d49ff4a7c07c  Product Load (0.9ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0  Couldn't find template for digesting: products/index   (1.2ms)  SELECT COUNT(*) FROM "products"Completed 200 OK in 37ms (Views: 21.5ms | ActiveRecord: 2.2ms)cache: [GET /api/products?page=?] miss, store

Cache miss, new content stored!

Now, we call "curl" against the same URL for the 2nd application again and we now see what we wanted in the log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:14:10 -0300Processing by Api::ProductsController#index as */*  Parameters: {"page"=>"1"}  Rendered text template (0.0ms)Completed 200 OK in 24ms (Views: 0.3ms | ActiveRecord: 0.0ms)

Down from 62ms to 24ms!! And in the 1st application log we see:

12345678

Started GET "/api/products?page=?" for 127.0.0.1 at 2016-03-23 14:14:10 -0300Processing by Api::ProductsController#index as */*  Parameters: {"page"=>"?"}"ccf372c24cd259d0943fa3dc99830b10", ccf372c24cd259d0943fa3dc99830b10  Product Load (1.2ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0  Couldn't find template for digesting: products/indexCompleted 304 Not Modified in 12ms (ActiveRecord: 1.2ms)cache: [GET /api/products?page=?] stale, valid, store

A cache hit! Content is stale and valid, so return just 304, the 2nd application acknowledges and fetch the still valid content from its own cache and return to Curl.

Conclusion

If you remove ETAGs from the 1st application, the 2nd one will not break and vice-versa, because it's optional. If "ETAG" and "If-None-Match" headers are present in received HTTP response, we can use, otherwise they will work as before.

If the 2nd application is itself another API you should also add ETAGs for it, and so on. In this example we didn't, just because I wanted to simplify the scenario. But instead of being just a simple one-to-one proxy, it could be one of those "porcelain" APIs that fetch data from several other smaller microservices, compile down in a single structure and return it. You should create ETAGs that could be the returning ETAGs from all the other microservices digested together in a single ETAG, for example. Because you're just receiving headers and fetching their content from an internal cache, it's quite cheap. Something like this pseudo-code:

12345678910111213

defindex  url1 = "http://somehost1.foo/some_endpoint/1"  url2 = "http://somehost1.foo/some_endpoint/2"  etag1 = etag_from(url1)  etag2 = etag_from(url2)  etag = Digest::MD5.hexdigest(etag1 + etag2)if stale?(etag: etag, public: true)    body1 = JSON.parse fetch_from(url1)    body2 = JSON.parse fetch_from(url2)    result = do_some_processing(body1, body2)    render json: result.to_jsonendend

Another thing: you can add any vanilla HTTP Cache between your microservices, to add authorization, security, or just plain extra caching, it's just HTTP with proper headers. But the more you exchange "304" between your services, the less processing and the less bandwidth you're spending. It should be noticeably efficient in most cases. But again, it's not always cheap or trivial to generate the cache keys/ETAGs to begin with, so this is the point to take more care.

And if you're creating heavy Javascript apps that also consume those APIs, I "believe" the Ajax calls properly cache HTTP content and send back the correct "If-None-Match" and in case they receive 304s, your application should get the normal "success" triggers. I didn't test this when I was writing this post but I think this is the case indeed. So you should automatically get better performance in your front-end application for free if you add proper ETAGs in your APIs.

This is particularly useful for APIs that return data that don't change too often. If it changes every second, or every minute, you should not see too much gains. But if it's something like this example: products lists that only change once every day or every week, or ZIP code lists, or Previous Orders in an e-commerce. Any data that change infrequently is a good candidate. And the larger the dataset, the larger the benefits you will see (if it's a megabyte long listing, for example). As usual, this is also no Silver Bullet, but in this case it is not so much work to add ETAGs and there are near to zero side-effects, so why not?

ETAG is just one of many other HTTP feature you should be using, CORS is another one (Research Rack Cors).

If you're from Brazil, you should watch Nando Vieira's entire course on the broad subject of Rails Caching.

To be honest, I'm not sure how effective this technique can actually be in all kinds of scenarios so I am very interested in hearing your feedback in case you use something like this in your applications.

Atualização 07/04: o preview já saiu antecipando o lançamento em Junho. Se você for membro do programa Insider, baixe e teste já. Deixe suas impressões nos comentários.

Ontem foi um dia histórico para a Microsoft, eu acho que pode ser um divisor de águas - na minha visão pessoal, pelo menos. Desde ontem estou tentando pensar em como deixar isso registrado para a posteridade.

Prólogo - 1995 a 2005, 1a Década

Digo isso para nós, desenvolvedores de software. Conhecemos alguns divisores assim na história. Por exemplo, o lançamento do Turbo Pascal em 1983, o lançamento do Visual Basic em 1991, o lançamento do Java em 1995, as primeiras distros de Linux como o Slackware a partir de 1993, o lançamento do OS X em 2000 e a transição de PowerPC para Intel em 2005.

Os anos 80 e 90 foram conturbados, uma época de crescimento acelerado, tudo ficava obsoleto muito rápido. Numa década, de 1989 até 1999, eu passei de Basic, pra dBase, pra Clipper, pra Fox Pro, pra Visual Basic, pra Delphi, pra Java, pra multimídia (Flash, Director/Shockwave), pra Internet (HTML, Javascript, ASP, PHP, Perl). Nunca experimentei uma década onde as coisas andaram tão rápido. Fui de um computador XT de 4Mhz pra Pentium III de quase 1Ghz nessa 1 década.

Chegamos ao final do século, ao ápice da Bolha da Internet, tivemos 9/11 em 2001 e o crash. Mas a Internet chegou para ficar e dispositivos começaram a despontar com os Palms e iPods. Em 2001 a Microsoft lançou seu Windows XP e amargaria a próxima década levando seu legado nas costas. Ao contrário, um ano antes a pequena Apple iniciara seu levante, os novos iMac, iBook e iPod, o novo OS X baseado em BSD UNIX e um passo de cada vez em criar um sistema perfeito para consumidores e desenvolvedores, a promessa da NEXT criando forma uma década depois de sua incepção.

Então, em 2005 a Apple forjaria aliança com uma das mais antigas companheiras da Microsoft, a Intel e finalmente Macs rodariam no mesmo processador que o Windows, dando um salto estratosférico em adoção. Esse salto ainda possibilitaria mais: já que o OS X foi feito agnóstico de processador, era questão de tempo até rodar em processadores ARM, menores e mais eficientes, e foi o que aconteceu em 2007 com o lançamento dos iPhones, o resto é história.

2005 a 2015 - 2a Década

Enquanto isso a Microsoft ficou pra trás, teve o fracasso com a promessa do Windows Vista, um Windows 7 até competente com a sequência do controverso Windows 8. Finalmente, com o lançamento do Windows 10 no final de 2014 é que as coisas começaram a mudar. Os efeitos da morte de Steve Jobs no final de 2011 começaram a ficar mais aparentes: a Apple parou.

De 2011 até hoje tivemos novas versões, apenas. Nada significativo. Nada novo. O último grande iPhone foi o 5 de 2012. O último grande felino do OS X foi o Mountain Lion de 2011. Enquanto isso o Google passou à frente no mundo mobile quando o KitKat foi lançado em 2013, pra mim a primeira versão que ficou em nível de igualdade com os iOS.

Em resumo, de 1999 até 2005, no geral, havia 2 opções para desenvolvedores de software enterprise: Java e .NET. Sim, PHP tinha boa adoção mas na maioria como customizações de Wordpress e Magento, não para a mesma categoria que Java ou .NET e o resto era nicho. Mas a partir de 2006 iniciou uma nova geração de desenvolvedores para a chamada Web 2.0, o mundo pós-Google, SEO, UX, iterações rápidas, Agilidade com "A" maiúsculo. Foi onde Ruby, Python, depois Node aceleraram exponencialmente.

A partir de 2010 a mundo mobile acelerou e, em particular, um tipo de desenvolvimento integrado: baseado em "Cloud" onde uma "app" nativa fala com serviços em servidores remotos, particularmente da Amazon AWS, que estreou não fazia muito tempo, em 2006.

A Microsoft ficou de fora desse movimento e teve que gastar os 10 anos seguintes à ressaca do Vista correndo atrás. Mantendo a liderança em mercados puramente enterprise com licenças de SQL Server, Office, mas à margem do "cutting-edge" do desenvolvimento de software.

Na verdade eu diria que o divisor de águas veio em 2013, com a saída de Ballmer e o fim da "Era Extendida de Gates" com a entrada de Satya Nadella. Obviamente a Microsoft é uma entidade gigante e muitos cérebros já pesquisavam fazia alguns anos e muito disso só veio aparecer agora. Mas digo "Eras" mais no sentido simbólico, o fim da cultura Highlander da Microsoft "there can only be one".

A profecia de 1995 de que a Internet destruiria a hegemonia do Windows aconteceu. E no mundo pós-Moore onde a computação se tornou não só acessível como ubíquita. Pense nisso: em 1995, ter acesso a um cluster de "super" computadores para realizar computação distribuída de alto processamento era algo apenas acessível em universidades ou grandes corporações. Hoje? Crie agora mesmo um cluster de quantas dúzias de computadores quiser no EC2 por um mês pelo equivalente a um videogame.

Eu fui usuário de DOS desde 1988, mudei para Windows 95 logo que foi lançado e passei quase 10 anos primariamente desenvolvendo em Windows em ambiente corporativo. Passei de Visual Basic para ASP para .NET ao mesmo tempo que fazia Java, PHP. Mas eu queria poder usar Perl, PHP, Python, Ruby nativo de Linux. Mas também não queria abrir mão de Office, Photoshop.

A resposta que já esperávamos era o OS X. Com o lançamento do Tiger a coisa ficou séria. Eu mesmo experimentei com o Mac Mini G4 em 2004. Era excepcional ter uma usabilidade muito melhor que Windows XP e acesso a tudo de UNIX que queria numa única máquina.

Quando a Intel lançou o Core Duo e depois o Core 2 Duo de 64-bits e o Tiger passou a rodar em Intel em 2005, não havia o que pensar: era hora de migrar. Agora eu passaria a década seguinte em ambiente puramente OS X e desenvolvendo em outro tipo de plataforma, com outros tipos de ferramentas e outro tipo de cultura de desenvolvimento. Foi toda minha história com Ruby e OS X nos últimos 10 anos, que vocês devem conhecer bem se acompanham este blog.

2015 em diante, Epílogo

2015 chegou. 4 anos depois da morte de Steve Jobs. Foi um gap importante porque deu tempo para as outras plataformas não-Mac finalmente se equipararem e superarem em muitas áreas. Estamos definitivamente na Era do Software como Serviço. Não "compramos" mais nada, "assinamos" serviços.

Passamos da Era Pós-PC. Passamos da Era Smartphone. Estamos na Era Serviços. Tudo é um serviço. Nossa identidade é um perfil do Facebook. Nossa voz é o Zap Zap. E tudo que fazemos gera dados. Estamos na Era Big Data. Um monte de dados inúteis que nenhum SQL Server do mundo é capaz de manter.

Precisamos de coisas novas que consigam lidar com esse novo paradigma: Cassandra, Riak, Hadoop, HBase, Spark, Elasticsearch, Go, Rust, Elixir. Todos eles "rodam" em Windows. Alguém tem que gastar tempo fazendo ports, criando patches de compatibilidade, empacotando instaladores, e - o principal - a maioria dos desenvolvedores de cada uma dessas tecnologias está ou em Linux ou em OS X, quase nenhum está ativamente desenvolvendo essa nova geração de ferramentas dentro do Windows. Todos estão usando Emacs, Vim ou Sublime Text, quase ninguém quer usar Visual Studio.

E isso volta ao início do meu post: é uma nova Era para a Microsoft. Ela sofreu os últimos 10 anos correndo atrás e, pela primeira vez, parece que ela tem chances reais de dar a volta por cima e voltar a ser relevante para desenvolvedores não-.NET novamente.

Para isso 2 coisas aconteceram ontem.

2016 - O Ano do Linux no Desktop

Primeiro, o anúncio do "Bash está chegando ao Windows". Que na verdade é muito mais do que simplesmente o Bash no Windows. É uma associação com a Canonical - que já se iniciou com o anúncio do SQL Server rodando em Linux - que na prática é um subssistema Linux rodando paralelamente à kernel do Windows e traduzindo chamadas de sistema Linux para Windows. A idéia original veio no falecido Projeto Astoria ou Windows Bridge for Android, cuja idéia na época era pra permitir apps Android de rodar em Windows. Tecnicamente o Android é um tipo de Linux, daí foi um salto para ontem.

Assim será possível pegar um binário feito para rodar em Ubuntu e rodá-lo no Windows sem modificar nada!

Desde que começamos a desenvolver com Ruby, muitos que utilizam Windows gostariam de poder rodar Ruby nativamente no Windows. Existe uma distribuição mas não é boa, e não é culpa dos mantenedores. O Ruby, assim como Python, PHP e tudo mais, foi feito para funcionar em Linux. Para portar para Windows é preciso fazer "wrappers" e fora nao ter um ambiente com compiladores como GCC por exemplo, para compilar pacotes de extensões. Tudo isso é trivial no Linux mas uma verdadeira dor de cabeça no Windows. Não vale o trabalho, portanto a única solução era rodar uma máquina virtual (como Virtual Box) e usar algo como Vagrant para facilitar o trabalho e usar o verdadeiro Ruby de Linux.

Tudo de Linux que "funciona" em Windows, seja Python ou PHP mesmo, nunca funcionou 100%, e nós simplesmente nos acostumamos a isso.

O que a Microsoft vai lançar no Anniversary Update não é uma máquina virtual, um hypervisor, é uma camada muito fina de tradução no nível mais baixo: de uma kernel Linux "virtual" para a kernel do Windows. Tudo vai pensar que está rodando no Linux. É o que o projeto Wine vem tentando fazer para rodar aplicações Windows no Linux, só que ao contrário. Também não é a mesma coisa do Cygwin, que exige recompilar tudo do Linux para ter um ambiente semi-Linux sobre o Windows.

Provavelmente não haverá ambiente gráfico secundário num primeiro momento, portanto nada de X, então nada de GNOME ou KDE. E nem precisa, estamos num mundo Web: nosso ambiente gráfico agora é HTML e Javascript e Chrome e Firefox todos existem nativos em Windows. Para coisas como compilar gems com extensões nativas, até mesmo o GCC vai funcionar! Não é uma adaptação é exatamente o mesmo binário, no mesmo ambiente, podendo contar com tudo que estamos acostumados em ambientes UNIX. Ter o Bash, ter o SSH, no Windows, é só a ponta do Iceberg.

O segundo anúncio é o lançamento das ferramentas Xamarin gratuitamente e como open source! A Microsoft adquiriu a Xamarin de Miguel De Icaza recentemente, eles desenvolveram o .NET open source por pura engenharia reversa, reimplementando tudo como open source e agora a Microsoft finalmente fez a coisa inteligente. Não somente os adquiriu como abriu suas ferramentas. A última coisa que poderia segurar um desenvolvedor em Mac seria a necessidade de desenvolver para iOS usando o XCode que já vem no OS X, mas agora você pode fazer isso em Windows gratuitamente!

Portanto, a conta ficou bem mais simples agora. O Windows 10 finalmente é a primeira versão respeitável de Windows desde o breve intervalo que foi o 7. Vamos esquecer o que 8 existiu da mesma forma que já esquecemos do Vista. O XP finalmente morreu, levou mais de uma década pra isso, junto com os horrorosos Internet Explorer 6, 7, 8. O Google nos fez o favor de tornar o Chrome ubíquito e serviços como Facebook obrigaram os usuários a migrar mais rápido também.

Com o Windows 10 Anniversary Update provavelmente vamos poder fazer:

1234	apt-get install build-essential\curl -sSL https://get.rvm.io \| bashrvm install 2.3.0bundle install

E tudo deve funcionar!

No mesmo ambiente ainda poderemos usar Xamarin Studio e com um pouco de esforço pra relembrar os músculos de C# vamos poder criar apps iOS e Android. Poderemos usar Office 365 nativo, todas as ferramentas mais recentes do Adobe CS6 e poderemos até mesmo jogar The Division, tudo na mesma máquina!

De uma só vez, o Windows 10 será a escolha com o melhor custo-benefício tanto do ponto de vista de comodidade de uso, opções de desenvolvimento, e hardware competente na forma de Surface Book, Lenovo Yoga 3, Razer Blade Stealth que finalmente alcançaram e ultrapassaram o lendário acabamento dos Macbook.

2016 finalmente é o famigerado Ano do Linux no Desktop, só que não como o povo de Free Software gostaria que fosse. Não se pode ter tudo.

Update 04/07: the preview is already out ahead of the June release. If you're an Insider member, try it.

This is the translation of my original post in Brazilian Portuguese.

Yesterday was a pretty remarkable and historical day for Microsoft, I even think it might be an Era dividing moment, at least from my personal perspective. Since then I was trying to figure out a way to register this moment for posterity.

Prologue - 1995 to 2005, the 1st Decade

By Era dividing moment, I mean it for us, software developers. We have been through moments like this before. For example, the release of Turbo Pascal in 1983, the release of Visual Basic in 1991, the release of Java in 1995, the first Linux distros such as Slackware in 1993, the release of OS X in 2000 and its transition from PowerPC to Intel processors in 2005.

The 80's and 90's were convoluted, an era of real accelerated change and growth, everything became obsolete over night. In one single decade, from 1989 to 1999, I transitioned from Basic to dBase, to Clipper, to Fox Pro, to Visual Basic, to Delphi, to Java, to multimedia (Flash, Director/Shockwave), to the Internet (HTML, Javascript, ASP, PHP, Perl). I have never experienced so much change in such a short amount of time. I went from a 4Mhz XT PC to a Pentium III with 1Ghz in this same decade.

The new century opened with the apex of the Internet Bubble, we had 9/11 and the Bubble Crash. But the Internet came, stayed and blossomed. Personal devices were born with Palm and iPod. In 2001 Microsoft released its Windows XP and it would make it suffer the bitter sweet legacy for the next decade and half. On the opposite side, a year before, the still much smaller Apple would initiate its explosive come back with the brand new and highly desirable iMac, iBook and iPod. It would launch the BSD UNIX based OS X, creating the perfect system for both consumers and developers, the missed promise neither NEXT nor the earlier Apple never achieved.

2005 would reach its peak with the also explosive revelation of the love affair between Apple and Intel, traditionally married to Microsoft, and OS X would sing on Intel processors, allowing another giant leap for Apple marketshare growth. This revealed something else: OS X could sing on PowerPC, Intel and possibly something else, being built to be agnostic. It was a matter of time until it could run on smaller and more energy efficient ARM processors, which was what happened in 2007 with the launch of the iPhone. Arguably the most successful technology launch in history.

2005 to 2015 - 2nd Decade

On the midst of all this, Microsoft ate dust from the competitors, miserably failed the Windows Vista promise, it had a dull albeit competent Windows 7, but again failed with Windows 8. Finally, with the release of Windows 10 by the end of 2014 things started to change. The effects of Steve Jobs passing by the end of 2011 are now obvious: Apple has stopped.

Since 2011 we had nothing but small incremental improvements. Nothing of significance, nothing new at all. I believe the last big iPhone launch was iPhone 5 in 2012. The last big feline release of OS X was Mountain Lion in 2011. In the meantime Google stepped ahead in the mobile front when KitKat was released in 2013, in my personal oppinion, the first time it stood on equal footing with iOS.

In summary, from 1999 until 2005, in general, we had 2 major mainstream options for software developers: Java and .NET. Yes, PHP had respectable adoption but it was never much more than customizing Wordpress or Magento most of the time, never the same league as Java or .NET, all the rest were niche groups. But from 2006 we saw a whole new generation of developers in the so called "Web 2.0", the post-Google, SEO, UX, quick iterations, Agility with capital "A". It was the dawn for Ruby, Python, then Node.

2010 saw the mobile development world blossoming and speeding up, particularly a singular kind of application: the "Cloud" variety, where a native "app" connected to remote servers to offload data and processing, particularly to Amazon AWS services. AWS started in 2006, by the way.

Microsoft played catch up all this time and stayed out of this tidal change, suffering the hungover of Vista. It kept its purely enterprise profit centers with SQL Server, Office licences, but out of the "cutting-edge".

In fact, I'd state that the Era dividing moment came in 2013 when Ballmer stepped out, closing the "Gates Extended Era" and allowing Satya Nadella to step up as the leader of the new dawn. There are obviously smart brains in Microsoft architecting all of what we are discussing now from many years back, but I say "era dividing moment" in symbolic ways, as the end of the Highlander-"there can only be one"-Culture.

The 1995 Prophecy that the Internet would destroy the Windows hegemony actually came to be. And in the post-Moore Era computation became not only accessible but ubiquitous. Think about it: to have a cluster of "super" computers back in 1995 you needed serious investments that only universites and big corporations could afford. Today? Create your own cluster of EC2 instances right now and pay the equivalent of a popular videogame console for a month of serious computation.

I was an MS-DOS user since 1988, I changed to Windows 95 since it was launched and I spent the next following 10 years primarily developing software on Windows in the enterprise (yes, I know my ways in the undergrounds of the dared Registry and Windows debugging). I did Visual Basic, ASP, .NET, Java, PHP. But I wanted to be able to use Perl, PHP, Python, Ruby, natively in Linux. But I also still wanted and needed Office and Photoshop.

The answer was clearly OS X. With the release of Tiger it became serious. I started in a Mac Mini G4 in 2004. It had exceptional usability, far ahead of Windows XP, and access to all the UNIX I ever wanted in a single box.

The Core Duo and 64-bit Core 2 Duo was decisive for Intel and Apple. Tiger would sing on Intel in 2005, then it was a no-brainer: it was time to change. Now I would spend the following decade in OS X developing software in a different platform, with different tools and culture. It was my whole Ruby story that you know well if you have been following my blog since 2006.

2015 on forward, Epilogue

2015 arrived, almost 4 years after Jobs passing. An important gap which gave enough time for non-Mac platforms to finally catch up and surpass in many fronts. We are in the Software as Service Era. We don't "buy" software no more. We "subscribe" to services.

We are passed the Post-PC Era. We are passed the Smartphone Era. We are in the Everything as a Service Era. Our very identity is but a Facebook profile. Our voice is Whatsapp or Snapchat. Everything we go create tons of data. We are in the Big Data Era. Volumes of useless information that no SQL Server could ever dream to manage.

We need new stuff to crack into this new paradigm: Cassandra, Riak, Hadoop, HBase, Spark, Elasticsearch, Go, Rust, Elixir. They all "run" on Windows. Someone has to port them, make patches, package into installers. And most - if not all - of the developers for all these technologies are using either a Linux distro or OS X. Virtually no cutting-edge developer is willingly on Windows.

And this is the closing of my post opening: a New Dawn for Microsoft. It has suffered the last 10 years playing catch up and, for the first time since forever, it seems to have a real shot in making a real comeback, to become relevant to non-.NET developers.

2 things happened yesterday that could allow for that.

2016 - The Year for Linux on the Desktop

First up, the announcement that "Bash is coming to Windows". It's actually way more than simply Bash. It's a partnership with Canonical - which started a few months back since the announcement of SQL Server running on Linux, another shocker by the way. The reality is an entire Linux subsystem running alongside the Windows kernel, a super thin translation of Linux syscalls into Windows syscalls. The original idea came from the desire of running unmodified Android apps on Windows, this is the already deceased Project Astoria or Windows Bridge for Android. Technically, Android is a flavor of Linux, so you can see how it evolved.

Therefore, one can theoretically get an unmodified ELF binary made to run under Ubuntu and run it nativelly on Windows, no patches, no recompilation, just works! Reminds me of the Rosetta component on OS X Tiger to allow unmodified PowerPC binaries to be emulated on Intel, but much more efficient as it's not a heavyweight processor emulation, and also much less convoluted than Universal Binaries, which required recompilation back in the days of migrating OS X to Intel.

Ever since we started learning Ruby, many wanted to use their Windows machines but couldn't. There is a Windows distribution of Ruby but I can't quite recommend it despite the many efforts of highly competent developers such as Luis Lavena. Ruby, Python, PHP and everything built on Linux is meant to work just on Linux, period. To make them run elsewhere requires serious hours of testing, patching, wrapping around, adapting, cutting corners, and all this needs to be done in each new release, and they will never work quite the same. Moreover, dependencies such as rubygem libraries sometimes require a compilation step of native extensions, then you have a huge headache of trying to make a compiler such as GCC work.

It's living hell, seriously.

The recommendation is to emulate Linux itself under Virtual Box with the help of fancy tooling such as Vagrant and use native, pure Ruby under the environment where it was designed to work properly. But it's a hassle, because now you have a heavy OS on top of another heavy OS and a lot of emulation and virtualization on the way. We just got used to it.

What Microsoft is promising with its Anniversary Update for Windows 10 is not virtualization, but a very very thin layer of translation in the kernel itself. Everything in user land will "think" that its running on top of a normal Linux kernel. It's the reverse of what the Wine project has been trying to do to run unmodified Windows binaries on top of Linux distros. And it's not the same as Cygwin, which requiries modifications and recompilation of all Linux tools to run on Windows.

It will also probably not support a secondary Window manager right now, so no X, no GNOME, no KDE. And we don't need them, this is a Web world, if we can fire up a web server, we can have our GUI over any commodity web browser. And for stuff such as the native extension for Ruby gems, for example, we will have the full GCC toolchain working perfectly. And again: this does not require patches, recompilation, it's unmodified binary compatibility. Native execution with full performance.

The second big announcement is Xamarin's tools being released for free and as open source. Microsoft has acquired Xamarin recently, they spent almost a decade reimplementing all .NET as open source, pure and efficient reverse engineering of the proprietary tools and libraries, and they became a force to be reckoned with, specially in the games world with platforms as Unity built on top of Mono and the more recent set of tools for iOS and Android development using C#. So Microsoft played very smart, first by acquiring them, second by releasing their tools for free. It would be the last reason to stay on the Mac: the only platform with free tools to create native iOS apps. No more.

Therefore, the choice became much simpler now than ever before. Windows 10 is finally the first respectable Windows version if we disconsider the short life of Windows 7. Let's forget that Windows 8 ever existed the same way we already forgot the Vista fiasco. XP is finally dead. Internet Explorer 6 (and 7, and 8) are also finally as good as dead. Google made us all a favor by pushing Chrome and services such as Facebook forced users to upgrade quicker as well.

With Windows 10 Anniversary Update we will possibly be able to natively do the following in a cmd.exe console:

bashapt-get install build-essential\curl -sSL https://get.rvm.io | bashrvm install 2.3.0bundle install

And it should just work!

In the very same environment we will be able to use Xamarin Studio and with some effort to remember C# from our muscle memory we should be able to create iOS and Android apps in no time. And we can use Office 365, alongside all the most recent versions of Adobe CS6 apps, and finally, we can play The Division over Steam, just for fun, on the very same machine!

In one single day, Microsoft has made Windows 10 the most compelling choice for developers. And with manufacturers finally nailing Macbook levels of machining and refinement with the incredible Surface Book or Razer Blade Stealth or even the Lenovo Yoga 3, the math is becoming more and more beneficial to Microsoft.

So, 2016 is finally the so ill-called "Year of Linux on the Desktop", just not as the Free Software folk idealized. But hey, we can't have everything. I think it's a good enough compromise. This is a very risky gambit, but if Microsoft actually deliver all this, I'm ready to leave OS X.

"The Times They Are a-Changin"

So, it's finally here. I wrote about the announcement at the Build event a few days ago here. Now Microsoft is teasing us a bit more by releasing the first Insider Build Preview. It is labeled "Build 14316.rs1_release.160402-2217" to be exact, be sure to have this one.

To get it, you must have the following requirements:

Have an activated Windows 10 64-bits
Sign up for the Windows Insider Program. If it's your first time it can take up to 24 hours to activate
In your Windows Update Settings, Advanced Options, you must choose to "Get Started" in the Inside option and also choose the "Fast" Ring option. Now, when you Check for Updates, the Preview 14316 should show up. If not, wait until your account is refreshed by Microsoft.

You must have, at the very least, 10GB of available space in your main partition, so keep that in mind. Another caveat, if you're like me and you're testing under Virtualbox, also bear in mind that to resize a virtual drive you must delete your snapshots, and the merging process can take a ridiculously long time.

Once it's installed, you must go to the Developer options in the "Update & Security" Control Panel and change your profile to "Developer Mode". This "Bash on Windows" feature will show up under the Windows Features list. It's intended for developers only not to be used in production servers.

Once you have the Preview installed you must be able to fire up the old cmd.exe console (or any other better console such as ConEmu) and type in "bash", it will prompt you to now install the userland Canonical packaged for Ubuntu. Takes another while. The whole process take a lot of time, by the way, so make sure you reserve half a day at least. Don't do it at work :-)

One problem I stumbled upon right away is that networking was not working properly. Someone narrowed it down to DNS not being added to /etc/resolv.conf so you must add it manually. Just add the Google DNS (8.8.8.8 and 8.8.4.4) to the resolv.conf file and you're up.

Finally, you should be inside bash, as a root user. So, if you're a Windows user, you must know right now that it's insecure and an anti-practice to run as root, so don't do it.

The very first thing you must do is manually create an unprivileged Linux user and add it to the sudo group as Andrew Malton blogged first:

123	useradd new_username -m -s /bin/bash passwd new_usernameusermod -aG sudo new_username

Then you can log into this user shell with su - [your user] everytime!

From here, we can assume it's a plain Ubuntu 14.04 and you should be able to follow my very old post on how to setup a developer Ubuntu environment, or any other tutorial you can find over Google. These basic development packages that I always install first all run:

sudo apt-get install curl build-essential openssl libcurl4-openssl-dev libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt-dev libgmp-dev autoconf libc6-dev ncurses-dev automake libtool bison subversion redis-server libhiredis-dev memcached libmemcached-dev imagemagick libmagickwand-dev exuberant-ctags ncurses-term ack-grep git git-svn gitk ssh libssh-devsudo dpkg-divert --local --divert /usr/bin/ack --rename --add /usr/bin/ack-grep

This should give you all compilation toolchain essentials as well as some well known command line tools such as ack (way better than grep). It also adds Redis and Memcached. They all run.

As good practice you should add the following to your /etc/bash.bashrc configuration file:

123	export LANGUAGE=en_US.UTF-8export LANG=en_US.UTF-8export LC_ALL=en_US.UTF-8

And after that you should also configure locale to UTF-8:

12	sudo locale-gen en_US.UTF-8sudo dpkg-reconfigure locales

Just to warm up a little bit, we can start with one of the things that were impossible in Windows: get a stable Ruby installation. For that, RVM is said to not work (as ZSH is also not working for /dev/pts and symlink problems in this preview, see more below). But RBENV is working! You must install it first and they the ruby-build plugin:

Interestingly, it takes a lot of time to complete and during the process CPU usage goes to the roof. Installing Go is even longer! Something is still unstable underneath as it shouldn't take the CPU to 100% for so long.

CPU through the roof

If I check how much memory I have in the system it's more clear:

root@localhost:/mnt/c/Users/fabio# free -h             total       used       free     shared    buffers     cachedMem:          1.0G       342M       664M         0B         0B         0B-/+ buffers/cache:       342M       664MSwap:           0B         0B         0B

I am not sure if this is somehow just a "hard-coded" value for memory as many people are reporting the same "664M" free regardless of what we are running. But something is incomplete here. Swap is also zero and you can't add any as far as I can tell. Swapon, fallocate, none of those work yet.

root@localhost:/mnt/c/Users/fabio# swapon -sswapon: /proc/swaps: open failed: No such file or directoryroot@localhost:/mnt/c/Users/fabio# fallocate -l 4G swapfilefallocate: swapfile: fallocate failed: Invalid argument

I actually tried to create a swap file using dd instead of fallocate and add it to the /etc/fstab but it didn't work:

123456789

root@localhost:/mnt/c/Users/fabio# free -m             total       used       free     shared    buffers     cachedMem:          1006        342        664          0          0          0-/+ buffers/cache:        342        664Swap:            0          0          0root@localhost:/mnt/c/Users/fabio# cat /etc/fstabLABEL=cloudimg-rootfs   /        ext4   defaults        0 0/swapfile       none    swap    sw      0 0

So it sounds like memory is "hard-coded". Follow this issue #92 if you want to know how it develops out.

But worse than that, shared memory has very low limits and strange behavior. Postgresql will install but won't start up at all:

123456789101112

root@localhost:/mnt/c/Users/fabio# pg_createcluster 9.3 main --startCreating new cluster 9.3/main ...  config /etc/postgresql/9.3/main  data   /var/lib/postgresql/9.3/main  locale en_US.UTF-8FATAL:  could not create shared memory segment: Invalid argumentDETAIL:  Failed system call was shmget(key=1, size=48, 03600).HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter, or possibly that it is less than your kernel's SHMMIN parameter.        The PostgreSQL documentation contains more information about shared memory configuration.child process exited with exit code 1initdb: removing contents of data directory "/var/lib/postgresql/9.3/main"Error: initdb failed

And if we try to expand the SHMMAX limit this is what we get:

root@localhost:/mnt/c/Users/fabio# sysctl -w kernel.shmmax=134217728sysctl: cannot stat /proc/sys/kernel/shmmax: No such file or directoryroot@localhost:/mnt/c/Users/fabio# echo 134217728 >/proc/sys/kernel/shmmaxbash: /proc/sys/kernel/shmmax: Operation not permitted

So, no Postgresql for the time being. Some people were able to complete the Go Lang installation (I gave up after a very very long time waiting for apt-get to finish) complained that Go also crashed on shared memory requirements. Follow the Issue #32 and Issue #146 to see if anyone can make it work.

I also tried to install Node.js. No lucky with package installs:

1234	curl -sL https://deb.nodesource.com/setup_5.x \| sudo -E bash -...E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.Error executing command, exiting

NVM installs, sort of, with many errors in git:

...error: unable to create file test/slow/nvm run/Running "nvm run 0.x" should work (No such file or directory)error: unable to create file test/slow/nvm run/Running "nvm run" should pick up .nvmrc version (No such file or directory)error: unable to create file test/slow/nvm use/Running "nvm use iojs" uses latest io.js version (No such file or directory)error: unable to create file test/slow/nvm use/Running "nvm use node" uses latest stable node version (No such file or directory)error: unable to create file test/slow/nvm use/Running "nvm use v1.0.0" uses iojs-v1.0.0 iojs version (No such file or directory)error: unable to create file test/slow/nvm use/Running "nvm use" calls "nvm_die_on_prefix" (No such file or directory)

And this is what happens if I try to install the most recent version:

1234567891011121314

akitaonrails@localhost:~/.nvm$ nvm install 5.10.1Downloading https://nodejs.org/dist/v5.10.1/node-v5.10.1-linux-x64.tar.xz...######################################################################## 100.0%tar: bin/npm: Cannot create symlink to ‘../lib/node_modules/npm/bin/npm-cli.js’: Invalid argumenttar: Exiting with failure status due to previous errorsBinary download failed, trying source.######################################################################## 100.0%Checksums emptytar: bin/npm: Cannot create symlink to ‘../lib/node_modules/npm/bin/npm-cli.js’: Invalid argumenttar: Exiting with failure status due to previous errorsBinary download failed, trying source.Detected that you have 1 CPU thread(s)Number of CPU thread(s) less or equal to 2 will have only one job a time for 'make'Installing node v1.0 and greater from source is not currently supported

Issue #9 points to non-implemented symlink support.

Another very annoying thing is the lack of Pseudo-Terminals (/dev/pts), this is possibly one of the reasons ZSH won't work. Follow Issue #80.

You should also be able to copy over your SSH private keys to ".ssh" and start git cloning from Github or git push-ing to Heroku in no time.

123456789101112131415161718192021

akitaonrails@localhost:~$ ssh-keygen  -t rsaGenerating public/private rsa key pair.Created directory '/home/akitaonrails/.ssh'.kitaonrails/.ssh/id_rsa):Enter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /home/akitaonrails/.ssh/id_rsa.Your public key has been saved in /home/akitaonrails/.ssh/id_rsa.pub.The key fingerprint is:f8:16:b1:be:85:31:0c:71:f8:c2:d3:72:ab:48:42:9a akitaonrails@localhostThe key's randomart image is:+--[ RSA 2048]----+|      ...        ||      .o         ||     ..o.        ||  .   =++o       || +    .=S.       ||E . .  o.=       ||   o . .= .      ||    . .. o       ||        .        |+-----------------+

At least, Elixir does seem to work:

123456789101112

akitaonrails@localhost:~$ iexErlang/OTP 18 [erts-7.3] [source-d2a6d81] [64-bit] [async-threads:10] [hipe] [kernel-poll:false]Interactive Elixir (1.2.3) - press Ctrl+C to exit (type h() ENTER for help)iex(1)> defmodule HelloWorld do; def say(name) do; IO.puts("Hello #{name}"); end; endiex:1: warning: redefining module HelloWorld{:module, HelloWorld,<<70, 79, 82, 49, 0, 0, 5, 240, 66, 69, 65, 77, 69, 120, 68, 99, 0, 0, 0, 147, 131, 104, 2, 100, 0, 14, 101, 108, 105, 120, 105, 114, 95, 100, 111, 99, 115, 95, 118, 49, 108, 0, 0, 0, 4, 104, 2, ...>>, {:say, 1}}iex(2)> HelloWorld.say("Fabio")Hello Fabio:ok

Not so fast ...

akitaonrails@localhost:~$ mix new ex_test** (ErlangError) erlang error: :terminated    (stdlib) :io.put_chars(#PID<0.25.0>, :unicode, [[[[[[[] | "\e[32m"], "* creating "] | "\e[0m"], ".gitignore"] | "\e[0m"], 10])    (mix) lib/mix/generator.ex:26: Mix.Generator.create_file/3    (mix) lib/mix/tasks/new.ex:76: Mix.Tasks.New.do_generate/4    (elixir) lib/file.ex:1138: File.cd!/2    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2

You will find more information on the pseudo-project Microsoft opened over Github to keep track of Issues from testers like me. You can follow the list of opened and closed issues there. You will see many things that work, but also many other things that won't work until a new release is available to fix everything I mentioned here.

Conclusion

So, is "Bash on Windows" a good development environment for Linux users to have over Windows?

As far as Preview 14316, not yet. The keyword here is "yet". It's shaping up nicely, if they can actually fix all the issues opened so far, it will be very usable very fast.

We need proper memory controls, a well implemented pseudo-terminal support, proper shared memory controls, proper symlinks, proper upstart from Ubuntu so the installed services suchs as Redis or Memcached (which I installed and run) can be properly restarted when I boot the environment (they don't come up if I boot the machine, for example).

Some people went as far as being able to trick X11 and run GUI apps such as Firefox and even XFCE over this environment already, so it's really promising. This idea does work.

Once those issues are ironed out, yes, I believe we are closer than ever to finally use Windows as a viable and full featured development environment. Let's see if by the Windows 10 Anniversary Release in June we will have a stable Ubuntu installation.

One thing that I think they did very wrong was to tie the Linux Subsystem to the Windows 10 installation. It should've been a separated installer, so the team can release build updates without having to bundle it all together with the entire Windows OS.

If you were intending to move to Windows for your Linux-based developments, not yet. Hold your horses a bit longer.

TL;DR: este é um dos meus famosos posts Off-Topic, portanto, longos! Vá diretamente à conclusão para ver aonde quero chegar. Mas por favor não critique antes de ler o texto todo, será mais produtivo.

Em tempos em que todo mundo parece ter uma opinião ativa sobre certos assuntos, eu prefiro não proferir ativamente nenhuma, nem a favor, nem contra. Porque ambos têm certeza absoluta que estão certos, é um jogo de soma zero, e eu odeio soma zero.

Se eu tiver uma opinião e ativamente forçá-la sobre as pessoas, muitas vão acabar concordando, muitos vão odiosamente discordar. Muitos vão comprar meu ponto de vista. Muitos vão me odiar e ativamente tentar desacreditar o ponto de vista ou o autor. E o principal: todos nós só temos a perder se entrarmos nesse tipo de visão maniqueísta.

Isso significa ficar "em cima do muro"? Também não é isso. Uma coisa que eu tenho há anos no meu perfil do Quoraé isso:

"I never live for the sake of another person and I never ask anyone to live for the sake of mine. I only accept voluntary trade for mutual benefit."

Que quer dizer:

"Eu nunca vivo por causa de outra pessoa e eu nunca peço a qualquer pessoa para viver por minha causa. Eu aceito somente trocas voluntárias para mútuo benefício."

Portanto, antes de continuar, eu peço, encarecidamente, que você pense por si mesmo - não pelos outros, nem muito menos por mim -, independente se concorda ou discorda do que eu tenho a dizer.

Dito isso, quero começar com um pequeno exemplo - de magnitude microscópica perto das grandes questões da atualidade, claro.

"Bem Vindos"

Em 07 de Julho de 2016, às 1:52:39 completará 10 anos que publico posts dedicados à comunidade de tecnologia, particularmente de Ruby on Rails. Eu disse o seguinte, após escolher cuidadosamente minhas palavras:

Vamos ver até onde essa comunidade pode chegar.
Ruby on Rails poderá ser muito ou nada, tudo vai depender de como o mercado vai encarar a novidade. Mas muita coisa pode ser feita agora. Para começar, aprendendo sobre o assunto.
Vou postar os principais assuntos sobre a plataforma aqui e espero que todos colaborem com ideias e sugestões ou mesmo críticas e opiniões.
Infelizmente ainda existem muitos desafios a serem vencidos. Para começar, materiais de Rails em português virtualmente inexistem. Sites brasileiros idem. Portanto quando digo "começar do zero", estou falando sério.
O maior desafio será convencer o mercado. E isso não se faz da noite para o dia. Significa que ainda não será possível deixar o legado do Java totalmente de lado. Vamos começar um período de transição no qual tentaremos as duas coisas em paralelo.
Os pioneiros sempre caminham por território árduo, mas a recompensa dos primeiros sempre será maior também. Esse é o sentido do investimento.

Meus Princípios

Se você ainda não sabia (nunca leu meus Off-Topic) vamos deixar claro: eu não sou altruísta (no sentido literal da palavra, não no jeito errado como é interpretado). Nada do que fiz até hoje foi puramente em prol de algum tipo de ideologia de "bem maior".

Meu objetivo é muito simples: eu só quero o direito de poder produzir, nos meus próprios termos.

No caminho, o que eu fizer deixará efeitos colaterais, como ajudar os outros. Por exemplo, hoje tenho uma pequena empresa chamada Codeminer 42 que emprega mais de 60 pessoas. Pessoas com famílias, que encontram não só seu sustento mas sua própria auto-melhoria, através dos resultados que eu e meus amigos buscamos e conseguimos conquistar juntos. E isso é um efeito colateral direto.

Eu organizo a Rubyconf Brasil, faz 9 anos junto com a Locaweb. Os palestrantes, patrocinadores, eu, a própria Locaweb, têm um ponto de encontro anual onde empresas conhecem novas pessoas pra contratar, profissionais encontram seu próximo emprego, empreendedores se juntam para testar idéias que depois podem gerar valor a mais pessoas, e assim por diante. É uma oportunidade de mais de 1.200 participantes todo ano de produzir alguma coisa. Isso é um efeito colateral indireto.

Um grupo como o Ruby on Rails Brasil no Facebook - que não fui eu quem criou - tem mais de 8.500 participantes trocando idéias, aprendizados, experiências.

A base disso é o mundo open source - a maior experiência capitalista do mundo do software e um exemplo vivo para todos do que é um Mercado Livre Laissez-faire. O melhor lugar para manter e melhorar commodities de tecnologia.

Empresas investem tempo e dinheiro contratando programadores para manter projetos abertos não porque são "boazinhas" - independente do que digam - mas porque essas tecnologias realmente trazem muitos benefícios diretos e indiretos aos seus negócios.

Porque se fossem fazer sozinhas, precisariam investir muito mais do que dividir o problema entre dezenas de outras empresas para todas lucrarem juntas. É a plataforma ideal para indivíduos demonstrarem suas capacidades ao mundo, melhorando suas próprias capacidades, buscando novas oportunidades. Mais do que isso, é um dos lugares mais brutais e agressivos - e por isso mesmo muito bom - para um indivíduo aprender a lidar com pressão, comunicação, soluções técnicas criativas, exatamente porque não tem recursos infinitos.

Todos lucram, não só de maneira monetária, seja com novo aprendizado, seja com novas oportunidades. E mesmo aqueles que têm seu lucro na pura satisfação de ajudar os outros de alguma forma (porque ajudar os outros não é altruísmo, é uma troca justa: troca do que você sabe produzir pela recompensa de ter sua consciência mais tranquila, afinal ninguém ajuda quem não gosta - o que seria altruísmo de verdade).

O que eu NÃO fiz, e NÃO faço

Quando eu comecei essa jornada - lembrando, eu não fui o primeiro - sempre busquei fazer coisas para meu benefício próprio, sabendo que isso poderia trazer benefícios para outras pessoas e, com benefícios mútuos, todos andariam mais rápido e melhor. Minha proposição nunca foi promessas, seja de recompensas materiais ou morais, apenas a promessa de trabalhar no que se gosta.

Se fosse puramente por moral ou ideologia, nunca teria dado certo. Ou pior, se tivesse dado certo, seria frágil, por ter a fundação errada.

Porque se eu realmente tivesse acreditado que "Ruby on Rails é a única melhor tecnologia" eu teria forçado essa visão nos outros. Muitos teriam acreditado. Mas pior, nós teríamos que acreditar que outras tecnologias não são boas e teríamos lutado contra elas, independente das evidências. Isso teria sido desastroso.

Eu sempre soube que Rails é bom em certos casos, em particular no contexto do crescente mercado de startups a partir de 2004. Software as a Service, Infrastructure as a Service, Platform as a Service. Muitos pensaram como eu, independentemente, e daí nasceram gemas como Github, o maior mercado livre de software já criado.

Se eu tivesse acreditado que meu jeito era o certo e o dos outros sendo automaticamente errado, eu teria me frustrado e fracassado amargamente. Meu pensamento natural teria sido:

"Por que esse povo não me segue se meu caminho é certamente o correto?"

Eu nunca tentei convencer ninguém de que meu ponto de vista é correto "pelo bem" de alguma coisa ou "porque é free software" ou "porque atende alguma visão política". Eu sempre evangelizei tentando ensinar como lucrar.

Porque num nicho com as características como a de Rails se acumularam profissionais de qualidade ímpar. Porque todo novo programador tinha como "role model" esses pioneiros. Porque todo Rubista almejava qualidade em primeiro lugar. Porque a nova geração de startups estava demonstrando como fazer produtos lucrativos sem a torneira aberta de investidores alucinados como em 2000. Era esse o discurso da 37signals em 2005: todo mundo queria fazer o próximo Basecamp. E fizeram! De Engine Yard, a Heroku, a Github, a Pivotal Labs, a Living Social, a Shopify, a Twitter, a Groupon, a Zendesk, a Hulu, a Square, a Airbnb, a Soundcloud.

Importa que eles não são puristas 100% feito em Ruby? Lógico que não! Ninguém está defendendo uma visão extremista e religiosa. Este não é o "Culto do Ruby". Rails é um meio para um fim, nunca o fim. Quem adota Rails queria filtrar os melhores programadores - porque precisava gostar muito da arte de programar pra aprender Rails em 2006. Quem adota Rails queria qualidade de vida na rotina de programação - porque os princípios por trás do design do Ruby e do Rails criaram uma visão única de como software poderia ser feito. Ninguém escolheu Rails como se escolhe um time de futebol ou uma afiliação política. Todos que escolheram almejaram algum tipo de ganho pra si próprio e, pensando assim, criaram um gigantesco ecossistema que acabou beneficiando muito mais gente.

O que eu não fiz? Não fiz promessas. Não dei certificados. Não garanti ministérios. Não prometi "um futuro brilhante". Não adicionei visão política à comunidade. Talvez a única promessa tenha sido de muito trabalho à frente, oportunidades de produção. Releiam o último parágrafo do meu primeiro post:

"Os pioneiros sempre caminham por território árduo, mas a recompensa dos primeiros sempre será maior também. Esse é o sentido do investimento."

Esse é o espírito de um empreendedor: arriscar o que se possui, não o que os outros têm.

É proposital porque desde o começo eu raramente palestro nos eventos que eu mesmo organizo. Por isso desde o começo existem outras tecnologias em exposição num evento de Ruby (tivemos palestras de Node.js desde 2009, antes de existir eventos de Javascript). Por isso a "Ruby"conf tem trilhas de todas as outras novas tecnologias em crescimento: porque só temos a lucrar com isso.

É minha forma de poder produzir nos meus próprios termos, com o que eu gosto.

O "Ruby-way" Não é convencer os outros que usar Ruby é o único caminho, mas sim que trilhar o caminho mais difícil, o da qualidade, de se importar com o que faz, provavelmente pode trazer resultados melhores. É o que um empreendedor faz: assume riscos calculados e trabalha brutalmente por si próprio. É o que um artista faz: diferente do comum.

O caminho de um empreendedor, ou um artista, ou alguém que produz, é necessariamente individualista. Porque vamos contra o status-quo, contra o que é considerado o "certo" hoje. Contra o que é "comumente aceito" por uma "maioria".

"Loucura é querer resultados diferentes fazendo sempre a mesma coisa." - Albert Einstein

Seremos sempre apedrejados por isso, mas são pessoas que escolhem o caminho diferente que todos vão tentar impedir. E são elas que nos fizeram atingir as grandes conquistas da humanidade. Uma das coisas que me fez reconhecer o real significado logo de cara foi quando o criador do Rails, o até hoje controverso David Heinemeir Hansson - alguém que eu concordo e discordo em vários aspectos - disse logo nos primórdios da comunidade:

“Primeiro, eles o ignoram. Depois, o ridicularizam. Finalmente, eles brigam contra você. E então, você vence.” - Mahatma Gandhi

É pecado sequer imaginar que não é o Sol que gira em torno da Terra. Precisou da visão individualista e absolutamente "egoísta" de um "Galileo vs Inquisição" para ajudar a abrir caminho para hoje não só já termos ido à Lua mas estarmos nos tornando excepcionalmente bons em mandar maquinário para o espaço.

Uma Nação Nazista

"Somente um Sith pensa em absolutos." - Obi-wan Kenobi

Existe um termo para quem pensa em absolutos no mundo real: Extremistas Terroristas. Ser extremista, por si só, não é ruim. É forçar sua visão extrema - o campo de um terrorista - que leva ao lado negro da Força.

Um terrorista é perigoso, porque ele absolutamente acredita que está certo.

"Porque algumas pessoas não estão procurando por alguma coisa lógica, como dinheiro. Eles não podem ser comprados, antagonizados, ensinados, ou negociados. Algumas pessoas somente querem ver o mundo pegar fogo." - Alfred, The Dark Knight

Que é a razão de porque não se deve tentar negociar com terroristas. É impossível, você sempre vai perder.

E o pior: o lado negro da Força sempre pode ser sedutor. Porque ele é facilmente travestido de algo "bom":

"Não tente me ensinar, Obi-wan! Eu vejo através das mentiras dos Jedi. Eu não tenho medo do lado negro como você. Eu trouxe paz, liberdade, justiça e segurança ... ao meu Novo Império!" - Anakin Skywalker, Episode III

Não é difícil entender como surgiram pessoas tão inequivocadamente reconhecidas como "o mal". Coletivistas, extremistas, como Adolf Hitler, Joseph Goebbels, Heinrich Himmler, Hermann Göring.

Mas eles são apenas algumas pessoas. A questão principal é mais importante e mais difícil: como se convence toda uma Nação, como o povo Alemão, a aceitar e agir em favor da ideologia de um partido como o Nacional Socialista até o ponto do genocídio de 11 milhões de pessoas no Holocausto!?

Existem muitas teorias para o que realmente aconteceu durante a Segunda Guerra Mundial e é um assunto até hoje delicado, mas eu tenho uma pequena teoria pessoal (não é suportada por evidências, por isso este 'disclaimer'):

Apesar de saber que muitos devem ter cometido esses atos com intenções puramente maléficas, é muito difícil de acreditar que num curto espaço de tempo, milhares de alemães de repente se tornaram transformaram automaticamente em agentes do Mal.

Naquela época, a Alemanha havia sofrido a humilhação pós-Primeira Guerra, a economia estava em frangalhos após terem sofrido a Grande Depressão pós-1929. Num contexto desse, uma ideologia política populista, pelo bem social da nação do Partido Nacional Socialista ganhou força. A idéia do "Terceiro Reich" era sedutora por trazer de volta a esperança do futuro brilhante de uma "Nova Alemanha".

E eles encontraram alguém para realmente odiar, no caso, os Judeus. E a idéia anti-semita pegou tração no discurso populista.

Agora, em nome do "futuro brilhante da Nova Alemanha" pode ter levado até o extremo do Holocausto. E mesmo assim, como isso se justifica?

Se vocês não conhecem as teorias psicológicas de Stanley Migram, recomendo que assistam ao filme The Experimenter (2015) para entender.

Em resumo, o Experimento da Autoridade de Migram - que foi repetido inúmeras vezes ao longo dos anos - demonstra como não é tão difícil como o senso comum imagina, que um grupo de pessoas executem a coisa errada - mesmo tendo consciência disso - em nome de um "bem maior".

Obviamente estou sendo simplista na explicação, mas vale ter isso em mente: a maioria das pessoas reage com comportamento de rebanho (Herd Behavior).

Não é muito longe da Lei do Mínimo Esforço, afinal, porque gastar tempo pesquisando se uma coisa pode ser "certa" ou "errada" quando é muito mais fácil seguir o que a "maioria" pensa que é certo?

"Um Pouco de Conhecimento é Uma Coisa Perigosa"

Não é completa ignorância, mas sim "um pouco de conhecimento", que pode ser a coisa mais perigosa de todas; como já nos alertou o grande poeta Alexander Pope.

"A little learning is a dangerous thing; Drink deep, or taste not the Pierian spring : There shallow draughts intoxicate the brain, And drinking largely sobers us again." - Alexander Pope

E uma das coisas que eu gosto de escrever é justamente este alerta: não sabemos tudo, nunca saberemos tudo, e tudo que sabemos pode estar errado. Se você viver constantemente com isso em mente, estará seguro.

A Raíz de Todo o Mal, os Inimigos da Razãoé o que eu falo faz tempo: o pensamento "religioso".

E não digo isso puramente para falar mal da religião dos outros, mesmo porque a "religião" a que me refiro não é meramente Cristianismo, Judaísmo, Xintoísmo, etc. É pior: é a Religião do Pouco Conhecimento.

Vemos exemplos desse tipo de coisa todos os dias, basta ler as notícias:

Basta procurar por "morte" nas notícias e muitas delas são, infelizmente, resultado da ignorância ou pior, "pouco conhecimento" das pessoas. Não é algo que vai se resolver automaticamente, mas nós já resolvemos milhares desses problemas. Felizmente, o século XX em diante foi um grande avanço para eliminar todos os preconceitos, conhecimentos ultrapassados, e ignorância geral da população global.

O problema: nenhuma dessas pessoas tem dúvidas se está errado. Todos tem certeza que estão certos, fazendo o que é moralmente correto - dado o pouco que sabem.

Dogmatização é a consequência. E Dogmas são Inquestionáveis. Você segue uma cartilha inquestionável? Você provavelmente já foi dogmatizado. Ninguém gosta de ser provado errado, especialmente se acreditar em alguma mentira com muito afinco por muito tempo, isso se torna sua identidade. E quem quer ter sua própria identidade quebrada?

Cisnes Negros e o Antídoto do Pouco Conhecimento

Finalmente, chegamos ao ponto. Como se resolve os problemas do mundo?

Com a "filosofia científica". E não é necessário ser um cientista para pensar dessa forma.

O mundo é injusto, é um fato. É impossível resolver o problema de todo mundo de uma só vez. Mas podemos resolver um problema de cada vez, ao longo do tempo. Sinto muito aos impacientes.

E eu acho que começa com uma coisa muito simples: raciocínio crítico não-baseado puramente em indução simples.

Para mim, Karl Popper me ensinou uma das coisas mais valiosas no meu jeito de pensar. Eu não procuro falsos-gurus, falsos-especialistas, falsos-líderes, falsos-heróis ou falsos-ícones.

Não preciso de ninguém de "autoridade" para me dizer o que é o certo. Nem tampouco preciso do consenso de alguma forma de "maioria" para validar minhas ações.

Há quem advogue uma "afirmação inviolável", "a" forma correta; seja pela tradição, seja por escrituras, seja pela vontade de uma entidade abstrata e amorfa como "a sociedade", "as pessoas"; ou pior, por "provas invioláveis", então sei que se tratam de falsos profetas.

A explicação para você se tornar incorruptível é simples. Deixe-me fazer a explicação clássica:

"Imagine que todos os cisnes do mundo são brancos."

Digamos que sua afirmação vem do fato que nos 40 anos que viveu, você, sua família, seus amigos, somente viram cisnes brancos com seus próprios olhos. Sua conclusão natural e "lógica", por "indução", é extrapolar para uma lei geral, absoluta, e dizer que "todos os cisnes do mundo são brancos".

Isso vem do seu pouco conhecimento. Não importa se foram 40 anos. Não importa que você já viu 1.000 cisnes durante sua vida, e todos eram brancos.

Você claramente nunca viajou pra Austrália, ou pra Nova Zelândia, onde encontraria a variedade de cor preta.

Mas, pelo seu "pouco conhecimento", você chegou a uma conclusão errada. E, pior, extrapolou e proferiu essa afirmação a outros, que agora pensam como você. É sua responsabilidade.

"Eu sei que vacinas são ruins porque eu ouvi casos de várias crianças que foram vacinadas e morreram, portanto nunca vou vacinar meus filhos."

"Eu sei que remédios não funcionam mas homeopatia funciona, porque minhas tias usam faz anos e nunca tiveram problemas de saúde, portanto nunca vou usar remédios."

"Eu sei que o capitalismo é coisa de gente maléfica, porque eu conheci empresários na minha cidade que exploravam meus amigos e vizinhos, portanto o sistema capitalista tem que cair."

"Eu sei que o socialismo funciona, porque meus amigos que visitaram Cuba ficaram muito impressionado com o que ouviram dos hospitais de lá, e por isso acho que tinha que usar em todo lugar."

Você faz isso todos os dias: vê uma notícia com título chamativo, vê dezenas de "Likes" e concorda que está certo e, pior, ainda compartilha para mais gente, disseminando mentiras.

"Uma mentira repetida mil vezes torna-se verdade" - Joseph Goebbels, Ministro da Propaganda Nazista

O problema é a "procura da verdade". A grande maioria das coisas - tirando teoremas matemáticos - raramente é possível mostrar a prova correta. E muitas vezes recorremos ao reverso, mostrar que é incorreto. Nenhuma quantidade de evidências é suficiente para provar qualquer teoria como absolutamente verdade. O máximo que podemos fazer é "desprovar" uma teoria, porque para isso bastar uma única evidência que a refute.

"Afirmações extraordinárias exigem evidências extraordinárias." - Carl Sagan, Cosmos

E eu adiciono: nem mesmo com evidências extraordinárias podemos afirmar a descoberta de uma "verdade absoluta", apenas podemos dizer que temos um conhecimento sólido o suficiente para usar mas que pode ser desprovada a qualquer momento.

É importante ter esse conceito em mente pois isso nos torna mais fortes contra rumores, pseudo-ciência, superstições, "ditos comuns", tradição ultrapassada e todo tipo de falsas-"verdades".

Qualquer teoria que não dá formas claras de desprová-la não é uma boa teoria e, certamente, não é científica. Qualquer afirmação que depende de uma divindade que não se pode provar a existência, ou qualquer "moral" inviolável, ou uma autoridade "inquestionável", é pseudo-ciência e é errada.

A Moral Cinzenta dos "Justiceiros Sociais"

Esse é um assunto que evito ao máximo me envolver porque esta geração dos chamados Justiceiros Sociais (Social Justice Warriors - SJW) é particulamente perversa. Não todos! Apenas uma pequena minoria muito influentes. Verdadeiros lobos em pele de cordeiro.

Disclaimer: considero quem luta por causas sociais uma pessoa perversa? NÃO Existem diversas iniciativas sociais legítimas, com excelentes resultados, com pessoas com real afinco em resolver um problema legítimo. Espero somente coisas boas a essas pessoas. Aqui estou falando somente de quem está desvalorizando essas pessoas. O objetivo desta seção é justamente separar o real do ilegítimo.

Não sou contra as causas sociais, a maioria das pessoas civilizadas não é. Certamente existe muito a se fazer para continuar a melhorar o mundo. E no último século certamente melhoramos o mundo de maneira extraordinária, principalmente se compararmos aos séculos anteriores ao XX.

Ao contrário do que muitos dizem e uma coisa que eu repito desde 2013: "O Mundo Hoje está Melhor". Acredite que o mundo está piorando e aí você começará a justificar atos perversos como moralmente corretos.

E não foi graças a eventos e movimentos pontuais, mas pelo progresso contínuo de gente que "produz", não de gente que "rouba". E roubar não é somente diretamente tirar um bem material de alguém. Alguém pode roubar mais que isso: seu tempo, sua moral, sua mente e sua liberdade.

Roubo da Liberdade é Escravidão.

Até o século XIX todos precisavam trabalhar arduamente somente para tirar seu sustento básico. Hoje temos uma sobra enorme, um surplus, porque descobrimos formas mais eficientes e aplicamos para fins individuais. Isso acabou ajudando mais do que apenas o indivíduo que as criou. O motor a vapor, a eletricidade, a agricultura, a medicina, tudo criação de gente egoísta, cujo único objetivo é produzir, acabou por tornar o mundo ordens de magnitude melhor. Nenhuma revolução social na história pode clamar pelos mesmos resultados.

Você se importa em diminuir a pobreza? Milton Friedman já explicava porque o Livre Mercado é melhor para os pobres.

Existem apenas 3 liberdades básicas que realmente importam:

o direito à liberdade de expressão;
o direito a produzir e manter para si o que se produz via trocas voluntárias, sem roubo ou escravidão;
o direito a procurar e conquistar seus próprios objetivos, sem obrigação de servir à vontade de outros.

Um indivíduo produtivo "egoísta" tem como único objetivo produzir, criar. Nunca roubar.

Ninguém tem direito de restringir a liberdade de qualquer outra pessoa, forçando sua própria moral cinzenta, sua própria agenda, seus próprios objetivos, se não for pelo meio exclusivo de "trocas voluntárias". Nenhuma moral cinzenta tem direito de se justificar a imoralidade da escravidão de um indivíduo, por nenhuma justificativa, não importa por qual objetivo.

E esse é o problema: um falso justiceiro social tem como único objetivo ser um justiceiro social; ter a aceitação e validação dos demais. E para isso ele necessita da existência do sofrimento alheio, qualquer um, para justificar sua existência e sua identidade. Portanto o objetivo não é acabar com o sofrimento alheio. Se uma forma de sofrimento acaba, logo será necessário outro tipo de sofrimento para se levantar uma causa, e esse é o círculo vicioso. Pois seu objetivo não é produzir, é se alimentar do sofrimento dos outros, é a escravidão do sofrimento alheio à validação de sua própria identidade.

Um produtor, um criador, não está necessariamente interessado em "causas". Certamente não é uma regra. Independente dos motivos, ele quer criar. E isso basta. Para um falso justiceiro, isso não basta, ele precisa tomar de quem produz em benefício de alguma "causa". Um justiceiro quer tirar o que você tem, mas não o que ele próprio tem - porque ele não produz - em nome de alguma moral cinzenta. E não é por qualquer causa, somente as que interessam a ele. É a velha e eterna briga entre os que produzem e os que tomam, a moral cinzenta de Robin Hood.

E isso é uma pena pois desvaloriza quem realmente dedica sua vida a causa sociais. Sejam médicos, bombeiros, policiais, soldados, e diversos anônimos que não gastam seu tempo postando em redes sociais, julgando os outros e criando discórdia, mas que trabalham silenciosamente, sem expectativa de reconhecimento público.

Quer reconhecer um real justiceiro social dos parasitas sociais? Procure por Não demonstração de busca por reconhecimento.

Conclusão

O objetivo deste longo artigo? Apenas expor conceitos universais, meus princípios pessoais, que são invioláveis e incorruptíveis. Os principais pontos:

A única coisa realmente moral são as trocas voluntárias. Nunca forçar a escravidão, física ou moral, de qualquer indivíduo.
Não existe "verdade" decretada por algum grupo ou maioria. Existem idéias, algumas válidas outras inválidas. Mas se não forem idéias falseáveis, refutáveis, provavelmente são falsas por premissa, não importa a aparente moral que representam ou a quantidade de "evidências" que existam.
Não existe "autoridade moral", isso é o domínio dos charlatães. Duvide de qualquer um que force sua propriedade (sua ação é sua propriedade, sua liberdade é sua propriedade) em nome de qualquer tipo de "causa", não importa quão justa ela pareça. Não procure por heróis.
A única coisa que realmente nos tirou da Idade Média e nos lançou à modernidade foi graças às mentes que produziram para benefício próprio (sem exigir unilateralmente a tomada da propriedade de ninguém, seja material ou mental).

Veja o século XX e compare ao século XIX e anteriores. Ninguém explicou isso melhor que Hans Rosling, da Gapminder:

Não se torne uma vítima da minoria dos falsos justiceiros sociais, aprenda mais sobre esse perigoso grupo Orwelliano - nada menos do que a "Polícia do Pensamento", como profetizado no universo despótico de 1984. Esse tipo de pensamento Não é novidade, existiu por todo o século XX, mas o advento da Internet e principalmente da Redes Sociais nos trouxe um ambiente selvagem onde reinam os rumores, notícias falsas, pseudo-ciência em geral. É onde o Racionalismo Crítico pode te salvar.

Princípios invioláveis e incorruptíveis são simples. No mundo axiomático da matemática, 2 + 2 é 4, não existe "maioria" ou grupo que, por clamor ou voto, pode mudar esse resultado para 5 ou 6, não importa a moral ou autoridade de quem faz o clamor. Você não precisa de validação! Você precisa, sim, assumir a responsabilidade pelas próprias ações e não justificar seus erros como sendo o resultado de seguir a moral cinzenta de outra pessoa.

Repetindo:

"Eu nunca vivo por causa de outra pessoa e eu nunca peço a qualquer pessoa para viver por minha causa. Eu aceito somente trocas voluntárias para mútuo benefício."

E com esse princípio vamos continuar a evoluir o século XXI como fizemos no século XX, não importa quem nos tenta roubar, quer você concorde com esta exposição ou não.

Já estamos fazendo isso.

Uma frase que eu disse no meu artigo anterior pode ter causado estranheza para algumas pessoas:

"A base disso é o mundo open source - a maior experiência capitalista do mundo do software e um exemplo vivo para todos do que é um Mercado Livre Laissez-faire. O melhor lugar para manter e melhorar commodities de tecnologia."

Pelos princípios de Liberdade de "Free" Software, muitos erroneamente atribuem o mundo de código aberto e livre como sendo uma grande experiência "socialista". E não é, ela é absolutamente capitalista, é um Laissez-Fair, Laisser-Passer, eu iria mais longe e diria mesmo que é quase um real Livre Mercado, a utopia capitalista - que não existe hoje e muitos acham que só existiu nos primeiros 20 anos do século XX nos EUA.

Como a C|NET reportou em "Código Aberto: É sobre capitalismo, não brindes gratuitos"

"O segredo é usar software de código aberto como um meio para um fim, não o fim em si mesmo. Código aberto é um meio de distribuição barata, uma maneira de colocar software nas mãos de potenciais compradores por pouco ou nenhum custo. É uma maneira de tornar a experiência de software social e menos arriscada, porque os usuários podem experimentar antes de comprar e porque eles podem customizar (ou pagar alguém para customizar) o software para suas necessidades por um custo menor do que software proprietário."

Ou neste outro artigo da C|NET "Desculpe, socialistas: Código Aberto é um jogo capitalista"

"[Sarah] Grey escreve que 'existem alternativas ao capitalismo'. Ela está certa. Infelizmente, código aberto não é uma delas. Código aberto é a essência do Livre Mercado do Capitalismo."

A maioria das pessoas tem idéias muito distorcidas de porque "capitalismo é o império do mal dos tiranos" ou porque "socialismo é justiça, igualdade e democracia para todos", quando na realidade é exatamente o oposto.

Aliás, uma noção que muitos não temé que ou você é defensor do socialismo ou você é defensor da liberdade. Ambos são diametralmente opostos.

Vejamos como Livre Mercado e interesses individualistas é o único caminho que leva a um efeito colateral positivo: a possibilidade de realmente fazer a diferença em "causas sociais". Através de princípios capitalistas, não socialistas.

O princípio? Como eu disse no meu artigo anterior:

"Eu nunca vivo por causa de outra pessoa e eu nunca peço a qualquer pessoa para viver por minha causa. Eu aceito somente trocas voluntárias para mútuo benefício."

O que é Livre Mercado em Software Livre?

Vamos entender:

Um "Livre Mercado" é a condução de transações sem coerção.

Eu posso usar um software livre gratuitamente, é a licença que seu proprietário me deu. Eu posso contribuir com partes de código a esse software, sem a coerção, por minha própria escolha. O proprietário do software tem o direito de aceitar ou não minha contribuição, com base nos critérios que ele quiser.

Um software é propriedade privada, copyright, de seus autores. Ter o código "aberto" não o torna domínio público e você não pode clamar direitos se este software não tiver uma licença que lhe dê esse privilégio. Enforçar ou não esse copyright é um direito exclusivo de seu proprietário.

Laissez-Fair é a intervenção mínima ou quase inexistente de uma entidade governamental ou reguladora. Nenhuma entidade controlando a oferta e a procura. Deixando a dinâmica desse mercado - quem oferta, quem adquire, a moeda e os preços - totalmente na mão de seus participantes. As trocas são voluntárias, quem quer contribuir código deve convencer os demais de seus méritos baseados unicamente na moeda "competência técnica". Pelo menos esse é o "ideal", no mundo real existem obviamente pressões sociais e políticas em alguns casos.

Convencer os outros é sobre demonstrar, com resultados, porque sua idéia tem valor. A partir do momento em que você acha que os demais deveriam apoiar sua "causa" porque você tem certeza que ela tem valor e "quem discorda é um imbecil e merece morrer", você acabou de desvalorizar e fazer um desserviço à sua causa. Se você não consegue convencer os demais do seu valor, não são os outros que são burros ou preconceituosos, provavelmente é você que foi incompetente.

"Coerção é o domínio dos Tiranos."
"Tentar regular a propriedade dos outros via coerção é o domínio dos Tiranos."

"Preços" não se refere meramente a transações financeiras com moeda, como seria numa Economia Capitalista. "Valor", o termo mais geral, pode ser determinado por qualquer coisa que tenha oferta e procura. Quem configura esse "preço" é a competição.

Um Livre Mercado tem sua competição desregulamentada. Por causa disso, uma verdade muitas vezes não mencionada quando se fala de software livre é que para cada 1 (um) projeto famoso como PostgreSQL que existe, dezenas de outros tentaram e morreram, ou porque não eram tão bons quanto ou porque falharam em convencer os outros de seus méritos. Para cada uma distro Linux que dá certo, centenas de outras deram errado ou vivem meramente num pequeno nicho - o que por si só é valor suficiente pra muita gente.

Eles morrem, porque não há nenhuma "agência reguladora" tentando mantê-los vivos artificialmente, muito menos importa se existia alguma agenda social por trás. E isso é justo, porque ou a justiça é realmente cega, ou não é justiça, é apenas vigilantismo e coerção.

E o preço não é o valor do código que entra, mas quanto vale para um programador ter seu código envolvido num projeto famoso. É uma inversão de valores em relação a empresas comerciais: normalmente se paga em dinheiro a um programador para ele produzir um código proprietário. Num projeto de software livre é o programador que deve "pagar" - com seu tempo livre e sua capacidade - para ter seu código aceito num projeto famoso porque ele recebe algum valor como retorno.

Software Livre = Propriedade Privada + Licença de Uso

Como bem lembrou Jeff Atwood, do Coding Horror em "Escolha uma Licença, qualquer Licença":

"Porque eu não indiquei explicitamente uma licença, eu declarei implicitamente um copyright sem explicar como outros poderiam usar meu código. Já que código está sem licença, eu poderia teoricamente forçar o copyright a qualquer momento e demandar que as pessoas parem de usar meu código. Desenvolvedores experientes não tocariam em código sem licença porque eles não tem direito legal para usá-lo."

Como testar a propriedade privada num mundo de software livre? Eu posso, a qualquer momento, apagar meu código de um repositório público que eu criei. Aconteceu recentemente com o controverso caso do Leftpad que quebrou o projeto de muita gente. "Quebrou a Internet" como muitos exageraram.

Concordar ou discordar do Koçulu ou da Kik é irrelevante. Koçulu exerceu seu direito: ele destruiu sua própria propriedade e isso é legítimo. É sua propriedade, ele faz o que quiser com ela, não importa quem clame o contrário ou quanto prejuízo isso possa ter causado. A situação foi ruim? Então vamos aprender a criar um ambiente mais robusto sem retirar a propriedade privada de seu autor. E seguindo o exemplo, uma possível solução foi a mudança na política de unpublish do NPM.

Na comunidade Ruby tivemos um episódio assim. Claro, foi muito menor, com muito menos consequências ou repercussão, em 2009, com o sumiço de Why. Aliás, notaram que antigamente qualquer "mimimi" na comunidade Ruby logo era chamado de "Ruby Drama"? Notaram que ninguém mais fala disso? Por que "dramas" se tornaram tão comuns e tão banais em todas as comunidades que deixou de ser percebido como exclusivo da comunidade Ruby. Como evidência, o último tweet da conta @rubydramasé de 2013.

E quando existe conflito pode acontecer os famosos casos de "fork", onde um projeto pode ser clonado e uma segunda comunidade pode acabar se formando se os critérios fizerem sentido o suficiente e o resultado for realmente melhor. Exemplo recente foi o fork do Node.js para IO.js, o período de transição e negociação e o resultado de fazerem "merge" de volta sob a liderança da nova Node Foundation.

Existiram diversos forks que deram certo, como o Webkit a partir do KHTML, o Ubuntu a partir do Debian, o próprio OS X a partir do BSD.

Projetos devem poder Fracassar. E nada pode forçar seu Sucesso.

Num Livre Mercado não existe garantias de sucesso, só existem garantias de que você pode tentar.

Livre Mercado é "justo" (como em "justiça cega") e, por consequência, não é igualitário nem "moral" dependendo do seu ponto de vista pessoal. Aliás, "ponto de vista" é necessariamente tendencioso e pessoal, não objetivo, por isso não serve de parâmetro para justiça. Para entender isso assista O Povo Contra Larry Flint. Você pode discordar completamente de Larry Flint, mas nem por isso pode condená-lo. E ao não condená-lo, não significa que você apóie o que Flint diz ou faz.

Sucesso em código, no geral, independe de qualquer característica pessoal de seus autores, somente do mérito técnico do resultado do seu código. A beleza do código é que ele é anônimo desprovido de qualidades pessoais ou agendas políticas. Mesmo se temporariamente ele acabar desvalorizado por causa de vínculos políticos, quanto menos afiliações um projeto tiver, mais chances tem de sobreviver ao longo prazo.

Só porque você criou um projeto e abriu seu código, não quer dizer absolutamente nada. Muito menos que vai ter sucesso, não importa suas boas intenções, não importa sua história, não interessa sua afiliações.

Essa é justamente a melhor parte do mundo de Software Livre, porque a competição, os interesses individuais, garantem que haverá alguns poucos que darão muito certo, muitos que terão pouco ou nenhuma tração e a maioria que simplesmente vai desaparecer da existência e da memória das pessoas.

Um exemplo? Alguém se lembra do sistema operacional móvel Symbian da Nokia, que era proprietário e nos seus últimos dias anunciaram que iam abrir seu código na tentativa de ver se ganhava alguma tração? Muito pouco, muito tarde.

Livre Mercado é a manifestação econômica da Natureza (Darwiniana) de Seleção Natural, que eu já expliquei num screencast/palestra. O software que tiver as melhores características e que continuar se mutando e adaptando às demandas do ambiente, é quem terá melhores chances de reproduzir (ser usado e distribuído) e ser selecionado para sobreviver. Existem mais projetos morrendo todos os dias do que prosperando. E isso não é uma falha ou fracasso do sistema, é a garantia de sua robustez.

Apache HTTP vem perdendo cada vez mais espaço para NGINX. MySQL veio perdendo espaço primeiro para outros RDBMS mais competentes como PostgreSQL e depois para NoSQL como MongoDB e agora até pra forks como MariaDB. Mozilla Firefox veio perdendo espaço drasticamente para Google Chrome. Perl perdeu espaço para Python e diversas novas linguagens como Go.

Software Livre "Socialista" é um Paradoxo

Num mundo "socialista" teríamos necessariamente o sucateamento do software.

Como toda propriedade seria monopólio da entidade "governo", não haveria competição, apenas incentivo a conluio, cartel e monopólios. Todo projeto ruim - se atendesse à agenda política de alguém - se manteria vivo, a despeito de outro se provar tecnicamente melhor. A escolha não seria de quem usa, mas de quem controla. Não haveria acidentes como leftpad, mas seria pior: existiria tanto lixo que o mundo de software livre simplesmente perderia relevância.

Sem benefícios aos voluntários como: propaganda em benefício próprio, exercitar suas capacidades técnicas em projetos desafiadores, distribuir seu software livremente, alimentar seu ego e suas métricas de vaidade; nenhum programador de real capacidade técnica perderia seu tempo se esforçando. Pior: se num governo socialista o governo decretasse arbitrariamente que todos deveriam contribuir para determinado projeto por causa de algum "bem social", ninguém daria o melhor de si, faria apenas o mínimo para cumprir determinada "cota".

Quando não há competição e não há oferta e procura derivada de valor, não existe mais inovação. Inovação surge de uma vontade individualista de resolver seu próprio problema do seu melhor jeito, independente do que os outros clamarem. Por isso mesmo o mundo de free software começa com a famosa frase:

"Todo bom trabalho de software começa coçando uma coceira pessoal de um desenvolvedor." - Eric Raymons, The Cathedral and the Bazaar

Inovação é naturalmente transgressora. Inovação é naturalmente competitiva. Quando alguém inventa um robô de fábrica, está eliminando empregos, portanto para muitos a Inovação também é anti-social. Uber realmente acaba com o emprego de taxistas. Airbnb acaba com o emprego de corretores. Uma máquina para sacar dinheiro (ATM) acaba com o emprego de bancários.

Em software, você acha que o cara de Hurd ficou muito contente com Linux? Acha que o cara de Symbiam achou legal o Android? Acha que o cara de Clipper ficou feliz com Java? Acha que o cara de MySQL está contente com PosgreSQL? Se essas pessoas forem resistentes à dinâmica do mercado, ficou para trás e, honestamente, está valendo menos e menos e provavelmente também vai perder seu emprego. Injusto? Não, absolutamente justo.

Inovação é anti-socialista porque acaba com a necessidade de força bruta de trabalho e trabalho intelectual é necessariamente individual, porque não existe tal coisa como uma "mente coletiva", assim como não existe um "estômago coletivo".

Um software proprietário não é necessariamente ruim porque não é livre. Ele é ruim quando representa um monopólio. Por exemplo, se o Windows tivesse sido mesmo a única opção, teríamos testemunhado o sucateamento da tecnologia. No longo prazo não teria sido bom nem pra própria Microsoft.

A Microsoft era um monopólio de fato. Podemos discutir eternamente se era mesmo ou não, se seu julgamento foi justo ou não. Mas o advento do software livre, particularmente dos sistemas operacionais baseados em GNU e Linux e depois o advento da Internet comercial, que desvinculou a necessidade de softwares nativos instalados localmente, quebrou esse monopólio. Com ou sem o Departamento de Justiça, o monopólio teria sido derrubado por pura dinâmica do mercado. Não no curto prazo, mas no longo prazo.

Imagine se, num futuro alternativo distópico, a Microsoft tivesse convencido o governo federal que o Windows era tão essencial à segurança nacional que seria do benefício da nação apoiar e manter seu monopólio?

Tudo teria parado. Até hoje estaríamos usando Pentium 100Mhz (porque, quem precisa de mais velocidade?), rodando um derivado de Windows 98 (porque, pra que criar um NT ou 2000 se o 98 já atende?), com Internet discada (porque, quem precisa de banda larga?), onde somente alguns poucos consumidores profissionais estariam usando (porque, pra que a população em geral precisa de computadores?)

Se alguém com a cabeça dos anos 80 e 90 tivesse sido decretado como a referência de "moral" e "necessidade social", pra que escolher evoluir qualquer coisa? Deixe as coisas como está. Mudanças fazem a população sofrer, pessoas perdem seus empregos, perdem sua segurança e estabilidade - ora, Inovação é sobre insegurança e instabilidade.

Não crie e-mail, isso pode prejudicar o emprego dos carteiros. Não crie websites, isso pode prejudicar quem trabalha nas fábricas de impressão de papel. Não crie software livre, isso pode prejudicar quem trabalha como programador do governo.

É isso que acontece quando se deposita as decisões que deveriam ser do mercado às mãos de um indivíduo ou mesmo uma entidade. Ela naturalmente se torna tirânica e despótica, porque é a única postura popularmente aceita como "moralmente correta".

Todo ditador, tirano, tem o mesmo discurso: "para o bem da população" e o inverso também é verdade: todos que tem o discurso de "para o bem da população, não importa os meios", quer se tornar um ditador tirânico.

Tirania e o Fim do Software Livre

Uma situação que começou a se tornar comum é a tentativa de desapropriação de uma propriedade privada para benefícios tirânicos usando como desculpa quaisquer causas sociais. Novamente, para deixar claro, não há culpa nas causas sociais em si, na maioria legítimas, mas sim em quem se auto-entitulou seus representantes ilegítimos.

É muito fácil pegar a conquista de outra pessoa e se apropriar dela em nome de uma "causa maior".

Não se engane, isso é simplesmente roubo. E não importa quantos apoiem esse tipo de coisa - coisa que é muito fácil nesse mundo de redes sociais. Roubo é errado, sempre, não existem justificativas.

Veja um cenário: você é um desenvolvedor que quis resolver um problema e criou um pedaço de software. Como achou que outros poderiam ajudar a evoluir sua solução, você disponibilizou como código aberto. Durante 20 anos muitos desenvolvedores se beneficiaram desses códigos e ajudaram a melhorá-lo. É a história de centenas de projetos de software livre.

Agora, em nome de diversas causas, não importa quais, alguém intervém e, por pura pressão social e propaganda, etiquetam todos os que estão há anos trabalhando, como verdadeiras párias anti-sociais. A mídia especializada - sempre à espreita de controvérsias para atrair mais leitores - noticia exatamente dessa forma.

E com o advento das Redes Sociais, por motivos que nada tem a ver com o mérito técnico e o valor que esse código aberto outrora tinha, vê seus colaboradores expostos e automaticamente julgados e condenados como criminosos perversos e anti-sociais.

Os donos do projeto eventualmente cedem e, por um pequeno período de tempo o projeto é obrigado a rodar sob intervenção despótica, onde suas identidades, passados, carreiras, vida particular, estarão sob constante escrutínio. Não mais seu código, mas suas opiniões pessoais. Não mais seu código, mas suas atitudes. Não mais seu código, mas sua aparência.

Consequentemente, se essa intervenção continuar, o valor do projeto decai drasticamente. Não há mais valor que o programador voluntário pode receber, pelo contrário, participar se tornou uma "liability", portanto a reação correta é deixar de participar. Um a um quem realmente estava produzindo código começa a desaparecer. E essa é uma forma perversa de tomar uma propriedade e remover seu valor.

Isso já começou a acontecer, ainda não tivemos casos que chegaram tão longe, mas é o que acontece quando você adiciona uma camada política sobre um projeto que era puramente livre. É assim que você destrói um Livre Mercado, dando poder político a quem não concorda com trocas voluntárias, apenas em remover propriedades de quem se "julga" que não merece para dar a quem se "julga" que merece.

Eu postei sobre isso em 2011 se estiver interessado.

"Eu sou um arquiteto. Eu sei o que se resulta pelos princípios sobre os quais foi construído. Estamos nos aproximando de um mundo onde eu não posso me permitir viver. Minhas idéias são minha propriedade. Eles foram tirados de mim à força, por quebra de contrato. Nenhuma escolha me foi dada."
"Acreditaram que meu trabalho pertencia a outros, para fazerem com ele o que quisessem. Eles clamaram por mim sem meu consentimento - que era meu dever servir a eles sem escolha ou recompensa."
"Agora vocês sabem porque eu dinamitem Courtland [condomínio de baixo custo]. Eu desenhei Courtland. Eu tornei possível. Eu destruí. Eu concordei em desenhá-lo pelo propósito de vê-lo como eu queria. Esse foi o preço que dei pelo meu trabalho. Eu não fui pago. Meu prédio foi desfigurado pelo capricho de outros que tomaram todos os benefícios do meu trabalho e não me deram nada em retorno."
"Eu vim aqui dizer que eu não reconheço o direito de qualquer um a um minuto da minha vida, ou a qualquer parte da minha energia, nem a qualquer conquista minha - não importa quem clame por isso!"
"Tinha que ser dito: O mundo está perecendo em uma orgia de auto-sacrifício. Eu vim aqui ser ouvido em nome de todos os homens independentes que ainda existem no mundo. Eu queria ditar meus próprios termos. Eu não me importo de trabalhar ou viver por causa de outros."
"Meus termos são: os DIREITOS de uma pessoa de existir apenas por si mesmo."

Isto é um aviso do pior caso que pode acontecer.

Conclusão

Toda esta exposição é para tirar o falso verniz "social" tão comumente associado a software livre.

Significa que se alguém contribui para software livre puramente porque acredita que isso vai trazer benefícios a causas sociais, é algo ruim? Claro que não, como eu disse, cada um se voluntaria porque obtém algum benefício nessa troca. Para alguns o pagamento é simplesmente a sensação de estar ajudando alguém. Alguns gostam de trocar seu código por consciência. Muitos fazem isso fora de software, chama-se "doação" ou "trabalho voluntário". E obviamente não existe nenhum problema justamente porque continua sendo uma troca voluntária.

Agora, resolver os problemas do mundo é algo extremamente complicado. É extremamente complicado até de começar a explicar. Mesmo pessoas que estão nisso com muito afinco tem dificuldade em obter os resultados que gostaria.

Bill Gates é notório por ter a visão de influenciar positivamente em realmente resolver os problemas do mundo e ele tem o melhor aviso à nossa geração de programadores que, independente de classe social, posse, raça, gênero, temos acesso à Internet e recursos suficiente para nos dar ao luxo de sermos "Drama Queens" em discussões irrelevantes em redes sociais sobre "salvar o mundo". E quem está fora desse circuito, realmente executando em vez de só falar?.

"Ótimo, vá até os centros da Infosys em Bangalore, mas saia do oásis e vá só 3 milhas um pouco mais pra fora e procure pelo cara que vive sem privada, sem água encanada ... O mundo não é plano e PCs não são, na hierarquia das necessidades humanas, entres os Top 5" - Bill Gates

Você está empenhado em tentar transformar uma propriedade privada (projeto de software livre) numa plataforma política para uma suposta "causa social"? Pense de novo. Você provavelmente está desperdiçando seu tempo em mais maneiras do que imagina.

Para fechar, vou dar um exemplo de um caso real que me foi relatado - mas removendo nomes e detalhes para proteger os envolvidos:

A: - Puxa vida, ninguém dá chance à minha [minoria].
B: - Não diga isso, porque você não me ajuda a resolver estes problemas neste projeto open source, daí você pode palestrar sobre isso.
(depois de algum tempo ... A some, B cansa de esperar e chama C, que ajuda a resolver os problemas e ganha o espaço de palestrar)
A: - tá vendo B, você é parte dos opressores que não dão espaço a gente como eu e preferiu dar a oportunidade pro C, que nem precisa e faz parte dos opressores.

E antes que você discorde porque se trata de conceitos de Ayn Rand (tem muita gente que não gosta dela, não sei porque), não seja um idiota e avalie a idéia por causa de seu autor, avalie o argumento que foi dado pelo seu próprio mérito. Muitos me criticam porque acham que eu "sigo" Ayn Rand, e não poderiam estar mais enganados: Ayn Rand simplesmente foi a única que pôs por escrito exatamente o que eu sempre pensei e como sempre agi desde que me conheço por gente. Poderia ter sido qualquer outra pessoa.

Aliás, da mesma forma, eu não sou "seguidor" de Bastiat, Friedman, Hayek, Menger ou qualquer outro. Eu sigo minha vida nos meus próprios termos, por princípios que eu mesmo defini anos atrás e essas referências são apenas isso: referências.

Considero um insulto à minha inteligência simplesmente me taxarem de seguidor de alguém e eu não respondo bem a "Ad Hominem" ou outros tipos de falácias.

"Voilà! In view, a humble vaudevillian veteran cast vicariously as both victim and villain by the vicissitudes of Fate. This visage, no mere veneer of vanity, is a vestige of the vox populi, now vacant, vanished. However, this valorous visitation of a bygone vexation stands vivified and has vowed to vanquish these venal and virulent vermin vanguarding vice and vouchsafing the violently vicious and voracious violation of volition! The only verdict is vengeance; a vendetta held as a votive, not in vain, for the value and veracity of such shall one day vindicate the vigilant and the virtuous. Verily, this vichyssoise of verbiage veers most verbose, so let me simply add that it's my very good honor to meet you and you may call me V." - V for Revenge

I just read a very well written and important article from OSS contributor Solnic and I have to agree with him in almost every technical point.

First of all, it's inevitable but I still hate dramatic headlines, even though I write like this myself sometimes. "My time with Rails is up" like it's saying "And you should leave Rails now too if you're smart". I dismissed the article entirely because of that. I was about to post a counter-rant to that without reading it properly, but now that I have, I wrote this new article from scratch.

Solnic is right: Ruby, by itself, has little to no future alone. Rails is a real monopoly in this community and most OSS projects are targeting Rails. And yes, Rails do encourage some bad practices and anti-patterns. This can be very discouraging to many OSS contributors, specially because to change the direction of a huge Elephant takes humongous effort.

To make it clear, the dialectic technical arguments he makes are all true. But half the article - as he stated himself - is just rant, pure rethoric. And I think it would be good to balance it out, which is what I will try to do in this post.

Accepting Reality

The reality is this: Rails is tailor made for Basecamp.

We all know that, or we should. Basecamp-like apps are not too difficult to make, at least in terms of architecture. You don't need fancy super performant languages with super duper highly concurrent and parallel primitives. Also a reality is that 80% of the web applications are Basecamp-like (disclosure: my feelings for years of experience in consulting). Which is why Rails has endured so far.

It's like content management systems, or CMS. Most of them are blog-like systems. And for that you should go ahead and install Wordpress. The very same arguments made against Rails can be done against Wordpress. And you should never, ever, tweak Wordpress to be anything but a blog-system. Try to make it into an e-commerce for high traffic, and you will suffer.

To make it clear: Wordpress has one of the most offensive source codes I've ever seen. I would hate having to maintain that codebase. I'm sorry if anyone from the Wordpress base of contributors is reading this, I say this without malevolence. And you know what? Possibly half of all CMSs in the world are Wordpress. Over a million websites.

Then you have the case for Magento2. Big rewrite over the original Magento, written using all the dreaded Zend stuff that everybody else dislikes. But it's huge. If you need a fast turn-key solution for e-commerce, look no further.

Do Wordpress plugins work with Magento? Nope. They are 2 fragmented, independent and isolated communities. But they both generate a lot of revenue, which is what covers the cost of redundancy between them. And this is not even counting Drupal, Joomla. PHP is one big ocean of disconnected islands. Countries with severe immigration laws.

Fragmentation is no stranger to the Javascript world. But it's a different kind of value generation. Facebook, Google, Microsoft, they all want to be the thought-leaders in the fast evolving Millenials generation. It's a long term strategy. And one of the elements of this game is the Browser. But not only in terms of Chrome vs Firefox vs IE, but also on how applications are implemented.

Facebook came up with React. Google came up with Polymer and Angular. The Node guys went through a power struggle with Joyent which almost resulted in further fragmentation but they settled for the Node Foundation.

Apple went all on war against Adobe's Flash and then only now Google is turning them off in Chrome, but they are all looting on the consequences for all the attention it brings in the Web Development communities.

Apple wants native to succeed and Swift to be the one language to lead it all. Google has conflicting strategies because they want native Instant Apps to succeed but if it fails, plan B continues to be for them to dominate HTML5/CSS3 based web apps with Angular. Facebook don't want to have their fate being decided by the power struggle between Apple and Google.

It's a complex power struggle unfolding, and you can see that it's not about technical prowess, it's not about value generation. It's about ego, influence and power. Very fitting for the YouTuber generation. And the web technologies are being held hostage in this siege, if you havent's noticed.

Then there is the issue that Ruby's future is now tightly coupled with Rails. This is a reality and if you're a Rubyist that don't like Rails, I feel bad for you. But not so much. For example, if Hanami is interesting I believe at least one company invested on it. If no one is using it, then it doesn't matter how technically superior it is. If Rom.rb is great someone should be using it, otherwise what's the point? Why create a technical marvel that no one wants? But if there is at least one company using it, it's enough reason to keep going, regardless of what happens to Rails or what DHH says or does.

People think that because something is "technically superior" everybody else should blindly adopt. But this is not how the market works.

Of all the cosmic-size events going on out there, I really don't sweat it that much if Ruby stays tied to Rails. What would it do without it?

All communities face fragmentation at some point. It's very difficult and expensive to maintain cohesiveness for a long time. The only community that I think achieved that through sheer force of regulation is Microsoft's .NET stack. It doesn't mean that there were no pressure from the outside. Rails itself played a big role into influencing the move from old-ASP.NET to ASP.NET MVC. Now they finally acquired Xamarin before .NET could steer out of their control in open source platforms they don't control.

Ruby on Rails is the only other "cohesive" community I've seen. With the upside that Basecamp doesn't need hundreds of thousands of developers to exist. A niche market would suffice, enough for the framework to evolve gradually through OSS processes. Which is why I always question the history and origins of tools and technologies to make my decisions on where to use them, not just technical prowess.

Rails works because it doesn't have to play politics with Apple, Facebook, Microsoft, Google or any other committees (by the way, by default, I never trust committees). Those who depend on Rails will do the house-keeping, directly. Heroku, Github, New Relic, Shopify, and many talented developers.

3 Laws of Market Reality

It's easy to over-analyse something after the fact. 10 years down the road, I can easily trace back an optimal path, avoiding all boobtraps and obstacles along the way. Doesn't make me any genius, just shows that I can connect the - now clearly visible - dots.
No solution implementation is perfect. If it actually solves a real problem it's bound to be imperfect. If it solves a real problem in a fast paced changing market, the more imperfect.
Either you build tools because your core business applications depend on it or you build tools to sell. The former will usually be better - in terms of market-fit - than the latter. So if you have to blindly choose, go with the former.

So, first of all, I will always prefer tools that solve a real problem made by those that actually depend on them. Otherwise, the shoemaker's son will end up barefoot. Case in point: I will definitely use Angular, if I have to. But I would never, ever, begin a new business that depends solely on Angular to survive. Why? Because Google doesn't need it. It didn't blink to give up GWT, it didn't have to think twice to decide to rewrite Angular 2 in an incompatible way to Angular 1, and so on.

Second of all, I will always see what other external factors will influence the fate of the technology. Imagine that I spent a whole lot of time writing scripts, libraries, tools for Grunt. Then people decide it's bad. Now Gulp is the better choice - and you will find plenty of technical reasons. Now you invest a lot of time writing everything you had for Grunt to Gulp. Then, for plenty of other reasons people decide that Webpack is the best choice. And there you go again. NIH (Not-invented-here) syndrome gallore.

This is clearly a small bubble. It's tulips all over again. There are too many big players (Facebook, Google, Apple, Microsoft, Mozilla, etc) with big pockets, plenty of time and resources. This is how an experimental lab works in public. Lots and lots of experimental alternatives and several businesses blindly choosing depending on the best sales pitch of the week.

Sometimes this kind of situation makes the monopoly of ASP.NET on the Microsoft camp and the Rails monopoly on the Ruby camp seen innocuous. And yes, I compared Rails to .NET here. They are the 2 most comparable stacks. Possibly comparable to Spring faction in the Java camp. If you remember the history, Spring was like Rails in the Java community, rising up against the humongous complexity of the official J2EE stack back in 2002. And then Spring itself became the new J2EE-like behemoth to beat.

This is a millenia old dillema:

"You either die a hero, or live long enough to see yourself become the villain."

Why is Rails a problem now?

As I said before, I agree to almost every technical problem that Solnic outlined.

Rails is indeed the brainchild of David Hansson (DHH). DHH is a person, with a very big ego, and a business to run. You can't expect any person to be reasonable all the time, specially one that pushed something up from zero, both a business and a technology platform.

When it started, people defected from Java, .NET, PHP even Python, in droves and they all acknowledged how interesting Rails was compared to J2EE, ASP.NET, Plone. It offered not only productivity but technical enjoyment. We were discussing the wondreous world of dynamic languages, open classes, injecting behavior on the fly (aka monkey patching), we all stood up and aplauded dropping all unnecessary abstrations.

We could not have enough of our Ruby fix, spitting out all the Perl-like magic we would accomplish in a language that didn't feel ugly as PHP or bureacratic like Java or C#. And they all laughed.

The Golden Age from 2004 to 2006 saw a never ending stream of celebratory masturbation of Perl-like coding prowess. We learned Ruby through Why, the Lucky Stiff most obscure black magic, remember that? It was everything but clean and modular architectures.

Then we entered the Silver Age, from 2007 to around 2011. Rails actually went too far, too fast. Suddenly we saw big companies popping up from everywhere! Twitter, Github, Engine Yard, Heroku, Zendesk, Airbnb, everybody drunk the Rails cool aid. The opportunity was there to offer something for the enterprise. Merb was ahead of its time and it was pitched the wrong way. I do think that confronting the almighty Rails upfront, at that point, was not smart. You should expect overreaction and it did came and it was a swift blow. I will be honest and say that I was very aprehensive in 2009 and 2010 to see if the Rails 3 pseudo-rewrite, pseudo-Merb-merge would actually come through.

2011 to 2016 was the Bronze Age, a bittersweet period. That's because many new languages have emerged and finally reached "usable" state. From JS's V8 and Node.js, to Rust, to Clojure, to Elixir, and even some gems from the past started to get attention, such as Scala and even Haskell. The most important change of all: 2010 saw the advent of the Walled Garden App Store, commercially available native applications for smartphone and tablets. Forget web development: mobile was getting it all.

And that's when all the big companies started to show their deep pockets. Apple releases Swift. Google released Dart, Angular, Go. Microsoft released Typescript, ASP.NET MVC then vNext and had it's hands full working on Windows 10. Facebook entered the game late by releasing React and then React Native.

Rails can now be considered a "problem" for the very same reasons that made it popular in the first place. And this is bound to happen to any technology.

But you can't change the architecture of Rails too much, otherwise you risk breaking down very big chunks of the projects that are deployed in production now. And when someone has to rewrite big chunks of a project you might as well consider rewriting it in something else entirely.

Many people are doing exactly that, specially for APIs and web applications implemented as SPAs talking to APIs. The APIs can be written in Go, Elixir, Scala and avoid Ruby altogether. You lose the fast turn around of the Rails ecosystem, but if you can afford it (you're a Unicorn startup with deep pockets), why not?

But again, for the 90% of small to medium projects out there, you can still get the best punch for the buck using Rails and all the libraries available for Rails. It's like saying, if you want to build a blog, go for Wordpress and you will get the best benefit for the limited resources you have. Don't try to be fancy and write an SPA blog using Go APIs with React from scratch. Feasible, but not worth it.

If you're a medium company already using Rails for some time, first of all make sure you adhere to basic best practices. Add tests and specs if you haven't already. Steadily refactor code, remove duplication, upgrade gems. Then you should consider adding an abstration layer such as Trailblazer and possibly consider componentizing parts of your application as Rails Engines or removing those parts into separated Rails-API applications to be consumed, if possible. But do one step at a time, as needed.

One rarely benefits from big bang rewrites from scratch.

Conclusion

So yes, for developers such as Solnic the Rails community is probably a frustrating place to be. But it's also an addiction that's hard to drop because Rails is so much larger than any other competitor in any other new and fancy platform, you always feel bad for being the underdog.

Rails went from underdog to mainstream in 5 years. Possibly the fastest growth any web framework ever achieved. The life of a web developer from 1995 to 2003 was not particularly interesting. Rails did a lot to improve it. And if anyone thinks they can do better, just do it. What's the point of writing about Rails? More than just code competing against code, results should compete against results.

Active Record's architecture will indeed hurt hard maybe 10% of the cases out there. Active Support does not have a better alternative so far, and just removing it won't bring anything of value for the end user. Replacing a big component such as Active Record for something "better" such as an improved version of DataMapper or Rom.rb as the default again won't bring so much value, specially for the hundreds of applications out there. You're telling everybody to just rewrite everything. And if I would have to rewrite, I would definitely do a new application using Rails + Trailblazer or go straight to Hanami. But most people would decide in favor of ditching Ruby altogether.

Could Active Record be better? Sure! We have old Data Mapper, Sequel and ROM.rb to prove it. But the real question is: could it be done better back in 2004 when it was first created? I don't think so. Now even the creator of DataMapper advocates for No-ORM. In 2004 "NoSQL" wasn't even a thing. The best we had back then was Hibernate, way before JPA! And for all intents and purposes, Active Record still does much better than average. But if you're big, you should be careful. That's all.

The other communities will face the same predicaments we are now facing in the Rails community. It's inevitable. Everything is so much easier when you have a small community that even if you break things it won't be too bad. It's much harder to maintain something that actually became big beyond your most optimistic scenarios.

I do understand the conservative approach DHH is taking by not making big disruptions. If this is something he is doing because he believes in conservative moves or because he doesn't understand better architectural options is not up to me to judge, but it's a valid move that will alienate advanced developers like Solnic but still allow for beginners to jump right into it without worrying too much right now about too many abstractions.

Update: DHH commented on this section later:

@akitaonrails It’s not out of conservatism. But a difference of opinion and values. I love Active Record. Love callbacks.
— DHH (@dhh) May 24, 2016

@akitaonrails Will everyone love the same techniques and methods as me? Of course not (see RSpec for just one popular example!). That’s fine
— DHH (@dhh) May 24, 2016

People forget that abstractions are very nice for advanced developers that had suffered the lack of them. But beginners will always suffer if presented with too many architectural choices upfront. Understanding GoF Design Patterns, Data-Driven Design, SOLID, Enterprise Architectures, etc is very overwhelming. Experienced people often forget the learning curve when they were themselves beginners, and at that time Rails was so attractive, so sexy. Remember that feeling? Of accomplishment in the face of the knowledge that some nice witch left super useful black magic behind?

Rails has won for it's simplicity for beginners, having a "rails"-like guidance for experienced people as well, and somewhat acceptable flexibility for more advanced developers. Will it be able to maintain another 10 years in face of the many smaller alternatives out there trying to recreate everything from scratch? Time will tell.

I think it would be a good fit to finish with Bob Dylan:

Come gather 'round people
Wherever you roam
And admit that the waters
Around you have grown
And accept it that soon
You'll be drenched to the bone.
If your time to you
Is worth savin'
Then you better start swimmin'
Or you'll sink like a stone
For the times they are a-changin'.
...

If you're following me you will see that I have been sidetracking to Elixir recently. I will write a big introduction before diving into Crystal, so bear with me.

I'll say that the Erlang OTP and it's internals do intrigue me more than just the novelty of the language syntax. I believe that if you want to build highly distributed and also highly reliable systems you will either use Erlang (through Elixir) or you will end up replicating most of what Erlang already has with OTP.

This is true if you see that there are many Java/Scala-based distributed systems such as Hadoop, Spark, Cassandra, and they all could have benefitted from Erlang's OTP architecture. On the other hand, if you follow Cassandra you will see that even Java still has a hard time competing with C++ if you compare the clone called ScyllaDB.

I believe Elixir (perhaps with HiPE) can compete in the same league as Java for distributed systems avoiding people to leave the Erlang platform for C or Java as has happened with CouchDB back in 2012 because of lack of interest and exotic syntax. And because Java already has a big ecosystem, Clojure is in the game primarily because it can interface directly with it.

I think Go is also a Java contender but for other use cases, particularly in the systems tools space where you have lots of C/C++ and also glue code in Perl and Python. An obvious example being Docker orchestration. Yes, you can do microservices, crawlers and other stuff but I am not so keen to build big "applications" in it, althought there is nothing preventing it. Again, it's just a personal opinion.

Rust is a big contender to low-level systems development. I think it can replace C in many use cases and be used to implement libraries and components to be used by other languages, with the added benefits of a modern language that tries to avoid security hazards such as the recent Heartbleed debacle. But because of it's "Borrow system" I find it extremely bureacratic, specially if you're coming from highly dynamic languages such as Ruby, Python or even Swift or Elixir. I'm not saying it's a bad thing, just something that take way more time to become comfortable with than I expected.

Crystal is something between Rust and Go. LLVM is something you must have in your radar, because of Apple's support towards Swift it's better than ever. Crystal is similar to RubyMotion in terms of both being similar to Ruby but not fully compatible and both being front-end parsers to LLVM. RubyMotion, on the other hand, is closer to Ruby and can even use some Ruby libraries almost without changes.

You do have to take your hats off to Ary Borenszweig and his contributors. It's very impressive how far they got in such a short period of time and without having deep pockets from Mozilla, Apple or Google.

Limitations

Crystal borrows heavily from Ruby but it's not a goal to reach any level of compatibility. It's a strong and static type language with Type Inference. It does not have any runtime component such as RubyMotion or Swift, so it has no notion of introspection or reflection over objects.

Mainly, you have neither "#eval" nor "#send". The lack of runtime evaluation is not so bad as you do have compilation-time AST manipulation through a good enough system of Macros (more on that later).

But the lack of "#send" does hurt a bit. It's the one thing that makes Crystal farther away from Ruby dynamic object flexibility. But it's also understandable why it's not a priority to have it.

As of version 0.17.0, Crystal has one huge limitation: it's using a Boehm-Demers-Weiser conservative garbage collector.

@asterite @headius I assumed as much... the Boehm GC, although handy for starting a new language, will only get you so far
— Jason Frey (@Fryguy9) May 29, 2016

The language is currently implemented as a single-threaded process. This means that you probably can't max out all CPUs of your machine with just a single process. Although I may be wrong here, it's just my first impressions.

Charles Nutter (from JRuby fame) gives a warn about this:

@fryguy9 From their site: https://t.co/uOjIT0c8ji It does say something about "when" parallelism happens. Risky to start single-threaded.
— Charles Nutter (@headius) May 29, 2016

It's a double-edged sword. Using a "generic" plug-and-play GC such as Boehm - which is not a bad thing in itself, but it's possibly not nearly as powerful as the JVM's own set of high performance GC such as the brand new G1GC.

Making the language be single-threaded by default also means that the entire standard library and all the ecosystem is built assuming there is no parallelism. So there are no race-conditions and hence, no usage of proper synchronization anywhere. If one day they decide to add multi-threading, all existing code will not be thread-safe. This is probably the meaning of Charles' warning.

I don't think it's a pressing problem though. You probably won't have a highly parallel system built with Crystal but maybe it's not the use case. In that case you should be using Elixir + OTP, Scala + Akka, for example.

Concurrency

Not having access to native Threads doesn't mean that you can't have Concurrency. Concurrency is not Parallelism. Since version 0.7.0 they have added support for non-blocking I/O and also Fibers and Channels.

In a nutshell, a Fiber is a special kind of coroutine. It's a "piece of execution" that can "pause" itself and yield control of the execution back to its caller. It can pause itself but it can't be paused from the outside like Erlang's Scheduler can pause its running processes, for example.

In this case we can have a "Fiber Scheduler" of sorts, that can "resume" other Fibers. It is currently single-threaded, of course. And it works as an Event Loop. Whenever you have a non-blocking I/O operation waiting for file reading, network packet sending and stuff like that, it can yield control back to other fibers to resume work.

This makes it work more or less like Node.js. Javascript is also a single-threaded language and Node.js is basically an implementation of an event loop. Functions declare other anonymous functions as callbacks and it works based on the I/O triggers to callback those functions on every tick of the reactor.

On top of that you can add "Channels" as popularized by Google's Go language. You can start as many channels as you like. Then you can spawn Fibers in the Scheduler. Fibers can execute and keep sending messages through the Channels. Execution control is yielded to whoever is expecting to receive from the same Channels. Once one of them receives and executes, control is sent back to the Scheduler to allow other spawned Fibers to execute, and they can keep "pinging" and "ponging" like this.

Speaking of ping-pong, you have this snippet in the "Go by Example" site:

12345678910111213141516

package mainimport "fmt"func ping(pings chan<- string, msg string) {    pings <- msg}func pong(pings <-chan string, pongs chan<- string) {    msg := <-pings    pongs <- msg}func main() {    pings := make(chan string, 1)    pongs := make(chan string, 1)    ping(pings, "passed message")    pong(pings, pongs)    fmt.Println(<-pongs)}

And this is mostly the same thing but implemented in Crystal:

123456789101112131415

defping(pings, message)  pings.send messageenddefpong(pings, pongs)  message = pings.receive  pongs.send messageendpings = Channel(String).newpongs = Channel(String).newspawn ping pings, "passed message"spawn pong pings, pongsputs pongs.receive# => "passed message"

Same thing, but more pleasant to my eyes, again a personal opinion - which is expected if you're also a Rubyist.

So Crystal has Node.js/Javascript-like Event Loop in the form of a Fiber Scheduler and a Go-like Channel/CSP mechanism to coordinate concurrent processing. It's a "cooperative multitasking" mechanism. It's not good if you have CPU intensive processing, such as heavy number crunching, data processing. It works best for I/O intensive processing.

If you have a Fiber doing super heavy processing, it will block the Scheduler, as is also true for Node.js. The current implementation of the Scheduler is bound to a single native thread so it can't use other available native threads of the system for now.

Because the Channel implementation is quite new, it's not used throughout the standard library. But the standard implementation for HTTP::Server uses Fibers. And because it is compiled down to a native binary it's way faster than Javascript, as this "fast-http-server" shows:

fast-http-server (Crystal) 18348.47rpm at 8.67ms avg. response time
http-server (Node.js) 2105.55rpm at 47.92ms avg. response time
SimpleHTTPServer (Python) 785.14rpm at 1.91ms avg. response time

We're talking about an order of magnitude of 8 times faster than Node.js and more than 20 times faster than Python.

As usual, you shouldn't blindly trust synthetic benchmarks but depending on which kind of processing you decide to compare against, you will see Crystal being pretty on par with Rust, Go, Swift, D.

Type Inference

Crystal is a statically typed language. But it won't require you to declare every single type beforehand. It's quite smart in its Type Inference so if you just type:

12	a = 1b = "HELO"

It will know that "a" is an "Int32" and that "b" is a "String". By the way, contrary to Ruby, all Strings must be double-quoted. But it becomes particularly more complicated when you're dealing with complicated data structures such as JSON.

In Ruby I can just parse a JSON String and immediatelly explore its structure like this:

1	parsed_json = JSON.parse(response.body)["files"].first["id"]

I can't do this in Crystal, instead the recommended approach is to declare a schema-like structure like this:

12345678910111213141516171819

classFilesResponseJSON.mapping({ok: {type: Bool},files: {type: Array(FilesItemResponse)}  })endclassFilesItemResponseJSON.mapping({id: {type: String},name: {type: String},filetype: {type: String},created: {type: Time, converter: Time::EpochConverter}timestamp: {type: Time, converter: Time::EpochConverter}permalink: {type: String}  })end...# parsed_json = JSON.parse(response.body)["files"].first["id"]parsed_json = FilesResponse.from_json(@body).files.first.id

I am very used to Ruby's duck typing and the ability to query objects in runtime. It's just not the same in Crystal and it can become quite tedious to change your mindset to think about types beforehand. The compiler will scream a hell of a lot during your development cycle until you become acquainted with this concept.

I did an experiment with JSON parsing and the result is a project I called "cr_slack_cleanup" which showcases both this idea of JSON Schemas as well as Fibers and Channels as I explained in the previous section.

Update: after I posted this article @LuisLavena stepped in to correct me: you can do it like Ruby, without schemas:

@AkitaOnRails @CrystalLanguage re: JSON, you can parse it kinda-like Ruby, see JSON::Any: https://t.co/MD7PYy5AVH
— Luis Lavena (@luislavena) June 1, 2016

And it will be like this:

12345678910111213141516171819202122

require "json"data = <<-JSON  {"files": [        {"id": 1,"name": "photo.jpg"        },        {"id": 99,"name": "another.jpg"        }    ]  }  JSONobj = JSON.parse(data)obj["files"].each do |item|  puts item["id"].as_iend

So yeah, JSON parsing can be almost identical to what you would do with Ruby, although he also recommends Schemas as it's a bit faster anyway.

Macros

The last important feature worth mentioning is the presence of Macros. As you don't have control over your code in runtime, you don't have "eval", you don't have "send", then you would not have any way to do proper metaprogramming.

Macros bring back some of Ruby's metaprogramming. For example, in Ruby, when you include a Module to a Class, there is the "included?" hook in the Module where you can add some metaprogramming such as "class_eval" stuff.

In Crystal, a Module does have an "included" hook but now it's a Macro. You can do something like this:

123456789101112131415161718192021222324

moduleFoo  macro included    {% if@type.name == "Bar" %}defself.hello      puts "Bar"end    {% else%}defself.hello      puts "HELLO"end    {% end%}endendmoduleBar  include FooendmoduleSomething  include FooendBar.hello # => "Bar"Something.hello # => "HELLO"

It feels like having "ERB" templates but for code. A macro is code building code in compile time. In the resulting AST it's as if you wrote the boring repetitive code yourself. The compiled native binary doesn't care. If you're from C it's like pre-processing but having control of an AST tree of Nodes instead of just manipulating the source code. You can even have something akin to the venerable "#method_missing".

12345678910

classFoo  macro method_missing(name, args, block)    {{pp name}}endendFoo.new.hello_world# => name = "hello_world"Foo.new.bla_bla("bla")# => name = "bla_bla"

Any macro function can receive an AST as the argument and you can manipulate this AST in any way as you see fit. For example:

12345678

module Foo  macro teste(ast)    puts {{ast.stringify}}  endendFoo.teste "Hello World".split(" ").join(" - ")# => ("Hello World".split(" ")).join(" - ")

In this example you just get a normalized version of the code you passed as argument. And the code won't be "executed" at that location.

So, you can define new methods and call different versions of methods depending on the combination of elements that you get from the AST. One example of what is possible is this experiment from developer Oleksii that I bundles in this small project I called "cr_chainable_methods" and we can build code that is very different from either Crystal or Ruby:

result = pipe "Hello World"  .>> Foo.split_words  .>> Foo.append_message("Bar")  .>> Bar.add_something  .>> Foo.join  .>> unwrap

(Yes, a subset of what a real Elixir Pipe can do.)

Crystal Shards

Finally, one aspect I like about Crystal is that it has task management and dependency management already figured out. No pointless discussions on which implementation of packages and dependencies we "should" be using.

You can just write a simple script into a file such as "test.cr" and run this file like this:

1	crystal test.cr

Or, you can build and compile to a native binary before executing like this:

12	crystal build test.cr --release./test

And you can also start in a proper project structure and development lifecycle like this:

crystal init app testcd testcrystal build src/test.cr --release./testgit add .git commit -m "initial commit"git push -u origin master

It sets up a proper project directory structure, a Git repository is pre-initialized for you (why would you be programming without Git!?) with proper ".gitignore" file.

You will also find a "shard.yml" file where you can declare your application or library name, version, author, and also its dependencies like this:

123456789101112

name: kemalversion: 0.12.0dependencies:radix:github: luislavena/radixversion: 0.3.0kilt:github: jeromegn/kiltversion: 0.3.3author:  - Serdar Dogruyol <dogruyolserdar@gmail.com>

This example is from the Kemal web framework by Serdar. It depends on the Radix and Kilt libraries. You must run crystal deps and it will fetch from Github before you can compile.

More than that: every project receives a proper "spec" directory where you can write Rspec-like tests and you can run them using crystal spec. And the spec runner results will look like this:

$ crystal spec........................................................Finished in 8.28 milliseconds56 examples, 0 failures, 0 errors, 0 pending

All built-in, no external libraries needed.

And 8 milliseconds to run the entire test suite of a micro web framework? Not too shabby. If you add up start up time for the executable, it still is around 1 second of total running time. Pretty impressive.

Conclusion

Crystal is not meant to be a Ruby replacement like JRuby. So it will never run existing and complicated Ruby projects such as Ruby on Rails or even Sinatra for that matter.

To me Crystal fills a gap in some use cases where I would use Rust or Go. It has the raw performance of an LLVM-optimized native binary, a reasonably fast Go-like concurrency system (and much faster and less resource intensive than Node.js), and the added benefit of a much more enjoyable Ruby-like syntax set.

You can even start experimenting with Crystal to build Ruby native extensions without the need of FFI and without having to write C. There is an attempt port of ActiveSupport as a native extension to MRI Ruby, written as a proof of concept in Crystal.

If you're a Rubyist you will find yourself pretty much at home with most of the standard library. You will miss a thing or two but most of the time it will feel like Ruby. The official Guide is good enough to get started and there is a comprehensive API documentation just to check if that favorite Ruby API is there or not, and if not what the replacements are. Most of the time you will think "hmm, in Ruby I would use Enumerator#each ... and yes, sure enough, there is Enumerator#each as expected." The API documentation could use some more love, so if you want to contribute this is a good place to start.

If you want to check out a curated list of Crystal libraries and application, go straight to "Awesome Crystal". You can also see the more dynamic "Trending Projects in Github".

For the most part, your proficiency in Ruby will payoff to build small systems where raw performance and concurrency are really necessary. This is exactly the use case for Mike Perham's port of Sidekick.cr. Say you have gigabytes of files to crunch. Or you have thousands of external websites to crawl, parse and organize. Or you have petabytes of data in your database waiting to be number crunched. Those are tasks that MRI Ruby won't help but Crystal may well be the quick answer.

You can even get one of the many micro web frameworks such as Kemal and deploy your HTTP API providers directly to Heroku now. This will give you Go-levels of performance, and this is very compelling for me as I really also dislike Go's archaic smelling syntax.

So, Rust, Go, Javascript, all very fast, but with very questionable syntaxes and not so enjoyable. They are way more mature, and their ecosystems are much larger. But unless I really need to, I'd rather choose Crystal for the use cases I described above, and I think you will find some good use for it too.

Update: after I posted, the creator of Crystal, @Asterite had a few things to add as well:

@AkitaOnRails @waj so it's likely that we'll have something similar. We aren't there yet because we are not Google, Mozilla nor Apple
— Ary Borenszweig (@asterite) May 31, 2016

After playing around with languages different than Ruby (such as Elixir or Crystal), it was time to go back to Ruby and see how JRuby is performing nowadays.

JRuby always pursued the goal of being a competent MRI replacement. If you have not followed what's going on, JRuby changed version schemes when they switched from the 1.7 series to the 9.x series. You should read this blog post that explains it.

In summary, if you want MRI 2.2 compatibility you must use JRuby 9.0.x and if you want MRI 2.3 compatibility you must use JRuby 9.1.x series. The most current release being 9.1.2.0. Anything prior to that you can refer to this table of versions from Heroku's documentation.

There are important recommendations as well:

Ideally you should use Rails 4.2. Try to be at least above 4.0, and you can turn on config.threadsafe! by default in the "config/environments/production.rb" file. To understand this subject, refer to Tenderlove's excellent explanation.
If you're deploying to Heroku, don't bother trying the free or 1X dyno, which only gives you 512Mb of RAM. While it is enough for most small Rails applications (even with 2 or 3 Puma concurrent workers), I found that even the smallest apps can easily go above that. So you must consider at least the 2X dyno. In any server configuration, always consider more than 1Gb of RAM.

Gems

There are several gems with C extensions that just won't work. Some of them have drop-in replacements, some don't. You should refer to their Wiki for a list of cases.

In my small sample application - which is nothing more than a content website backed by a Postgresql database, ActiveAdmin to manage content, RMagick + Paperclip (yes, it's an old app) to handle image upload and resizing, there was not a lot to change. The important bits of my "Gemfile" end up looking like this:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374

source 'https://rubygems.org'ruby '2.3.0', :engine => 'jruby', :engine_version => '9.1.1.0'# ruby '2.3.1'gem 'rails', '~> 4.2.6'gem 'devise'gem 'haml'gem 'puma'gem 'rack-attack'gem 'rack-timeout'gem 'rakismet'# Databasegem 'pg', platforms: :rubygem 'activerecord-jdbcpostgresql-adapter', platforms: :jruby# Cachegem 'dalli'gem "actionpack-action_caching", github: "rails/actionpack-action_caching"# Admingem 'activeadmin', github: 'activeadmin'gem 'active_skin'# Assetsgem 'therubyracer', platforms: :rubygem 'therubyrhino', platforms: :jrubygem 'asset_sync'gem 'jquery-ui-rails'gem 'sass-rails'gem 'uglifier',     '>= 1.3.0'gem 'coffee-rails', '~> 4.0.0'gem 'compass-rails'gem 'sprockets', '~>2.11.0'# Image Processinggem 'paperclip'gem 'fog'gem 'rmagick', platforms: :rubygem 'rmagick4j', platforms: :jrubygroup :testdo  gem 'shoulda-matchers', require: falseendgroup :test, :developmentdo  gem "sqlite3", platforms: :ruby  gem "activerecord-jdbcsqlite3-adapter", platforms: :jruby# Pretty printed test output  gem 'turn', require: false  gem 'jasmine'  gem 'pry-rails'  gem 'rspec'  gem 'rspec-rails'  gem 'capybara'  gem 'poltergeist'  gem 'database_cleaner', '< 1.1.0'  gem 'letter_opener'  gem 'dotenv-rails'endgroup :productiondo  gem 'rails_12factor'  gem 'rack-cache', require: 'rack/cache'  gem 'memcachier'  gem 'newrelic_rpm'end

Notice how I paired gems for the :ruby and :jruby platforms. After doing this change and bundle install everything, I ran my Rspec suite and - fortunatelly - they all passed on the first run without any further changes! Your mileage will vary depending on the complexity of your application, so have your tests ready.

Puma

In the case of Puma, the configuration is a bit trickier, mine looks like this:

123456789101112131415161718

web_concurrency = Integer(ENV['WEB_CONCURRENCY'] || 1)if web_concurrency > 1  workers web_concurrency  preload_app!endthreads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)threads threads_count, threads_countrackup      DefaultRackupport        ENV['PORT']     || 3000environment ENV['RACK_ENV'] || 'development'on_worker_boot do# Worker specific setup for Rails 4.1+# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-bootActiveRecord::Base.establish_connectionend

It's a bit different than you might find in other documentations. The important part is to turn off workers and preload_app when loading over JRuby. It will complain and crash. On my original MRI deploy I am using a small 1X dyno and I can leave WEB_CONCURRENCY=2 and RAILS_MAX_THREADS=5 but on the JRuby deploy I set it to WEB_CONCURRENCY=1 (to turn off workers) and RAILS_MAX_THREADS=16 (because I am assuming JRuby can handle more multithreading than MRI).

Another important bit, most people still assume that MRI can't take advantage of native parallel threads at all because of the dared GIL (Global Interpreter Lock), but this is not entirely true. MRI Ruby can parallelize threads on I/O waits. So, if a part of your app is waiting for database to process and return rows, for example, another thread can take over and do something else, in parallel. It's not totally multi-threaded, but it can do some concurrency so setting a small amount of threads for Puma might help a bit.

Do not forget to set the Pool size to at least the same number of RAILS_MAX_THREADS. You can either use the config/database.yml for Rails 4.1+ or add an initializer. Follow Heroku's documentation on how to do so.

Initial Benchmarks

So, I was able to successfully deploy a secondary JRuby version of my original MRI-based Rails app.

The original app is still in a Hobby Dyno, sized at 512Mb of RAM. The secondary app needed a Standard 1X Dyno, sized at 1Gb of RAM.

The app itself is super simple and as I'm using caching - as you always should! -, the response time is very low, on the tens of milliseconds.

I tried to use the Boom tool to benchmark the requests. I did a lot of warming up on the JRuby version, running the benchmarks multiple times and even using Loader.io for added pressure.

I am running this test:

1	$ boom -n 200 -c 50 http://foo-my-site/

The MRI version performs like this:

12345678910111213141516171819202122232425262728293031323334

Summary:  Total:    16.4254 secs  Slowest:  9.0785 secs  Fastest:  0.8362 secs  Average:  2.6551 secs  Requests/sec: 12.1763  Total data:   28837306 bytes  Size/request: 144186 bytesStatus code distribution:  [200] 200 responsesResponse time histogram:  0.836 [1] |  1.660 [57]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  2.485 [57]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  3.309 [33]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  4.133 [22]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  4.957 [16]    |∎∎∎∎∎∎∎∎∎∎∎  5.782 [6] |∎∎∎∎  6.606 [3] |∎∎  7.430 [1] |  8.254 [3] |∎∎  9.079 [1] |Latency distribution:  10% in 1.2391 secs  25% in 1.5910 secs  50% in 2.1974 secs  75% in 3.4327 secs  90% in 4.5580 secs  95% in 5.6727 secs  99% in 8.1567 secs

And the JRuby version performs like this:

123456789101112131415161718192021222324252627282930313233

Summary:  Total:    15.5784 secs  Slowest:  7.4106 secs  Fastest:  0.5770 secs  Average:  2.3224 secs  Requests/sec: 12.8383  Total data:   28848475 bytes  Size/request: 144242 bytesStatus code distribution:  [200] 200 responsesResponse time histogram:  0.577 [1] |  1.260 [23]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎  1.944 [62]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  2.627 [51]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  3.310 [24]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  3.994 [26]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎  4.677 [8] |∎∎∎∎∎  5.361 [1] |  6.044 [0] |  6.727 [1] |  7.411 [3] |∎Latency distribution:  10% in 1.1599 secs  25% in 1.5154 secs  50% in 2.0781 secs  75% in 2.8909 secs  90% in 3.7409 secs  95% in 4.2556 secs  99% in 6.7685 secs

In general, I'd say that they are around the same. As this is not a particularly CPU-intensive processing, and most of the time is spent going through the Rails stack and hitting Memcachier to pull back the same content all the time, maybe it's only fair to expect similar results.

On the other hand, I'm not sure I'm using the tool in the best way possible. The log says something like this for every request:

1234567891011121314

source=rack-timeout id=c8ad5f0c-b5c1-47ec-b88b-3fc597ab01dc wait=29ms timeout=20000ms state=readyStarted GET "/" for XXX.35.10.XXX at 2016-06-03 18:54:34 +0000Processing by HomeController#home as HTMLRead fragment views/radiant-XXXX-XXXXX.herokuapp.com/en (6.0ms)Completed 200 OK in 8ms (ActiveRecord: 0.0ms)source=rack-timeout id=c8ad5f0c-b5c1-47ec-b88b-3fc597ab01dc wait=29ms timeout=20000ms service=19ms state=completedsource=rack-timeout id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 wait=35ms timeout=20000ms state=readyStarted GET "/" for XXX.35.10.XXX at 2016-06-03 18:54:36 +0000Processing by HomeController#home as HTMLRead fragment views/radiant-XXXX-XXXXX.herokuapp.com/en (6.0ms)Completed 200 OK in 9ms (ActiveRecord: 0.0ms)source=rack-timeout id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 wait=35ms timeout=20000ms service=21ms state=completedat=info method=GET path="/" host=radiant-XXXX-XXXXX.herokuapp.com request_id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 fwd="XXX.35.10.XXX" dyno=web.1 connect=1ms service=38ms status=200 bytes=144608

The times reported by the Boom tool are much larger (2 seconds?) than the processing times in the logs (10ms?). Even considering some overhead for the router and so on, still it's a big difference, I wonder if it's being queued for too long because the app is not being able to respond more of the concurrent requests.

The amount of requests divided by the number of concurrent connections will bring the overall performance and throughput down if you increase it too much, I wasn't able to go too much above 14 rpm with this configuration, though.

If you have more experience with http benchmarking and you can spot something wrong I am doing here, please let me know in the comments section below.

Conclusion

JRuby continues to evolve, and you might benefit if you already have a large set of Dynos of large servers around. I wouldn't recommend it for small to medium applications.

I've seen many orders of magnitude improvements in specific use cases (I believe it was a very high traffic API endpoint). This particular case I tested is probably not its sweet spot and changing from MRI to JRuby didn't give me too much advantage, so in this case I would recommend sticking to MRI.

Startup time is still an issue. There is an entry in their Wiki giving some recommendations, but even in the Heroku deploy I ended up having R10 errors (Boot Timeout) every once in a while for this small app.

I didn't try increasing the dynos to the Performance tier introduced last year. I would bet that JRuby would be better at those and more able to leverage the extra power of having from 2.5GB up to 14GB if you have really big traffic (on the order of thousands or tens of thousands of requests per minute). With MRI the recommendation would be using small-ish dynos (2X or at most Performance-M dynos) and scale horizontally. With JRuby you could have less dynos with larger sizes (Performance-L, for example). Again, depends on your case.

Don't take the benchmarks above as "facts" to generalize everywhere, they are just there to give you a notion of an specific use case of mine. Your mileage will vary, so you must test it out yourself.

Another use case (that I did not test) is not to just "port" an MRI app to JRuby but leverage JRuby's unique strenghts as this post from Heroku explains, in the case of using JRuby with Ratpack, for example.

All in all, JRuby is still a great project to experiment. MRI itself came a long way as well and 2.3.1 is not bad at all. Most of the time it's down to your entire architecture choices, not just the language. If you didn't try it yet, you definitely should. It "just works".

I have this old pet project of mine called Manga Downloadr that I published in 2014. It was a very rough version. I was experimenting with Typhoeus asynchronous requests and in the end the code ended up becoming super messed up.

The nature of the original Manga Downloader is no different from a web crawler, it fetches HTML pages, parses them in order to find collections of links and keeps fetching until a set of images are downloaded. Then I organize them in volumes, optimize them to fit the Kindle screen resolution, and compile them down into PDF files. This makes this project an interesting exercise in trying to make concurrent HTTP requests and process the results.

A year later I was learning Elixir. The Manga Downloadr was a nice candidate for me to figure out how to implement the same thing in a different language. You can follow my learning process in this series of posts.

Finally, I've been learning more about Crystal, a Ruby-inspired platform that can compile Ruby-like source code into LLVM-optimized native binaries. And as a bonus it features a Go-like CSP channels and fibers to allow for concurrent code.

So I adapted my Elixir version to Crystal and the result is this code you can find over Github as akitaonrails/cr_manga_downloadr.

It works very well and performs really fast, limited mainly by how many requests MangaReader can respond concurently and the speed/reliability of the Internet connection. So, as my original Ruby version was terrible code, it was a good time to rewrite it. And as Crystal is surprisingly close to Ruby I decided to port it over.

The port was almost too trivial

It was mostly copying and pasting the Crystal code without the Type annotations. And I had to replace the lightweight Fibers and Channel implementation for concurrency over to traditional Ruby Threads.

The new version 2.0 for the Ruby version of the tool can be found in this Github repository: akitaonrails/manga-downloadr.

Ruby, Ruby everywhere!

Moving between Ruby and Crystal is not so difficult. The Crystal team did a fantastic job implementing a very solid Standard Library (stdlib) that very closely resembles MRI Ruby's.

For example, let's compare a snippet from my Crystal version first:

123456789101112131415161718

deffetch(page_link : String)  get page_link do |html|    images = html.xpath("//img[contains(@id, 'img')]").as(XML::NodeSet)    image_alt = images[0]["alt"]    image_src = images[0]["src"]if image_alt && image_src      extension      = image_src.split(".").last      list           = image_alt.split("").reverse      title_name     = list[4..-1].join("")      chapter_number = list[3].rjust(5, '0')      page_number    = list[0].rjust(5, '0')      uri = URI.parse(image_src)CrMangaDownloadr::Image.new(uri.host as String, uri.path as String, "#{title_name}-Chap-#{chapter_number}-Pg-#{page_number}.#{extension}")else      raise Exception.new("Couldn't find proper metadata alt in the image tag")endendend

Now let's check the ported Ruby version:

123456789101112131415161718192021

deffetch(page_link)  get page_link do |html|    images = html.css('#img')    image_alt = images[0]["alt"]    image_src = images[0]["src"]if image_alt && image_src      extension      = image_src.split(".").last      list           = image_alt.split("").reverse      title_name     = list[4..-1].join("")      chapter_number = list[3].rjust(5, '0')      page_number    = list[0].rjust(5, '0')      uri = URI.parse(image_src)Image.new(uri.host, uri.path, "#{title_name}-Chap-#{chapter_number}-Pg-#{page_number}.#{extension}")else      raise Exception.new("Couldn't find proper metadata alt in the image tag")endendend

It's uncanny how similar they are, down to stdlib calls such as URI.parse or Array methods such as split.

Once you remove the Type annotations from the Crystal version, it's 99% Ruby.

Ruby doesn't care if you're trying to call a method in a Nil object - at source code compiling time. Crystal does compile-time checkings and if it feels like the method call will be over Nil, it won't even compile. So this is one big win to avoid subtle bugs.

In Rails we are used to the dared #try method. Ruby 2.3 introduced the safe navigation operator.

So, in Ruby 2.3 with Rails, both of the following lines are valid:

12	obj.try(:something).try(:something2)obj&.something&.something2

In Crystal we can do the following:

1	obj.try(&.something).try(&.something2)

So, it's close. Use with care.

As I mentioned before, Crystal is close to Ruby but it's not meant to be compatible, so we can't just load Rubygems without porting. In this example I don't have Nokogiri to parse the HTML. But this is where the stdlib shines: Crystal comes with a good enough XML/HTML and JSON parsers. So we can parse HTML as XML and use plain XPath instead.

Instead of Ruby's Net::HTTP we have HTTP::Client (but their methods and semantics are surprisingly similar).

There are other differences, for example, this is the main file that requires all the others in Ruby:

1234567891011

...$LOAD_PATH.unshift File.join(File.dirname(__FILE__), "lib")require "manga-downloadr/records.rb"require "manga-downloadr/downloadr_client.rb"require "manga-downloadr/concurrency.rb"require "manga-downloadr/chapters.rb"require "manga-downloadr/pages.rb"require "manga-downloadr/page_image.rb"require "manga-downloadr/image_downloader.rb"require "manga-downloadr/workflow.rb"

And this is the Crystal version of the same manifest:

12	require "./cr_manga_downloadr/*"...

On the other hand, we need to be a bit more explicit in each Crystal source code file, and declare the specific dependencies where needed. For example, in the pages.cr file it starts like this:

require "./downloadr_client"require "xml"moduleCrMangaDownloadrclassPages< DownloadrClient(Array(String))  ...

Crystal has less room for "magic" but it's able to maintain a high level of abstraction anyway.

A Word about Types

We can spend the next decade masturbating over all there is about Types, and it will be extremelly boring.

The only thing you must understand: the compiler must know the method signature of classes before you can use them. There is no Runtime component that can introspect objects on the fly, like in Ruby and other dynamic languages (even Objective-C/Swift can do more dynamic stuff than Crystal).

Most of the time the Crystal compiler will be smart enough to infer the types for you, so you don't need to be absolutely explicit. You should follow the compiler's lead to know when to use Type Annotations.

What may really scare you at first is the need for Type Annotations, to understand Generics and so forth. The compiler will print out every scary error dumps that you will need to get used to. Most scary errors will usually be a Type Annotation missing or you trying to call a method over a possible Nil object.

For example, if I change the following line in the page_image_spec.cr test file:

# line 8:image = CrMangaDownloadr::PageImage.new("www.mangareader.net").fetch("/naruto/662/2")# line 10:# image.try(&.host).should eq("i8.mangareader.net")image.host.should eq("i8.mangareader.net")

The commented out line recognizes that the image instance might come as Nil, so we add an explicit #try call in the spec.

If we try to compile without this recognition, this is the error the compiler will dump on you:

1234567891011121314

$ crystal spec                                                        [Error in ./spec/cr_manga_downloadr/page_image_spec.cr:10: undefined method 'host' for Nil (compile-time type is CrMangaDownloadr::Image?)    image.host.should eq("i8.mangareader.net")          ^~~~=============================================================================Nil trace:  ./spec/cr_manga_downloadr/page_image_spec.cr:8        image = CrMangaDownloadr::PageImage.new("www.mangareader.net").fetch("/naruto/662/2")...

There is a large stacktrace dump after that snippet above, but you only need to pay attention to the first few lines that already says what's wrong: "undefined method 'host' for Nil (compile-time type is CrMangaDownloadr::Image?)". If you know how to read, you shouldn't have any problems most of the time.

Now, the Chapters, Pages, PageImage (all subclasses of DownloadrClient) are basically the same thing: they do HTTP::Client requests.

This is how the Pages class is implemented:

1234567891011

...moduleCrMangaDownloadrclassPages< DownloadrClient(Array(String))deffetch(chapter_link : String)      get chapter_link do |html|        nodes = html.xpath_nodes("//div[@id='selectpage']//select[@id='pageMenu']//option")        nodes.map { |node| [chapter_link, node.text as String].join("/") }endendendend

#get is a method from the DownloadrClient superclass that receives a String chapter_link and a block. The block receives a parsed html collection of nodes and we can play with it, return an Array of Strings.

That's why we have the (Array(String)) when inheriting from DownloadrClient. Let's see how the DownloadrClient superclass is implemented.

12345678910111213141516

moduleCrMangaDownloadrclassDownloadrClient(T)    ...defget(uri : String, &block : XML::Node -> T)      response = @http_client.get(uri)case response.status_codewhen301        get response.headers["Location"], &blockwhen200        parsed = XML.parse_html(response.body)        block.call(parsed)end      ...endendend

You can see that this class receives a Generic Type and it uses it as the return type for the yielded block in the #get method. The XML::Node -> T is the declaration of the signature for the block, sending XML::Node and receiving whatever the type T is. At compile time, imagine this T being replaced by Array(String). That's how you can create classes that can deal with any number of different Types without having to overloading for polymorphism.

If you come from Java, C#, Go or any other modern static typed language, you probably already know what a Generic is.

You can go very far with Generics, check out how our Concurrency.cr begins:

1234567891011

classConcurrency(A, B, C)  ...deffetch(collection : Array(A)?, &block : A, C? -> Array(B)?) : Array(B)?    results = [] of B    collection.try &.each_slice(@config.download_batch_size) do |batch|      engine  = if@turn_on_engineC.new(@config.domain)end      channel = Channel(Array(B)?).new      batch.each do |item|      ...

And this is how we use it in the workflow.cr:

private deffetch_pages(chapters : Array(String)?)  puts "Fetching pages from all chapters ..."  reactor = Concurrency(String, String, Pages).new(@config)  reactor.fetch(chapters) do |link, engine|    engine.try( &.fetch(link) )endend

In this example, imagine A being replaced by String, B also being replaced by String and C being replaced by Pages in the Concurrency class.

This is the "first-version-that-worked" so it's probably not very idiomatic. Either this could be solved with less Generics exercizing or maybe I could simplify it with the use of Macros. But it's working quite ok as it is.

The pure Ruby version ends up just like this:

12345678910

classConcurrencydefinitialize(engine_klass = nil, config = Config.new, turn_on_engine = true)    ...enddeffetch(collection, &block)    results = []    collection&.each_slice(@config.download_batch_size) do |batch|      mutex   = Mutex.new      threads = batch.map do |item|      ...

This version is much "simpler" in terms of source code density. But on the other hand we would have to test this Ruby version a lot more because it has many different permutations (we even inject classes through engine_klass) that we must make sure responds correctly. In practice we should add tests for all combinations of the initializer's arguments.

In the Crystal version, because all types have been checked at compile-time, it was more demanding in terms of annotations; but we can be pretty sure that if it compiles, it will run as expected.

I'm not saying that Crystal doesn't need any specs.

Compilers can only go so far. But how much is "so far"? Whenever you're "forced" to add Type Annotations, I will state that those parts are either trying to be too smart or they are intrinsically complex. Those are parts that would require extra levels of tests in Ruby and, if we can add the annotations properly, we can have less tests (we don't need to cover most permutations) in the Crystal version (exponencial complexity of permutations could go down to a linear complexity, I think).

A Word about Ruby Threads

The main concepts you must understand about concurrency in Ruby is this:

MRI Ruby has a GIL, a Global Interpreter Lock, that forbids code to run concurrently.
MRI Ruby does have access and exposes native Threads since version 1.9. But even if you fire up multiple Threads, they will run sequentially because only one thread han hold the Lock at a time.
I/O operations are the exception: they do release the Lock for other threads to run while the operation is waiting to complete. The OS will signal the program through OS level poll.

This means that if your app is I/O intensive (HTTP requests, File reads or writes, socket operations, etc), you will have some concurrency. A web server, such as Puma for example, can take some advantage of Threads because a a big part of the operations are involve receiving HTTP requests and sending HTTP responses over the wire, which would make the Ruby process idle while waiting.

If your app is CPU intensive (heavy algorithms, data crunching, stuff that really make the CPU hot) then you can't take advantage of native Threads, only one will run at a time. If you have multiple cores in your CPU, you can Fork your process to the number of cores available.

You should check out grosser/parallel to make it easy.

This is why Puma also has a "worker" mode. "Worker" is the name we usually give to forked children of processes.

In the case of this downloader process, it will perform thousands of HTTP requests to scrap the needed metadata from the MangaReader pages. So it's definitely much more I/O intensive than CPU intensive (the CPU intensive parts are the HTML parsing and later the image resizing and PDF compiling).

A sequential version of what has to be done, in Ruby, looks like this:

1234567891011

deffetch_sequential(collection, &block)  results = []  engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil  collection&.each_slice(@config.download_batch_size) do |batch|    batch.each do |item|      batch_results = block.call(item, engine)&.flatten      results += ( batch_results || [])endend  resultsend

If we have 10,000 links in the collection, we first slice it to what @config.download_batch_size and we iterate over those smaller slices, calling some block and accumulating the results. This is naive algorithm, as you will find out in the next section, but bear with me.

In Elixir you can fire up micro-processes to make the HTTP requests in parallel. In Crystal you can fire up Fibers and wait for the HTTP requests to complete and signal the results through Channels.

Both are lightweight ways and you can have hundreds or even thousands running in parallel. Manga Reader will probably complain if you try that many at once, so the limit is not in the code, but in the external service.

To transform the sequential version in a concurrent one, this is what we can do in Crystal:

12345678910111213141516171819202122232425

deffetch(collection : Array(A)?, &block : A, C? -> Array(B)?) : Array(B)?  results = [] of B  collection.try &.each_slice(@config.download_batch_size) do |batch|    channel = Channel(Array(B)?).new    batch.each do |item|      spawn {        engine  = if@turn_on_engineC.new(@config.domain)end        reply = block.call(item, engine)        channel.send(reply)        engine.try &.close      }end    batch.size.times do      reply = channel.receiveif reply        results.concat(reply.flatten)endend    channel.close    puts "Processed so far: #{results.try &.size}"end  resultsend

Fetching a huge collection and slicing it in smaller 'batches' is easy. Now we have a smaller batch collection. For each item (usually an URI) we spawn a Fiber and call a block that will request and process the results. Once it's finished processing it sends the results over a channel.

Once we finish iterating over the batch and spawning that many Fibers we can "wait" for them by doing channel.receive, which will start receiving results as soon as the Fibers finish requesting/processing each URI.

We accumulate the results and go over the next batch of the collection until finished. The amount of concurrency is determined by the size of the batch (it's like what I did with 'poolboy' over Elixir where we start a fixed number of processes to run in parallel and avoid doing a Denial of Service to Manga Reader).

By the way, this Crystal implementation is similar to what you would do if you were in Go, using Channels.

In the Ruby version you can fire up native Threads - which has a lot of overhead to spawn! - and assume the HTTP requests will run almost all in parallel. Because it's I/O intensive, you can have them all in parallel. This is what it looks like:

123456789101112131415161718

deffetch(collection, &block)  results = []  collection&.each_slice(@config.download_batch_size) do |batch|    mutex   = Mutex.new    threads = batch.map do |item|Thread.new {        engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nilThread.current["results"] = block.call(item, engine)&.flatten        mutex.synchronize do          results += ( Thread.current["results"] || [] )end      }end    threads.each(&:join)    puts "Processed so far: #{results&.size}"end  resultsend

Threads are all initialized in a "paused" state. Once we instantiate those many Threads, we can #join on each of them and await for them all to finish.

Once each Thread finishes the same URI request/process, the results must be accumulated in a global storage, in this case a simple array called results. But because we might have the chance of 2 or more threads trying to update the same array, we might as well synchronize the access (I'm not sure if Ruby's Array access is already synchronized, but I guess not). To synchronize access we use a Mutex, which is a fine-grained lock, to make sure only 1 thread can modify the global array at once.

To prove that Ruby can support concurrent I/O operations, I have added 2 methods to the Concurrent class, the first is just #fetch and it's the Thread implementation above. The second is called #fetch_sequential and it is the sequential version also shown at the beginning of this section. And I added the following spec:

123456789101112131415161718192021

it "should check that the fetch implementation runs in less time than the sequential version"do  reactor = MangaDownloadr::Concurrency.new(MangaDownloadr::Pages, config, true)  collection = ["/onepunch-man/96"] * 10WebMock.allow_net_connect!begin    concurrent_measurement = Benchmark.measure {      results = reactor.fetch(collection) { |link, engine| engine&.fetch(link) }    }    sequential_measurement = Benchmark.measure {      results = reactor.send(:fetch_sequential, collection) { |link, engine| engine&.fetch(link) }    }/\((.*?)\)$/.match(concurrent_measurement.to_s) do |cm|/\((.*?)\)/.match(sequential_measurement.to_s) do |sm|# expected for the concurrent version to be close to 10 times faster than sequential        expect(sm[1].to_f).to be > ( cm[1].to_f * 9 )endendensureWebMock.disable_net_connect!endend

Because it uses WebMock, I first disable it during this spec. I create a fake collection of 10 real links to MangaReader. And then I benchmark the Thread-based concurrent version and the plain sequential version. Because we have 10 links and they are all the same you can assume that the sequential version will be almost 10 times slower than the Thread-based version. And this is exactly what this spec compares and proves (the spec fails if the concurrent version is not at least 9x faster).

To compare all versions of the Manga Downloadrs I let download an compile an entire manga collection, in this case one "One-Man Punch", which has almost 1,900 pages/images. I am just measuring the fetching and scrapping processes and skipping the actual image downloading, image resizing and PDF generation as they take the majority of the time and the resizing and PDF part are all done by ImageMagick's mogrify and convert tools.

This is how long this new Ruby version takes to fetch and scrap almost 1,900 pages (using MRI Ruby 2.3.1):

1	12,42s user 1,33s system 23% cpu 57,675 total

This is how long the Crystal version takes:

1	4,03s user 0,40s system 7% cpu 59,207 total

Just for fun I tried to run the Ruby version under JRuby 9.1.1.0. To run with JRuby just add the following line in the Gemfile:

1	ruby "2.3.0", :engine => 'jruby', :engine_version => '9.1.1.0'

Bundle install, run normally, and this is the result:

1	47,80s user 1,99s system 108% cpu 45,967 total

And this is how long the Elixir version takes:

1	11,38s user 1,04s system 85% cpu 14,590 total

Reality Check!

If you just look at the times above you might get to the wrong conclusions.

First of all, it's an unfair comparison. The Elixir version uses a very different algorithm than the Ruby and Crystal versions.

In Elixir I boot up a pool of processes, around 50 of them. Then I start 50 HTTP requests at once. Once each process finishes, it releases itself back to the pool and I can fire up another HTTP request from the queue of links. So it's a constant stream of at most 50 HTTP requests, constantly.

The Crystal and Ruby/JRuby versions slice the 1,900 links into batches of 40 links and then I fire up 40 requests at once. This implementation waits all 40 to finish and they fire up another 40. So it's never a constant stream, it's bursts of 40 requests. So each batch is slowed down by the slowest request in the batch, not giving a chance for other requests to go through.

It's a difference in architecture. Elixir makes it much easier to do streams and Crystal, with it's CSP style, makes it easier to do bursts. A better approach would be to queue up the 1,900 links and use something like Sidekiq.cr to go over one link at a time (spawning 40 fibers to serve as a "pool", for example).

The Elixir version has a better efficient architecture, which is why it takes no more than 15 seconds to fetch all the image links while the Crystal version takes almost a full minute to finish (the accumulation of the slowest requests in each batch).

Now, you will be surprised that the Crystal version is actually a bit slower than the Ruby version! And you won't be too surprised to see JRuby being faster at 45 seconds!

This is another evidence that you should not dismiss Ruby (and that you should try JRuby more often). As I explained before, it does support concurrency in I/O operations and the applications I tested are all I/O heavy.

The difference is probably in the maturity of Ruby's Net::HTTP library against Crystal's HTTP::Client. I tried many tweaks in the Crystal version but I couldn't get much faster. I tried to do larger batches, but for some reason the applications just hangs, pauses, and never releases. I have to Ctrl-C out of it and retry until it finally goes through. If someone knows what I am doing wrong, please don't forget to write in the comments section below.

Some of this is probably due to MangaReader's unreliable servers, they probably have some kind of DoS preventions, throttling connections or something like this.

Anyway, when they go through, because the Ruby and Crystal algorithm are essentially the same, they take roughtly the same time to complete. So what's missing is for me to evolve this algorith to either use something like Sidekiq or implement an internal queue/pool of workers scheme.

Conclusion

The goal of this experiment was to learn more of Crystal's ability and how easy would it be to go back and forth with Ruby.

As you can see, there are many differences, but it's not so difficult. I may be missing something, but I stumbled upon some difficulties when pushing HTTP::Client + Fibers too hard, as I explained above. If you know what I am doing wrong, please let me know.

The difference between the Elixir vs Ruby/Crystal algorithms shows not just language performance differences, but also the importance of architecture and algorithms in the overall performance. This test was not conclusive, just a hint that my naive Fibers implementation needs some more work, and that Elixir's natural handling of parallel processes make it easier to achieve higher levels of parallelism.

This serves as a taste of what Crystal can do, and also that Ruby is still in the game. But also that Elixir surely is something to look close.

Yesterday I posted how I build a new implementation of the Manga Reader Downloader in Crystal, ported to Ruby, tested on JRuby and compared against Elixir. Just to recap, these were the results to fetch chapters, pages and image links of a sample manga:

MRI Ruby 2.3.1: 57 seconds
JRuby 9.1.1.0: 45 seconds
Crystal 0.17.0: 59 seconds
Elixir: 14 seconds

I also said how it was an unfair comparison as the Elixir version uses a different - and obviously more efficient - algorithm.

It was the "first-version-that-worked" so I decided to go ahead and improve the implementations. In the Ruby/JRuby version I added the thread gem to have a good enough implementation of a Thread pool that works for Ruby and JRuby. I probably should have used Concurrent-Ruby but I was having some trouble to make FixedThreadPool to work.

Anyway, now all versions will have a constant pool of requests running and we can make a better comparison.

Another thing that may have skewed the results against Crystal is that it seems to have a faulty DNS resolver implementation, so for now I just added the Manga Reader IP address directly to my /etc/hosts file to avoid getaddrinfo: Name or service not known exceptions.

To begin, let's test the same Elixir implementation, and as expected the result is still the same:

1	11,49s user 0,82s system 77% cpu 15,827 total

Now, the MRI Ruby with the "batch burst" algorithm was taking 57 seconds and this new implementation using a ThreadPool runs much better:

1	12,67s user 0,92s system 50% cpu 27,149 total

In Crystal, it's a bit more complicated as there is no way to implement the equivalent of a "Fiber Pool". What we have to do is do an infinite loop until the last process signals a loop break. Within the loop we create a maximum number of fibers, wait for each one to signal that it finished through an individual channel and loop again to create a new fiber, and so on. Compared to yesterday's 59 seconds, this is much better:

1	5,29s user 0,33s system 26% cpu 21,166 total

The JRuby version is not so fast though. Still better than yesterday's 45 seconds but now it's losing even to MRI:

1	49,24s user 1,41s system 146% cpu 34,602 total

I tried to use the --dev flag for faster start up time and it does improve a bit, getting it closer to MRI:

1	22,26s user 0,99s system 76% cpu 30,320 total

Not sure if it can be improved more though, any tips are welcome - don't forget to comment below.

So, Elixir is still at least twice as fast than Crystal at this point. But this also demonstrates how a different algorithm does make a huge difference where it matters. I can probably tweak it a bit more but this should suffice for now.

Changing the Ruby implementation to use ThreadPool

123456789101112131415161718192021222324

require "thread/pool"moduleMangaDownloadrclassConcurrency    ...deffetch(collection, &block)      pool    = Thread.pool(@config.download_batch_size)      mutex   = Mutex.new      results = []      collection.each do |item|        pool.process {          engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil          reply = block.call(item, engine)&.flatten          mutex.synchronize do            results += ( reply || [] )end        }end      pool.shutdown      resultsend    ...

The idea is that we will have a fixed number of spawn native threads (in this case, determined by @config.download_batch_size). As one thread finishes it will pop a new link from the collection Array, essentially working as a "queue" to be depleted.

The results are accumulated in the results Array. Because many threads may want to modify it at once we have to synchronize access through a Mutex.

This way we always have a fixed amount of workers performing requests constantly instead of slicing the collection Array and doing bursts as in the previous version.

Changing the Crystal implementation to simulate a Fibers Pool

The Crystal version became a bit more complicated as I didn't find a pool library to pull. I found a rough implementation of a pool in this stackoverflow post and I adapted it here:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162

moduleCrMangaDownloadrclassConcurrencyBreakException< Exception; endclassConcurrency(A, B, C)definitialize(@config : Config, @turn_on_engine = true); enddeffetch(collection : Array(A)?, &block : A, C? -> Array(B)?) : Array(B)?      results = [] of B      pool @config.download_batch_size doif (collection.try(&.size) || 0) > 0          item = collection.try(&.pop)          engine = @turn_on_engine ? C.new(@config.domain) : nil          reply = block.call(item as A, engine)          results.concat(reply.try(&.flatten) as Array(B))else          raise ConcurrencyBreakException.newendend      resultsenddefworker(&block)      signal_channel = Channel::Unbuffered(Exception).new      spawn dobegin          block.callrescue ex          signal_channel.send(ex)else          signal_channel.send(Exception.new(nil))endend      signal_channel.receive_openddefpool(max_num_of_workers = 10, &block)      pool_counter = 0      workers_channels = [] of Channel::ReceiveOp(Channel::Unbuffered(Exception), Exception)      loop dowhile pool_counter < max_num_of_workers          pool_counter += 1          workers_channels << worker(&block)end        index, signal_exception = Channel.select(workers_channels)        workers_channels.delete_at(index)        pool_counter -= 1if signal_exception.is_a?(ConcurrencyBreakException)breakelsif signal_exception.message.nil?# does nothing, just signalling to continueelse          puts "ERROR: #{signal_exception.message}"endendendendend

This can be daunting at first but it's not so difficult to understand. The pool method will first spawn as many fibers as determined by the pool size:

1234	while pool_counter < max_num_of_workers pool_counter += 1 workers_channels << worker(&block)end

This snippet returns a number of channels that each worker method creates and returns. The worker method also spawns a new Fiber to do the actual work and once it finishes it signals into the channel it created.

Then, it waits for one of the created channels to receive something here:

1	index, signal_exception = Channel.select(workers_channels)

Then it checks if one of the Fibers sent a ConcurrencyBreakException to signal the loop to break, otherwise it will loop back to the previous while, which will spawn more worker calls and create new channels and new Fibers to do more work.

The fetch method is similar to the Ruby version with the ThreadPool, so in essence those worker and pool methods are simulating a "FiberPool" of sorts.

But again, this is very I/O intensive and both the Ruby and Crystal versions take advantage of the fact that they can do more work while waiting for one HTTP request to finish.

Conclusion

Even though the algorithms are roughly similar, Elixir is still winning by a very large margin.

Either it's evidence that the decades old and battle tested BEAM (Erlang's VM) is far more efficient to make concurrent and parallel processing or it's evidence that I still missed something in the algorithms :-)

The Ruby and Crystal versions are very easier to understand, the Elixir version is still a little rough in the edges - I need to improve my Elixir skills.

So, the summary with the new results is this:

MRI Ruby 2.3.1: 27 seconds
JRuby 9.1.1.0: 30 seconds
Crystal 0.17.0: 21 seconds
Elixir: 15 seconds

And these are the links for each implementation on Github:

Let me know if you have ideas to make them even faster!

In my eyes, Crystal may become the ideal solution to make our beloved Ruby gems faster. Up until now we have been using C-based extensions to accelerate CPU-bound code in Ruby. Nokogiri, for example, is a wrapper to provide a nice API on top of libxml, which is a huge library in C.

But there are many opportunities to accelerate Rails applications as well. For example, we just saw the release of the gem "faster_path", this time written in Rust and bridged through FFI (Foreign Function Interface). The author's claim is that Sprockets has to compute lots of paths and making this library, natively compiled and optimized with Rust, added a huge improvement in the asset pipeline task.

Sam Saffrom, from Discourse, also built a very small gem called "fast_blank" which is a tiny library written in C that reimplements ActiveSupport's String#blank? method to be up to 9x faster. Because Rails digests volumes of strings, checking if they are blank everytime, this adds some performance (depends on your app, of course).

The Holy Grail to native-level performance is to be able to write close-to-Ruby code instead of having to hack low-level C or having the high learning curve of a language such as Rust. More than that, I'd like to avoid having to use FFI. I am not an expert in FFI but I remember understanding that it adds overhead to make the bindings.

By the way, it's important to disclose right now: I am not a C expert by any means of the imagination, far from that. So I have very little experience dealing with hard core C development. Which is again, why this possibility of writing in Crystal is even more appealing to me. So if you are a C expert and you spot something silly I am saying about it, please let me know in the comments section below.

My exercise is to rewrite the C-based Fast Blank gem in Crystal, add it to the same Gem to compile under Crystal if it's available or fallback to C, and make the specs pass so it's a seamless transition for the user.

To achieve that I had to:

Extend the Gem's extconf.rb to generate different Makefiles (for C and Crystal) that are able to compile under OS X or Linux (Ubuntu at least) - OK
Make the specs pass in the Crystal version - Almost (it's ok for all intents and purposes but an edge case)
Make the performance be faster than Ruby and close to C - Not so much yet (under OS X the performance is quite good, but under Ubuntu it doesn't scale so well for large string)

You can check out the results so far on my fork over Github and follow the Pull Request discussion as well.

Comparing C and Crystal

Just to have us started, let's check out a snippet of Sam's original C version:

1234567891011121314151617181920

static VALUErb_str_blank(VALUE str){  rb_encoding *enc;char *s, *e;  enc = STR_ENC_GET(str);  s = RSTRING_PTR(str);if (!s || RSTRING_LEN(str) == 0) return Qtrue;  e = RSTRING_END(str);while (s < e) {int n;unsignedint cc = rb_enc_codepoint_len(s, e, &n, enc);if (!rb_isspace(cc) && cc != 0) return Qfalse;    s += n;  }return Qtrue;}

Yeah, quite scary, I know. Now let's see the Crystal version:

12345678910111213141516

struct Char  ...# same way C Ruby implements itdefis_blankself == '' || ('\t'<= self<= '\r')endendclassString  ...defblank?returntrueifself.nil? || self.size == 0    each_char { |char| returnfalseif !char.is_blank }returntrueendend

Hell yeah! If you're a rubyist I bet you can understand a 100% of the snippet above. It's not "exactly" the same thing (as the specs are not fully passing yet), but it's damn close.

The Quest for a Makefile to Crystal

I've researched many experimental Github repos and Gists out there. But I didn't find one that has it all so I decided to tweak what I found until I got to this version:

Obs: again, I am not a C expert. If you have experience with Makefiles I know this one can be refactored to something nicer than this. Let me know in the comments below.

123456789101112131415161718192021222324252627282930313233343536373839404142434445

ifeq "$(PLATFORM)"""PLATFORM := $(shell uname)endififeq "$(PLATFORM)""Linux"UNAME = "$(shell llvm-config --host-target)"CRYSTAL_BIN = $(shell readlink -f `which crystal`)LIBRARY_PATH = $(shell dirname $(CRYSTAL_BIN))/../embedded/libLIBCRYSTAL = $(shell dirname $(CRYSTAL_BIN) )/../src/ext/libcrystal.aLIBRUBY = $(shell ruby -e "puts RbConfig::CONFIG['libdir']")LIBS = -lpcre -lgc -lpthread -levent -lrt -ldlLDFLAGS = -rdynamicinstall: allall: fast_blank.sofast_blank.so: fast_blank.o  $(CC) -shared $^ -o $@ $(LIBCRYSTAL) $(LDFLAGS) $(LIBS) -L$(LIBRARY_PATH) -L$(LIBRUBY)fast_blank.o: ../../../../ext/src/fast_blank.cr  crystal build --cross-compile --release --target $(UNAME) $<.PHONY: cleanclean:  rm -f bc_flags  rm -f *.o  rm -f *.soendififeq "$(PLATFORM)""Darwin"CRYSTAL_FLAGS = -dynamic -bundle -Wl,-undefined,dynamic_lookupinstall: allall: fast_blank.bundlefast_blank.bundle: ../../../../ext/src/fast_blank.cr  crystal $^ --release --link-flags "$(CRYSTAL_FLAGS)" -o $@clean:  rm -f *.log  rm -f *.o  rm -f *.bundleendif

Most people using Crystal are on OS X, including the creators of Crystal. LLVM is under Apple's umbrella and their entire ecosystem relies heavily on LLVM. They spent many years migrating the C front-end first, then the C back-end away from GNU's standard GCC to Clang. And they were able to make their both Objective-C and Swift compile down to LLVM's IR and that's how both can interact back and forth natively.

Then, they improved the ARM backend support and that's how they can have an entire iOS "Simulator" (not a dog slow emulator like Android) where the iOS apps are natively compiled to run over Intel's x86_64 processor while in development and then quickly recompile to ARM when ready to package to the App Store.

This way you can run natively, test quickly, without the slowness of an emulated environment. By the way, I will say this once: Google's biggest mistake is not supporting LLVM as they should and reinventing the wheel. If they had, Go could already be used to implement for Android and Chromebooks as well as x86 based servers and they could put away all the Java/Oracle debacle.

But I digress.

In OS X you can pass a "-bundle" link-flag to crystal and it will probably use clang underneath to generate the binary bundle.

On Ubuntu crystal just compiles down to an object file (.o) and you have to manually invoke GCC with the "-shared" option to create a shared-object. To do that we have to use the "--cross-compile" and pass an LLVM target triplet so it generates the .o (this requires the llvm-config tool).

Shared Libraries (.so) and Loadable Modules (.bundle) are different beasts, check this documentation out for more details.

Keep in mind that benchmarking binaries built with different compilers can make a difference. I am not an expert but out of pure anecdote I believe Ruby under RVM on OS X is compiled using OS X's default Clang. On Ubuntu it's compiled under GCC. This seems to make Ruby on OS X "so slightly" inneficient in synthetic benchmarks.

On the other hand, Crystal binaries linked with GCC feels "so slightly" inneficient on Ubuntu, while Ruby on Ubuntu feels a bit faster, having been compiled and linked with GCC.

So when we compare Fast Blank/OS X/bit faster with Ruby/OS X/slower against Fast Blank/Ubuntu/bit slower with Ruby/Ubuntu/bit faster, it seems to give a wider advantage to the OS X benchmark comparison against the Ubuntu benchmark, even though individual computation times are not so far from each other.

I will come back to this point in the benchmarks section.

Finally, everytime you have a rubygem with a native extension, you will find this bit in their gemspec files:

Gem::Specification.new do |s|  s.name = 'fast_blank'  ...  s.extensions = ['ext/fast_blank/extconf.rb']  ...

When the gem is installed through gem install or bundle install it will run this script to generate a proper Makefile. In a pure C extension it will use the built-in "mkmf" library to generate it.

In our case, if we have Crystal installed, we want to use the Crystal version, so I tweaked the extconf.rb to be like this:

123456789101112131415

require 'mkmf'ifENV['VERSION'] != "C"&& find_executable('crystal') && find_executable('llvm-config')# Very dirty patchingdefcreate_makefile(target, srcprefix = nil)    mfile = open("Makefile", "wb")    cr_makefile = File.join(File.dirname(__FILE__), "../src/Makefile")    mfile.print File.read(cr_makefile)ensure    mfile.close if mfile    puts "Crystal version of the Makefile copied"endendcreate_makefile 'fast_blank'

So, if it finds crystal and llvm-config (which in OS X you have to add the proper path like this: export PATH=$(brew --prefix llvm)/bin:$PATH).

The Rakefile in this project declares the standard :compile task as the first one to run, and it will execute the extconf.rb, which will generate the proper Makefile and run the make command to compile and link the proper library in the proper lib/ path.

So we will end up with lib/fast_blank.bundle on OS X and lib/fast_blank.so on Ubuntu. From there we can just have require "fast_blank" from any Ruby file in the gem and it will have access to the publicly exported C function mappings from the Crystal library.

Mapping C-Ruby to Crystal

Now, any direct C extension - without FFI, fiddle or other "bridges" - will ALWAYS have a much better advantage.

The reason is that you literally have to "copy" data from C-Ruby to Crystal/Rust/Go or whatever other language you're binding. While with a C-based extension you can operate directly in the memory space with the data without having to move it or copy it away.

For example. First, you have to bind the C functions from C-Ruby to Crystal. And we accomplish that with Paul Hoffer's Crystalized Ruby mappings. It's an experimental repository that I helped a bit to clean up in order for him to later extract this mapping library into its own Shard (shards are the same as gems for Crystal). For now I had to simply copy the file over to my Fast Blank.

Some of the relevant bits are like this:

12345678910111213141516171819202122232425262728293031323334353637383940414243

lib LibRuby  type VALUE = Void*  type METHOD_FUNC = VALUE -> VALUE  type ID = Void*  ...# strings  fun rb_str_to_str(value : VALUE) : VALUE  fun rb_string_value_cstr(value_ptr : VALUE*) : UInt8*  fun rb_str_new_cstr(str : UInt8*) : VALUE  fun rb_utf8_encoding() : VALUE  fun rb_enc_str_new_cstr(str : UInt8*, enc : VALUE) : VALUE  ...# exception handling  fun rb_rescue(func : VALUE -> UInt8*, args : VALUE, callback: VALUE -> UInt8*, value: VALUE) : UInt8*end...classStringRUBY_UTF = LibRuby.rb_utf8_encodingdefto_rubyLibRuby.rb_enc_str_new_cstr(self, RUBY_UTF)enddefself.from_ruby(str : LibRuby::VALUE)    c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)# FIXME there is still an unhandled problem: then we receive \u0000 from Ruby it raises "string contains null bytes"# so we catch it with rb_rescue, but then we can't generate a Pointer(UInt8) that represents the unicode 0, instead we return a plain blank string# but then the specs fail    new(c_str)ensure""enddefself.cr_str_from_rb_cstr(str : LibRuby::VALUE)    rb_str = LibRuby.rb_str_to_str(str)    c_str  = LibRuby.rb_string_value_cstr(pointerof(rb_str))enddefself.return_empty_string(arg : LibRuby::VALUE)    a = 0_u8    pointerof(a)endend

Then I can use these mappings and helpers to build a "Wrapper" class in Crystal:

12345678910111213141516171819202122232425

require "./lib_ruby"require "./string_extension"moduleStringExtensionWrapperdefself.blank?(self : LibRuby::VALUE)returntrue.to_ruby ifLibRuby.rb_str_length(self) == 0    str = String.from_ruby(self)    str.blank?.to_rubyrescuetrue.to_rubyenddefself.blank_as?(self : LibRuby::VALUE)returntrue.to_ruby ifLibRuby.rb_str_length(self) == 0    str = String.from_ruby(self)    str.blank_as?.to_rubyrescuetrue.to_rubyenddefself.crystal_value(self : LibRuby::VALUE)    str = String.from_ruby(self)    str.to_rubyendend

And this "Wrapper" depends on the "pure" Crystal library itself like with the snippets for the Char struct and String class extensions I showed in the first section of the article above.

Finally, I have a main "fast_blank.cr" file that externs those Wrapper functions so C-Ruby can see them as plain String methods:

1234567891011

require "./string_extension_wrapper.cr"fun init = Init_fast_blankGC.initLibCrystalMain.__crystal_main(0, Pointer(Pointer(UInt8)).null)  string = LibRuby.rb_define_class("String", LibRuby.rb_cObject)LibRuby.rb_define_method(string, "blank?", ->StringExtensionWrapper.blank?, 0)LibRuby.rb_define_method(string, "blank_as?", ->StringExtensionWrapper.blank_as?, 0)  ...end

This is mostly boilerplate. But now check out what I am having to do in the wrapper, in this particular snippet:

defself.blank?(self : LibRuby::VALUE)returntrue.to_ruby ifLibRuby.rb_str_length(self) == 0  str = String.from_ruby(self)  str.blank?.to_rubyrescuetrue.to_rubyend

I am receiving a C-Ruby String casted as a pointer (VALUE) then I go through the lib_ruby.cr mappings to get the C-Ruby string data and copy it over into a new instance of Crystal's internal String representation. So at any given time I have 2 copies of the same string, one in the C-Ruby memory space and another in the Crystal memory space.

This happens with all FFI-like extensions but it doesn't happen to the pure C implementation. In Sam Saffrom's C implementation it directly works with the same address in C-Ruby's memory space:

123456789

static VALUErb_str_blank(VALUE str){  rb_encoding *enc;char *s, *e;  enc = STR_ENC_GET(str);  s = RSTRING_PTR(str);  ...

It receives a pointer (direct memory address) and goes. And this is huge advantage for the C version. When you have a big volume of medium to large sized strings being copied over from C-Ruby to Crystal, it adds a noticeable overhead that can't be removed.

String mapping Caveat

I still have a problem though. There is one edge case I was not able to overcome yet (help is most welcome). When C-Ruby passes a unicode "\u0000" I am unable to create the same character in Crystal and I end up passing just an empty string ("") which is not the same thing.

The way to deal with it is to receive a Ruby String (VALUE) and get the C-String from it this way:

12	rb_str = LibRuby.rb_str_to_str(str)c_str = LibRuby.rb_string_value_cstr(pointerof(rb_str))

If the "str" is the "\u0000" (under Ruby 2.2.5 at least) C-Ruby raises a "string contains null bytes" exception. Which is why I rescue from this exception like this:

1	c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)

When an exception is triggered I have to pass the pointer to another function to rescue from it:

1234	defself.return_empty_string(arg : LibRuby::VALUE) a = 0_u8 pointerof(a)end

But this is not correct, I am just passing the pointer to a "0" character, which is "empty". Therefore, specs are not passing correctly:

123456789101112131415161718

Failures:  1) String provides a parity with active support function     Failure/Error: expect("#{i.to_s(16)} #{c.blank_as?}").to eq("#{i.to_s(16)} #{c.blank2?}")       expected: "0 false"            got: "0 true"       (compared using ==)     # ./spec/fast_blank_spec.rb:22:in `block (3 levels) in <top (required)>'     # ./spec/fast_blank_spec.rb:19:in `times'     # ./spec/fast_blank_spec.rb:19:in `block (2 levels) in <top (required)>'  2) String treats  correctly     Failure/Error: expect("\u0000".blank_as?).to be_falsey       expected: falsey value            got: true     # ./spec/fast_blank_spec.rb:47:in `block (2 levels) in <top (required)>'

Ary gave a simple tip later, I will add it to the conclusion below.

The Synthetic Benchmarks (careful on how you interpret them!)

The original Rails ActiveSupport implementation of String#blank? looks like this:

1234567891011121314151617181920

classString# 0x3000: fullwidth whitespaceNON_WHITESPACE_REGEXP = %r![^\s#{[0x3000].pack("U")}]!# A string is blank if it's empty or contains whitespaces only:##   "".blank?                 # => true#   "   ".blank?              # => true#   "　".blank?               # => true#   " something here ".blank? # => false#defblank?# 1.8 does not takes [:space:] properlyif encoding_aware?self !~ /[^[:space:]]/elseself !~ NON_WHITESPACE_REGEXPendendend

It's mainly a regular expression comparison, which can be a bit slow. Sam's version is a more straight forward loop through the string to compare each character with what's considered "blank". There are many unicode codepoints that are considered blank, some are not, which is why the C and Crystal versions are similar, but they are different from Rails' version.

In the Fast Blank gem there is a benchmark Ruby script to compare the C-extension against Rails' Regex based implementation.

The Regex implementation is called "Slow Blank". It's particularly slow if you pass a real empty String, so in the benchmark Sam added a "New Slow Blank" that checks through String#empty? first, and this version is faster in this edge case.

The fast C version is called "Fast Blank" but although you can consider ir "correct" it's not compatible with all the edge cases from Rails. So he implemented a String#blank_as? which is compatible with Rails. Sam calls it "Fast Activesupport".

In my Crystal version I did the same, having both String#blank? and String#blank_as?.

So, without further ado, here is the C Version over OS X benchmark for empty strings, and we exercise each function many times within a few seconds to have more accurate results (check out Evan Phoenix's "benchmark/ips" to understand the "iteration per second" methodology).

1234567891011121314151617

================== Test String Length: 0 ==================Warming up _______________________________________          Fast Blank   191.708k i/100ms  Fast ActiveSupport   209.628k i/100ms          Slow Blank    61.487k i/100ms      New Slow Blank   203.165k i/100msCalculating _______________________________________          Fast Blank     20.479M (± 9.3%) i/s -    101.414M in   5.001177s  Fast ActiveSupport     21.883M (± 9.4%) i/s -    108.378M in   5.004350s          Slow Blank      1.060M (± 4.7%) i/s -      5.288M in   5.001365s      New Slow Blank     18.883M (± 6.9%) i/s -     94.065M in   5.008899sComparison:  Fast ActiveSupport: 21882711.5 i/s          Fast Blank: 20478961.5 i/s - same-ish: difference falls within error      New Slow Blank: 18883442.2 i/s - same-ish: difference falls within error          Slow Blank:  1059692.6 i/s - 20.65x slower

It's super fast. Rails' version is 20x slower on my machine.

Now, Crystal version over OS X

1234567891011121314151617

================== Test String Length: 0 ==================Warming up _______________________________________          Fast Blank   174.349k i/100ms  Fast ActiveSupport   174.035k i/100ms          Slow Blank    64.684k i/100ms      New Slow Blank   215.164k i/100msCalculating _______________________________________          Fast Blank      8.647M (± 1.6%) i/s -     43.239M in   5.001530s  Fast ActiveSupport      8.580M (± 1.3%) i/s -     42.987M in   5.010759s          Slow Blank      1.047M (± 3.7%) i/s -      5.239M in   5.008907s      New Slow Blank     19.090M (± 9.3%) i/s -     94.672M in   5.009057sComparison:      New Slow Blank: 19090034.8 i/s          Fast Blank:  8647459.7 i/s - 2.21x slower  Fast ActiveSupport:  8580487.9 i/s - 2.22x slower          Slow Blank:  1047465.3 i/s - 18.22x slower

As I explained before, even checking empty strings, the Crystal version is slower than the Ruby check for String#empty? (New Slow Blank) because I have the string copying routine of the Wrapper mappings. This adds overhead that is perceptible over many iterations. It's still 18x faster than Rails, but it loses to C-Ruby.

Finally, Crystal version over Ubuntu

1234567891011121314151617

================== Test String Length: 0 ==================Warming up _______________________________________          Fast Blank   255.883k i/100ms  Fast ActiveSupport   260.915k i/100ms          Slow Blank   105.424k i/100ms      New Slow Blank   284.670k i/100msCalculating _______________________________________          Fast Blank      8.895M (± 9.8%) i/s -     44.268M in   5.037761s  Fast ActiveSupport      8.647M (± 8.2%) i/s -     43.051M in   5.020125s          Slow Blank      1.736M (± 3.9%) i/s -      8.750M in   5.048253s      New Slow Blank     22.170M (± 6.2%) i/s -    110.452M in   5.004909sComparison:      New Slow Blank: 22170031.0 i/s          Fast Blank:  8895113.3 i/s - 2.49x slower  Fast ActiveSupport:  8646940.8 i/s - 2.56x slower          Slow Blank:  1736071.0 i/s - 12.77x slower

Notice that it's around the same ballpark, but the Rails version on Ubuntu runs almost twice as fast compared to its counterpart in OS X, which makes the comparison against the Crystal library go down from 18x to 12x.

The benchmark keeps comparing agains strings of larger and larger sizes, from 6, to 14, to 24, up to 136 characters in length.

Let's get just the last test case of 136 characters. First with C version on OS X:

1234567891011121314151617

================== Test String Length: 136 ==================Warming up _______________________________________          Fast Blank   177.521k i/100ms  Fast ActiveSupport   193.559k i/100ms          Slow Blank    89.378k i/100ms      New Slow Blank    60.639k i/100msCalculating _______________________________________          Fast Blank     10.727M (± 8.7%) i/s -     53.256M in   5.006538s  Fast ActiveSupport     11.600M (± 8.3%) i/s -     57.681M in   5.009692s          Slow Blank      1.872M (± 5.7%) i/s -      9.385M in   5.029243s      New Slow Blank      1.017M (± 5.3%) i/s -      5.094M in   5.022994sComparison:  Fast ActiveSupport: 11600112.2 i/s          Fast Blank: 10726792.8 i/s - same-ish: difference falls within error          Slow Blank:  1872262.5 i/s - 6.20x slower      New Slow Blank:  1016926.7 i/s - 11.41x slower

The C-version is consistently much faster in all test cases and in the 136 characters it's still 11x faster than Rails in pure Ruby.

Now the Crystal version over OS X:

1234567891011121314151617

================== Test String Length: 136 ==================Warming up _______________________________________          Fast Blank   127.749k i/100ms  Fast ActiveSupport   126.538k i/100ms          Slow Blank    94.390k i/100ms      New Slow Blank    60.594k i/100msCalculating _______________________________________          Fast Blank      3.283M (± 1.8%) i/s -     16.480M in   5.021364s  Fast ActiveSupport      3.235M (± 1.3%) i/s -     16.197M in   5.008315s          Slow Blank      1.888M (± 4.4%) i/s -      9.439M in   5.009458s      New Slow Blank    967.950k (± 4.7%) i/s -      4.848M in   5.018946sComparison:          Fast Blank:  3283025.1 i/s  Fast ActiveSupport:  3234586.5 i/s - same-ish: difference falls within error          Slow Blank:  1887800.5 i/s - 1.74x slower      New Slow Blank:   967950.2 i/s - 3.39x slower

It's also faster, but just by 2 to 3 times compared to pure Ruby, a far cry from 11x. But my hypothesis is that the mapping and copying of so many string over adds a large overhead that the C version does not have.

And the Crystal version over OS X:

1234567891011121314151617

================== Test String Length: 136 ==================Warming up _______________________________________          Fast Blank   186.810k i/100ms  Fast ActiveSupport   187.306k i/100ms          Slow Blank   143.439k i/100ms      New Slow Blank    98.308k i/100msCalculating _______________________________________          Fast Blank      3.517M (± 3.9%) i/s -     17.560M in   5.000791s  Fast ActiveSupport      3.485M (± 3.8%) i/s -     17.419M in   5.006427s          Slow Blank      2.755M (± 4.2%) i/s -     13.770M in   5.008490s      New Slow Blank      1.551M (± 4.3%) i/s -      7.766M in   5.017853sComparison:          Fast Blank:  3516960.7 i/s  Fast ActiveSupport:  3484575.5 i/s - same-ish: difference falls within error          Slow Blank:  2754669.0 i/s - 1.28x slower      New Slow Blank:  1550815.2 i/s - 2.27x slower

Again, the Ubuntu versions of both Crystal library but also the Ruby binary runs faster and the comparison shows no more than twice as much faster. And the pure Ruby's String#empty? is in the same ballpark as Crystal's version.

Conclusion

The most obvious conclusion is that I probably did a mistake in choosing Fast Blank as my first proof of concept. The algorithm is too trivial and a simple check for String#empty? in pure Ruby is orders of magnitude faster than the added overhead of mapping and string copying to Crystal.

Also, any use case where you have a huge amount of small bits of data being transferred from C-Ruby to Crystal or any FFI-based extension will have the overhead of data copying, which a pure C-version will not have. Which is why Fast Blank is better done in C.

Any other use case where you have less amounts of data, or data that can be transferred in bulk (less calls from C-Ruby to the extension, with arguments with a larger size, and with more costly processing) are better candidates to benefit from extensions.

Again, not everything gets automatically faster, we always have to figure out the use case scenarios first. But because it's so much easier to write in Crystal and benchmark, we can make faster proofs of concepts and scrap the idea if the measurements prove that we won't benefit as much.

The Crystal documentation recently received a "Performance Guide". It's very useful for you to avoid common pitfalls that harms overall performance. Even though LLVM is quite competent in heavy optimization, it can't do everything. So read it through to improve your general Crystal skills.

That being said, I still believe that this exercise was well worth it. I will probabaly do some more. I'd really want to thank Ary (Crystal creator) and Paul Hoffer for the patience in helping me out through many of quircks I found along the way.

While I was finishing this post, Ary pointed out that I could probably ditch Strings altogether and work directly with an array of bytes, which is a good idea and I will probably try that. I think I made it clear by now that the whole String copying adds a very perceptible overhead as we saw in the benchmarks above. Let me know if someone is interested in contributing as well. With a few more tweaks I believe we can have a Crystal version that can at least compete against the C version while also being more readable and maintainable for most Rubyists, which is my goal.

I hope the codes I published here will serve as boilerplate examples for more Crystal-based Ruby extensions in the future!

07/23/16: coincidentally I posted this review a day before of the final release :-) So the final build is 14393 and it's available for free right now!

It's been 3 months since I posted my initial impressions on Windows 10 Anniversary Edition most important feature: Bash on Windows. My conclusion at the time was that it was not ready for prime time yet.

My conclusion as of right now, on July 26th is that it's finally usable enough for web developers, particularly for Ruby developers who always suffered through lack of Windows support.

It's alive!

Installation process is the same. You must be signed up to the Windows Insider program, wait at least one day if this is your first time. Enable the Fast ring of updates, install the Preview edition from normal Windows Update and then turn on the "Windows Subsystem for Linux (Beta)" feature.

Last time I was testing Windows 10 over Virtualbox over Ubuntu 14.04 on a Dell Inspiron notebook (8 cores i7 with 16GB of RAM). It was super slow, I don't recommend at all. Now I am back on my trustworthy Macbook Pro 2013 with 16GB of RAM and SSD running VMWare Fusion, and Windows 10 flies here. Super recommended.

Once you do all that, you can start the "Bash on Ubuntu on Windows" (a mouth full). The first good surprise is that it prompts you to register a new username instead of just falling back to Root. As I said in my previous post, it's good practice to create a new user and add it to the sudoers group, and this is what it does. So you can install packages using "sudo apt-get install".

You should follow my previous post for all the packages and configurations I normally do in a development Linux box.

RVM now works! I was able to install Ruby 2.3.1 through RVM without any problems whatsoever.

I was able to git clone a small Rails project and properly bundle install. All gems were downloaded, native extensions compiled without any flaws.

Somethings still don't work. For example, you won't be able to finish the Postgresql 9.3 installation. It will download and install but the cluster setup fails as you can follow through this issue thread.

But you don't need to have everything installed under Ubuntu, you can just fallback to the native Postgresql for Windows and edit your config/database.yml to point to server 127.0.0.1 and port 5432. On the Ubuntu side you just need to install libpq-dev so the pg gem can compile its native extensions and that's it.

Smaller services such as Memcached or Redis install properly with apt-get but they won't auto-start through Upstart. But you can start them up manually and use something as Foreman to control the processes. They both start and work good enough, so you can test caching in your Rails projects and also test Sidekiq workers.

I know that this is still a Preview, so there are bug fixes and possibly some new features that might be included in the final release in August. One nitpick I have is that every command I run with sudo takes a few seconds to start, so it's annoying, but it works in the end.

Node.js 6.3.1 successfully installs and runs. I was able to npm i and node server.js on Openshift's Node example repository.

Crystal 0.18.7 successfully installs and it was able to properly compile my Manga Downloader project. It executed my built-in performance test in 15 minutes. Not lightning fast performance but it runs correctly until the end. (And so, yes, Crystal runs on Windows as well now!).

Go 1.6 works. I just did a go get to install Martini and just ran the simple "Hello World" server. Compiles, starts and executes very fast as expected.

Unfortunatelly, Elixir 1.3.1 crashes, I don't know why yet.

123	$ iexCrash dump is being written to: erl_crash.dump...doneerl_child_setup closed

Actually, Erlang itself crashes by just trying to run erl. None of the Elixir tools work as a result. No iex, no mix. The funny thing is that it was working in the initial Preview. So either it's something in the new Preview or something in the newest Erlang releases. There are open issues regarding this problem, so let's hope it gets fixed soon.

"The Best Environment for Rails on Windows"

I have this very old 2009 post to guide developers that are locked on Windows to implement Rails projects. The first advice is to avoid Ruby for Windows. I really commend the efforts of great developers such as Luis Lavena, who invested a lot of time to make it work well enough. But unfortunately the reality is that Ruby is made for Linux environments, it binds to native extensions in C that has lots of dependencies that are not easy to make available on Windows.

So the best option up until now was to install Vagrant (through Virtualbox or, even better, VMWare) as the runtime and use Windows-available editors such as Sublime Text 3 or Atom.

Now you can avoid the Vagrant/virtualization part and directly use "Bash on Ubuntu on Windows". It's so complete that I can even use ZSH and install complex dotfiles such as YADR. You can run Postgresql for Windows and connect to localhost:5432 in your Rails application or any web application that requires Postgresql for example.

You can install ConEmu as a better terminal emulator than the stupidly bad cmd.exe default console. Follow this article to get news on its support of Bash on Windows. Right now you have to edit the "Ctrl-V" hotkey to something else ("Ctrl-Shift-V"), arrow keys don't work well om Vim, and you can add a Default Task for Bash like this: %windir%\system32\bash.exe ~ -cur_console:p

Developing Rails on Windows

Conclusion

So, yes, at least from this version I tested (installed on July 25th) this is a usable "Bash on Ubuntu on Windows" and web developers from Ruby to Javascript to Crystal to Go can really start testing and trying to make Windows 10 Anniversary their primary platform for serious Linux-based development.

Add Sublime Text or Atom (or Visual Studio, if that's your thing) and you should be good to go. I expect the next releases to fix some of the bugs and performance issues, but compared to what we saw in April, it's a huge improvement.

Kudos to Microsoft for that!

And if you want to know more hardcore details on how the Windows Subsystem for Linux is actually implemented, I recommend this series of blog posts from MSDN itself:

Fortunately things usually change for the best, and practices that "worked ok" in the past might be obsolete by now.

That happens to old blog posts, including my own as I've been posting for several years.

In this post I will fix what I posted (in pt-BR):

They are still good posts to understand the mechanics of what I will propose here, so you should read them if you're interested in how things work.

TL;DR is: use Cloudinary (with Attachinary) and install a CDN (Cloudfront) right now as it's super trivial to do.

Uploading files is not a task to be taken lightly. It's one of those tasks that you must not consider that it "works" just because it works in your development machine, for a few trivial uploads. Production level behavior changes drastically and it can bring your entire project down depending on how heavy it depends on user generated content.

There are a lot of moving parts to understand. But in the most basic solutions you just create an HTML multipart form with a file input field and submit it to some controller endpoint directly. This controller receives the blob binary and you save it to some local directory in the server.

This is the worst possible solution and unfortunatelly this is the one you will find the most through the web.

There are many problems to deal with this simple implementation:

We are in the era of 12+ megapixel cameras. Those are multi-megabyte sized files. Each upload will take a long time, specially if the user is trying to upload using unreliable or slow wireless networks.

Once the user's browser connects to the web application it will hold that connection and block the entire process through the duration of the request-response cycle. Making this simple to understand, if you only have 1 single process in the server, it will be unable to respond to any other request until the upload finishes.

Fortunately most deployments will use NGINX at the front of the web application - as they should -, so this effect is minimized as NGINX will receive the entire blob before passing it to the web application process underneath, making it not so problematic in most cases.

But if you're on Heroku, the router layer has a 30 seconds timeout limit. If the upload takes more than that (which is common with users in unreliable networks) this layer will cut the connection down before it finishes. Users will retry and this can cause your request queues to grow very fast.

Avoid uploading anything to Heroku. I consider this a feature as it forces application deployed on Heroku to use good practices as I will explain below.

You should also NOT save files to the local filesystem if you want to scale horizontaly, as one server will not be able to see what's in the other server unless you're in AWS EC2 or other IaaS, with shared mounted volumes, for example. In Heroku, you can't rely on anything in the filesystem as the machines are volatile and whenever a dyno restarts it loses whatever was not there during the deployment.

So you must upload what you received to an external storage, such as AWS S3, and this is also slow if you receive lots of uploaded files. You can rely on NGINX or Heroku's router layer to handle the heavy receiving of the files but you will make the request-response cycle super slow if you synchronously process and upload the files to S3.

The next option you think of is to add Sidekiq or some other Async Job mechanism to do this other heavy lifting. Now you have to create front-end placeholders to show to the users while the background job is not finished sending the files to S3.

The last option, if you dig deeper, is browser-based direct uploads to S3 and just posting the URLs to your web application. This is the ideal solution, but it's not easy to implement in your stack.

Once you nail everything above, you still do the mistake of putting the S3 URLs directly in the IMG tags in your HTML, which is not recommended by Amazon as S3 is recommended just for "storage" purposes. This is the easy part to fix, but most projects forget about that.

Cloudinary or Carrierwave Direct

In my previous posts I explained each of the above issues in more detail and I offer some (complicated) solutions, such as using the Refile project. At that time the Carrierwave plugins for Direct Upload were not ready.

Nowadays the solution is actually super easy and this is what I recommend: go straight to Cloudinary. This is a Software as a Service solution for proper direct uploads and dynamic image processing. You should definitelly explore the Attachinary gem to make the process easier.

If you have legacy Carrierwave, Paperclip, Shrine, Dragonfly, Refile or others, one jerry rigged - but reasonable - alternative is to just add Attachinary to the mix.

Let's say you have an old user.avatar uploader. You can just create a new user.new_avatar with Attachinary/Cloudinary. In the HTML form upload you start using just the Attachinary helpers. And in the HTML where you show the image itself, you can make a helper to check for @user.new_avatar? and show using cl_image_tag(@user.avatar.path instead, otherwise you show the old URLs.

You don't need to totally migrate everything to Cloudinary, just keep the old assets where they are and start putting the new assets in the Cloudinary configuration. It has a Free tier that can hold up to 75 thousand images and allow a traffic of up to 5GB. And with just USD 44 a month you have 10 times that. So it's really cheap if your application is serious about file uploading.

If you don't want to commit to an external service yet, and you're using Carrierwave, another option you must consider is testing out CarrierwaveDirect to test direct uploading to S3 from the browser. This will at least avoid the complexities of setting up async uploads from the web application to S3.

Update: Janko Marohnić posted a very good comment that I think is important to quote directly:

Great writeup, I wholeheartedly agree that we probably don't "just want something simple", especially concerning synchronous uploads and CDNs.
Shrine actually has a Cloudinary integration, and it's pretty advanced too; it supports direct uploads, setting upload options, storing Cloudinary metadata (hello responsive breakpoints) etc :). Also, the Cloudinary gem ships with a CarrierWave integration. I think it's beneficial to use Shrine or CarrierWave with Cloudinary, because then you get to keep your file attachments library, and just change the underlying storage service.
As for CarrierwaveDirect, I don't think it's actually very useful, because backgrounding you need to do all by yourself anyway, so basically all it gives you is the ability to generate a direct-upload form to S3. But generating request parameters and URL to S3 is something that's already built in into the official aws-sdk gem, CarrierwaveDirect just seems to be reimplementing th at logic. And the advantage of generating this through aws-sdk is that it's not HTML-specific, so you could setup an endpoint which returns this information as JSON, and now you have your multiple file uploads directly to S3 :)

As a disclaimer I didn't test CarrierwaveDirect myself and I remember it being quite broken 2 years ago. And I've heard good things about Shrine along the way but never did a proper testing. I'd recommend giving it a try if you want to migrate away from Carrierwave.

Don't serve assets from S3, use a CDN, any CDN

If you're sticking to S3 uploading, at the very least avoid serving those files from S3. In my "Heroku Encyclopedia" post I recommend using "S3 Assets Sync". DO NOT DO IT: use a CDN instead, it's faster, easier and the correct solution

The first thing you want to do is sign up to AWS's Cloudfront CDN service. Always use a CDN for all your assets, not only the ones internal to your web application but also every user uploaded content. It's super simple to create a new CDN endpoint pointing to your S3 bucket.

Carrierwave will default to the S3 bucket host, but once you have the Cloudfront endpoint you can easily change all the uploaded files URLs like this:

12345678910111213

# ./config/initializers/carrierwave.rbCarrierWave.configure do |config|  config.fog_credentials = {:provider => "AWS",:aws_access_key_id => ENV['AWS_ACCESS_KEY_ID'],:aws_secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'],  }  config.fog_directory = ENV['S3_BUCKET']# use only one of the following 2 settings# config.fog_host = "http://#{ENV['S3_BUCKET']}.s3.amazonaws.com"  config.fog_host = ENV['S3_CDN'] # for cloudfrontend

You must obviously configure the environment variable S3_CDN to point to the Cloudfront endpoint specific to the particular S3 bucket you're using.

The important part is: serve ALL ASSETS from a CDN, no matter what. Never directly from your web application, avoid directly serving from S3. It's almost trivial to implement and if you don't like AWS for any reason, you can choose Fastly, Cloudflare and many others.

They all work the same way. There is NO excuse to not use a CDN no matter the size of your application.

I started evangelizing Git in 2007. It was a very tough sell to make at the time.

Outside of the kernel development almost no one wanted to learn it and we had very worthy competitors, from Subversion, to Mercurial, to Bazaar, to Darcs, to Perforce, and so on. But those of use that dug deeper knew that Git had the edge and it was a matter of time.

Then GitHub showed up in 2008 and the rest is history. For many years it was just "cool" to be in GitHub. The Ruby community drove GitHub up into the sky. Finally it became the status quo and the one real monopoly in information repositories - not just software source code, but everything.

I always knew that we should have a "local" option, which is why I tried to contribute to Gitorious way back in 2009. Other options arose, but eventually GitLab appeared around 2011 and picked up steam in the last couple of years.

GitHub itself raised USD 350 million in funding and one of its required goals is to nail the Enterprise Edition for big corporations that don't want their data outside their closed gardens. Although GitHub hosts every single open source project out there, they are themselves closed-source.

GitLab Inc. started differently with an open source-first approach with their Community Edition (CE) and having both a GitHub-like hosted option as well as a supported Enteprise Edition for fearsome corporations. They already raised USD 5.62 million in funding, and they are the most promising alternative to GitHub so far.

Of course, there are other platforms such as Atlassian's Bitbucket. But I believe Atlassian's strategy is slower and they have a larger suite of enterprise products to sell first, such as Confluence and Jira. I don't think they ever posed much of a competition against GitHub.

GitLab really started accelerating in 2015 as this commit graph shows:

contributions

It's been steadily growing since 2011, but they seem to have crossed the first tipping point around late 2014, from early adopters to the early majority. This became more important as GitHub announced their pricing changes in May.

They said they haven't committed to a dead line to enforce the change, so organizations can opt out of the new format for the time being. They are changing from "limited repositories and unlimited users" to "unlimited repositories and limited users".

The Cost-Benefit Conundrum

For example, if you have up to 8 developers in the USD 50/month (20 private repositories), the change won't affect you, as you will pay USD 25/month for 5 users and USD 9 for additional users (total of USD 52/month).

Now, if you have a big team of 100 developers currently in the Diamond Plan of USD 450/month (300 private repositories), you would have to pay USD 25/month + 95 times USD 9, which totals a staggering USD 880/month! Double the amount!

This is an extra USD 10,560 per year!

And what does GitLab affords you instead?

You can have way more users and more repositories in a USD 40/month virtual box (4GB of RAM, 60GB SSD, 4TB transfer).

And it doesn't stop there. GitLab also has very functional GitLab Multi Runner which you can install in a separate box (actually, at least 3 boxes - more on that below).

You can easily connect this runner to the build system over GitLab so every new git push trigger the runner to run the automated test suite in a Docker image of your choosing. So it's a fully functional, full featured Continuous Integration system nicely integrated in your GitLab project interface:

Pipeline

CI Runner

Reminds of you anything? Yep, it's a fully functional alternative to Travis-CI, Semaphore, CircleCI or any other CI you're using with a very easy to install procedure. Let's say you're paying Travis-CI USD 489/month to have 10 concurrent jobs.

You can install GitLab Runner in 3 boxes of USD 10/month (1GB RAM, 1 Cores, 30GB SSD) and have way more concurrent jobs (20? 50? Auto-Scale!?) that runs faster (in a simple test, one build took 15 minutes over Travis took less than 8 minutes at Digital Ocean).

So let's make the math for a year's worth of service. First considering no GitHub plan change:

USD 5,400 (GitHub) + USD 5,868 (Travis) = USD 11,268 a year.

Now, the GitLab + GitLab Runner + Digital Ocean for the same features and unlimited users, unlimited repositores, unlimited concurrent builds:

USD 480 (GitLab) + USD 840 (Runner box) = USD 1,320 a year.

This is already almost 8.5x cheaper with almost no change in quality.

For the worst case scenario, compare it when GitHub decides to enforce the new plans:

USD 10,560 (GitHub new plans) + USD 5,868 (Travis) = USD 16,428

Now the GitLab option is 11x cheaper! You're saving almost USD 15,000 a year! This is not something you can ignore in you cost sheet.

As I said, the calculations above are only significant in a scenario of a 100 developers. You must do your own math taking into account your team size and number of active projects (you can always archive unused projects).

Even if you don't have 100 developers. Let's consider the scenario for 30 developers in the new GitHub per user plans and a smaller Travis configuration for 5 concurrent jobs:

USD 3,000 (GitHub new plan) + USD 3,000 (Travis) = USD 6,000

It's 4.5x cheaper in the Digital Ocean + GitLab suite option.

Heck, let's consider the Current GitHub plan (the Platinum one, for up to 125 repositories):

USD 2,400 (GitHub current plan) + USD 3,000 (Travis) = USD 5,400

Still at least 4x more expensive than a GitLab-based solution!

And how long will it take for a single developer to figure out the set up and migrate everything from GitHub over to the new GitLab installation? I will say that you can reserve 1 week of work for the average programmer to do it following the official documentation and my tips and tricks below.

Installing GitLab CE

I will not bore you with what you can readily find over the Web. I highly recommend you start with the easiest solution first: Digital Ocean's One-Click Automatic Install. Install it in at least a 4GB RAM machine (you will want to keep it if you like it).

Of course, there is a number of different installation options, from AWS AMI images to Ubuntu packages you can install manually. Study the documentation.

It will cost you USD 40 for a month of trial. If you want to save as much as tens of thousands of dollar, this is a bargain.

GitLab has many customization options. You can lock down your private GitLab to allow only users with an official e-mail from your domain, for example. You can configure OAuth2 providers so your users can quickly sign in using their GitHub, Facebook, Google or other accounts.

A Few Gotchas

I've stumbled upon a few caveats in the configuration. Which is why I recommend that you plan ahead - study this entire article ahead of time! -, do a quick install that you can blow away, so you can "feel" the environment before trying to migrate all your repos over to your brand new GitLab. As a reference, this is a part of my /etc/gitlab/gitlab.rb:

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455

# register a domain for your server and place it here:external_url "http://my-gitlab-server.com/"# you will want to enable [LFS](https://git-lfs.github.com)gitlab_rails['lfs_enabled'] = true# register your emailsgitlab_rails['gitlab_email_from'] = "no-reply@my-gitlab-server.com"gitlab_rails['gitlab_support_email'] = "contact@my-gitlab-server.com"# add your email configuration (template for gmail)gitlab_rails['smtp_enable'] = truegitlab_rails['smtp_address'] = "smtp.gmail.com"gitlab_rails['smtp_port'] = 587gitlab_rails['smtp_user_name'] = "-- some no-reply email ---"gitlab_rails['smtp_password'] = "-- the password ---"gitlab_rails['smtp_domain'] = "my-gitlab-server.com"gitlab_rails['smtp_authentication'] = "login"gitlab_rails['smtp_enable_starttls_auto'] = truegitlab_rails['smtp_openssl_verify_mode'] = 'peer'# this is where you enable oauth2 integrationgitlab_rails['omniauth_enabled'] = true# CAUTION!# This allows users to login without having a user account first. Define the allowed providers# using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none.# User accounts will be created automatically when authentication was successful.gitlab_rails['omniauth_allow_single_sign_on'] = ['github', 'google_oauth2', 'bitbucket']gitlab_rails['omniauth_block_auto_created_users'] = truegitlab_rails['omniauth_providers'] = [  {"name" => "github","app_id" => "-- github app id --","app_secret" => "-- github secret --","url" => "https://github.com/","args" => { "scope" => "user:email" }  },  {"name" => "google_oauth2","app_id" => "-- google app id --","app_secret" => "-- google secret --","args" => { "access_type" => "offline", "approval_prompt" => '', hd => 'codeminer42.com' }  },  {"name" => "bitbucket","app_id" => "-- bitbucket app id --","app_secret" => "-- bitbucket secret id --","url" => "https://bitbucket.org/"  }]# if you're importing repos from GitHub, Sidekiq workers can grow as high as 2.5GB of RAM and the default [Sidekiq Killer](http://docs.gitlab.com/ee/operations/sidekiq_memory_killer.html) config will cap it down to 1GB, so you want to either disable it by adding '0' or adding a higher limitgitlab_rails['env'] = { 'SIDEKIQ_MEMORY_KILLER_MAX_RSS' => '3000000' }

There are dozens of default variables you can override, just be careful on your testings.

Every time you change a configuration, you can just run the following commands:

12	sudo gitlab-ctl reconfiguresudo gitlab-ctl restart

You can open a Rails console to inspect production objects like this:

1	gitlab-rails console

I had a lot of trouble importing big repos from GitHub, but after a few days debugging the problem with GitLab Core Team developers Douglas Alexandre, Gabriel Mazetto, a few Merge Requests and some local patching and I was finally able to import relatively big projects (more than 5,000 commits, more than 1,000 issues, more than 1,200 pull requests with several comments worth of discussion threads). A project of this size can take a couple of hours to complete, mainly because it's damn slow to use GitHub's public APIs (they are slow and they have rate limits and abuse detection, so you can't fetch everything as fast as your bandwidth would allow).

(By the way, don't miss GitLab will be over at Rubyconf Brazil 2016, on Sep 23-24)

Migrating all my GitHub projects took a couple of days, but they all went through smoothly and my team didn't have any trouble, just adjusting their git remote URLs and they're done.

The import procedure from GitHub is quite complete, it brings not only the git repo per se, but also all the metadata, from labels to comments and pull request history - which is the one that usually takes more time.

But I'd recommend waiting for at least version 8.11 (it's currently 8.10.3) before trying to import large GitHub projects.

If you're on Bitbucket, unfortunatelly there are less features in the importer. It will mostly just bring the source code. So be aware of that if you extensively depend on their pull request system and you want to preserve this history. More feature will come and you can even help them out, they are very resourceful and willing to make GitLab better.

Side-track: Customizations for every Digital Ocean box

Assume that you should run what's in this section for all new machines you create over Digital Ocean.

First of all, they come without a swap file. No matter how much RAM you have, the Linux OS is meant to work better by combining a swap file. You can read more about it later, for now just run the following as root:

fallocate -l 4G /swapfilechmod 600 /swapfilemkswap /swapfileswapon /swapfilesysctl vm.swappiness=10sysctl vm.vfs_cache_pressure=50

Edit the /etc/fstab file and add this line:

1	/swapfile none swap sw 0 0

Finally, edit the /etc/sysctl.conf file and add these lines:

12	vm.swappiness=10vm.vfs_cache_pressure = 50

Don't forget to set the default locale of your machine. Start by editing the /etc/environment file and adding:

12	LC_ALL=en_US.UTF-8LANG=en_US.UTF-8

Then run:

12	sudo locale-gen en_US en_US.UTF-8sudo dpkg-reconfigure locales

Finally, you should have Ubuntu automatically install stable security patches for you. You don't want to forget machines online without the most current security fixes, so just run this:

1	sudo dpkg-reconfigure --priority=low unattended-upgrades

Choose "yes" and you're done. And of course, for every fresh install, it's always good to run the good old:

1	sudo apt-get update && sudo apt-get upgrade

This is the very basics, I believe it's easier to have an image with all this ready, but if you use the standard Digital Ocean images, these settings should do the trick for now.

Installing the CI Runner

Once you finish your GitLab installation, it's super easy to deploy the GitLab Runner. You can use the same machine but I recommend you install it in a separate machine.

If you don't know what a runner is, just imagine it like this: It's basically a server connected to the GitLab install. When it's available and online, whenever someone pushes a new commit, merge request, to a repository that has a gitlab-ci-yml file present, GitLab will push a command to the runner.

Depending on how you configured the runner, it will receive this command and spawn a new Docker container. Inside the container it will execute whatever you have defined in the gitlab-ci.yml file in the project. Usually it's fetching cached files (dependencies, for example), and run your test suite.

In the most basic setup, you will only have one Runner and any subsequent builds from other users will wait in line until they finish. If you've used external CI services such as Travis-CI or CircleCI, you know that they charge for some number of concurrent builds. And it's very expensive.

The less concurrent builds available, the more your users will have to wait for feedback on their changes, and less productive you will become. People may even start to avoid adding new tests, or completely ignore the tests, which will really hurt the quality of your project over time. If there is one thing you must not do is not having good automated test suites.

Gabriel Mazetto pointed me to a very important GitLab CI Runner feature: auto-scaling. This is what they use in their hosted offering over at GitLab.com.

You can easily set up a runner that can use "docker-machine" and your IaaS provider APIs to spin up machines on the fly to run as many concurrent builds as you want, and it will be super cheap!

For example, on Digital Ocean you can be charged USD 0.06 (6 cents) per hour of usage of a 4GB machine. Over at AWS EC2 you can be charged USD 0.041 per hour for an m3.medium machine.

There is extensive documentation but I will try to summarize what you have to do. For more details I highly recommend you to study their official documentation.

Start by creating 3 new machines at Digital Ocean, all in the same Region with private networking enabled! I will list a fake private IP address just for the sake of advancing in the configuration examples:

a 1GB machine called "docker-registry-mirror", (ex 10.0.0.1)
a 1GB machine called "ci-cache", (ex 10.0.0.2)
a 1GB machine called "ci-runner", (ex 10.0.0.3)

Yeah, they can be small as very little will run on them. You can be conservative and choose the 2GB RAM options just to be on the safe side (and pricing will still be super cheap).

Don't forget to execute the basic configuration I mentioned above to enable a swapfile, auto security update and locale regeneration.

SSH in to "docker-registry-mirror" and just run:

1234	docker run -d -p 6000:5000 \ -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \ --restart always \ --name registry registry:2

Now you wil have a local Docker images registry proxy and cache at 10.0.0.1:6000 (take note of the real private IP).

SSH in to "ci-cache" and run:

mkdir -p /export/runnerdocker run -it --restart always -p 9005:9000 \        -v /.minio:/root/.minio -v /export:/export \        --name minio \        minio/minio:latest /export

Now you will have an AWS S3 clone called Minio running. I didn't know this project even existed, but it is a nifty little service written in Go to clone the AWS S3 behavior and APIs. So now you can have your very own S3 inside your infrastructure!

After Docker spin ups, it will print out the Access Key and Secret keys, make notes. And this service will be running at 10.0.0.2:9005.

You can even open a browser and see their web interface at http://10.0.0.2:9005 and use the access and secret keys to login. Make sure you have a bucket named "runner". The files will be stored at the /export/runner directory.

Minio Dashboard

Make sure the bucket name is valid (it must be a valid DNS naming, for example, DO NOT use underlines).

Open this URL from your freshly installed GitLab-CE: http://yourgitlab.com/admin/runners and take note of the Registration Token. Let's say it's 1aaaa_Z1AbB2CdefGhij

Admin Area for Runner Registration Token

Finally, SSH in to "ci-runner" and run:

123456789

curl -L https://github.com/docker/machine/releases/download/v0.7.0/docker-machine-`uname -s`-`uname -m` > /usr/local/bin/docker-machinechmod +x /usr/local/bin/docker-machinecurl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bashsudo apt-get install gitlab-ci-multi-runnerrm -Rf ~/.docker # just to make sure

Now you can register this new runner with your GitLab install, you will need the Registration Token mentioned above.

1	sudo gitlab-ci-multi-runner register

You will be asked a few questions, and this is what you can answer:

12345678910111213

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci )https://yourgitlab.com/ciPlease enter the gitlab-ci token for this runner1aaaa_Z1AbB2CdefGhij # as in the example abovePlease enter the gitlab-ci description for this runnermy-autoscale-runnerINFO[0034] fcf5c619 Registering runner... succeededPlease enter the executor: shell, docker, docker-ssh, docker+machine, docker-ssh+machine, ssh?docker+machinePlease enter the Docker image (eg. ruby:2.1):codeminer42/ci-ruby:2.3INFO[0037] Runner registered successfully. Feel free to start it, but if it'srunning already the config should be automatically reloaded!

Let's make a copy of the original configuration, just to be safe:

1	cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.bak

Copy the first few lines of this file (you want the token), it will look like this:

12345678

concurrent = 1check_interval = 0[[runners]]  name = "my-autoscale-runner"  url = "http://yourgitlab.com/ci"  token = "--- generated runner token ---"  executor = "docker+machine"

The important part here is the "token". You will want to take note of it. And now you also will want to create a new API Token over at Digital Ocean. Just Generate a New Token and take note.

You can now replace the entire config.toml file for this:

1234567891011121314151617181920212223242526272829303132333435

concurrent = 20check_interval = 0[[runners]]  name = "my-autoscale-runner"  url = "http://yourgitlab.com/ci"  token = "--- generated runner token ---"  executor = "docker+machine"  limit = 15  [runners.docker]    tls_verify = false    image = "codeminer42/ci-ruby:2.3"    privileged = false  [runners.machine]    IdleCount = 2                   # There must be 2 machines in Idle state    IdleTime = 1800                 # Each machine can be in Idle state up to 30 minutes (after this it will be removed)    MaxBuilds = 100                 # Each machine can handle up to 100 builds in a row (after this it will be removed)    MachineName = "ci-auto-scale-%s"   # Each machine will have a unique name ('%s' is required)    MachineDriver = "digitalocean"  # Docker Machine is using the 'digitalocean' driver    MachineOptions = ["digitalocean-image=coreos-beta","digitalocean-ssh-user=core","digitalocean-access-token=-- your new Digital Ocean API Token --","digitalocean-region=nyc1","digitalocean-size=4gb","digitalocean-private-networking","engine-registry-mirror=http://10.0.0.1:6000"    ]  [runners.cache]    Type = "s3"   # The Runner is using a distributed cache with Amazon S3 service    ServerAddress = "10.0.0.2:9005"  # minio    AccessKey = "-- your minio access key --"    SecretKey = "-- your minio secret key"    BucketName = "runner"    Insecure = true # Use Insecure only when using with Minio, without the TLS certificate enabled

And you can restart the runner to pick up the new configuration like this:

1	gitlab-ci-multi-runner restart

As I said before, you will want to read the extensive official documentation (and every link within).

If you did everything right, changing the correct private IPs for the docker registry and cache, the correct tokens, and so forth, you can log in to your Digital Ocean dashboard and you will see something like this:

Digital Ocean CI Setup

And from the ci-runner machine, you can list them like this:

# docker-machine lsNAME                                                ACTIVE   DRIVER         STATE     URL                         SWARM   DOCKER    ERRORSrunner-xxxx-ci-auto-scale-xxxx-xxxx   -        digitalocean   Running   tcp://191.168.0.1:2376            v1.10.3runner-xxxx-ci-auto-scale-xxxx-xxxx   -        digitalocean   Running   tcp://192.169.0.2:2376           v1.10.3

They should not list any errors, meaning that they are up and running, waiting for new builds to start.

There will be 2 new machines listed in your Digital Ocean dashboard, named "runner-xxxxx-ci-auto-scale-xxxxx". This is what IdleCount = 2 does. If they stay idle for more than 30 minutes (IdleTime = 1800) they will be shut down so you don't get charged.

You can have several "runner" definitions, each with a limit of builds/machines that can be spawned in Digital Ocean. You can have other runner definitions for other providers, for example. But in this example we are limited to at most 15 machines, so 15 concurrent builds.

The concurrent limit is a global setting. So if I had 3 runner definitions, each with a limit of 15, they would still be globally limited to 20 as defined in the concurrent global variable.

You can use different providers for specific needs, for example, to run OS X builds or Rapsberry PI builds or other exotic kinds of builds. In the example I am keeping it simple and just setting many builds in the same provider (Digital Ocean).

And don't worry about the monthly fee for each machine. When used in this manner, you will be paying per hour.

Also, make sure you spinned up all your machines (docker-registry, minio cache, CI runner) all with private networking enabled (so they talk through the internal VLAN instead of having to go all the way through the public internet) and that they are all in the same region data center (NYC1 is New York 1 - New York has 3 sub-regions, for example). Don't start machines in different regions.

Because we have Docker proxy/cache and Minio/S3 cache, your builds will take take longer the first time (let's say, 5 minutes), and then subsequent build will fetch everything from the cache (taking, let's say, 1:30 minute). It's fast and it's convenient.

Setting up each Project for the Runner

The Runner is one of the newest pieces of the GitLab ecosystem so you might have some trouble at first to figure out a decent configuration. But once you have the whole infrastructure figured out as described in the previous section, now it's as easy as adding a .gitlab-ci.yml file to your root directory. Something like this:

123456789101112131415161718192021222324252627282930

# This file is a template, and might need editing before it works on your project.image: codeminer42/ci-ruby:2.3# Pick zero or more services to be used on all builds.# Only needed when using a docker container to run your tests in.# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-serviceservices:  - postgres:latest  - redis:latestcache:key: your-project-nameuntracked: truepaths:    - .ci_cache/variables:RAILS_ENV: 'test'DATABASE_URL: postgresql://postgres:@postgresCODECLIMATE_REPO_TOKEN: -- your codeclimate project token --before_script:  - bundle install --without development production -j $(nproc) --path .ci_cache  - cp .env.sample .env  - cp config/database.yml.example config/database.yml  - bundle exec rake db:create db:migratetest:script:    - xvfb-run bundle exec rspec

My team at Codeminer 42 prepared a simple Docker image with useful stuff pre-installed (such as the newest phantomjs, xvfb, etc), so it's now super easy to enable automated builds within GitLab by just adding this file to the repositories. (Thanks to Carlos Lopes, Danilo Resende and Paulo Diovanni - who will be talking about Docker at Rubyconf Brasil 2016, by the way)

GitLab-CI even supports building a pending Merge Request, and you can enforce the request so it can only be merged if builds pass, just like in GitHub + Travis. And as Code Climate is agnostic to Repository host or CI runner, you can easily integrate it as well.

Project Force Successful Build to Merge

Conclusion

The math is hard to argue against: the GitLab + GitLab-CI + Digital Ocean combo is a big win. GitLab's interface is very familiar so users from GitHub or Bitbucket will feel quite at home in no time.

We can use all the Git flows we're used to.

GitLab-CE is stil a work in progress though, the team is increasing their pace but there are currently more than 4,200 open issues. But as this is all Ruby on Rails and Ruby tooling, you can easily jump in and contribute. No contribution is too small. Just by reporting how to reproduce a bug is help enough to assist the developers to figure out how to improve faster.

But don't shy away because of the open issues, it's fully functional as of right now and I have not found any bugs that could be considered show stoppers.

They have many things right. First of all, it's a "simple" Ruby on Rails project. It's a no-thrills front-end with plain JQuery. The choice of HAML for the views is questionable but it doesn't hurt. They use good old Sidekiq+Redis for asynchronous jobs. No black magic here. A pure monolith that's not difficult to understand and to contribute.

The APIs are all written using Grape. They have the GitLab CE project separated from other components, such as the GitLab-Shell and GitLab-CI-Multi-Runner.

They also forked Omnibus in order to be able to package the CE Rails project as a ".deb". Everything is orchestrated with Docker. And when a new version is available, you only need to apt-get update && apt-get ugprade and it will do all the work of backing up and migratind Postgresql, updating the code, bundling in new dependencies, restarting the services and so forth. It's super convenient and you should take a look at this project if you have complicated Rails deployments into your own infrastructure (out of Heroku, for example).

I am almost done moving hundreds of repositories from both BitBucket and GitHub to GitLab right now and the developers from my company are already using it in a daily basis without any problems. We are almost at the point where we can disengage from BitBucket, GitHub and external CIs.

You will be surprised how easy your company can do it too and save a couple thousand dollars in the process, while having fun doing it!

In my quest to create my own private infrastructure I've already moved away from Pivotal Tracker (saving this for a future post), Bitbucket, Github, Travis and into GitLab and GitLab Runner. There are more services I will move away from in the future, so expect more posts like this.

In my previous post I forgot to mention minimal firewall and SSL configuration, but you can check it here.

One of the most difficult pieces to move away from is Slack. Don't get me wrong, it's a great tool and it gets the job done. The free-tier is really competent but I really want some of the payed features such as history of all conversations with search capabilities and uploads without limit.

Codeminer Rocket

The Cost Conundrum

But then you have to commit to pay USD 8/month per user (USD 6.67 if you pay ahead of time). If you're a small team (around a dozen people) it's well worth it and you should definitely pay and get over with.

If you have more than 20 people then it starts to become a conundrum. On my GitLab cost calculations, some people argued that saving USD 15k yearly is not such a big deal, and they're not wrong.

If you're from outside of the US coastal cities, specially from countries in South America, Asia where currency can be more than 3 times devalued, than it quickly starts to become a big deal.

Apart from that, we are currently offloading a whole lot of out IPs and knowledge into the care of companies we don't really know and where we don't hold any real stakes.

If you're a medium to big company it really starts to make sense to want to have more control over your own property. Of course, it costs some work and maintenance, so it's not for everybody.

What I will argue is that if you have 150 people or more, and you're still growing, it's not such a big burden to jump into your own infrastructure.

You avoid paying around USD 12,000 a year or more, and instead you will pay USD 960 a year in Digital Ocean boxes and MMS plus a few hours every month in basic maintenance. That's 12 times cheaper.

Again, you shouldn't do it just for the cost savings as the maintenance burden can easily not be worth it if you or your team are not devops savvy and you don't have anyone to keep monitoring it. For most companies, the conversation history is not so important and even the free-tier will be more than enough.

Cost is less relevant depending on your context, and there are many different uses for group chats. For example, if you're an open source community, you would try Gitter instead.

There are also many other alternatives that you may want to try.

For the sake of this article, let's assume you're a small to medium company, in need of private rooms, many external users (for example, your clients), with at least 50 people. Then this can start to make sense. Any other scenario and we have to analyze it differently.

Why I chose Rocket.chat

With the cost reasoning out of the way, then it's a matter of choosing which alternative to use.

I tried MatterMost first and I really wanted to like it. It has a very similar feature set to Slack and most importantly, it's written in Go. So it's both lightweight, very fast, and not so difficult to contribute. It uses MySQL or PostgreSQL which I like.

Unfortunatelly, it has an important show-stopper for my needs. You can create both public channels or private groups, but anyone can delete a private group. If you want permission control for that you must use their "Enterprise" offering and pay USD 20/year per user.

It's still at least 3 times cheaper than Slack, so it may be a good option for you.

I really want to have full-control, not just make it cheaper. I could pay Slack already if I didn't care, so I would not just pay an alternative and not have full-control (including the ability to tweak and improve the entire code base).

Then, it took me back to Rocket.chat. It's feature set also rivals Slack, it's good looking enough. But, it's made in Meteor. Now, I don't have anything against Meteor and I really think a Slack-clone is exactly the kind of use case where you could use Meteor to its full potential.

Technically I really think it's a downside to be forced to use Mongodb (Meteor requires Mongo). For small installations I really prefer to have PostgreSQL. MongoDB is competent, but for medium to big installations, and I will show you why below.

I strongly dislike the culture of writing software without minimal care such as having a reasonably complete test suite. You browse through the many packages that comprise Rocket.chat and several of them have no tests whatsoever. Some packages do have jasmine tests, but the majority is lacking.

This is one example I picked randomly, out of the few test files I found:

1234	describe 'rocketchat:katex Client', -> it 'should exist', -> expect(RocketChat.katex).toBeDefined()

This is sad, sorry to say that, but it is. Not surprisingly I've seen katex issues while using the interface. And if you think this was just a poor choice, I picked another test file:

1234	describe 'rocketchat:markdown Client', -> it 'should exist', -> expect(RocketChat.Markdown).toBeDefined()

Really?

So, the code itself is not so pretty, you have to keep this in mind. I'd much rather use Mattermost (which is much better structured and with enough tests), but it's not entirely free, so it's not an option to me. For my scenario, I'd rather have sloppy code (that minimally "works") that I can tweak than code I can't see, for this particular venture. Many inexperienced developers may argue about the necessity of having automated tests, but not having them also impacts this:

Rocket.chat contribution graph

If we see how the project is evolving over time, we can see that interest is diminishing. The lack of good automated tests also make it more difficult for new people to contribute without adding regressions, make contribution reviews more difficult and time consuming.

On the other hand the feature set is good enough as it is. There is only so much you can add to a Slack-clone. The upside is that if the code doesn't change too much, I can tweak it more without having many things breaking all the time. It's the complete opposite from the impressive contribution boost over at GitLab, for example.

Doing a few days of testing through the administration settings, user interface, having dozens of users doing real conversation, convinced me that even though the code quality is bad, the feature set is indeed there. I think I can live with that. Your mileage may vary though.

I am using Franz to open Slack, Rocket.chat, Hangouts, Messenger and any number of other communicators in one single app, which is great. And Rocket.chat does provide mobile apps based off PhoneGap. Again, it's good, not great.

Many people have heavy bots or integrations into their Slack configuration. The alternatives such as Rocket.chat do support incoming and outgoing webhooks and most of the integrations should be possible (with some tweaking and hacking such as this Giphy example), so make a list of all important integrations and do a Google research on that before continuing.

Planning the Infrastructure

All that having being said, I'd recommend you try Rocket.chat's demo server first to see if it has the minimal features you need. And also that you install a dry-run in a small Digital Ocean box with the intention to blow it off later.

For a single-instance, stand-alone installation, temporary dry-run, I recommend you start by installing the MongoDB One-Click Application and follow this instruction to install everything in one machine.

This is the worst possible production environment, which is why I am being so repetitive into saying that you should destroy this box after you do your testing.

As I've said before, I am assuming a scenario where you have 50 or more people, with real clients, important communication going on. You don't want to keep a simple installation like this for production usage.

For a minimal production installation you will need at the very least 4 boxes with 1 GB of RAM each. Now, why 4?

This is an important section for most of the beginner Javascript programmers out there playing with the so called "MEAN" stack (MongoDB, Express, Angularjs, Node.js). I've seen people deploy single-instance, stand-alone "MEAN" apps for production usage. And this is super bad.

MongoDB is NOT MEANT to function with just one single instance!

More than that:

DO NOT start an even number of MongoDB nodes, or at least install an Arbiter

So, the minimal setup for MongoDB is a Three-member Replica Set, no less!

The article "MongoDB Gotchas & How To Avoid Them" is a little bit old but you should be aware of several of the documented gotchas, including adding replica sets and avoid waiting too long to start sharding you data.

The most important part of making Rocket.chat reliable is to make your MongoDB minimally reliable, and this is a hidden cost you must consider.

Installing a Three-Member Replica Set

For the sake of this article I will assume that you created 3 (three) MongoDB One-Click Applications on Digital Ocean's at the very least 1 GB RAM boxes (2 GB RAM machines are a good choice if you're in doubt, but never the 512 Mb) and another bare Ubuntu 14.04 box to be the web server. Create all machines in the same sub-region, all of them with private networking and backup enabled.

Take note of the public IPs and private IPs, for the sake of this article let's say you have this:

Public IP     Private IP    Hostname192.160.10.1  10.0.0.1      mongo1.yourdomain.com192.160.10.2  10.0.0.2      mongo2.yourdomain.com192.160.10.3  10.0.0.3      mongo3.yourdomain.com192.160.10.4  10.0.0.4      yourdomain.com

In the commands and code snippets below, make sure to replace the fake IPs for your own private IPs.

As I said in my GitLab post, create a swap file, configure the locale, set it up for automatic security updates, and do upgrade the Ubuntu packages.

The next first thing: configure the firewall in all machines (assuming ufw package is already installed):

sudo ufw resetsudo ufw default deny incomingsudo ufw default allow outgoingsudo ufw allow httpssudo ufw allow sshsudo ufw allow in on eth1 to any port 27017sudo ufw enable

We are allowing MongoDB's 27017 port only for machines in the same private networking (through eth1, the public IP go through eth0). We're also allowing port 22 for SSH and although I am not listing it here in this article you should also consider installing and configuring fail2ban. Digital Ocean also has a good post on UFW for more details.

Next thing to do on the MongoDB machines is to Disable Transparent Huge Pages (THP). For that create a file /etc/init.d/disable-transparent-hugepages, with this content:

1234567891011121314151617181920212223242526272829

#!/bin/sh### BEGIN INIT INFO# Provides:          disable-transparent-hugepages# Required-Start:    $local_fs# Required-Stop:# X-Start-Before:    mongod mongodb-mms-automation-agent# Default-Start:     2 3 4 5# Default-Stop:      0 1 6# Short-Description: Disable Linux transparent huge pages# Description:       Disable Linux transparent huge pages, to improve#                    database performance.### END INIT INFOcase $1 in  start)    if [ -d /sys/kernel/mm/transparent_hugepage ]; then      thp_path=/sys/kernel/mm/transparent_hugepage    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then      thp_path=/sys/kernel/mm/redhat_transparent_hugepage    else      return 0    fi    echo 'never' > ${thp_path}/enabled    echo 'never' > ${thp_path}/defrag    unset thp_path    ;;esac

Now you can register it as a service:

123	sudo chmod 755 /etc/init.d/disable-transparent-hugepagessudo update-rc.d disable-transparent-hugepages defaultssudo /etc/init.d/disable-transparent-hugepages start

Stop MongoDB in all 3 machines:

1	sudo service mongod stop

Now comes a complicated procedure that you should not do wrong. It's only complicated because you have to do it in all 4 machines minding their correct private IP addresses. If you already have your own DNS server, you can skip this, otherwise follow through carefully.

You will have to edit both /etc/hostname and /etc/hosts. Each machine will have a particular name and they should be able to find the other machines by name. Let's start by editing the /etc/hostname on each machine. Use the table in the beginning of this section, for example, in the machine with public IP 192.160.10.1 you must edit it's hostname to be mongo1.yourdomain.com, and so on for all the machines.

Once you have it done, you have to replace the line 127.0.0.1 localhost in the /etc/hosts in each of them for the following:

127.0.0.1 localhost mongo1.yourdomain.com10.0.0.1 mongo1.yourdomain.com10.0.0.2 mongo2.yourdomain.com10.0.0.3 mongo3.yourdomain.com10.0.0.4 yourdomain.com

This snippet assumes you're in the machine with private IP 10.0.0.1, of course. Make sure you change mongo1.yourdomain.com in the first line for the hostname of the machine you're in. If this is too difficult for you, you definitely should not try to maintain your own infrastructure, so beware.

Finally, let's edit the /etc/mongod.conf with these changes:

123456789

storage:  dbPath: /mongodb-data...net:  port: 27017  bindIp: 0.0.0.0...replication:  replSetName: rs0

This should be exactly the same in all machines. And you must create this /mongodb-data directory like this:

12	sudo mkdir /mongodb-datasudo chown -R mongodb:mongodb /mongodb-data

Now go ahead on machine mongo1.yourdomain.com, start up with sudo service mongod start and fire up the mongo command, and you should see something like this:

1234	# mongoMongoDB shell version: 3.2.8connecting to: test>

Let's now configure the replica set by issuing this command in the MongoDB command-line interface:

123	rs.initiate()rs.add('mongo2.yourdomain.com')rs.add('mongo3.yourdomain.com')

And you can now log in to the secondary machines and sudo service mongod start in all of them. They should all start joining the replica set and syncing.

You can check the status of the replication like this:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556

> rs.conf(){"set" : "rs0","date" : ISODate("2016-08-09T17:33:10.466Z"),"myState" : 1,"term" : NumberLong(-1),"heartbeatIntervalMillis" : NumberLong(2000),"members" : [    {"_id" : 1,"name" : "mongo1.yourdomain.com:27017","health" : 1,"state" : 1,"stateStr" : "PRIMARY","uptime" : 11727,"optime" : Timestamp(1470763990, 3),"optimeDate" : ISODate("2016-08-09T17:33:10Z"),"electionTime" : Timestamp(1470752553, 1),"electionDate" : ISODate("2016-08-09T14:22:33Z"),"configVersion" : 6,"self" : true    },    {"_id" : 2,"name" : "mongo2.yourdomain.com:27017","health" : 1,"state" : 2,"stateStr" : "SECONDARY","uptime" : 11693,"optime" : Timestamp(1470763988, 4),"optimeDate" : ISODate("2016-08-09T17:33:08Z"),"lastHeartbeat" : ISODate("2016-08-09T17:33:10.026Z"),"lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.887Z"),"pingMs" : NumberLong(0),"syncingTo" : "mongo3.yourdomain.com:27017","configVersion" : 6    },    {"_id" : 3,"name" : "mongo3.yourdomain.com:27017","health" : 1,"state" : 2,"stateStr" : "SECONDARY","uptime" : 11693,"optime" : Timestamp(1470763988, 4),"optimeDate" : ISODate("2016-08-09T17:33:08Z"),"lastHeartbeat" : ISODate("2016-08-09T17:33:09.994Z"),"lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.655Z"),"pingMs" : NumberLong(0),"syncingTo" : "mongo1.yourdomain.com:27017","configVersion" : 6    }  ],"ok" : 1}

Each box should be syncing data to another box, one of them will be marked "PRIMARY" and the others will be "SECONDARY". If one of them "dies" for some reason, one of the others will be elected as PRIMARY until you add a replacement box. Read more details on this procedure in Digital Ocean's post about implementing replication sets.

Adding your MongoDB environment to MMS

Another thing beginners will not realise is that it's a good idea to monitor your services. At the very least you should install MongoDB Cloud Manager agents into your servers. Just set up an account at MongoDB Atlas.

It's beyond the point of this article to explain Atlas, but you should have no trouble installing the agents with the Group ID and API Key provided. Then install the monitoring agent and have the Manager figure out your existing deployment and replica set.

The basic subscription for monitoring will cost you an extra USD 39 a month, but believe me when I say it's worth to add all your MongoDB deployments under MMS.

MMS

Replica Set Graphs

Installing a multi-instance Node.js service

Another mistake beginners in the MEAN stack do is to start up a single Node.js instance. Yes, Node.js asynchronous I/O nature makes it "concurrent enough" in 1 single thread. But you do want to maximize the rented machine so you should spin up at least one Node.js instance for each CPU core. A 1 GB RAM machine has just 1 core, but you can spin up at least 2 instances.

First of all, we must set up the yourdomain.com (192.160.10.4 or 10.0.0.4 in the example) to support Node.js, let's do it:

sudo apt-get install -y npm curl graphicsmagick nginx git bcsudo npm install -g nsudo npm install -g foreversudo npm install -g forever-servicesudo n 0.10.40

If you haven't already, create a more restricted sudo user:

123	adduser rocket # add a password when askedusermod -aG sudo rocketsu - rocket # and always ssh in with `ssh rocket@yourdomain.com`

Within rocket's home directory let's install the Rocket.chat codebase:

cd /home/rocketcurl -L https://rocket.chat/releases/latest/download -o rocket.chat.tgztar zxvf rocket.chat.tgzmv bundle Rocket.Chatcd Rocket.Chat/programs/servernpm install

And now let's add a way for the machine to start the Rocket.chat whenever it reboots:

cd ~/Rocket.Chatsudo forever-service install -s main.js -e "ROOT_URL=https://rocketchat42.com/ MONGO_URL=mongodb://mongo1.yourdomain.com:27017/rocketchat MONGO_OPLOG_URL=mongodb://mongo1.yourdomain.com:27017/local PORT=3001" rocketchat1sudo forever-service install -s main.js -e "ROOT_URL=https://rocketchat42.com/ MONGO_URL=mongodb://mongo1.yourdomain.com:27017/rocketchat MONGO_OPLOG_URL=mongodb://mongo1.yourdomain.com:27017/local PORT=3002" rocketchat2sudo start rocketchat1sudo start rocketchat2

Let's also create SSL certificates (and for that you must have a properly registered domain, of course). Again, Digital Ocean's documentation on Let's Encrypt is very good. In summary, all you have to do is:

1	sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Now edit the /etc/nginx/sites-available/default and add this inside the server block:

server {  ...  location ~ /.well-known {          allow all;  }  ...}

Now you can:

sudo service nginx reloadsudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048cd /opt/letsencrypt./letsencrypt-auto certonly -a webroot --webroot-path=/usr/share/nginx/html -d yourdomain.com -d www.yourdomain.com

Again, replace yourdomain.com with your own registered domain, of course. The letsencrypt command will prompt you to enter your e-mail address and accept terms of service.

Let's Encrypt is a fantastic free SSL provider. Their certificates are short lived and are meant to expire in 90 days, so you must set up auto-renewal. Start sudo crontab -e and add the following lines:

12	30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log35 2 * * 1 /etc/init.d/nginx reload

This should take care of SSL. Now you can edit your /etc/nginx/sites-available/default and replace it with this template:

12345678910111213141516171819202122232425262728293031323334353637383940414243444546

# Upstreamsupstream backend {    server 127.0.0.1:3001;    server 127.0.0.1:3002;}server {  listen 443 ssl;  server_name yourdomain.com;  ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;  ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;  ssl_prefer_server_ciphers on;  ssl_dhparam /etc/ssl/certs/dhparam.pem;  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';  ssl_session_timeout 1d;  ssl_session_cache shared:SSL:50m;  ssl_stapling on;  ssl_stapling_verify on;  add_header Strict-Transport-Security max-age=15768000;  location / {    proxy_pass http://backend;    proxy_http_version 1.1;    proxy_set_header Upgrade $http_upgrade;    proxy_set_header Connection "upgrade";    proxy_set_header Host $http_host;    proxy_set_header X-Real-IP $remote_addr;    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;    proxy_set_header X-Forward-Proto http;    proxy_set_header X-Nginx-Proxy true;    proxy_redirect off;  }}server {  listen 80;  server_name yourdomain.com www.yourdomain.com;  return 301 https://$host$request_uri;}

Just for the sake of completeness, again: replace yourdomain.com for your domain. Usually the /etc/nginx/sites-enabled/default is a symlink to /etc/nginx/sites-available/default, just check that as well. This configuration will load balance between the 2 node.js instances we configured and started before. And you can add as many as you want following the same forever-install procedure and adding the new instances to the upstream section in the nginx configuration above.

If everything is correct, you can restart nginx again: sudo service nginx restart.

And now you should have https://yourdomain.com already up and running. The first user that sign up will be the site administrator and from there you can customize Rocket.chat internally.

Conclusion

As you can see, this is not a simple procedure to follow and I am assuming you have experience managing your own infrastructure. If you don't you definitely should NOT do this by yourself.

If you do it correctly, you should have a functional Slack-clone with minimal reliability (thanks in large part to Digital Ocean) and minimal cost (USD 80/month in the configuration I described that should be enough for more than a 100 users). Rocket.chat also offer more documentation for other environments including Docker configurations you might want to try. But the procedure above is sufficient for my needs.

I said it already but it's better to repeat it: don't fall for the trap of installing everything in a single box, with single instance MongoDB, and without proper monitoring. It's asking for trouble.

For now I am in the middle of the roll out. All developers in my company are already in the new deployment and soon half of the clients should also migrate (some will not be able to leave Slack just yet), but with Franz and the ability of full history and searchability this shoudn't be a concern even if a developer stays offline for some period of time.

I also don't advocate that everybody should be online and responding in real-time. It's unfeasible, unproductive, creates unnecessary tension. People should participate when they have free time, and they should be able to concentrate without worrying that they are missing something. That's why they should opt-out of being notified in the more busy channels and just enable notification on the private groups that matter.

Coincidentally Jason Fried just posted about concerns over group chats at Signal v. Noise, but the gist is that group chat should be purposeful not yet another tool to create tension. People should definitelly get offline when they need to fully concentrate in their work and have the opportunity to catch up with interesting conversations later. And really important communication should go through e-mail or other tradicional ways, a simple @all don't cut it for company-wide announcements for example.

I hope this exercise gives you more perspective on what you can have and also raise awareness on the need for companies to regain more control over their own data, particularly knowledge. Erase your communication channels and you're losing years-worth of knowledge that can be invaluable.