How do I write microservices in Ruby?

"Editor's note" the author of this article is Pierpaolo Frasa, which describes in detail the various aspects of writing microservices in Ruby in a detailed case. Department of Domestic ITOM management platform OneAPM compiled rendering.

Recently, we all think that we should adopt a micro-service architecture. But how many tutorials do you have? Let's take a look at this article about writing microservices in Ruby.

Everyone is talking about microservices, but I haven't seen a few decent tutorials on using Ruby to write MicroServices. This may be because many Ruby developers still prefer the rails architecture (there's nothing wrong with rails itself, but there's a lot more that Ruby can do.) ）

So I came up with a piece of power. Let's start by looking at how to write and deploy MicroServices in Ruby.

Imagine this scenario: we need to write a micro-service whose responsibility is to send an email. It received the following information:

{  
  ‘provider‘: ‘mandrill‘,  
  ‘template‘: ‘invoice‘,  
  ‘from‘: ‘support@company.com‘,  
  ‘to‘: ‘user@example.com‘,  
  ‘replacements‘: {    
  ‘salutation‘: ‘Jack‘,    
  ‘year‘: ‘2016‘
  }
}

Its task is to replace some of the variables in the template, and then send the invoice email to [email protected]. (We use Mandrill as a provider of email APIs, and sadly, Mandrill will stop the service.) ）

This example is great for using microservices because it is small and focuses on a single function point, and the interface is clearly defined. So when we decide to rewrite our messaging infrastructure at work, we do that.

If we have a micro-service, we need to find a way to send some information to it. That is, the method of passing Message Queuing. There are many optional messaging systems that you can choose from. We have chosen RABBITMQ here, because:

• It's popular, and it's coded according to the standard (AMQP).

• It is already bound to multiple languages and is therefore ideal for multi-lingual environments. I like to use Ruby to write apps (and I think it's better than other languages), but I don't think Ruby applies to all of the problems at the moment, and I don't think it will be like that in the future. Therefore, we may also need to write a mail-sending application with elixir (it will not be very difficult to write).

• It is flexible enough to accommodate a variety of workflows – it can be adapted to a simple workflow that handles Message Queuing in the background (this is the focus of this article), and can also accommodate complex message exchange workflows (even RPC). There are many examples on the website.

• This panel is useful for accessing its admin panel from a browser.

• It has many managed solutions (you can find resources in your favorite Package manager for development).

• It was written in Erlang, and Erlang's handlers handled the concurrency problem very well.

It's very easy to put messages in a queue with RABBITMQ, like this:

require ‘bunny‘
require ‘json‘

connection = Bunny.new
connection.start
channel = connection.create_channel
queue = channel.queue ‘mails‘, durable: true

json = { ... }.to_json
queue.publish json
connection.close

bunnyis the standard gem of RABBITMQ, when we do not pass any item toBunny.newit, it will assume that RABBITMQ has a standard certificate that islocalhost:5672running on it. Then we (through a series of settings) connect to a message queue named "mails". If the queue does not already exist, the system will create the queue, and if it does, the system will connect directly. We can then post any messages directly to this queue (for example, the invoice message above us). Here we use JSON, but in fact, you can use any format you like (BSON, Protocol buffers, or whatever), rabbitmq don't care.

Now that we've solved the producer side, we still need an app to accept and process the message. We are using the snearkers. Sneakers is a compression gem around the RABBITMQ. If you want to do some background processing, it will expose you to the subset of RABBITMQ you are most likely to use, but the bottom is still rabbitmq. With sneakers (sneakers is inspired by Sidekiq), we can set up a "worker" to process our message sending request:

require ‘sneakers‘
require ‘json‘
require ‘mandrill_api/provider‘

class Mailer
  include Sneakers::Worker
  from_queue ‘mails‘

  def work(message)
    puts "RECEIVED: #{message}"
    option = JSON.parse(message)
    MandrillApi::Provider.new.deliver(options)
    ack!
  end
end

We have to explicitly read the message from which queue (that is, "mails"), and the method of consume messagework, we first parse the message (previously we have said in JSON format – but again, you can choose any format, RABBITMQ or sneakers do not care about format issues). We then scattered the message biographies to some of the actual working classes inside. Finally, we must notify the system that the message has been received, otherwise RABBITMQ will put the message back in the queue. If you want to reject a message, or do something else, there is a method in the Snearkers wiki. In order to master the situation, we also added a log function in it (we will explain later why the log is standard output).

But a program cannot have only one class. So we need to build a project structure – it's a bit strange for rails developers, because usually we just need to runrails newand everything is set up. Here I would like to expand a bit more. It's almost like this when our project tree is finished:

.
├── Gemfile
├── Gemfile.lock
├── Procfile
├── README.md
├── bin
│   └── mailer
├── config
│   ├── deploy/...
│   ├── deploy.rb
│   ├── settings.yml
│   └── setup.rb
├── examples
│   └── mail.rb
├── lib
│   ├── mailer.rb
│   └── mandrill_api/...
└── spec
    ├── acceptance/...
    ├── acceptance_helper.rb
    ├── lib/...
    └── spec_helper.rb

Part of this is self-explanatory, such as theGemfile(\.lock)?Readme. We do not have to explain the spec folder too much, just know, as usual we have two helper files in this directory, one (spec_helper.rb) for quick unit testing, and anotheracceptance_helper.rbfor acceptance testing. Acceptance testing requires setting up more things (for example, simulating a real HTTP request).libfolders are also less relevant to our theme, and we can see that there is onelib/mailer.rb(this is the worker class we defined above), and the rest of the file is dedicated to personalized service. Theexamples/mail.rbfile is the formation code for the sample message, as in the previous paragraph. We can use it to initiate manual testing at any time. Now I'd like to focus on theconfig/setup.rbpaper. This is the file that we usually load at the beginning (even if it is inspec_helper.rb). So we don't need it to do too many things (or your test will get very slow). In our case, it is this:

require ‘bundler/setup‘

lib_path = File.expand_path ‘../../lib‘, __FILE__
$LOAD_PATH.unshift lib_path

ENVIRONMENT = ENV[‘ENVIRONMENT‘] || ‘development‘

require ‘yaml‘
settings_file = File.expand_path ‘../settings.yml‘, __FILE__
SETTINGS = YAML.load_file(settings_file)[ENVIRONMENT]

if %w(development test).include? ENVIRONMENT
  require ‘byebug‘
end

The most important thing here is to set the load path. First, we introducebundler/setup, so that we can introduce each gem by the name of the gem. Next, we add the service Lib folder to the load path. This means that we can do a lot of things, such as introducingmandrill_api/provider, which can be<project_root>/ lib/mandrill_api/providerfound from within. The reason we do this is because we don't like the relative path. Please note that we do not use automatic loading in rails. We also did not callBundler.require, as this would introduce all the gems in the Gemfile. This means that you have to explicitly call the dependencies (Gems or LIB files) you need (which I think is good).

In addition, I like the multi-environment of rails. In the example above, we are loading through the UNIX environment variableENVIRONMENT. We also need to make some settings (such as the RABBITMQ connection option, or the key for some of the APIs that our service uses). These should depend on the environment, so we loaded a Yaml file and turned it into a global variable.

Finally, such code ensures that, as soon as it is introduced in the development and testing process, you can add Byebug (the Debug tool for Ruby 2.x) at any time. If you're worried about speed (it does take some time), you can take it off, put it in when you need it, or add a monkey patch:

if %w(development test).include? ENVIRONMENT
  class Object
    def byebug
      require ‘byebug‘
      super
    end
  end
end

Now we have a worker class, and an approximate project structure. We just need to notify sneakers to run the worker, which is what we did in thebin/mailer:

#!/usr/bin/env ruby
require_relative ‘../config/setup‘
require ‘sneakers/runner‘
require ‘logger‘
require ‘mailer‘
require ‘httplog‘

Sneakers.configure(
  amqp: SETTINGS[‘amqp_url‘],
  daemonize: false,
  log: STDOUT
)
Sneakers.logger.level = Logger::INFO
Httplog.options[:log_headers] = true

Sneakers::Runner.new([Mailer]).run

Note that this is executable (look at the beginning of the #!), so we do not needrubyto command, can run directly. First, we load the settings file (where we have to use a relative path) and then load the other needed things, including our mail worker class.

The important thing here is to configure the sneakers: theamqpparameter will accept a URL for the RABBITMQ connection, which can be loaded from the settings. We can notify sneakers to run in the foreground and record the log as standard output. Next, we give sneakers an array of worker classes, let sneakers run this array. We also need a library with logs so that we can observe the situation dynamically. The Httplog Gem records all outgoing requests, which is useful for communicating with external APIs (where we also let it record HTTP headers, but this is not the default setting).

Now runbin/mailer, it will turn out like this:

... WARN: Loading runner configuration...
... INFO: New configuration:
#<Sneakers::Configuration:0x007f96229f5f28 ...>
... INFO: Heartbeat interval used (in seconds): 2

But the actual output is actually much more verbose!

If you let it run and then run our formation script in another terminal window, we will get the following result:

... RECEIVED: {"provider":"mandrill","template":"invoice", ...}
D, ... [httplog] Sending: POST
https://mandrillapp.com:443/api/1.0/messages/send-template.json
D, ... [httplog] Data: {"template_name":"invoice", ...}
D, ... [httplog] Connecting: mandrillapp.com:443
D, ... [httplog] Status: 200
D, ... [httplog] Response:
[{"email":"user@example.com","status":"sent", ...}]
D, ... [httplog] Benchmark: 1.698229061003076 seconds

(This is also a simplified version!) ）

The amount of information here is quite large, especially in the beginning, and, of course, you can remove some of the logs as needed afterwards.

The above gives the basic structure of the project, but also what to do? Well, there's one more difficult part: deployment.

There are a number of things to keep in mind when deploying microservices (or, in general, any application), including:

• You will want to make it a daemon (that is, let it run in the background). We can do this when we set up sneakers, but I prefer not to do that-in the process of development, I want to see the log output and can use itCTRL+Cto kill the process.

• You'll want a reasonable diary. What is reasonable is to make sure that the log file does not fill the hard disk at last, or that it is so huge that it takes a lifetime to retrieve it (for example, a circular log).

• You'll want to restart the server for some reason, or if the program crashes.

• You will want to have some standardized commands that you can use to start/stop/restart programs when you need them.

You can do this on your own in Ruby, but I think there's a better solution: use something out of the box to handle these tasks-your operating system (Sidekiq's creator, Mike Perhammm, agrees with me). For us, this is meant to be usedsystemdbecause this is the program that runs on our servers (and most of today's Linux systems), but I don't want to spark a war of words here. Upstart or daemontools may also be possible.

"There are a lot of things you have to think about when deploying microservices. "From @tainnor
Click to tweet

To run our microservices with SYSTEMD, we need to create some configuration files. This can be done by hand, but I prefer to use a tool called Foreman. With foreman, we can specify all the processes that need to beProcfilerun in:

mailer: bin/mailer

Here we have only one process, but you can specify more than one. We have specified a process called "Mailer", which will runbin/mailerthe executable file. Benefits of Foreman now, it can export this configuration file to many initialization systems, including SYSTEMD. For example, from this simple procfile, it can create a lot of files; As I said earlier, we can specify multiple processes in the profile, and multiple such files can specify a dependency level. The top of the hierarchy is amailer.targetfile that relies on amailer-mailer.targetfile (and if there are multiple processes in our procfile, itmailer.targetwill depend on multiple child target files).mailer-mailer.targetfiles are also dependent onmailer-mailer-1.service(this type of file can have multiple, we only need to explicitly set the value of the thread concurrency is greater than 1). The final file looks like this:

[Unit]
PartOf=-.target

[Service]
User=mailer_user
WorkingDirectory=/var/www/mailer_production/releases/16
Environment=PORT=5000
Environment=PATH=
/home/deploy/.rvm/gems/ruby-2.2.3/gems/bundler-1.11.2:...
Environment=ENVIRONMENT=production
ExecStart=/bin/bash -lc ‘bin/mailer‘
Restart=always
StandardInput=null
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%n
KillMode=process

The specifics are not important. However, as can be seen from the code above, we have identified the user, the work path, the command to start running the service, and it is clear that each failure should be restarted, as well as logging and added to the system log. We have also set some environment variables, including path. I'll talk about this later.

With this, the system behavior we wanted before was realized. It can now run in the background and will restart every time it encounters a failure. You can also runsudo systemctl enable mailer.targetit so that it starts running when the system starts. As for the standard output log, it is re-written to the system log. For Systemd, that isjournald, a binary logger (so the problem of dumps no longer exists). We can check our log output in the following ways:

$ sudo journalctl -xu mailer-mailer-1.service
-- Logs begin at Thu 2015-12-24 01:59:54 CET, end at ... --
Feb 23 10:00:07 ... RECEIVED: {"from": ...}
...

You can givejournalctlmore options, for example, to filter by date.

In order for Foreman to generate the Systemd file, we must set the export process in the deployment. I don't know if you've used Capistrano 2 or Capistrano 3 or something like that (such as Mina). Below you will see the shell commands you may need. The hardest part of the task is how to set the environment variables correctly. To make sure that Foreman can write out just the variables in the startup script, we can run the following code from the deployed project Root to put them in a.envfile first:

$ echo "PATH=$(bundle show bundler):$PATH" >> .env
$ echo "ENVIRONMENT=production" >> .env

(I omitted the port variable here-this variable is automatically generated by foreman.) We don't need it for our service either. ）

We then told Foreman to export the variables of the file we just created.envto Systemd.

$ sudo -E env "PATH=$PATH" bundle exec foreman  export systemd /etc/systemd/system  -a mailer -u mailer_user -e .env

This command is very long, but in the end it is runningforeman export systemd, specifying the directory to which the file should be placed (as far as I know it/etc/systemd/systemis the standard directory), the user running the command, and the environment in which the file is loaded.

Then we reload all the things:

$ sudo systemctl daemon-reload$ sudo systemctl reload-or-restart mailer.target

Next, we enable the service to keep it running after the server starts:

$ sudo systemctl enable mailer.target

After that, our service can be started and kept running on the server and ready to accept all the messages sent.

I have covered many aspects of this article, but I would like to let you see the panorama behind writing and deploying MicroServices. Obviously, if you really want to master these things yourself, you have to go into it. But I think I've told you what techniques can be studied.

We wrote a similar mail service a few months ago and so far we have been very satisfied with the results. The mail service is relatively independent, has a well-defined API, and is rigorously tested independently, so we believe it will meet our expectations. and its sound restart mechanism is like a trade fuse for us-some sidekiq work programs occasionally bug, so we have to add monit to solve the problem-can fully use the operating system comes with the tool, feel great.

This article is compiled and collated by OneAPM engineers. OneAPM can provide you with an end-to-end ruby application performance solution, and we support all common Ruby frameworks and application servers to help you quickly discover system bottlenecks and pinpoint the root cause of the anomalies. Minute-level deployment, instant experience, Ruby monitoring has never been easier. To read more technical articles, please visit the OneAPM Official technology blog.

This article was transferred from OneAPM official blog

Original address: Https://dzone.com/articles/writing-a-microservice-in-ruby

How do I write microservices in Ruby?