The Ultimate Guide to HTTP Requests in Ruby

If you’d like to get information from a website, or if you’d like to submit forms, upload files, etc.

You’ll need to send an HTTP request & then process the response.

In this article you’ll learn how to:

• Make a simple HTTP request using net/http
• Send SSL requests
• Submit data using a POST request
• Send custom headers
• Choose the best HTTP client for your situation

Let’s do this!

How to Send an HTTP Request

Ruby comes with a built-in http client, it’s called net/http & you can use it to send any kind of request you need.

Here’s a net/http example:

require 'net/http'

Net::HTTP.get('example.com', '/index.html')

This will return a string with the HTML content of the page.

But often you want more than the HTML content.

Like the HTTP response status.

Without the response status you don’t know if your request was successful, or if it failed.

This is how you get that:

response = Net::HTTP.get_response('example.com', '/')

response.code
# 200

Now if you want the response content you call the body method:

response.body

How to Use the HTTParty Gem

There are many gems that can make things easier for you.

One of these gems is httparty.

Here’s how to use it:

require 'httparty'

response = HTTParty.get('http://example.com')

response.code
# 200

response.body
# ...

The benefits of using an HTTP gem:

• It’s easier to use.
• There is no separate get_response method, get already gives you a response object.
• As you’ll see in the next section they make SSL request transparent

Sending SSL Requests

If you try to send an SSL request with net/http:

Net::HTTP.get_response("example.com", "/", 443)

You get:

Errno::ECONNRESET: Connection reset by peer

You’d have to do this instead:

net = Net::HTTP.new("example.com", 443)

net.use_ssl = true
net.get_response("/")

Save yourself some work & use a gem 🙂

How to Submit Data With a Post Request

A GET request is used to request information.

Like downloading an image, css, javascript…

But if you want to submit information use a POST request.

Here’s an example:

HTTParty.post("http://example.com/login", body: { user: "test@example.com", password: "chunky_bacon" })

To upload a file you’ll need a multipart request, which is not supported by HTTParty.

You can use the rest client gem:

require 'rest-client'

RestClient.post '/profile', file: File.new('photo.jpg', 'rb')

Or the Faraday gem:

require 'faraday'

conn =
Faraday.new do |f|
  f.request :multipart
  f.request :url_encoded

  f.adapter :net_http
end

file_io = Faraday::UploadIO.new('photo.jpg', 'image/jpeg')

conn.post('http://example.com/profile', file: file_io)

How to Send Custom HTTP Headers

You can send custom headers with an HTTP request.

This helps you send extra data with your request, including:

• Cookies
• User-agent
• Caching preferences
• Content-type
• Accept (to get JSON back, or something else)

Here’s how:

Faraday.new('http://example.com', headers: { 'User-Agent' => 'test' }).get

I’m creating a Faraday object (using new), then calling get on it.

Or you can do something like this:

Faraday.post(
  "https://example.com/login/oauth/access_token",
  URI.encode_www_form(
    code: params[:code],
    client_id: ENV["example_client_id"],
    client_secret: ENV["example_client_secret"],
  ),
  {
    "Content-Type" => "application/x-www-form-urlencoded",
    "Accept" => "application/json"
  }
)

Which works great for OAuth type of requests.

Choosing The Best Ruby HTTP Client

There are many HTTP clients available in Ruby.

But which one should you choose?

To help you make this decision I prepared a comparison table for you.

Here it is:

REPO	STARS	RECENT_COMMIT_COUNT	LAST_COMMIT	LATEST_RELEASE	CREATED_DATE
lostisland/faraday	4817	26	2020-07-09 09:43:09	2020-03-29 13:46:47	2009-12-10 17:14:55
rest-client/rest-client	4913	0	2019-08-25 21:41:17	2019-08-21 22:50:21	2009-12-07 19:34:29
typhoeus/typhoeus	3739	2	2020-05-17 19:33:54	2020-01-15 16:24:17	2009-02-18 23:14:50
jnunemaker/httparty	5220	11	2020-06-10 18:40:13	2020-06-10 18:40:13	2008-07-28 20:55:11
excon/excon	1018	14	2020-06-30 11:30:12	2020-06-17 12:07:02	2009-10-25 17:50:46
httprb/http	2623	13	2020-03-30 12:34:38	2020-02-26 17:33:37	2011-10-06 04:19:10

Looking at popularity & how well maintained these gems are is a good first step into evaluating all the options.

But what are the technical differences between them?

Many of these gems are wrappers around the net/http library:

• excon
• httparty
• rest client

While Faraday is an adapter gem, it can use different implementations.

And Typhoeus is a wrapper around the C libcurl library.

Typhoeus For Multi-Threading

Typhoeus supports concurrent requests without having to write any concurrent code.

This makes Typhoeus the best choice if you are looking to maximize your requests per second.

To use Typhoeus with multi-threading:

require 'typhoeus'

hydra   = Typhoeus::Hydra.hydra
request = Typhoeus::Request.new("https://www.rubyguides.com")

hydra.queue(request)
hydra.queue(request)

hydra.run

Faraday Gem For Adaptability

Faraday allows you to choose any implementation from net/http, to Typhoeus, or even rest-client all from the same interface.

It defaults to net/http, but you can change it like this:

Faraday.default_adapter = :typhoeus

Another interesting thing about Faraday is that it supports middleware.

Middleware is like a plugin that helps you modify the request or response in a specific way.

Example:

require 'faraday_middleware'

client =
  Faraday.new do |f|
    f.response :json
    f.adapter :net_http
  end

client.get('https://api.github.com/repos/vmg/redcarpet/issues')

This middleware will parse the response as JSON automatically.

If you want to send JSON instead of parsing it, use f.request :json instead of f.response :json.

Faraday Video

Bonus: How to Debug Your HTTP Requests

If you want to see exactly what your code is sending so that you can make sure it’s working correctly…

…then I have a nice little trick for you.

You can use this one-liner:

ruby -rsocket -e "trap('SIGINT') { exit }; Socket.tcp_server_loop(8080) { |s,_| puts s.readpartial(1024); puts; s.puts 'HTTP/1.1 200'; s.close }

This creates a server on port 8080 that prints everything it receives, returns an HTTP status code & then closes the connection.

You can send requests against localhost:8080 to see them.

Like this:

Faraday.new('http://localhost:8080', headers: { 'User-Agent' => 'test', 'foo' => '1234' }).get

# GET / HTTP/1.1
# User-Agent: test
# Foo: 1234
# Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
# Accept: */*
# Connection: close
# Host: localhost:8080

Other options include using a gem like httplog, or a tool like mitmproxy.

Summary

You learned how to send all kinds of HTTP request, starting from the basic GET to request information, to the POST request to submit information.

You have seen an overview of the most popular Ruby gems for working with HTTP requests. My favorite is faraday because of the flexibility it offers, but feel free to pick another gem if it fits your needs better.

If you enjoyed this post please share it with your friends so they can enjoy it too 🙂

Thanks for reading!