Using OpenSSL with memory BIOs
For a couple of projects I've been using SSL/TLS to secure data transport, but everytime when I start to use the openSSL library, it's tough to find the correct documentation. The library is poorly documented but when you have some experience with it I'm sure it all makes sense. When you don't have the experience it's hard to find the correct information and see the bigger picture.
I'm far from an expect on SSL/TLS or the openSSL library. Though after having it used for a couple of projects I'm getting more familiar and got a basic understanding. I'll try to explain how I use it. Please contact me at info@this_domain if you see any errors/mistakes.
How TLS works
The Transport Layer Security protocol is used to secure your data that you e.g. want to receive and send to a client over a network. For example the Twitter API is only accessible over HTTPS. This means that the HTTP protocol is encrypted using TLS. TLS is based on asymmetric and symetric keys.
asymetric keys use a public and private key to encrypt and decrypt data. When data is encrypted using a private key only users with the public key can decrypt the data. When data is encrypted with the public key, only users with the private key can decrypt the data. Encrypting or decrypting data with asymetric keys takes more time then symetric keys. (RSA)
when using symmetric keys one can encrypt and decrypt data with the same key. symmetric keys are often used to encrypt large amount of data because it's easier to compute then asymetric keys. (DES, 3DES, AES, RC4, RC2)
TLS uses asymetric keys to exchange a symmetric key. Once the symmetric key is available between sender and receiver they are used to encrypt/decrypt the large partion of application data.
TLS FLOW
TLS is a protocol which means that the flow of data is predefined. In most cases
the data flows as shown in the snippet below. Important to know is the handshake, which
kicks off the communication between a sender and receiver (e.g. browser and webserver).
The CertificateRequest
is an important aspect of TLS. A Certificate is used to
check if the keys are created by an authority which is trusted. Also a certificate
contains the public key. Also note that the handshake hops 4 times between client
and server.
Client Server ------ ------ (1) ClientHello --------> | ServerHello | Certificate* | ServerKeyExchange* | CertificateRequest* | (2) | <-------- ServerHelloDone | Certificate* | ClientKeyExchange |--- HANDSHAKE CertificateVerify* | [ChangeCipherSpec] | (3) | Finished --------> | [ChangeCipherSpec] | (4) | <-------- Finished | Application Data <-------> Application Data
BIOs
This part will describe how I've used openSSL in a experimental
application to secure data. In the code which will follow below I'm similating a client and
server in one application. The client starts the handshake by sending a ClientHello
to the
server.
OpenSSL uses the a concept of BIOs which is an input/output abstraction allowing us to
e.g. use data from a file, memory or network in a similar way. Because I want to manage
my own memory I'm using a memory bio. These BIOs always tend to confuse me because
of the BIO_read(), BIO_write(), SSL_read(), SSL_write()
functions and the BIO objects
you use with them. To make it a bit easier and less confusing with all these read(), write()
functions I name my BIOs not "read" and "write" BIO, but input bio
and output bio
. Somehow
this makes more sense to me. For example, I use an input BIO to store data that comes from
the network and then I use SSL_read()
to retrieve the unencrypted data.
SSL_read()
read unencrypted data which is stored in the input BIO.SSL_write()
write unencrypted data into the output BIO.BIO_write()
write encrypted data into the input BIO.BIO_read()
read encrypted data from the output BIO.
So when do you use what function?
- Use
BIO_write()
to store encrypted data you receive from e.g. a tcp/udp socket. Once you've written to an input BIO, you useSSL_read()
to get the unencrypted data, but only after the handshake is ready. - Use
BIO_read()
to check if there is any data in the output BIO. The output BIO will be filled by openSSL when it's handling the handshake or when you callSSL_write()
. When there is data in your output BIO, use BIO_read to get the data and send it to e.g. a client. UseBIO_ctrl_pending()
to check how many bytes there are stored in the output bio. Seekrx_ssl_handle_traffic()
in the code listing at the bottom of this post.
Using OpenSSL
Before you can use anything of the OpenSSL library you need to initialize it:
SSL_library_init(); SSL_load_error_strings(); ERR_load_BIO_strings(); OpenSSL_add_all_algorithms();
When you're ready you can shutdown the library by using:
ERR_remove_state(0); ENGINE_cleanup(); CONF_modules_unload(1); ERR_free_strings(); EVP_cleanup(); sk_SSL_COMP_free(SSL_COMP_get_compression_methods()); CRYPTO_cleanup_all_ex_data();
To use openSSL you create a SSL_CTX
object which is a context that
keeps track of shared information like the certificate to use, private
key to use etc.. See the krx_ssl_ctx_init()
function in the souce code
at the bottom of this document.
Once you have a SSL_CTX
object you can create a SSL
object for a
particulair connection. You can create many SSL
objects for each connection
you want to handle and share one SSL_CTX
object.
When you're using openSSL as a server you need to make sure that the
SSL object knows that it should act as a server which can be done by calling
SSL_set_accept_state(ssl)
. When SSL is used as a server it will wait for a
ClientHello
message and makes sure that a ServerHello, CertificateRequest etc..
is send back to the client. When you want your SSL object to act like a client
you need to call SSL_set_connect_state(ssl)
and call start filling your output
BIO by calling SSL_do_handshake()
.
As long as the handshake is not yet ready, you have to make sure
that the function SSL_do_handshake()
is called everytime the
input or output BIOs change. For example, when the server starts,
and a clients connects, you call BIO_write() and fill the input
BIO with data you got on your socket, then call
SSL_do_handshake(ssl)
as long as the handshake isn't ready
yet. You can check this by using SSL_is_init_finished(ssl)
.
After the handshake has been completed you can start filling the
BIOs by calling SSL_read and SSL_write with your application data
that you want to secure.
Update: tracking state
Some people have told me you don't need to track state of SSL. Though in some situations
it's good to know when the handshake (initialisation) has finished. OpenSSL provides a
function SSL_is_init_finished()
to check this. As long as SSL_is_init_finished()
returns
false you should handle the handshake, which is a 4 step procedure as described above
(see the flow example). When SSL_is_init_finished()
returns false you need to call
SSL_do_handshake(). OpenSSL
keeps state internally and calling SSL_do_handshake()
should be enough to allow openSSL
to perform the handshake. But, it's important that you check the return value of the
call to SSL_do_handshake()
because it tells you want functions you need to call next,
especially when using memory bios. As described in the manual: "if the underlying BIO is
non-blocking, SSL_do_handshake() will also return when the underlying
BIO could not satisfy the needs of SSL_do_handshake() to continue the
handshake. In this case a call to SSL_get_error() with the return
value of SSL_do_handshake() will yield SSL_ERROR_WANT_READ or
SSL_ERROR_WANT_WRITE." So A very verbose way of handling this could be something
like:
if (!SSL_is_init_finished(ssl)) { /* NOT INITIALISED */ r = SSL_do_handshake(ssl); if (r < 0) { r = SSL_get_error(ssl, r); if (SSL_ERROR_WANT_READ == r) { pending = BIO_ctrl_pending(out_bio); if (pending > 0) { read = BIO_read(out_bio, buffer, POLY_SSL_BUFFER_SIZE); if (read > 0) { listener->onSslBufferEncryptedData(buffer, read); } } } } } else { /* SSL IS INITIALISED */ pending = BIO_ctrl_pending(out_bio); if (pending > 0) { read = BIO_read(out_bio, buffer, POLY_SSL_BUFFER_SIZE); if (read > 0) { listener->onSslBufferEncryptedData(buffer, read); } } pending = BIO_ctrl_pending(in_bio); if (pending > 0) { while((read = SSL_read(ssl, buffer, POLY_SSL_BUFFER_SIZE)) > 0) { listener->onSslBufferDecryptedData(buffer, read); } } } }
If you implement your own socket functions you need to make sure that the SSL handshake has been finished before your start enrypting your application data. A function to check if you need to read more data from your socket is:
int SslBuffer::wantsMoreData() { int r = 0; if (SSL_is_init_finished(ssl)) { return -1; } r = SSL_do_handshake(ssl); if (r < 0) { r = SSL_get_error(ssl, r); if (SSL_ERROR_WANT_READ == r) { return 0; } } return -1; }
Example code
The following code listing shows how you can use OpenSSL with Memory BIOs.
/* Create server/client self-signed certificate/key (self signed, DONT ADD PASSWORD) openssl req -x509 -newkey rsa:2048 -days 3650 -nodes -keyout client-key.pem -out client-cert.pem openssl req -x509 -newkey rsa:2048 -days 3650 -nodes -keyout server-key.pem -out server-cert.pem */ #include <stdio.h> #include <stdlib.h> #include <openssl/err.h> #include <openssl/dh.h> #include <openssl/ssl.h> #include <openssl/conf.h> #include <openssl/engine.h> /* SSL debug */ #define SSL_WHERE_INFO(ssl, w, flag, msg) { \ if(w & flag) { \ printf("+ %s: ", name); \ printf("%20.20s", msg); \ printf(" - %30.30s ", SSL_state_string_long(ssl)); \ printf(" - %5.10s ", SSL_state_string(ssl)); \ printf("\n"); \ } \ } typedef void(*info_callback)(); typedef struct { SSL_CTX* ctx; /* main ssl context */ SSL* ssl; /* the SSL* which represents a "connection" */ BIO* in_bio; /* we use memory read bios */ BIO* out_bio; /* we use memory write bios */ char name[512]; } krx; void krx_begin(); /* initialize SSL */ void krx_end(); /* shutdown SSL */ int krx_ssl_ctx_init(krx* k, const char* keyname); /* initialize the SSL_CTX */ int krx_ssl_init(krx* k, int isserver, info_callback cb); /* init the SSL* (the "connection"). we use the `isserver` to tell SSL that it should either use the server or client protocol */ int krx_ssl_shutdown(krx* k); /* cleanup SSL allocated mem */ int krx_ssl_verify_peer(int ok, X509_STORE_CTX* ctx); /* we set the SSL_VERIFY_PEER option on the SSL_CTX, so that the server will request the client certificate. We can use the certificate to get/verify the fingerprint */ int krx_ssl_handle_traffic(krx* from, krx* to); /* some debug info */ void krx_ssl_server_info_callback(const SSL* ssl, int where, int ret); /* purely for debug purposes; logs server info. */ void krx_ssl_client_info_callback(const SSL* ssl, int where, int ret); /* client info callback */ void krx_ssl_info_callback(const SSL* ssl, int where, int ret, const char* name); /* generic info callback */ int main() { /* startup SSL */ krx_begin(); /* create client/server objects */ krx server; krx client; /* init server. */ if(krx_ssl_ctx_init(&server, "server") < 0) { exit(EXIT_FAILURE); } if(krx_ssl_init(&server, 1, krx_ssl_server_info_callback) < 0) { exit(EXIT_FAILURE); } printf("+ Initialized server.\n"); /* init client. */ if(krx_ssl_ctx_init(&client, "client") < 0) { exit(EXIT_FAILURE); } if(krx_ssl_init(&client, 0, krx_ssl_client_info_callback) < 0) { exit(EXIT_FAILURE); } printf("+ Initialized client.\n"); /* kickoff handshake; initiated by client (e.g. browser) */ SSL_do_handshake(client.ssl); krx_ssl_handle_traffic(&client, &server); krx_ssl_handle_traffic(&server, &client); krx_ssl_handle_traffic(&client, &server); krx_ssl_handle_traffic(&server, &client); /* encrypt some data and send it to the client */ char buf[521] = { 0 } ; sprintf(buf, "%s", "Hello world"); SSL_write(server.ssl, buf, sizeof(buf)); krx_ssl_handle_traffic(&server, &client); krx_ssl_shutdown(&server); krx_ssl_shutdown(&client); krx_end(); return EXIT_SUCCESS; } void krx_begin() { SSL_library_init(); SSL_load_error_strings(); ERR_load_BIO_strings(); OpenSSL_add_all_algorithms(); } void krx_end() { ERR_remove_state(0); ENGINE_cleanup(); CONF_modules_unload(1); ERR_free_strings(); EVP_cleanup(); sk_SSL_COMP_free(SSL_COMP_get_compression_methods()); CRYPTO_cleanup_all_ex_data(); } int krx_ssl_ctx_init(krx* k, const char* keyname) { int r = 0; /* create a new context using DTLS */ k->ctx = SSL_CTX_new(DTLSv1_method()); if(!k->ctx) { printf("Error: cannot create SSL_CTX.\n"); ERR_print_errors_fp(stderr); return -1; } /* set our supported ciphers */ r = SSL_CTX_set_cipher_list(k->ctx, "ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH"); if(r != 1) { printf("Error: cannot set the cipher list.\n"); ERR_print_errors_fp(stderr); return -2; } /* the client doesn't have to send it's certificate */ SSL_CTX_set_verify(k->ctx, SSL_VERIFY_PEER, krx_ssl_verify_peer); /* enable srtp */ r = SSL_CTX_set_tlsext_use_srtp(k->ctx, "SRTP_AES128_CM_SHA1_80"); if(r != 0) { printf("Error: cannot setup srtp.\n"); ERR_print_errors_fp(stderr); return -3; } /* load key and certificate */ char certfile[1024]; char keyfile[1024]; sprintf(certfile, "./%s-cert.pem", keyname); sprintf(keyfile, "./%s-key.pem", keyname); /* certificate file; contains also the public key */ r = SSL_CTX_use_certificate_file(k->ctx, certfile, SSL_FILETYPE_PEM); if(r != 1) { printf("Error: cannot load certificate file.\n"); ERR_print_errors_fp(stderr); return -4; } /* load private key */ r = SSL_CTX_use_PrivateKey_file(k->ctx, keyfile, SSL_FILETYPE_PEM); if(r != 1) { printf("Error: cannot load private key file.\n"); ERR_print_errors_fp(stderr); return -5; } /* check if the private key is valid */ r = SSL_CTX_check_private_key(k->ctx); if(r != 1) { printf("Error: checking the private key failed. \n"); ERR_print_errors_fp(stderr); return -6; } sprintf(k->name, "+ %s", keyname); return 0; } int krx_ssl_verify_peer(int ok, X509_STORE_CTX* ctx) { return 1; } /* this sets up the SSL* */ int krx_ssl_init(krx* k, int isserver, info_callback cb) { /* create SSL* */ k->ssl = SSL_new(k->ctx); if(!k->ssl) { printf("Error: cannot create new SSL*.\n"); return -1; } /* info callback */ SSL_set_info_callback(k->ssl, cb); /* bios */ k->in_bio = BIO_new(BIO_s_mem()); if(k->in_bio == NULL) { printf("Error: cannot allocate read bio.\n"); return -2; } BIO_set_mem_eof_return(k->in_bio, -1); /* see: https://www.openssl.org/docs/crypto/BIO_s_mem.html */ k->out_bio = BIO_new(BIO_s_mem()); if(k->out_bio == NULL) { printf("Error: cannot allocate write bio.\n"); return -3; } BIO_set_mem_eof_return(k->out_bio, -1); /* see: https://www.openssl.org/docs/crypto/BIO_s_mem.html */ SSL_set_bio(k->ssl, k->in_bio, k->out_bio); /* either use the server or client part of the protocol */ if(isserver == 1) { SSL_set_accept_state(k->ssl); } else { SSL_set_connect_state(k->ssl); } return 0; } void krx_ssl_server_info_callback(const SSL* ssl, int where, int ret) { krx_ssl_info_callback(ssl, where, ret, "server"); } void krx_ssl_client_info_callback(const SSL* ssl, int where, int ret) { krx_ssl_info_callback(ssl, where, ret, "client"); } void krx_ssl_info_callback(const SSL* ssl, int where, int ret, const char* name) { if(ret == 0) { printf("-- krx_ssl_info_callback: error occured.\n"); return; } SSL_WHERE_INFO(ssl, where, SSL_CB_LOOP, "LOOP"); SSL_WHERE_INFO(ssl, where, SSL_CB_HANDSHAKE_START, "HANDSHAKE START"); SSL_WHERE_INFO(ssl, where, SSL_CB_HANDSHAKE_DONE, "HANDSHAKE DONE"); } int krx_ssl_handle_traffic(krx* from, krx* to) { // Did SSL write something into the out buffer char outbuf[4096]; int written = 0; int read = 0; int pending = BIO_ctrl_pending(from->out_bio); if(pending > 0) { read = BIO_read(from->out_bio, outbuf, sizeof(outbuf)); } printf("%s Pending %d, and read: %d\n", from->name, pending, read); if(read > 0) { written = BIO_write(to->in_bio, outbuf, read); } if(written > 0) { if(!SSL_is_init_finished(to->ssl)) { SSL_do_handshake(to->ssl); } else { read = SSL_read(to->ssl, outbuf, sizeof(outbuf)); printf("%s read: %s\n", to->name, outbuf); } } return 0; } int krx_ssl_shutdown(krx* k) { if(!k) { return -1; } if(k->ctx) { SSL_CTX_free(k->ctx); k->ctx = NULL; } if(k->ssl) { SSL_free(k->ssl); k->ssl = NULL; } return 0; }