CIDR Notation Explained Simply

A range of IP addresses can be expressed very simply with CIDR notation. I don’t want to discuss why classless inter-domain routing was developed, but only demonstrate how you can figure out what the CIDR notation for a range of IP addresses, or determine what the range of IP addresses are represented by a CIDR notation.

How to write an IP address as four octets

Consider the following IP address: 192.168.0.1

The IP address (IPv4 is only discussed in this CIDR tutorial) is made up of four octets. The first one is 192 which in binary notation would be 1100 0000 and I’ve written here with a space to make reading easier. In binary each 1 represents a value depending on which bit in the byte it is. Here are the values from left to right:

128 64 32 16 8 4 2 1

So 1100 0000 in binary is equal to 128 + 64 in decimal, or 192. If we write out the binary representation for each octet:

192.168.0.1

1100 0000 – 1010 1000 – 0000 0000 – 0000 0001

CIDR notation

CIDR notation looks like an IP address followed by a slash with a number, for example:

192.168.0.0/24

The number after the slash is the bit mask for the network. Simply put, it tells us how many bits are the same for each IP on the subnet. This also tells us which parts of the IP addresses can vary, and that gives us the range.

192.168.0.0/24 indicates that the first 24 bits are all the same on this range of IP addresses. Lets look at the first 24 bits:

1100 0000 – 1010 1000 – 0000 0000

That is the first 24 bits of 192.168.0.0. This means the remaining 8 bits can be either 0 or 1, but on this range of IP addresses the first 24 bits will always be the same.

If we set the part that can be 0 or 1 to all 1’s we can get the maximum IP value on the range:

1100 0000 – 1010 1000 – 0000 0000 – 1111 1111

All 1’s is 255, so the maximum IP on this range is 192.168.0.255.

The CIDR notation for the range of IP addresses from 192.168.0.0 thru 192.168.0.255 is thus 192.168.0.0/24.

Here is another range of IP addresses: 128.1.0.1 thru 191.255.255.254. How can this range of IP addresses be written in CIDR notation?

128.1.0.1 can be written in binary as

1000 0000 – 0000 0001 – 0000 0000 – 0000 0001

191.255.255.254 can be written as

1011 1111 – 1111 1111 – 1111 1111 – 1111 1110

We can see that the bits that are the same in this range, starting with the left most octet are

10

So there are only 2 bits in the mask, making the CIDR notation:

128.1.0.1/2

Here is another example of calculating the CIDR notation for a range of IP addresses:

69.4.128.0 – 69.4.159.255

Writing each IP in binary form we have

0100 0101 – 0000 0100 – 1000 0000 – 0000 0000

0100 0101 – 0000 0100 – 1001 1111 – 1111 1111

So we can see the bit mask, which is the left most set of bits that are the same are

0100 0101 – 0000 0100 – 100

which is 19 bits, so this range of IP addressed can be expressed in CIDR notation as

69.4.128.0/19

Hopefully this CIDR notation tutorial can help you quickly determine the CIDR notation for a given range of IP addresses, or how to determine the range if IP addresses represented by CIDR notation.

Bash one-liners

Get a list of URLs for a website using wget:

wget -r -nv -nd –delete-after “your-url-goes-here” 2>&1 | sed -n ‘/->/{/URL:/s/.*URL:\([^ ]*\).*/\1/;/^[ ]*$/d;p}’

Capture Fatal PHP Errors for Logging

When code runs without a user interface, for example a script invoked by cron, there often is not an easy way to see what errors are occuring. Another example would be php invoking another php script using exec(), for example. Here are some common fatal errors: unknown class referenced, unknown method called and parse error. Forgot a semicolon? Thats a parse error.

This post combines ideas others have used and gives an example of how to capture and log fatal php errors.

The php function set_error_handler() allows you to create a callback for handling errors. However, you cannot catch fatal errors using a callback registered with set_error_handler().

You can register a callback for php to invoke at the very end of execution via register_shutdown_function(). There is a way to see if there were any errors during script execution by calling error_get_last().

The problem is, how do we call register_shutdown_function()? If the script you want to monitor for errors has en error, like a parse error, and you put the register_shutdown_function() call in that script it will not work, because of the parse error. And using “include” or “require” will not work either because that essentially is still the same script, as far as php is concerned.

You can however use an INI file to tell php to automatically prepend a php file before invoking the script you want to monitor. By combining this with the log4php as the logging library you can create a very powerful error handler. You can read my log4php how-to if you are not familiar with the log4php library.

Imagine this is the script to be called by cron, it has a parse error. The code is saved in a file called mycron.php.

<?php
$foo = ;
?>

I put mycron.php in its own directory: /home/tom/bin.

In that same directory create php.ini telling php what files to automatically prepand and append:

; php.ini contents:
auto_prepend_file = /home/tom/bin/runtime_start.php

The contents of runtime_start.php is:

<?php

require_once 'log4php/Logger.php';

Logger::configure('logconfig.xml');

class Runtime
{
    function Runtime()
    {
        register_shutdown_function(array($this, 'shutdown'));
    }

    function shutdown()
    {
        $logger = Logger::getLogger('root');

        $e = error_get_last();

        if(is_null($e)) {
	    $logger->debug('Script ended normally');
	} else {
            $logger->error($e['message']);
        }
    }

    function finish()
    {
        $logger = Logger::getLogger('root');
        $logger->debug('Script ended normally');
    }
}

$runtime = new Runtime();
?>

One of the nice things about this approach is that the script I’m executing does not have to be altered in any way to add the error capturing and logging. You do however need to invoke it in a particular way. You have to tell php where to find the INI file:

php -c /home/tom/bin -f /home/tom/bin/mycron.php

You can get the full list of php command line invocation options using “php –help”.

When run the log file will contain a message like the following:

2011-09-15 11:08:01 PDT [ERROR] root: syntax error, unexpected ‘:’

Burn CDs from the Command Line

To burn cd images from the command line in Debian squeeze you need two packages:

(1) genisoimage
(2) wodim

To make an ISO image of a directory simply type:

genisoimage -o outfile_name some_directory

To burn the image use wodim. First figure out which optical drive to use by asking wodim for a list:


tom@osmium:~$ wodim -devices
tom@osmium:~$ wodim -devices
wodim: Overview of accessible drives (1 found) :
-------------------------------------------------------------------------
0 dev='/dev/scd0' rwrw-- : 'HP' 'CD-Writer+ 8000'
-------------------------------------------------------------------------

We need the device as in the /dev device so we can use it to tell wodim which device to use:

wodim dev=/dev/scd0 -v -data path_to_iso_file

Configuration options for wodim are in /etc/wodim.conf.

To see list of options for the drive:

wodim dev=/dev/scd0 driveropts=help -checkdrive

log4php example

This log4php example demonstrates how to the log4php library in a non-trivial way. log4php is a version of the Apache log4J package ported to PHP5. Please refer to the apache project for complete details.

log4php is a powerful logging mechanism that allows you to use a configuration file to specify what types of messages you want to route to various destinations (eg: file, email, database record). This makes it easy to change what messages are logged or where they are sent without tinkering around with code in the application.

In concept log4php has three core entities that work together:

(1) Loggers – loggers know what level of message to capture, they use one or more Appenders; loggers can be defined in a heirarchial manner, always stemming from the root logger. Loggers are named, and the periods in the name indicate the heirarchial levels (phpfunk, phpfunk.descendant, phpfunk.descendant.furtherDescenant, etc).

(2) Appenders – determine where the logged messages are sent and use Layouts to determine how the message should be formatted.

(3) Layouts specify the formating of the logged message.

In this example we want to capture all log messages to a file called default.log. We also want to capture serious errors to error.log and send an error message via email to the webmaster to let him know something is wrong.

First lets look at how simple the code is. I’m throwing an Exception just for fun. The configuration that I’ll explain below is the key to harnessing the power of log4php.

require_once 'log4php/Logger.php';
Logger::configure('logconfig.xml');

// get logger by name, if this name isn't listed 
// as a logger name in the configuration 
// file you will get the root level logger
$logger = Logger::getLogger('phpfunk');
$logger->info("beginning log4php example");
try {
    throw new Exception("log4php example error");
}
catch(Exception $e) {
    $logger->error($e->getMessage());
}

The log files now contain:
::::::::::::::
/tmp/default.log
::::::::::::::
2011-04-13 13:25:35 [INFO] phpfunk: beginning log4php example (at /home/tom/testlog.php line 8)
2011-04-13 13:25:35 [ERROR] phpfunk: log4php example error (at /home/tom/testlog.php line 12)
::::::::::::::
/tmp/error.log
::::::::::::::
2011-04-13 13:25:35 [ERROR] phpfunk: log4php example error (at /home/tom/testlog.php line 12)

And in addition an email was sent with the same message body as the message captured in error.log.

Now lets dig into the log4php configuration. I’m using an XML file for configuration because I want to use a Filter in the Appender. At the bottom of the configuration file I set the logging level and appender to use for the root logger and another logger named phpfunk, which descends from the root because the root logger is always the top most logger.

Appenders are referenced by name in the appender_ref element in the logger mark up. For phpfunk I have specified two appenders, a file and one to send messages via email. Both of those appenders further use filtering to control what level messages are captured in the logs; in this case ERROR and FATAL messages.

The root logger records DEBUG and higher messages. The Appender it uses writes these messages to default.log.

I want to use phpfunk logger, for this example, to capture all messages to default.log, but only capture ERROR and FATAL messages to error.log and also to email ERROR and FATAL messages to the webmaster. If I set the default logger level to ERROR for phpfunk then any INFO level messages it receives are discarded. By setting the level to ANY I ensure that INFO level messages get passed up to the default logger’s Appender and appear in default.log.

Here is the log4php XML configuration and you can download the example log4php configuration.

<?xml version="1.0"?>
<log4php:configuration xmlns:log4php="http://logging.apache.org/log4php/">
    <appender name="default" class="LoggerAppenderFile">
        <param name="file" value="/tmp/default.log" />
        <param name="datePattern" value="Ymd" />
        <layout class="LoggerLayoutPattern">
            <param name="ConversionPattern" value="%d{Y-m-d H:i:s} [%p] %c: %m (at %F line %L)%n" />
        </layout>
    </appender>
    <appender name="errorLog" class="LoggerAppenderFile">
        <param name="file" value="/tmp/error.log" />
        <param name="datePattern" value="Ymd" />
        <layout class="LoggerLayoutPattern">
            <param name="ConversionPattern" value="%d{Y-m-d H:i:s} [%p] %c: %m (at %F line %L)%n" />
        </layout>
        <filter class="LoggerFilterLevelRange">
            <param name="LevelMin" value="error" />
            <param name="LevelMax" value="fatal" />
        </filter>
        <filter class="LoggerFilterDenyAll" />
    </appender>
    <appender name="emailNotice" class="LoggerAppenderMail">
        <param name="from" value="webmaster@phpfunk.com" />
        <param name="to" value="tom@phpfunk.com" />
        <param name="subject" value="Error" />
        <layout class="LoggerLayoutPattern">
            <param name="ConversionPattern" value="%d{Y-m-d H:i:s} [%p] %c: %m (at %F line %L)%n" />
        </layout>
        <filter class="LoggerFilterLevelRange">
            <param name="LevelMin" value="error" />
            <param name="LevelMax" value="fatal" />
        </filter>
        <filter class="LoggerFilterDenyAll" />
    </appender>
    <root>
        <level value="DEBUG" />
        <appender_ref ref="default" />
    </root>
    <logger name="phpfunk" additivity="true">
        <!-- with additivity set to TRUE the default is also called, eg parent -->
        <level value="ALL" />
        <appender_ref ref="errorLog" />
        <appender_ref ref="emailNotice" />
    </logger>
</log4php:configuration>

So you see with a single configuration file its easy to switch around what level messages are captured, how they are handled (written to file, emailed) and how they are formatted. Hopefully you will find this log4php example to be a nice addition to the information presented in the log4php project documentation.

How to Get Raw Request Variables in PHP

Turns out php will tinker with names of HTTP request parameters in some circumstances. The reason is historical: register globals was a dumb idea that allowed request parameters to be converted into php variables. A request parameters is something like ?id=42 in a GET request, or the name of a form input in a POST request, to use common examples. The problem was, register globals couldn’t convert a variable with a space or dot in the name into a valid php variable because dot and space are not allowed in php variable names. In other words ?object.id=42 would become in $_GET object_id with a value of 42. Eventhough the use of register globals is frowned upon today because its a security vulnerability, the behavior of PHP remains unchanged as far as tinkering with request parameter names.

So how can you get at the raw values? I found these two mechanisms. For HTTP GET requests you can read the raw query string from $_SERVER[‘QUERY_STRING’]. For POST requests you have to call file_get_contents(‘php://input’), which is a trick I learned from this post.

Caching Static Content with mod_expires and Apache

Visitors to your website will experience faster page loads if you take some steps to enable caching of static content, such as CSS files, javascript and images. One simple way to enable such caching is to use mod_expires with Apache. If you run your own server you can easily enable this module. If you are on a shared host you should check with your hosting provider to see if its enabled. If you are looking for reasonably priced virtual private hosting you should checkout rimuhosting.com.

Here is the official documentation for the module.

I found that with ExpiresByType specifying text/javascript did not set the expiration time for javascript files. However, changing text/javascript to application/javascript did work.

If you are serving dynamic content do not set a default expires time (ie with ExpiresDefault). You do not want to cache dynamic pages like a shopping cart or other dynamically generated HTML.

Learn efficient MySQL

Having a good understanding of database schema design, how to use indices and write table joins is important for any developer. If you use mysql there is an excellent book that covers such information in a clear and concise manner.

Click on the cover image to visit amazon.com:

Traversing XML with JavaScript

Here are some excellent references on walking thru a XML blob with JavaScript. I was a little rusty. The main issue was that every element has a child node that is a text node, even if that text is empty. I was laboring under the misconception that somehow the text portion of an element was something we could get with a method call….as you can with getAttribute(attrName) for example. Not so. I was confusing “elements” with “nodes”. That’s bad.

I’m always looking for API: give me all the methods and attributes in one place so I can use it as a quick reference…this one seems like a decent reference list of methods and properties for nodes in an XML DOM.

IBM developer works usually has good tutorials. This particular one gives you a fast overview of the DOM and especially points out that elements and nodes are not the same. But the code portions are in Java, not JavaScript.    And for some reason its not on developer works anymore.  There is a PDF of it here.