DSL Bandwidth Monitoring Tool Part 1

Ever how much you rely on / abuse the internet? As the Internet of Things emerges and more devices are sharing bandwidth within the home I became curious as to how much data my home actually uses. Exploring the connectivity options available in my area narrowed my choices down to an unlimited but slow access tethered to old copper wires and faster cellular options billed by megabytes used. I decided to statistically identify if a faster cellular connection would be a cost effective solution compared to an unlimited wired option. This was accomplished using a home brewed monitoring tool that will be described in detail below should you be interested in examining your own usage.


The first step in creating this tool is to identify a location for storing the data. I chose to use a MySQL database to be the home. A table was created with the following structure:



 Once the repository is defined the next steps are to collect the data then populate the table. I started with checking out available SNMP options. To do this, I accessed the administration page of my Comtrend ADSL 3112u router and enabled SNMP. The image below shows the default settings:




 Once the agent was enabled, SNMP access rights were granted to users on the local LAN:






 Now that the SNMP is enabled and access rights granted locally to the LAN which limits security risk from the outside world, the next step is to identify what data can be collected via SNMP. To accomplish this I took a snmpwalk from my Linux terminal:

snmpwalk -v2c -cpublic <IP_ADDRESS_OF_ROUTER>

This command lists all available SNMP version 2 (-v2c) values in the public community (-cpublic) for the device at the IP Address given. The output looks something like this if your mibs are enabled:

SNMPv2-MIB::sysDescr.0 = STRING: WA31-412CTU-C06_R02.A2pvbF039t.d26b
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.16972
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (8868373) 1 day, 0:38:03.73
SNMPv2-MIB::sysContact.0 = STRING: unknown
SNMPv2-MIB::sysName.0 = STRING: Comtrend
SNMPv2-MIB::sysLocation.0 = STRING: unknown
SNMPv2-MIB::sysServices.0 = INTEGER: 72
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (105861570) 12 days, 6:03:35.70
SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB
SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMPv2 entities
SNMPv2-MIB::sysORUpTime.1 = Timeticks: (0) 0:00:00.00
IF-MIB::ifNumber.0 = INTEGER: 7
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifIndex.5 = INTEGER: 5
IF-MIB::ifIndex.6 = INTEGER: 6
IF-MIB::ifIndex.7 = INTEGER: 7
IF-MIB::ifDescr.1 = STRING: eth1
IF-MIB::ifDescr.2 = STRING: eth2
IF-MIB::ifDescr.3 = STRING: eth3
IF-MIB::ifDescr.4 = STRING: eth0
IF-MIB::ifDescr.5 = STRING: ppp0.1
IF-MIB::ifDescr.6 = STRING: dsl0
IF-MIB::ifDescr.7 = STRING: atm0
IF-MIB::ifType.1 = INTEGER: ethern
and so on.....
 
Each identifier listed in the tree was compared against the administration pages to identify which values could be retrieved via SNMP. Since SNMP follows published standards I was able to find the Object Identifiers (OID) for the RXbytes and TXbytes of the WAN interface (.5 as shown by the ppp0.1 Descriptor above) at .iso.3.6.1.2.1.2.2.1.10.x and .iso.3.6.1.2.1.2.2.1.16.x respectively as expected by the standards. Notice the last .x of each OID. This is the index of the interface you are looking for, WAN or .5 in this case.

Now that we know which OID holds the data on the router by comparing the snmpwalk output with the webui for accuracy we can query that specific value for the interface by using snmpget:

snmpget -v2c -Oqv -c public <IP_ADDRESS_OF_ROUTER> .iso.3.6.1.2.1.2.2.1.10.5

This command gets the version 2c (-v2c) information in the public community (-c public) with output options (-O) that are quick (q) for easy parsing and values (v) only for the OID (.iso.3.6.1.2.1.2.2.1.10.5) which is our RXbytes. The output is only the value of the RXBytes transmitted over the WAN interface. Change the OID to see the TXBytes as described earlier.

SNMP has taken care of two main values we are looking to store. However, I chose to expand this tool to log some other information while it is monitoring the system. To accomplish the logging of these values I utilized two more tools. 

The next value I was looking to store is the IP address of the WAN interface. (This stemmed from my curiosity of how often my public IP address changes and whether or not it would be possible to host a server with it. Check out my Dyanmic DNS blog for more details on that possiblity. ) To accomplish this I found the easiest method is to use ipecho.net. Parsing the webpage for the value was not necessary as there is a plain version available to retrieve the data only http://ipecho.net/plain. So we can write the value directly to a variable using the non-interactive network downloader wget to quietly (-q) output to STDOUT (-O -) then echoed as seen here:

 wget http://ipecho.net/plain -O - -q

  Values are related to the connection speed and collected courtesy of speedtest.net and a python tool found on GitHub. (Note on this tool utilizes file transfers to test speed so the cumulative data used is skewed. Not a problem for an unlimited plan but costly per/mb if utilized frequently.) The tool was easily installed per instructions provided leaving only the timestamp missing from completing our table.

To accomplish the time stamp, I chose to let MySQL take care of that detail by defaulting the column to the current time for each insert as shown in the table structure above. 

Now that all methods for collecting data have been identified it is time to pull it all together with the help of BASH. The script below stores variables utilizing command substitution of the output of the tools discussed earlier and with a little help from grep. The grep command parses the output files with regular expression pattern matching. Once all of the variables are set, they are inserted into the MySQL table. (Note the use of the Internet Address to Number function INET_ATON()as it will convert the dotted decimal notation to an integer for ease of storage.) Finally, temporary files are removed for housekeeping purposes.


#!/bin/bash

### Acquire Data from Router
IPAddress=$(wget http://ipecho.net/plain -O - -q ; echo)
RXBytes=$(snmpget -v1 -Oqv -c public <IP_ADDRESS_OF_ROUTER> .iso.3.6.1.2.1.2.2.1.10.5)
TXBytes=$(snmpget -v1 -Oqv -c public <IP_ADDRESS_OF_ROUTER> .iso.3.6.1.2.1.2.2.1.16.5)

### Acquire Speed Data
/usr/local/bin/speedtest-cli --simple > /tmp/speedtestdata.txt

### Extract values from speed data required
PingSpeed=$(cat /tmp/speedtestdata.txt | grep -oP '(?s)(?<=Ping: ).*(?= ms)')
DownloadSpeed=$(cat /tmp/speedtestdata.txt | grep -oP '(?s)(?<=Download: ).*(?= Mbit/s)')
UploadSpeed=$(cat /tmp/speedtestdata.txt | grep -oP '(?s)(?<=Upload: ).*(?= Mbit/s)')


echo $PingSpeed
echo $DownloadSpeed
echo $UploadSpeed

###Insert Variables into Database
mysql --host=<HOSTNAME> --user=<USERNAME> --password=<PASSWORD> <DB_NAME> << EOF
SET @IPAddress="$IPAddress";
SET @RXBytes="$RXBytes";
SET @TXBytes="$TXBytes";
SET @PingSpeed="$PingSpeed";
SET @DownloadSpeed="$DownloadSpeed";
SET @UploadSpeed="$UploadSpeed";

insert into Statistics (IPAddress,RXBytes,TXBytes,PingSpeed,DownloadSpeed,UploadSpeed) values(INET_ATON(@IPAddress),@RXBytes,@TXBytes,@PingSpeed,@DownloadSpeed,@UploadSpeed);
EOF

### Cleanup Temporary Files
rm /tmp/speedtestdata.txt

This script is then executed at 16 minutes after every hour of every day via the cron:

16 * * * * /home/user/Scripts/LogDSLUsage.sh 

Once enough data is collected it can be analyzed. Check out part 2 for details on how this can be done programmatically.