Visualizing the Heap on Embedded Systems – Part II

In the last article, I described a method to acquire heap allocation data from an embedded system. Next, I’ll describe how to visualize the data. First, though, to make things easier, I’ll acquire some real data from a regular Linux application.

For simplicity, I profiled a Linux application built with gcc, but the same principle applies to an embedded application. Using gcc, there is a useful shortcut to wrap the allocation function calls, which doesn’t require editing the code or modifying the objects directly. The linker ld provides a built-in option –wrap which will replace a symbol with __wrap_symbol which in turn can call __real_symbol to call the original function. You can pass this option through gcc to the linker as appropriate:

CFLAGS = -g -O0 -Wall -Wl,--wrap,malloc -Wl,--wrap,free -Wl,--wrap,calloc -Wl,--wrap,realloc

Next, just define your wrappers in a fashion similar to the previous article:

#include <stdio.h>

void *__wrap_malloc(size_t size)
{
	void *ret;

	ret = __real_malloc(size);
	fprintf(stderr, "m,%d,0x%8.8x\n", size, (unsigned int)ret);
	return ret;
}

void __wrap_free(void *ptr) {
	__real_free(ptr);
	fprintf(stderr, "f,,0x%8.8x\n", (unsigned int)ptr);
	return;
}

void *__wrap_realloc(void *ptr, size_t size)
{
	void *ret = __real_realloc(ptr, size);
	fprintf(stderr, "f,,0x%8.8x\n", (unsigned int)ptr);
	fprintf(stderr, "m,%d,0x%8.8x\n", size, (unsigned int)ret);
	return ret;
}

void *__wrap_calloc(size_t nmemb, size_t size)
{
	void *ret = __real_calloc(nmemb, size);

	fprintf(stderr, "c,%d,0x%8.8x\n", size*nmemb, (unsigned int)ret);
	return ret;
}

Now, I just ran the newly compiled program and directed stderr to a file, and then ran the malloc_analyze.py program as before.

Finally, we can produce something useful from this data to analyze our heap usage. First, it’s good to take a look at the overall heap usage over time (or in this case, over number of allocation calls). You can do this easily enough in a pylab session:

x = csv2rec('fetchorigin_analyzed.txt', names=('type','size','addr','total','count'))
plot(x.total)
xlabel('Number of allocations')
ylabel('Heap Allocated (bytes)')

Total Heap Usage

With this graph, you can now choose some points at which you’d like to visualize the heap usage and fragmentation. For this example, I chose three points: 400, 800 and 900. I wrote a short script to take the analyzed data and plot it as a bar, color-coded to based on how much space is in use at a given point in the memory space. The general idea is:

  1. Find the first (lowest) address where heap memory is allocated.
  2. Map each addressable byte in the memory region to a 1 or 0 depending on whether it is in use
  3. Bin the data into reasonable sized chunk (e.g. 512 bytes) and calculate how much of the space is in use
  4. Produce a graph with a colored box for each bin, coded by how full it is.

I wrote a simple script to do this. It’s certainly not very efficient, but it works just fine for small data sets that I was dealing with. The script needs to be adjusted for the start address and points in time at which to produce a graph.

import csv
import sys
import cPickle
import numpy as N
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.colors as colors
import time

binsize = int(sys.argv[2])

fig = plt.figure()
p = 1

for end in [400, 800, 900]:
    mem = csv.reader(open(sys.argv[1]), delimiter=',')

    ax = fig.add_subplot(9,1,p)
    p = p + 1

    mmap = []
    mmap.extend([0 for x in range(0,1024*160)])

    mdata = []
    for row in mem:
        mdata.append(row)

    for row in mdata[0:end]:

        size = long(row[1])
        addr = long(row[2], 16) - 0x12c0010

        if row[0] == 'm' or row[0] == 'c':
            for loc in range(addr, addr + size):
                mmap[loc] = 1

        elif row[0] == 'f':
            for loc in range(addr, addr + abs(size)):
                mmap[loc] = 0

    print str(len(mdata[0:end]))

    mmapbin = []
    mmapbin.extend([0 for x in range(0,len(mmap) / binsize)])

    for x in range(0, len(mmap)):
        mmapbin[x / binsize] = mmapbin[x / binsize] + mmap[x]

    patches = ax.bar(N.arange(len(mmapbin)), [1 for x in mmapbin], linewidth=0, width=1)

    fracs = [float(x) / binsize for x in mmapbin]
    norm = colors.normalize(0, max(fracs))

    for thisfrac, thispatch in zip(fracs, patches):
        color = cm.jet(norm(thisfrac))
        thispatch.set_facecolor(color)

    ax.set_yticks([0, 1])
    ax.set_ylabel(str(end))

    fig.show()

And the result you get is a nice picture that can give you an intuitive view of how your heap is fragmented.

Visualization of the heap at three points in time

The graph could be cleaned up, but you can get a very good idea of what the heap looks like just from this graph. The x-axis units are in bin sizes, and could be converted to memory addresses to be more useful.

  • Share/Bookmark
Posted in embedded | Tagged , , , | Leave a comment

Operation Aurora

I have a general distrust for Mcafee, as the maker of the world’s most poorly performing virus scanner / security suite. However, when I came across a Wired article about security holes in the Perforce version control system, I was lured in with interest. I don’t use Perforce, but the catchy headline of stealing Google source code via their source control system seemed intriguing.

After reading, the article, I knew I was going to be disappointed when got as far as:

The communication contained a link to a website hosted in Taiwan that downloaded and executed a malicious JavaScript, with a zero-day exploit that attacked a vulnerability in the user’s Internet Explorer browser.

This was in direct contradiction of dire warnings just above from Mcafee about SCMs being wide open for exploit. The attackers got in via an Internet Explorer vulnerability, not a Perforce one.

I read the Mcafee paper anyway, and found that the researchers had discovered a number of legitimate security weakness in Perforce to put its makers to shame:

  • P4Web authentication can be bypassed by replay attack
  • P4Web access levels implemented via hidden controls and can be bypassed by URL manipulation
  • Session token not used for change password feature

It also mixes in some general security concerns that apply to every othr source control system in the world, or every other Windows service that exists, such as running with too high privileges, or the source code being stored in plaintext.

The complaint about the source code being stored unencrypted on the user’s local system seems particularly absurd. Given that the attacker has compromised the system and gained access (elevated privileges or not), it doesn’t matter how encrypted the files are, the attacker already has access to them so long as the user has authenticated and can decrypt them.

Mcafee goes on to make a number of recommendations, the most sensible, if impractical, one being to disable P4Web. But, the whitepaper leaves me wondering what its real purpose was. They gloss over the “zero-day exploit” in Internet Explorer to focus on vulnerabilities in Perforce that would not have otherwise been exposed beneath the corporate firewall.

Once a developer workstation is compromised and a pipe is available to the external internet, there’s really no stopping the attacker from obtaining the confidential source code, no matter how secure the SCM is. That is, unless, perhaps, the developer simply does no work on the source code. And that’s always an option if you want to be truly secure.

  • Share/Bookmark
Posted in security | Tagged , , , | Leave a comment

FTDI USB Serial Port Enumeration on Windows

FTDI is one of the two major manufacturers of USB UART chips, and they provide regularly updated and reliable drivers. Most devices I’ve used don’t bother to implement a custom VID/PID, so you can just use the drivers directly from FTDI.

When you connect a large number of USB serial ports, it is often difficult to determine which port belongs to which physical connector, so FTDI has implemented a mechanism that persists COM port enumeration on Windows. The mechanism works by uniquely identifying a port based on the serial number stored in an eeprom associated with the device. Unfortunately, it turns out to be really persistent and I spent half a day trying to unpersist the mapping after I plugged in a second device to test it. The result was my COM port which I wante to be COM3 – COM11 became COM21-COM29. While there’s nothing really wrong with this, I wanted the lower ports back.

FTDI Workaround for a Single Device

FTDI presents a workaround in the App Note Advanced Driver Options for using a single device. The Ignore Hardware Serial Number flag will force all devices to be treated as the same device and enumerate from the lowest available COM port. This is only an option if you want to connect just one device.

Removing Persistent Enumeration Entirely

Since I found no documentation on how to wipe out all persistent enumeration (ftclean doesn’t do it), I started searching the registry. The persistent enumeration appears to be stored by both the serenum driver and the ftdibus driver.

  1. Run FTClean to remove the driver and existing COM ports
  2. Delete the FTDI enumeration registry settings
    1. Remove HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\FTDIBUS
    2. Remove HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\FTDIBUS
    3. Remove HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\FTSER2K
  3. Delete serenum enumeration only for the ports you wish to remove
    1. Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\serenum\Enum, remove each REG_SZ entry for sequential COM ports you want to remove. Then, adjust the Count REG_DWORD and NextInstance REG_DWORD to point to the next available COM port.

I don’t know if all of these steps are strictly necessary, as I haven’t yet tried to reduce the procedure. However, this should eliminate all persistent mapping and the next time you install an FTDI device, the serial ports should start enumerating again from the number specified in the NextInstance value.

  • Share/Bookmark
Posted in troubleshooting | Tagged , , , , | Leave a comment

Visualizing the Heap on Embedded Systems

Debugging a memory leak can seem trivial compared to debugging fragmentation. Faced with such a problem recently, I decided I really needed to visualize what the heap looked like to determine how to fix the problem.

Many embedded systems avoid using dynamic memory allocation entirely to avoid just this kind of problem, among others. But the larger your system is, the more likely you are to want it to make life easier and more efficient. If your system is large enough, you may end up running embedded Linux and have all the power that entails. But if you’re stuck in the middle, you have an embedded system running some kind of RTOS using malloc and not nearly enough memory to do much of any useful debugging on target. In my case, we do have some tools for memory tracking and enough memory to theoretically use them, but they didn’t do what I wanted.

I was left uninspired after a number of Google searches, so I decided to blaze my own path and develop my own tool. There are three primary steps to visualizing heap data in this approach:

  1. Instrument ([mc]|re)alloc and free calls
  2. Process raw data
  3. Visualize processed data

I had previously worked on the first two steps, and saved my work, but it was really a one-off hack. This time I decided to automate the entire process.

Instrument ([mc]|re)alloc and free calls

The first step involves modifying your code to run on target with sufficient instrumentation to record memory allocation.

Instrument memory allocation calls

The first thing you need to do is modify ([mc]|re)alloc and free calls to so that they can do extra processing. If you’ve got the source code to your malloc and free, then you’re done; you can modify it directly. The same applies if you consistently use a malloc wrapper in your code. In my case, neither was true, so I had to do more work.

There are are least two options:

  1. Replace all calls with wrapped calls
  2. Link in replacement versions of ([mc]|re)alloc and free

The second option seemed more difficult than what I wanted to deal with since these functions are not weakly linked, and this would require some linker magic. So, I opted for option 1.

I wrote a simple Python script to replace all calls to malloc, calloc and free with instrumented versions: imalloc, icalloc and ifree. realloc is not used in our codebase. The script is not a lexer, but a few simple regular expressions were sufficient to catch all instances of these calls (and the ones inside comments, for good measure).

import fileinput
import os
import fnmatch
import re
import sys

malloc_re = re.compile(r'\bmalloc[(](.*?)[)]')
calloc_re = re.compile(r'\bcalloc[(](.*?)[)]')
free_re = re.compile(r'\bfree[(](.*?)[)]')

for root, dirs, files in os.walk('.'):
    candidates = fnmatch.filter(files, '*.c')
    for name in candidates:
        print name

        for line in fileinput.input(os.path.join(root, name), inplace=1):

            line = re.sub(malloc_re, r'imalloc(\1)', line)
            line = re.sub(calloc_re, r'icalloc(\1)', line)
            line = re.sub(free_re, r'ifree(\1)', line)

            sys.stdout.write(line)

My litmus test for this script was whether the code linked. It did.

Writing imalloc

The next step is highly platform-dependent and will vary given your needs and restrictions. The basic idea is to write wrapper functions that record the parameter of malloc. So, first, just call the appropriate real function inside the wrappers. Next, decide how you want to collect your data. You could store it in memory, but that’s almost certainly not practical given the constraints mentioned in this article. I decided to push the data out through a UART running at 115200 baud. You can potentially use any external interface, though choosing one that requires use of malloc would likely be a bad idea. I choose the UART simply because it was the first thing that came to mind, but it in fact has a number of desirable features:

  1. Relatively fast (compared to some other choices, like the JTAG debugger messaging interface)
  2. Extremely simple (doesn’t use malloc, simple interface requires little stack use)
  3. Already bound to stdio in this application

The second point is important to keep in mind. You may call malloc all over the place in your code, in the context of different threads. In my case, some of the threads are huge with 8K of stack space, but others are tiny (a couple hundred bytes) with little room to spare. It is safest to keep the instrumentation as lightweight as possible so you don’t blow the stack. I was overly optimistic the first time I tried this and just called printf directly. I ended up settling with an implementation that seems to work well enough:

static char str[64];
...
str = sprintf("m,%8.8x,%8.8x", size, ptr);
puts(str);

This generates a line of CSV, with the type of call (malloc), size allocated, and address of the allocation. A similar implementation for free does the same thing, but leaves the size parameter blank since it is unknown. My very first attempt included str on the stack, but that blew up in some of the tighter threads. Hopefully you’ll realize the big mistake I made here without much thought quicker than I did. To my meager defense, when I initially did this, I was only interested in initial mallocs at startup.

Being lazy, I stuck this code in an existing C file and turned off the annoying stop on warnings compiler flag we had enabled so the new wrapper functions would link. I could have been more thorough and either modified stdlib.h or included a new header file, but I didn’t.

This ran fine until multiple threads started calling malloc and my analysis script started reporting errors. Of course, my wrapper function is missing the global lock that malloc uses for thread-safety. I added global lock calls around the wrappers to create the final code:

void *imalloc(size_t size) {
    void *ptr;
    static char str[64];

    __global_lock_acquire();
    ptr = malloc(size);
    str = sprintf("m,%8.8x,%8.8x", size, ptr);
    puts(str);
    __global_lock_release();

    return ptr;
}

Running the instrumented code on target produces a nice long list of CSV which I can record indefinitely with a terminal program attached to the UART:

m,12,0x50000000
m,10,0x5000000C
c,40,0x50000034
f,,0x50000000
f,,0x5000000C
m,12,0x50000000

We’ve now acquired all the data we need from the target. By itself, it’s not very insightful, but once it is processed we can learn a great deal from it.

Processing the Raw Data

The raw data contains only the information known from the parameters of the functions called. Thus, we don’t have the size of the allocation freed in the free lines. This can be taken care of easily enough with a small script and linear search backward through the data. For example, given

f,,0×50000000

We don’t know how much was allocated at 0×50000000. However, if we step backward through the list, we can look for the first allocation line we find with the same address:

m,12,0×50000000

We now know the allocation size (12) and can fill in the missing value and move on to the next entry.

My data was interlaced with other debug trace, so I passed it through a line filter first. Then, I wrote another script to backfill the free addresses:

import csv
import sys

mem = csv.reader(open(sys.argv[1]), delimiter=',')

mallocd = {}
total = 0
num = 0

for row in mem:
    if row[0] == 'm' or row[0] == 'c':
        row[1] = str(int(row[1]))
        if row[2] in mallocd:
            # This section was for debugging problems with the data
            if int(row[1], 16) == int(mallocd[row[2]][1],16):
                pass
            else:
                pass
        mallocd[row[2]] = row
        total += int(row[1])
        row.append(str(total))
        row.append(str(num))
        print ','.join(row)
    elif row[0] is 'f':
        try:
            row[1] = str(-int(mallocd[row[2]][1]))
            del mallocd[row[2]]
        except KeyError:
            row[1] = str(0)
        total += int(row[1])
        row.append(str(total))
        row.append(str(num))
        print ','.join((row))
    else:
        print row
        raise Exception
    num += 1

The script parses the raw data as csv and does a simple reverse linear search for each free entry until it finds a malloc or calloc with matching address. If all goes well, a backfilled version of the csv file will print to stdout. However, it is important to ensure that your data are valid and the script above does very little validation. I added some additional debugging statements initially to ensure that everything matched up. There are a number of problems that can occur (and did occur for me):

  • Not all allocations are recorded. This could happen if your IO device (a UART in this case) is not initialized before the first allocation occurs.
  • Allocations don’t match deallocations. This can happen because of missing output (IO not functional during certain periods) or if you get overzealous trying to save memory like I did. When I allocated the string array as a static function variable, it worked great until the RTOS was started and threads started overwriting the data. Don’t forget to acquire a lock!

What’s Next

In a subsequent article, I’ll describe how to visualize the processed data.

  • Share/Bookmark
Posted in embedded | Tagged , , , | Leave a comment

Debugging USB Device Installation on Windows

Recently, I ran into a problem trying to install a USB keyboard on Windows XP at work. Given that HID devices always just work with Windows, I wasn’t quite sure how to go about troubleshooting such a ridiculous problem.

My first step was to investigate the problem, repeating the process to see what went wrong. When I plugged in the keyboard, it would show up as Unknown Device. Searching for drivers would fail. I already had another USB keyboard installed, and this one worked fine. So I tried installing a third keyboard, this time a fancy multimedia keyboard, and it also exhibited the same behavior.

It turned out that my keyboard was not just a Human Interface Device, despite its appearance. It was actually a USB composite device and hub. I spent half a day trying to figure out what was wrong, and ended up learning a lot more about Windows USB driver installation.

Microsoft provides several MSKB articles on troubleshooting USB driver installation, most of which are useless. However, KB314464 contains the key to the problem:  setupapi.log. This log file records installations of drivers and the process Windows went through to determine how to match a driver to newly discovered device. Once I opened this file and watched the install fail for the keyboard, it became obvious that it was searching for a USB composite device driver, but not finding one. It turned out my usb.inf was missing. I don’t know how it got deleted, but it did. Once I copied over a replacement from another PC, the keyboard installed perfectly.

Device Tree for Logitech Illuminated Keyboard (Windows 7)

Here’s what happened with the when I plugged in the keyboard:

[2010/01/04 10:11:28 1276.7 Driver Install]
#-019 Searching for hardware ID(s): usb\vid_046d&pid_c318&rev_5501,usb\vid_046d&pid_c318
#-018 Searching for compatible ID(s): usb\devclass_00&subclass_00&prot_00,usb\devclass_00&subclass_00,usb\devclass_00,usb\composite
#-198 Command line processed: C:\WINNT\system32\services.exe
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#W059 Selecting best compatible driver failed. Error 0xe0000228: There are no compatible drivers for this device.
#W157 Default installer failed. Error 0xe0000228: There are no compatible drivers for this device.

Windows failed to find an exact driver for the vid/pid combination and tried to find a compatible driver based on the device class. The device is a usb\composite device, which should map to a standard Windows driver, but it instead reported that the search failed. At this point, I should have looked for an inf file with this hwid string: usb\composite. If I had, I would have discovered it missing in my directory, but present in another machine where I could install a keyboard, in usb.inf.

The Generic Models section of usb.inf includes the hwid:

; =================== Generic ==================================

[Generic.Section]
%USB\COMPOSITE.DeviceDesc%=Composite.Dev,USB\COMPOSITE
%USB\UNKNOWN.DeviceDesc%=BADDEVICE.Dev,USB\UNKNOWN

This directs to DDInstall sections Composite.Dev.*:

[Composite.Dev]
AddReg=CommonClassParent.AddReg
CopyFiles=CommonClassParent.CopyFiles

; For Windows NT...

[Composite.Dev.NT]
CopyFiles=CommonClassParent.CopyFiles
DelReg=Composite.DelReg.NT

[Composite.DelReg.NT]
HKR,,EnumPropPages32

[Composite.Dev.NT.Services]
AddService = usbccgp, 0x00000002, CommonClassParent.AddService

The Composite.Dev section is actually for Windows 2000 and the decorator .NT on Composite.Dev.NT specifies a generic driver for Windows XP and later.

Once I placed the usb.inf file back into the %SYSTEMDIR%\inf directory, installation of the USB Composite Device magically worked:

[2010/01/04 13:10:39 1264.3 Driver Install]
#-019 Searching for hardware ID(s): usb\vid_046d&pid_c318&rev_5501,usb\vid_046d&pid_c318
#-018 Searching for compatible ID(s): usb\devclass_00&subclass_00&prot_00,usb\devclass_00&subclass_00,usb\devclass_00,usb\composite
#-198 Command line processed: C:\WINNT\system32\services.exe
#I022 Found "USB\COMPOSITE" in C:\WINNT\inf\usb.inf; Device: "USB Composite Device"; Driver: "USB Composite Device"; Provider: "Microsoft"; Mfg: "(Standard USB Host Controller)"; Section name: "Composite.Dev".
#I023 Actual install section: [Composite.Dev.NT]. Rank: 0x00002003. Effective driver date: 07/01/2001.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [Composite.Dev] in "c:\winnt\inf\usb.inf".
#I320 Class GUID of device remains: {36FC9E60-C465-11CF-8056-444553540000}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "USB\VID_046D&PID_C318\5&1AF8ED3F&0&1".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [Composite.Dev.NT.Interfaces] from "c:\winnt\inf\usb.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "USB\VID_046D&PID_C318\5&1AF8ED3F&0&1".
#I121 Device install of "USB\VID_046D&PID_C318\5&1AF8ED3F&0&1" finished successfully.

After that, the remaining the remaining device tree hierarchy installed successfully and the keyboard was enabled:

[2010/01/04 13:10:43 1264.7 Driver Install]
#-019 Searching for hardware ID(s): usb\vid_046d&pid_c318&rev_5501&mi_00,usb\vid_046d&pid_c318&mi_00
#-018 Searching for compatible ID(s): usb\class_03&subclass_01&prot_01,usb\class_03&subclass_01,usb\class_03
#-198 Command line processed: C:\WINNT\system32\services.exe
#I022 Found "USB\Class_03&SubClass_01" in C:\WINNT\inf\input.inf; Device: "USB Human Interface Device"; Driver: "USB Human Interface Device"; Provider: "Microsoft"; Mfg: "(Standard system devices)"; Section name: "HID_Inst".
#I023 Actual install section: [HID_Inst.NT]. Rank: 0x00003101. Effective driver date: 07/01/2001.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [HID_Inst] in "c:\winnt\inf\input.inf".
#I320 Class GUID of device remains: {745A17A0-74D3-11D0-B6FE-00A0C90F57DA}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "USB\VID_046D&PID_C318&MI_00\6&7451BA8&0&0000".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [HID_Inst.NT.Interfaces] from "c:\winnt\inf\input.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "USB\VID_046D&PID_C318&MI_00\6&7451BA8&0&0000".
#I121 Device install of "USB\VID_046D&PID_C318&MI_00\6&7451BA8&0&0000" finished successfully.
[2010/01/04 13:10:47 1264.11 Driver Install]
#-019 Searching for hardware ID(s): usb\vid_046d&pid_c318&rev_5501&mi_01,usb\vid_046d&pid_c318&mi_01
#-018 Searching for compatible ID(s): usb\class_03&subclass_00&prot_02,usb\class_03&subclass_00,usb\class_03
#-198 Command line processed: C:\WINNT\system32\services.exe
#I022 Found "USB\Class_03" in C:\WINNT\inf\input.inf; Device: "USB Human Interface Device"; Driver: "USB Human Interface Device"; Provider: "Microsoft"; Mfg: "(Standard system devices)"; Section name: "HID_Inst".
#I023 Actual install section: [HID_Inst.NT]. Rank: 0x00003202. Effective driver date: 07/01/2001.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [HID_Inst] in "c:\winnt\inf\input.inf".
#I320 Class GUID of device remains: {745A17A0-74D3-11D0-B6FE-00A0C90F57DA}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "USB\VID_046D&PID_C318&MI_01\6&7451BA8&0&0001".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [HID_Inst.NT.Interfaces] from "c:\winnt\inf\input.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "USB\VID_046D&PID_C318&MI_01\6&7451BA8&0&0001".
#I121 Device install of "USB\VID_046D&PID_C318&MI_01\6&7451BA8&0&0001" finished successfully.
[2010/01/04 13:10:51 1264.15 Driver Install]
#-019 Searching for hardware ID(s): hid\vid_046d&pid_c318&rev_5501&mi_00,hid\vid_046d&pid_c318&mi_00,hid_device_system_keyboard,hid_device_up:0001_u:0006,hid_device
#-198 Command line processed: C:\WINNT\system32\services.exe
#I022 Found "HID_DEVICE_SYSTEM_KEYBOARD" in C:\WINNT\inf\keyboard.inf; Device: "HID Keyboard Device"; Driver: "HID Keyboard Device"; Provider: "Microsoft"; Mfg: "(Standard keyboards)"; Section name: "HID_Keyboard_Inst".
#I023 Actual install section: [HID_Keyboard_Inst.NT]. Rank: 0x00001002. Effective driver date: 07/01/2001.
#I022 Found "HID\VID_046D&PID_C318&Mi_00" in C:\WINNT\inf\oem52.inf; Device: "Logitech HID-Compliant Keyboard"; Driver: "Logitech HID-Compliant Keyboard"; Provider: "Logitech"; Mfg: "Logitech"; Section name: "HIDFiltInstWakeEnbl".
#I023 Actual install section: [HIDFiltInstWakeEnbl.NT]. Rank: 0x00000001. Effective driver date: 06/17/2009.
#I022 Found "HID\VID_046D&PID_C318&Mi_00" in C:\WINNT\inf\oem52.inf; Device: "Logicool HID-Compliant Keyboard (106 keys)"; Driver: "Logicool HID-Compliant Keyboard (106 keys)"; Provider: "Logitech"; Mfg: "Logicool"; Section name: "HIDFiltInstWakeEnblJ".
#I023 Actual install section: [HIDFiltInstWakeEnblJ.NT]. Rank: 0x00000001. Effective driver date: 06/17/2009.
#I022 Found "HID_DEVICE" in C:\WINNT\inf\input.inf; Device: "HID-compliant device"; Driver: "HID-compliant device"; Provider: "Microsoft"; Mfg: "(Standard system devices)"; Section name: "HID_Raw_Inst".
#I023 Actual install section: [HID_Raw_Inst.NT]. Rank: 0x00001004. Effective driver date: 07/01/2001.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [HIDFiltInstWakeEnbl] in "c:\winnt\inf\oem52.inf".
#I320 Class GUID of device remains: {4D36E96B-E325-11CE-BFC1-08002BE10318}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#I063 Selected driver installs from section [HIDFiltInstWakeEnbl] in "c:\winnt\inf\oem52.inf".
#I320 Class GUID of device remains: {4D36E96B-E325-11CE-BFC1-08002BE10318}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "HID\VID_046D&PID_C318&MI_00\7&2C4789AA&0&0000".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [HIDFiltInstWakeEnbl.NT.Interfaces] from "c:\winnt\inf\oem52.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "HID\VID_046D&PID_C318&MI_00\7&2C4789AA&0&0000".
#I121 Device install of "HID\VID_046D&PID_C318&MI_00\7&2C4789AA&0&0000" finished successfully.
[2010/01/04 13:10:56 1264.19 Driver Install]
#-019 Searching for hardware ID(s): hid\vid_046d&pid_c318&rev_5501&mi_01&col01,hid\vid_046d&pid_c318&mi_01&col01,hid_device_system_consumer,hid_device_up:000c_u:0001,hid_device
#-198 Command line processed: C:\WINNT\system32\services.exe
#I022 Found "HID_DEVICE_UP:000C_U:0001" in C:\WINNT\inf\hidserv.inf; Device: "HID-compliant consumer control device"; Driver: "HID-compliant consumer control device"; Provider: "Microsoft"; Mfg: "Microsoft"; Section name: "HIDSystemConsumer".
#I023 Actual install section: [HIDSystemConsumer]. Rank: 0x00001003. Effective driver date: 07/01/2001.
#I022 Found "HID_DEVICE" in C:\WINNT\inf\input.inf; Device: "HID-compliant device"; Driver: "HID-compliant device"; Provider: "Microsoft"; Mfg: "(Standard system devices)"; Section name: "HID_Raw_Inst".
#I023 Actual install section: [HID_Raw_Inst.NT]. Rank: 0x00001004. Effective driver date: 07/01/2001.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [HIDSystemConsumer] in "c:\winnt\inf\hidserv.inf".
#I320 Class GUID of device remains: {745A17A0-74D3-11D0-B6FE-00A0C90F57DA}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "HID\VID_046D&PID_C318&MI_01&COL01\7&3DCDBE1&0&0000".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [HIDSystemConsumer.Interfaces] from "c:\winnt\inf\hidserv.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "HID\VID_046D&PID_C318&MI_01&COL01\7&3DCDBE1&0&0000".
#I121 Device install of "HID\VID_046D&PID_C318&MI_01&COL01\7&3DCDBE1&0&0000" finished successfully.
[2010/01/04 13:10:58 1264.23 Driver Install]
#-019 Searching for hardware ID(s): hid\vid_046d&pid_c318&rev_5501&mi_01&col02,hid\vid_046d&pid_c318&mi_01&col02,hid_device_up:ff00_u:0001,hid_device
#-198 Command line processed: C:\WINNT\system32\services.exe
#I022 Found "HID_DEVICE" in C:\WINNT\inf\input.inf; Device: "HID-compliant device"; Driver: "HID-compliant device"; Provider: "Microsoft"; Mfg: "(Standard system devices)"; Section name: "HID_Raw_Inst".
#I023 Actual install section: [HID_Raw_Inst.NT]. Rank: 0x00001003. Effective driver date: 07/01/2001.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [HID_Raw_Inst] in "c:\winnt\inf\input.inf".
#I320 Class GUID of device remains: {745A17A0-74D3-11D0-B6FE-00A0C90F57DA}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "HID\VID_046D&PID_C318&MI_01&COL02\7&3DCDBE1&0&0001".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [HID_Raw_Inst.NT.Interfaces] from "c:\winnt\inf\input.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "HID\VID_046D&PID_C318&MI_01&COL02\7&3DCDBE1&0&0001".
#I121 Device install of "HID\VID_046D&PID_C318&MI_01&COL02\7&3DCDBE1&0&0001" finished successfully.
[2010/01/04 13:10:59 1264.29]
#-198 Command line processed: C:\WINNT\system32\services.exe
#I140 Installing device class: "DriverInterface" {D41DD63A-1395-4419-AE14-A534F5F2AD29}.
#I141 Class install completed with no errors.
[2010/01/04 13:10:59 1264.27 Driver Install]
#-019 Searching for hardware ID(s): logitech_raw_pdo
#-198 Command line processed: C:\WINNT\system32\services.exe
#I022 Found "LOGITECH_RAW_PDO" in C:\WINNT\inf\oem53.inf; Device: "Logitech Driver Interface"; Driver: "Logitech Driver Interface"; Provider: "Logitech"; Mfg: "Logitech"; Section name: "NullInst".
#I023 Actual install section: [NullInst.NT]. Rank: 0x00000000. Effective driver date: 06/17/2009.
#-166 Device install function: DIF_SELECTBESTCOMPATDRV.
#I063 Selected driver installs from section [NullInst] in "c:\winnt\inf\oem53.inf".
#I320 Class GUID of device remains: {D41DD63A-1395-4419-AE14-A534F5F2AD29}.
#I060 Set selected driver.
#I058 Selected best compatible driver.
#-166 Device install function: DIF_INSTALLDEVICEFILES.
#I124 Doing copy-only install of "{C1FCC185-55B3-4E00-814B-C588A13525E1}\VID_046D&PID_C318&REV_5501&MI_00&HIDFILT\8&AEA4138&0&00".
#-166 Device install function: DIF_REGISTER_COINSTALLERS.
#I056 Coinstallers registered.
#-166 Device install function: DIF_INSTALLINTERFACES.
#-011 Installing section [NullInst.NT.Interfaces] from "c:\winnt\inf\oem53.inf".
#I054 Interfaces installed.
#-166 Device install function: DIF_INSTALLDEVICE.
#I123 Doing full install of "{C1FCC185-55B3-4E00-814B-C588A13525E1}\VID_046D&PID_C318&REV_5501&MI_00&HIDFILT\8&AEA4138&0&00".
#I121 Device install of "{C1FCC185-55B3-4E00-814B-C588A13525E1}\VID_046D&PID_C318&REV_5501&MI_00&HIDFILT\8&AEA4138&0&00" finished successfully.

USB Driver Installation Troubleshooting Steps for Windows XP / Server 2003

  1. Look in device manager to determine the device that failed to install (with a yellow exclamation mark )
  2. View the properties to determine its hardware and compatible ids
  3. Search setupapi.log for the device installation and examine the output
  4. If no driver was found, search the %SystemRoot%\Inf directory for the vid/pid or compatible id.

In a future article, I will cover the differences for newer versions of Windows (Vista, Server 2008, 7).

  • Share/Bookmark
Posted in troubleshooting | Tagged , , , | 1 Comment

how to prevent feedback on your website

Fidelity has done a remarkable job of unintentionally making it impossible to submit feedback on their website, 401k.com. I decided to send feedback after I noticed the new login form was obviously completely untested before being deployed:

  1. The username field assumes you are entering a social security number and masks all but the last four ‘digits’ of your input, given a sufficiently long username.
  2. The tab ordering changed so I had to tab more than four times to get from the username to password field (I gave up after four tabs).

Once you log into the site, there is no option to submit feedback that I could find. When you log out, there is a contact link, but website feedback is hidden under a link: Participant in a 401(k) Plan Managed by Fidelity. This link is right above General Investing Inquiry, so it wasn’t particularly obvious it was what I wanted.

After I wrote my message, a dialog box popped up saying I had entered invalid characters.

Of course, after review of the message, none of these characters were present in the message. I tried removing parentheses with no luck. Then I figured it was the slash (/) character. Still, no luck. So I started removing newlines. Eventually I just kept removing bits of the message until it got sent.

Turns out it was in fact the slash character, though removing it the first time didn’t work. All told, I have now submitted six comments to Fidelity, including a rant on the form through which I couldn’t manage to submit my comments.

  • Share/Bookmark
Posted in usability | Tagged , , | Leave a comment

makeshift iscsi san

Aware that my aging PATA drives will slowly but surely descent toward an inevitable death along with the data contained on them, I decided to revive an old forsaken Soyo motherboard (free after rebate, if Soyo wasn’t in the business of fraud…) as a SAN. It’s previous incarnation as a dvr failed miserably, since it just wasn’t powerful enough. Somehow, I had lost the CMOS battery, and it wouldn’t boot, but once I stole an even older one it eventually booted. Now, I just needed to set up a SAN and I would be all set. Once again disenchanted by Ubuntu and not having considered something like FreeNAS, I decided to install Debian Squeeze. A few hours later I figured out that the squeeze installer is broken and installed Lenny instead as a base system. Everything went well, except for the installing grub. Maybe I didn’t read what it said carefully enough, but I really wanted it to install stage 1 on the mbr of my install drive, not the primary master IDE drive. As it turned out, my primary master happened to be my newest, and least reliable PATA drive that I bought in 2005 and now reports SMART errors. I have to press F1 every time I boot to ignore the SMART errors and usually it doesn’t actually boot until the second try, and only when I use the boot selection menu. On a side now, the Debian installer is a pleasure to work with for installing an lvm-based system. It’s a nightmare and completely manual on Ubuntu. lvm over dmraid was also seamless, just with one addition to special boot parameters:

dmraid=true

At this point everything was up and running. Admittedly, the bottom of my coffee table is not the best location for a server, but it’s temporary and mostly out the way. It’s quite nice to work with too, having lots of space to spread everything out over. I broke two SATA power connectors trying to move drives around in my primary PC’s case.

Configuring an iSCSI Target on Debian Lenny

Setting up the server only requires a few steps for the installation of iSCSI Enterprise Target. Instructions are available on the Debian wiki.

aptitude install iscsitarget
aptitude install iscsitarget-modules

Since I was running a stock Lenny kernel, I didn’t need to compile anything, but you can use module-assistant if you need to for some reason. Next, you need to configure /etc/ietd.conf. Examples are included so it should be fairly self-explanatory. You’ll need to choose an iSCSI qualified name. If you’re on a private network it doesn’t really matter, just choose something that will be unique on your network if you might have another iSCSI target. I included my box’s hostname. I set up my /etc/ietd.conf like this:

Target iqn.2009-08.org.debian.vault:vault1
    Lun 0 Path=/dev/hdd,Type=fileio
    Lun 1 Path=/dev/hda,Type=fileio
    Lun 2 Path=/dev/hdc,Type=fileio
    Lun 3 Path=/dev/hde,Type=fileio
    Lun 4 Path=/dev/hdf,Type=fileio
    Lun 5 Path=/dev/hdi,Type=fileio

Initially, I had added a an IncomingUser clause to the Target, but it wasn’t immediately obvious to me how to specify a username for CHAP authentication in Windows 7, so I just removed it. I added six LUNs for the six non-Linux drives I had connected and specified the drive itself so I would get the entire drive, rather than individual partitions. You can also specify more targets, but there’s no reason to do so in this case. Be sure to restart the server with /etc/init.d/iscsitarget restart. Apparently you can also say invoke-rc.d iscsitarget restart now. But, so far I’ve had little luck remembering this, and I wonder how I am supposed to be aware of these new features without following development mailing lists in detail… One of the things I’ve always hated about iSCSI is typing in the qualified names and IPs manually. It seems like a lot of wasted effort. Fortunately, the iSNS protocol exists and is equivalent to DNS in the TCP/IP realm. You can install an iSNS server as well on the same host. It doesn’t require any configuration on Debian. Just run:

aptitude install isns

Now, the next step was to connect my Windows 7 box as an iSCSI initiator. You can search for iSCSI in the Control Panel and the first time you run it, it will ask you to enable to iSCSI service and present you with an iSCSI Target dialog. There are a lot of tabs and options to configure, but with iSNS, you should be able to get up and running quickly.

Go to the Discover tab and add your host’s IP address as an iSNS server. After waiting a few moments, or clicking refresh enough times, your target portal at the same IP address should show up in the Target portals list above. Switch to the Targets tab and you should now see the Target name you chose in your /etc/ietd.conf file in the list with hopefully the status Connected.

If you’ve made it this far, everything is working properly. The only step left is to add your volumes so Windows will see them. The easy way to do this is to go to the Volumes and Devices tab and click Auto Configure. It should add a verbose string to the Volume List and when you go to Disk Management you should see your new disks. You’ll be able to tell they are iSCSI targets by looking at the hardware type and seeing that they are IET VIRTUAL-DISK SCSI devices.

Now, I can see all my additional NTFS partitions in explorer!

I decided to purchase a gigabit switch, since I had been planning to for some time, to hopefully increase the throughput to the SAN disks. Unfortunately, all I really got was disappointment.

My transfer rate seems to peak out at around 11MB/s, which is only just over 10% utilization of my gigabit link. It’s clearly gigabit speed, though, since you wouldn’t get such a high transfer rate on 100mbit. I read some complaints about the D-Link DGS-2208 on newegg about it not playing well with a mix of gigabit and 100mbit devices. I figured it was a configuration issue on the user’s part. I doubt there is anything wrong with the router, but I’ll need to do some diagnostic tests and try to isolate the problem. I haven’t yet actually seen gigabit speeds on this router.

  • Share/Bookmark
Posted in san | Tagged , , , | Leave a comment