Moving to new site

Monday, February 01, 2016

I'll be continuing my site off the blogger platform, but retaining this site for historical purposes. Please follow me to my new home at https://emil.lerch.org/. My first post goes into more detail about the move.


Probing support for architecture-specific native DLLs in .NET

Friday, June 13, 2014


Or...How I've tamed the Oracle beast

Over the last couple decades, very little has changed with regards to Oracle client software. Install hundreds of MB of code, update network/admin/tnsnames.ora, and finally you can make a database connection.

With a little (well, a lot) of elbow grease, I've made it possible for .NET applications to run as "AnyCPU" and work in 32 or 64 bit process space on either a 32 or 64 bit OS. It's still an 84MB download (mostly due to a large dll containing all of Oracle's error messages), but it's bin-deployable, and only 20MB more than the combination of installing each of Oracle's "Instant Lite" processor-specific installation packages. More importantly, since it's bin-deployed, you can have 32 and 64 bit processes running side by side on the same machine, something that's been problematic to do with full client installs.

The fruits of my labor are now on the the NativeProbing repository on GitHub: https://github.com/elerch/NativeProbing. The focus of the repository is not on Oracle per-se, but my goal was to achieve XCopy deployment for Oracle connectivity without massive installs or gnarly configuration.

.Net assemblies are loaded through the Fusion process, and typically follow a strict path through an AnyCPU chain, 64 bit chain or 32 bit chain. If using the <probing> element to alter flow and add platform-specific assemblies, the first time .NET hits an assembly not matching the underlying processor architecture you'll get a BadImageFormatException. This is especially nasty; running 32 bit on a 64 bit OS is not enough to see the exception, rather, you need to be 32 bit on a 32 bit OS with your 64 bit assemblies listed first in the <probing> directory list. Therefore you need to hook into the assembly resolution process itself similar to this process: http://stackoverflow.com/a/9951658/113225

However, native libraries contain a special challenge. A different Windows process kicks in when loading a native library, and in the case of Oracle, the .Net assemblies used load native assemblies. My code on GitHub therefore uses the SetDllDirectory kernel32.dll native function to add the correct platform-specific directory to the list of paths to check. The two techniques in concert work to automatically load the appropriate DLLs.

More details of my travels are on GitHub, but the bottom line is that I have code that can augment the probing pipeline to include the correct architecture specific DLLs. This can be used for any architecture-specific DLLs with matching version numbers. SQLLite is a possible candidate for using a similar technique.

Of course, Oracle does not have matching version numbers, which is where part two of this story unfolds. To make Oracle.DataAcess.dll work, I had to resort to extreme measures. The details are on the OracleDLLHacking.md file in my repository, but the high level process was this: 

  1. Disassemble Oracle.DataAccess.dll with ildasm
  2. Remove the public key token and change the version number in the IL
  3. Reverse the branch logic where an exception is thrown if the assembly is "incompatible" with the native code
  4. Reassemble the library with ilasm
  5. Repeat for the other processor architecture





Visual Studio unit testing slow when NetBIOS over TCP/IP is enabled

Monday, January 13, 2014

According to MS Connect, this doesn't happen. People have reported it, however, in VS 2010, 2012, and I've experienced the problems in VS 2013. It's also listed at this StackOverflow question: http://connect.microsoft.com/VisualStudio/feedback/details/768230/slow-running-in-test-runner

I've now added this handy PowerShell command to my "initialize a new machine" setup PowerShell script:

# Disable NetBios over TCP/IP on all interfaces
# to prevent weird Visual Studio slowdowns during unit tests
Get-ChildItem hklm:system/currentcontrolset/services/netbt/parameters/interfaces | foreach{ $item = $_; Set-ItemProperty -Path ($item.ToString().Replace("HKEY_LOCAL_MACHINE", "hklm:")) -Name NetbiosOptions -Type DWord -Value 2 -ea "SilentlyContinue" }


REST Basics

Friday, July 12, 2013


Obligatory upper-right blog image
I've created a video on the basic use of REST from a consuming developer's perspective.  It covers the following:

Normal HTTP Methods:

  • POST
  • GET
  • PUT
  • DELETE
  • PATCH

I cover the use of PATCH to replace special-purpose functions in MVC, singular vs plural urls and search apis.  I also demonstrate the use of the methods at a browser console (w/jQuery). Length - 19:23.



AngularJs HTML5 routing and IE9

Thursday, May 02, 2013

We seem to be pushing the limits of AngularJS, and we've only started using it. The framework is very promising but definitely very new and a little rough in some areas.

We'd like to avoid hash urls in our solutions. The backend can respond to our routes just fine, and our backend framework is able to serve up the correct content (not necessarily the same content) for any area of the application. Using HTML5 mode and the history.pushState api, we can get clean URLs and no 404 errors on refresh...awesome.

Enter, IE9 (top light blue line). Our current browser standards are IE9+, Desktop/Android Chrome, FF, and IOS Safari. All browsers support history.pushState except IE9. The browser has had a recent significant drop in usage, but we can't ignore it quite yet.

At first, we thought we could set $locationProvider.html5Mode(true) and it would either a) break in IE9 calling a function that didn't exist (but we could define as window.location.assign(url)), or b) do a browser-based redirect based on window.location.assign.  Well, RTFM:  what it does is actually relatively painful.  It falls back to hashbang syntax.  This is similar to history.js, and IMHO, is broken in the same way as that library.  Now you have URLs you can't share between browsers without crazy (and slow) workarounds.

I tried to work around the problem by providing a history.pushState implementation.  This fooled Angular into thinking it was compliant, but caused some redirect loops and ultimately didn't work out.  Next, I tried first intercepting routeChangeStart and calling event.preventDefault to no avail, then locationChangeStart also without success (I'm informed this does work, but for some reason I didn't see it, at least not in IE9).

Finally, it hit me...since we're not programmatically changing locations (just using anchor tags), we could simply determine if pushState is available and perform the following functions if it is not:

  1. Avoid defining any routes with the route provider
  2. Take our <script id='whateverroutewearecurrentlyon' type='text/ng-template'/> that is automatically generated on the backend for the current URL and manually add it to the ng-view element.
  3. Remove the ng-view attribute (for completeness) and add an ng-controller attribute pointing to the controller for the current route.
  4. Disable any pre-loading for pages other than the one on which we reside.
This technique can only work in a pretty specific set of circumstances, but I think a properly designed backend system should be able to meet the criteria:
  1. Any URL used by history.pushState actually generates appropriate content from the server.
  2. The server at any URL provides the page layout and the application code.  Providing the current page content in the script tag is nice (that's what we do) but it's not strictly required (#2 above could be done via Ajax).
  3. The application code is light enough that you (and your users) can put up with full page loads in non-compliant browsers until users upgrade.  I wouldn't want to try this technique on gmail. ;-)


Enabling Quartz jobs in ASP.NET applications that will run despite restart

Friday, January 11, 2013

With IIS 7.5, you can now auto-start applications and have them continuously run.  However, implementing System.Web.Hosting.IProcessHostPreloadClient means having a Process method with some significant restrictions.  In a Spring.Net environment, the IOC Container's context is not yet started, and it will fail in rather spectacular ways if you try to crank it up with a hack.  Even if you manage to do this, Quartz does not start up, so your jobs still will fail to run.

I spent some time on this problem and discovered that I do have access to System.Web.HttpRuntime.AppDomainAppVirtualPath, which allows us to automatically fire an initial request at the app automatically, if we're careful.  This will crank up Application_Start and the rest of the Spring (and Quartz) machinery, allowing us to keep an app running 100% of the time.  This initial request, however, will be thrown away, even if fired asynchronously, if initiated during the Preload() method.  I ended up using a thread to get around this problem, allowing the requests to fire 500ms after the Preload method runs.

In certain cases, we can't just use http://localhost/yourvirtualpathhere.  You might have multiple sites listening to different host headers, or you might be running SSL and don't want to hit the application using localhost.  To cover those cases, I devised a web.config appSettings scheme where additional URLs can be applied to Preload (space delimited).  You can also specify an IP address or machine name, so the web.config can stay static but different URLs can be applied as you move through environments.  Here is the code:


public class Preloader : System.Web.Hosting.IProcessHostPreloadClient
    {
        public void Preload(string[] parameters)
        {
            var uris = System.Configuration.ConfigurationManager.AppSettings["AdditionalStartupUris"];
            StartupApplication(AllUris(uris));
        }

        public void StartupApplication(IEnumerable<Uri> uris)
        {
            new System.Threading.Thread(o =>
            {
                System.Threading.Thread.Sleep(500);
                foreach (var uri in (IEnumerable<Uri>)o) {
                    var client = new System.Net.WebClient();
                    client.DownloadStringAsync(uris.First());
                }
            }).Start(uris);
        }

        public IEnumerable<Uri> AllUris(string userConfiguration)
        {
            if (userConfiguration == null)
                return GuessedUris();
            return AllUris(userConfiguration.Split(' ')).Union(GuessedUris());
        }

        private IEnumerable<Uri> GuessedUris()
        {
            string path = System.Web.HttpRuntime.AppDomainAppVirtualPath;
            if (path != null)
                yield return new Uri("http://localhost" + path);
        }

        private IEnumerable<uri> AllUris(params string[] configurationParts)
        {
            return configurationParts
                .Select(p => ParseConfiguration(p))
                .Where(p => p.Item1)
                .Select(p => ToUri(p.Item2))
                .Where(u => u != null);
        }

        private Uri ToUri(string value)
        {
            try {
                return new Uri(value);
            }
            catch (UriFormatException) {
                return null;
            }
        }

        private Tuple<bool string> ParseConfiguration(string part)
        {
            return new Tuple<bool, string>(IsRelevant(part), ParsePart(part));
        }

        private string ParsePart(string part)
        {
            // We expect IPv4 or MachineName followed by |
            var portions = part.Split('|');
            return portions.Last();
        }

        private bool IsRelevant(string part)
        {
            var portions = part.Split('|');
            return
                portions.Count() == 1 ||
                portions[0] == System.Environment.MachineName ||
                HostIpAddresses().Any(a => a == portions[0]);
        }

        private IEnumerable<string> HostIpAddresses()
        {
            var adaptors = System.Net.NetworkInformation.NetworkInterface.GetAllNetworkInterfaces();
            return adaptors
                    .Where(a => a.OperationalStatus == System.Net.NetworkInformation.OperationalStatus.Up)
                    .SelectMany(a => a.GetIPProperties().UnicastAddresses)
                    .Where(a => a.Address.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork)
                    .Select(a => a.Address.ToString());
        }
    }


Installing Cyanogenmod 7.2 on Verizon Droid 2 Running 2.3.4

Friday, December 14, 2012

They said it couldn't be done.  If you install 2.3.4 OTA update from Verizon, you were stuck with a version that was un-rootable.  And rooting is the first step in getting a custom Android build on a phone.  Specifically:

There is currently (as of May 2012) no way to root a Droid 2 with this system version.
As a word of warning, the procedures for getting Cyanodenmod are as scary as the link above, and with 2.3.4, you're really in the deep end.  Here's a summary of what I did.  I'm not providing step-by-step directions for two reasons; first, if you can't get there from the summary, you're probably in over your head and you have a good chance of bricking your phone.  Secondly, I am providing links to my source materials with more details.  I'm also taking you through the method I went, which generally used tools rather than adb commands/doing it by hand.  I purchased two of the tools...I think the $8 I spent was worth it, but go read the wiki if you want to do it the hard way.

Rooting

The rooting procedures on the Cyanogenmod wiki don't work, obviously.  What you need to do is download this iso: http://www.mediafire.com/?mim304k214kl41h.  MD5 is 7baee9c34f6ef7ad0b4fa219ae387c68.  The forum post regarding this is here: http://androidforums.com/droid-x-all-things-root/603489-how-root-2-3-4-4-5-621-magic-md5-does-not-require-milestone-sbf.html.

The ISO can be burned to a bootable CD or USB (using Unetbutin).  I went the USB route.  Once booting off the USB, the scripts on the ISO walk you through what needs to be done.  This was by far the easiest part of the install.

Custom Recovery ROM and Bootloader

Not having gone down these paths before, I didn't have a conceptual understanding of how Android boot works.  That turns out to be pretty important at this point, and it tripped me up for a bit.
  1. Bootloader is run, presumably by the hardware
  2. Bootloader looks around, checks out everything, and loads the ROM image.  Depending on the environment it can run the recovery ROM or the normal boot ROM (your phone).
So, now that the phone is rooted, we need a custom bootloader because the Android one will verify that it's the Verizon image, and we don't want that.  We need a custom recovery ROM as well...one that will let us flash new Android versions.  So, the next step is to install the Droid 2 Recovery Bootstrap by ClockworkMod: http://market.android.com/details?id=com.koushikdutta.droid2.bootstrap.  Also install the ROM Manager: https://play.google.com/store/apps/details?id=com.koushikdutta.rommanager.license&feature=more_from_developer#?t=W251bGwsMSwxLDEwMiwiY29tLmtvdXNoaWtkdXR0YS5yb21tYW5hZ2VyLmxpY2Vuc2UiXQ..

Use the ROM Manager to Flash ClockworkMod Recovery.  Once that's done, install the Recovery Bootstrap (remember, we need the Bootstrap to let us get into our shiny new Recovery).  I missed this part and I kept going into the Android recovery...it was very confusing.

Prepping and Hacking the Cyanogen install

Download:
  1. Cyanogen latest version: http://wiki.cyanogenmod.org/wiki/Devices_Overview#Motorola_Droid_2.  Right now, this is 7.2.0, available at: http://get.cm/get/jenkins/2824/cm-7.2.0-droid2.zip
  2. Google apps (gets you the market): http://goo-inside.me/gapps/gapps-gb-20110828-signed.zip.  That is the mirror link as the primary didn't work for me.  The cyanogenmod page on this is at: http://wiki.cyanogenmod.org/wiki/Latest_Version#Google_Apps
Copy the Google apps zip file to the SD card.  I don't know if this needs to be in the root directory, but Cyanogen instructions mention putting it into root.  It's certainly easier, anyway.

Don't copy the Cyanogen zip file to the SD card yet.  Our last step in this path is that there is a safety check in the install to verify the kernel version so it doesn't accidentally brick your phone.  With 2.3.4 we're safe, but Cyanogen doesn't know this.  So, you need to unzip the Cyanogen zip file above, and alter the /system/etc/check_kernel script (with a decent text editor - not notepad).  Wipe out everything except the first line.  Then the second line should be simply "exit 0".  More details are here: http://forum.cyanogenmod.org/topic/50469-installing-cm-from-234/#entry335479

Zip the file with the changed script and put it on your SD card from the phone.  

Installing Cyanogen

Reboot the phone into ClockworkMod recovery by first powering down.  Then hold power and the X key on the keyboard until it comes up.  You'll see ClockworkMod come up (maybe).  I remember having an issue where nothing seems to happen.  If you have that problem, hit both volume keys at the same time (if that fails, just play around with volume a bit).  You'll then see shiny new text from ClockworkMod recovery, and maybe even an error message.  

At this point, it took me a while to figure out, and the ROM manager was useless for me.  The directions and forums don't know a whole lot about that error message.  You're supposed to be able to use up/down volume to arrow around the options, and power selects the option you want.  However, power just made the phone go blank for me.  I think what's happening is that the droid 2 does not have /dev/tty0 and there is a bug in the recovery ROM where the power button selects /dev/tty0 as the device when pressed.  However, there is another way to navigate...

What I finally figured out is that navigation can operate with the keyboard.  Slide it open, and left = up, right = down (which makes sense since you have the phone in portrait mode.  Up = right, Enter = enter, and Del = back.  This knows where the screen is, and I'm sure it's much more pleasant than trying to navigate around selecting files with volume keys/power.

With that working, you can perform the following steps:
  1. Back up your existing ROM
  2. Select and install the Cyanogen ROM
  3. Select and install the Google Apps ROM (I think this just unpacks apks and puts them in a special directory, but I'm not sure).  I thought it was way weird I was installing two ROMs.
  4. Lastly, wipe all cache/data.  I don't remember if I performed factory reset - I think I did not.  Apparently the phone will get into a "boot loop" unless this step is performed, but if you forget for some reason, you can get out of that problem: http://forum.xda-developers.com/showthread.php?t=1832130
Congrats, Cyanogen is installed!  It's like a brand new phone, so you'll have to sign in, install apps, yada yada.

Other thoughts

I noticed the Alt Lock button does not work after I installed.  Apparently this is a special Motorola hardware key and cannot be used by Cyanogen.  However, pressing Alt twice will do the same thing...I'm sure that's a default Android behavior I had no idea about.  I am also missing the Motorola calendar widget terribly, but I'm sure I'll find a suitable replacement.  Cyanogen is so much better/faster.  They really had me at "Add Access Point" in the wireless settings. ;-)

Update

I noticed that my WiFi does not seem to connect. It connects but the advanced settings says IP address unavailable. I tried a custom kernel but to no avail. It appears to be a dhcp problem only, and I think it is a problem with cm 7.2. I have worked around the problem by using a static IP address and DNS server, but the static DNS settings in the settings don't seem to work. Instead I installed Set DNS and that works properly. For WiFi hotspot support I installed Barnacle WiFi tether, but I haven't had a chance to play with it too much.


Emil's Wicked Cool Blog