Categories
Photography

Canon 7D native ISO tests

Recently I picked up a Canon 7D which I’m really enjoying so far. While researching the camera I found various references to both 100 and 160 being the native ISO and as I do quite a bit of still photography from a tripod was keen to determine the optimum ISO setting and also the best ISO multiples to use for available light shooting. Between various forum posts there seemed to be about a 50/50 mix of people suggesting one or the other was the native ISO so I decided to do my own test. I printed an ISO 12233 test chart replica from Cornell University on an A4 piece of paper and set it up inside a lightbox in a room with no direct sunlight. I shot using the 7D with a Canon 70-200mm f/2.8L IS USM II lens at f/2.8 at around 100mm focal length, tripod mounted with a remote cable release, manually focused once at the start of the sequence with the camera set to aperture priority mode. While I’m sure there are plenty of things that could be done better I thought this was typical of the kind of lighting situation where I’d be tempted to knock up the ISO speed. I imported the RAW files in Adobe Lightroom 3 beta 2 and didn’t perform any operations other than changing the colour balance and exposure equally on all shots to lighten them up for printing.

The results? ISO 100 and 160 were very close, but I felt 160 was just a little more defined so I might settle on that for tripod shooting. I don’t have an expert eye but I couldn’t spot any nasty steps in the ISO range that other’s have reported, in my mind it was a pretty even transition with the each ISO speed. ISO 400 remained excellent so I wouldn’t hesitate to go that high for quality shots. What I’d set out to find is if automatic ISO was a useful mode and I feel that it is. From what I gather Canon don’t release specific details on the native ISO of their cameras but the results made me wonder if on a modern DSLR it’s a myth, I’d assume the ISO setting equates to a sampling period for the sensor and can’t think of any reason it couldn’t be infinitely variable over a range. Of course at the lower end of the range there could be an optimum level such as 160 below which the sensor becomes saturated and needs to be attenuated which may explain why ISO 160 appeared marginally better than ISO 100.

Categories
GPS / GIS SQL / Database

Geocoding using OSM data and SQL/Server 2008

I’ve just completed importing the OpenStreetMap data into SQL/Server 2008 using the new geography data types. Originally I linked the data to Manifold GIS to verify the import process but have just started on my first real project using the data which is address geocoding for a GPS tracking site located in Australia. The OSM data doesn’t contain enough street number information to be useful for most areas but the aim is to provide nearest street names at no charge, versus some local commercial geocoding services that charge around ten cents per address.

Loading the OSM ‘ways’ as a series of around 250,000 linestrings seems to be giving a performance of around 40 geocoding operations per second. As you’d expect performance of the STDistance method depends heavily on the distance, so in the SQL/Server stored procedure I first start by locating streets within 100 meters, followed by 1Km, 10Km and finally 100Km. The series of queries only attempts to find street names not matched by the previous query, and for Australia these numbers seem to give a match rate of about 90%, 9%, 1% and not much for the final distance that results in good performance.

For optimisation of the spatial index I found that changing the default of medium for each level in the index grid to high gave approximately a 100% increase in performance. It makes sense that the finest level of granularity would work best when looking for the nearest distance to points. The following was what I ended up using for the spatial index definition, changing the cells per object didn’t seem to have a great deal of effect for this dataset / application so I left it at the default value of 16.

CREATE SPATIAL INDEX [spatial_IX] ON [dbo].[Ways] 
(
	[LineString]
) USING  GEOGRAPHY_GRID 
WITH (
    GRIDS =(LEVEL_1 = HIGH,LEVEL_2 = HIGH,LEVEL_3 = HIGH,LEVEL_4 = HIGH), 
    CELLS_PER_OBJECT = 16, PAD_INDEX  = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, 
    ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) 
ON [PRIMARY]
Categories
GPS / GIS Programming

JScript to automatically retrieve / check web page

Recently I discovered a problem with Manifold WMS where if the page wasn’t accessed for a while it would cause the w3wp.exe IIS worker thread to loop. Until I find out the underlying cause I’ve written the map_keepalive.wsf script below to automatically retrieve a web page and scheduled it to run every 5 minutes. I assume something gets spun down and the concurrent WMS requests cause a problem, the map_test.asp page that this script retrieves is just a default IMS page generated from within Manifold.

<package>
  <job id="js">
     <script language="JScript">
	var finished = false;
	var timeout = 0;
	var timeoutSecs = 30;
	var forAppending = 8;
	var xmlhttp = WScript.CreateObject("MSXML2.XMLHTTP");
	xmlhttp.open("GET", "http://www.satsleuth.com", false);
	xmlhttp.onreadystatechange = function() {
		if(xmlhttp.readyState == 4)
			finished = true;
	}
	try {
		xmlhttp.Send("");
	} catch (e) {
	}
	while ((!finished) && (timeout < timeoutSecs))
	{
		timeout++;
		WScript.Sleep(1000);
	}
	var msg = ""
	if (!finished)
		msg = "Timeout occurred";
	else if (xmlhttp.status != 200)
		msg = "Error retrieving map_test.asp, error code: " + xmlhttp.status;
	if (msg != "") {
		var oFSO = WScript.CreateObject("Scripting.FileSystemObject");
		var oFile = oFSO.OpenTextFile("map_keepalive.log", forAppending, true);
		var dt = new Date();
		oFile.WriteLine(dt.toLocaleString() + ": " + msg);
		oFile.Close();
	}
      </script>
   </job>
</package>
Categories
SQL / Database

Using MySQL variables to limit a select query

I discovered today when writing a stored procedure for a web reporting application that MySQL doesn’t support using variables for the LIMIT clause in a SELECT statement. I didn’t really want to write a dynamic query, one workaround that I found while Googling looked promising:

SET SQL_SELECT_LIMIT = RowLimit (can be a variable)
SELECT ….
SET SQL_SELECT_LIMIT = default

However it is applied at a global level and I needed different limits within a sequence of four queries formed into a union to get the top N results for a series of categories. Because I was giving the user the limited choice of retrieving the top 10, 20, 50 or 100 items I ending up using a CASE statement with each WHEN clause containing a duplicate of the entire query. Certainly not a pretty solution but works and is reasonably maintainable by leaving the LIMIT clause on a seperate line so any changes can be cut & pased down the procedure.

Categories
GPS / GIS Programming

Manifold GIS IMS (Internet Map Server) error 0x80004005

I’m working on an iPhone 3GS mapping application using Manifold as a map tile server via the Web Map Service (WMS) protocol. I did initial development using a Vista x64 development machine on my intranet via wifi but had considerable difficulty moving to my Windows Server 2008 64-bit machine so that I could access the map tiles remotely via the 3G network. It’s been a while since I’d deployed a Manifold IMS solution but I remembered the usual steps:

  • Web site must be running under 32-bit mode because the COM object is 32-bit only
  • Map files, config.txt and any linked resources must be available to the ‘NETWORK SERVICE’ account
  • Same account needs access to Windows\serviceprofiles\networkservice\AppData\Local\Temp

    After my initial attempts failed I did numerous searches of the Manifold site and other resources and worked through the Manifold “Problems with the Internet Map Server” guide but kept getting the 80004005 error that I thought pointed at a permissions error. I thought to install the excellent SysInternals Process Monitor utility and the first error that became apparent was a file permissions error writing to the file “c:\Manifold.log”. I created a blank file with that name and granted full control permissions to the NETWORK SERVICE account used by IIS 7 and found several errors including the following:

    Invalid component name
    Incompatible component type
    Can’t generate unique name for temporary file

    Creating a new blank map file solved the problem so it appeared to be a problem within the Manifold map file itself, I decided to copy back a fresh copy of the map from my development machine so I could try removing a few elements until it worked but found it worked first go with the fresh copy. I didn’t modify anything with the map file so can only conclude it got corrupted at some initial stage while I had configuration problems. Anyway to avoid similar problems in the future I’ll make sure that during Manifold IMS deployment I always include a blank “c:\Manifold.log” file with appropriate permissions and be careful to archive any changes no matter how small they seem in source code control as a recovery point.

    I love Manifold GIS in general and it represents great value for money, but it would be nice if the object model was structured so that it would throw back a response code in the form of a text message on the call to MapServer.Create so it could be passed back to the client browser. Additionally rather than the non-standard practice of writing a text file to the root directory it would also be nice to see it write a meaningful errors to the Windows application event log.

  • Categories
    Programming SQL / Database

    Using JET OLE DB under ASP and Vista x64

    I’m currently performing some work on classic ASP pages that during development are retrieving data from an existing Microsoft Access database. After receiving the following error I found that Microsoft haven’t and have no plans to release JET OLE DB drivers for 64-bit platforms. Fair enough it’s hardly the way of the future but I wanted to be able to perform some development on the pages before migrating to SQL/Server.

    Provider=Microsoft.Jet.OLEDB.4.0
    ADODB.Connection error ‘800a0e7a’
    Provider cannot be found. It may not be properly installed.

    After some digging around I found the solution was to change the IIS 7 settings to run the application in 32-bit mode. That can be performed under Vista x64 using the following steps:

  • Click Start > All programs > Administrative tools > Internet Information Server (IIS) Manager
  • Under the machine name click on Application Pools.
  • Right-mouse click on DefaultAppPool and select Advanced Settings…
  • Under the general section set “Enable 32-Bit Applications” to true.
  • Press OK and IIS should now be running in 32-bit mode, no other actions are necessary.
  • Categories
    Programming

    Getting detailed ASP errors under IIS 7.0

    Some time ago I was doing maintenance work on some classic ASP pages and couldn’t get detailed error reports sent back to the browser. Using Internet Explorer I disabled “Show friendly HTTP error messages” but still received the generic HTTP 500 type error messages from IIS 7.0 running under Windows Server 2008. The same also occurred under FireFox. The modifications I was making to the ASP pages were fairly limited so I gave up on trying to get it to work and ending up sticking with examining the IIS logs on the server for debugging information.

    Today I started work on maintaining some more complex ASP pages developed by someone else under Vista x64 and IIS 7 and encountered the same problem. I also noticed Vista locks the IIS log files while the web server is running which further added to my determination to get ASP errors including line numbers back to the browser. It appears setting “Send Errors To Browser” in “Debugging Properties” for the ASP properties doesn’t work at a virtual directory level, it only works when enabled at the root level for the entire site. Enabling it at the higher level got back the detailed error reports as per the default bahaviour of IIS 6.

    Categories
    Programming

    ASP site map generator

    This is a simple ASP site map generator that I wrote some time ago using VBScript, it’s for a human-readable sitemap not the XML sitemaps used by Google and other search engines. It uses the Scripting.FileSystemObject to scan the same directory level that the ASP is placed at and returns a link to any htm or html pages found along with the last modification time and the title of the page. It was fairly quickly written so the parsing used for the title tag is just to suit the format I was using and it might need some other tweaking to suit your own site.

    Because I don’t actively use the script anymore I haven’t updated to XHTML 1.0 Strict so on the W3C Markup Validation Service it will only pass as HTML 4.01 Transitional. Currently presentation elements are embedded in the HTML rather than a CSS style sheet. Here is the main logic for the page:

    &amp;lt;table border="1"&amp;gt;
    &amp;lt;tr&amp;gt;&amp;lt;td&amp;gt;&amp;lt;b&amp;gt;Page name&amp;lt;/b&amp;gt;&amp;lt;/td&amp;gt;&amp;lt;td ALIGN="RIGHT"&amp;gt;&amp;lt;b&amp;gt;Modified&amp;lt;/b&amp;gt;&amp;lt;/td&amp;gt;&amp;lt;td&amp;gt;&amp;lt;b&amp;gt;Title&amp;lt;/b&amp;gt;&amp;lt;/td&amp;gt;&amp;lt;/tr&amp;gt;
    &amp;lt;%
    	Dim objFSO, objFolder, objFiles, objFile, strFile, Title, CurLine, LCaseLine, TitlePos, PageCount, FontColour
    
    	Private Sub GetTitle(FileName)
    		Dim fso, file, strPhysicalPath
    
    		Title = ""
    		set fso = Server.Createobject("Scripting.FileSystemObject")
    		strPhysicalPath = Server.MapPath(FileName)
    		set file = fso.opentextfile(strPhysicalPath, 1)
    		do until file.AtEndOfStream or Len(Title) &amp;gt; 0
    			CurLine = file.ReadLine
    			LCaseLine = LCase(CurLine)
    			TitlePos = InStr(LCaseLine, "&amp;lt;title&amp;gt;")
    			if TitlePos then
    				Title = Right(CurLine, Len(CurLine) - TitlePos - 6)
    				Title = Left(Title, InStr(LCaseLine, "&amp;lt;/title&amp;gt;") - 8 )
    			end if
    		loop
    		file.close
    	End Sub
    
    	Set objFSO = Server.CreateObject("Scripting.FileSystemObject")
    	Set objFolder = objFSO.GetFolder(Server.MapPath("."))
    	Set objFiles = objFolder.Files
    	PageCount = 0
    	For Each objFile in objFiles
    		strFile = LCase(objFile.Name)
    		If Right(strFile, 4) = ".htm" or Right(strFile, 5) = ".html"  then
    			GetTitle(objFile.Name)
    			if Len(Title) &amp;gt; 0 then
    				PageCount = PageCount + 1
    				if objFile.DateLastModified &amp;gt;= Date - 7 then
    					FontColour = "&amp;lt;font color=""#FF0000""&amp;gt;"
    				else
    					FontColour = "&amp;lt;font color=""#000000""&amp;gt;"
    				end if
    				Response.Write("&amp;lt;tr&amp;gt;&amp;lt;td&amp;gt;&amp;lt;a href=""" &amp;amp; strFile &amp;amp; """&amp;gt;" &amp;amp; _
    					strFile &amp;amp; "&amp;lt;/a&amp;gt;&amp;lt;/td&amp;gt;&amp;lt;td align=""RIGHT""&amp;gt;" &amp;amp; _
    					FontColour &amp;amp; DateValue(objFile.DateLastModified) &amp;amp; "&amp;lt;/font&amp;gt;" &amp;amp; _
    					"&amp;lt;/td&amp;gt;&amp;lt;td&amp;gt;" &amp;amp; Title &amp;amp; "&amp;lt;/td&amp;gt;&amp;lt;/tr&amp;gt;")
    			end if
    		end if
    	next
    	Response.Write("&amp;lt;tr&amp;gt;&amp;lt;td&amp;gt;&amp;amp;nbsp;&amp;lt;/td&amp;gt;&amp;lt;td&amp;gt;&amp;amp;nbsp;&amp;lt;/td&amp;gt;&amp;lt;td&amp;gt;&amp;lt;b&amp;gt;Total pages: " &amp;amp; PageCount &amp;amp; "&amp;lt;/b&amp;gt;&amp;lt;/td&amp;gt;&amp;lt;/tr&amp;gt;")
    %&amp;gt;
    &amp;lt;/table&amp;gt;
    
    Categories
    Programming SQL / Database

    Creating compressed ZIP files using VBScript

    SQL/Server database backups tend to compress fairly well, in my experience typically down to around 25% of the original file size. However I discovered backup data compression is only a feature of SQL/Server 2008 Enterprise Edition and it’s not a feature that inspired me to spend thousands extra on a license. I’ve attached a VBScript procedure that I wrote to automatically compress my SQL/Server backups but of course it can be used for any application where you’d like to compress the entire contents of a directory automatically.

    I can’t take credit for the innovative technique of creating a ZIP header in code to make Windows create a compressed ZIP folder. That was a part of a code snippet I found elsewhere quite some time ago. For some reason the MoveHere method of Shell.NameSpace didn’t seem to remove the source file so when move mode is selected I stuck with deleting the source file afterwards. This version displays ZIP compression progress in a dialog box but you could use “ZipFile.CopyHere InFilename, 4” to disable the progress dialog box.

    ' VBScript to move or copy all files in a folder to a compressed ZIP file
    
    Option Explicit
    Const MoveMode = False
    Const BackupDir = "E:\DB Backups"
    Const TimeoutMins = 10 ' Timeout for individual file compression operation
    
    Sub MoveToZip(InFilename, OutFilename)
    	Dim FSO : Set FSO = CreateObject("Scripting.FileSystemObject")
    	Dim Timeout : Timeout = 0
    	FSO.CreateTextFile(OutFilename, true).WriteLine "PK" & Chr(5) & Chr(6) & String(18, 0)
    	Dim Shell : Set Shell = CreateObject("Shell.Application")
    	Dim ZipFile: Set ZipFile = Shell.NameSpace(OutFilename)
    	ZipFile.CopyHere InFilename
    	Do Until ZipFile.items.Count = 1  or Timeout > TimeoutMins * 600
    		Wscript.Sleep 100
    		Timeout = Timeout + 1
    	Loop
    	If MoveMode and ZipFile.items.Count = 1 Then FSO.DeleteFile(InFilename)
    	Set Shell = Nothing
    	Set FSO = Nothing
    	Set ZipFile = Nothing
    End Sub
    
    Dim FSO : set FSO = CreateObject("Scripting.FileSystemObject")
    Dim Folder : Set Folder = FSO.GetFolder(BackupDir)
    Dim Files : Set Files = Folder.Files
    Dim File
    For Each File In Files
    	If InStr(UCase(File.Name), ".ZIP") = 0 Then
    		MoveToZip BackupDir & "\" & File.Name, BackupDir & "\" & FSO.GetBaseName(File.Name) & ".zip"
    	End If
    Next
    
    Categories
    Programming

    Posting to a forum programmatically using Delphi

    I’m a member of a web site where we run various games in the forums and for several that can be tedious to keep track of manually I’ve written some code in Borland Delphi to automate the task. The forum itself is phpBB and I use the web browser component included with Delphi to edit posts directly in the forum. Being able to post to a website as though a user submitted a form is handy for these sorts to tasks where you don’t have access to the underlying forum database. I’ve posted the more interesting parts of the code that format the request and setup the correct headers:

    var
      strData: String;
      PostData, Headers: OleVariant;
    ...................
    strData := 'message=' + HTTPEncode(MessageData) + '&notify=on&mode=editpost&p=' + IntToStr(PostID) + '&post=+Post+';
    
    PostData := VarArrayCreate([0, Length(strData) - 1], varByte);
    
    for i := 1 to Length(strData) do
      PostData[i-1] := Ord(strData[i]);
    
    Headers := 'Content-Type: application/x-www-form-urlencoded' + #10#13;
    
    WebBrowser.Navigate('http://example.com/posting.php?mode=reply&t=' + MainTopicID.Text, EmptyParam, EmptyParam, PostData, Headers);