Using MySQL variables to limit a select query

I discovered today when writing a stored procedure for a web reporting application that MySQL doesn’t support using variables for the LIMIT clause in a SELECT statement. I didn’t really want to write a dynamic query, one workaround that I found while Googling looked promising:

SET SQL_SELECT_LIMIT = RowLimit (can be a variable)
SELECT ….
SET SQL_SELECT_LIMIT = default

However it is applied at a global level and I needed different limits within a sequence of four queries formed into a union to get the top N results for a series of categories. Because I was giving the user the limited choice of retrieving the top 10, 20, 50 or 100 items I ending up using a CASE statement with each WHEN clause containing a duplicate of the entire query. Certainly not a pretty solution but works and is reasonably maintainable by leaving the LIMIT clause on a seperate line so any changes can be cut & pased down the procedure.

Manifold GIS IMS (Internet Map Server) error 0x80004005

I’m working on an iPhone 3GS mapping application using Manifold as a map tile server via the Web Map Service (WMS) protocol. I did initial development using a Vista x64 development machine on my intranet via wifi but had considerable difficulty moving to my Windows Server 2008 64-bit machine so that I could access the map tiles remotely via the 3G network. It’s been a while since I’d deployed a Manifold IMS solution but I remembered the usual steps:

  • Web site must be running under 32-bit mode because the COM object is 32-bit only
  • Map files, config.txt and any linked resources must be available to the ‘NETWORK SERVICE’ account
  • Same account needs access to Windows\serviceprofiles\networkservice\AppData\Local\Temp

    After my initial attempts failed I did numerous searches of the Manifold site and other resources and worked through the Manifold “Problems with the Internet Map Server” guide but kept getting the 80004005 error that I thought pointed at a permissions error. I thought to install the excellent SysInternals Process Monitor utility and the first error that became apparent was a file permissions error writing to the file “c:\Manifold.log”. I created a blank file with that name and granted full control permissions to the NETWORK SERVICE account used by IIS 7 and found several errors including the following:

    Invalid component name
    Incompatible component type
    Can’t generate unique name for temporary file

    Creating a new blank map file solved the problem so it appeared to be a problem within the Manifold map file itself, I decided to copy back a fresh copy of the map from my development machine so I could try removing a few elements until it worked but found it worked first go with the fresh copy. I didn’t modify anything with the map file so can only conclude it got corrupted at some initial stage while I had configuration problems. Anyway to avoid similar problems in the future I’ll make sure that during Manifold IMS deployment I always include a blank “c:\Manifold.log” file with appropriate permissions and be careful to archive any changes no matter how small they seem in source code control as a recovery point.

    I love Manifold GIS in general and it represents great value for money, but it would be nice if the object model was structured so that it would throw back a response code in the form of a text message on the call to MapServer.Create so it could be passed back to the client browser. Additionally rather than the non-standard practice of writing a text file to the root directory it would also be nice to see it write a meaningful errors to the Windows application event log.

  • Using JET OLE DB under ASP and Vista x64

    I’m currently performing some work on classic ASP pages that during development are retrieving data from an existing Microsoft Access database. After receiving the following error I found that Microsoft haven’t and have no plans to release JET OLE DB drivers for 64-bit platforms. Fair enough it’s hardly the way of the future but I wanted to be able to perform some development on the pages before migrating to SQL/Server.

    Provider=Microsoft.Jet.OLEDB.4.0
    ADODB.Connection error ‘800a0e7a’
    Provider cannot be found. It may not be properly installed.

    After some digging around I found the solution was to change the IIS 7 settings to run the application in 32-bit mode. That can be performed under Vista x64 using the following steps:

  • Click Start > All programs > Administrative tools > Internet Information Server (IIS) Manager
  • Under the machine name click on Application Pools.
  • Right-mouse click on DefaultAppPool and select Advanced Settings…
  • Under the general section set “Enable 32-Bit Applications” to true.
  • Press OK and IIS should now be running in 32-bit mode, no other actions are necessary.
  • Getting detailed ASP errors under IIS 7.0

    Some time ago I was doing maintenance work on some classic ASP pages and couldn’t get detailed error reports sent back to the browser. Using Internet Explorer I disabled “Show friendly HTTP error messages” but still received the generic HTTP 500 type error messages from IIS 7.0 running under Windows Server 2008. The same also occurred under FireFox. The modifications I was making to the ASP pages were fairly limited so I gave up on trying to get it to work and ending up sticking with examining the IIS logs on the server for debugging information.

    Today I started work on maintaining some more complex ASP pages developed by someone else under Vista x64 and IIS 7 and encountered the same problem. I also noticed Vista locks the IIS log files while the web server is running which further added to my determination to get ASP errors including line numbers back to the browser. It appears setting “Send Errors To Browser” in “Debugging Properties” for the ASP properties doesn’t work at a virtual directory level, it only works when enabled at the root level for the entire site. Enabling it at the higher level got back the detailed error reports as per the default bahaviour of IIS 6.

    ASP site map generator

    This is a simple ASP site map generator that I wrote some time ago using VBScript, it’s for a human-readable sitemap not the XML sitemaps used by Google and other search engines. It uses the Scripting.FileSystemObject to scan the same directory level that the ASP is placed at and returns a link to any htm or html pages found along with the last modification time and the title of the page. It was fairly quickly written so the parsing used for the title tag is just to suit the format I was using and it might need some other tweaking to suit your own site.

    I’ve just loaded a copy at http://www.satsleuth.com/site_map.asp so you can take a look at the script in action. Because I don’t actively use the script anymore I haven’t updated to XHTML 1.0 Strict so on the W3C Markup Validation Service it will only pass as HTML 4.01 Transitional. Currently presentation elements are embedded in the HTML rather than a CSS style sheet. Here is the main logic for the page:

    <table border="1">
    <tr><td><b>Page name</b></td><td ALIGN="RIGHT"><b>Modified</b></td><td><b>Title</b></td></tr>
    <% 
    	Dim objFSO, objFolder, objFiles, objFile, strFile, Title, CurLine, LCaseLine, TitlePos, PageCount, FontColour
    
    	Private Sub GetTitle(FileName)
    		Dim fso, file, strPhysicalPath
    
    		Title = ""
    		set fso = Server.Createobject("Scripting.FileSystemObject") 
    		strPhysicalPath = Server.MapPath(FileName) 
    		set file = fso.opentextfile(strPhysicalPath, 1)
    		do until file.AtEndOfStream or Len(Title) > 0
    			CurLine = file.ReadLine
    			LCaseLine = LCase(CurLine)
    			TitlePos = InStr(LCaseLine, "<title>")
    			if TitlePos then
    				Title = Right(CurLine, Len(CurLine) - TitlePos - 6)
    				Title = Left(Title, InStr(LCaseLine, "</title>") - 8 )
    			end if
    		loop
    		file.close
    	End Sub
    
    	Set objFSO = Server.CreateObject("Scripting.FileSystemObject") 
    	Set objFolder = objFSO.GetFolder(Server.MapPath("."))
    	Set objFiles = objFolder.Files
    	PageCount = 0
    	For Each objFile in objFiles 
    		strFile = LCase(objFile.Name) 
    		If Right(strFile, 4) = ".htm" or Right(strFile, 5) = ".html"  then
    			GetTitle(objFile.Name)
    			if Len(Title) > 0 then
    				PageCount = PageCount + 1
    				if objFile.DateLastModified >= Date - 7 then
    					FontColour = "<font color=""#FF0000"">"
    				else
    					FontColour = "<font color=""#000000"">"
    				end if
    				Response.Write("<tr><td><a href=""" & strFile & """>" & _
    					strFile & "</a></td><td align=""RIGHT"">" & _
    					FontColour & DateValue(objFile.DateLastModified) & "</font>" & _
    					"</td><td>" & Title & "</td></tr>")
    			end if
    		end if
    	next
    	Response.Write("<tr><td>&nbsp;</td><td>&nbsp;</td><td><b>Total pages: " & PageCount & "</b></td></tr>")
    %>
    </table>
    

    Creating compressed ZIP files using VBScript

    SQL/Server database backups tend to compress fairly well, in my experience typically down to around 25% of the original file size. However I discovered backup data compression is only a feature of SQL/Server 2008 Enterprise Edition and it’s not a feature that inspired me to spend thousands extra on a license. I’ve attached a VBScript procedure that I wrote to automatically compress my SQL/Server backups but of course it can be used for any application where you’d like to compress the entire contents of a directory automatically.

    I can’t take credit for the innovative technique of creating a ZIP header in code to make Windows create a compressed ZIP folder. That was a part of a code snippet I found elsewhere quite some time ago. For some reason the MoveHere method of Shell.NameSpace didn’t seem to remove the source file so when move mode is selected I stuck with deleting the source file afterwards. This version displays ZIP compression progress in a dialog box but you could use “ZipFile.CopyHere InFilename, 4” to disable the progress dialog box.

    ' VBScript to move or copy all files in a folder to a compressed ZIP file
    
    Option Explicit
    Const MoveMode = False
    Const BackupDir = "E:\DB Backups"
    Const TimeoutMins = 10 ' Timeout for individual file compression operation
    
    Sub MoveToZip(InFilename, OutFilename)
    	Dim FSO : Set FSO = CreateObject("Scripting.FileSystemObject")
    	Dim Timeout : Timeout = 0
    	FSO.CreateTextFile(OutFilename, true).WriteLine "PK" & Chr(5) & Chr(6) & String(18, 0)
    	Dim Shell : Set Shell = CreateObject("Shell.Application")
    	Dim ZipFile: Set ZipFile = Shell.NameSpace(OutFilename)
    	ZipFile.CopyHere InFilename
    	Do Until ZipFile.items.Count = 1  or Timeout > TimeoutMins * 600
    		Wscript.Sleep 100
    		Timeout = Timeout + 1
    	Loop
    	If MoveMode and ZipFile.items.Count = 1 Then FSO.DeleteFile(InFilename)
    	Set Shell = Nothing
    	Set FSO = Nothing
    	Set ZipFile = Nothing
    End Sub
    
    Dim FSO : set FSO = CreateObject("Scripting.FileSystemObject")
    Dim Folder : Set Folder = FSO.GetFolder(BackupDir)
    Dim Files : Set Files = Folder.Files
    Dim File
    For Each File In Files
    	If InStr(UCase(File.Name), ".ZIP") = 0 Then
    		MoveToZip BackupDir & "\" & File.Name, BackupDir & "\" & FSO.GetBaseName(File.Name) & ".zip"
    	End If
    Next
    

    Posting to a forum programmatically using Delphi

    I’m a member of a web site where we run various games in the forums and for several that can be tedious to keep track of manually I’ve written some code in Borland Delphi to automate the task. The forum itself is phpBB and I use the web browser component included with Delphi to edit posts directly in the forum. Being able to post to a website as though a user submitted a form is handy for these sorts to tasks where you don’t have access to the underlying forum database. I’ve posted the more interesting parts of the code that format the request and setup the correct headers:

    var
      strData: String;
      PostData, Headers: OleVariant;
    ...................
    strData := 'message=' + HTTPEncode(MessageData) + '&notify=on&mode=editpost&p=' + IntToStr(PostID) + '&post=+Post+';
    
    PostData := VarArrayCreate([0, Length(strData) - 1], varByte);
    
    for i := 1 to Length(strData) do
      PostData[i-1] := Ord(strData[i]);
    
    Headers := 'Content-Type: application/x-www-form-urlencoded' + #10#13;
    
    WebBrowser.Navigate('http://example.com/posting.php?mode=reply&t=' + MainTopicID.Text, EmptyParam, EmptyParam, PostData, Headers);
    

    GPS / Asset tracking AJAX web page

    I’ve just completed an AJAX based GPS tracking page for a client based on the OpenLayers Javascript library using the OpenStreetMap map tile server for data, although it can be easily be adapted to other data sources. The following page offers a login to the demonstration system for evaluation:

    GPS tracking page

    Data is served from an SQL database with a modular set of stored procedures translating data from its raw format allowing integratation with a wide range of data sources and GPS tracking devices. Columns displayed can be customised on a per-user basis and dynamically generated SQL queries allow great flexibility in the data generated. Please contact me if you are interesting in licensing the software for your own site.

    MySQL: Access denied for user ‘root’@’localhost’ (using password: YES)

    After Googling and reading through all the usual reasons for this error when trying to connect to a MySQL database I was starting to get frustrated because I could connect to MySQL server using several GUI applications so knew my root password was correct. The problem only cropped up when I wanted to launch MySQL from a bash script to execute a stored procedure via a cron job. When I stepped back and took a look it turned out to be a fairly obvious problem, I was trying as per this example (obviously not the real password):

    mysql -u root -pABC$123

    I normally like to include a few special characters in my passwords and use more than eight characters to negate the possibility of them being cracked using reasonable length rainbow tables. Of course the dollar sign was causing bash environment variable substitution so it was simply a matter of prefixing the dollar sign with a backslash:

    mysql -u root -pABC\$123

    OSM planet.osm disk requirements

    After loading the planet.osm file (planet-090916.osm.bz2) into an Ubuntu 9.04 virtual machine the total disk usage was 133GB after the import. The total dropped to 115GB after doing a VMWare shrink operation however it looks like to account for future near-term expansion of the OSM data I should allow for at least 200GB of storage. I’d originally hoped to store the data on a small but fast SAS 15K RAID array on my server but the current version will only barely fit so I’ll need to consider other alternatives. It looks like costs are dropping fairly fast for some of the larger 15K SAS drives so I might defer the decision for a few months as the system isn’t required for production until the start of the year. Meanwhile I’ll just slip a 1TB SATA drive into the server for testing.