DISCONTINUED: Interested parties in purchasing the source code can contact me via email.
Ready for some fun stuff? Let’s play with ABXVision and create the 2D Augmented Reality program shown in the teaser video.
A quick note first.
I did not find a suitable free Webcam framework for OSX. I talked with Christian from MonkeyBread about creating a cheap spinoff of his excellent plugin suite that just contains the Webcam functionality, but it appears it’s not that easy to make. (I’m saving up myself to buy them also, so some donations would be welcome) If someone knows an alternative (free or not) that works well with ABXVision, or finds it a challenge to write one, mail me. I’m sure a lot of OSX peeps would love this.
If no webcam is found this demo will use an included picture to demonstrate the workings of ABXVision. If you already have the MBS plugins, you can make the changes in the code. I’ll point out where it is needed.
Enough about that, Alain. We want some code!
And right you are! Here we go.
We’ll need a couple of steps to go from our grabbed webcam picture to the augmented one.
Here are the steps we’re going to do. You’ll notice there are quite a few and we want to do this real-time. Thanks to our ABXVImage from the previous article, we’ll be able to do so. (Click to enlarge)
First we need to do some initialization in the open event of the form. I’ve documented the code.
' pauze the loop Timer1.Enabled = false ' set DEBUG on to show extra info (like corners) Debug = true ' camera initialization CameraWidth = 320 CameraHeight = 240 ' Windows ONLY! for OSX, this should be from the plugin you are using! #if TargetWin32 MyCam = new ABXVWebCamWindows dim Ret as boolean Ret = MyCam.Start(canCam, cameraWidth, cameraHeight) if ret = false then MsgBox &amp;quot;Unable to initialize the webcam! The demo program will continue using the demo picture.&amp;quot; canCam.Backdrop = DemoPicture end if #else ' write here the code to initialize your webcam plugin ' for this demo, I'll use a static Picture canCam.Backdrop = DemoPicture #endif CameraFocalLength = CameraWidth ' the width of the picture is a good autoFocusLength 'Glyp initialization GlyphSizeMM= 113 ' in milimetre, the real length GlyphSizeMMHalf = glyphSizeMM / 2 ' a list of all the sizes of Glyphs we are going to have in our database GlyphSizes = Array(5,8) ' add some glyph templates to our database AddGlyphTemplate(cornerTemplate, &amp;quot;Alwaysbusy's Corner&amp;quot;, &amp;quot;&amp;quot; + _ &amp;quot;00000&amp;quot; + _ &amp;quot;01100&amp;quot; + _ &amp;quot;00100&amp;quot; +_ &amp;quot;01010&amp;quot; + _ &amp;quot;00000&amp;quot;, 5) AddGlyphTemplate(xojoTemplate, &amp;quot;Made With Xojo&amp;quot;, &amp;quot;&amp;quot; + _ &amp;quot;00000000&amp;quot; + _ &amp;quot;01011010&amp;quot; + _ &amp;quot;01100100&amp;quot; +_ &amp;quot;01011000&amp;quot; + _ &amp;quot;01111010&amp;quot; + _ &amp;quot;01000100&amp;quot; + _ &amp;quot;01111010&amp;quot; + _ &amp;quot;00000000&amp;quot;, 8) ' start the loop Timer1.Enabled = true
1. You’ll see I use a rather small camera size (320×240). Although this framework is heavly optimized, it is written in Xojo, not C++. You can always resize the picture after all the processing is done to get a bigger picture.
AddGlyphTemplate(cornerTemplate, “Alwaysbusy’s Corner”, “” + _
“00000” + _
“01100” + _
“01010” + _
The ‘5’ is the size of the glyph. You can create glyphs up to 23, but the bigger, the more difficult it will be to detect them.
Also beware of creating doubles! These two are exactly the same:
Sub AddGlyphTemplate(PictureTemplate as Picture, Name as String, GlyphValues as String, GlyphSize as integer) ' check if the template is nil. If true, create a random blank one if PictureTemplate = nil then PictureTemplate = new Picture(128,128,32) PictureTemplate.Graphics.ForeColor = RGB(Rnd * 255, Rnd * 255, Rnd * 255) PictureTemplate.Graphics.FillRect 0,0,128,128 end if ' make an ABXVImage from the Xojo picture dim arpic as new ABXVImage arpic.SetPicture(PictureTemplate, ABXVMASKTYPE.MASK_NONE) ' Create the Glyph dim tmpGlyph as ABXVGlyph = ABXVGlyphRecognizer.CreateGlyphTemplateFromString(Name, GlyphValues, GlyphSize) ' set the image to the glyph tmpGlyph.Image = arpic ' add it to the database GlyphDatabase.Append tmpGlyph End Sub
And all we have to do now is following the schema above. We do this in a timer (or thread).
' temporary blok the loop, we're busy! if isBusy then Return isBusy = true ' the center of the image dim cx as Integer = CameraWidth/2 dim cy as Integer = CameraHeight/2 ' some temporary pictures dim BackwardQuadrilateralTransformedPicture as Picture ' take a picture #if TargetWin32 GrabbedPicture = MyCam.GrabPicture ' in case no webcam was attached, use the demo picture if GrabbedPicture = nil then GrabbedPicture = new Picture(CameraWidth, CameraHeight, 32) GrabbedPicture.Graphics.DrawPicture DemoPicture,0,0 end if #else ' grab the picture with your WebCam plugin ' for the demo, the fixed demo picture is used GrabbedPicture = new Picture(CameraWidth, CameraHeight, 32) GrabbedPicture.Graphics.DrawPicture DemoPicture,0,0 #endif ' some temporaty ABVXImages dim GrabbedImage, GrayImage, QuadrilateralTransformedImage, BackwardQuadrilateralTransformedImage as ABXVImage ' convert the grabbed picture to a ABXVImage for processing GrabbedImage = new ABXVImage GrabbedImage.SetPicture(GrabbedPicture, ABXVMASKTYPE.MASK_NONE) ' convert to a grayscaled image. We want to keep this for later and continue on the GrabbedImage GrabbedImage.GrayScaleTo8Bit(ABXVGRAYSCALETYPE.GRAYSCALE_BT709) GrayImage = GrabbedImage.Clone ' run some filters ' first a difference edge filter ABXVEdgeDetector.DifferenceEdgeDetect(GrabbedImage) ' next change everything to black or White (2bit), using a threshold ABXVBinarization.Threshold(GrabbedImage, 40) ' lets find some blobs in our B/W picture. dim BlobCounter as new ABXVBlobCounter ' the size must minimum 32 pixels . FilterBlobs = true means filter all smaller ones out BlobCounter.minWidth = 32 BlobCounter.minHeight = 32 BlobCounter.FilterBlobs = true BlobCounter.FindBlobs(GrabbedImage) ' get the found blobs dim Blobs(-1) as ABXVBlob = BlobCounter.GetObjectsInformation ' a temporary blob to do the loop dim Blob as ABXVBlob ' temporary tables to hold the edgepoints of a blog (the border, so to day) dim EdgePoints(-1) as ABXVIntPoint dim LeftEdgePoints() as ABXVIntPoint dim RightEdgePoints() as ABXVIntPoint ' temporary table to hold the corners dim Corners(-1) as ABXVIntPoint ' Our final table that will hold all the found Glyphs dim FoundGlyphs(-1) as ABXVGlyph ' The glyph we found dim FoundGlyph as ABXVGlyph ' The Bounding Rectangle of the Found Glyph dim BoundingRect as ABXVRectangle for each Blob in Blobs ' find all the EdgePoints EdgePoints = BlobCounter.GetBlobsEdgePoints(Blob) ' check if the EdgePoints form a Quadrilateral, if yes, return the corners if ABXVShapeChecker.IsQuadrilateral(EdgePoints, Corners) then ' It's a Quadrilateral, let's find the left and right edges BlobCounter.GetBlobsLeftAndRightEdges(Blob, LeftEdgePoints, RightEdgePoints) ' using our gray image, we check if the difference of the edges is big enough with the surrounding pixels (B&amp;lt;-&amp;gt;W) dim Difference as Double = ABXVGlyphRecognizer.CalculateAverageEdgesBrightnessDifference(LeftEdgePoints, RightEdgePoints, GrayImage) if Difference &amp;gt; 20 then ' Yes it is! ' Transform the found Quadrilateral to a 2D Glyph QuadrilateralTransformedImage = ABXVTransform.QuadrilateralTransform(grayimage, corners,GlyphSizeMM,GlyphSizeMM, true) ' Run a Otsu Threshold filter, returning a B/W 2Bit image ABXVAdaptiveBinarization.OtsuThreshold(QuadrilateralTransformedImage) ' look if the found Glyph matches one in our database FoundGlyph = ABXVGlyphRecognizer.FindGlyph(QuadrilateralTransformedImage, Corners, GlyphDatabase, GlyphSizes) if FoundGlyph &amp;lt;&amp;gt; nil then if FoundGlyph.RecognizedGlyph &amp;lt;&amp;gt; nil then ' yes, found and recognized! ' 2D augmented reality ' we do a Backward Quadrilateral Transformation with the picture we want to show over the glyph. A rectangle is returned where it should be placed BackwardQuadrilateralTransformedImage = ABXVTransform.BackwardQuadrilateralTransform(FoundGlyph.RecognizedGlyph.Image, FoundGlyph.RecognizedQuadrilateral, true, BoundingRect) ' translate the ABVXImage to a Xojo picture BackwardQuadrilateralTransformedPicture = BackwardQuadrilateralTransformedImage.GetPicture ' and draw it on our original picture, using the returned BoundingRect GrabbedPicture.Graphics.DrawPicture BackwardQuadrilateralTransformedPicture,BoundingRect.Left, BoundingRect.Top if Debug then ' if debug, draw the edges in red dim point as ABXVIntPoint for each point in edgePoints GrabbedPicture.Graphics.Pixel(point.x,point.y) = &amp;amp;cFF0000 next end if ' and add it to our found list to do something with it later FoundGlyphs.Append FoundGlyph end if end if end if end if next ' draw the result to the canvas canAR.Refresh ' ok we're done. Next! isBusy = false
And that’s it! In just a 100 lines of code, we created magic! Check out the help for more information on what each method does. A lot of methods contain reference links to the theory behind the function.
And what’s next? 3D Augmented Reality of course. Alwyn Bester has gracefully offered to expand the excellent X3 Core framework so that it will work very well with ABXVision.
So, a lot more fun to come in the upcoming weeks!
Download the latest framework from the ABXVision page to get version 1.0.2 with the 2D Augmented Reality demo.
If it is the first time you use ABXVision, read the Getting Started document.