Xojo ABXVision – 2D Augmented Reality example

DISCONTINUED: Interested parties in purchasing the source code can contact me via email.

2D Augmented Reality Example

2D Augmented Reality Example

Ready for some fun stuff? Let’s play with ABXVision and create the 2D Augmented Reality program shown in the teaser video.

A quick note first.
I did not find a suitable free Webcam framework for OSX. I talked with Christian from MonkeyBread about creating a cheap spinoff of his excellent plugin suite that just contains the Webcam functionality, but it appears it’s not that easy to make. (I’m saving up myself to buy them also, so some donations would be welcome) If someone knows an alternative (free or not) that works well with ABXVision, or finds it a challenge to write one, mail me. I’m sure a lot of OSX peeps would love this.

If no webcam is found this demo will use an included picture to demonstrate the workings of ABXVision. If you already have the MBS plugins, you can make the changes in the code. I’ll point out where it is needed.

Enough about that, Alain. We want some code!

And right you are! Here we go.

We’ll need a couple of steps to go from our grabbed webcam picture to the augmented one.

Here are the steps we’re going to do. You’ll notice there are quite a few and we want to do this real-time. Thanks to our ABXVImage from the previous article, we’ll be able to do so. (Click to enlarge)

The flow

The flow

First we need to do some initialization in the open event of the form. I’ve documented the code.

   ' pauze the loop
  Timer1.Enabled = false

  ' set DEBUG on to show extra info (like corners)
  Debug = true

  ' camera initialization
  CameraWidth = 320
  CameraHeight = 240

  ' Windows ONLY! for OSX, this should be from the plugin you are using!
  #if TargetWin32
    MyCam = new ABXVWebCamWindows
    dim Ret as boolean
    Ret = MyCam.Start(canCam, cameraWidth, cameraHeight)
    if ret = false then
      MsgBox "Unable to initialize the webcam! The demo program will continue using the demo picture."
      canCam.Backdrop = DemoPicture
    end if
  #else
    ' write here the code to initialize your webcam plugin
    ' for this demo, I'll use a static Picture
    canCam.Backdrop = DemoPicture
  #endif

  CameraFocalLength = CameraWidth ' the width of the picture is a good autoFocusLength
  'Glyp initialization
  GlyphSizeMM= 113 ' in milimetre, the real length
  GlyphSizeMMHalf = glyphSizeMM / 2

  ' a list of all the sizes of Glyphs we are going to have in our database
  GlyphSizes = Array(5,8)
  ' add some glyph templates to our database
  AddGlyphTemplate(cornerTemplate, "Alwaysbusy's Corner", "" + _
  "00000" + _
  "01100" + _
  "00100" +_
  "01010" + _
  "00000", 5)
  AddGlyphTemplate(xojoTemplate, "Made With Xojo", "" + _
  "00000000" + _
  "01011010" + _
  "01100100" +_
  "01011000" + _
  "01111010" + _
  "01000100" + _
  "01111010" + _
  "00000000", 8)

  ' start the loop
  Timer1.Enabled = true

Notes:
1. You’ll see I use a rather small camera size (320×240). Although this framework is heavly optimized, it is written in Xojo, not C++. You can always resize the picture after all the processing is done to get a bigger picture.

2. AddGlyphTemplate
This method adds a glyph template to our database. The GlyphValues is a string that describes the white and black values of our glyph. e.g.
Glyph
will be:

AddGlyphTemplate(cornerTemplate, “Alwaysbusy’s Corner”, “” + _
“00000” + _
“01100” + _
“00100” +_
“01010” + _
“00000”, 5)
The ‘5’ is the size of the glyph. You can create glyphs up to 23, but the bigger, the more difficult it will be to detect them.

Also beware of creating doubles! These two are exactly the same:

GlyphSame

Sub AddGlyphTemplate(PictureTemplate as Picture, Name as String, GlyphValues as String, GlyphSize as integer)
  ' check if the template is nil. If true, create a random blank one
  if PictureTemplate = nil then
    PictureTemplate = new Picture(128,128,32)
    PictureTemplate.Graphics.ForeColor = RGB(Rnd * 255, Rnd * 255, Rnd * 255)
    PictureTemplate.Graphics.FillRect 0,0,128,128
  end if
  ' make an ABXVImage from the Xojo picture
  dim arpic as new ABXVImage
  arpic.SetPicture(PictureTemplate, ABXVMASKTYPE.MASK_NONE)

  ' Create the Glyph
  dim tmpGlyph as ABXVGlyph = ABXVGlyphRecognizer.CreateGlyphTemplateFromString(Name, GlyphValues, GlyphSize)
  ' set the image to the glyph
  tmpGlyph.Image = arpic
  ' add it to the database
  GlyphDatabase.Append tmpGlyph
End Sub

And all we have to do now is following the schema above. We do this in a timer (or thread).

  ' temporary blok the loop, we're busy!
  if isBusy then Return
  isBusy = true

  ' the center of the image
  dim cx as Integer = CameraWidth/2
  dim cy as Integer = CameraHeight/2

  ' some temporary pictures
  dim BackwardQuadrilateralTransformedPicture as Picture

  ' take a picture
  #if TargetWin32
    GrabbedPicture = MyCam.GrabPicture
    ' in case no webcam was attached, use the demo picture
    if GrabbedPicture = nil then
      GrabbedPicture = new Picture(CameraWidth, CameraHeight, 32)
      GrabbedPicture.Graphics.DrawPicture DemoPicture,0,0
    end if
  #else
    ' grab the picture with your WebCam plugin
    ' for the demo, the fixed demo picture is used
    GrabbedPicture = new Picture(CameraWidth, CameraHeight, 32)
    GrabbedPicture.Graphics.DrawPicture DemoPicture,0,0
  #endif

  ' some temporaty ABVXImages
  dim GrabbedImage, GrayImage, QuadrilateralTransformedImage, BackwardQuadrilateralTransformedImage as ABXVImage

  ' convert the grabbed picture to a ABXVImage for processing
  GrabbedImage = new ABXVImage
  GrabbedImage.SetPicture(GrabbedPicture, ABXVMASKTYPE.MASK_NONE)

  ' convert to a grayscaled image. We want to keep this for later and continue on the GrabbedImage
  GrabbedImage.GrayScaleTo8Bit(ABXVGRAYSCALETYPE.GRAYSCALE_BT709)
  GrayImage = GrabbedImage.Clone

  ' run some filters
  ' first a difference edge filter
  ABXVEdgeDetector.DifferenceEdgeDetect(GrabbedImage)

  ' next change everything to black or White (2bit), using a threshold
  ABXVBinarization.Threshold(GrabbedImage, 40)

  ' lets find some blobs in our B/W picture.
  dim BlobCounter as new ABXVBlobCounter
  ' the size must minimum 32 pixels . FilterBlobs = true means filter all smaller ones out
  BlobCounter.minWidth = 32
  BlobCounter.minHeight = 32
  BlobCounter.FilterBlobs = true
  BlobCounter.FindBlobs(GrabbedImage)

  ' get the found blobs
  dim Blobs(-1) as ABXVBlob = BlobCounter.GetObjectsInformation
  ' a temporary blob to do the loop
  dim Blob as ABXVBlob

  ' temporary tables to hold the edgepoints of a blog (the border, so to day)
  dim EdgePoints(-1) as ABXVIntPoint
  dim LeftEdgePoints() as ABXVIntPoint
  dim RightEdgePoints() as ABXVIntPoint

  ' temporary table to hold the corners
  dim Corners(-1) as ABXVIntPoint

  ' Our final table that will hold all the found Glyphs
  dim FoundGlyphs(-1) as ABXVGlyph

  ' The glyph we found
  dim FoundGlyph as ABXVGlyph

  ' The Bounding Rectangle of the Found Glyph
  dim BoundingRect as ABXVRectangle

  for each Blob in Blobs
    ' find all the EdgePoints
    EdgePoints = BlobCounter.GetBlobsEdgePoints(Blob)
    ' check if the EdgePoints form a Quadrilateral, if yes, return the corners
    if ABXVShapeChecker.IsQuadrilateral(EdgePoints, Corners) then
      ' It's a Quadrilateral, let's find the left and right edges
      BlobCounter.GetBlobsLeftAndRightEdges(Blob, LeftEdgePoints, RightEdgePoints)

      ' using our gray image, we check if the difference of the edges is big enough with the surrounding pixels (B<->W)
      dim Difference as Double = ABXVGlyphRecognizer.CalculateAverageEdgesBrightnessDifference(LeftEdgePoints, RightEdgePoints, GrayImage)
      if Difference > 20 then
        ' Yes it is!

        ' Transform the found Quadrilateral to a 2D Glyph
        QuadrilateralTransformedImage = ABXVTransform.QuadrilateralTransform(grayimage, corners,GlyphSizeMM,GlyphSizeMM, true)

        ' Run a Otsu Threshold filter, returning a B/W 2Bit image
        ABXVAdaptiveBinarization.OtsuThreshold(QuadrilateralTransformedImage)

        ' look if the found Glyph matches one in our database
        FoundGlyph = ABXVGlyphRecognizer.FindGlyph(QuadrilateralTransformedImage, Corners, GlyphDatabase, GlyphSizes)
        if FoundGlyph <> nil then
          if FoundGlyph.RecognizedGlyph <> nil then
            ' yes, found and recognized!

            ' 2D augmented reality
            ' we do a Backward Quadrilateral Transformation with the picture we want to show over the glyph. A rectangle is returned where it should be placed
            BackwardQuadrilateralTransformedImage = ABXVTransform.BackwardQuadrilateralTransform(FoundGlyph.RecognizedGlyph.Image, FoundGlyph.RecognizedQuadrilateral, true, BoundingRect)
            ' translate the ABVXImage to a Xojo picture
            BackwardQuadrilateralTransformedPicture = BackwardQuadrilateralTransformedImage.GetPicture
            ' and draw it on our original picture, using the returned BoundingRect
            GrabbedPicture.Graphics.DrawPicture BackwardQuadrilateralTransformedPicture,BoundingRect.Left, BoundingRect.Top

            if Debug then
              ' if debug, draw the edges in red
              dim point as ABXVIntPoint
              for each point in edgePoints
                GrabbedPicture.Graphics.Pixel(point.x,point.y) = &cFF0000
              next
            end if

            ' and add it to our found list to do something with it later
            FoundGlyphs.Append FoundGlyph
          end if
        end if
      end if
    end if
  next

  ' draw the result to the canvas
  canAR.Refresh

  ' ok we're done. Next!
  isBusy = false

And that’s it! In just a 100 lines of code, we created magic! Check out the help for more information on what each method does. A lot of methods contain reference links to the theory behind the function.

And what’s next? 3D Augmented Reality of course. Alwyn Bester has gracefully offered to expand the excellent X3 Core framework so that it will work very well with ABXVision.

So, a lot more fun to come in the upcoming weeks!

Download the latest framework from the ABXVision page to get version 1.0.2 with the 2D Augmented Reality demo.

If it is the first time you use ABXVision, read the Getting Started document.

Greetings,

Alwaysbusy

Click here to Donation if you like my work

Advertisements

About Alwaysbusy

My name is Alain Bailleul and I'm the Senior Software Architect/Engineer at One-Two. I like to experiment with new technologies, Computer Vision and A.I. My projects are programmed in B4X , Xojo, C#, java, HTML, CSS and JavaScript. View all posts by Alwaysbusy

3 responses to “Xojo ABXVision – 2D Augmented Reality example

  • François Jouen

    Salut Alain
    Je m’amuse en ce moment avec XoJo et OpenCV
    Voici un exemple de code qui permet d’accéder à sa webcam sous Mac OSX (marche aussi avec Windows et Linux)

    const opencv= “OpenCV2-0”
    const CV_CAP_ANY = 0 ; première caméra disponible

    Soft Declare Function cvNamedWindow lib opencv (name As CString, flag as Integer) as Integer
    Soft Declare Function cvCreateCameraCapture lib opencv ( index as Integer) as ptr
    Soft Declare Function cvQueryFrame lib opencv (capture as Ptr) as ptr
    Soft declare Function cvWaitKey lib opencv ( delay as Integer) as integer
    Soft Declare Sub cvShowImage lib opencv (name As CString, image as Ptr)
    Soft Declare Sub cvResizeWindow lib opencv (name As CString, size as CvSize)

    dim wName as string
    dim w as Integer
    dim capture as Ptr
    dim img as Ptr

    dim key As Integer
    dim foo as integer
    dim size as CvSize

    wName = “Test Window [q to Quit]”

    capture = cvCreateCameraCapture (CV_CAP_ANY)
    if capture = nil then
    msgBox (“Error”)
    end if

    //we need a CvSize structure for the camera size
    size.width =320
    size.height = 240

    w = cvNamedWindow (wName, 1)
    key = 113 ‘(#q)

    while foo key
    img = cvQueryFrame(capture)
    cvShowImage (wName, img)
    cvResizeWindow(wName,size)
    foo = cvWaitKey(1)
    wend

    Amicalement

    • Alwaysbusy

      Thanks François for this snippet! OpenCV is a very nice framework. I did not know you could also use it on Mac.

    • Dinuka Perera

      Hi François Jouen,

      How did you manage to include the functions of OpenCV into XOJO?
      Which dll file did you point to in the soft declare function?

      Thanks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: