Category Archives: Augmented Reality

Xojo ABXVision – 2D Augmented Reality example

DISCONTINUED: Interested parties in purchasing the source code can contact me via email.

2D Augmented Reality Example

2D Augmented Reality Example

Ready for some fun stuff? Let’s play with ABXVision and create the 2D Augmented Reality program shown in the teaser video.

A quick note first.
I did not find a suitable free Webcam framework for OSX. I talked with Christian from MonkeyBread about creating a cheap spinoff of his excellent plugin suite that just contains the Webcam functionality, but it appears it’s not that easy to make. (I’m saving up myself to buy them also, so some donations would be welcome) If someone knows an alternative (free or not) that works well with ABXVision, or finds it a challenge to write one, mail me. I’m sure a lot of OSX peeps would love this.

If no webcam is found this demo will use an included picture to demonstrate the workings of ABXVision. If you already have the MBS plugins, you can make the changes in the code. I’ll point out where it is needed.

Enough about that, Alain. We want some code!

And right you are! Here we go.

We’ll need a couple of steps to go from our grabbed webcam picture to the augmented one.

Here are the steps we’re going to do. You’ll notice there are quite a few and we want to do this real-time. Thanks to our ABXVImage from the previous article, we’ll be able to do so. (Click to enlarge)

The flow

The flow

First we need to do some initialization in the open event of the form. I’ve documented the code.

   ' pauze the loop
  Timer1.Enabled = false

  ' set DEBUG on to show extra info (like corners)
  Debug = true

  ' camera initialization
  CameraWidth = 320
  CameraHeight = 240

  ' Windows ONLY! for OSX, this should be from the plugin you are using!
  #if TargetWin32
    MyCam = new ABXVWebCamWindows
    dim Ret as boolean
    Ret = MyCam.Start(canCam, cameraWidth, cameraHeight)
    if ret = false then
      MsgBox "Unable to initialize the webcam! The demo program will continue using the demo picture."
      canCam.Backdrop = DemoPicture
    end if
  #else
    ' write here the code to initialize your webcam plugin
    ' for this demo, I'll use a static Picture
    canCam.Backdrop = DemoPicture
  #endif

  CameraFocalLength = CameraWidth ' the width of the picture is a good autoFocusLength
  'Glyp initialization
  GlyphSizeMM= 113 ' in milimetre, the real length
  GlyphSizeMMHalf = glyphSizeMM / 2

  ' a list of all the sizes of Glyphs we are going to have in our database
  GlyphSizes = Array(5,8)
  ' add some glyph templates to our database
  AddGlyphTemplate(cornerTemplate, "Alwaysbusy's Corner", "" + _
  "00000" + _
  "01100" + _
  "00100" +_
  "01010" + _
  "00000", 5)
  AddGlyphTemplate(xojoTemplate, "Made With Xojo", "" + _
  "00000000" + _
  "01011010" + _
  "01100100" +_
  "01011000" + _
  "01111010" + _
  "01000100" + _
  "01111010" + _
  "00000000", 8)

  ' start the loop
  Timer1.Enabled = true

Notes:
1. You’ll see I use a rather small camera size (320×240). Although this framework is heavly optimized, it is written in Xojo, not C++. You can always resize the picture after all the processing is done to get a bigger picture.

2. AddGlyphTemplate
This method adds a glyph template to our database. The GlyphValues is a string that describes the white and black values of our glyph. e.g.
Glyph
will be:

AddGlyphTemplate(cornerTemplate, “Alwaysbusy’s Corner”, “” + _
“00000” + _
“01100” + _
“00100” +_
“01010” + _
“00000”, 5)
The ‘5’ is the size of the glyph. You can create glyphs up to 23, but the bigger, the more difficult it will be to detect them.

Also beware of creating doubles! These two are exactly the same:

GlyphSame

Sub AddGlyphTemplate(PictureTemplate as Picture, Name as String, GlyphValues as String, GlyphSize as integer)
  ' check if the template is nil. If true, create a random blank one
  if PictureTemplate = nil then
    PictureTemplate = new Picture(128,128,32)
    PictureTemplate.Graphics.ForeColor = RGB(Rnd * 255, Rnd * 255, Rnd * 255)
    PictureTemplate.Graphics.FillRect 0,0,128,128
  end if
  ' make an ABXVImage from the Xojo picture
  dim arpic as new ABXVImage
  arpic.SetPicture(PictureTemplate, ABXVMASKTYPE.MASK_NONE)

  ' Create the Glyph
  dim tmpGlyph as ABXVGlyph = ABXVGlyphRecognizer.CreateGlyphTemplateFromString(Name, GlyphValues, GlyphSize)
  ' set the image to the glyph
  tmpGlyph.Image = arpic
  ' add it to the database
  GlyphDatabase.Append tmpGlyph
End Sub

And all we have to do now is following the schema above. We do this in a timer (or thread).

  ' temporary blok the loop, we're busy!
  if isBusy then Return
  isBusy = true

  ' the center of the image
  dim cx as Integer = CameraWidth/2
  dim cy as Integer = CameraHeight/2

  ' some temporary pictures
  dim BackwardQuadrilateralTransformedPicture as Picture

  ' take a picture
  #if TargetWin32
    GrabbedPicture = MyCam.GrabPicture
    ' in case no webcam was attached, use the demo picture
    if GrabbedPicture = nil then
      GrabbedPicture = new Picture(CameraWidth, CameraHeight, 32)
      GrabbedPicture.Graphics.DrawPicture DemoPicture,0,0
    end if
  #else
    ' grab the picture with your WebCam plugin
    ' for the demo, the fixed demo picture is used
    GrabbedPicture = new Picture(CameraWidth, CameraHeight, 32)
    GrabbedPicture.Graphics.DrawPicture DemoPicture,0,0
  #endif

  ' some temporaty ABVXImages
  dim GrabbedImage, GrayImage, QuadrilateralTransformedImage, BackwardQuadrilateralTransformedImage as ABXVImage

  ' convert the grabbed picture to a ABXVImage for processing
  GrabbedImage = new ABXVImage
  GrabbedImage.SetPicture(GrabbedPicture, ABXVMASKTYPE.MASK_NONE)

  ' convert to a grayscaled image. We want to keep this for later and continue on the GrabbedImage
  GrabbedImage.GrayScaleTo8Bit(ABXVGRAYSCALETYPE.GRAYSCALE_BT709)
  GrayImage = GrabbedImage.Clone

  ' run some filters
  ' first a difference edge filter
  ABXVEdgeDetector.DifferenceEdgeDetect(GrabbedImage)

  ' next change everything to black or White (2bit), using a threshold
  ABXVBinarization.Threshold(GrabbedImage, 40)

  ' lets find some blobs in our B/W picture.
  dim BlobCounter as new ABXVBlobCounter
  ' the size must minimum 32 pixels . FilterBlobs = true means filter all smaller ones out
  BlobCounter.minWidth = 32
  BlobCounter.minHeight = 32
  BlobCounter.FilterBlobs = true
  BlobCounter.FindBlobs(GrabbedImage)

  ' get the found blobs
  dim Blobs(-1) as ABXVBlob = BlobCounter.GetObjectsInformation
  ' a temporary blob to do the loop
  dim Blob as ABXVBlob

  ' temporary tables to hold the edgepoints of a blog (the border, so to day)
  dim EdgePoints(-1) as ABXVIntPoint
  dim LeftEdgePoints() as ABXVIntPoint
  dim RightEdgePoints() as ABXVIntPoint

  ' temporary table to hold the corners
  dim Corners(-1) as ABXVIntPoint

  ' Our final table that will hold all the found Glyphs
  dim FoundGlyphs(-1) as ABXVGlyph

  ' The glyph we found
  dim FoundGlyph as ABXVGlyph

  ' The Bounding Rectangle of the Found Glyph
  dim BoundingRect as ABXVRectangle

  for each Blob in Blobs
    ' find all the EdgePoints
    EdgePoints = BlobCounter.GetBlobsEdgePoints(Blob)
    ' check if the EdgePoints form a Quadrilateral, if yes, return the corners
    if ABXVShapeChecker.IsQuadrilateral(EdgePoints, Corners) then
      ' It's a Quadrilateral, let's find the left and right edges
      BlobCounter.GetBlobsLeftAndRightEdges(Blob, LeftEdgePoints, RightEdgePoints)

      ' using our gray image, we check if the difference of the edges is big enough with the surrounding pixels (B<->W)
      dim Difference as Double = ABXVGlyphRecognizer.CalculateAverageEdgesBrightnessDifference(LeftEdgePoints, RightEdgePoints, GrayImage)
      if Difference > 20 then
        ' Yes it is!

        ' Transform the found Quadrilateral to a 2D Glyph
        QuadrilateralTransformedImage = ABXVTransform.QuadrilateralTransform(grayimage, corners,GlyphSizeMM,GlyphSizeMM, true)

        ' Run a Otsu Threshold filter, returning a B/W 2Bit image
        ABXVAdaptiveBinarization.OtsuThreshold(QuadrilateralTransformedImage)

        ' look if the found Glyph matches one in our database
        FoundGlyph = ABXVGlyphRecognizer.FindGlyph(QuadrilateralTransformedImage, Corners, GlyphDatabase, GlyphSizes)
        if FoundGlyph <> nil then
          if FoundGlyph.RecognizedGlyph <> nil then
            ' yes, found and recognized!

            ' 2D augmented reality
            ' we do a Backward Quadrilateral Transformation with the picture we want to show over the glyph. A rectangle is returned where it should be placed
            BackwardQuadrilateralTransformedImage = ABXVTransform.BackwardQuadrilateralTransform(FoundGlyph.RecognizedGlyph.Image, FoundGlyph.RecognizedQuadrilateral, true, BoundingRect)
            ' translate the ABVXImage to a Xojo picture
            BackwardQuadrilateralTransformedPicture = BackwardQuadrilateralTransformedImage.GetPicture
            ' and draw it on our original picture, using the returned BoundingRect
            GrabbedPicture.Graphics.DrawPicture BackwardQuadrilateralTransformedPicture,BoundingRect.Left, BoundingRect.Top

            if Debug then
              ' if debug, draw the edges in red
              dim point as ABXVIntPoint
              for each point in edgePoints
                GrabbedPicture.Graphics.Pixel(point.x,point.y) = &cFF0000
              next
            end if

            ' and add it to our found list to do something with it later
            FoundGlyphs.Append FoundGlyph
          end if
        end if
      end if
    end if
  next

  ' draw the result to the canvas
  canAR.Refresh

  ' ok we're done. Next!
  isBusy = false

And that’s it! In just a 100 lines of code, we created magic! Check out the help for more information on what each method does. A lot of methods contain reference links to the theory behind the function.

And what’s next? 3D Augmented Reality of course. Alwyn Bester has gracefully offered to expand the excellent X3 Core framework so that it will work very well with ABXVision.

So, a lot more fun to come in the upcoming weeks!

Download the latest framework from the ABXVision page to get version 1.0.2 with the 2D Augmented Reality demo.

If it is the first time you use ABXVision, read the Getting Started document.

Greetings,

Alwaysbusy

Click here to Donation if you like my work


Xojo ABXVision: First Augmented Reality 3D Demo

DISCONTINUED: Interested parties in purchasing the source code can contact me via email.

This is one of the first expreriments using ABXVision and X3 Core together! Some work needs to be done for the positioning on the glyph, but we’re getting there. I promise I stop playing with it (but it’s so much fun!) and start writing the articles real soon.

Click here to Donation if you like my work


Xojo: ABXVision – The use of ABXVImage and filters

DISCONTINUED: Interested parties in purchasing the source code can contact me via email.

ABXVImage and Filters

ABXVImage and Filters

This is the first article on the use of the new ABXVision framework. I’ve decided to start with the most fundamental class: ABXVImage.

ABXVImage is the core image object that will be used in every project you create with ABXVision. It’s a new ‘picture’ object in Xojo that is especially written to optimize the algorithms you can use (Color filters, Edge detectors, Blurs, Corner detectors, Shape Detectors, etc…).

To make it clear if we are working with a Xojo picture or an ABXVImage, I’ve split up the terms:
Whenever I use picture, I mean a Xojo picture. Whenever I use Image, I mean an ABXVImage.

So the first thing we need to do is convert a picture to an image and back:

  ' create a ABXVImage from a Xojo Picture
  dim img as new ABXVImage
  img.SetPicture(walleorig)

Img contains the data of the picture, ready to be processed. We can now make use of the filters. Some basic filters (like grayscaling, a thing you’ll be doing all the time), are also build-in the ABXVImage class.

For example, if we want to grayscale the image for further processing, we can use the following code:

   img.GrayScaleTo8Bit()

You’ll notice the 8Bit part in the method name. A lot of the methods in ABXVision (like binarization) use this 8Bit pointer to perform their work. As a grayscaled image has the same value in its Red, Green and Blue channel, there is no need to run the whole filter over all 3 channels. Keeping record of one channel will do.

Once this function is performed, the ABXVImage will have its BitDepth property set to ABXVBITDEPTH.BIT8. Now, all other methods can check this property and see if it is ok to perform the action.

If, for some reason we want it back as a 32Bit (24Bit in many cases where no alpha is used), we can use the following code:

   img.GrayScaled8BitTo32Bit()

This will also set the BitDepth property of the image to ABXVBITDEPTH.BIT32.

Now, after our hard work, we want to have a Xojo picture with the result. This can be done with the GetPicture() method:

   mBuffer = img.GetPicture()

Beware of overusing the GetPicture() method! If you have to run a lot of filters after each other and have no need to see every single step, there is no need to use the GetPicture() method. GetMethod() is mostly called after everything is done, e.g. when the image is analyzed, a glyph is found and the augmented part is drawn on the image.

Notes:

1. GetPicture() will do a GrayScaled8BitTo32Bit() automatatically if it was a 8 Bit image.

2. To be complete, most methods can handle 32 bit picture (R, G, B + Alpha/Mask), but it’s not advised to use this kind of image. Those methods will also perform their action on the alpha channel, and this takes time. The SetPicture() method will set the image as ABXVMASKTYPE.MASK_NONE, so the Alpha channel is ignored. If for some reason you want your Alpha channel, use the Mask parameter of the SetPicture() method. You can use ABXVMASKTYPE.MASK_OLD (This is the old Realbasic format where the mask is also a complete picture object) or ABXVMASKTYPE.MASK_NEW (This is the new Xojo picture format, with an extra real Alpha channel. This is experimental and not fully tested)

3. A lot of optimization is done using #Pragmas in the framework. This means it will run a lot faster once compiled, but will do also less error checks. Always make sure you follow the rules of the syntax of the method. e.g. if the method asks an array with 4 points, don’t give it one with 3 points. Your program will crash!

Here are the pragmas used:

  #if not DebugBuild
    #pragma DisableBackgroundTasks
    #pragma DisableBoundsChecking
    #pragma StackOverflowchecking False
    #pragma NilObjectChecking False
  #endif

There are many filters available in ABXVision. I’ve tried to split them up so they made some sense. (Click the titles to jump to the help)

ABXV2SourceFilters
Provides a number of 2 source filters – filters which take two input images to produce the result image.
ABXVAdaptiveBinarization
Provides several adaptive binarization filters, which are aimed to find binarization threshold automatically and then apply it to the source image.
ABXVBinarization
Provides different binarization filters, which may be used as in image processing, as in some computer vision tasks.
ABXVColorFilters
Provides number of image processing filters, which allow to filter pixels depending on their color values.
These image processing filters may be used to keep pixels, which color falls inside or outside of specified range, and fill the rest of pixels with specified color.
ABXVConvolution
Provides convolution filters and a set of derived filters, which allow to perform image convolution with common kernels.
ABXVEdgeDetector
Provides a number of edge detection filters, which may suite different tasks providing different performance.
ABXVMorphology
Provides a set of filters from mathematical morphology.
All of the filters may be applied as using default structuring element, as using custom specified structuring element.
ABXVTransform
Provides transformations methods, which allow to perform image transformationq.

In the attached demo project, some of those filters are used to show the different techniques used. You’ll notice most of the filters have multiple calling methods. They are mostly split up as follows:

1. call(img1). The action is performed on the image.
2. img2 = call(img1). The action is performed on img and returned in img2 without altering img1.
3. call(img1, rect). Perform the action on the image itself, but only on the part described by the rectangle.
4. img2 = call(img1, rect). Perform the action on the image itself, but only on the part described in the rectangle and also only return this part as img2, without altering img1

I suggest you read up on the help and play with the ABXVImage object and familiarize with it. The demo project contains some filters, but you can try out others.

This concludes the introduction. I felt it was needed to talk about ABXVImage first before going into other subjects as this is a core object. The source code and framework can be found on the ABXVision page (see top of this page)

You need to read the Getting Started document as you have to copy the framework into the demo projects.

In my next article, we’ll tackle Augmented Reality, as seen in the teaser video!

Happy Coding!

Alwaysbusy

Click here to Donation if you like my work


Xojo: ABXVision – Version 1.0 Released

ABXVisionDISCONTINUED: Interested parties in purchasing the source code can contact me via email.

This is the first public release of ABXVision! I’ve spend a lot of time setting it up. Creating the documentation for the framework also took a fair amount of time, but as this project will grow in the future, it’s best I start documenting it from the beginning.

To centralize everything about ABXVision, I’ve created a new menu item on top of this page. Here you’ll be able to find the latest version, the help and the projects I’ll create with the ABXVision framework in the upcoming articles.

Latest version can be downloaded here

ABXVision

Halfway through writing the documentation, I realized this would take longer than expected so I stopped and build a tool to help me maintain the documentation: ABXDocumentor. I have a very good feeling about this tool and will probably update it and release it on this blog.

In the upcoming weeks I will try to write some demos using the framework. The first one will handle some basic functionality like ABXVImage as this object is really important. Later we’ll move on to Augmented Reality, Motion detection, maybe something on playcard recognition. Who knows were it takes us.

I just finish with this note: This is version 1.0. Not all functionality was tested so I can use your feedback to update it. I hope you join me on this ongoing journey and show us some projects your created with the framework.

Alwaysbusy

Click here to Donation if you like my work


Xojo: ABXVision 1.0 progress update

DISCONTINUED: Interested parties in purchasing the source code can contact me via email.

ABXVision
Some readers are getting impatient to try out ABXVision. So, I decided to pause programming new features for now and started to document the current framework. This is a big task but is needed for this kind of framework.

The main purpose of ABXVision is giving students and researchers the tools to do Computer Vision and A.I. with a RAD tool like Xojo. Without being a programming genius with a Phd in C++ or ASM, they can experiment with cutting edge algorithms with the ease of creating simple desktop apps.

Some of you who are familiar with Aforge will notice the resemblance. A lot of code is a direct translation of this framework to Xojo (mixed with own functions, Xojo optimalisations, Xojo wrappers to make it simpler to use, and translated code from other frameworks). I think it will make it easier to try out some of the functionalities by checking out the .NET examples. The code will be very similar when you try it out in Xojo.

The first release will contain GlyphDetectors, Color Filters, Edge Filters, Blur Filters, Motion Detectors, Blob finders, Transformations, Binarizers, Augmented Reality, etc… So a complete toolbox to start Computer Vision with Xojo!

Future features will be Fuzzy Logic, Neural Networks, Machine Learning, Genetic Algotithms, a lot more filters, Contour detectors, Corner Detectors, Interesting Point algorithms, Moment Computers, Kinematic… etc.

Carry on coding!

Alwaysbusy


Xojo: Augmented Reality with ABXVision first look

DISCONTINUED: Interested parties in purchasing the source code can contact me via email.

Augmented Reality in Xojo

Augmented Reality in Xojo

This is a teaser for an upcoming article on Alwaysbusy’s Corner on Augmented Reality with ABXVision!

ABXVision is a framework in 100% pure Xojo (except for the WebCam). It does blobdetection, Glyph recognition, Augmented Reality in 2D and 3D (still working on the 3D part) in real time.

For the moment I’m getting good results (under 0.05 seconds to do the job), but still some work to do before public release.

If someone can help me with a free solution for the Webcam on OSX, please PM me.

And without further ado, here is the teaser video. Enjoy 🙂

If you want to receive an email when the upcoming article is published, follow the blog on the top right.

See you soon!

Click here to Donation if you like my work


RealBasic: Computer Vision -> FAST Corner Detection


As promised, here is the FAST (Features from Accelerated Segment Test) implementation in RealBasic! And as you’ll notice, this is extremely fast!

Wiki:

AST is an acronym standing for Accelerated Segment Test. This test is a relaxed version of the SUSAN corner criterion. Instead of evaluating the circular disc only the pixels in a Bresenham circle of radius r around the candidate point are considered. If n contiguous pixels are all brighter than the nucleus by at least t or all darker than the nucleus by t, then the pixel under the nucleus is considered to be a feature. This test is reported to produce very stable features.[17] The choice of the order in which the pixels are tested is a so called Twenty Questions problem. Building short decision trees for this problem results in the most computationally efficient feature detectors available.

The first corner detection algorithm based on the AST is FAST (Features from Accelerated Segment Test). Although r can in principle take any value, FAST uses only a value of 3 (corresponding to a circle of 16 pixels circumference), and tests show that the best results are achieved with n being 9. This value of n is the lowest one at which edges are not detected. The order in which pixels are tested is determined by the ID3 algorithm from a training set of images. Confusingly, the name of the detector is somewhat similar to the name of the paper describing Trajkovic and Hedley’s detector.

It is the RealBasic translation of the nonmaximal FAST-9 written by Edward Rosten http://www.edwardrosten.com/work/fast.html
With this library you can find Corners very fast within a picture.

and it can do this under e.g. different light intensities:

Again, the RealBasic engine has a simple interface so you can focus on the fun stuff!

    dim FCD as new ABFastCornerDetector
  
  dim tmpXYTraining(-1) as ABXY
  dim i as integer
  dim j as integer
  
  dim pic as Picture
  dim tmpPic as Picture
  
  dim DownSample as integer
  dim build2Dim as Boolean = true
  
  Dim f as FolderItem
  
  dim ft1 as new filetype
  ft1.name = "picture"
  ft1.Extensions = ".png;.bmp;.jpg;.jpeg"
  
  dim dlg as OpenDialog
  dlg = new OpenDialog
  
  dlg.Title = "Pick picture"
  dlg.PromptText = "Pick a picture to compare"
  dlg.Filter=ft1
  if dlg.InitialDirectory = nil then
    dlg.InitialDirectory = GetFolderItem("")
  end if
  
  f = dlg.ShowModal
  if f <> nil then
    if f.Exists then
      pic = f.OpenAsPicture
    else
      Return
    end if
  else
    Return
  end if
  
  ' these are then numbers you can play with
  FCD.threshold_detection = 20
  FCD.threshold_scoring = 0
  DownSample = 2
  ' do the FAST detection
  tmpXYTraining = FCD.detectFASTCornerFeatures(pic,DownSample, build2Dim)
  
  tmpPic = NewPicture(pic.Width, pic.Height, 32)
  tmpPic.Graphics.DrawPicture pic,0,0
  
  tmpPic.Graphics.ForeColor = &cFF0000
  
  ' and draw them
  for i = 0 to UBound(tmpXYTraining)
    tmpPic.Graphics.DrawLine tmpXYTraining(i).x*DownSample - 3, tmpXYTraining(i).y*DownSample, tmpXYTraining(i).x*DownSample + 3, tmpXYTraining(i).y*DownSample
    tmpPic.Graphics.DrawLine tmpXYTraining(i).x*DownSample, tmpXYTraining(i).y*DownSample-3, tmpXYTraining(i).x*DownSample, tmpXYTraining(i).y*DownSample+3
  next
  
  Canvas1.Graphics.FillRect 0,0, 640,480
  Canvas1.Graphics.DrawPicture tmpPic,0,0

You can download the FAST engine from: http://www.gorgeousapps.com/ABFAST.zip

Until the next post!

Click here to Donation if you like my work


%d bloggers like this: