PuzzleDetectors Class Reference

Detector for layered puzzle scene: glove and puzzle pieces. More...

Inheritance diagram for PuzzleDetectors:

Collaboration diagram for PuzzleDetectors:

Public Member Functions
def	__init__ (self, detCfg=None, detInst=None, processors=None)
	Constructor for layered puzzle scene detector. More...
def	adapt (self)
	Adapt the layer detection models. More...
def	correct (self)
	Apply correction process to the individual detectors. More...
def	detect (self, I)
	Apply predict, measure, correct process to source image. More...
def	emptyDebug (self)
def	emptyState (self)
	Get and empty state to recover its basic structure. More...
def	getDebug (self)
def	getState (self)
	Get the complete detector state, which involves the states of the individual layer detectors. More...
def	info (self)
def	loadFrom (fPtr)
def	measure (self, I)
	Apply detection to the source image pass. More...
def	predict (self)
	Generate prediction of expected measurement. More...
def	process (self, I)
	Apply entire predict to adapt process to source image. More...
def	saveTo (self, fPtr)
	Save the instantiated Detector to given HDF5 file. More...

Static Public Member Functions
def	buildFromCfg (theConfig)
	Instantiate from stored configuration file (YAML). More...
def	calibrate2config (theStream, outFile)
def	load (inFile)

Public Attributes
	depth
	glove
	hand
	First, perform any specified pre-processing. More...
	imGlove
	Package the processed layers started with the glove. More...
	imPuzzle
	mask
	params
	workspace

Detailed Description

Detector for layered puzzle scene: glove and puzzle pieces.

Puzzle pieces are really flat, non-background elements on the work mat. Anything high enough off the work mat is not a puzzle piece, but presumed to be a hand or other equivalent puzzle manipulation mechanism.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
			detCfg = None,
			detInst = None,
			processors = None
	)

Constructor for layered puzzle scene detector.

Parameters

[in]	detCfg	Detector configuration (from CfgPuzzleScene).
[in]	processors	Image processors for the different layers.
[in]	detInst	Detection instances for the different layers.

Reimplemented in PuzzleCalibrator, and PuzzleDetectors_v2.

Member Function Documentation

adapt()

def adapt

(

self

)

Adapt the layer detection models.

This part is tricky as there may be dependencies across the layers in terms of what should be updated and what should not be. Applying simple filtering to establish what pixels should adapt and which ones shouldn't.

Reimplemented in PuzzleCalibrator.

buildFromCfg()

def buildFromCfg (   theConfig )

static

Instantiate from stored configuration file (YAML).

calibrate2config()

def calibrate2config (   theStream,   outFile  )

static

correct()

def correct

(

self

)

Apply correction process to the individual detectors.

Apply naive correction on a per detector basis. As a layered system, there might be interdependencies that would impact the correction step. Ignoring that for now since it does not immediately come to mind what needs to be done.

Reimplemented in PuzzleCalibrator.

detect()

def detect	(		self,
			I
	)

Apply predict, measure, correct process to source image.

Running detect alone elects not to adapt or update the underlying models. The static model is presumed to be sufficient and applied to the RGBD stream.

Parameters

[in]

I

Source RGB-D image (structure/dataclass).

Reimplemented in PuzzleCalibrator.

emptyDebug()

def emptyDebug

(

self

)

Reimplemented in PuzzleCalibrator.

emptyState()

def emptyState

(

self

)

Get and empty state to recover its basic structure.

Parameters

[out]

estate

The empty state.

Reimplemented in PuzzleCalibrator.

getDebug()

def getDebug

(

self

)

Reimplemented in PuzzleCalibrator.

getState()

def getState

(

self

)

Get the complete detector state, which involves the states of the individual layer detectors.

Parameters

[out]

state

The detector state for each layer, by layer.

Reimplemented in PuzzleCalibrator, and PuzzleDetectors_v2.

info()

def info

(

self

)

Reimplemented in PuzzleCalibrator.

load()

def load (   inFile )

static

Reimplemented in PuzzleDetectors_v2.

loadFrom()

def loadFrom

(

fPtr

)

Reimplemented in PuzzleDetectors_v2.

measure()

def measure	(		self,
			I
	)

Apply detection to the source image pass.

Parameters

[in]

I

An RGB-D image (structure/dataclass).

Shrink region associated to the surface just to clean up potential sources of confusion and promote recovery of puzzle pieces that are fully captured. Combine with puzzle mat mask region (negated) to get presumed puzzle piece area.

Reimplemented in PuzzleCalibrator, and PuzzleDetectors_v2.

predict()

def predict

(

self

)

Generate prediction of expected measurement.

The detectors are mostly going to be static models, which means that prediction does nothing. Just in case though, the prediction methods are called for them.

Reimplemented in PuzzleCalibrator.

process()

def process	(		self,
			I
	)

Apply entire predict to adapt process to source image.

@param[in]  I   Source RGB-D image (structure/dataclass).

Reimplemented in PuzzleCalibrator.

saveTo()

def saveTo	(		self,
			fPtr
	)

Save the instantiated Detector to given HDF5 file.

The save process saves the necessary information to re-instantiate a PuzzleDetectors class object.

Parameters

[in]

fPtr

An HDF5 file point.

Reimplemented in PuzzleCalibrator.

Member Data Documentation

depth

depth

glove

glove

hand

hand

First, perform any specified pre-processing.

The post processing here is hard-coded rather than a private member function invocation.

Note

Not dealing with pre-processor, but it might be important.

Todo:

Figure out how to use the improcessor.

Second, invoke the layer detectors and post-processor to differentiate the actual semantic layers of the scene. The layer detectors should be considered as raw detectors that need further polishing to extract the desired semantic layer information. These layers are further processed by customized track pointers and filters.

Glove hand region recovery. The glove regions should be above the surface and pass the glove detector. Otherwise, they could be confounding puzzle pieces that look like the glove. Anything near the surface and not the black mat is most likely a puzzle piece. Capture these, as they can be interpreted as regions not being the black mat surface but level with the surface. Note that "level with the surface" is only accurate up to the depth camera's depth sensitivity. Some can be a little too noisy to really capture fine details.

Code below is an attempt to employ detection hysteresis for the glove. A much smaller set is established as the definitely glove seed region. That seed region should subsequently get expanded (later code) into a bigger region based on connectivity. It has been very troublesome to find that sort of operation in existing libraries. Many attempts were made but they all do the wrong thing. Amazing that such a feaure does not exist. It's a core component of standard detection focused image processing strategies. So silly.

lessGlove = scipy.ndimage.binary_erosion(gDet.fgIm, kernel, 5) NOTE 2023/12/06 Trying out something new in line below vs line above.

imGlove

imGlove

Package the processed layers started with the glove.

Next, remove any parts of the not surface layer that intersect with the expanded glove region. May remove adjacent puzzle piece area; that's OK since we can't rely on those pieces having been fully measured/captured. After that

imPuzzle

imPuzzle

mask

mask

params

params

workspace

workspace

---

The documentation for this class was generated from the following file:

• /home/mary/python/Surveillance/Surveillance/layers/PuzzleScene.py