HandByDepth Class Reference

Detector based on depth only, no glove appearance model. More...

Inheritance diagram for HandByDepth:

Collaboration diagram for HandByDepth:

Public Member Functions
def	__init__ (self, detCfg=None, detInst=None, processors=None)
	Constructor for layered puzzle scene detector assuming no glove is being used. More...
def	adapt (self)
	Adapt the layer detection models. More...
def	correct (self)
	Apply correction process to the individual detectors. More...
def	detect (self, I)
	Apply predict, measure, correct process to source image. More...
def	emptyDebug (self)
	Get empty puzzle scene detector debug state. More...
def	emptyState (self)
	Get empty puzzle scene detector state to recover its basic structure. More...
def	getDebug (self)
	Get debug information if available. More...
def	getState (self)
	Get the complete detector state, which involves the states of the individual layer detectors. More...
def	info (self)
def	loadFrom (fPtr)
def	measure (self, I)
	Apply detection to the source image pass. More...
def	predict (self)
	Generate prediction of expected measurement. More...
def	process (self, I)
	Apply entire predict to adapt process to source image. More...
def	saveTo (self, fPtr)
	Save the instantiated Detector to given HDF5 file. More...

Static Public Member Functions
def	buildFromCfg (theConfig)
	Instantiate from stored configuration file (YAML). More...
def	calibrate2config (theStream, outFile)
def	load (inFile)

Public Attributes
	depth
	hand
	First, perform any specified pre-processing. More...
	imHand
	Package the processed layers started with too high. More...
	imPuzzle
	mask
	params
	workspace

Detailed Description

Detector based on depth only, no glove appearance model.

Note

We still call it a glove to maintain compatibility with the glove-based model. In principle, we should call it HandByDepth. Will have to figure that out eventually. Do we rigidly stick to Glove or do we move to Hand and then augment with glove capabilities? Or, do we modify internal member variables to refer to hand so that it make sense whether or not there is a glove on the hand? Need to resolve. 2025/07/22 PAV.

Changing to HandByDepth and committing to eventually aligning member variables with this change. Right now, need to recall how everything was implemented and do better job at documenting things for easier development in the future. 2027/07/22 PAV.

How can we generalize this so that a general robot can use it and even employ tracking to interpret and learn from human (demonstrator)? 2025/07/22 PAV.

Constructor & Destructor Documentation

__init__()

def __init__	(		self,
			detCfg = None,
			detInst = None,
			processors = None
	)

Constructor for layered puzzle scene detector assuming no glove is being used.

Parameters

[in]	detCfg	Detector configuration.
[in]	processors	Image processors for the different layers.
[in]	detInst	Detection instances for the different layers.

Member Function Documentation

adapt()

def adapt

(

self

)

Adapt the layer detection models.

This part is tricky as there may be dependencies across the layers in terms of what should be updated and what should not be. Applying simple filtering to establish what pixels should adapt and which ones shouldn't.

buildFromCfg()

def buildFromCfg (   theConfig )

static

Instantiate from stored configuration file (YAML).

calibrate2config()

def calibrate2config (   theStream,   outFile  )

static

correct()

def correct

(

self

)

Apply correction process to the individual detectors.

Apply naive correction on a per detector basis. As a layered system, there might be interdependencies that would impact the correction step. Ignoring that for now since it does not immediately come to mind what needs to be done.

detect()

def detect	(		self,
			I
	)

Apply predict, measure, correct process to source image.

Running detect alone elects not to adapt or update the underlying models. The static model is presumed to be sufficient and applied to the RGBD stream.

Parameters

[in]

I

Source RGB-D image (structure/dataclass).

emptyDebug()

def emptyDebug

(

self

)

Get empty puzzle scene detector debug state.

emptyState()

def emptyState

(

self

)

Get empty puzzle scene detector state to recover its basic structure.

Translated from Matlab, which required access to this in advance to manage streaming data storage. Python isn't as strongly typed.

Parameters

[out]

estate

The empty state.

getDebug()

def getDebug

(

self

)

Get debug information if available.

Status: Not available.

getState()

def getState

(

self

)

Get the complete detector state, which involves the states of the individual layer detectors.

Parameters

[out]

state

The detector state for each layer, by layer.

info()

def info

(

self

)

load()

def load (   inFile )

static

loadFrom()

def loadFrom

(

fPtr

)

measure()

def measure	(		self,
			I
	)

Apply detection to the source image pass.

Parameters

[in]

I

An RGB-D image (structure/dataclass).

predict()

def predict

(

self

)

Generate prediction of expected measurement.

The detectors are mostly going to be static models, which means that prediction does nothing. Just in case though, the prediction methods are called for them.

process()

def process	(		self,
			I
	)

Apply entire predict to adapt process to source image.

Parameters

[in]

I

Source RGB-D image (structure/dataclass).

saveTo()

def saveTo	(		self,
			fPtr
	)

Save the instantiated Detector to given HDF5 file.

The save process saves the necessary information to re-instantiate a HandByDepth class object.

Parameters

[in]

fPtr

An HDF5 file point.

Member Data Documentation

depth

depth

hand

hand

First, perform any specified pre-processing.

Note

Not dealing with pre-processor, but it might be important.

Todo:

Figure out how to use the improcessor.

Second, invoke the layer detectors and post-processor to differentiate the actual semantic layers of the scene. The layer detectors should be considered as raw detectors that need further polishing to extract the desired semantic layer information. These layers are further processed by customized track pointers and filters. The post processing here is hard-coded rather than a private member function invocation.

The hand regions above the surface get recovered here. Unfortunately, the Realsense is not the best depth detector and fails to capture regions near the work surface. The best thing to do is to expand the too high regions, to excise nearby areas from workmat detection region.

Again, "level with the surface" is only accurate up to the depth camera's depth sensitivity. If not the Realsense, the depth might still be a little too noisy to really capture fine details.

imHand

imHand

Package the processed layers started with too high.

Next, remove any parts of the not surface layer that intersect with the expanded glove region. May remove adjacent puzzle piece area; that's OK since we can't rely on those pieces having been fully measured/captured. After that

imPuzzle

imPuzzle

mask

mask

params

params

workspace

workspace

---

The documentation for this class was generated from the following file:

• /home/mary/python/Surveillance/Surveillance/layers/PuzzleScene.py