Color Deconvolution / "Depainting"

[oshfdk]

Did you do the training on labeled data? How exactly?

I take an original TIFF into an image editor, mark some areas of interest with a distinctive color (specific to the desired predicted class) and cover the rest with neutral grey. Then the training script reads the original TIFF and the marked file, extracts the desired values from the marked file, treating each distinct color value as a separate class (and ignoring grey ares), then extracts 3x3 kernels of the same pixels from the original TIFFs, shuffles them (can be skipped for models that don't process data sequentially, e.g., linreg, but important for models that work on batched data) and trains the model. 

I've labelled 4 images this way, using those that show various colors.  On each image I only label (manually, with a pixel brush) a small subset of features, maybe 0.1% or less of all that is present. Normally it takes no more than 10-15 minutes to label an image this way. When adding marks, I'm primarily targeting areas which can be clearly identified visually, but could be challenging for an algorithm with no spacial awareness (e.g., faint lines with clearly visible beginning and ending, lines coming partially under paint, etc). Normally I split my labels between 50% of clearly identifiable features and 50% of features that I can only extrapolate from spacial clues, but the model will have to learn them the hard way using local pixel values only. After finishing the training, I applied the resulting model to the whole set.

The following image shows the markup used for f69r, the original on the left, the image after adding color labels in the middle and the final markup image as seen by the model on the right. For training the model receives the leftmost and the rightmost images as its only input (other than the hyperparameters of the model). For prediction it only receives the original TIFFs with no annotations.

Note that the model is trained to separate various paint colors and ink into different layers, not to remove color, I "converted" it to paint removal tool by setting white color to all paint classes in post processing step. It's likely the results could be better if I train it to specifically remove paint by leaving just two training classes - ink and everything else.