Questions regarding mIoU, accuracy, FID #39

Activity

1. taesungp commented on Apr 28, 2019

   

   taesungp

   on Apr 28, 2019

   Contributor

   More actions

   Hello,

   1. I did not crop, but resized the whole image to the same size as generated ones (usually 256x256).
   2. (512x256 for Cityscapes and 256x256 for the others).
   3. You are all correct about the models.
   4. I downsampled the labels using nearest neighbor interpolation.
2. Godo1995 commented on Apr 28, 2019

   

   Godo1995

   on Apr 28, 2019

   Author

   More actions

   Thank you so much for your quick response.
3. Godo1995

   closed this as completedon Apr 28, 2019
4. SolidShen commented on Jul 6, 2019

   

   SolidShen

   on Jul 6, 2019

   More actions

   Hello !
   Can you achieve the mIoU score as the paper mentioned?(62.3 on Cityscapes). I follow your guide and only get 48.7 mIoU on cityscapes val set.
5. zkchen95 commented on Jul 25, 2019

   

   zkchen95

   on Jul 25, 2019

   Last edited by zkchen95

   More actions

   Hello !
   Can you achieve the mIoU score as the paper mentioned?(62.3 on Cityscapes). I follow your guide and only get 48.7 mIoU on cityscapes val set.

   I resize the label as the generator photo(512x256), and the result is 53.5 mIoU, 91.0 accu,
   and resize both the label and the generator photo to (1024x512), the result is 58 mIoU, 92.9 accu

   I have a question for author, what size of the label and generator's photo when evaluation?
6. ShihuaHuang95 commented on Aug 28, 2019

   

   ShihuaHuang95

   on Aug 28, 2019

   More actions

   @ZzzackChen @SolidShen Hi, guys. I have tested the pretrained models (Cityscapes and ADE20k), and I got 64.07 and 43.02 (both represent mIoU). I have downsampled the labels using nearest neighbor interpolation as the authors suggested, (512x256 for cityscapes and 256x256 for others). However, I am confused about the unexpected higher scores than the scores in the paper, especially for ADE20k. @taesungp Would you like to presenet more details for evaluation?
7. fido20160817 commented on May 29, 2023

   

   fido20160817

   on May 29, 2023

   Last edited by fido20160817

   More actions

   Hi,

   Thank you for sharing this awesome code! Base on this issue, I understand that you are not going to release the evaluation code, and I'm working on reimplementing them myself. I have the following questions:

   1. When computing the FID scores, do you compare to the generated images the original images or the cropped images (the same size as the generated ones)?
   2. What are the image sizes you used for evaluation? Do you generate higher resolution ones for evaluation or just use the default size (512x256 for cityscape, and 256x256 for the others)?
   3. What are the pre-trained segmentation models and code base you use for each datasets? Based on the paper, I assume these are the ones you use. Could you please confirm them?

   • COCO stuff: code: kazuto1011/deeplab-pytorch model:
     deeplabv2_resnet101_msc-cocostuff164k-100000.pth
   • ADE20K: code: CSAILVision/semantic-segmentation-pytorch model: baseline-resnet101-upernet
   • Cityscapes: code: fyu/drn model: drn-d-105_ms_cityscapes.pth

   4. When you evaluate mIoUs and accuracies, do you upsample the images or downsample the labels? If so, how do you interpolate them?

   Thanks in advance.

   Best, Godo

   Hi, link for "baseline-resnet101-upernet" is invalid now, can you share this model with me?
8. fido20160817 commented on May 29, 2023

   

   fido20160817

   on May 29, 2023

   More actions

   I have successfully downloaded by information from https://github.com/CSAILVision/semantic-segmentation-pytorch/blob/master/demo_test.sh and https://github.com/CSAILVision/semantic-segmentation-pytorch/blob/master/config/ade20k-resnet101-upernet.yaml