MoA-Net: Self-supervised Motion Segmentation. Pia Bideau, Rakesh R Menon, Erik Learned-Miller

MoA-Net: Self-supervised Motion Segmentation Pia Bideau, Rakesh R Menon, Erik Learned-Miller University of Massachusetts Amherst College of Information and Computer Science

Motion Segmentation P Bideau, E Learned-Miller, ECCV 2016: It s moving! A probabilistic model for causal motion segmentation P Bideau, A RoyChoudhury, R Menon, E Learned-Miller, CVPR 2018: The best of both worlds: Combining CNNs and geometric constraints for hierarchical motion segmentation P Bideau, R Menon, E Learned-Miller, Workshop ECCV 2018: MoA-Net: Unsupervised Motion Segmentation

Overview Motivation How do humans know what is moving in the world and what is not? Approach: Motion Segmentation Rotation compensation Learning Motion Patterns: MoA-Net Results Future Research Questions

Motivation

Motivation stationary scene moving object no observer motion stationary scene moving object observer motion

Motivation All motions result in changes of the retinal image. What is the problem about retinal image motion? photoreceptors are slow motion detection in our brain is challenging Need to stabilize the image, to reduce retinal image motion

Motivation

Approach: Motion Segmentation MoA-Net optical flow rotation compensated flow step 1: rotation compensation angle field motion segmentation step 2: motion segmentation

Rotation Compensation rotation + translation optical flow magnitude is dependent on scene depth optical flow angle is dependent on scene depth translation optical flow magnitude is dependent on scene depth optical flow angle is independent of scene depth

Rotation Compensation only camera translation and object motion optical flow angle field

Motion Segmentation MoA-Net optical flow rotation compensated flow step 1: rotation compensation angle field motion segmentation step 2: motion segmentation

Motion Segmentation Definition Def.: Moving Object A moving object is a connected image region that undergoes some independent motion. The connected image region can be of any size and shape.

Motion Segmentation Generating training data

Motion Segmentation Generating training data Generating connected object regions. Splitting each object into n subregions. Assigning to each motion region a translational 3D direction. Smoothing motion boundaries inside moving objects. Adding random gaussian noise.

<latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> <latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> <latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> <latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> Motion Segmentation Generating training data Generating connected object regions. Splitting each object into n subregions. = atan( fv + yw, fu + xw ) =atan( V 0 + yw, U 0 + xw ) Assigning to each motion region a translational 3D direction. Smoothing motion boundaries inside moving objects. Adding random gaussian noise.

Motion Segmentation Generating training data Generating connected object regions. Splitting each object into n subregions. Assigning to each motion region a translational 3D direction. Smoothing motion boundaries inside moving objects. Adding random gaussian noise.

Motion Segmentation MoA-Net angle field motion segmentation step 2: motion segmentation

Segmentation Results video frame ground truth Jain et al. Tokmakov et al. ours

<latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> <latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> <latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> Segmentation Results Motion Segmentation: Sintel J Mean J Recall J Decay F Mean F Recall F Decay " " # " " # <latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> Tokmakov [1, 2] 50.46 55.43 44.50 53.43 35.04 39.75 Jain et al. [3] 29.63 24.98 36.07 28.65 14.70 31.20 ours 54.77 54.47 26.57 59.71 61.38 14.79

Segmentation Results

Future Research Questions angle field magnitude Importance of the flow magnitude for estimating the scene depth dealing with estimated (noisy) optical flow

-Questions-