MoA-Net: Self-supervised Motion Segmentation. Pia Bideau, Rakesh R Menon, Erik Learned-Miller

Transkrypt

1 MoA-Net: Self-supervised Motion Segmentation Pia Bideau, Rakesh R Menon, Erik Learned-Miller University of Massachusetts Amherst College of Information and Computer Science

2 Motion Segmentation P Bideau, E Learned-Miller, ECCV 2016: It s moving! A probabilistic model for causal motion segmentation P Bideau, A RoyChoudhury, R Menon, E Learned-Miller, CVPR 2018: The best of both worlds: Combining CNNs and geometric constraints for hierarchical motion segmentation P Bideau, R Menon, E Learned-Miller, Workshop ECCV 2018: MoA-Net: Unsupervised Motion Segmentation

3 Overview Motivation How do humans know what is moving in the world and what is not? Approach: Motion Segmentation Rotation compensation Learning Motion Patterns: MoA-Net Results Future Research Questions

4 Motivation

5 Motivation stationary scene moving object no observer motion stationary scene moving object observer motion

8 Motivation All motions result in changes of the retinal image. What is the problem about retinal image motion? photoreceptors are slow motion detection in our brain is challenging Need to stabilize the image, to reduce retinal image motion

9 Motivation

11 Approach: Motion Segmentation MoA-Net optical flow rotation compensated flow step 1: rotation compensation angle field motion segmentation step 2: motion segmentation

12 Approach: Motion Segmentation MoA-Net optical flow rotation compensated flow step 1: rotation compensation angle field motion segmentation step 2: motion segmentation

13 Rotation Compensation rotation + translation optical flow magnitude is dependent on scene depth optical flow angle is dependent on scene depth translation optical flow magnitude is dependent on scene depth optical flow angle is independent of scene depth

14 Rotation Compensation only camera translation and object motion optical flow angle field

15 Motion Segmentation MoA-Net optical flow rotation compensated flow step 1: rotation compensation angle field motion segmentation step 2: motion segmentation

16 Motion Segmentation Definition Def.: Moving Object A moving object is a connected image region that undergoes some independent motion. The connected image region can be of any size and shape.

17 Motion Segmentation Generating training data

18 Motion Segmentation Generating training data Generating connected object regions. Splitting each object into n subregions. Assigning to each motion region a translational 3D direction. Smoothing motion boundaries inside moving objects. Adding random gaussian noise.

20 <latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> <latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> <latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> <latexit sha1_base64="jcr+bgiupgfxbeorwr2wjartaye=">aaacknicbzbnswmxeiazftb6vfxojvjesrxsiqaxpelfo4lbcm0ps2nwhmazszirlqw/x4t/xyshrbz6q0w/dlodcly8zwyteynecoou++hmzm7nlyzmlvllk6tr64wnzzqju824z2iz67sadjdccr8fsn6xaa5rihk96f0oef2baynidyv9hlciufcifazqwu3crro7hige0ayjsafbduqhya3crx/qw9avp9b36rst7y2pvzek7ulrrbijon+fnxffmqnrdug12ylzgngftiixdc9nsjwbrsekh+sbqeejsb7c84avcijuwtno1ahdtu6hhrg2tyedut8nmoim6ueb7ywau2aadc3/wcpf8lsvczwkybublwptstgmw9xor2joupatakaf/stlxdda0kabtyf40yf/fbwjiudwvjvjyvv8ekeobjmduiieosfvckwuiu8yesiv5i28o8/oq/phfi5bz5zjzbb5vc7xn6odowu=</latexit> Motion Segmentation Generating training data Generating connected object regions. Splitting each object into n subregions. = atan( fv + yw, fu + xw ) =atan( V 0 + yw, U 0 + xw ) Assigning to each motion region a translational 3D direction. Smoothing motion boundaries inside moving objects. Adding random gaussian noise.

26 Motion Segmentation MoA-Net angle field motion segmentation step 2: motion segmentation

28 Segmentation Results video frame ground truth Jain et al. Tokmakov et al. ours

29 <latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> <latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> <latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> <latexit sha1_base64="gqmw2dqec4tllt+0miwq63aa5ec=">aaab8xicbvdlsgmxfl2pr1pfvzdugkvwvwze0juu3lisyb/ydiwtztrqtdikgusz+hduxcji1r9x59+ytrpq1gobwzn3khtpmahurod9o8la+sbmvng7tlo7t39qpjxqgpvqyhpucaxbitfmcmkallvb2olmja4fa4wj25nfemlaccuf7crhquwgkkeceuukx25fjsxrwo175ypx9ebaq8tpsqvy1hvllxemacykpyiy0/g9xayz0zztwaalbmpyquiidfjhuulizojsvveunzmljyol3zmwz9xfiyzexkzi0e3gxa7nsjct//m6qy2ug4zljlvm0svhusqwvxh2pu5zzagve0ci1dztiumqaektk6nksvcxt14lzyuq71x9+8tk7savowgncarn4mmv1oao6taachke4rxekeev6b19leylkm8cwx+gzx/92jez</latexit> Segmentation Results Motion Segmentation: Sintel J Mean J Recall J Decay F Mean F Recall F Decay " " # " " # <latexit sha1_base64="eiq6itqci3xxiustr8cvblw1si8=">aaab73icbvdlsgnbeoz1gemr6thlyba8hv0r9cqblx4jmacks+idzczdzmfwmvklhpyefw+kepv3vpk3tpi9agjbq1hvtxdxlapuro9/eyura+sbm4wt4vbo7t5+6ecwyvsmkattjzrurwiy4jlvlbectvlnmikea0bdm6nffgtaccxv7shlyyj9ywno0tqp1cls1fo9dutlv+lpqjzjkjmy5kh1s1+dnqjzwqslao1pb35qwzfqy6lgk2inmyxfosq+azsqmwemhm/unzbtp/rirlqraclm/t0xxssyurk5zgttwcx6u/e/r53z+cocc5lmlkk6xxrnglhfps+thtemwjfybknm7lzcb6irwhdr0yuqll68tbrnlccvbhcx5ep1hkcbjueezicas6jcldsgdhqepmmrvhkp3ov37n3mw1e8foyi/sd7/aftnjay</latexit> Tokmakov [1, 2] Jain et al. [3] ours

30 Segmentation Results

32 Future Research Questions angle field magnitude Importance of the flow magnitude for estimating the scene depth dealing with estimated (noisy) optical flow

33 -Questions-