AMAS: Attention Model for Attributed Sequence Classification

Transkrypt

1 <latext sha1_base64="u5xncmv0ktxombnzjkhvjvo2ai=">aaab/3cbzdptsjaeman+a/xh+rrsymx8ural3ok8ciro4ajngs7tmug7xazuzuhdqcfwks+gjfj1ufxcxwnf+hbwc/z5jdvzjkzxyg508bzvp3sxubw9k55t7k3f3b4vd0+6eg0uxtbnowpegyjrs4etg0zhb+lqpkehlvh+hzw7z6h0wvd2ymuhilfjekdhwupcdf1ctexvvlncd/ajqukg1qp70hynnehsgcqj1z/ekcxkdkmcp5v+plesoyx9wkkqao8vmpu/fcokm3spv9wrhz9+9ethktj0looxnrnq1njp/q/uye90eormymyjoylgucdek7uzf7pappizplbcqml3vpsocdu2nautmywv4nhascn4qzmsq+eq7lu+82qnzpfrgc7ghc7bh2toqbna0aykmbzak7w5z8678+f8llpltjfzcktyvn4b5aywgg==</latext> <latext sha1_base64="a4gdxlto5jabzev2+erjztv9cjg=">aaab/3cbzbltgjbeizr8ix4ql266uhmxjeznrokccmsozwsmjcepgy69prmuntmyisfb3crr3bn3hout+a1bgawav5jj1/+qkpv/0eudau++0utrz3dvek+6wdw6pjk/lpwvvhqwlyyrgivtegggwx2dlccowmcmkucowek7t5vfoesvnyppppgn5er5khnffjrydkubuuk27vxyhsgpddbxi1b+wf/jbmaytsmeg17nluyvymksozwfmpn2pmkjvqefysshqh9rpfqtnyzz0hcwnlnzrk4f6dygk9tqkbgdezvv1+bmf7veasjbp+mysq1ktlwupokymmz/tyzcitnaoeyxe2thi2poszydfa2pmliiu5mjzumt57djrrrvc/yvvupn/kmnabl3anhtxahrrqhbywgmelvmkb8+y8ox/o57k14oqz57a5+sx50wwgw==</latext> <latext sha1_base64="hs+0g4zhdq59q6az3y7yjjbno18=">aaab/3cbzblsgnbeizr4vgv9slm8yguaozutblwe2wec0dkh0dgqsjj0ztxepeiyspibbpyi7cetrpihxsjpmwt+0pdxvxvv/qdscg1c99spbgxube8ud0t7+wehr+xjk5zouswwyrkrqe5anqoey9nwi7ajfdioengoxnezevsjlezj/ggmev2idmmeckantr5k/7pfrrhvdy6ydl4ofcjv6jd/eooeprhghgmqdddzpfezqgxnaqelxqprujamq+xajgme2s/mp07jhxugjeyufbehc/fvreyjrsdrydsjakz6ttyz/6t1uxpe+hmpzwowzotfysqiscjs32taftijjhyou9zestiksqmtwdpsyqhcne8ldlkvnuc1qf1vfus37uvwj3pqahnca6x4men1kaodwgcgyg8wcu8oc/ou/phfc5ac04+cwplcr5+aejeloq=</latext> <latext sha1_base64="cni8g7zineynnzmemfrpe11gc=">aaab/3cbzblsgnbeizr4vgv9slm8yguaoziugy4cbloybyrb6ojvjk56zprthcemwhscthsgdupuonsbr2elmyrj/apj4q4qq/gmpudau++0unja3tneku6w9/ypdo/lxsusnqwlyzilivcegggwpswm4edrcmkucgwh47tzvf2esvmkfjqtx5ehzepoapgwg+yf90vv9yqoxdzby+hcurq9ms/vuhc0ghjwwtvuuu50vgzvyyzgdnsl9uokrvtixytxjrc7wfzu6fkwjodebkvtquft3iqor1pmosj0rnso9wpuz/9w6qqlv/yzhmjuys8wmbxejgt2bzlgcpkrewuukw5vjwxefwxgpro0jzvdhtelmwy3moo69c6qnqw791krz5nviqzoidl8oagalchbjsbwrbe4bxengfn3flwphetbsefoyulov+/6newhq==</latext> <latext sha1_base64="zj4uys8vpwn7odxdrfwhvojj5tu=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xnja3tqu1+s6djlpfomneasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgiv2u4j+6k8cigbwgom682mcvfkagtrmuk27xe1mp6pkccygr/rsdqllqzqarowsrqd72wrvjg8te+awvvzlgyfsx0dgi61hkw+vetxper43jv/rdvmttvozl0lqqllpojav2mr4hcwouajmxmgcyhs3t2l2tbvlxsy/syvnbgpgmfdsmmq+h0xw0dwhft+6jfz1kvez7amddiqiokdtdi1uuacx9ia+0df6kx04zafq1kdsp1r4dtfmoxu/xeu4uw==</latext> <latext sha1_base64="e3suspg05b38qeervndokx/7ouk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xnja3tqu1+s6djlpfomneasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgaxq3hnrtwouafkcbrrxap+9igzpbniwqbxuejcx/ywqw5mavnjlnssudekauhzkgohuz5o9ot60tiddwnkvdz6wfx0zjbqerb5vrtq86/nempyv101n2opnxcapacmm8juybpjcba44aqyesmlkfpc3orzm1wugrv/zjy0gsayv6xyzd5hbbbq/oewhzrntrxruzlti8o0be6by10tw6qr3e0bv6qt/op/thlj2qu59knvlh2uuz5ez9asxkufq=</latext> <latext sha1_base64="4aybh8rx2pm0qpnvgvrg0n1dlk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaola6ihhgbeqpultfjnmpvh0nsh1qfex/8rdywdhyycqtgwjlyzae3r13d3p+irg2rvtwchywl5zxyquvtfwnzwqtvvwg41qxalnyxorrpxoel9a23ah4tbtqybfq8yex437ngztmsbw3owt6er1ihnjgjaw8mt8lilteyarmwounimzu6ulpcot48nmy74cv+zkvkh0dvpdwmxoukohdoteu/eqzxci3dseb6qajrqutde7amxxcynqbomqnzd4amn1floboqv3qphosyir1a10jji9d9bli3x/uwcxayk/ulwrp2ryojkdajylfkjonpdsbk//1uqkjw/2myyq1innvojav2mr4hcwouajmxmgcyhs3t2l2rbvlxsy/tsvnbgpgmfdsmmq2h3nw0dwft+6jfpriqmy2kv76aardibo0tw6qw3e0cv6rf/ou/tulj2qu/+voqxcs42mytn5ackwufy=</latext> <latext sha1_base64="suskbt71ekg0cgn6xemzyv6ml8m=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4faijggfgqlqgtvxkodffqunetgoqovxafgizaymrblhtbtpyzuth537x1zpbkrg2nvdeczawv1bx1jeqm1vbo7vu3n5hj5l0gajsnrtqduilqftubhwlcqgcsdgmrhejfupl6a0t+sdgaxqj+la8ogzaqzlu6wxqmrnjy/lel59bwgr319ffjlpnu8alxskxf3aapfbytxotgxhphg4bys+k3frxun3qtwoclqkoybn33sxcmlitbgao1l3paafu2u4e1bue5mgllihhudxsklj0p18srjar9yjczqoe6xbe/fvre5jrudxymmymmc93xub//w6myla/zzlndmg2xrrlalsejxofodcatnzavltu/yvzmfwxg5j+zjushcmbyvg0yzd6hrdfpnbkr77z6xu2z0tqqoun0jag6rxfobt2nmlodx2hb/rt+xa2hnfzn6jopzw5qdpl1h4bvpe4za==</latext> <latext sha1_base64="e3suspg05b38qeervndokx/7ouk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xnja3tqu1+s6djlpfomneasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgaxq3hnrtwouafkcbrrxap+9igzpbniwqbxuejcx/ywqw5mavnjlnssudekauhzkgohuz5o9ot60tiddwnkvdz6wfx0zjbqerb5vrtq86/nempyv101n2opnxcapacmm8juybpjcba44aqyesmlkfpc3orzm1wugrv/zjy0gsayv6xyzd5hbbbq/oewhzrntrxruzlti8o0be6by10tw6qr3e0bv6qt/op/thlj2qu59knvlh2uuz5ez9asxkufq=</latext> <latext sha1_base64="1vmfuq/vvbb/jx6wo0glc55xy=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4goiyiarec1ibru5zk2x6jr7ycqkd+qn8dmwmgkg26bgbzc2dlrud/1tu6qcq6n571xnixfpewv1bxq+sbm1ra7s9vwsayytfgevuuua2cs2gzbgq8pqpohah4dazxo/7jcyjne/lghn0ytqxpokmgmv5luugennz8uu5hfdfqvjk99exru6aj0enpj0frs5hgcg5bpnr4ryjngjcazj/vpfrxvh3rjwvcclqkoybn33sxsmlitbgao1h3paaxu2u4e1buu5mgllib7uphsklj0l18vljab9yjczqoe6xby/fvre5jrydxymmymmc92xuz//u6mymavzzlndmg2wrrlalsejxkfodcatnaavltu/yvzmfwxg5j+1juv7cmbqvg0yzdahedfuhbor77z6xu2z0sqqox10ag6rxfobt2fmlodx2hb/rt+xdwhnfznabopzzzq1pl1h4bwdc4zq==</latext> <latext sha1_base64="suskbt71ekg0cgn6xemzyv6ml8m=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4faijggfgqlqgtvxkodffqunetgoqovxafgizaymrblhtbtpyzuth537x1zpbkrg2nvdeczawv1bx1jeqm1vbo7vu3n5hj5l0gajsnrtqduilqftubhwlcqgcsdgmrhejfupl6a0t+sdgaxqj+la8ogzaqzlu6wxqmrnjy/lel59bwgr319ffjlpnu8alxskxf3aapfbytxotgxhphg4bys+k3frxun3qtwoclqkoybn33sxcmlitbgao1l3paafu2u4e1bue5mgllihhudxsklj0p18srjar9yjczqoe6xbe/fvre5jrudxymmymmc93xub//w6myla/zzlndmg2xrrlalsejxofodcatnzavltu/yvzmfwxg5j+zjushcmbyvg0yzd6hrdfpnbkr77z6xu2z0tqqoun0jag6rxfobt2nmlodx2hb/rt+xa2hnfzn6jopzw5qdpl1h4bvpe4za==</latext> <latext sha1_base64="zj4uys8vpwn7odxdrfwhvojj5tu=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xnja3tqu1+s6djlpfomneasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgiv2u4j+6k8cigbwgom682mcvfkagtrmuk27xe1mp6pkccygr/rsdqllqzqarowsrqd72wrvjg8te+awvvzlgyfsx0dgi61hkw+vetxper43jv/rdvmttvozl0lqqllpojav2mr4hcwouajmxmgcyhs3t2l2tbvlxsy/syvnbgpgmfdsmmq+h0xw0dwhft+6jfz1kvez7amddiqiokdtdi1uuacx9ia+0df6kx04zafq1kdsp1r4dtfmoxu/xeu4uw==</latext> <latext sha1_base64="55emaey4r0asqk+uwtj5pqz7rjm=">aaacncbvhltswfjxdv5rjdnj2u+hgtsjmigqr0yyzzjo0hwazlskdkmbk+torkwujys3b+a/6g/mfs+8xvml40twuehzopvdhnbukgmvjefeo++tps+cvk+rz7xzvxf112/goskugxflrkkmaduguisr4ubanfva40dajfh/3fynv0bpnsg7s0lhedol5bfn1fjkr/+zhxdz2d1luytfvokwyh9+/1lkph/v6vttkxd1sbaidepd53oleksj3nhughiutnttujxa/viuz0fkll/k19vem1vv/gykbi0uvldv/4wdxowxsane1trgffss8pmpwjscazhpsynv3czejjy9clfodc4i+wcxgukhulwtv2/4mcxlpv4saqy2pw+rc3ju/1zpmj+oucyzqzinmjuzqjbbk8xqeouqjmxmycyhs3f8vsrrvlx5qzyvlt9ms6pnhhzmcazgntax+ifxvb2ugvxqe9ral4ghrpfazrei8tqx+ed88fpudw34167n49s1yln3qk9cr/9a0f7xqu=</latext> <latext sha1_base64="suskbt71ekg0cgn6xemzyv6ml8m=">aaacyxcbza7t8mwfixd8c6vagmxwqjaag4faijggfgqlqgtvxkodffqunetgoqovxafgizaymrblhtbtpyzuth537x1zpbkrg2nvdeczawv1bx1jeqm1vbo7vu3n5hj5l0gajsnrtqduilqftubhwlcqgcsdgmrhejfupl6a0t+sdgaxqj+la8ogzaqzlu6wxqmrnjy/lel59bwgr319ffjlpnu8alxskxf3aapfbytxotgxhphg4bys+k3frxun3qtwoclqkoybn33sxcmlitbgao1l3paafu2u4e1bue5mgllihhudxsklj0p18srjar9yjczqoe6xbe/fvre5jrudxymmymmc93xub//w6myla/zzlndmg2xrrlalsejxofodcatnzavltu/yvzmfwxg5j+zjushcmbyvg0yzd6hrdfpnbkr77z6xu2z0tqqoun0jag6rxfobt2nmlodx2hb/rt+xa2hnfzn6jopzw5qdpl1h4bvpe4za==</latext> <latext sha1_base64="zj4uys8vpwn7odxdrfwhvojj5tu=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xnja3tqu1+s6djlpfomneasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgiv2u4j+6k8cigbwgom682mcvfkagtrmuk27xe1mp6pkccygr/rsdqllqzqarowsrqd72wrvjg8te+awvvzlgyfsx0dgi61hkw+vetxper43jv/rdvmttvozl0lqqllpojav2mr4hcwouajmxmgcyhs3t2l2tbvlxsy/syvnbgpgmfdsmmq+h0xw0dwhft+6jfz1kvez7amddiqiokdtdi1uuacx9ia+0df6kx04zafq1kdsp1r4dtfmoxu/xeu4uw==</latext> <latext sha1_base64="e3suspg05b38qeervndokx/7ouk=">aaacyhcbva9t8mwehxdvykfbwgdxajcykaolkj0rgcaerafplakhodsrdpozdugksr/4y+wsrcxwojbzqatj1t6evfe3en5edauo57yvlaxlldk69xnja3tqu1+s6djlpfomneasnn2oqxelhccpgkvfai1/aoz+8hpcfx0bphst7m0qgh9gb5cfn1fjkq/m9aelrnuzkjlx6coi8u7u6ydpsoj1utuzlcwvb5osucl3z7jgqqypuwdg7kfgaxq3hnrtwouafkcbrrxap+9igzpbniwqbxuejcx/ywqw5mavnjlnssudekauhzkgohuz5o9ot60tiddwnkvdz6wfx0zjbqerb5vrtq86/nempyv101n2opnxcapacmm8juybpjcba44aqyesmlkfpc3orzm1wugrv/zjy0gsayv6xyzd5hbbbq/oewhzrntrxruzlti8o0be6by10tw6qr3e0bv6qt/op/thlj2qu59knvlh2uuz5ez9asxkufq=</latext> Abstract AMAS: Attenton Model for Attrbuted Sequence Classfcaton Zhongfang Zhuang Xangnan Kong Elke Rundenstener Classfcaton over sequental data s mportant for a wde range of applcatons from nformaton retreval, anomaly detecton to genomc analyss. Neural network approaches, n partcular recurrent neural networks, have been wdely used n such tasks due to ther strong capablty of feature learnng. However, recent nnovatons n sequence classfcaton learn from not only the sequences but also the assocated attrbutes, called attrbuted sequences. Whle recent work shows the attrbuted sequences to be useful n real-world applcatons, neural attenton models have not yet been explored for attrbuted sequence classfcaton. Ths paper s the frst to study the problem of attrbuted sequence classfcaton wth the neural attenton mechansm. Ths s challengng that now we need to assess the mportance of each tem n each sequence consderng both the sequence tself and the assocated metadata. We propose a framework, called AMAS, to classfy attrbuted sequences usng the nformaton from the sequences, metadata, and the computed attenton. Emprcal results on real-world datasets demonstrate that the proposed AMAS framework sgnfcantly mproves the performance of classfcaton over the state-of-the-art methods on attrbuted sequences. 1 Introducton Classfcaton over sequental data s mportant for a wde range of applcatons over a varety of research areas, ncludng document classfcaton n nformaton retreval and gene classfcaton n genomc analyss. The conventonal approaches to sequence classfcaton focus on only the sequental data, whereas the metadata assocated wth the sequences s often dscarded. However, metadata s has been shown to be useful for a varety of topcs [30, 26]. In the lterature, the data type of a sequence and ts assocated metadata as a unty s called attrbuted sequence. Attrbuted sequences are heterogeneously structured and thus not naturally represented as fxed-sze vectors. For example, genes can be represented as attrbuted sequences, where each gene conssts of a DNA sequence and a set of attrbutes {zzhuang, xkong, rundenst}@wp.edu, Computer Scence Department, Worcester Polytechnc Insttute Attrbuted Sequences p1 - p4 <latext sha1_base64="u5xncmv0ktxombnzjkhvjvo2ai=">aaab/3cbzdptsjaeman+a/xh+rrsymx8ural3ok8ciro4ajngs7tmug7xazuzuhdqcfwks+gjfj1ufxcxwnf+hbwc/z5jdvzjkzxyg508bzvp3sxubw9k55t7k3f3b4vd0+6eg0uxtbnowpegyjrs4etg0zhb+lqpkehlvh+hzw7z6h0wvd2ymuhilfjekdhwupcdf1ctexvvlncd/ajqukg1qp70hynnehsgcqj1z/ekcxkdkmcp5v+plesoyx9wkkqao8vmpu/fcokm3spv9wrhz9+9ethktj0looxnrnq1njp/q/uye90eormymyjoylgucdek7uzf7pappizplbcqml3vpsocdu2nautmywv4nhascn4qzmsq+eq7lu+82qnzpfrgc7ghc7bh2toqbna0aykmbzak7w5z8678+f8llpltjfzcktyvn4b5aywgg==</latext> <latext sha1_base64="cni8g7zineynnzmemfrpe11gc=">aaab/3cbzblsgnbeizr4vgv9slm8yguaoziugy4cbloybyrb6ojvjk56zprthcemwhscthsgdupuonsbr2elmyrj/apj4q4qq/gmpudau++0unja3tneku6w9/ypdo/lxsusnqwlyzilivcegggwpswm4edrcmkucgwh47tzvf2esvmkfjqtx5ehzepoapgwg+yf90vv9yqoxdzby+hcurq9ms/vuhc0ghjwwtvuuu50vgzvyyzgdnsl9uokrvtixytxjrc7wfzu6fkwjodebkvtquft3iqor1pmosj0rnso9wpuz/9w6qqlv/yzhmjuys8wmbxejgt2bzlgcpkrewuukw5vjwxefwxgpro0jzvdhtelmwy3moo69c6qnqw791krz5nviqzoidl8oagalchbjsbwrbe4bxengfn3flwphetbsefoyulov+/6newhq==</latext> Classfcaton on Attrbutes Classfcaton on Sequences Classfcaton on Attrbuted Sequences P4 can only be <latext sha1_base64="55emaey4r0asqk+uwtj5pqz7rjm=">aaacncbvhltswfjxdv5rjdnj2u+hgtsjmigqr0yyzzjo0hwazlskdkmbk+torkwujys3b+a/6g/mfs+8xvml40twuehzopvdhnbukgmvjefeo++tps+cvk+rz7xzvxf112/goskugxflrkkmaduguisr4ubanfva40dajfh/3fynv0bpnsg7s0lhedol5bfn1fjkr/+zhxdz2d1luytfvokwyh9+/1lkph/v6vttkxd1sbaidepd53oleksj3nhughiutnttujxa/viuz0fkll/k19vem1vv/gykbi0uvldv/4wdxowxsane1trgffss8pmpwjscazhpsynv3czejjy9clfodc4i+wcxgukhulwtv2/4mcxlpv4saqy2pw+rc3ju/1zpmj+oucyzqzinmjuzqjbbk8xqeouqjmxmycyhs3f8vsrrvlx5qzyvlt9ms6pnhhzmcazgntax+ifxvb2ugvxqe9ral4ghrpfazrei8tqx+ed88fpudw34167n49s1yln3qk9cr/9a0f7xqu=</latext> correctly classfed wth attrbuted sequences. Fgure 1: Usng both attrbutes and sequences s essental for predctng the of p 4. (e.g., PPI, gene ontology) ndcatng the propertes of the gene. Network traffc can also be modeled as attrbuted sequences. Namely, each network transmsson sesson conssts of a sequence of packages beng sent (or receved) by a router and a set of attrbutes characterzng the network traffc, (e.g., date, tme, sze sender prorty etc). Desgnng a classfer to correctly predct the class of an attrbuted sequence s benefcal for applcatons such as automatc network traffc proflng for cyber securty. However, t has been shown that t s challengng to classfy sequences by themselves [31, 24, 19, 25]. Wth dependences between attrbutes and sequences, the attrbuted sequence classfcaton must tackle several mportant desgn challenges. In ths paper, we study the problem of attrbuted sequence classfcaton. Ths problem s dfferent from pror sequence classfcaton work [24, 19] as we now need to extract feature vectors from not only the sequences but also ncorporatng the metadata as the attrbutes, along wth the dependences between attrbutes and sequences. We summarze the challenges below. Attrbute-sequence dependences. Contrary to the smplfyng assumpton that the attrbutes and sequences are ndependent, there are varous dependences between them. Dependences between at- 109

2 (a) Classfcaton on data attrbutes [1] (c) Image classfcaton wth attenton [20]. 0.5 (b) Sequence classfcaton [31] logn search search book search (d) Sequence classfcaton wth attenton [28] logn search search book search (e) Sequence classfcaton wth attrbute-guded attenton. (ths paper) Fgure 2: A comparson of related classfcaton problems. trbutes and sequences rase problems when learnng to classfy attrbuted sequence data. For example, n network traffc data, the devce type of the router(.e., an attrbutes) may affect the pattern of sendng/recevng TCP/UDP packets (.e., sequences). Snce conventonal sequence classfcaton approaches focus on a sngle data type only, these dependences would thus not be captured. Attenton from both attrbutes and sequences. Recent sequence learnng research [34] has used neural attenton models to mprove the performance of sequence learnng, such as document classfcaton [34]. However, the attenton mechansm focuses on learnng the weght of certan tme steps or sub-sequences n each sequental nstance, wthout regards to ts assocated attrbutes. Wth nformaton from the attrbutes, the weght of tem or sub-sequence may be drastcally dfferent from the weght calculated by the attenton mechansm usng only sequences, whch would consequently have dfferent classfcaton results. To address the above challenges, we propose an endto-end soluton for attrbuted sequence classfcaton wth an attenton model, called AMAS. The AMAS model ncludes three man components: a Attrbute Net to encode the nformaton from attrbutes, a Sequence Net to encode the nformaton from sequences, and an Attenton Block learnng the attenton from not only the sequences but also the hybrd of nformaton from both attrbutes and sequences. Our paper offers the followng contrbutons: We formulate the problem of attrbuted sequence classfcaton. We desgn a deep learnng framework, called AMAS, wth two attenton-based models to explot the nformaton from attrbutes and sequences. We demonstrate that the proposed models sgnfcantly mprove the performance of attrbuted sequence classfcaton usng performance experments and case studes. 2 Problem Formulaton 2.1 Prelmnares. We ntroduce the key defntons n ths secton. The mportant notatons are summarzed n Table 1. Defnton 1. (Sequence) Gven a fnte set I com- posed of r categorcal tems, a sequence s = x (1),, x (t) s an ordered lst of t tems, where x (t) I. We use the subscrpt to dstngush dfferent sequence nstances. Followng common pre-processng steps for deep learnng, we apply zero-paddng on the sequences of varable-length, n whch each sequence s padded to the maxmum length of the sequences n a dataset usng 0 s. The second step s to one-hot encode each sequence s [10]. We denote the one-hot encoded form of sequence s as a matrx s = x (1),, x (t). Learnng models are capable of dsregardng the paddng so that the paddng has no effect on the tranng of models. Defnton 2. (Attrbuted Sequence) Gven a u- dmensonal attrbute vector v s composed of u attrbutes, an attrbuted sequence p s a par composed of an attrbute vector v and a one-hot encoded sequence s, denoted as p = (v, s ). 2.2 Problem Defnton. We formulate the attrbuted sequence classfcaton problem as the problem of fndng the parameters θ of a predctor Θ that mnmzes the predcton error of class s. Intutvely, we want to maxmze the possblty of correctly predctng s when gven a tranng set P = {p 1,, p k } of k attrbuted sequences. Thus, we formulate the tranng process as an optmzaton process: (2.1) arg mn θ Pr(l ) log Pr (Θ (p )) 110

3 Table 1: Important Mathematcal Notatons Notaton Descrpton R The set of real numbers. P A set of attrbuted sequences. r The number of all possble tems n sequences. s A sequence of categorcal tems. x (t) The t-th tem n sequence s. t max The maxmum length of sequences. s A one-hot encoded sequence n the form of a matrx s R t max r. x (t) A one-hot encoded tem at t-th tme step. v An attrbute vector. p An attrbuted sequence..e.,, p = (v, s ) p A feature vector of attrbuted sequence p. µ Attenton weghts. α Attenton vector. Our goal s to fnd the parameters that mnmze the categorcal cross-entropy loss between the predcted s usng parameters n functon Θ and the true s for all attrbuted sequences n the dataset. 3 Attrbuted Sequence Attenton Mechansm The proposed AMAS model has three components, one Attrbute Net for learnng the attrbute nformaton, one Sequence Net to learn the sequental nformaton, and one Attenton Block to learn the attenton from both attrbutes and sequences. 3.1 Network Components Attrbute Net. We buld Attrbute Net usng fully connected neural network denoted as: (3.2) f (A; W r, b r ) = tanh(w r A + b r ) where W a and b a are two tranable parameters n the Attrbute Net, denotng the weght matrx and bas vector, respectvely. We use the actvaton functon tanh here n our Attrbute Net based on our emprcal studes. Other choces, such as ReLu or sgmod, may work equally well n other real-world scenaros. When gven an attrbuted sequence p = (v, s ), Attrbute Net takes the attrbutes as nput and generates an attrbute vector r = f(v ; W r, b r ). Dfferent from prevous work n [4, 27] usng stacked fully connected neural network as autoencoder, where the tranng goal s to mnmze the reconstructon error, our goal of Attrbute Net s to work together wth other network components to maxmze the possblty of predctng the correct s Sequence Net. Dfferent from the attrbutes beng unordered, tems n our sequences have a temporal orderng. The nformaton about the sequences s n both the tem values and the orderng of tems. The temporal orderngs requre a model that s capable of handlng the dependences between dfferent tems. There have been extensve studes on usng recurrent neural networks (RNN) to handle temporal dependences. However, RNN suffers from the problem of explodng and vanshng gradent durng the tranng, where the gradent value becomes too large or too small and thus the network becomes untranable. Long Short- Term Memory (LSTM) [13] s desgned as one varaton and expanson of the RNN to handle such ssues. LSTM s capable of rememberng values over long tme ntervals by ntroducng addtonal nternal varables (.e., varous gates and cell states ). We use an LSTM to handle the dependences. Wth a varable X (t) at tme t, the Sequence Net can be expressed as: (3.3) (t) = σ W X (t) + U h (t 1) + b f (t) = σ W f X (t) + U f h (t 1) + b f o (t) = σ W o X (t) + U o h (t 1) + b o g (t) = tanh W c X (t) + U c h (t 1) + b c c (t) = f (t) c (t 1) + (t) g (t) h (t) = o (t) tanh c (t) where denotes the btwse multplcaton, σ s a sgmod actvaton functon, (t), f (t) and o (t) are the nternal gates of the LSTM, and c (t) and h (t) are the cell and hdden states of the LSTM, respectvely. We denote the Sequence Net as: (3.4) g(x (t) ; W s, U s, b s ) = h (t) where W s = [W, W f, W o, W c ], U s = [U, U f, U o, U c ] and b s = [b, b f, b o, b c ]. Wth the sequence s = x (1),, x (t) as part of an attrbuted sequence p, the hdden states for nput x (t) are g x (t) ; W s, U s, b s = h (t) Attenton Block. Recent work [20, 6] has dentfed that even LSTM-based solutons cannot fully handle the sequence learnng on long sequences that the nformaton over a long tme may be lost. One popular soluton to ths problem s to ncorporate the attenton mechansm nto the model. The attenton mechansm effectvely summarzes the data wth the am to leverage the mportance of each tem n the sequental nput. Attrbuted Sequence Attenton (ASA). Dfferent from the common sequence attenton models, we now need to ncorporate the attrbute nformaton nto 111

4 <latext sha1_base64="+newlwycyydy8+le/yujfrp7bbk=">aaacbxcbzc7sgnbfiznvcz41ralabbkuzaabmwsrnbxcs7hnnj2wtizowwfyesqx0aw30eo7h1oxwcx8njsovj/ghg4z/ncm78wrumyd49jy2t7z3dkt75f2dw6pjyslpw2dwewrjgwqm2anjapogwoydkuczbmgnwr8n6t3nkbpmokhm5eqczwunkueg2c9rpinebd92q9ug1owl78oyqfvvkjzr/xeg4xydsiqhrxuhye0cy6voytbtbxzdrktmr5cz6hahhsczw+e+pfogfhpptwtxp+7fydyzlwe8mr1cmxgeru2m/+r9axjb+occmkncljylfrmm8yf/d4fuaxesikdtbr1t/pkhbumxmw0tmxkoqiyt8sumxa1h3vox9dcx/dbtd4omqhc3sbrlcibladnvattrbbhl2gv/tmpxvv3of3uwjd8iqzm7qk7+sxbhszxw==</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="gtmv6ivthka+4horls3tcoxpqlm=">aaacfhcbvdlssnafj34rpuvfdy4gsxcxvgsxey4kblcvybbqyt6u07dpjgzkumn/wa9zqj7gtt+79an/dszufbt1w4xdovdzd8wloplksb2nldw19y7o0vd7e2d3bnw8o2zjkbiuwjxgkuh6rwfkilcuuh24sgaqeh443vs39zmiyalwxk1caiydjnpkffacs3jfkduypptueayh7sqxhzxdz65zswqwvpgzwixpiiknf3zpz+iabjaqcgnuvzsk1zosorlenw7csyklhzag9tumsghtsaf4mn2llgp1i6akvnqp/l1issdkjpl2zp5wlx7+5/us5d84kqvjrefiz4/8hgmv4bwmpgacqoittqgvtgffdeqeoupxnvclyccyjyvdtp2yg/lph1zszw/syr1rtfrcz2gu1rfnrpgddratdrcfd2hf/sk3oxn4934md5nqytgcxoe5mb8/qigqz6q</latext> <latext sha1_base64="xg38qqc07e6ma225thsezgyyqsy=">aaab/ncbzdlsgmxfiyzxmu9jyorn8e1ivlrhe6lljpsok9qdsomttthazkgsemgz4km5ckolw53dn252s9dwhwif/zmhc/jhklftpo/bwvldw9/ylg2vt3d29/bdg8o2tlkfsqsnlfhdcgncqcatqw0jxaki4hejnwh8o613honsnbh3zjjwnfq0jhzkwvusd9yanm5f9ykomppacxp3novvxat5mcbn8aqgudn0v/qdbkeccimz0rrne9iegvkgykbycj/vrci8rkpssygqjzrizufn8mw6axgnyj5h4mz9pzehrvwer7atizps7wp+v+tl5r4jsokkkhas8xxsmdjohtlocakoinm1hawff7k8qjpba2nrgydcff/piytc9rvuu7r1jvfhguwak4bvxgg2tqbw3qbc2aqqaewst4c56cf+fd+z3rjjfzbh4i+fzb/fhlm0=</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="ke/ptvz4zdrkas+upaaqfp1c+2q=">aaacfhcbvdlssnafj34rpuvfdy4crahlxjn7osuomygn1ag8nketmonuzczeqomb/hb7jvt3anbt37bf6gkzyl23rgwugce7mh48emsmxb38ba+sbm1nzpp7y7t39wab4dd2sucajterfi9hwsgveobuuvg14saic+g64/uc397missn+r6yxuceecrpqgpwwppn0egi19on0nhn0ia0qj17vlzpprng1ewzrltgfqaaclc/8gqwjkotafwfyyr5jx8pnsvcummjkg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqrnaziwfjkpxtvaluccyjkvdtpocg+rpfovozrf2zvgs+ohm7qoaob12jbmqfmojgp7qc3pfb8az8w58gj/z1twjudlbczc+fgefdp6p</latext> <latext sha1_base64="l7u7aa1n2mxz5g7ot1dxzmoeikg=">aaab/ncbzdlsgmxfiyzxmu9jyorn8e1ivlphtdftx0wcfeob2htjppq5nmsdjcgqz8ftcufhhrc7jzbuzbwwjrd4gp/5zdofkjyag2nvftrk1vbg5tl3bku3v7b4fu0xfhj6ncpi0tlqhehdrhvjc2oyarnlqe8yrbjs5ndw7j0rpmoh7m5uk4ggkaewxmtyk3dob5fem85a+zfutungf65d56fa8mjcxxaw/gaoo1ardr8ewwsknwmcgto77njrbhpshmjg8peg1kqhp0ij0lqreq6y+fk5vldoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykysntaxsq/cxv7wknxrnt3znvrrnio4soapnoap8ca0aoalaoa0wymazeavvzppz4rw7h4vwnaeyoqf/5hz+ao/blmw=</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="ppmkzzdoerbr3fzpsrauq4anq2e=">aaacf3cbzc7sgnbfiznvcz4w7usm8egxmkwq4wwazuuecwfknwznzwkq2z3h7kiyqk+hw9gq49gj7awpogv4ersmmqfbj7+cw7nzb8jzpt2vg9nzxvtfwmzt5xf3tnd23cpdusqnzjcjay8lc2ikoasgzpmmkntscbxxkerdw7h9cyjsmxs5f4pbqqx6swsyyjr1grd47aii9w2gndrjyf7yio6zbdx56pqlxglbyk8dp4mcmmauj+tdspntekmnkvmv3ha4yijwjheb5tlegcb2qhrqsjqgfwstl4zwmxu6ujtk+xknj+7fyzesg3jyhbgrpfvym1s/ldrgd29ctkwckmhodnfxcoxtve4d9xheqjmqwuesmzvxbrpjkhapja3xyebbm8jyzfzghzahflnzld16hxjlllemn6bqvky+uurlvubxveevp6aw9ojfn2xl3ppzpaeukm5s5qnnyvn4b3lufna==</latext> <latext sha1_base64="h6mcvzrsgm5unpml8x+0mrweya=">aaacf3cbzc7sgnbfiznvcz4w7usm8egxmkwm0blge3kcoycsvxmjyfjknndys5cwbafwwew1uewe1tln8dxchiptoipax//oydz5g8fz0p73reztr6xubwd28nv7u0fhlphxw2vgemhthoeyfzifhawq10zzaeljjao5namr7etevmrpgjjfk/harorgcsszyjr1grc046iqtwxgnaxjaf7sis6yfflyyxwc17jmwqvgj+hapqrfrg/nv5ctqsxppwo1fy9obspkzprdlm+yxqiqkdkag2lmyladdppfzj8yz0e7fsvljjqft3iwruumotj0r0uo1xjuy/9xarvdvumlhdeq09mvufyj3sb+4xcvtzsqvcjbo3yjokklbtu1vyysraaoyyve3gx85hfrrlkm/5ztuqvomcugmnam8te1qqaqqqe6ougjvabx9oy8o+/oh/m5a11z5jmnaeho1y/cwj+b</latext> <latext sha1_base64="sgwewobqlwop/xciuubipjmyn+4=">aaacf3cbzdlsgmxfiyz9vbrrepk3aslubewgte6lljpsok9qkcomfs0du1mqpiryjd4hd6aw30ed+lwpu/ga5hefrb1h8dhf87hnpyh5ewb1/12cmvrg5tb+e3czu7e/khx8kp40rrancyx6odeg2crdawzhboswvehbxa4eh2um89gtisju7nwejxkehe+owsy62geojlewi/kytliqnyq1o2abv0lrkgwhir7lr4fbw5lnbc9ad44/dmgidove647nstnntkmcsgkfqjbejoa+hyjiga3u2nx8jwuxv6ub8r+ykdp+7fzqircctj2cmkferk3m/2qdxprvumlzgigornf/yrje+njhrjhffddxxyivczeumqkekntw1hsyihcmcufwwy3nioq9c8qnw79xsttbpki9o0rkqiw9doyqqotpqiiqe0at6rw/os/pufdfs9acm585rgtyvn4b2ywfmg==</latext> <latext sha1_base64="mnwra8armv44xzwgxnstrea/fm=">aaacfxcbzc7tsmwfiadcvlfmbgylgokmpsjswwvmlpwcr6kdoqoe5ja9vjlntbqqi8bw/aco/ahlzeqjea7fnqft+ydkn/5yjc/whgjolhefbkm1sbm3vlhcre/shh0f28ulhjamk0kyjt2qviao46gtmebqexjifhdobpo7wb37bfkxjh7quwfereyxcxkl2l+ftyquyahqsbcjinphroa9tlv7ttvp+7mhdfblackcrv8+2cwtggaqawpj0r1xudolynsm8ohrwxsbylqcrlb32bmilbenv9aj+nm8rhis2lnz67fycyek1jqltgre9vqu1mflfrz/q8nblwcxsdtfdlaptjnwcz2ngiznanz8aifqycyumyyij1sazps2pgemasv4xybroaxd57rugr53qo1mkveznamlvemuuken1eqt1eyu5egfvai369l6tz6sz0vryspmttgsrk9f6wefka==</latext> <latext sha1_base64="fihe3xjkd6ju2rqnmqg1cqc+qw=">aaacfhcbvdlssnafj3uv62vqodgtbaidwfj3oy4kblcvybbqyt6u07ddijmxohxpygh+bwp8gduhxvf/gbttosboubc4dz7uuejh8zkpvtfxultfwnza3ydmvnd2//wdw86sgoeqtajgkr6plyaqmc2ooqbr1yaa59bl1/cpv73ucqkkb8xk1jcem84jsgbcsteebjimrq7afpoppoq1pthr10ljlprnp1ewzrltgfqaiclc/8gqwjkotafwfyyr5jx8pnsvcummgqg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqrnaziwfjkpxtvaluccyjjvddpocg+rphnvdzs/s6unztfrgz2m1rddrpgddreldrgbd2hf/sk3oxn4934md7nqywjudlgczc+fgedc56o</latext> <latext sha1_base64="vbafdyn+0zk7w+2mnb7mw66wfo=">aaab/ncbzdlsgmxfiyz9vbrbvrcuqkwos4sm250wxdtzqv7gxycmmmmdu0yickizrjwvdy4umstz+hotzftz6gtpwq+/nmo5+spjkpaen63u1pb39jckm9xdnb39g/cw6ootlkfsrsnlfg9cgncqcbtqw0jpaki4hej3whyo6t3h4nsnbh3zpjwnfi0jhzkwvucdyanm5f9ygomppas+hd56fa9ujcxxaw/gcoo1ardr8ewwsknwmcgto77njrbhpshmjg8mkg1kqhp0ij0lqreq6y+fk5plfoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykysntaxq/cxv7wknau6b/noqzaarrxlcaroqa344bo0qbo0qbtgkifn8arencfnxxl3phatjaeyoqz/5hz+ao5vlms=</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="4stvxtr8vdpbr6fgmjp1ow2tlui=">aaacencbvdlssnafj34rpuvhzs3g0wom5k40wxbtzcv7apagbtm3bozbjmjkin+qs/wk1+gjtx6w/4bf6gkzyl23rgwugce7mh48ecke0439ba+sbm1nzpp7y7t39wab8dt1wusaotgvfidn2gdmblc00h24sgyq+h44/uc39znixsjxr6cxdeiyexglggjefzppyr67afpoppyq1rvhrvmplv1jwz8cpxc1jbbzqe/dmfrjqjqwjkvi914n1icvsm8ohk/ctbtghezkcnqgchkag6sx9h+mmsrbjm0ijwfq34uuhepnq99s5lnvspel/3m9rac3g5sjoneg6pxrkhcsi5xxgydmatv8aghkpmsmi6jjfsbwha+jpfiakyysmngxe5hlbsvaq7hd06l3g6kqezdi6qyexxqi4aqilakin9ije0zv1bl1bh9bnfhxnkm5o0aksr18taz42</latext> <latext sha1_base64="adkmcmxvsu2nnlos2dzqtmth55m=">aaab+ncbzc7tsmwfiadcvllsliylehlavkwgcsxnkxspqtsfyxke1ajuw7ycqkedhyqahvp6ejbfbbtnayy9z+vsfc3so/0gyqo3nftuljc2t7z3ybmvv/+dwyk0ed3wskkw6oggj6kdie0yf6rhqgollrrcpgolf05t5vfdalkajudmzsqkoxolgfcnjrdctdwpmpmh9d6rm5be5kfb8xreqnad/ajqofa7dl+gowsnnadgdj64hvsbblshmjg8sow1uqpevjmraoecc6yban5/dcomyj8o+yedc/t2ria71jee2kymz0au1uflfbzca+driqjcpiqivf8upgyab8xzgcqcdztzqfhreyvee6qqnjatg3bx/3yonqvg77lw6/wbbvxlmepoan14imr0aqt0aydgmejeaav4m15cl6cd+dj2vpypkt8efo5w9qajqf</latext> <latext sha1_base64="+newlwycyydy8+le/yujfrp7bbk=">aaacbxcbzc7sgnbfiznvcz41ralabbkuzaabmwsrnbxcs7hnnj2wtizowwfyesqx0aw30eo7h1oxwcx8njsovj/ghg4z/ncm78wrumyd49jy2t7z3dkt75f2dw6pjyslpw2dwewrjgwqm2anjapogwoydkuczbmgnwr8n6t3nkbpmokhm5eqczwunkueg2c9rpinebd92q9ug1owl78oyqfvvkjzr/xeg4xydsiqhrxuhye0cy6voytbtbxzdrktmr5cz6hahhsczw+e+pfogfhpptwtxp+7fydyzlwe8mr1cmxgeru2m/+r9axjb+occmkncljylfrmm8yf/d4fuaxesikdtbr1t/pkhbumxmw0tmxkoqiyt8sumxa1h3vox9dcx/dbtd4omqhc3sbrlcibladnvattrbbhl2gv/tmpxvv3of3uwjd8iqzm7qk7+sxbhszxw==</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="ppmkzzdoerbr3fzpsrauq4anq2e=">aaacf3cbzc7sgnbfiznvcz4w7usm8egxmkwq4wwazuuecwfknwznzwkq2z3h7kiyqk+hw9gq49gj7awpogv4ersmmqfbj7+cw7nzb8jzpt2vg9nzxvtfwmzt5xf3tnd23cpdusqnzjcjay8lc2ikoasgzpmmkntscbxxkerdw7h9cyjsmxs5f4pbqqx6swsyyjr1grd47aii9w2gndrjyf7yio6zbdx56pqlxglbyk8dp4mcmmauj+tdspntekmnkvmv3ha4yijwjheb5tlegcb2qhrqsjqgfwstl4zwmxu6ujtk+xknj+7fyzesg3jyhbgrpfvym1s/ldrgd29ctkwckmhodnfxcoxtve4d9xheqjmqwuesmzvxbrpjkhapja3xyebbm8jyzfzghzahflnzld16hxjlllemn6bqvky+uurlvubxveevp6aw9ojfn2xl3ppzpaeukm5s5qnnyvn4b3lufna==</latext> <latext sha1_base64="xg38qqc07e6ma225thsezgyyqsy=">aaab/ncbzdlsgmxfiyzxmu9jyorn8e1ivlrhe6lljpsok9qdsomttthazkgsemgz4km5ckolw53dn252s9dwhwif/zmhc/jhklftpo/bwvldw9/ylg2vt3d29/bdg8o2tlkfsqsnlfhdcgncqcatqw0jxaki4hejnwh8o613honsnbh3zjjwnfq0jhzkwvusd9yanm5f9ykomppacxp3novvxat5mcbn8aqgudn0v/qdbkeccimz0rrne9iegvkgykbycj/vrci8rkpssygqjzrizufn8mw6axgnyj5h4mz9pzehrvwer7atizps7wp+v+tl5r4jsokkkhas8xxsmdjohtlocakoinm1hawff7k8qjpba2nrgydcff/piytc9rvuu7r1jvfhguwak4bvxgg2tqbw3qbc2aqqaewst4c56cf+fd+z3rjjfzbh4i+fzb/fhlm0=</latext> <latext sha1_base64="gtmv6ivthka+4horls3tcoxpqlm=">aaacfhcbvdlssnafj34rpuvfdy4gsxcxvgsxey4kblcvybbqyt6u07dpjgzkumn/wa9zqj7gtt+79an/dszufbt1w4xdovdzd8wloplksb2nldw19y7o0vd7e2d3bnw8o2zjkbiuwjxgkuh6rwfkilcuuh24sgaqeh443vs39zmiyalwxk1caiydjnpkffacs3jfkduypptueayh7sqxhzxdz65zswqwvpgzwixpiiknf3zpz+iabjaqcgnuvzsk1zosorlenw7csyklhzag9tumsghtsaf4mn2llgp1i6akvnqp/l1issdkjpl2zp5wlx7+5/us5d84kqvjrefiz4/8hgmv4bwmpgacqoittqgvtgffdeqeoupxnvclyccyjyvdtp2yg/lph1zszw/syr1rtfrcz2gu1rfnrpgddratdrcfd2hf/sk3oxn4934md5nqytgcxoe5mb8/qigqz6q</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="h6mcvzrsgm5unpml8x+0mrweya=">aaacf3cbzc7sgnbfiznvcz4w7usm8egxmkwm0blge3kcoycsvxmjyfjknndys5cwbafwwew1uewe1tln8dxchiptoipax//oydz5g8fz0p73reztr6xubwd28nv7u0fhlphxw2vgemhthoeyfzifhawq10zzaeljjao5namr7etevmrpgjjfk/harorgcsszyjr1grc046iqtwxgnaxjaf7sis6yfflyyxwc17jmwqvgj+hapqrfrg/nv5ctqsxppwo1fy9obspkzprdlm+yxqiqkdkag2lmyladdppfzj8yz0e7fsvljjqft3iwruumotj0r0uo1xjuy/9xarvdvumlhdeq09mvufyj3sb+4xcvtzsqvcjbo3yjokklbtu1vyysraaoyyve3gx85hfrrlkm/5ztuqvomcugmnam8te1qqaqqqe6ougjvabx9oy8o+/oh/m5a11z5jmnaeho1y/cwj+b</latext> <latext sha1_base64="l7u7aa1n2mxz5g7ot1dxzmoeikg=">aaab/ncbzdlsgmxfiyzxmu9jyorn8e1ivlphtdftx0wcfeob2htjppq5nmsdjcgqz8ftcufhhrc7jzbuzbwwjrd4gp/5zdofkjyag2nvftrk1vbg5tl3bku3v7b4fu0xfhj6ncpi0tlqhehdrhvjc2oyarnlqe8yrbjs5ndw7j0rpmoh7m5uk4ggkaewxmtyk3dob5fem85a+zfutungf65d56fa8mjcxxaw/gaoo1ardr8ewwsknwmcgto77njrbhpshmjg8peg1kqhp0ij0lqreq6y+fk5vldoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykysntaxsq/cxv7wknxrnt3znvrrnio4soapnoap8ca0aoalaoa0wymazeavvzppz4rw7h4vwnaeyoqf/5hz+ao/blmw=</latext> <latext sha1_base64="ke/ptvz4zdrkas+upaaqfp1c+2q=">aaacfhcbvdlssnafj34rpuvfdy4crahlxjn7osuomygn1ag8nketmonuzczeqomb/hb7jvt3anbt37bf6gkzyl23rgwugce7mh48emsmxb38ba+sbm1nzpp7y7t39wab4dd2sucajterfi9hwsgveobuuvg14saic+g64/uc397missn+r6yxuceecrpqgpwwppn0egi19on0nhn0ia0qj17vlzpprng1ewzrltgfqaaclc/8gqwjkotafwfyyr5jx8pnsvcummjkg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqrnaziwfjkpxtvaluccyjkvdtpocg+rpfovozrf2zvgs+ohm7qoaob12jbmqfmojgp7qc3pfb8az8w58gj/z1twjudlbczc+fgefdp6p</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="sgwewobqlwop/xciuubipjmyn+4=">aaacf3cbzdlsgmxfiyz9vbrrepk3aslubewgte6lljpsok9qkcomfs0du1mqpiryjd4hd6aw30ed+lwpu/ga5hefrb1h8dhf87hnpyh5ewb1/12cmvrg5tb+e3czu7e/khx8kp40rrancyx6odeg2crdawzhboswvehbxa4eh2um89gtisju7nwejxkehe+owsy62geojlewi/kytliqnyq1o2abv0lrkgwhir7lr4fbw5lnbc9ad44/dmgidove647nstnntkmcsgkfqjbejoa+hyjiga3u2nx8jwuxv6ub8r+ykdp+7fzqircctj2cmkferk3m/2qdxprvumlzgigornf/yrje+njhrjhffddxxyivczeumqkekntw1hsyihcmcufwwy3nioq9c8qnw79xsttbpki9o0rkqiw9doyqqotpqiiqe0at6rw/os/pufdfs9acm585rgtyvn4b2ywfmg==</latext> <latext sha1_base64="vbafdyn+0zk7w+2mnb7mw66wfo=">aaab/ncbzdlsgmxfiyz9vbrbvrcuqkwos4sm250wxdtzqv7gxycmmmmdu0yickizrjwvdy4umstz+hotzftz6gtpwq+/nmo5+spjkpaen63u1pb39jckm9xdnb39g/cw6ootlkfsrsnlfg9cgncqcbtqw0jpaki4hej3whyo6t3h4nsnbh3zpjwnfi0jhzkwvucdyanm5f9ygomppas+hd56fa9ujcxxaw/gcoo1ardr8ewwsknwmcgto77njrbhpshmjg8mkg1kqhp0ij0lqreq6y+fk5plfoemajsk8yohd/t2siaz3lke3kyiz1cm1m/lfrpya+ctiqzgqiwitfccqgseasczkmddphyqvttecveykysntaxq/cxv7wknau6b/noqzaarrxlcaroqa344bo0qbo0qbtgkifn8arencfnxxl3phatjaeyoqz/5hz+ao5vlms=</latext> <latext sha1_base64="fihe3xjkd6ju2rqnmqg1cqc+qw=">aaacfhcbvdlssnafj3uv62vqodgtbaidwfj3oy4kblcvybbqyt6u07ddijmxohxpygh+bwp8gduhxvf/gbttosboubc4dz7uuejh8zkpvtfxultfwnza3ydmvnd2//wdw86sgoeqtajgkr6plyaqmc2ooqbr1yaa59bl1/cpv73ucqkkb8xk1jcem84jsgbcsteebjimrq7afpoppoq1pthr10ljlprnp1ewzrltgfqaiclc/8gqwjkotafwfyyr5jx8pnsvcummgqg0rcjmkej6cvkcchsded5c+sc60mrsaseryzurfxshuk5dx2/maewyl4v/ef1ebtdusnmckobk/himkuky/dglibrlgpjpgiqrnaziwfjkpxtvaluccyjjvddpocg+rphnvdzs/s6unztfrgz2m1rddrpgddreldrgbd2hf/sk3oxn4934md7nqywjudlgczc+fgedc56o</latext> <latext sha1_base64="2ydszne4xjmbmh06scqwyesuh50=">aaacghcbvhlsgmxfm2m7/qqulbweyyc6ktra4el3ypypvos1djnonhmyyq5jryjab/9iv0m8wbweh1ssnhm4998fjkaqujed9oo7c/ml0vjkzxvtfwozurx9qjnmmewrctqoaaabjfqmdwiee4v0dgq8bqmr8f1p1dqmfywyxs6md0ihnegtww8qvvvra2zuzlet48xwerx5/c1xkphvab7rsfrgyiwtebqwucvynxoqi959woabkqtxv1sm5v2+sotjl4warp+efx615x94k8cwgjahmm796mcvtfgwgzrmuk27xetnp6fkccbajs40pjqn6qc6fkoag+7nk4skfgzeeejsk8apgf/duq01noub1yzu/o/9bg5h+1bmavj/nms0msdzdfgucmwspjcchv8cmgflamel2vsxeqklm2o/5tsvlx54n4p1hvz1yry8no6jxxde7bjdersm9tebokiendelaqnb1eemftkbzq6z57rukxvkqnudcqehfqr3itvgufaxg==</latext> <latext sha1_base64="mnwra8armv44xzwgxnstrea/fm=">aaacfxcbzc7tsmwfiadcvlfmbgylgokmpsjswwvmlpwcr6kdoqoe5ja9vjlntbqqi8bw/aco/ahlzeqjea7fnqft+ydkn/5yjc/whgjolhefbkm1sbm3vlhcre/shh0f28ulhjamk0kyjt2qviao46gtmebqexjifhdobpo7wb37bfkxjh7quwfereyxcxkl2l+ftyquyahqsbcjinphroa9tlv7ttvp+7mhdfblackcrv8+2cwtggaqawpj0r1xudolynsm8ohrwxsbylqcrlb32bmilbenv9aj+nm8rhis2lnz67fycyek1jqltgre9vqu1mflfrz/q8nblwcxsdtfdlaptjnwcz2ngiznanz8aifqycyumyyij1sazps2pgemasv4xybroaxd57rugr53qo1mkveznamlvemuuken1eqt1eyu5egfvai369l6tz6sz0vryspmttgsrk9f6wefka==</latext> <latext sha1_base64="adkmcmxvsu2nnlos2dzqtmth55m=">aaab+ncbzc7tsmwfiadcvllsliylehlavkwgcsxnkxspqtsfyxke1ajuw7ycqkedhyqahvp6ejbfbbtnayy9z+vsfc3so/0gyqo3nftuljc2t7z3ybmvv/+dwyk0ed3wskkw6oggj6kdie0yf6rhqgollrrcpgolf05t5vfdalkajudmzsqkoxolgfcnjrdctdwpmpmh9d6rm5be5kfb8xreqnad/ajqofa7dl+gowsnnadgdj64hvsbblshmjg8sow1uqpevjmraoecc6yban5/dcomyj8o+yedc/t2ria71jee2kymz0au1uflfbzca+driqjcpiqivf8upgyab8xzgcqcdztzqfhreyvee6qqnjatg3bx/3yonqvg77lw6/wbbvxlmepoan14imr0aqt0aydgmejeaav4m15cl6cd+dj2vpypkt8efo5w9qajqf</latext> <latext sha1_base64="4stvxtr8vdpbr6fgmjp1ow2tlui=">aaacencbvdlssnafj34rpuvhzs3g0wom5k40wxbtzcv7apagbtm3bozbjmjkin+qs/wk1+gjtx6w/4bf6gkzyl23rgwugce7mh48ecke0439ba+sbm1nzpp7y7t39wab8dt1wusaotgvfidn2gdmblc00h24sgyq+h44/uc39znixsjxr6cxdeiyexglggjefzppyr67afpoppyq1rvhrvmplv1jwz8cpxc1jbbzqe/dmfrjqjqwjkvi914n1icvsm8ohk/ctbtghezkcnqgchkag6sx9h+mmsrbjm0ijwfq34uuhepnq99s5lnvspel/3m9rac3g5sjoneg6pxrkhcsi5xxgydmatv8aghkpmsmi6jjfsbwha+jpfiakyysmngxe5hlbsvaq7hd06l3g6kqezdi6qyexxqi4aqilakin9ije0zv1bl1bh9bnfhxnkm5o0aksr18taz42</latext> (a) Attrbuted Sequence Attenton (ASA). (b) Attrbuted Sequence Hybrd Attenton (ASHA). Fgure 3: Two types of attenton for attrbuted sequence classfcaton n ths paper, where µ = µ (1),, µ (t) s the attenton weghts. the learnng process. Here, we desgn the Attenton Block as follows: Frst, we need to compute the attenton weght µ (t) at t-th tme as: g x (t) = h (t) exp g x (t) µ (t) = t j=1 g x (j) Then, the attenton weght s multpled wth the hdden state at each tme step: (3.5) α (t) = µ (t) g x (t), t = 1, 2,, t The attenton weght µ (t) at t tme s randomly ntalzed and ncrementally adjusted durng the tranng process. The output at each tme step s then augmented wth the outputs from Attrbute Net as: p (t) = f(v ) α (t) At the last tme step t, we denote p = p (t) to smply the notaton. Attrbuted Sequence Hybrd Attenton (ASHA). Dfferent from the prevous ASA approach, the outputs of Attrbute Net and Sequence Net are augmented wth the Attenton Block. The attenton weght s wrtten as: d v, x (t) µ (t) = = f(v ) g d exp t j=1 d v, x (t) x (t) v, x (j) Then, the vectors used for classfcaton s: (3.6) α (t) = µ (t) d v, x (t), t = 1, 2,, t 3.2 Attrbuted Sequence Classfcaton. In the soluton of attrbuted sequence classfcaton wthout attenton, the Attrbute Net and the Sequence Net are frst concatenated as: (3.7) p = d v, x (t) = r h (t) Here, denotes the concatenaton and t denotes the last tem n s. Although all attrbuted sequences n the dataset are zero-padded to the maxmum length t max, the padded zero values are masked and not used n the computaton. We model the process of predctng the for each attrbuted sequence as: σ(w pp + b p), ASA or No Attenton Θ(p ) = σ W pα (t ) + b p, ASHA where σ s a sgmod actvaton functon and ˆl = Θ(p ) s the predcted. The W p and b p are both tranable n our model. 3.3 Tranng Regularzaton. We adopt multple strateges for dfferent components n our AMAS network. We emprcally select the followng regularzaton strateges n our model based on: (1). For Sequence Net, we apply l 2 -regularzaton to the recurrent unt. (2). 112

5 Table 2: Compared Methods Name Data Used Attenton Note BLA Attrbutes No [12] BLS Sequences No [28] BLAS Attrbutes Sequences No [35] SOA Sequences Yes [34] ASA Attrbutes Sequences Yes Ths paper ASHA Attrbutes Sequences Yes Ths paper Dropout wth a rate of 0.5 s used to regularze the fully connected layer n Attrbute Net. (3). Lastly, we use Dropout wth a rate of 0.2 n other fully connected layers n the model. Based on our observatons, usng regularzaton on Attenton Block has no sgnfcant mpact on the performance of AMAS Optmzer. We use an optmzer that computes the adaptve learnng rates for every parameters, referred to as Adaptve Moment Estmaton (Adam) [15]. The core dea s to keep (1). an exponentally decayng average of gradents n the past and (2). a squared past gradents. Adam counteracts the bases as: ω (t) = β 1ω (t 1) + (1 β 1 )m (t) 1 β t 1 ν (t) = β 2ν (t 1) + (1 β 2 ) m (t) 2 1 β t 2 where β 1 and β 2 are the decay rates, and m (t) s the gradent. We adopt β 1 = 0.9 and β 2 = as n [15]. Fnally, the Adam updates the parameters as: γ (t+1) = γ (t) ρ ν (t) + ω (t) where ρ s a statc learnng rate and s a constant wth a small value to avod dvson errors, such as dvson by zero. We emprcally select ρ = Experments 4.1 Datasets. Our soluton has been motvated by use case scenaros observed at Amadeus corporaton. For ths reason, we work wth the log fles of an Amadeus [2] nternal applcaton. The log fles contan user sessons n the form of attrbuted sequences. In addton, we apply our methodology to real-world, publcly avalable Wkspeeda data [29] and Reddt data [16]. For each type of data, we sample two subsets and conduct experments ndependently. We summarze the data descrptons as follows: Amadeus data (AMS-1, AMS-2) 1. We sampled sx datasets from the log fles of an nternal applcaton at Amadeus IT Group. Each attrbuted sequence s composed of a user profle contanng nformaton (e.g., system confguraton, offce name) and a sequence of functon names nvoked by web clck actvtes (e.g., logn, search) ordered by tme. Wkspeeda data (Wk-1, Wk-2). Wkspeeda s an onlne game requrng partcpants to clck through from a gven start page to an end page usng fewest clcks [29]. We select fnshed paths and extract several propertes of each path (e.g.,, the category of the start path, tme spent per clck). We also sample sx datasets from Wkspeeda. The Wkspeeda data s avalable through the Stanford Network Analyss Project 2 [17]. Reddt data (Reddt-1, Reddt-2). Reddt s an onlne forum. Two datasets that contan the content of reddt submssons are used. The Reddt data s avalable through the Stanford Network Analyss Project 3. We use 60% of the nstances n each dataset for the tranng and the rest 40% for testng. In the tranng, we holdout 20% of the tranng nstances for valdaton. 4.2 Compared Methods. We evaluate our two approaches, namely ASA and ASHA and compare them wth the followng baselne methods. We summarze all compared methods used n ths research n Table 2. BLA s bult usng a fully connected neural network to reduce the dmensonalty of the nput data, and then classfy each nstance. BLS classfes sequences only data usng an LSTM. BLAS utlzes the nformaton from both attrbutes and sequences. The resultng embeddngs generated by BLAS are then used for classfcaton. SOA bulds attenton on the sequence data for classfcaton, whle the attrbute data s not used. 4.3 Expermental Settng. Our paper focuses on mult-class classfcaton problem. We thus use accuracy as the metrc to evaluate the performance. A hgher accuracy score depcts more correct predctons of class s. For each method, we holdout 20% as the valdaton dataset randomly selected from tranng dataset. 1 Personal nformaton s not collected

6 (a) ASM-1, 83 classes (b) Wk-1, 64 classes (c) Reddt-1, 140 classes (d) ASM-2, 83 classes (e) Wk-2, 64 classes (f) Reddt-2, 140 classes Fgure 4: Performance comparson on all sx datasets. For each expermental settng, we report the top-1 top-10 accuracy for each method. We ntalze our network usng the followng strateges: orthogonal matrces are used to ntalze the recurrent weghts, normalzed random dstrbuton [9] s used to ntalze weght matrces n Attrbute Net, and bas vectors are ntalzed as zero vector Accuracy Results. In Fgure 4, we compare the performance of our ASA and ASHA solutons wth the other state-of-the-art methods n Table 2. ASHA acheves the best performance of top-1 accuracy on most datasets. In most cases, ASHA outperforms other solutons sgnfcantly. We also observe a sgnfcant performance mprovement by ASA compared to other methods. That s, although the top-1 accuracy performance of ASA s beneath that of ASHA, t stll outperforms SOA wth sequence-only attenton and all other methods wthout attenton. The two closest compettors, the SOA utlzng the attenton mechansm and classfyng each nstance based on only the sequental data, and BLAS usng nformaton from both attrbutes and sequences, but wthout the help from an attenton mechansm, are outperformed by the our proposed models. 4.5 Parameter Senstvty Analyss Adaptve Samplng Accuracy. As ponted out n recent work [4], adaptve samplng s capable of mprovng the effcency of the optmzaton processes by adaptng the tranng sample sze n each teraton (.e., epoch). In ths set of experments, we evaluate the two models wth varyng adaptve samplng rates. We adopt the adaptve samplng functon as: N τ = N 1 λ (τ 1) Here, τ denotes the epoch number, N τ denotes the number of nstances used n the τ-th epoch and λ the rate of adaptve samplng. We choose λ = 1, 1.001, 1.005, and 1.01 n our experments, where λ = 1 means no adaptve samplng. The results presented n Fgure 5 shows that the adaptve samplng wth the above samplng rates can acheve smlar performance as the non-adaptve approach yet now wth much less tranng data Tranng wth Adaptve Samplng. Wth the contnuously ncreasng amount of tranng nstances, we expect the hstory of tranng loss to be jttery when a model encounters prevously unseen new nstances. Dfferent from prevous experments, where we use Early Stoppng strategy to avod overfttng, we now set a fxed number of 144 epochs for the ASHA model and 97 epochs for ASA model and collect the hstory of tranng and valdaton to study the adaptve tranng strategy. In Fgure 6a, we observe the tranng wth adaptve samplng s more aggressve compared to the non-adaptve approach. From Fgure 6b we conclude that wth a hgher adaptve rate, the model more easly becomes overftted. Smlar concluson can also be made from Fgure 6d. Selectng a hgher adaptve samplng rate can shorten the tranng tme but rskng a hgher chance of overfttng. 4.6 Case Studes. Fgure 7 demonstrates the weghts of each word of ten nstances from the Reddt-2 dataset. Hgher weghts are represented wth a darker color, whle lower weghts are represented wth a lghter 114

7 (a) ASA model on AMS-1 dataset. (b) ASA model on Wk-1 dataset. (c) ASA model on Reddt-1 dataset. (d) ASHA model on AMS-1 dataset. (e) ASHA model on Wk-1 dataset. (f) ASHA model on Reddt-1 dataset. Fgure 5: The performance comparson between non-adaptve tranng and adaptve samplng. (a) tran_loss of ASHA. (b) val_loss of ASHA. (c) tran_loss of ASA. (d) val_loss of ASA. Fgure 6: Comparson of the hstory of tranng and valdaton losses. color. Comparng the three cases, we fnd that the SOA has the most polarzed weghts among the three cases. Ths may be caused by the fact that the attenton produced by SOA s solely based on the sequences, whle ASHA and ASA have been nfluenced by attrbute data. 5 Related Work 5.1 Deep Learnng Deep learnng has receved sgnfcant nterests n recent years. Varous deep learnng models and optmzaton technques have been proposed n a wde range of applcatons such as mage recognton [14, 32] and sequence learnng [5, 24, 33, 22]. Many of these applcatons nvolve the learnng of a sngle data type [5, 24, 33, 22], whle some applcatons nvolve more than one data type [14, 32]. The applcaton of deep learnng n sequence learnng s popular wth one of the most popular works, sequence-to-sequence [24], usng a long short-term memory model n machne translaton. The hdden representatons of sentences n the source language are transferred to a decoder to reconstruct n the target language. The dea s that the hdden representaton can be used as a compact representaton to transfer smlartes between two sequences. Mult-task learnng [18] examnes three mult-task learnng settngs for sequence-to-sequence models that am to share ether an encoder or a decoder n an encoder-decoder model settng. Although the above work s capable of learnng the dependences wthn a sequence, none of them focuses on learnng the dependences between attrbutes and sequences. Multmodal deep neural networks [14, 23, 32] are desgned for nformaton sharng across multple neural networks, but none of these works focuses on the attrbuted sequence classfcaton problem we target here. 5.2 Attenton Network. Attenton network [20] has ganed a lot of research nterest recently. The attenton network has been appled n varous tasks, ncludng mage captonng [32, 21], mage generaton [11], speech recognton [6] and document classfcaton [34]. The goal of usng an attenton network n these tasks s to make the neural network focus on the nterestng parts of each nput, such as, a small regon of an mage or words that are most helpful to classfyng documents. 115

8 (a) ASHA (b) ASA (c) SOA Fgure 7: Weghts of each words from 10 nstances n Reddt-2. Hgher weghts are darker. There are varatons of attenton networks, ncludng herarchcal attenton [34] and dual attenton [21]. 5.3 Sequence Mnng. Recent work n sequence mnng area ams to fnd the most frequent subsequence patterns [19, 8]. Several recent works [3, 19] focus on fndng the most frequent subsequence that meets certan constrants. That s, they fnd the set of sequental patterns satsfyng varous lngustc constrants (e.g., syntactc, symbolc). Many sequence mnng works focus on frequent sequence pattern mnng. Recent work n [19] targets fndng subsequences of possble non-consecutve actons constraned by a gap wthn sequences. [7] ams at solvng pattern-based sequence classfcaton problems usng a parameter-free algorthm from the model space. [8] bulds a subsequence nterleavng model for mnng the most relevant sequental patterns. However, none of the above works supports attrbute data alongsde the sequences, nor do they classfy attrbuted sequences. 6 Concluson In ths paper, we propose a AMAS framework wth two models for classfyng attrbuted sequences. Our ASHA and ASA models progressvely ntegrate the nformaton from both attrbutes and sequences whle weghtng each tem n the sequence to mprove the classfcaton accuracy. Expermental results demonstrate that our models sgnfcantly outperform state-of-the-art methods. 7 Acknowledgement We would lke to thank the Amadeus IT Group for the generous supports n fundng, computng resources and data access. References [1] Z. Akata, F. Perronnn, Z. Harchaou, and C. Schmd, Label-embeddng for attrbute-based classfcaton, n Proceedngs of the IEEE Conference on Computer Vson and Pattern Recognton (CVPR), 2013, pp [2] Amadeus, Amadeus IT Group. com. Accessed: