論文サーベイ
@article{zhuang2023optimizing, title={On optimizing the communication of model parallelism}, author={Zhuang, Yonghao and Zheng, Lianmin and Li, Zhuohan and Xing, Eric and Ho, Qirong and Gonzalez, Joseph and Stoica, Ion and Zhang, Hao and Z…
Cai, Zixian, et al. "Synthesizing optimal collective algorithms." Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2021. どんなもの？ 詳細 論文メタ情報 AI向けハードウェア DGX-1 Gigabyte Z52 …
滝澤真一朗, 遠藤敏夫, and 松岡聡. "次世代光インターコネクトでの MPI 通信に関する研究." コンピュータ ソフトウェア 26.3 (2009): 3_5-3_19. 概要 背景 どんなもの? 技術や手法のキモはどこ? どうやって有効だと検証した? OCS経路選択アルゴリズム Switc…
Sato, Ken-ichi. "How optical technologies can innovate intra data center networks." 2021 International Conference on Computer Communications and Networks (ICCCN). IEEE, 2021. paper: https://ieeexplore.ieee.org/document/9522206 光スイッチを…
Wang, Weiyang, et al. "Topoopt: Co-optimizing network topology and parallelization strategy for distributed training jobs." arXiv preprint arXiv:2202.00433 (2022). [paper] 概要 どんなもの? Metaにおける分散DNNトレーニングジョブの解析 それを…
Liu, Gengchen, et al. "Architecture and performance studies of 3D-Hyper-FleX-LION for reconfigurable all-to-all HPC networks." SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020. bib…
Yang, Hao, et al. "Which can Accelerate Distributed Machine Learning Faster: Hybrid Optical/Electrical or Optical Reconfigurable DCN?." 2022 Optical Fiber Communications Conference and Exhibition (OFC). IEEE, 2022. @inproceedings{yang2022c…
@article{zhang2015improved, title={An improved generalized conjugate residual squared (IGCRS2) algorithm suitable for distributed parallel computing}, author={Zhang, Li-Tao and Dong, Xiao-Na and Gu, Tong-Xiang and Zuo, Xian-Yu and Liu, Xin…
@article{zheng2022alpa, title={Alpa: Automating Inter-and Intra-Operator Parallelism for Distributed Deep Learning}, author={Zheng, Lianmin and Li, Zhuohan and Zhang, Hao and Zhuang, Yonghao and Chen, Zhifeng and Huang, Yanping and Wang, Y…
Tanaka, Masahiro, et al. "Automatic graph partitioning for very large-scale deep learning." 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2021. @inproceedings{tanaka2021automatic, title={Automatic gra…
Wang, Minjie, Chien-chin Huang, and Jinyang Li. "Supporting very large models using automatic dataflow graph partitioning." Proceedings of the Fourteenth EuroSys Conference 2019. 2019. @inproceedings{wang2019supporting, title={Supporting v…
@article{shoeybi2019megatron, title={Megatron-lm: Training multi-billion parameter language models using model parallelism}, author={Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanza…
@inproceedings{ren2021zero, title={$\{$ZeRO-Offload$\}$: Democratizing $\{$Billion-Scale$\}$ Model Training}, author={Ren, Jie and Rajbhandari, Samyam and Aminabadi, Reza Yazdani and Ruwase, Olatunji and Yang, Shuangyan and Zhang, Minjia a…
@inproceedings{rajbhandari2020zero, title={Zero: Memory optimizations toward training trillion parameter models}, author={Rajbhandari, Samyam and Rasley, Jeff and Ruwase, Olatunji and He, Yuxiong}, booktitle={SC20: International Conference…
Chen, Tianqi, et al. "Training deep nets with sublinear memory cost." arXiv preprint arXiv:1604.06174 (2016). @article{chen2016training, title={Training deep nets with sublinear memory cost}, author={Chen, Tianqi and Xu, Bing and Zhang, Ch…
@article{伊藤祐貴2018gpu, title={GPU メモリ管理の実行時最適化による大規模深層学習の高速化}, author={伊藤祐貴 and 今井晴基 and 根岸康 and 河内谷清久仁 and 松宮遼 and 遠藤敏夫 and others}, journal={研究報告ハイパフォーマンスコンピューティン…
https://dl.acm.org/doi/10.1145/3458817.3476209 paper: @inproceedings{10.1145/3458817.3476209, author = {Narayanan, Deepak and Shoeybi, Mohammad and Casper, Jared and LeGresley, Patrick and Patwary, Mostofa and Korthikanti, Vijay and Vainbr…
@article{shazeer2018mesh, title={Mesh-tensorflow: Deep learning for supercomputers}, author={Shazeer, Noam and Cheng, Youlong and Parmar, Niki and Tran, Dustin and Vaswani, Ashish and Koanantakool, Penporn and Hawkins, Peter and Lee, Hyouk…
https://dl.acm.org/doi/abs/10.1145/3341301.3359646?casa_token=L-sKQKrRoE4AAAAA%3AYKo9NPdnPyG6IouMN5jfTHTCYFAGORDxen32GKAteeSG-ROhqx_OX-hVOfuyHiVBXLLJH0RPujhFPEk @inproceedings{narayanan2019pipedream, title={PipeDream: generalized pipeline …
@article{huang2019gpipe, title={Gpipe: Efficient training of giant neural networks using pipeline parallelism}, author={Huang, Yanping and Cheng, Youlong and Bapna, Ankur and Firat, Orhan and Chen, Dehao and Chen, Mia and Lee, HyoukJoong a…
深層学習において、学習データと学習モデルの巨大化が最新のトレンドになっています。 そこで学習時間の削減のために複数のマシンを用いてモデルを訓練する試みが行われており、 分散深層学習(distributed deep learning)などという呼ばれ方で一つの分野にな…
Lindauer, Marius, Holger Hoos, and Frank Hutter. "From sequential algorithm selection to parallel portfolio selection." International Conference on Learning and Intelligent Optimization. Springer, Cham, 2015. ポートフォリオ最適化とは(1) 資…
Nils Boysen and Konrad Stephan. “A survey on single crane scheduling in automated stor- age/retrieval systems”. In: European Journal of Operational Research 254.3 (2016), pp. 691– 704. issn: 0377-2217. doi: https://doi.org/10.1016/j.ejor.2…
Jeroen P. van den Berg and A.J.R.M. Gademann. “Simulation study of an automated storage/retrieval system”. In: International Journal of Production Research 38.6 (2000), pp. 1339–1356. doi: 10.1080/002075400188889. eprint: https://doi.org/1…
2020年の論文, "MaxCut algorithm selection"の検索でヒット; Moussa, Charles, Henri Calandra, and Vedran Dunjko. "To quantum or not to quantum: towards algorithm selection in near-term quantum optimization." Quantum Science and Technology 5.4…
