PhoneCanvas+: 深度カメラ付きのスマートフォンをキャンバスとして用いた3Dスケッチシステム

Translated Abstract

3D modeling software often presents challenges for beginners because of steep learning curves. To address these issues, we developed PhoneCanvas+, a 3D sketching system designed with low installation and learning costs. The system allows users to create lines and surfaces, as well as delete 3D sketches, through mid-air hand gestures above the smartphone. These gestures mimic everyday hand movements, such as holding a pen to draw lines or moving a hand sideways to create surfaces. We conducted a user study with 12 3D modeling beginners to evaluate PhoneCanvas+ in terms of sketch objects, time, and usability. Results showed that participants completed sketching tasks in about 1 minute with about 10 minutes of practice in a simplified setup.

References

• [1]  Blender Foundation: Blender, https://www.blender.org/ (2024). 最終閲覧日: 2024年12月5日.
• [2]  Autodesk, Inc.: Maya, https://www.autodesk.com/products/maya/overview (2024). 最終閲覧日: 2024年12月5日.
• [3]  Maxon Computer GmbH: ZBrush, https://www.maxon.net/ja/zbrush (2024). 最終閲覧日: 2024年12月5日.
• [4]   Kazmi , I. K.,  You,  L. and  Zhang , J. J.: A Survey of Sketch Based Modeling Systems, Proceedings of the 2014 11th International Conference on Computer Graphics, Imaging and Visualization, CGIV ’14, USA, IEEE Computer Society, (online), 10.1109/CGiV.2014.27 (2014).
• [5]  Google LLC: Tilt Brush, https://support.google.com/tiltbrush (2024). 最終閲覧日: 2024年12月5日.
• [6]  Gravity Sketch: Gravity Sketch, https://www.gravitysketch.com/ (2024). 最終閲覧日: 2024年12月5日.
• [7]   Kwan , K. C. and  Fu,  H.: Mobi3DSketch: 3D Sketching in Mobile AR, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, New York, NY, USA, Association for Computing Machinery, pp. 1–11 (online), 10.1145/3290605.3300406 (2019).
• [8]  Google LLC: Just a Line, https://experiments.withgoogle.com/justaline (2024). 最終閲覧日: 2024年12月5日.
• [9]  武山侑輝，西川宜利，志築文太郎：PhonePedestal: 深度カメラ付きのスマートフォンを台座とした3Dモデリング，研究報告ヒューマンコンピュータインタラクション(HCI)，No. 38, pp. 1–8 (2024).
• [10]   Takeyama,  Y.,  Nagayama,  K.,  Choi,  M. and  Shizuki,  B.: PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a Canvas, Proceedings of the 2024 ACM Symposium on Spatial User Interaction, SUI ’24, New York, NY, USA, Association for Computing Machinery, pp. 1–12 (online), 10.1145/3677386.3682078 (2024).
• [11]   Jiang,  Y.,  Zhang,  C.,  Fu,  H.,  Cannavò,  A.,  Lamberti,  F.,  Lau , H. Y. K. and  Wang,  W.: Hand-Painter - 3D Sketching in VR with Hand-Based Physical Proxy, Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA, Association for Computing Machinery, pp. 1–13 (online), 10.1145/3411764.3445302 (2021).
• [12]   Drey,  T.,  Gugenheimer,  J.,  Karlbauer,  J.,  Milo,  M. and  Rukzio,  E.: VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality, Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, New York, NY, USA, Association for Computing Machinery, pp. 1–14 (online), 10.1145/3313831.3376628 (2020).
• [13]   Machuca , M. D. B.,  Asente,  P.,  Stuerzlinger,  W.,  Lu,  J. and  Kim,  B.: Multiplanes: Assisted Freehand VR Sketching, Proceedings of the 2018 ACM Symposium on Spatial User Interaction, SUI ’18, New York, NY, USA, Association for Computing Machinery, pp. 36–47 (online), 10.1145/3267782.3267786 (2018).
• [14]   Machuca , M. D. B.,  Stuerzlinger,  W. and  Asente,  P.: Smart3DGuides: Making Unconstrained Immersive 3D Drawing More Accurate, Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology, VRST ’19, New York, NY, USA, Association for Computing Machinery, pp. 1–13 (online), 10.1145/3359996.3364254 (2019).
• [15]   Yu,  X.,  DiVerdi,  S.,  Sharma,  A. and  Gingold,  Y.: ScaffoldSketch: Accurate Industrial Design Drawing in VR, The 34th Annual ACM Symposium on User Interface Software and Technology, UIST ’21, New York, NY, USA, Association for Computing Machinery, pp. 372–384 (online), 10.1145/3472749.3474756 (2021).
• [16]   Qian,  J.,  Zhou,  T.,  Young-Ng,  M.,  Ma,  J.,  Cheung,  A.,  Li,  X.,  Gonsher,  I. and  Huang,  J.: Portalware: Exploring Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable Paradigm, Proceedings of the 2021 ACM Designing Interactive Systems Conference, DIS ’21, New York, NY, USA, Association for Computing Machinery, pp. 205–219 (online), 10.1145/3461778.3462098 (2021).
• [17]   Chen , X. A.,  Grossman,  T.,  Wigdor , D. J. and  Fitzmaurice,  G.: Duet: Exploring Joint Interactions on a Smart Phone and a Smart Watch, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, New York, NY, USA, Association for Computing Machinery, pp. 159–168 (online), 10.1145/2556288.2556955 (2014).
• [18]   Wells,  T.,  Potts,  D. and  Houben,  S.: A Study into the Effect of Mobile Device Configurations on Co-Located Collaboration using AR, Proceedings of the ACM on Human-Computer Interaction, Vol. 6, No. MHCI, pp. 1–23 (online), 10.1145/3546735 (2022).
• [19]   Bae,  S.-H.,  Balakrishnan,  R. and  Singh,  K.: ILoveSketch: As-Natural-As-Possible Sketching System for Creating 3D Curve Models, Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST ’08, New York, NY, USA, Association for Computing Machinery, pp. 151–160 (online), 10.1145/1449715.1449740 (2008).
• [20]   Arora,  R.,  Habib Kazi,  R.,  Grossman,  T.,  Fitzmaurice,  G. and  Singh,  K.: SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, New York, NY, USA, Association for Computing Machinery, pp. 1–15 (online), 10.1145/3173574.3173759 (2018).
• [21]   Kari,  M. and  Holz,  C.: HandyCast: Phone-based Bimanual Input for Virtual Reality in Mobile and Space-Constrained Settings via Pose-and-Touch Transfer, Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA, Association for Computing Machinery, pp. 1–15 (online), 10.1145/3544548.3580677 (2023).
• [22]   Chulpongsatorn,  N.,  Willett,  W. and  Suzuki,  R.: HoloTouch: Interacting with Mixed Reality Visualizations Through Smartphone Proxies, Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA ’23, New York, NY, USA, Association for Computing Machinery, pp. 1–8 (online), 10.1145/3544549.3585738 (2023).
• [23]   Pietroszek,  K.,  Wallace , J. R. and  Lank,  E.: Tiltcasting: 3D Interaction on Large Displays using a Mobile Device, Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, UIST ’15, New York, NY, USA, Association for Computing Machinery, pp. 57–62 (online), 10.1145/2807442.2807471 (2015).
• [24]   Büschel,  W.,  Mitschick,  A.,  Meyer,  T. and  Dachselt,  R.: Investigating Smartphone-based Pan and Zoom in 3D Data Spaces in Augmented Reality, Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI ’19, New York, NY, USA, Association for Computing Machinery, pp. 1–13 (online), 10.1145/3338286.3340113 (2019).
• [25]   Hursale,  S.,  Makwana,  H.,  Panchal,  A.,  Bhakkad,  C. and  Vairagar,  S.: Smartphone’s Based Gesture Recognition in Air, Proceedings of the 2015 International Conference on Advanced Research in Computer Science Engineering & Technology (ICARCSET 2015), ICARCSET ’15, New York, NY, USA, Association for Computing Machinery, pp. 1–5 (online), 10.1145/2743065.2743114 (2015).
• [26]   Vinayak , Ramanujan, D.,  Piya,  C. and  Ramani  K.: MobiSweep: Exploring Spatial Design Ideation Using a Smartphone as a Hand-held Reference Plane, Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, TEI ’16, New York, NY, USA, Association for Computing Machinery, pp. 12–20 (online), 10.1145/2839462.2839490 (2016).
• [27]   Chen , X. A.,  Schwarz,  J.,  Harrison,  C.,  Mankoff,  J. and  Hudson , S. E.: Air+Touch: Interweaving Touch & In-Air Gestures, Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14, New York, NY, USA, Association for Computing Machinery, pp. 519–525 (online), 10.1145/2642918.2647392 (2014).
• [28]  Šimoník, M.: Record3D, https://record3d.app/ (2024). 最終閲覧日: 2024年12月5日.
• [29]  Apple Inc.: CMMotionManager, https://developer.apple.com/documentation/coremotion/cmmotionmanager (2024). 最終閲覧日: 2024年12月5日.
• [30]  Google LLC: MediaPipe, https://ai.google.dev/edge/mediapipe/solutions/guide (2024). 最終閲覧日: 2024年12月5日.
• [31]  Apple Inc.: Apple Support, https://support.apple.com/en-us/102381 (2024). 最終閲覧日: 2024年12月5日.
• [32]   Brooke,  J.: SUS: A Quick and Dirty Usability Scale, Usability Evaluation in Industry, Vol. 189 (1995).
• [33]   Kim,  Y.,  An,  S.-G.,  Lee , J. H. and  Bae,  S.-H.: Agile 3D Sketching with Air Scaffolding, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, New York, NY, USA, Association for Computing Machinery, pp. 1–12 (online), 10.1145/3173574.3173812 (2018).
• [34]  Drascic, D. and Milgram, P.: Perceptual issues in augmented reality, Stereoscopic Displays and Virtual Reality Systems III (Bolas, M. T., Fisher, S. S., Bolas, M. T., Fisher, S. S. and Merritt, J. O., eds.), Vol. 2653, International Society for Optics and Photonics, SPIE, pp. 123–134 (online), 10.1117/12.237425 (1996).
• [35]   Diaz,  C.,  Walker,  M.,  Szafir ,  D. A. and  Szafir, D.: Designing for Depth Perceptions in Augmented Reality, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 111–122 (online), 10.1109/ISMAR.2017.28 (2017).
• [36]   Wanger,  L.,  Ferwerda,  J. and  Greenberg,  D.: Perceiving spatial relationships in computer-generated images, IEEE Computer Graphics and Applications, Vol. 12, No. 3, pp. 44–58 (online), 10.1109/38.135913 (1992).
• [37]   Hart , S. G. and  Staveland , L. E.: Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, Human Mental Workload (Hancock, P. A. and Meshkati, N., eds.), Advances in Psychology, Vol. 52, North-Holland, pp. 139–183 (online), https://doi.org/10.1016/S0166-4115(08)62386-9 (1988).
• [38]   Lewis ,  J. R. and  Sauro, J.: Item Benchmarks for the System Usability Scale, J. Usability Studies, Vol. 13, No. 3, pp. 158–167 (2018).