PhoneCanvas+: 深度カメラ付きのスマートフォンをキャンバスとして用いた3Dスケッチシステム

武山 侑輝; 崔 明根; 志築 文太郎

doi:10.11184/his.27.2_47

Translated Abstract

3D modeling software often presents challenges for beginners because of steep learning curves. To address these issues, we developed PhoneCanvas+, a 3D sketching system designed with low installation and learning costs. The system allows users to create lines and surfaces, as well as delete 3D sketches, through mid-air hand gestures above the smartphone. These gestures mimic everyday hand movements, such as holding a pen to draw lines or moving a hand sideways to create surfaces. We conducted a user study with 12 3D modeling beginners to evaluate PhoneCanvas+ in terms of sketch objects, time, and usability. Results showed that participants completed sketching tasks in about 1 minute with about 10 minutes of practice in a simplified setup.

References

[1] Blender Foundation: Blender, https://www.blender.org/ (2024). 最終閲覧日: 2024年12月5日.
[2] Autodesk, Inc.: Maya, https://www.autodesk.com/products/maya/overview (2024). 最終閲覧日: 2024年12月5日.
[3] Maxon Computer GmbH: ZBrush, https://www.maxon.net/ja/zbrush (2024). 最終閲覧日: 2024年12月5日.
[4] Kazmi , I. K., You, L. and Zhang , J. J.: A Survey of Sketch Based Modeling Systems, Proceedings of the 2014 11th International Conference on Computer Graphics, Imaging and Visualization, CGIV ’14, USA, IEEE Computer Society, (online), 10.1109/CGiV.2014.27 (2014).
[5] Google LLC: Tilt Brush, https://support.google.com/tiltbrush (2024). 最終閲覧日: 2024年12月5日.
[6] Gravity Sketch: Gravity Sketch, https://www.gravitysketch.com/ (2024). 最終閲覧日: 2024年12月5日.
[7] Kwan , K. C. and Fu, H.: Mobi3DSketch: 3D Sketching in Mobile AR, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, New York, NY, USA, Association for Computing Machinery, pp. 1–11 (online), 10.1145/3290605.3300406 (2019).
[8] Google LLC: Just a Line, https://experiments.withgoogle.com/justaline (2024). 最終閲覧日: 2024年12月5日.
[9] 武山侑輝，西川宜利，志築文太郎：PhonePedestal: 深度カメラ付きのスマートフォンを台座とした3Dモデリング，研究報告ヒューマンコンピュータインタラクション(HCI)，No. 38, pp. 1–8 (2024).
[10] Takeyama, Y., Nagayama, K., Choi, M. and Shizuki, B.: PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a Canvas, Proceedings of the 2024 ACM Symposium on Spatial User Interaction, SUI ’24, New York, NY, USA, Association for Computing Machinery, pp. 1–12 (online), 10.1145/3677386.3682078 (2024).
[11] Jiang, Y., Zhang, C., Fu, H., Cannavò, A., Lamberti, F., Lau , H. Y. K. and Wang, W.: Hand-Painter - 3D Sketching in VR with Hand-Based Physical Proxy, Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA, Association for Computing Machinery, pp. 1–13 (online), 10.1145/3411764.3445302 (2021).
[12] Drey, T., Gugenheimer, J., Karlbauer, J., Milo, M. and Rukzio, E.: VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality, Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, New York, NY, USA, Association for Computing Machinery, pp. 1–14 (online), 10.1145/3313831.3376628 (2020).
[13] Machuca , M. D. B., Asente, P., Stuerzlinger, W., Lu, J. and Kim, B.: Multiplanes: Assisted Freehand VR Sketching, Proceedings of the 2018 ACM Symposium on Spatial User Interaction, SUI ’18, New York, NY, USA, Association for Computing Machinery, pp. 36–47 (online), 10.1145/3267782.3267786 (2018).
[14] Machuca , M. D. B., Stuerzlinger, W. and Asente, P.: Smart3DGuides: Making Unconstrained Immersive 3D Drawing More Accurate, Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology, VRST ’19, New York, NY, USA, Association for Computing Machinery, pp. 1–13 (online), 10.1145/3359996.3364254 (2019).
[15] Yu, X., DiVerdi, S., Sharma, A. and Gingold, Y.: ScaffoldSketch: Accurate Industrial Design Drawing in VR, The 34th Annual ACM Symposium on User Interface Software and Technology, UIST ’21, New York, NY, USA, Association for Computing Machinery, pp. 372–384 (online), 10.1145/3472749.3474756 (2021).
[16] Qian, J., Zhou, T., Young-Ng, M., Ma, J., Cheung, A., Li, X., Gonsher, I. and Huang, J.: Portalware: Exploring Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable Paradigm, Proceedings of the 2021 ACM Designing Interactive Systems Conference, DIS ’21, New York, NY, USA, Association for Computing Machinery, pp. 205–219 (online), 10.1145/3461778.3462098 (2021).
[17] Chen , X. A., Grossman, T., Wigdor , D. J. and Fitzmaurice, G.: Duet: Exploring Joint Interactions on a Smart Phone and a Smart Watch, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’14, New York, NY, USA, Association for Computing Machinery, pp. 159–168 (online), 10.1145/2556288.2556955 (2014).
[18] Wells, T., Potts, D. and Houben, S.: A Study into the Effect of Mobile Device Configurations on Co-Located Collaboration using AR, Proceedings of the ACM on Human-Computer Interaction, Vol. 6, No. MHCI, pp. 1–23 (online), 10.1145/3546735 (2022).
[19] Bae, S.-H., Balakrishnan, R. and Singh, K.: ILoveSketch: As-Natural-As-Possible Sketching System for Creating 3D Curve Models, Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST ’08, New York, NY, USA, Association for Computing Machinery, pp. 151–160 (online), 10.1145/1449715.1449740 (2008).
[20] Arora, R., Habib Kazi, R., Grossman, T., Fitzmaurice, G. and Singh, K.: SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, New York, NY, USA, Association for Computing Machinery, pp. 1–15 (online), 10.1145/3173574.3173759 (2018).
[21] Kari, M. and Holz, C.: HandyCast: Phone-based Bimanual Input for Virtual Reality in Mobile and Space-Constrained Settings via Pose-and-Touch Transfer, Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA, Association for Computing Machinery, pp. 1–15 (online), 10.1145/3544548.3580677 (2023).
[22] Chulpongsatorn, N., Willett, W. and Suzuki, R.: HoloTouch: Interacting with Mixed Reality Visualizations Through Smartphone Proxies, Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA ’23, New York, NY, USA, Association for Computing Machinery, pp. 1–8 (online), 10.1145/3544549.3585738 (2023).
[23] Pietroszek, K., Wallace , J. R. and Lank, E.: Tiltcasting: 3D Interaction on Large Displays using a Mobile Device, Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, UIST ’15, New York, NY, USA, Association for Computing Machinery, pp. 57–62 (online), 10.1145/2807442.2807471 (2015).
[24] Büschel, W., Mitschick, A., Meyer, T. and Dachselt, R.: Investigating Smartphone-based Pan and Zoom in 3D Data Spaces in Augmented Reality, Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI ’19, New York, NY, USA, Association for Computing Machinery, pp. 1–13 (online), 10.1145/3338286.3340113 (2019).
[25] Hursale, S., Makwana, H., Panchal, A., Bhakkad, C. and Vairagar, S.: Smartphone’s Based Gesture Recognition in Air, Proceedings of the 2015 International Conference on Advanced Research in Computer Science Engineering & Technology (ICARCSET 2015), ICARCSET ’15, New York, NY, USA, Association for Computing Machinery, pp. 1–5 (online), 10.1145/2743065.2743114 (2015).
[26] Vinayak , Ramanujan, D., Piya, C. and Ramani K.: MobiSweep: Exploring Spatial Design Ideation Using a Smartphone as a Hand-held Reference Plane, Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction, TEI ’16, New York, NY, USA, Association for Computing Machinery, pp. 12–20 (online), 10.1145/2839462.2839490 (2016).
[27] Chen , X. A., Schwarz, J., Harrison, C., Mankoff, J. and Hudson , S. E.: Air+Touch: Interweaving Touch & In-Air Gestures, Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14, New York, NY, USA, Association for Computing Machinery, pp. 519–525 (online), 10.1145/2642918.2647392 (2014).
[28] Šimoník, M.: Record3D, https://record3d.app/ (2024). 最終閲覧日: 2024年12月5日.
[29] Apple Inc.: CMMotionManager, https://developer.apple.com/documentation/coremotion/cmmotionmanager (2024). 最終閲覧日: 2024年12月5日.
[30] Google LLC: MediaPipe, https://ai.google.dev/edge/mediapipe/solutions/guide (2024). 最終閲覧日: 2024年12月5日.
[31] Apple Inc.: Apple Support, https://support.apple.com/en-us/102381 (2024). 最終閲覧日: 2024年12月5日.
[32] Brooke, J.: SUS: A Quick and Dirty Usability Scale, Usability Evaluation in Industry, Vol. 189 (1995).
[33] Kim, Y., An, S.-G., Lee , J. H. and Bae, S.-H.: Agile 3D Sketching with Air Scaffolding, Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, New York, NY, USA, Association for Computing Machinery, pp. 1–12 (online), 10.1145/3173574.3173812 (2018).
[34] Drascic, D. and Milgram, P.: Perceptual issues in augmented reality, Stereoscopic Displays and Virtual Reality Systems III (Bolas, M. T., Fisher, S. S., Bolas, M. T., Fisher, S. S. and Merritt, J. O., eds.), Vol. 2653, International Society for Optics and Photonics, SPIE, pp. 123–134 (online), 10.1117/12.237425 (1996).
[35] Diaz, C., Walker, M., Szafir , D. A. and Szafir, D.: Designing for Depth Perceptions in Augmented Reality, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 111–122 (online), 10.1109/ISMAR.2017.28 (2017).
[36] Wanger, L., Ferwerda, J. and Greenberg, D.: Perceiving spatial relationships in computer-generated images, IEEE Computer Graphics and Applications, Vol. 12, No. 3, pp. 44–58 (online), 10.1109/38.135913 (1992).
[37] Hart , S. G. and Staveland , L. E.: Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, Human Mental Workload (Hancock, P. A. and Meshkati, N., eds.), Advances in Psychology, Vol. 52, North-Holland, pp. 139–183 (online), https://doi.org/10.1016/S0166-4115(08)62386-9 (1988).
[38] Lewis , J. R. and Sauro, J.: Item Benchmarks for the System Usability Scale, J. Usability Studies, Vol. 13, No. 3, pp. 158–167 (2018).

Corresponding author

Register with J-STAGE for free!