Graph-Based Human-Object Interaction Detection for Automated Worker Accountabil
Journal
Journal of Computing in Civil Engineering
Journal Volume
40
Journal Issue
3
Start Page
04026025
ISSN
08873801
Date Issued
2026-05-01
Author(s)
Khosiin, Mik Wanul
Suryo, Eko Andi
Negara, Kartika Puspa
Aknuranda, Ismiarta
Abstract
Effectively monitoring worker accountability in construction project controls requires addressing the complex, dynamic interactions between workers and objects. However, existing human-object interaction (HOI) approaches are typically limited to single actions and single-scale object sizes, lack spatial context, and overlook how large slab structures require distinguishing between local interaction areas (local objects) and broader object zones (global objects). To overcome these challenges, this study proposes a comprehensive HOI detection framework that leverages convolutional neural networks and graph attention networks to generate a robust worker accountability monitoring (WAM) system. The research identifies various worker actions, such as tying, transporting, and pouring, as key performance indicators, while also capturing noninteractive behaviors to provide a comprehensive view of on-site activity. Objects are grouped into three categories - concrete, formwork, and steel rebar - and analyzed across three object sizes: big, medium, and small. Additionally, the bounding box design of the objects incorporates both local and global scenarios, which consider the area of the associated objects. The system achieves notable performance, with a mean average precision (mAP) of 0.830 for object detection and mAP scores of 0.553 and 0.502 for HOI tasks involving human interactions in local and global contexts, respectively. These results demonstrate accurate detection and action recognition across diverse scenarios. The WAM system generates essential data: worker, object, and action information supporting holistic project management systems, including productivity monitoring, quality control, and safety inspections. By enabling managers to better understand worker interactions and crew activities on construction sites, this approach provides actionable insights to enhance future performance.
Publisher
American Society of Civil Engineers (ASCE)
Type
journal article