Robust visual-based tracking using deep learning with image enhancement for reach stackers in container terminals

Introduction

Container ports worldwide are rapidly implementing automation technology to varying degrees, leading to the development of fully automated container terminals. Additionally, once considered futuristic, smart seaports are now adopting technologies ranging from artificial intelligence (AI)-driven equipment handling to autonomous shipping solutions. Advanced technology can enhance the safety and consistency of container handling operations, which are crucial for maintaining high operational efficiency and sustainability in smart ports connected to global supply chains [1]. Despite these advancements, major ports continue to face various inefficiencies related to safety globally, highlighting the demand for efficient and resilient supply chain solutions. Current innovations serve as foundational elements for optimizing port management, enhancing efficiency, and increasing competitiveness in the face of disruptions. Given the enormous volume of global goods, reach stackers have become essential machines for stacking and handling containers, replacing conventional equipment such as forklifts. The container transfer process during narrow and rapid maneuvers, which relies on human operators, poses significant risks. Providing reach stackers with enhanced visual assistance is vital for preventing potential accidents and dangers, particularly when various vehicles cross paths, encounter obstacles, or when personnel are within the maneuvering range. As an advanced automation technology for terminal equipment, stacker-assistance devices help prevent collisions and accidents by assisting drivers in safeguarding both workers and port equipment. One such assistance method involves improving visualization through cameras that monitor container-handling operations in container yards. The vision system displays nearby objects (targets) on a screen and provides information on their size, position, and movement. Recently, camera-based approaches have outperformed traditional sensor-based schemes in port environments due to their cost-effectiveness, and onboard computers can effectively process deep learning algorithms [2]. This combination allows data frames from the camera to be preprocessed using computer vision algorithms, facilitating safer handling for reach stacker drivers. Computer vision applications in maritime logistics have recently expanded significantly, offering active protection and warnings in critical situations [3]. Visual object tracking is essential across various applications, including autonomous vehicles, video surveillance, human-machine interaction, security, and analysis of dynamic behavior. Multiple object tracking (MOT) can detect and track multiple targets in port environments, accurately identifying objects in each frame and representing them as a set of trajectories, thus providing efficient visual information for various port operations through an approach known as tracking-by-detection [4], [5]. As illustrated in Fig. 1, multiple objects might be tracked in the driver assistance system of an autonomous vehicle. Each object’s tracking route is drawn in the subsequent frames to show the desired trajectory. During the handling process, the reach stacker may need to focus more on tracking a specific container than other types of objects in the assistance system, providing the driver with additional visualization support and attention to the particular container.

Thus, single object tracking (SOT) focuses on tracking an individual target at a time, making it suitable for tasks that require attention on a specific object rather than multiple ones. Models within the detection-free tracking category are expected to accurately track any given object. The object is defined in the initial frame of the video and should be detected and tracked in subsequent frames. Both SOT and MOT are commonly employed in computer vision tasks, wherein each object is enclosed within a bounding box in every frame. For each subsequent frame, the task involves moving and applying scaling transformations to the previous bounding box to accurately enclose the target in the current frame [6]. SOT is a fundamental task in computer vision with a wide range of applications. Its objective is to track a single object’s trajectory across sequential frames, given its initial appearance; SOT can continue monitoring the initial object even in the presence of multiple other objects [7]. This capability is particularly relevant in reach stacker environments, where the driver must monitor container handling amidst numerous containers visible in the camera’s field of view. Two critical prerequisites for effective trackers are real-time or near-real-time updates and the ability to operate under challenging conditions, such as fog, rain, cloud cover, and nighttime. Numerous methods have been proposed to address the challenges of SOT. Among these algorithms, the actor-critic tracking (ACT) tracker achieves real-time performance and high accuracy due to its simplicity, significantly accelerating the tracking process [8].

Actor-critic tracking has been well-known as a robust method for real-time processing since its introduction in 2018, with a guaranteed 30 frames per second (FPS) on the OTB-2015, considered the standard object tracking benchmark. The ACT technique provides continuous action to move the bounding box to the target, which is trained via the deep deterministic policy gradient (DDPG) algorithm. ACT was chosen to ensure its balance, speed, and precision. ACT suits the need for SOT to track a specific container in dynamic port environments. In contrast, modern MOT trackers, such as DeepSORT [9] and BoT-SORT [10], focus on multiple objects and provide robustness; however, their dependency on a detector reduces tracking efficiency. Another method, named MDNet [11], achieves high precision in object tracking, with a precision of 0.909 compared to 0.859 for ACT in OTB 2015. However, it is only capable of 1 FPS, compromising speed for high robustness. The A3CT utilizes a state-of-the-art tracker to learn an effective tracking policy at 90 FPS on the OTB-100 benchmark, enabling autonomous tracking of the target object. It has a lower precision than ACT (0.568 compared to 0.859). Like A3CT, the GOTURN [12] can run at 100 FPS but has lower precision than ACT (0.534 compared to 0.859) on the OTB-100 benchmark. The ACT baseline is competitive on our dataset, but the novel integration with the image enhancement module (DCP) can increase precision by 11 % while achieving 27 FPS on our dataset. This provides a robust solution to container tracking under adverse weather conditions. Related works in computer vision systems for autonomous vehicles offer valuable insights into the requirements of operating under various environmental conditions. For instance, due to its capability to perform complex mathematical equations at high processing speeds, the field programmable gate array (FPGA) is utilized in a lane detection system for daytime and nighttime vision, making the lane detection system suitable for a fully autonomous vehicle to operate under varying conditions [13], [14]. Additionally, a performance evaluation of lane detection and tracking techniques shows the efficiency of the Hough transform in environmental conditions. The proposed integration of ACT with DCP can enhance visual tracking accuracy in complex port environments [15], [16].

Most single-object tracking algorithms rely on sequential actions [17] or continuous actions [8], [18], [19], [20] to move the bounding boxes. The optimization-driven DDPG method is a popular deep reinforcement learning (DRL) algorithm. This algorithm is a well-known on-policy gradient-based technique for actor-critic structures that can handle continuous action spaces. Due to its robustness and stability, DDPG has been applied in various areas, including video games, robotic manipulation, autonomous driving, industrial automation, energy management for optimizing power systems, and financial trading for portfolio optimization, as reported in [21], [22], [23]. For instance, DDPG enables robotic grasping tasks by continuously learning control policies, as well as smooth lane changing in autonomous driving and obstacle avoidance for UAV navigation, highlighting the need for continuous and stable control policies in dynamic environments [21]. Thus, the DDPG algorithm provides a continuous action space framework that can improve the real-time tracking of SOT problems [8].

Container and code detection can facilitate the generation of the object’s first appearance. This method verifies the correct container and generates a bounding box around it, providing the initial input for the tracking model [24]. Although the ACT tracker performs well on popular benchmarks [25], these benchmarks do not consider adverse conditions as evaluation criteria. In port environments characterized by such conditions, including fog, rain, cloud cover, and nighttime operations, image quality is often compromised, significantly impacting tracking performance. In these scenarios, an image-dehazing module is typically employed before the main model, enhancing the accuracy and efficiency of the combined system. Moreover, real-time performance necessitates that dehazing modules be simple and efficient [26]. Object tracking techniques play a pivotal role in managing the complex scenarios of container terminals, where the movement of various entities—such as containers, cranes, yard trucks, and personnel—requires continuous monitoring and analysis. Despite extensive research efforts, key challenges remain due to factors such as deformation, illumination variations, rotation, rapid motion, and occlusion. The proposed method facilitates visual container tracking during handling operations, offering a cost-effective solution for container management. The reach stacker first lifts the container from its initial position, begins the handling operation, moves the container to the final position, lowers it, and concludes the operation. During this process, the model tracks the container in each sequential frame, providing the reach stacker driver with essential visual information. This tracking method utilizes a real-time ACT model, combined with an image-dehazing algorithm, to monitor the handling of containers by the reach stacker. Compared to the ACT model alone, the integrated model with dehazed images enhances tracking efficiency, addressing challenges posed by adverse conditions in the port environment. Consequently, this enables the successful application of a container tracker, as the algorithm continuously tracks the container’s movement in subsequent frames. This capability improves the driver’s visibility during container handling operations. The achievements of this study are described as follows.

•
The precision has been improved by 11 % (0.8841 compared to 0.7910 at CLE 20 pixels) for container handling under adverse conditions, thereby enhancing driver visibility and operational safety in the dynamic port environment.
•
The novel integration of DCP with ACT addresses existing visual detection challenges in adverse conditions, thereby tackling a research gap – the adverse weather effects on port operations.
•
The proposed method achieves real-time tracking at 27 FPS on mid-tier hardware, ensuring the deployment for the reach stacker in a real container port environment.
•
The proposed method reduces misdetections under challenging weather scenarios by employing the SOT, which focuses on a single object — a handled container by a reach stacker — rather than the MOT method. It outperforms the efficiency of MOT trackers in an automated container terminal with similar-looking objects, improving reliability for reach stacker operations.
•
Integrating a computer vision model with deep learning can significantly improve container handling operations.

The remainder of this paper is organized as follows: Section 2 provides a brief review of related work on object tracking models. Section 3 outlines the problem formulation. Section 4 describes the methodology, detailing the ACT framework and image enhancement techniques. Section 5 presents experimental results that demonstrate the effectiveness of the proposed model. Finally, Section 6 concludes with insights and directions for future research.