Detection of Human Movement Direction Using Optical Flow Analisys on Multiple Camera Angles

The active movement of children poses a safety risk in the absence of adult supervision. To reduce the risk of accidents in children, an automatic detection system for the direction of children's movements is crucially needed. In this study, detection of the direction of human movement based on image processing was carried out with the input of videos produce from 4 CCTV installed in each corner of the room. The system will detect the direction of object movement with classification of orientation, namely front, back, right and left. The detection method used in this research is Optical Flow. Optical Flow will calculate the value of the direction or orientation of the movement of an object. The orientation obtained is then accumulated with HOOF (Histogram Orientation of Optical Flow), where HOOF will collect the orientation of objects on the whole frame according to a 8-part Cartesian angle. The results of the orientation with Optical Flow will be compared with the direction of detection measured manually to determine whether the detection of movement direction using Optical Flow is running well. According to the results, it is known that the Optical Flow method has succeeded in detecting the direction of movement accurately based on different camera angles.


INTRODUCTION
In 2019, The total population of Indonesia has reached 266,91 million. The largest population is people in productive age (15 -64 years old) and the second largest is children (0 -14 years old), with 66,17 million or about 24,8% of the total population [1]. Large number of children means that we need to be more concern with their development. Children, especially in the very young age, tend to be more active and have great curiosity on what to be seen or heard [2]. They need to be monitored and supervised by adult in order to prevent from the dangerous thing or action that can harm them. One of the solutions is installing a CCTV (Closed Circuit Television) cameras around the house. However, CCTV camera only record events and unable to analyze the movement [3], it is necessary to have an integrating system which able to recognize the direction or the movement of a person through CCTV camera automatically. Several researchers have conducted research on the detection of human movement [4]- [7]. Camerabased human movement detection systems (non CCTV) are also widely developed, various methods have been developed to improve performance such as, [8] performs pixel-based motion detection. [9] detects the movement of objects using cameras and sensors. [10] detected movement during sleep using periodic limb motion detection (PLM). [11] performs face movement detection using template matching. [12] motion detection with a focus on head movement.
In this paper, we studied about the detection of human movement using videos acquired from CCTV. The camera is only able to record and showing the pictures or videos without analyzing the movement of person. However, if we utilized the camera and used the photos or videos to combine it with video processing technology, we will able to analyze and made prediction with it. The main parameter that we investigated is the detection of human movement and its direction. The direction of human movement is something that can be seen by the human eye and can be processed in the brain where the direction is.
Research [13] had been conducted a study on human movement recognition using Optical Flow histograms. In order to obtain the direction of human movement, a derivative formula from Optical Flow is needed, namely the Lucas-Kanade algorithm. The algorithm works by finding the orientation or value. The orientation values are grouped into Histogram Orientation of Optical Flow or HOOF with a reference angle of Cartesian coordinates by dividing it into 8 parts. In the first part, the orientation is between the angles of 0° to 45°, in the second part it is grouped between the angles of 45° to 90°, in the third part it is grouped from 90° to 135°, the fourth section is 135° to 180°, the fifth section is 180°. to 225°, sixth section 225° to 270°, seventh section 270° to 315° and eighth section 315° to 360°. However, this study of HOOF only uses 8 parts which are the minimum references. They lacked of accuracy of the direction of human movement.
The study in [14] determined the direction of object movement based on the Optical Flow feature. The direction or orientation in the optical flow is calculated for each pixel that moves in each frame. The orientation is very large and need to be simplified. To shorten the calculation, the HOOF technique is used which makes the orientation of each pixel in a frame grouped into a single histogram. There were more than one object in the video being tested. In this research, the direction or orientation is is calculated for each object so that each frame is divided into 16 grids. The orientation in the grids have been accumulated into the HOOF to simplify the reading of direction or movement in the frame. However, the comparison between the results of motion direction detection using Optical Flow with the actual detection results is not stated in the paper, thus the research that had been carried out was adding the results of the comparison of motion direction detection using Optical Flow detection with actual detection. This paper using research in [13] and [14] as key papers to study.

Data
We used the data of CCTV recorded video that installed in 4 corners of the room. The resolution of the video is 432×240 pixels. We recorded video with human as an object and applied several scenarios, such as the object walking towards the front, back, left, and right. The direction of the object's movement is relative to each camera as shown in Figure 2 and an illustration of the scenario of the object's movement at each camera orientation is shown in Figure 1.
The orientation of camera 1 shows the object moving towards the front of camera. While the orientation of camera 3 shows the object moving to wards the back away from the camera. From the left and right point a view, the camera 2 shows the object moving to the right and the camera 4 shows the object moving to the left of the camera.

Optical Flow
Optical Flow is a method used to find the value of the motion vector by calculating the field of optical flow in the image. Optical Flow in moving object detection works by estimating the movement of a part of an image based on the derivative of light intensity or I (x, y, t) in an image sequence. Optical Flow assumes that the flow or intensity of light in an image is basically a constant value [15] as it is shown in (1): In two-dimensional space, Optical Flow define how further an image pixel moves between two consecutive frames. A pixel at the initial position (x, y, t) with intensity ( , , ) will move with the derivative distance dx, dy and dt in between two frames as it is stated in (2). = Light intensity of Optical FLow that move for ∆x, ∆y dan ∆t.
H.O.T(Higher-Order-Terms) in (3) is an approximation solution, ude to very small value, it can be ignored to reach a simplified version in (4). Then we applied a differential equation based on time and we obtain (5). Since the derivative of the position function with respect to time is velocity, the equation in (5) becomes (6). With , is the velocity component of Optical Flow ( , , ) , and is the partial derivative of the image ( , , ) which turns Ix, Iy, It [10] then the equation becomes (7) : is the velocity component of Optical Flow ( , , ) , and is the partial derivative of the image ( , , ) which turns Ix, Iy, It [14] then the equation becomes (7) : with: = Pixel intensity gradient on the x-axis = Pixel intensity gradient on the y-axis = Pixel intensity gradient in the time domain = Pixel velocity on the x-axis = Pixel velocity on the y-axis Equation 7 yields the final Optical Flow equation to get the pixel velocity on the x-axis and y-axis. However, this equation cannot be solved because there are two unknown variables in one equation. This problem is called the aperture problem of Optical Flow. Hence, we need to develop method that able to determine and using Lucas-Kanade algorithm [15].

Lucas-Kanade Algorithm
The Lucas-Kanade algorithm is an Optical Flow algorithm that solves the previous Optical Flow equation by combining information from adjacent pixels [15]. This method assumes the image position between two frame images is small and close to constant. Then the equation in this method becomes (8).
( 1) Where q1, q2, q3 is a pixel in a frame, while (qi), (qi), (qi) is a partial derivative of the image that related to the coordinate position x, y and time domain. The equation (8) can be written in the matrix form in (9). with: If we want to find matrix then the equation can be written as in (10) and (11). In (11), the velocity of the image, and can be obtained. Optical flow assumes the small pixel of movement, however it has the drawback which unable to detect large movement. To cope up with it, we used pyramid method, namely pyramidal Lucas-Kanade [16].
The direction or orientation of the movement in Optical flow can be determined by using and [16]. The final equation to get the direction is the calculate the angle between the velocity of xaxis and y-axis as it is shown in (12), where ( , ) is the direction or orientation movement. Figure 3 shows the result of orientation movement in a frame according to the figure 2. Figure 4 shows the cartesian coordinate of x-axis and y-axis. While figure 5 shows the coordinate of the image, which has positive x-axis to the right but opposite from cartesian coordinate in figure 4, the positive y-axis is toward downwards.

Optical Flow Detection Movement
We utilized software Matlab to process the data of video recording from CCTV. We used Lucas-Kanade optical flow algorithm to detect the vector movement. There are several steps to reach the decision of the movement.

Step 1: Input Data Video
In the first step, we acquired the video recording of CCTV and used it as the input. Data from each camera was processed to generate parameters of the orientation in Optical flow. At the same time, we determined the starting point of the object and the movement direction based on the cartesian coordinate point of view. The data were stored and compared later with the result of the detection system.

Step 2: Frames Reading
Basically, the video is a collection of moving frames, thus we procced the video frame-by-frame [18].
Step 3: Change to Grayscale The next process was changed the frame image from RGB (Red-Blue-Green) into greyscale. It need to be done because optical flow is able to read only the changing in light intensity in gray degrees.

Step 4: The computation of Algorithm
Now the RGB image has been converted to a greyscale. Optical flow, by using Lucas-Kanade algorithm, calculated the orientation of the image. The result is in radian and we need to convert it into degrees. The negative value of the angle is added 360 degrees to make easier computation. Nan orientation (Not a Number) is an orientation with a value of 0. The result of NaN aims to remove the value 0 from HOOF, because HOOF tend to pick values that appears the most according to the angles. to the parts of the angle.

Step 5: Detection of the Object with GMM and Kalman Filter
The function of GMM (Gaussian Mixture model) and Kalman Filter in this study is to make it easier to visualize as well as speed up analyzing the orientation of human movement. Object detection with GMM is to segment objects or detect foreground (foreground/objects) by separating pixel values that do not match the Gaussian distribution in the background [19]. Kalman filter serves to track an object that has been defined by GMM. When the object is moving or the frame is running, the Kalman filter will predict the movement of the object in each frame.

Step 6: Compute the Orientation
The orientation that was calculated and analyzed is the orientation of the object. The orientation of the object was distinguished by the presence of a box on the object, to distinguish human objects from other moving objects.
Step 7: Orientation Coding In this step, the orientation obtained was encoded or converted into HOOF. HOOF is a histogram that serves to collect the orientation value of each pixel in each frame within a certain range of angles [5]. The 12 parts of the angle division is shown in the Table 1. Step

8: Decision of the Orientation
Lastly, the decision of human movement direction was determined by the highest number of orientations. It was visualized in the last frame.
Furthermore, to facilitate the classification we grouped the eight angle ranges into four directions, namely front, back, left and right as shown in Table 2. Classification relative to each camera as shown in Figure 1.

Evaluation
The evaluation of the detection system with optical flow is compared to the manual set point to obtain the accuracy. The comparison will be analyzed to determine whether the detection of the direction of human movement with Optical Flow has a good performance. Figure 6 shows a Cartesian diagram as a reference for determining direction.

Result of Human Movement Detection System with Optical Flow
The tests were conducted to find out the performance of detection system with Optical flow. One of the scenarios of the test is shown in Figure 7. While the rest of the result of scenarios is shown in Table 2. We took the 16 videos from 4 CCTV with different angle. Based on the result, we obtain that the optical flow is able to detect the direction of human movement with successful rate of 100%.

Performance of Orientation Direction ofMovement with HOOF Classifier and Optical Flow
Orientation direction is classified with histogram HOOF based on the cartesian diagram. Figure 8 shows one of the example of the histogram HOOF on camera 1 with the direction of human movement walk to the front side.  Initially, the angle orientation of optical flow on all frame video can not be the final direction of movement. This is because there is no information about the classification of orientation. Thus, at the final process, the detection of direction with optical flow needs to be classified with HOOF. The classification of HOOF collects all angle orientation of every pixel in each frame and classify it according to 8 diagrams. Every angle with the most orientation results will be the final decision of direction. the result from all test video is shown in Table 3.
The comparison of between the detection with optical flow and the result with manually detection of direction was done to get the accuracy of the detection. For each camera, we computed manually the angle with different point a view, which means that for every camera can be relatively different angle. Figure 9 shows the illustration of manually computation of angle. While the result of the comparison is shown in Table 4. The Accuracy yields 87,5%, which means out 14 out of 16 videos tested is compatible with manually detection. While 2 of 16 videos has wrong estimation, that is from camera 1 of the right side and camera 3 of the front side. Detection error occurs because the initial movement of the detected object is covered by immovable objects around the detection area.

Comparison with Other Research
In this part, we compare our experimental method and result with the previous results. Table 5 shows the different method and previous research as well as its accuracy. Explicitly in the table5, the accuracy of our method is lower than the research in [13] and [20]. However, the detection with optical flow method, especially with Lucas-Kanade algorithm has less complexity in term of it do not need training data as proposed in [20] and [14]. The method is similar with research in [13], yet the difference is they use more complex algorithm, Horn Schunk, while we implemented simplified version of algorithm with Lucas-Kanade. Additionally, we utilized more CCTV camera which produce more point a view to detect compared with the research in [20], [13], and [14] which only used one camera.

CONCLUSIONS
Detection system of human movement direction using optical flow is able to detect all of direction and orientation from four cameras, where each camera shows an object that moves with different scenarios, to the front, back, right and left side. The results show that the 16 tested scenarios are able to detect the direction of movement with the successful rate 100%. While the results of angle orientation show the accuracy of 87,5%. Moreover, compared to the previous research, the less complex method is the advantage of optical flow with Lucas-Kanade and HOOF diagram.

AUTHOR(S) BIOGRAPHY Elvira Sukma Wahyuni
Elvira Sukma Wahyuni received her bachelor degree in Informatic Engineering from Universitas Negeri Padang in 2012. She received her Master degree from Universitas Gadjah Mada in 2014. Currently, she is a lecturer in Department of Electrical Engineering, Universistas Islam Indonesia (UII). Her current research interests include image processing, computer vision, and machine learning.

Zulfika Iqbal
He was one of the students of bachelor degree of electrical engineering at University Islam Indonesia. He graduated in 2020.

Dzata Farahiyah
She received a bachelor degree in electrical engineering at Institute Technology of Sepuluh Nopember, Surabaya in 2010. Then, she pursued a Master degree in Communication Engineering, Electrical Engineering department, at University of Duisburg-Essen, in Germany and graduated in 2014. Currently, she is a lecturer in Universitas Islam Indonesia.