AUGMENTED REALITY BASED GESTURE DETECTION & OBJECT CREATION SYSTEM USING XCODE & ARKIT

In this modern era of mobile applications, Augmented Reality (AR) is becoming one of the emerging areas of implementation for researchers around the globe. While live gesture detection and 3D model creation still need more attention from researchers. In this paper, we present an interactive AR character that directly interacts with real objects. The interactive AR character automatically determines how to behave and to control real objects. This current research presents three studies that test the social psychological effects of Augmented Reality. In this research, we are using Apple’s IDE for native iOS development on swift, XCode for UI design and architectural functionalities of our application. We used the ARKIT library to import all the necessary functions and classes to manipulate and use according to our needs. Finally, we successfully deployed an IOS application which can detect live gestures of our hand movements and then create 3D models with the help of our hand gestures.


INTRODUCTION
Augmented Reality (AR) is a grand vision where the digital domain blends with the physical world. Information not only follows a person, but also her very gaze: looking at an object is enough to retrieve and display relevant information, amplifying her intelligence. Though research on AR has advanced over the last several decades, AR technology has yet to reach the mass-market. The minimum requirement for AR is a display, a camera for tracking, and a processing unit. These are also the components of camera phones, predicted to account for more than 80 percent of total worldwide mobile phone sales by 2010.
The average person learns better by observing and listening something than by simply reading something. We will be using this specific property of the human mind to accelerate learning. Although there are thousands of videos on the internet about almost every field of life but they're mostly from the perspective of video makers. By combining AR with learning and practicing we will provide immersive learning experience to our users in a way that the information settles in the long-term memory, with control over their interaction with the object they experiment as the way they want it. AR being the in the top 5 technologies of 2019 is momentous for the upcoming future as users incline towards augmented virtual experiences. With AR users can experience and interact with things that would be physically inaccessible or inconvenient otherwise. Our product will cater to users from students to professionals belonging to diversified fields of life and practice. The ability to touch and interact with augmented models in real time space is a new to the industry of Augmented Reality and our app is one of a kind with its ability to utilize real world spaces and objects to detect and write with our bare hands or using any object, from your ordinary pen, to any object with a pointed end. Users can write and then save their work to resume or view later.

RELATED WORK
This research describes experiences with development of an AR application for augmentation of an industrial robot. The paper focuses on the description of the application requirements and prototype implementation (Löfvendahl, 2014). Another application is Robotic Modeling Assistant (RoMA), an interactive fabrication system providing a fast, precise, hands-on and in-situ modeling experience. As a designer creates a new model using RoMA AR CAD editor, features are constructed concurrently by a 3D printing robotic arm sharing the same design volume.
The partially printed physical model then serves as a tangible reference for the designer as she adds new elements to her design. RoMA's proxemics-inspired handshake mechanism between the designer and the 3D printing robotic arm allows the designer to quickly interrupt printing to access a printed area or to indicate that the robot can take full control of the model to finish printing. RoMA lets users integrate real-world constraints into a design rapidly, allowing them to create well-proportioned tangible artifacts or to extend existing objects. We conclude by presenting the strengths and limitations of our current design (Peng et al., 2018).
An app that is based on usage of AR in historical recreation as well as it elaborates previous work that has been done in this area and gives knowledge about technologies that allows creating application for AR oriented historical site. This app will showcase various ancient artifacts and heritage sites across the world showing how important history is how we can learn from the immaculate and genius architecture and solutions of the past that those people came up with much lesser resources (Desai, 2018).
An Augmented Reality application for mobile devices that promoted and supports the learning of geometric figures. The application is named AGeRA, consists of a geometry book and software capable of reading special markers inserted into the book's content.
When this book is placed in front of the camera of a mobile device, 3D objects, sounds, animations, and other interactive elements leap from book pages making learning more immersive and interesting. Tests were made with teachers and students and showed good acceptance of the application to support the teaching of geometry (Neto, 2013).
An android based app has been developed in Indonesia help the Indonesian students to learn Batik. Pre-test and post-test are administered to check whether the application is improving the spatial intelligence of the students. This android based application was based on Augmented Reality Batik Ikonik (ARtikon) Joyful. First registration to the app is required by logging in with matching usernames and password. After successfully logging in the 3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 -4143 Edición Especial Special Issue Noviembre 2020 83 https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91 camera will be activated. All the 2D and the 3D objects will reflect in the Batik patterns by this app while the object will remain unidentified if the object is in 2D (Widiaty et al., 2017).
KUFSGT a school of Global tourism at Kyoto University of Foreign studies collaborated with MAVR, an immersive learning group in Japan, for augmented learning environments.
With the help of AReiantation, application which simplifies the workflow of AR contents so in seconds the idea can be prototyped. On the open campus they created an area for the activities they call ARVR experience zone. These activities included KUFSGO and MY HOMETOWN PROJECT (Hawkinson, 2019). An application was developed in India that scans the menu of any restaurant to provide ratings to each dish on the menu by the help of OCR. These rating were given by the sentiment analysis over the reviews provided by the customers. For a better experience, these menus have an AR system that projects the rating for the dishes (Wang, Chen, & Lang, 2015).
A researcher developed an innovation by the implementation of Augmented Reality technology in natural sciences learning of elementary school. The method that was used in this research was the method based on the Ball and George theory, which contained ten stages: Research and information collecting, Planning, Develop preliminary form of product, Preliminary field testing, Main product revision, Main field testing, Operational product revision, Operational field testing, Final product version and Dissemination and implementation (Fakhrudin, 2018).
A research was conducted; the purpose of this research was to calculate the impact of Augmented Reality mobile applications on the learning motivation of health science students of University of Cape Town. The ACRS (attention, relevance, confidence and satisfaction) model measured the impact of Augmented Reality on a student's motivation, and the Instructional Materials Motivation Survey guided the way to design the research instrument. A total of 78 participants used the Augmented Reality mobile application, and the result showed that satisfaction, attention and confidence factors were significantly increased. However, the decrease in relevance factor was proved to be insignificant (Khan et al., 2019 and other objects into real-world backdrops, and use their hands to manipulate those objects as if they were here. The developers are hoping the new system that can be a tool for artists, designers, game developers and others to experiment with Augmented Reality (AR). The team will present at the ACM Symposium on User Interface Software and Technology (UIST 2019) in New Orleans. The source code for Android is freely available for download on the researchers' website, and iPhone code will follow soon (Brown University, 2019).
Physics education applications AR, which has been developed extensively in recent years, have a lot of restrictions in terms of accuracy and performance. The purpose of our research is to develop a real-time simulation system for physics education that is based on parallel processing. In this research, we present a video see-through AR (Augmented Reality) system that includes an environment recognizer using a depth image of Microsoft's Kinect V2 and a real-time soft body simulator based on parallel processing using GPU (Graphic Processing Unit). Soft body simulation can provide more realistic simulation results than rigid body simulation, so it can be more effective in systems for physics education. We have designed and implemented a system that provides the physical deformation and movement of 3D volumetric objects and uses them in education. We plan to use the stand-alone AR device including one or more cameras to improve the system in the future (Sung et al., 2019).
Mobile devices are becoming a common target for Augmented Reality applications, especially for showing contextual information in buildings or construction sites. A prerequisite of contextual information display is the localization of objects and the device in the real world. In this research, we will present our approach to the problems of mobile indoor localization with a building model. The approach does not use external sensors or input. Accurate external sensors such as stationary cameras are expensive and difficult to set up and maintain. Relying on already existing external sources may also prove to be difficult, as especially inside buildings, Internet connections can be unreliable and GPS signals can be inaccurate. Therefore, we try to find a localization solution for Augmented Reality devices that can accurately localize itself only with data from internal sensors and preexisting information about the building. If a building has an accurate model of its geometry, we can use modern spatial mapping techniques and point-cloud matching to find a mapping between local device coordinates and global model coordinates. We use normal analysis and 2D template matching on an inverse distance map to determine Industrial Augmented Reality (AR) applications demand high on the visual consistency of virtual-real registration. To present, the marker-based registration method is most popular because it is fast, persistent, and convenient to obtain the registration matrix.
The registration matrix should multiply an o set matrix that describes the transformation between the attaching position and the initial position of the marker relative to the object.
However, the o set matrix is usually measured, calculated, and set manually, which is not accurate and convenient. This paper proposes an accurate and automatic marker object of set matrix calibration method. First, the normal direction of the target object is obtained by searching and matching the top surface of the CAD model. Then, the spatial translation is estimated by aligning the projected and the imaged top surface. Finally, all six parameters of the o set matrix are iteratively optimized using a 3D image alignment framework.
Experiments were performed on the publicity monocular rigid 3D tracking dataset and an automobile gearbox.
The average translation and rotation errors of the optimized o set matrix are 2.10 mm and 1.56 degree respectively. The results validate that the proposed method is accurate and automatic, which contributes to a universal o set matrix calibration tool for marker-based industrial AR applications (Yin et al., 2019). This current research presents three studies that test the social psychological effects of Augmented Reality.
Study 1 examined participants' task performance in the presence of embodied agents and replicated the typical pattern of social facilitation and inhibition. Participants performed a simple task better, but a hard task worse, in the presence of an agent compared to when participants complete the tasks alone. Study 2 examined nonverbal behavior. Participants meet an agent. Participants wearing the headset never sat directly on the agent when given the choice of two seats, and while approaching, most of the participants chose the 3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 -4143 Edición Especial Special Issue Noviembre 2020 86 https://doi.org/10.17993/3ctecno.2020.specialissue6.79-91 rotation direction to avoid turning their heads away from the agent. A separate group of participants chose a seat after removing the Augmented Reality headset, and the majority still avoided the seat previously occupied by the agent. Study 3 examined the social costs of using an Augmented Reality headset with others who are not using a headset. Participants talked in dyads, and Augmented Reality users reported less social connection to their partner compared to those not using Augmented Reality. Overall, these studies provide evidence suggesting that task performance, nonverbal behavior, and social connectedness are significantly affected by the presence or absence of virtual content (Miller et al., 2019).
In the research field of Augmented Reality (AR), applications using interactive characters have been developed as the form of giving users information such as LEGO assembly guidance and explanation about historical artifacts. Even though these characters respond to interaction with users, they could not create substantial effects or changes in a real space.
Therefore, this limitation makes users reduce their coexistence with the AR characters.
In this paper, we present an interactive AR character that directly interacts with real objects. The interactive AR character automatically determines how to behave and to control these objects. At first, we make working space populated by AR characters that has a real object with which the AR character can interact. As an interactive AR character, we implement ARMate, which presents realistic responses according to changes of real objects manipulated by a user in real time. We develop ToyCart as a physical object that includes hardware devices for movement, and ARMate can control ToyCart. Finally, we expect that our AR character can increase coexistence through real object-based interaction (Kang & Woo, 2011).

XCODE
Using apple's IDE for native iOS development on swift, we used XCode to design the UI and architectural functionalities of our application. We used the ARKIT library to import all the necessary functions and classes to manipulate and use according to our needs. We then designed the UI of the application based on the Apple's modern iOS design language.
The progression of development of our app was divided into 2 phases, first being the UI/ 3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 -4143  UX of the application, and second being the main programming and functionality of the application.
As the application is based on iOS and developed on XCode, we used the following libraries to construct our application.

ARKIT
Apple's AR Kit is a powerful library and tool that enables us to use already written code according to our functionality needs and modify it accordingly. Using the camera on iPhone or iPad it can detect live people in the real environment and track their motions and movements. We used this feature to track the hand or object movement that will enable us to draw and write in real time using the front or back camera of the device using our hands or objects. AR Kit also enabled us to make anything 3D after drawing it in 2D on any surface using its animation tools. ARKit is based on two main features, the camera in a location for a 3D space and the second main feature is the detection of horizontal plane.
To make it possible the camera of your phone assumes that it is in an actual 3D space and placing some 3D object will pinpoint in real 3D space. Then he ARKit detects the horizontal surface and places the object on top of it. So how the ARKit can perform this? This is done through a simple technique known as Visual Inertial Odometry (VIO). In 3D space, to track the location of the device with the help of motion sensors fused with camera frames, this technique is known as VIO. By detecting the edge points in the image with high contrast we can track the motion of the camera frames.
It estimates where the device is in the 3D space, by detecting how much of these points relatively moved to each other from one frame to another frame. This is the sole reason why ARKit couldn't work with a white wall in the background or when the device is moving too fast to detect the 3D space which causes to create blurred images. We can create a new ARKit project from New > Project > Augmented Reality App. It is more accessible to begin the AR tutorial with the official Apple ARKit sample, which make plane detection more feasible.

RESULTS
By successfully integrating everything together, this is the result of our AR application.
Firstly as seen below the yellow box detects your fingers or any other pointing object and then starts tracking it: After successfully track locking the finger/object, a red dot appears which will act as a pen to draw on the surface below:  Result of this can be seen below as after successfully tracking and drawing: Figure 3. The desired 3D model is drawn successfully.

CONCLUSION
AR is one of the fastest future technologies that will soon become a huge part of our everyday lives, and with huge conglomerates adopting it such as Google, Apple, Facebook, it will soon be a must with everything.
Keeping that in mind, and the fact that immersive learning is so much of a better experience than the more traditional forms of learning such as text and video, it is bound to get attention and appreciation for the awe factor it cases. Technology has come a long way and not to use it to its full potential would be a waste of all the hard work put in by scientists and engineers to bring that innovation to the table of everyday common man so he can rejoice life and maybe one step at a time make it easier and convenient for him.