ANALYTICS FOR SITUATIONAL AWARENESS USING MACHINE LEARNING WhitepaperSECTION A Title – ANALYTICS FOR SITUATIONAL AWARENESS USING MACHINE LEARNINGTechnical Area(s) addressed – Entity extraction and resolution within and across data sources.Disseminate fused air tracks and provide alerts for detected anomalies.Period of Performance – 18 monthsEstimated Cost – Name/Address of Company – 4WardTech, 7317 El Cajon Blvd, Ste 111, La Mesa, CATechnical and Contracting Points of Contact (phone, fax and email)- Andrew Parker [email protected] B – TASK OBJECTIVEAnalytics for Situation AwarenessMachine LearningThis BAA shall also provide cutting-edge machine learning capabilities to assist and automate user tasks. Potential applications include providing recommendations based on what data the user is interested in and has viewed, assisting with task management and the delegation of tasks, deriving alerts from patterns of life and indications and warnings, and learning and automating common user actions such as routine queries.Organizations are turning to machine learning analytics to help solve difficult problems and uncover new opportunities. Some of the common uses of machine learning are Fraud Detection, Optimizing Campaigns, Improving Operations, Reducing Risk, etc. SECTION C – TECHNICAL SUMMARY AND PROPOSED DELIVERABLESPredictive analytics is the use of data, statistical algorithms and machine learning techniques to identify the likelihood of future outcomes based on historical data. The goal is to go beyond knowing what has happened to provide the best assessment of what will happen in the future. When software analysis and prediction plays a significant role in the equation, we need to take a closer look at the advantages computer applications have in reaching this goal.As per the objectives defined, there are several areas where can apply predictive analysis using following examples. Recommendation – Although the human planner can still plan the personnel for complex and rule-ridden time shifts, now the application can recommend perfectly good assignments. It can learn from the planner’s past plans, as well as the past data of user time preferences. The application assists the planner rather than take over his job. The planner can overrule, alter, or just approve the automatically generated plan. His or her changes will affect the future recommendations of the system. Recommendation system is one of the applications of machine learning. When we want to recommend something to a user, the most logical thing to do is to find people with similar interests, analyze their behavior, and recommend our user the same items.Behind the scenes, these systems are powered by a recommender function. A recommender function takes in information about the user and predicts the rating the user would give the product. Following are some machine learning models using which we can implement recommendation system.1) Content-Based Recommendation (CBR)2) Collaborative filtering (CF)i) user-based collaborative filteringii) item-based collaborative filtering3) Matrix Decompositions4) Clustering5) Deep Learning Approach The use of Machine Learning in recommendation solely depends upon the objective and reformulating the problem into best-suited Machine Learning algorithm. Anomaly Detection – The application of Machine Learning techniques on monitoring the transaction may take several forms. We could shortly mention the ability of these types of algorithms to detect anomalies alleviating the need to specifically code threshold-based rules and also the capability of Machine Learning techniques to learn from the experts and generalize their strategies. When integrated into a transaction monitoring module, the system can alert the expert to pay her attention to outliers or serve as an early warning system, allowing to watch their development.Following are some techniques used in machine learning to detect outliers.Linear Regression Models (PCA, LMS)Probabilistic and Statistical Modeling Z-score or Extreme Value AnalysisInformation Theory Models Proximity Based ModelsEntity Extraction – Entity extraction feature pulls people, places, dates, companies, products, jobs, and titles from the source and determines their sentiment. To implement the entity extraction model, following machine learning techniques can be used.Maximum entropy modelingHidden-Markov models (HMM)Connectionists methods How to tackle the issues?Gathering/ Collection of Data:This is the first and important step in the hypothesis. This data may reside in any form like a file, database, internet etc. The more data we are able to retrieve the more accurate results we can predict.Data Exploration & Preparation:The quality of input decides the quality of your output. It makes sense to spend a majority of your time and effort to analyze the data. To fine grain the data we need to apply some techniques which are,Variable IdentificationUnivariate AnalysisBi-variate AnalysisMissing values treatmentOutlier treatmentVariable transformationVariable creationChoose Model:Model selection is a very tough task in the process because, the type of problem and the input data needs to be studied deeply by data scientist to choose the proper model.Training the Model:By cross-validating the dataset, we divide our data into two sections Training Data and Testing Data. The training dataset is used to train the model, based on given training data the model is able to judge the prediction. The more versatile our training set the more stable our prediction.Evaluation:The evaluation is the phase where we use our test data set to predict the results and based on the results we evaluate our model accuracy.Hyperparameter Tuning:In this phase, we may need to optimize our model prediction accuracy by fine tuning some model parameters.Prediction:This is the last phase of machine learning hypothesis where we predict the results using real data.The supported frameworks for machine learning hypothesis are as follows.Scikit-LearnTensorFlowMLlib(Spark)Azure ML StudioAmazon Machine LearningApache SingaApache MahoutCONCLUSION:Using Supervised and Unsupervised machine learning techniques, able to extract and detect entities from different data sources. As well as machine learning and analytical techniques helps to detect disseminating fused air tracks and providing alerts. Machine Learning approaches applied in systematic reviews of complex research fields such as classification, prediction, extraction, image and speech recognition, medical diagnosis, learning association, etc. Our goal should be to build the solution by using the power of machine learning & artificial intelligence to resolve the complex challenges with quality results.