MLP for IT Log analysis

IT logs contain valuable information on faults and troubleshooting, information security, applications, and infrastructure performance. Government regulations also sometimes require that IT organizations store the logs for a period of time.

MLP can process terabytes of IT operational logs in near real time continuously 24 hours a day. MLP can collect data from numerous sources in IT infrastructure, such as operating system (Windows, Linux, AIX, etc.), database, applications, network, security, middleware, SYSLOG, SNMP, network packet mirroring, netflow, etc. Data can also be collected remotely without running agents on the operating system. MLP can consolidate these logs into a central place, with fine-grained usage permissions attached to each data source. That way, only authorized personnel can access them.

The logs are then parsed using MLP's high-performance parsing engine. The parsers are designed by drag-and-drop in a fully graphical parser studio environment. The parsing process extracts key fields from the data and outputs them into the real time data processing environment as well as to the search index. The logs can also be indexed into a search index without parsing.

In MLP, the modeling of the data is done by a drag-and-drop model development environment. The models process real time streaming data through them. Models can also process previously stored historical data. The models create statistics and apply machine learning algorithms, and insert the resulting data into one or more database tables for reporting. Reports, dashboards and alerts are generated from the models and databases.

The user can drill down into the alerts and reports to get into the details of the raw data, to locate causes of operational problems. The user can also search into the logs to locate problems. However, locating issues, causes and correlations using refined information such as from machine learning algorithms is the preferred way to use MLP.

In summary, MLP provides a complete log data mining, analysis, search, refinement and investigation environment.

MLP for Banking Transactions

Banking transactions run on highly secure and protected computer systems. They need to be highly available and reliable. Any abnormal delay or failure in a banking transaction is considered detrimental to the banking customers, and consequently to the bank itself. Therefore, monitoring the transaction steps, and performance as they occur, are very important. In addition to this requirement, banks and stock brokerage firms may also want to capture unusual or illegal trading activity and financial fraud.

MLP has several ways to collect data to extract relevant information and to recreate transaction steps. One way is to collect log data from the servers that run the transactions. A second data source is from analyzing IP packets at the switches that run the transaction network. From the analysis of the packets, we can decode the relevant parameters such as delay times, account IDs, user names, amounts of money transfer and balance, types of transactions and channels, etc. A third way is to monitor the database that keeps the transaction records. A fourth way is to monitor the message queues. Other transaction monitoring products usually rely only on one of the data inputs. MLP has the unique ability to combine such diversity of data sources into a single modeling system, in near real time, proving very rich analysis and results.

Once data about transactions are collected in real time, MLP parses this and extracts the key fields. Then using MLP's flexible modeling tool, it can stitch together the transaction steps to form a complete analysis. Then, MLP conducts further analysis and machine learning to find faults, unusual delays, failures, fraud and other abnormal events. It can locate the steps that failed and show the analysis on a display. It can pinpoint users who are behaving abnormally. MLP can also predict transaction rates and potential congestions.

MLP can decode fixed transaction topology paths or completely arbitrarily defined transaction topologies. MLP can also display these in several powerful ways or large dashboards on the wall or small displays on the computer monitor. Abnormal transactions or users can trigger alerts to be sent via SMS, email, SYSLOG, SNMP trap, etc.

MLP for Network Packet Traffic and Netflow

Network packet traffic within the perimeters of the enterprise IT provides a lot of information about internal security conditions, internal application performance, network faults and so on. Network traffic at the perimeter of the network provides information about WAN performance, security attacks, external-facing service performance, etc.

There are mainly two ways to monitor network traffic. First, network packets can be mirrored at the Ethernet layer (layer-2) at switches. These packets need to be collected and decoded to reconstruct protocols such as TCP, UDP, HTTP, DNS, etc. The other way to monitor packet traffic is to program a router or layer-3 switch to decode netflow. Netflow decoding occurs within the router from which the flow information is sent via UDP packets to a netflow collector. Netflow is an industry standard for measuring network packet traffic flow at the IP layer. The primary types of netflow are version 5 and version 9.

MLP has receivers to process and decode packet traffic to reconstruct the flow information of protocols such as TCP, UDP, DNS, HTTP, FTP, etc. MLP also includes a netflow decoder. Therefore, MLP can process both of these kinds of data. It also performs key functions such as packet deduplication, event deduplication, creating bidirectional events from two or more unidirectional events, and more. MLP also can utilize dozens or even hundreds of receivers for packets and netflow. It can process the flow information in memory and extract the key features from them. It can apply rules to this to calculate security and performance metrics. The rules are implemented in an in-memory rule engine. MLP can also apply machine learning based anomaly detection on this data to detect abnormal activities and zero-day security threats within and outside the network. Anomaly detection based on machine learning creates a model of normal activities and automatically detects anything that lies outside the learned "profile". This multidimensional algorithms in MLP can detect previously unknown and potentially costly or troublesome issues in an IT environment.

MLP's flexible in-memory stream data modeling and processing engine can be used to configure the network traffic or netflow analysis to fit the customer's environment precisely without custom-writing software code. This is done entirely with drag-and-drop programming by non-programmers. Alternately, one can also deploy the standard application from Yosemei to accomplish this. The standard application for network traffic can also be tweaked to process the network flow data in customized fashion to fit the specific IT environment. The network traffic flow can be reported using MLP's flexible reporting system.

MLP can process very large amounts of network traffic data at the enterprise scale or even telecomm carrier scale. Its flexibility and scale are unmatched by dedicated network traffic analysis products.

MLP for IT Infrastructure and Operations

IT infrastructure often runs network, servers, applications, middleware, storage, security devices, etc. IT managers often would like to get the complete real time picture of how their infrastructure is running. Since the systems, applications and network are all dependent on one another, faults, performance bottlenecks, security issues and capacity problems can all be correlated. Events and logs from the infrastructure are available from file systems, operating systems, network devices, applications and databases. These events are correlated in order to obtain a total picture of the IT infrastructure operation.

MLP can collect the events and logs in various ways and parse them to extract the key parameters, then use various technologies such as rule engine, classification algorithms, anomaly detection, predictive intelligence, and search engine to cross correlate them and provide multi-dimensional views into the behavior of the IT infrastructure. MLP is designed in such a way that the user is notified of changes and anomalies in operations so they can proactively maintain their infrastructure. If the user is only provided with a search capability, they will often have to work in a reactive mode, searching for causes of problems they have experienced. A proactive model of maintaining IT operations is better because it prevents problems or reduces the damages caused by them. Therefore, warnings produced by machine learning algorithms help save the additional cost and labor associated with the traditional "reactive method".

Every IT environment is unique. By configuring real-time data analysis models and dashboards, MLP can be custom-configured to fit complex IT environments, without writing custom software applications.

MLP for Manufacturing

Modern factories are equipped with machines that can digitally provide data about their operational conditions. The data can be collected via electronic interfaces such as RS232, Modbus, etc., or from the machine's computer logs. The information collected includes the physical conditions such as pressure, temperature, position, vibration, dimensions of parts, rotational speed, electrical current, and voltage. It can also be about machine conditions, errors, data files, drawings, etc. There will be unstructured, semi-structured and structured data. Some have high volume such as terabytes per day and some have low volume such as several kilobytes a day. Some data have concentrated information, while some others have very little valuable information.

MLP can be used to collect, parse, process, store, and analyze factory data in real time. MLP can implement troubleshooting capability, root cause analysis ability and other problem resolving applications. MLP can also be used to run data analysis applications which help increase the yield of the production, reduce the wastage of resources or increase the units produces per hour. Predictive maintenance of machines can be performed using MLP-based applications to prevent catastrophic and costly failures. It can also be used to prevent outages by replacing parts or machines that are about to fail.

MLP is particularly suitable for both manufacturing assembly lines with human operators, as well as highly automated factories where robots do the majority of the work. It can monitor machines performing operations such as cutting, shaping, welding, joining, coating, riveting, assembling, forming, bending, examining, transporting, soldering, molding, etc. The knowledge of manufacturing processes can be embedded into the applications using flexible data modeling environment in MLP. Customized manufacturing analysis solutions can be made just by drag-and-drop programming.

The results can be displayed on factory-wall based dashboards, the foreman's display screen, or an operator's screen. Alerts and warning can be configured for operators as well as for managers. These can be customized with just drag-and-drop configuration, to show the relevant personnel what they need to see.

MLP for Energy Industry

Energy generation, transmission and distribution systems are becoming smarter. Data is being collected about these operations in order to optimize their cost and efficiency. Today, unprecedented potential exists to process real time big data and optimize energy operations. Power consumers are metered using smart meters. Their consumption behavior as well as demographic data can be analyzed. Fraud in consumption also need to be monitored and prevented, since billions of dollars are lost in fraud every year. The loss due to suboptimal energy distribution and poor efficiency is also very large.

Energy industry is currently in its early stages of digital transformation. Data is being collected from the power grid as well as consumers. The application of smart real-time data analysis provides tremendous value to the power industry participants – generation and distribution companies, consumers, the environment and the entire power ecosystem.

Recently, renewable energy sources such as wind and solar power are making the power sector more complex due to their special characteristics. In addition to electrical power, there are other power distribution systems, notably hot water and steam, which heat buildings in the coldest regions of the world. Such systems use thermal power station to heat water and supply it through heat exchangers to consumers via networks of hot water. Such heat energy grids also face similar problems similar to electrical power grids, and therefore need similar data analysis capability as electrical power grids.

MLP is capable of processing data from millions of consumer meters as well as power grids in order to create intelligent inferences and actions for the power industry. Applications running on MLP can be used to accomplish the following: save energy and reduce costs, forecast load, provide decision support, locate and resolve faults, predict equipment failure, accomplish smarter management, improve security, reduce risk, optimize power supply, reduce fraud, and prevent loss.

MLP for Transportation and Logistics

Transportation and logistics networks are being equipped with sensors that monitor their location, temperature, fuel, weather, pressure, equipment health, maintenance status, inventory, package handling, battery condition and other such information. Data can be collected and processed in near real time from these sensors. Combining this data with static data regarding inventories, vehicle fleets, and personnel, the resulting information can be made very rich and beneficial. IoT data analytics can help facilitate optimum operation, material management, communication links, energy management, traffic planning, passenger and driver information management, preventive and predictive maintenance, routing management, etc. It can also help cope with contingencies and changes in conditions such as road conditions, weather, accidents, geopolitical issues, etc.

MLP can collect, process, store, analyze, and respond in near real time for such data from millions of sources. MLP can be programmed to take automated action based on rules. It can also apply machine learning to find anomalies which may represent abnormal conditions, failures, security problems or fault. MLP can also be used to locate and determine the causes of such problems. Another application of MLP is predictive maintenance where preventive maintenance can save cost and reduce risks of accidents or delays. MLP can also provide dashboards and reports of operations to the executive management as well as to display in the operations center.

MLP, with its real time massive data processing ability, can be used to make customized solutions for your unique transportation and logistics data analysis needs.