Introduction

The National Security Telecommunications Advisory Committee (NSTAC) Report to the President on Zero Trust and Trusted Identity Management introduces Zero Trust (ZT) as “a cybersecurity strategy premised on the idea that no user or asset is to be implicitly trusted. It assumes that a breach has already occurred or will occur, and therefore, an entity should not be granted access to sensitive information by a single verification done at the enterprise perimeter. Instead, each user, device, application, and transaction must be continually verified.”

Enterprise adoption of Zero Trust is broad and growing. Venture Beat reports that 90% of organizations moving to the cloud are adopting a Zero Trust Strategy¹ while Gartner predicts that 10% of large enterprises will have a mature and measurable Zero Trust program in place by 2026.²

Visibility is a cornerstone of Zero Trust strategy because it ensures a comprehensive understanding of networks, identities, devices, and data. Here are four reasons why visibility is crucial:

1. Continuous Monitoring: Zero Trust relies on continuously validating all devices, users, and applications to ensure security. Visibility enables the detection of anomalous behavior or potential threats in real-time.
2. Contextual Awareness: Visibility provides contextual information about who is accessing what, when, and from where. This helps in making informed decisions about granting or denying access and determining the appropriate level of access.
3. Granular Control: With detailed visibility, organizations can implement more precise security policies based on the specific needs and risks associated with different users and devices.
4. Incident Response & Digital Forensics: In the event of a security incident, visibility enables quicker and more effective responses by providing detailed logs and activity reports. This helps trace the source, identify potential attackers, their motivation, and ultimately, the breach’s impact.

Logs are fundamental to achieving this visibility. By ensuring that each component logs its activities and then examining these logs holistically, organizations can lay the foundation for Zero Trust implementations. When AI/ML models are applied to logs, organizations can unlock advanced capabilities, such as pattern recognition and event correlation. This paper examines how security operations can utilize these techniques to enhance their Zero Trust strategies, addressing challenges such as managing large volumes of log data and minimizing false positives.

Scope

This paper aims to offer a distinct perspective on analyzing logs by examining the contents of log files and how they can be parsed and correlated to identify patterns that may indicate existing or potential threats. This document does not aim to rewrite the process by which Security Information and Event Management (SIEM) tools perform such an activity. This document aims to guide the Security Operations Center (SOC) in harnessing the power of raw events and identifying threats relevant to their organizations. This document is not limited to IT systems and includes Operational Technology (OT), Internet of Things (IoT), and physical security. This guidance should not be construed as limited to cloud environments but extended to any environment with log sources.

A Brief History of Logging

Logging in IT dates back to the early days of computing. It has been available on mainframe computers, in early operating systems, on networked systems, in applications, and cloud computing.

Logging reflects the evolution of computing systems, progressing from simple print logs in the 1950s to today’s advanced, AI-driven analytics. Over the decades, innovations in logging have been driven by the increasing complexity, scale, and interconnectedness of computing environments. What began as a passive record-keeping tool has become an active and essential component of modern cybersecurity and system management. Logging is no longer just a technical process but a strategic enabler of operational visibility, security, and informed decision-making. As we look ahead, integrating AI, machine learning, and observability tools will further solidify logging’s role as a cornerstone of modern digital infrastructure.

See “Appendix 4: A Brief History of Logging in Computing” in this paper for more historical details.

Overview of Logging

Logs are a foundational element of visibility, enabling the continuous monitoring and analysis of activities across IT, OT, IoT, and cloud environments. They record actions within a system, such as access attempts, network traffic, application events, and security incidents. These records are vital for understanding system and user behavior, detecting anomalies, and responding to potential threats.

By combining structured security logs with unstructured behavioral data, network activity, and real-time threat intelligence, security teams can react to known threats and proactively identify anomalies, insider risks, and emerging attack vectors before they escalate.

Structure of Logs

Logs are records of events, which are actions or occurrences within a system or component. These events serve as historical data points, offering visibility into the system’s performance and security. Each event documents specific “happenings,” ranging from routine operations to critical incidents, and provides insight into how the system functions.

Components of an Event

1. Event Data:

   • Represents the core details of the action or occurrence.
   • Describes the nature of the event, such as a service being invoked or a file being accessed.

2. Metadata:

   • Includes supplementary information about the event, such as:

     • Timestamp: When the event occurred.
     • Source: The event’s origin (e.g., application, user, device).
     • Severity: The importance or urgency of the event (e.g., critical, warning, informational).
     • Device ID: The identifier for the device where the event occurred.
     • Other: Relevant details depending on the system and context.

Example of an Event

Consider an event where a privileged service is called. The event data would capture the details of the service call and the resulting process creation as shown below:

Figure 1: Service Call Event Data

The metadata shown below provides contextual information, including the time and location of the service call, as well as the severity level of the action.

Figure 2: Service Call Metadata

By categorizing events based on severity, systems can ensure that urgent issues are addressed promptly while routine operations are logged for record-keeping and analysis. This structure provides a systematic approach to managing and prioritizing events in logs.

Log Formats

Log formats are critical to ensuring that log data is consistently structured, easily parsed, and effectively utilized for security analysis and event correlation. In the context of Zero Trust and advanced logging practices, standardized log formats provide the foundation for aggregating and analyzing data from diverse sources, including network devices, applications, cloud environments, and IoT systems. The relation to the log structure above is evident. As various log sources contribute to the overall visibility and monitoring capabilities, standardized formats enable interoperability and efficient processing of these logs. One such example is the use of standardized UTC-based timestamps across all log sources, which enables accurate event correlation and reduces ambiguity across time zones. Without consistent formats, integrating logs into broader security strategies, such as those required in a Zero Trust architecture (ZTA), would be inefficient and prone to errors.

Summary of Key Points

• Sources: Logs originate from diverse systems, including firewalls, endpoints, cloud services, and applications, each contributing critical data for analysis and investigation.
• Generation Methods: Logs may be created automatically by systems, triggered by events, initiated by users, or generated periodically.
• Formats and Structures: Logs can be structured (e.g., JSON), semi-structured (e.g., Syslog), or unstructured (e.g., plain text), influencing how they are processed and analyzed.
• Metadata: Logs include essential details, such as timestamps, event sources, and severity levels, to provide context and facilitate correlation.

Security of Logs

Logs should be secured to maintain continued visibility and protect the integrity of the logs. Log tampering can mean that the events that provide information about the actions may no longer be authentic. Any modification of timestamps on the logs would result in a lack of correlation between events. Therefore, maintaining the integrity of logs is crucial, and system architects should consider achieving this through techniques, such as digital signatures or cryptographic hashes. Logs might contain sensitive data that must be masked when shared or aggregated. Some examples are logging secrets, confidential actions, or personally identifiable information (PII). Workflows that add logs should ensure sensitive information is masked. Proper annotations should be added so that systems can handle sensitive information appropriately.

Utilizing AI Models to Analyze Log

With the rapid expansion of digital infrastructures and cloud-based business processes, the volume and velocity of log data continue to grow exponentially. Keeping up with the sheer volume and speed of log data makes it extremely difficult to identify potential or active threats. Security analysts require solutions that can analyze logs and indicators of compromise in real-time or near real-time to prevent delayed threat detection and the prolonged presence of malicious actors in the environment.
AI-powered analytics utilize machine learning to detect indicators of compromise, recognize behavioral patterns, and efficiently uncover hidden security threats. Using AI improves efficiency, accuracy, and proactive threat detection.

Event Correlation and Pattern Recognition

Logs are collected from an ever-increasing number of locations, including firewalls, gateways, access points, servers, endpoints, routers, applications, databases, cloud logs, and IoT devices. Extracting actionable signals from all the data and separating noise is becoming increasingly like finding a needle in a haystack while drowning in a sea of data.

AI can play a leading role in this world of ever-increasing log data, especially since having a large dataset of historical events can be an advantage, allowing ML algorithms to make predictions of future attacks.

By leveraging AI in security log correlation and event analysis, organizations can strengthen their security posture and make more informed decisions.

Adding a business context to log analysis further supports analysis. The general process of adding in a business context is as follows:

• Data categorization
  • Identify critical business processes and data, including their location (databases, servers, cloud, etc.) and the users or systems that access the data.
• Gather log sources and map logs to critical business processes and data
• Enrich the logs with additional context used within the organization, e.g., department, application name, type of transaction
• Identify legitimate events related to critical business data
• Create alerts for known abnormal activities

Introducing an AI/ML Model for Pattern Recognition

The SOC team understands the organization’s threats as they are aware of the organization’s security posture. This understanding can be enhanced by utilizing AI/ML models, which can help the SOC team identify patterns and anomalies.

Below are some examples:

• Anomaly Detection and Threat identification: Identifying deviations from the norm, such as unusual login times, abnormal file access patterns, or unexpected network traffic, indicating potential security incidents. AI can quickly identify patterns and anomalies that might indicate malicious activity.
• Reducing False Positives: AI can reduce the number of false positives. Learning from historical analysis can more accurately distinguish between actual threats and regular activity.
• Real-Time Analysis: AI can analyze security logs in real-time, enabling an immediate response to threats. This can significantly reduce the time between an incident’s occurrence and detection.
• Scalability: AI can be trained to cope with increased data volume and complexity, whereas manual security log analysis is resource-intensive and can become unmanageable with high data volumes.
• Behavioral Analysis: Monitoring user and entity behavior to detect insider threats or compromised accounts by identifying deviations from typical behavior patterns.
• Signature-based Detection: Matching log patterns against known threat signatures to identify known threats quickly.
• Automated Response: Upon detecting security threats through anomaly or behavioral analysis, automated response mechanisms can take action, such as isolating compromised systems and blocking malicious IP addresses.
• Threat Hunting: Proactively searching for threats by formulating and testing hypotheses based on patterns identified by AI/ML models in historical log data.
• Predictive Analytics: Predicting potential system failures or vulnerabilities that attackers could exploit. By utilizing machine learning algorithms, AI can forecast future security threats by analyzing historical data.
• Risk Assessment and Risk Aggregation: Analyzing patterns to assess risks and AI can correlate various low-risk and rare events that collectively indicate a significant risk.
• Behavioral Profiling: Creating typical behavior profiles for users and entities to detect anomalies.
• Insider Threat Detection: Identifying potentially malicious activities by insiders based on deviations from their usual behavioral patterns.
• Data Visualization: AI can visualize log data to help analysts understand complex patterns and relationships.
• System Performance Analysis: Monitoring logs to assess system performance and detect anomalies, such as unusual CPU or memory usage, that may signal underlying security issues.

Challenges in working on pattern recognition in security logs, manually:

• Data Volume and Variety: Security logs generate vast amounts of data from various sources, making real-time analysis challenging.
• False Positives/Negatives: It is crucial to strike a balance between detection accuracy and minimizing false positives (benign activities incorrectly flagged as threats) and false negatives (missed threats).
• Adaptability: Threat landscapes are constantly evolving, requiring models to be adaptable and regularly updated with new threat intelligence.
• Real-Time Processing: Log data must be processed in real-time, which poses a challenge for applying complex pattern recognition algorithms that require substantial computational resources.
• Nonstandard Data: data sources not mapped to the Open CyberSecurity Schema Framework (OCSF) and event classes.

Model interpretability ensures that the outputs of AI/ML models are understandable to security analysts, facilitating effective incident response.

Applying the Model to the Data

For training purposes, many methods exist to apply AI models to log data, with the common goal of processing local or distributed data while addressing security and privacy concerns. Each approach to training the model employs a distinct technical strategy and entails its trade-offs in terms of security, privacy, efficiency, communication overhead, and implementation complexity. The specific approach to train a model or models on log data is not primarily about simply applying a model to the data (which is what you typically do during inference). Instead, it’s a training approach where the model is trained on local data or collaboratively across multiple devices or servers that hold local data.

Notable training methods for applying AI models to log data include Centralized Learning, Distributed Learning, Split Learning, Ensemble Learning, Secure Multi-Party Computation (SMPC), Homomorphic Encryption, and Federated Learning.

Each method addresses different challenges, but many share similarities with federated learning, as they all aim to leverage distributed data or maintain privacy during the training process. However, they differ in how they distribute computations, manage data, and handle privacy. By keeping log data on local servers and only sharing model updates, federated learning offers a practical and secure approach to harnessing the power of AI across distributed systems. This method enhances privacy and security and efficiently uses computational resources across the network.

Federated learning represents a shift in how machine learning models are trained, particularly for applications like log data analysis, where data sensitivity and distribution are significant concerns. Centralized and distributed learning focuses on efficiency and performance when data is fully accessible. At the same time, split learning, ensemble methods, and privacy-preserving techniques like SMPC are more suited to environments where data sensitivity and regulatory compliance are paramount.
Here’s the key distinction:

• Traditional Model Application (Inference): You take a pre-trained model and apply it to new data to get predictions or classifications.
• Federated Learning: The focus is on training the model in a distributed manner. Each node (like an edge device or server) uses its local data (e.g., log data) to compute updates (such as gradients). These updates are then sent to a central aggregator, which combines them (often through methods like Federated Averaging) to update the global model. This process continues iteratively, ensuring that the model benefits from all the distributed data without the data ever leaving its local environment.

Taking a model to an aggregated log database can offer significant advantages for pattern recognition independent of training approaches. Aggregating log data itself has distinct benefits. It creates a unified, rich dataset that significantly enhances pattern recognition, correlation, anomaly detection, and overall system insight, thereby supporting more effective monitoring and maintenance.

Four ways to apply a machine-learning model to log data are:

• Anomaly Detection: Train a model to identify unusual patterns or outliers in log entries. This can help quickly detect security breaches, system errors, or performance bottlenecks without requiring manual inspection of every log line.
• Event Categorization and Root Cause Analysis: Categorize log entries into different types (e.g., errors, warnings, and informational messages). This helps organize the data but also aids in pinpointing recurring issues, enabling faster diagnosis and remediation.
• Predictive Analytics and Trend Forecasting: Apply models that analyze historical log data to forecast future events, such as system failures or performance degradations. This proactive approach enables preventive maintenance and more effective resource planning by predicting issues before they escalate.
• Automated Alert Prioritization: Train ML models to assess the severity and impact of log events and automatically prioritize alerts. This approach reduces alert fatigue and ensures a timely response to critical issues by learning from historical incidents and contextual signals.

Pattern Recognition and Alignment with Zero Trust

Log data contains information about who, what, when, and where systems, applications, and data were accessed in an organization. This information serves as input to the second step of the 5-Step Zero Trust implementation process: Mapping the Transaction Flows. The subset of access logs that pertain to the protect surface identified in Step 1 can be analyzed using AI/ML for access patterns.

Some of the outcomes of the log analysis are:

• The visibility to access patterns of sensitive data and movement within the organization helps detect compromises and data breaches.
• Visibility of users and user groups accessing sensitive data enables the development of zero-trust access policies that protect the data. This also includes cases where an overly permissive policy must be modified to detect insider threats.
• Visibility of sensitive data that has not been accessed for an extended period. Then, to prevent future compromise, the data can be archived or deleted to prevent any compromise of stale data.
• Visibility to second-order data movements that can indicate data breaches. For example, an authorised user may download sensitive data on their endpoint, which can then be exfiltrated outside the organization.

Visibility, Analytics & Automation, and Orchestration

One of the Zero Trust principles is to log everything, as logs are a source of information. However, not all events may be useful, as not all “happenings”/actions that are recorded are relevant. Additionally, as actions continue to unfold, the velocity of event generation is substantial. Thus, to get the logs to work for us, tackling the not-so-very-important events and their velocity is essential.

Logs provide critical information about the workings of components. Aggregated log sources are an ideal data lake for gaining feedback on ineffective policies, authentication failures, and brute force attacks, among other issues. From an automation and orchestration perspective, events are an ideal source of information for triggering policy changes, addressing configuration anomalies, and more.

The following figure illustrates one example of integrating Visibility and Analytics (V\&A) with Automation and Orchestration (A\&O).

Figure 3: Closed-Loop V\&A and A\&O Integration for Zero Trust IAM

Figure 4: Multi-Source Log Aggregation in V\&A/A\&O Feedback Loop

A user attempts to access a resource for which they do not have permission, as per the applied policy. If the user successfully accesses the resource, the action generates an event, which is then logged. Now, if an automation and orchestration engine were consuming the events, it would have a playbook entry that would lock the user out of the resource, thus limiting compromise.
To elaborate on the above scenario, unauthorized access can occur if the actor manipulates the access mechanisms, such as assuming roles in AWS or a Pass-the-Hash attack in a traditional Windows environment. The access request is granted because the ticket is valid, but the automation and orchestration engine can analyze the implicit trust and terminate the access.

This level of integration would depend on several factors, including technological and operational considerations. For example, response latency may be introduced if the logs aggregate many events, and the course correction would not be real-time, but it would be near real-time.

Challenges and Potential Solutions

Within a given timeframe, business solutions manage multiple business transactions across various components. The velocity of the transactions and the underlying component functioning funnel the velocity of log generation and the volume. The formats of the logs also vary.

When all the components’ logs are aggregated in a log database, the formats make it challenging to parse and identify the underlying messages.

There are two aspects here:

• One is that a lot of information on the “happenings”/actions, along with their metadata, is aggregated in the log database
• The second aspect is that while SIEMs and Extended Detection and Responses (XDRs) have AI capabilities, SOC analysts still need to understand the organization and the threats that apply so that they can write queries that help detect incidents, potential incidents, materialized threats, and threats that may potentially materialize.

Potential solutions for specific use cases:

1. A log database consumes application logs in JSON format and browser logs in syslog format. To identify the materialization of any threat, such as an SQL injection, the SOC team would need to correlate the relevant fields from the application and the browser to locate the origin of the SQL injection. This action would be challenging because the field names differ between JSON and Syslog formats. A potential solution would be to use a common information model (CIM) which can convert the JSON and syslog into common formats such that events from both logs can be correlated. The SOC team can use AI models for pattern recognition and write queries to create alerts in real/near-real time.
2. During a person-in-the-browser attack, multiple connections from the malicious site are established with the browser. It would be challenging to manually identify the velocity of attacks to locate the threat’s origin and determine whether it is genuine, a task that can be facilitated with AI models.

Business Benefits

Using AI to analyze log data can transform the way businesses operate. It helps run things more smoothly, strengthens your security, keeps strategy aligned with goals, and makes managing governance, risk, and compliance easier.

Operations

AI can simplify operations management by automating tasks that usually take manual time and effort. This can reduce costs and enable organizations to respond more quickly to issues.

• More Efficient and Cost-Effective: AI accelerates log analysis and reduces the manual work required. This frees up a team’s time and helps allocate resources where needed most.
• Handles More Data, Faster: As data grows, AI can keep pace with the volume, analyzing vast amounts of information in real-time. This helps things run more efficiently and ensures nothing important is missed.
• Integration with Existing Systems: AI-powered security solutions can integrate with existing systems, thereby reducing complexity and enhancing the overall security posture.
• Quick Incident Response: AI can detect problems in real / near-real time, enabling it to address them before they escalate. It also helps quickly track down the cause of issues during investigations.
• Learning Over Time: AI systems become smarter with each iteration, learning from past experiences. This helps reduce false alarms, allowing teams to focus on real threats rather than getting alert fatigue.

Security

AI enhances security efforts by rapidly identifying threats and forecasting future risks. It allows proactive measures to prevent events such as cyberattacks or data breaches.

• Looking Ahead with Predictive Analytics: AI doesn’t just examine current events; it also analyzes past data to anticipate future security issues. This helps identify and address risks early on, rather than reacting after the fact.
• Connecting the Dots: AI pulls data from various sources, making it easier to see the big picture and identify complex security threats that might otherwise go unnoticed.

Governance, Risk, and Compliance (GRC)

AI streamlines risk and compliance management, enabling organizations to stay ahead of regulations and policies.

• Stay Compliant: AI can automatically monitor data to ensure it adheres to all necessary rules and regulations, such as GDPR or HIPAA.
• Manage Risks Better: AI can detect patterns in log and operational data, identifying potential risks or violations before they escalate. It can also help monitor and flag anomalies, enabling proactive incident response.
• Generate Audit Trails: AI streamlines the creation of audit trails and compliance reports, providing a clear record to demonstrate compliance with regulations. This can also boost transparency and accountability within the organization.
• Enforce Policies Consistently: AI ensures that security policies are followed consistently and can alert if something isn’t right. It can even suggest improvements based on the data it processes.

Conclusion

Integrating Artificial Intelligence (AI) into log analysis marks a shift in how organizations manage operational efficiency, security, and strategic alignment. By automating the traditionally labor-intensive task of analyzing log data, AI reduces the cognitive load on security teams while improving their ability to detect and mitigate threats. AI’s ability to identify patterns, detect anomalies, and predict risks enables organizations to strengthen their Zero Trust architecture by continuously validating access and dynamically adapting defenses to changing threats.

AI-driven log analysis enhances decision-making by facilitating faster incident response, providing actionable insights, and optimizing resource allocation. Organizations gain comprehensive visibility across their ecosystems, helping reduce downtime, proactively handle threats, and enhance long-term security posture. AI’s role extends beyond technical improvements to providing strategic benefits, such as regulatory compliance, enhanced governance, and supporting business-aligned security strategies.

Despite its benefits, successful AI adoption involves addressing challenges, such as large log volumes, diverse data formats, false positives, and evolving threat landscapes. Organizations that invest in developing scalable AI solutions and maintaining updated models can unlock the potential of AI-driven log analysis.

Future Outlook

Federated Learning for Privacy-Enhanced Collaboration
Federated learning will allow organizations to collaboratively train threat detection models across distributed systems while keeping sensitive log data localized, enhancing security and data privacy.

AI-Powered Threat Hunting and Predictive Analytics
AI models will leverage real-time behavioral analytics and historical data to identify emerging threats proactively and anticipate future risks, thereby enhancing proactive defense measures.

Context-Aware Event Correlation
Future AI systems will integrate business context into log analysis, allowing for more precise identification of security incidents by correlating technical events with business-critical data and processes.

Automated Incident Response through AI-Orchestration Over Multiple Assets
AI will integrate seamlessly with Security Orchestration, Automation, and Response (SOAR) platforms, enabling faster, autonomous responses to mitigate threats before they escalate. User and identity behavior analysis can be performed across multiple log sources and assets, such as SSO systems, cloud platforms, and endpoint devices, providing a comprehensive view of user actions and enabling the accurate determination of their intent or context.

Glossary

See the CSA Glossary.

Useful References

NSTAC

• NSTAC Report to the President on Zero Trust and Trusted Identity Management

John Kindervag, Circle recordings and documentation

• Zero Trust Implementation and Guiding Principles Briefing by John Kindervag (Passcode: ZTimplement101!)
• ZT Data Protection and Privacy Briefing by John Kindervag (Passcode: DataPillar7!)

Palo Alto Networks

• The Zero Trust Learning Curve: Deploying Zero Trust One Step at a Time

CSA Zero Trust Advancement Center

• Zero Trust Advancement Center
• CSA Road To RSA Webinar Series - From Alert to Remediation: Millions of Alerts Analyzed to Shape Your Strategy

NIST

• NIST SP 800-207 Zero Trust Architecture
• NIST SP 800-207A A Zero Trust Architecture Model for Access Control in Cloud-Native Applications in Multi-Cloud Environments
• Discussion Draft of the NIST Cybersecurity Framework 2.0 Core
• NIST SP 1800-35 Implementing a Zero Trust Architecture, figure 1, page 54, 2nd preliminary draft

DoD Reference Architecture & Strategy

• Department of Defense (DoD) Zero Trust Reference Architecture Version 2.0
• DoD Zero Trust Strategy

CISA Maturity Model V2

• CISA Zero Trust Maturity Model Version 2.0

NSA
CIS: Advancing Zero Trust Maturity Throughout the Network and Environment Pillar
Splunk Blogs, Log Monitoring with AI: What Makes Monitoring Intelligent? April 2024, Accessed September 2024: Log Monitoring with AI: What Makes Monitoring Intelligent? | Splunk

Observo.ai, Observability 101: Why Log Management is Crucial for Business Success, Accessed September 2024: Observability 101: Why Log Management is Crucial for Business Success

NIST, Special Publication 800-92, Guide to Computer Security Log Management, September 2006:
SP 800-92, Guide to Computer Security Log Management | CSRC

NIST Special Publication 800-53, Revision 5, Security and Privacy Controls for Information Systems and Organizations, September 2020:
Security and Privacy Controls for Information Systems and Organizations

Rapid7, The CIS Critical Security Controls Explained - Control 6: Maintenance, Monitoring and Analysis of Audit Logs, Original April 2017, Updated March 2024: [The CIS Critical Security Controls Explained - Control 6: Maintenance, Monitoring and Analysis of Audit Logs

Rapid7 Blog](https://www.rapid7.com/blog/post/2017/04/21/the-cis-critical-security-controls-explained-control-6-maintenance-monitoring-and-analysis-of-audit-logs/)

ISMS.online, ISO 27002:2022, Control 8.15 – Logging, Revised Controls: [ISO 27002:2022, Control 8.15, Logging

ISMS.online](https://www.isms.online/iso-27002/control-8-15-logging/)

A Survey of Machine Learning Techniques for Anomaly Detection in Cybersecurity, October 2023, International Journal of Research in Engineering and Science 11(10):183-193, 11(10):183-193, Authors: Sayeth Saabith, Eastern University, Sri Lanka, Vinothraj Thangarajah, Eastern University, Sri Lanka: A survey of machine learning techniques for anomaly detection in cybersecurity

Automated Log Analysis and Anomaly Detection Using Machine Learning, October 2022, DOI:10.3233/FAIA220378, License CC BY-NC 4.0, Authors: Ali Hussain SHAHa, Daem PASHA a, Esmaeil Habib ZADEH a and Savas KONUR: (PDF) Automated Log Analysis and Anomaly Detection Using Machine Learning

Department of Homeland Security, Zero Trust Implementation Strategy, October 2023:
Department of Homeland Security Zero Trust Implementation Strategy

National Security Agency

Cybersecurity Information Sheet, Advancing Zero Trust Maturity Throughout the Visibility and Analytics Pillar, May 2024: Advancing Zero Trust Maturity Throughout the Visibility and Analytics Pillar

Appendix 1: AI/ML Tools and Model Providers

1. Elastic Stack (with Machine Learning Features)
   • Integrates machine learning into Elasticsearch for anomaly detection and insights from time-series log data.
   • Suitable for scalable log analysis with customizable ML pipelines.
2. Splunk Machine Learning Toolkit
   • Allows machine learning algorithms to be applied to logs directly in Splunk.
   • Enables predictive analytics, clustering, and anomaly detection.
3. IBM QRadar Advisor with Watson
   • Combines Watson’s cognitive capabilities with QRadar SIEM to detect and prioritize threats.
   • Enhances log analysis with machine learning and natural language processing.
4. Microsoft Azure Sentinel
   • Cloud-native SIEM with integrated AI/ML for large-scale log analysis.
   • Offers advanced anomaly detection and automated threat response.
5. Google Cloud Chronicle
   • Cloud-based platform with machine learning for log storage, threat hunting, and anomaly detection.
   • Leverages Google’s infrastructure for scalability and speed.
6. Anomaly Detection Algorithms (General)
   • Isolation Forest: Unsupervised learning for identifying outliers in log data.
   • Autoencoders (Deep Learning): Detects anomalies in complex datasets by learning compact representations.
7. Securonix Next-Gen SIEM
   • Applies advanced analytics, machine learning, and behavior analysis to detect threats in log data.
   • Focused on user and entity behavior analytics (UEBA).
8. Exabeam Advanced Analytics
   • Leverages behavioral analytics to detect anomalies in user and entity behavior.
   • It is ideal for forensic analysis and reducing alert fatigue in SOC environments.
9. Hugging Face Transformers
   • Adapts NLP models for text-heavy log analysis, such as categorization and anomaly detection.
   • Useful for extracting entities and understanding unstructured log data.
10. Posit
    • Enables statistical analysis and custom machine learning pipelines for log data
    • Supports exploratory analysis, visualization, and advanced model development.

Appendix 2: SIEM and SOAR Providers

1. Splunk
   • SIEM: Splunk Enterprise Security—Data analytics, real-time monitoring, incident response, advanced threat detection.
   • SOAR: Splunk Phantom—Automates threat detection and response and integrates with Splunk products.
2. IBM
   • SIEM: IBM QRadar—Threat detection, incident response, machine learning for prioritizing threats.
   • SOAR: IBM Resilient—Automation, orchestration, streamlined incident response workflows.
3. LogRhythm
   • SIEM: LogRhythm NextGen SIEM—Advanced analytics, log management, threat identification, and incident management.
4. McAfee
   • SIEM: McAfee Enterprise Security Manager (ESM)—Security analytics with real-time monitoring that integrates with McAfee products.
5. ArcSight (Micro Focus)
   • SIEM: ArcSight ESM—Threat detection, response, compliance management, security tool integration.
6. Rapid7
   • SIEM: Rapid7 InsightIDR—Threat detection and response with machine learning and analytics.
   • SOAR: Rapid7 InsightConnect—Automation and orchestration for security operations.
7. Securonix
   • SIEM: Securonix Next-Gen SIEM—Behavior analysis, machine learning, and advanced analytics for threat detection.
8. Exabeam
   • SIEM: Exabeam Advanced Analytics—Behavior analytics (UEBA), forensic analysis, and threat response.
9. Elastic
   • SIEM: Elastic SIEM (part of Elastic Stack)—Scalable real-time visibility and security event analysis using Elasticsearch.
10. Palo Alto Networks
    • SOAR: Cortex XSOAR—Security orchestration, incident management, and automation.
11. Hunters
    • SIEM: Hunters SOC Platform—Cloud-native SIEM alternative with automated threat detection, investigation, and response. Integrates with modern data lakes, emphasizes detection-as-code, and supports scalable SOC operations.

Appendix 3: Logs Generated by Zero Trust Pillars

When Zero Trust recommends logging everything, every pillar is responsible for logging events from its perspective. A transaction would encompass multiple pillars, and each pillar and its respective components would generate logs. This is important; the components should not assume the other component will generate the required log. While extremely useful for analyzing events, identifying threats and vulnerabilities, and predicting attacks, this model generates a significant amount of data.

The following are the types of events that are generally captured in an environment from each pillar’s perspective.

Identity

• Successful and failed login attempts, including timestamp, user ID, source IP, and device.
• MFA challenges: success/failure, the method used, and unusual activities.
• SSO events: identity provider used, services accessed, and errors.
• Password changes, resets, expirations, and failed attempts.

Device

• Device registration and enrollment details, including type, OS, identifiers, enrollment agent, and enrollment date.
• Device ownership assignments and changes: user identity, department, roles, and timestamp.
• Security posture assessments: compliance check results, successful or failed.
• Configuration changes: OS updates, security software, and network settings.

Data

• Data access events: who accessed the data, what data was accessed, when, and the method used.
• Data creation, modification, deletion: who, what, when, and the method used.
• Data transfers across systems: source, destination, type, encryption status, and errors.
• Changes in data classification: who authorized it, when, and why.

Network

• Network access events: user identity, source IP, destination, device used, timestamp, and authentication method.
• Firewall activity, including rule changes, policy updates, alerts, and blocked/allowed traffic.
• VPN connections: session start and end times, duration, IP addresses, and authentication methods.
• Network segmentation activity: inter-segment access attempts, policy changes, enforcement decisions, and outcomes (allowed/blocked).

Application

• Application access: who accessed, when, from where, and method used (web, mobile, API).
• User actions within applications: login, logout, and CRUD operations.
• Application authentication events, including success and failure, as well as MFA challenges pertinent to the application.
• API access and usage, including API calls, who made them, data accessed, rate limits, and anomalies.

Appendix 4: A Brief History of Logging in Computing

Logging in computing has a rich history that dates back to the early days of computing. Let us examine each decade, from the advent of mainframe computers to the present era of Zero Trust and Generative Artificial Intelligence.

1950s - Mainframe Computers

Logging began with mainframe computers, where logs were rudimentary. Log files were printed on paper and manually reviewed.

1960s - Operating Systems

Operating systems like IBM’s OS/360 introduced more sophisticated logging mechanisms. Logs included system events, errors, and job statuses, aiding in system management and troubleshooting.

1970s - UNIX and System Logs

UNIX’s development introduced system logs (Syslog), which use simple text files to store log data. The Syslog system, developed in the 1970s, became a standard for logging in UNIX and UNIX-like systems.

1980s - Networked Systems

The 1980s witnessed significant growth in computing, with networking becoming increasingly prevalent, logging expanding to encompass network events, and Simple Network Management Protocol (SNMP) gaining popularity in network management.

1990s - Distributed Systems and Web Servers

With the growth of distributed systems and the Internet, logging became crucial for web servers and applications. Windows NT, released in 1993, introduced the Event Log system, which provided a standardized way to log system, application, and security events.

2000s - Centralized Logging and Log Management

The 2000s were a significant decade in the evolution of computing and logging. Security Information and Event Management (SIEM) solutions were developed as the Internet matured. Centralized logging solutions emerged, allowing logs from multiple sources to be aggregated and analyzed in a single location.

2010s - Cloud Computing and Big Data

The advent of cloud computing introduced new challenges and solutions for logging, with cloud providers offering integrated logging services. Big Data technologies enabled the processing and analysis of large volumes of log data, providing deeper insights and real-time monitoring.

2020s - Observability and Advanced Analytics

The 2020s have witnessed a transformation in logging driven by advancements in AI, machine learning, and the increasing adoption of Zero Trust architectures. Observability has become a core concept, combining logging, metrics, and traces to provide a holistic view of system performance and security. Generative AI is integrated into log analysis, enabling automated anomaly detection, threat hunting, and the generation of predictive insights. The rise of edge computing, IoT, and 5G networks has also necessitated new approaches to distributed logging and security. Zero Trust principles emphasize continuous authentication and authorization, leveraging advanced logging to monitor and enforce granular access controls. As organizations adopt multi-cloud and hybrid environments, logging solutions have evolved to ensure seamless integration, scalability, and real-time insights.

Appendix 5: Log Sources

Logs originate from various systems and devices, each contributing specific data relevant to its function. The following tables categorize different log sources based on their function and use.

The first table highlights foundational logging sources, which are the core components of traditional security monitoring. These sources provide essential visibility into system and network activity, enabling organizations to detect threats, investigate incidents, enforce security policies, and maintain operational integrity. Capturing events related to access, authentication, system interactions, and user behavior enables the detection of anomalies, prevention of unauthorized activities, and support for compliance with security frameworks.

Traditional Logging Sources and Their Uses

Source	Detail Log Source	Use
Network Devices	Firewalls	Firewall logs display traffic as either allowed or denied based on established security policies. They also contain critical information about inbound and outbound network activity.
	Routers and Switches	Network devices’ logs. Related to network performance, traffic routing, and potential unauthorized access attempts.
	IDS/IPS	Logs capture network-based attacks and unauthorized access attempts related to suspicious activity.
Endpoint Devices	Antivirus/Antimalware	Logs track the detection, quarantine, or removal of malicious files on individual machines.
	Endpoint Detection and Response (EDR)	EDR logs provide detailed visibility into endpoint activities, including process execution, file modification, and network connection.
Operating Systems	Windows Event Logs	This includes security, application, and system event logs, which detail user logins, system performance, and security incidents.
	Linux/Unix Syslogs	Detailed file access, system errors, and user activity.
Applications	Web Servers	Requests, responses, errors, and other interactions with users or services.
	Databases	Track changes to query execution in databases and access attempts.
Cloud Services	Cloud infrastructure logs	Logs from cloud infrastructure that track user activity, configuration changes, and API interactions.
Identity and Access Management	Authentication logs	Authentication logs from identity providers or internal systems tracking login attempts, privilege escalations, or account lockouts.
Security Tools	AV, endpoint firewalls	Logs that detect suspicious activity and provide context for incident response.

The second table focuses on advanced AI/ML-enhanced logging, incorporating unstructured data, behavioral analytics, and external intelligence to improve threat detection and response. These sources offer a deeper, more contextualized view of security events, utilizing machine learning to identify patterns, anomalies, and emerging threats that traditional logs may overlook.

Advanced (AI/ML) - Logging Sources and Their Uses

Source	Detail Log Source	Use
Traditional Security Logs	Same as SIEM	It uses the same data sources as SIEMs, i.e., firewall logs, endpoint protection logs, and system event logs.
Unstructured Data	User behavior, social media	User activity, such as browsing history, email content, and document access patterns, provides insight into abnormal user behavior.
Network Traffic Logs and Packets D	Packet headers, payloads	Detects anomalies using raw network traffic data, including packet headers and payloads. This can identify lateral movement or data exfiltration.
Machine and Sensor Data	IoT devices, ICS data	Data from IoT devices or industrial control systems (ICS), including environmental sensor data and operational activity, can be used.
Application Logs and API Data	Application interactions	Extracts detailed insights from application interactions, session activity, API calls, and metadata.
User Activity Monitoring	Behavioral biometrics	Monitors subtle behaviors, such as typing patterns (keystrokes) or mouse movements. This helps detect unauthorized access or insider threats.
Behavioral Data and Risk Analytics	Risk scores, anomaly detection	Calculates risk scores from behavioral biometrics. This helps identify users’ bad behavior, providing deeper insights into potential threats.
Threat Intelligence Feeds	External threat intelligence feeds	Utilizes threat intelligence feeds to identify emerging threats, including global attacks, IP deny lists, and zero-day vulnerabilities.

Appendix 6: Details and Formats of Events

Severity Levels of Events

Not all events require immediate attention. To help prioritize responses, each event is assigned a severity level, which categorizes its criticality. For instance:

• Critical events might indicate system failures or security breaches.
• Informational events might simply log routine activities for monitoring purposes.

By categorizing events based on severity, systems can ensure that urgent issues are addressed promptly while routine operations are logged for record-keeping and analysis.

The following table provides an example of severity levels based on the Syslog protocol (RFC 5424):

Numerical Code	Severity	Description
0	Emergency	The system is unusable.
1	Alert	Action must be taken immediately.
2	Critical	Critical conditions.
3	Error	Error conditions.
4	Warning	Warning conditions.
5	Notice	Normal but significant conditions.
6	Informational	Informational messages.
7	Debug	Debug-level messages.

Reference: Syslog Protocol (RFC 5424)

Event Formats

Common Event Format (CEF)

The CEF is a log management standard defining a common security event log format. It is used to collect and analyze security events from various sources in a standardized way.

The CEF format is relevant to Zero Trust because it provides a standardized method for collecting and analyzing security events from various sources, which is essential for implementing a Zero Trust architecture. By using CEF, organizations can collect and analyze security events from various sources in a standardized way, which helps to identify potential security threats and vulnerabilities.

JSON

Logs in JSON format are events recorded as JSON objects in files. Such logs are easy to read, parse, analyze, and understand. JSON formats are not just an IT log feature, but are also used in utilities such as logs from wells and drills. The JSON Well Log Format explains how the JSON format can be utilized to capture information from wells and drills.

Windows Event format

EVTX files are proprietary binary file formats provided by Microsoft. The files can be viewed in XML format.

NCSA Common Log Format (CLF) for Session Initiation Protocol (SIP)

The Common Log Format for Session Initiation Protocol, also known as the NCSA Common Log Format, is described in RFC 6872. Generally used by web servers and web proxies, the Common Log Format is a standardized text file format. The CLF format, as described in the RFC, is based on Apache HTTPd’s common log format.

W3C Extended Log File Format

The W3C Extended Log File Format is an extension of the Common Log Format and is described in the W3C Working Draft. The extended log file contains a sequence of lines consisting of ASCII characters, terminated by either the LF or CRLF.

The Syslog format

The syslog format described in RFC5424 describes the standard format for syslog messages and
outlines the concept of transport mappings. It also describes structured data elements, which can transmit easily parseable, structured information and allow vendor extensions.

CEF and Syslog are particularly well-suited for Zero Trust due to their standardization and compatibility with diverse systems, while JSON adds flexibility for modern and nontraditional use cases. From an AI perspective, the key requirement is not the format itself but the quality, structure, and richness of the data within the logs. Standardized formats help streamline AI integration by reducing preprocessing overhead and ensuring consistent data for analysis.

Aggregation and Storage of Logs

In the sections above, we explored the source, structure, and formats of logs. Log storage involves aggregating data from distributed sources into centralized systems, utilizing technologies that support efficient storage, accessibility, and processing for analytics and compliance purposes. Organizations require storage solutions that can accommodate their scale, complexity, and diversity to utilize logs effectively.

Log Source

As detailed in the previous sections, logs are generated by various sources, including IT systems, OT environments, IoT devices, network components, and applications. These logs capture events and activities that provide visibility into system performance, security incidents, and operational behaviors.

Intermediate Log Storage

Before logs are aggregated into a centralized system, they often pass through an intermediate storage layer. This may be for a common information model where logs from different formats must be consolidated into a single format. This step ensures the reliability, scalability, and efficiency of log collection in environments with diverse sources and high data velocity. Intermediate storage acts as a buffer, allowing logs to be pre-processed, enriched, and temporarily stored before final aggregation.

Purpose of Intermediate Storage

Purpose	Description
Reliability	Prevents data loss during network outages or pipeline disruptions by temporarily storing logs.
Pre-Processing	Before sending them to central storage, filters normalize and enrich logs (e.g., adding metadata).
Scalability	Decouples log generation from aggregation, enabling systems to handle bursts in log volume effectively.
Latency Management	Optimizes log transmission by batching data or forwarding logs in near real-time to reduce delays.

Technologies for Intermediate Storage

Technology	Description	Example Tools
Message Queues	High-throughput, distributed systems for buffering and reliable delivery of log data.	Apache Kafka, RabbitMQ, Amazon SQS
Log Shippers with Buffers	Local storage options in log shippers allow temporary storage when downstream systems are unavailable.	Fluentd, Logstash, Beats
Edge Storage	Local storage on edge devices or gateways reduces dependency on continuous connectivity.	AWS IoT Greengrass, Azure IoT Edge
Temporary File Storage	Logs are stored as flat files on local disks and organized by time or source for later forwarding.	Custom File-Based Solutions

Appendix 7: Aggregation and Security of Logs

Logs from intermediate storage systems are transmitted to aggregation systems through:

• Direct Integration: Shippers like Fluentd and Logstash forward logs to platforms like Elasticsearch, Splunk, or cloud-based log services.
• Streaming Protocols: Tools like Kafka Streams enable continuous data flow into centralized systems.
• Batch Processing: Logs are sent in batches during non-peak times to optimize resource usage.

Intermediate storage is crucial in the log management pipeline. It ensures seamless data flow between diverse sources and centralized storage or analysis systems. This stage enhances reliability and scalability, laying the groundwork for effective aggregation and long-term log retention.

Log Aggregation

Log aggregation consolidates logs from diverse sources into a central system for unified analysis and management. This step is crucial for ensuring that data from various systems, such as IT infrastructure, OT devices, IoT systems, and applications, can be effectively correlated and analyzed.

The purposes of log aggregation are:

• Unified Analysis: Aggregating logs enables organizations to analyze data holistically, correlating events across systems to identify patterns and potential threats.
• Normalization and Standardization: Aggregation tools transform logs into consistent formats, ensuring compatibility and easier querying.
• Simplified Management: A centralized aggregation system reduces the complexity of accessing and analyzing logs from disparate sources.

Log Aggregation Workflows

1. Direct Aggregation
   • Logs are sent directly from the source to the aggregation system, bypassing intermediate storage.
   • This approach is common in environments with reliable connectivity and manageable log velocity.
   • Tools like Fluentd, Logstash, and Beats collect logs and forward them to a central repository or analysis platform.
2. Aggregation After Intermediate Storage
   • Logs are buffered or pre-processed in intermediate storage before being forwarded to the aggregation system.
   • This workflow benefits high-velocity environments or distributed systems where temporary storage enhances reliability and scalability.

Log Aggregation Technologies

Technology	Description	Key Tools	Destination (Aggregation Systems)
Log Shippers	Fluentd collects logs from diverse sources (e.g., Kubernetes pods, servers, containers), performs initial processing (e.g., formatting, routing), and forwards them to Logstash. Logstash then preprocesses and enriches the logs (e.g., parsing fields, adding metadata) before sending them to platforms like Elastic Stack or Splunk.	Fluentd Logstash	Elastic Stack Splunk Graylog
API Integrations	Applications or services capture logs and transmit them directly to aggregation platforms. For example, the CloudWatch Logs API collects logs from AWS services and sends them to CloudWatch Logs for querying. Similarly, the Splunk HEC allows custom applications to send structured event data directly to Splunk for analysis.	CloudWatch Logs API Splunk HEC	CloudWatch Logs Splunk Amazon OpenSearch Amazon S3
Streaming Platforms	Logs are streamed in real time from distributed sources to aggregation systems. For example, Apache Kafka collects logs from microservices and forwards them to Elastic Stack or Splunk. At the same time, Amazon Kinesis streams logs to destinations like CloudWatch Logs, S3, or OpenSearch for storage and further processing.	Apache Kafka Amazon Kinesis	Elastic Stack Splunk CloudWatch Logs Amazon S3 Amazon OpenSearch

Steps in Log Aggregation

1. Log Collection: Logs are gathered from sources using agents, shippers, or API integrations.
2. Pre-Processing: Logs may be filtered, enriched, or normalized during collection to reduce noise and improve analysis readiness.
3. Transmission: Logs are sent to the aggregation system in real-time or batches, depending on system requirements.
4. Normalization: Aggregation tools standardize logs into consistent formats, making them easier to query and analyze.
5. Indexing: Logs are indexed in the aggregation system to enable efficient search and retrieval.

Integration with Centralized Log Storage

Log aggregation prepares data for long-term storage and deeper analysis. Once aggregated, logs are typically sent to centralized storage platforms, where they can be retained for compliance, forensics, or advanced analytics. This transition ensures that aggregated logs are not only accessible but also secure and organized for future use.

By consolidating logs into a unified system, organizations enhance their ability to monitor, analyze, and respond to events across their environments. The next section explores how centralized storage systems retain and manage these aggregated logs.

Centralized Log Storage

Once logs have been aggregated, they are stored in centralized systems that support long-term retention, efficient querying, and advanced analytics. Centralized storage ensures that logs remain accessible and organized for real-time monitoring, compliance reporting, and forensic investigations.

Purpose of Centralized Storage

1. Long-Term Retention
   • Logs are retained for regulatory compliance, historical analysis, and audit purposes.
   • Retention policies vary based on business needs and industry standards, ranging from a few months to several years.
2. Real-Time Querying
   • Centralized storage enables fast searches and correlation of events across large datasets.
   • Supports Security Operations Center (SOC) teams in identifying and responding to threats quickly.
3. Advanced Analytics
   • Enables integration with AI/ML tools for anomaly detection, behavior analysis, and predictive analytics.
   • Facilitates root cause analysis and optimization of operational processes.

Storage Technologies

Technology	Description	Examples
Databases	Structured storage for logs, allowing SQL-based queries and relational data modeling.	MySQL, PostgreSQL, MongoDB
Log Indexing Systems	Index and store logs for fast, full-text searches and analytics.	Elasticsearch, OpenSearch
Cloud-Based Storage	Scalable and elastic storage for logs, often integrated with native analytics and monitoring tools.	AWS CloudWatch, Azure Monitor, Google Cloud Logging
Data Lakes	Store unstructured or semi-structured logs in their raw format for large-scale analysis and archival purposes.	Hadoop, Amazon S3, Azure Data Lake

Key Features of Centralized Storage

1. Scalability
   • Centralized systems can handle growing log volumes, supporting high-throughput environments like IoT or multicloud architectures.
2. Data Organization
   • Logs are indexed and categorized, making them easily searchable and accessible.
   • Metadata is often added during aggregation to enhance usability and accessibility.
3. Security and Access Control
   • Centralized storage platforms implement encryption, access control policies, and logging of access attempts to ensure security and compliance.
   • Ensures compliance with data protection regulations such as GDPR and HIPAA.
4. Integration with Monitoring and Analytics Tools
   • Centralized storage integrates with tools like Splunk, Kibana, or custom dashboards for visualization and alerting.
   • Supports automated responses by connecting with orchestration and automation systems.

Workflow for Centralized Storage

1. Ingestion: Aggregated logs are ingested into the centralized system through APIs or shippers.
2. Indexing: Logs are processed and indexed for optimized querying.
3. Access and Querying: Logs are accessed for real-time monitoring, historical analysis, or compliance reporting.
4. Archival: Older logs are moved to lower-cost storage tiers or cold storage for extended retention.
5. Retention Policies: Logs are stored according to predefined retention periods, with older logs archived or compressed as needed.

Benefits of Centralized Storage

• Efficiency: Centralized storage consolidates log data from diverse sources into a single repository, enabling rapid access for operational troubleshooting and security investigations by reducing the time spent locating and retrieving data from disparate systems.
• Compliance: Centralized log storage simplifies adherence to legal and industry regulations by ensuring all required data is retained in a traceable and easily auditable manner, helping organizations demonstrate accountability and meet regulatory obligations.
• Scalability: A centralized solution is designed to handle growing log volumes in dynamic environments, ensuring that storage and processing capabilities can scale as data sources expand without compromising performance.
• Advanced Capabilities: Centralized storage supports seamless integration with AI and machine learning tools, enabling intelligent analysis, automated anomaly detection, and predictive insights to improve security and operational decision-making.

Centralized storage is the final step in the log management pipeline, ensuring that logs are preserved, accessible, and usable for various operational and security needs. It forms the backbone of effective monitoring, analytics, and decision-making in modern IT ecosystems.

Security of Logs

Generation/Source: Log Creation by Systems and Devices

• Threats:
  • Logs could be intercepted during generation.
  • Insider threats may manipulate log creation mechanisms.
  • Misconfigured logging settings could expose sensitive information.
• Security Measures:
  • Enable secure logging protocols (e.g., TLS) to encrypt logs at the point of generation.
  • Authenticate and authorize log sources to ensure only legitimate devices generate logs.
  • Configure logs to exclude sensitive data, such as PII, where unnecessary.
  • Regularly audit and validate log configurations across systems and devices.

Intermediary Storage: Temporary Buffering and Pre-Processing

• Threats:
  • Logs in transit to intermediary storage may be intercepted or altered.
  • Buffer overflow attacks could exploit storage limitations.
  • Insufficient security controls may expose data stored temporarily.
• Security Measures:
  • Use encrypted communication channels (e.g., TLS or VPN) between sources and intermediate storage.
  • Secure intermediary storage with access controls and encryption.
  • Implement monitoring and alerts for anomalous activity in temporary storage.
  • Ensure data remains unaltered by applying integrity checks (e.g., hashing) during log buffering.

Aggregation: Consolidation and Normalization of Logs

• Threats:
  • Compromise during normalization could lead to altered or incomplete log data.
  • Insider threats may tamper with log aggregation processes.
  • High-volume environments may suffer from log loss or manipulation.
• Security Measures:
  • Authenticate and secure log shippers and aggregation tools to prevent unauthorized use.
  • Ensure integrity validation during and after aggregation using cryptographic methods (e.g., digital signatures).
  • Limit access to aggregation platforms through strict access control policies.
  • Monitor aggregation systems for signs of tampering, high error rates, or unexpected activity.

Utilization: Analysis and Use of Logs for Security and Operational Purposes

• Threats:
  • Logs accessed during analysis may expose sensitive information.
  • Attackers could target analytics systems to corrupt data or inject false patterns.
  • Insider threats could misuse analytics platforms to gain unauthorized insights.
  • Timestamps may be altered to disassociate events from their correlation.
• Security Measures:
  • Ensure analytics platforms implement role-based access control (RBAC) and monitor access logs.
  • Encrypt logs while in use, mainly if analyzed in shared or multi-tenant environments.
  • Utilize secure APIs and tools for querying logs, ensuring access is restricted to authorized users and systems.
  • Apply behavioral monitoring to identify misuse of analytics tools or suspicious queries.
  • Ensure log sources, logs in intermediary storage, and log aggregations are synced in time with a single source of Network Time Protocol (NTP).

Final Storage: Long-Term Retention and Archival of Logs

• Threats:
  • Long-term storage systems may become a target for attackers due to the richness of historical data.
  • Logs could be improperly accessed or deleted due to insufficient access control.
  • Storage misconfigurations can lead to data breaches.
• Security Measures:
  • Encrypt logs at rest using strong encryption standards.
  • Segment and isolate log storage systems from other infrastructure.
  • Implement retention policies and automated deletion mechanisms for expired logs to ensure data integrity and compliance with relevant regulations.
  • Use immutable storage technologies or blockchain-based solutions to prevent tampering.
  • Apply logging to the log management system (e.g., who accessed or modified logs).

Disposal: Secure Destruction After Logs Are No Longer Needed

• Threats:
  • Improper log disposal could result in data leakage.
  • Insider threats could retrieve logs marked for deletion.
• Security Measures:
  • Overwrite or securely erase logs in accordance with industry standards (e.g., NIST SP 800-88 for data sanitization).
  • Audit and verify log destruction processes to ensure compliance with policies.
  • Implement secure destruction workflows to ensure that only authorized personnel can initiate and oversee the disposal process.
  • Use tools that provide certificates of destruction for compliance purposes.

Appendix 8: Applying Model to the Data

Notable training methods for applying AI models to log data include:

• Centralized Learning

  • Description: All log data is aggregated in a central repository before training a model.
  • Benefits:
    • Access to the complete dataset may lead to more robust and comprehensive models.
    • Simpler data management and consistency in training.
  • Considerations:
    • May raise privacy or compliance concerns since sensitive log data is consolidated in a single location.
    • Requires high bandwidth and storage to manage all the aggregated data.

• Distributed Learning

  • Description: The training process is distributed across multiple parallel machines or nodes, each working on a different portion of the data.
  • Benefits:
    • Can reduce training time by leveraging parallel processing.
    • Often used when data is already partitioned across multiple servers.
  • Considerations:
    • Although computation is distributed, the data may still need to be aggregated or shared at some stage.
    • It requires careful coordination to ensure consistency and convergence of the global model. Description: The training process is spread across multiple machines or nodes that work in parallel on different data portions.
  • Benefits:
    • Can reduce training time by leveraging parallel processing.
    • Often used when data is already partitioned across multiple servers.
  • Considerations:
    • Although computation is distributed, the data may still need to be aggregated or shared at some stage.
    • Requires careful coordination to ensure consistency and convergence of the global model.

• Split Learning

  • Description: The neural network is partitioned into segments where the initial layers are trained on the client’s data, and subsequent layers are processed on a central server.
  • Benefits:
    • Allows privacy preservation by keeping raw data local while benefiting from centralized model refinement.
    • Reduces the amount of data that must be transferred between client and server.
  • Considerations:
    • Requires synchronization between client and server during training.
    • Implementation complexity can be higher compared to both centralized and federated approaches.

• Ensemble Learning

  • Description: Separate models are trained independently on different subsets of the log data, and their outputs are combined (e.g., via voting or averaging) to produce the final prediction.
  • Benefits:
    • Increases robustness and reduces overfitting by combining multiple perspectives.
    • Each model can specialize in recognizing specific patterns in different parts of the aggregated logs.
  • Considerations:
    • Ensemble methods can be computationally expensive both during training and inference.
    • Effective aggregation of model predictions is critical for performance.

• Secure Multi-Party Computation (SMPC) and Homomorphic Encryption

  • Description: Techniques that allow multiple parties to jointly compute a function (such as training a model) over their inputs while keeping those inputs private.
  • Benefits:
    • Enhanced security and privacy, as raw log data is never exposed.
    • Suitable for scenarios where strict data confidentiality is required.
  • Considerations:
    • Generally introduces computational overhead and can be slower than traditional training methods.
    • More complex to implement compared to centralized or distributed approaches.

• Federated Learning

  • Description: a decentralized machine learning paradigm that enables training AI models across multiple locations (or “clients”) where the data resides, rather than requiring that all data be centralized.
  • Benefits:
    • Log data often contains sensitive or proprietary information. Federated learning ensures that this data remains on local servers or devices, thereby mitigating risks associated with central data breaches.
    • Instead of transmitting vast quantities of log data to a central repository, only model updates (typically much smaller) must be communicated.
    • In many industries, data privacy laws (such as GDPR) restrict the sharing of user data. Federated learning helps organizations comply with these regulations by keeping the data local.
    • Log data can vary widely in distributed environments (e.g., multiple branches of a company or IoT devices in the field). Federated learning allows each node to contribute unique insights while still building a unified model that benefits from the diversity of its training data.
  • Considerations:
    • Although only model updates are transmitted, the iterative process of federated learning can still involve significant communication, especially in large-scale deployments.
    • The participating nodes might have vastly different computational resources and data volumes. The aggregation mechanism must account for these disparities to avoid biasing the model.
    • Ensuring that malicious nodes do not send harmful updates is an ongoing area of research. Techniques such as robust aggregation and anomaly detection are crucial in this context.
    • With highly non-independent data and identically distributed (non-IID) across clients, achieving global model convergence can be more challenging than in centralized training.

Several key reasons why aggregating log data and then applying a model can be beneficial:

• Comprehensive Context: The model gains a holistic system view when logs from various sources are aggregated. This comprehensive dataset helps identify patterns across different services or components that might be missed when logs are viewed in isolation.
• Enhanced Pattern Recognition: Aggregation increases the volume and diversity of data. This enriched dataset enables the model to identify subtle patterns and correlations—such as sequences of events leading to an anomaly—that may not be apparent when analyzing individual log streams.
• Improved Anomaly Detection: The model can better distinguish between normal and abnormal behavior with more data points. It can also identify rare events or outlier patterns that indicate potential issues, such as security breaches or system failures.
• Efficient Root Cause Analysis: An aggregated log database can help correlate events from multiple sources, making it easier to trace the origin of a problem. This unified view accelerates troubleshooting and helps pinpoint root causes quickly.
• Resource Optimization: Centralizing log data enables more efficient use of computational resources. Instead of running multiple analyses on fragmented data, a single model applied to the aggregated logs can more efficiently process and analyze the entire dataset.

Applying the model to logs can offer significant advantages for pattern recognition independent of specific learning approaches.
Federated learning isn’t primarily about simply applying a model to the data (which you typically do during inference). Instead, it’s a decentralized training approach where the model is trained collaboratively across multiple devices or servers that hold local data.

Here’s the key distinction:

• Traditional Model Application (Inference): You take a pre-trained model and apply it to new data to get predictions or classifications.

• Federated Learning: The focus is on training the model in a distributed manner. Each node (like an edge device or server) uses its local data (e.g., log data) to compute updates (such as gradients). These updates are then sent to a central aggregator, which combines them (often through methods like Federated Averaging) to update the global model. This process continues iteratively, ensuring that the model benefits from all the distributed data without the data ever leaving its local environment.

In summary, federated learning involves training a model on distributed data while preserving privacy and reducing data transfer rather than simply applying the model to the data after it has been trained.

Still, Federated learning can be particularly beneficial for analyzing security logs. It enables multiple organizations or devices to collaboratively improve threat detection models without sharing sensitive log data. This approach enhances threat detection accuracy and maintains the confidentiality of each organization’s data.

Advantages of applying the model to the data (Federated Learning):

• Data Privacy: Raw data is stored on client devices, reducing the risk of data breaches and enhancing user privacy.
• Security: By not transmitting raw data, federated learning mitigates risks associated with data interception during transmission.
• Efficiency: Training is distributed across multiple clients, potentially speeding up the process and leveraging the computational resources of all participants.

Challenges and Considerations

• Communication Overhead: Frequent communication between clients and the central server can introduce latency and bandwidth challenges.
• Heterogeneous Data: Clients may have data with varying distributions, which can impact the model’s performance.
• Privacy and Security: Although federated learning enhances privacy, model updates can still leak sensitive information.
• Model Convergence: Ensuring the model converges to a reasonable solution can be challenging, especially with non-IID (Independent and Identically Distributed) data.