Getty Images
How to set up automated log collection with PowerShell
Gathering logs from on-premises Windows Server systems or Office 365 cloud services is a necessary but tedious job. Try a PowerShell script to ease the pain.
IT is one of the few jobs where you actively go looking for trouble.
Administrators should get into the routine to check logs from both on-premises Windows Server systems and the Office 365 environment to avoid being caught by surprise. Part of the problem is the amount of work involved to gather the logs from disparate locations. But an automated log collection script written in PowerShell can run on a regular basis to keep admins apprised of any concerning developments, no matter where they originate.
Why reviewing security logs should be routine
One key component to a security assessment is whether IT workers or the security team review the logs regularly. In many cases, these logs are only examined when trying to resolve an ongoing issue or attempting to discover how an incident started. This makes the log review process for both the IT team and the end-user slow, time-consuming and often painful.
Over the past few years, cybersecurity events have increased and grown more sophisticated and will often outpace the security tools designed to detect them. The advantage of standard log checks is evidence of identifiable entries can help find the source of an attack. Many organizations that store events within security information and event management (SIEM) platforms still require proactive interaction and execution to resolve specific events. For other organizations that do not use a SIEM platform, there needs to be a mechanism to efficiently retrieve the entries. Many applications utilize different formats and getting the logs can be cumbersome.
Why is on-premises and cloud log collection difficult?
Many organizations have difficulties retrieving log entries from both on-premises and cloud services for several reasons. The most common is the format and authentication. For example, Windows Server event log format differs from audit logs from Office 365 services or even the direct logs for a specific service such as Exchange Online.
Authentication for on-premises log gathering tends to be much easier, whereas the same administrative work for a cloud service requires specific PowerShell modules, credentials and commands. For example, retrieving all entries from the Security event log on a Windows Server, you can use the Event Viewer interface and export as needed. To fetch the log entries for Exchange Online, you need to use PowerShell. First, you need to install and import the required module, connect to the service with the necessary account permissions then run the correct PowerShell command. It becomes much more complicated to gather log entries from multiple on-premises servers or cloud services.
How does PowerShell make it easier to retrieve logs?
Both the on-premises servers and cloud services such as Microsoft 365 support PowerShell to make automated log collection less of an obstacle. For example, you can create a script to query multiple Windows Server logs and another to query Microsoft 365 service logs, then store them into a particular location for deeper inspection. You could then convert these scripts into scheduled tasks for a fully automated method. With Microsoft cloud services, you can schedule PowerShell jobs within Azure to perform these tasks as needed.
How can you retrieve event logs from servers in the data center?
Windows Server features the Event Viewer interface for manual handling of event logs and entries. It is the easiest way to work with log entries. The Event Viewer displays the various locations such as Application, Security, System, as well as application specifics; for example, if you use Active Directory Federated Services (ADFS) on a server, there is a corresponding log that segments the entries for the application.
Within the Event Viewer, you can export the entries into a flat file then import the log into other applications for further querying and inspection.
Another option to export the log is to use PowerShell. A command called Get-WinEvent is designed to retrieve event log entries. For example, the following command retrieves all events from the Security log:
Get-WinEvent -LogName "Security"
To limit the entries returned, set the maximum number to return.
Get-WinEvent -LogName "Security" -MaxEvents 10
To filter the same log entries to a specific event ID, you use a Hashtable filter.
$id = "4798"
Get-WinEvent -FilterHashtable @{ LogName='Security'; Id=$id }
A standard PowerShell export command outputs the selected entries.
$path = "C:\Export\Results.csv"
$id = "4798"
$events = Get-WinEvent -FilterHashtable @{ LogName='Security'; Id=$id }
$events | Select ID, Message | Export-Csv -Path $path -NoTypeInformation
To perform the same export command for multiple servers, you can use the following command.
$path = "C:\Export"
$id = "4798"
$computers = @("DC","ADFS","SQL","WEB")
foreach ($computer in $computers)
{
$events = Get-WinEvent -ComputerName $server -FilterHashtable @{ LogName='Security'; Id=$id }
$events | Select ID, Message | Export-Csv -Path "$path\$server.csv" -NoTypeInformation
}
PowerShell automation makes a difficult but necessary job easier. This review process should be part of an IT department's routine.
How do you retrieve event logs from Microsoft 365 services?
For Office 365/Microsoft 365 customers, Microsoft offers PowerShell modules to connect, perform configuration tasks and gather items such as log entries. The various administration consoles also execute queries and give the feature to download the results, but the best approach is to utilize PowerShell, especially when merging results with on-premises server logs.
The complicated part of using PowerShell with Office 365 is to understand which PowerShell module to use and the various commands. Each service has its module and commands, with services such as Exchange Online and Security and Compliance features bundled into the same module. The two main ways to connect to Microsoft 365, no matter the module, is either with direct credentials or an app registration within Azure Active Directory (Azure AD). The most common approach is to use credentials.
You can also retrieve log entries within the Security and Compliance components, which is the central access for all audit logs. To retrieve audit log entries, use the command Search-UnifiedAuditLog or Search-AdminAuditLog, depending on if you need end-user or administrator entries. Office 365 also provides a dedicated command for searching Exchange Online mailbox entries called Search-MailboxAuditLog.
Using the standard audit log requires installing and importing the correct PowerShell module.
Install-Module ExchangeOnlineManagement Import-Module ExchangeOnlineManagement Connect-ExchangeOnline -UserPrincipalName [email protected]
After the PowerShell modules load and you connect to Microsoft 365, you can execute the required commands. A basic search of end-user log entries uses the following command.
Search-UnifiedAuditLog -StartDate 1/1/2022 -EndDate 2/1/2022
If you know the record types or entry types you need, you can filter them using the following commands.
Search-UnifiedAuditLog -StartDate 1/1/2022 -EndDate 2/1/2022 -RecordType SharePointFileOperation Search-UnifiedAuditLog -StartDate 1/1/2022 -EndDate 2/1/2022 -RecordType MicrosoftTeams Search-UnifiedAuditLog -StartDate 1/1/2022 -EndDate 2/1/2022 -RecordType ExchangeAdmin
You can use the same commands outlined for on-premises servers to export the entries.
$path = "C:\Export\Results.csv" $events = Search-UnifiedAuditLog -StartDate 1/1/2022 -EndDate 2/1/2022 -RecordType MicrosoftTeams $events | Select UserIds, Operations, AuditData | Export-Csv -Path $path -NoTypeInformation
As you can see, the PowerShell syntax is relatively simple for both on-premises Windows Server systems and cloud services. By combining the two areas of concern and scheduling them as tasks, IT teams will develop a proactive way to review and monitor logs to catch potential security incidents or to control issues before they cause disruptions that affect the entire organization.
