SharePoint Audit Logging. SharePoint 2007, 2010 and 2013 have a built-in audit logging capability that allows you to track site collection user and administrative activity. The Windows SharePoint Services 3.0 and SharePoint Foundation products also have this audit function but it is not exposed in the interface.
This app is designed to provide dashboards for Microsoft SharePoint 2010 or SharePoint 2013 and a set of starting dashboards covering health, audit and usage. The data collection provided by the TA-Sharepoint (http://apps.splunk.com/app/1908) includes the following:
- SharePoint Object Inventory
- Service and Performance Information
- Network Latency Information
- IIS Logs
- ULS Logs
- SharePoint Audit Logs
This app will not work until all installation is completed. The installation process will take some time and some adjustment to the permissions structures that govern the SharePoint farm.
The 'central Splunk instance' (consisting of search heads and indexers) can be any operating system. You must be running Splunk v6.2 or later on all Splunk servers within the central Splunk instance.
Splunk App for Microsoft SharePoint supports:
- Microsoft SharePoint 2010 running on Windows Server 2008R2
- Microsoft SharePoint 2013 running on Windows Server 2012 and 2012R2
Step 1: Install the Splunk App for SQL Server on all Content Servers
The backend storage for SharePoint is SQL Server. If you wish, install the Splunk App for SQL Server, which is available on Splunkbase, prior to commencing work on SharePoint. Note that there is a single panel in a single dashboard that utilizes the SQL Server data.
Step 2: Prepare the Splunk Indexers
Create the following indices in the Splunk Indexers:
- perfmon
- iis
- mssharepoint
In addition, you must deploy the TA-Sharepoint app to the Indexers or Heavy Intermediate Forwarders. This add-on can be found on Splunkbase at http://apps.splunk.com/app/1908 You will need to restart the splunkd process to register this app on each indexer. You can disable the inputs within this app as they are not necessary.
The TA-Sharepoint add-on handles augmented line breaking and data fixing of the ULS logs so that each transaction is available as a single event instead of multiple events. This is required for exception reporting within the dashboards.
Step 3: Designate an Inventory and Audit Reader
One of your SharePoint servers must be designated as an inventory and audit reader. This server will be used to gather the inventory information and to read the audit log within SharePoint. The Universal Forwarder that is installed on this SharePoint server will need special handling.
If you have multiple farms, then you must designate an Inventory and Audit Reader in each farm.
Step 4: Install Splunk on the other SharePoint servers
Each SharePoint server that is not the Inventory and Audit Reader host can get a standard 'Local System' type install of the latest Splunk Universal Forwarder (version v6.2 or later). It should be linked to a Splunk Deployment Server so that you can easily push updated apps to the Universal Forwarder. It should be sending data to the central Splunk instance.
Step 5: Install Splunk on the Inventory and Audit Reader
The Splunk Universal Forwarder that is running on the Inventory and Audit Reader must be running as a domain account. The domain accounts must have the following characteristics:
- Local Administrator
- Add-SPShellAdmin (PowerShell Farm Administrator)
To add the database permissions properly, ensure you run the following command using machine administrator from a farm administrator account that has shell admin privileges to SharePoint_Config:
Where DOMAINuser is the username of the user running the Splunk Universal Forwarder.
If you have multiple farms, then the user involved is probably different for each farm. Each farm will need their own Inventory and Audit Reader.
Step 6: Deploy Add-ons to the SharePoint servers
The following add-ons need to be deployed to each server:
- Splunk_TA_windows v4.64 or later
- TA-Sharepoint v0.2.0 or later
- SA-ModularInput-PowerShell v1.2.0 or later
Ensure you follow the instructions for enabling the Audit and Inventory inputs on the Audit and Inventory Reader
Step 7: Add cs-host to the SharePoint site logging
Normally, the full URL that the user types in is not available in the logs. You need to adjust the logging specification for each web site in IIS as follows:
- Open up the IIS Manager
- Select the SharePoint Site
- Select Logging
- Click on Select Fields
- Check the box next to cs-host
- Click on OK
- Under Actions, click on Apply
- Repeat steps 2-7 for each additional SharePoint site and host
This needs to be completed on each SharePoint host that answers IIS queries (which is generally all of them).
Step 8: Add Users to the SharePoint role
Part of the app install will also enable a sharepoint role. Add users that will view the SharePoint data to the sharepoint role. Without this step, they may not be able to see the relevant data. As an alternative to this process, you can also edit the eventtypes supplied with the app to indicate which index the data resides in.
Step 9: Wait 1 hour
We need some data from the inventory and this is gathered on a regular basis. In general, this data is collected within the first hour of operation.
Step 10: Regenerate Lookup Files
There is a dashboard under Searches and Reports -> Lookup Generators called the Lookup Table Builder. Use it to generate all the lookup tables.
Step 11: Check Errors
Log into Splunk and open the Splunk App for SharePoint. Select Health -> Farm Errors. All errors in this report are significant.
Errors
If the modular inputs show errors, they will show up in the splunkd.log file. The most common will be something to the effect 'SPFarm.Local null', which would indicate a permissions problem. Specifically, the user that is being used to run the Splunk Universal Forwarder on the Audit and Inventory reader has not been added to the SPShellAdmin list with Add-SPShellAdmin.
If you get Farm Errors showing an SqlException 80131904, then it is likely that you have not provided access to all the databases. Log on to the Central Administration host as a farm administrator with shell admin rights on SharePoint_Config; bring up a PowerShell host using 'Run as Administrator', and type the following:
Replace the DOMAINuser with the username of the user running the Splunk Universal Forwarder.
Performance Issues
The most normal performance issue is from the Inventory and Audit Reader. I recommend running this on a separate host if performance is an issue.
-->The SharePoint Migration Tool generates log files, summary and task level reports to help you manage, audit and troubleshoot your migration process.
How to view the reports
These reports can be viewed while the migration is taking place or after the jobs are complete.
Viewing task level reports:
- After the migration has begun, click View reports.
- A file folder will open listing the task level reports that have been generated for that specific task.
Viewing summary reports:
- After migration completes, click Migration details.
- A folder with the summary reports opens.
Summary Reports
Two types of summary reports are generated. The second type of summary report generated only if there is failures.
- SummaryReport.csv. This report contains a single row of data that gives the total picture total size, number of files migrated, duration.
- FailureSummary.csv. This report is created only when failures or errors happen during the migration process.
When assessing your migration jobs, we recommend that you first look at these summary reports. If a FailureSummary.csv file was not created, then no failures occurred during the job run.
Summary Report
Column | Description |
---|---|
Source | File path or URL of the location of the data being migrated. |
Destination | The URL of the Site and library to where the data will be migrated. |
Status | Status of each task (success, failure, in progress, not started) |
Total bytes | Total number of bytes scanned in source destination.. |
Total GB | Total number of gigabytes scanned in source destination. |
Count of files | Total number of scanned items |
Migrated bytes | The total number of bytes of data migrated. |
Migrated GB in current round | The total size of the files migrated, expressed in gigabytes. |
GB not migrated in current round | The total size of the files not migrated, expressed in gigabytes. |
Total scanned items | Total number of files and list items, including those that will be filtered out because of settings or potential scan issues. |
Total to be migrated in current round | The total number of files that were expected to migrate excluding those filtered out based on settings or potential scan issues. |
Migrated items in current round | The total number of files migrated. |
Items not migrated in current round | Number of files that did not migrate. |
Warning count in current round | Number of warnings generated. |
Start time | The time the migration task began. |
End time | The time the migration task ended. |
Duration | Length of time in minutes that the migration task took to complete. |
GB/hour | How many GB per migrated per hour. |
Round number | The round number is the incremental number of times the report has been generated based on the multiple passes the tool made to get all the content it could into the destination. |
Workflow ID | The ID number of the migration workflow. Many tasks can be in a single workflow. |
Task ID | The individual task number. |
Log Path | The location of the log files for each task. |
Failure Summary Report
This report is only generated if a failure occurs during the job run.
Column | Description |
---|---|
Source | File path or URL of the location of the data being migrated. |
Destination | The URL of the Site and library to where the data will be migrated. |
File name | Failed file or folder name or list items |
Extension | If it is a folder, then the extension is empty, else if it is a file, then extension is file extension. |
File size | Failed file or folder size or item size |
Content type | Folder or file. |
Status | Status of the file or folder that shows as 'Failed' in this report. |
Result category | Failed reason category based on the job process. |
Message | Failed reason detail description. |
Error code | Failed reason errorcode. |
Package number | The package number for the package includes the failed item . |
Migration job ID | The job id for the package includes the failed item. |
Incremental round | The last incremental round number that item failed. |
Task ID | The individual task number. |
Device name | The name of the device or computer that is running the migration job. |
Task Reports
When you need to do deeper investigation or a thorough verification of your migration task, the task level reports help you drill down into the specific details. The four recommended task level reports to use are:
- FileSummary.csv: This is similar to the overall summary report except that it aggregates the data just for a single task.
- FilesFailureReport.csv: This is the failure report at the item level. This is a filtered version of the filese report, showing only failures.
- FilesReport.csv: A list of all the items this task attempted to do
- ScanSummary.csv: This report gives statistical totals.
- StructureReport.csv: Structure report at the task level.
- StructureFailureReport.csv: Structure failure report at the task level.
- StructureFailureSummary.csv: This is an aggregate of all the structural task failure reports. This will only be generated if there are failures.
Files Summary
The FilesSummary.csv report is a summary report at the task level.
Column | Description |
---|---|
Incremental round | The round number added to the end of the report name (RO, R1, etc.) indicates if the scan or job has been rerun. |
Scanned | Total number of files scanned before migration. |
Item scan failures | Number of files that failed the scan and doesn't qualify for the migration. |
Filtered out items | Number of files not included in migration. |
Expected migrated file count | The total number of files that were expected to migrate excluding those filtered out based settings or scanned potential issues. The total number of files that were expected to migrate. |
Read | Total number of files read. |
Packaged | Total number of files packaged and ready to upload to the destination. |
Uploaded | Total number of files attempted to upload. |
ReUploaded | The total number of files that were re-uploaded. |
Submitted | Total number of files submitted. |
ReSubmitted | Total number of files resubmitted. |
Migrated | Total number of files migrated. |
Failed reading | Number of files that encountered an error or failure while being read. |
Failed packing | Number of files that encountered an error or failure while being packaged. |
Failed uploading | Number of files that encountered an error or failure while being uploaded. |
Failed submitting | Number of files that encountered an error or failure while being submitted. |
Failed querying | Number of files that encountered an error or failure while being queried. |
Device name | Name of the device or computer that is running the migration job. |
File Failure Report
The FilesFailureReport.csv, is only generated if an error resulting in a file being unable or failing to migrate.
Column | Description |
---|---|
Source | File path or URL of the location of the data being migrated. |
Destination | the URL of the tenant and library to where the data will be migrated. |
Item name | The name of the file migrated. |
Extension | The extension, indicating the file type. |
Item size | The size of the individual file. |
Content type | The file type. |
Status | Status indicating at what stage the file is. |
Result category | General code associated with the item to indicate what happened with that item. |
Message | Detailed error or informational message . |
Error code | Failed reason error code. |
Source item ID | ID of the item at the source. |
Destination item ID | ID ofthe item at the destination. |
Package number | ID generated for the package number during the transition. |
Migration job ID | The ID number of the job (which could contain one or more tasks). |
Incremental Round | The round number added to the end of the report name (RO, R1, etc.) indicates if the scan or job has been rerun. |
Task ID | The ID number of the Task. |
Device name | Name of the device or computer that is running the migration job. |
Files Report
The FilesReport.csv is a detailed report that provides data on each file within the task.
Column | Description |
---|---|
Source | File path or URL of the location of the data being migrated. |
Destination | the URL of the tenant and library to where the data will be migrated. |
File name | The name of the file migrated. |
Extension | The extension, indicating the file type. |
File size | The size of the individual file. |
Content type | The file type. |
Status | Status indicating at what stage the file is. |
Result category | General code associated with the item to indicate what happened with that item. |
Message | more detailed Error or informational message generated. |
Source item ID | ID of the item at the source. |
Destination item ID | ID ofthe item at the destination. |
Package number | ID generated for the package number during the transition. |
Migration job ID | The ID number of the job (which could contain one or more tasks). |
Incremental round | The round number added to the end of the report name (RO, R1, etc.) indicates if the scan or job has been rerun. |
Task ID | The ID number of the Task. |
Device name | Name of the device or computer that is running the migration job. |
Scan Summary
The ScanSummary.csv report provides the total stats for the scan -- a process that takes place before the actual migration begins.
Column | Description |
---|---|
Incremental round | The round number added to the end of the report name (RO, R1, etc.) indicates if the scan or job has been rerun. |
Total scanned items | Total number of folders, list items and files that have been scanned. |
Total scanned folders | Total number of folders scanned. |
Total scanned list items | Total number of list items scanned. |
Total scanned files | Total number of files scanned. |
Folders with issues | The number of folders with potential issues for the migration. |
Items with issues | The number of files with potential issues for migration. |
Items filtered out | Number of files that where filtered out based on settings in the tool. |
Folders to be migrated | Number of folders that will be migrated. |
Items to be migrated | Number of files that will be migrated. |
Total items to be migrated | Total number of folder and files that will be migrated. |
Device name | Name of the device or computer that is running the migration job. |
Structure report
Structure report at the task level.
Column | Description |
---|---|
Structure type | Site collection, site, list, field, content type, view |
Structure title | Display name of the object |
Operation | Skipped, created or updated. |
Status | Success, partial success, failure |
Details | Reason for failure. |
Source structure URL | Display the source URL. Site collection, site, and list will list the URL. Fields, content type. and view will display its container's URL. |
Destination structure URL | Display the source URL. Site collection, site, and list will list the URL. Fields, content type, and view will display its container's URL. |
Source structure ID | ID when available. |
Destination structure ID | ID when available. |
Time stamp | The time at which the action occurred. |
Structure failure report
This is a failure report at the task level. This report will only be generated if there is a failure.
Column | Description |
---|---|
Structure type | Site collection, site, list, field, content type, view |
Structure title | Display name of the object |
Operation | Skipped, created or updated. |
Status | Success, partial success, failure |
Details | Reason for failure. |
Source structure URL | Display the source URL. Site collection, site, and list will list the URL. Fields, content type. and view will display its container's URL. |
Destination structure URL | Display the source URL. Site collection, site, and list will list the URL. Fields, content type, and view will display its container's URL. |
Source structure ID | ID when available. |
Destination structure ID | ID when available. |
Time stamp | The time at which the action occurred. |
Structure failure summary
This is an aggregate of all the task failure reports. This will only be generated if there are failures.
Column | Description |
---|---|
Structure type | Site collection, site, list, field, content type, view |
Structure title | Display name of the object |
Operation | Skipped, created or updated. |
Status | Success, partial success, failure |
Details | Reason for failure. |
Source structure URL | Display the source URL. Site collection, site, and list will list the URL. Fields, content type. and view will display its container's URL. |
Destination structure URL | Display the source URL. Site collection, site, and list will list the URL. Fields, content type, and view will display its container's URL. |
Source structure ID | ID when available. |
Destination structure ID | ID when available. |
Time stamp | The time at which the action occurred. |