Wednesday, October 21, 2015

Automating Forensic Artifact Collection with Splunk and GRR

Recently I had the need for GRR to collect forensic artifacts when a Splunk alert was triggered. The point of this is to collect the forensics data when a incident ticket is generated to save IR staff time and eliminate redundant redundant tasks.

Example Scenario
When a pre-defined malicious event is seen, Splunk will send an email with event details to the ticketing system and IR folks will investigate. One of the first step in the example below is to acquire the files in question with GRR. To save time we will want to automate the collection of evidence.

AV does a horrible job of detecting malicious scripts like JS.Proslikefan.B (and anything malicious in general). However, with the help of WLS this is simple to detect and alert on. Splunk search:
`wlslogs` (EventID=4688 OR EventID=592) InternalName=wscript.exe BaseFileName!=wscript.exe
To briefly explain this alert, JS.Proslikefan maintains persistent by executing a 'random filename.lnk' file in the startup folder. The LNK file executes a randomly named copy of 'wscript.exe' in the appdata folder along with the malicious script. When a person logs on to an infected machine it would generate a WLS event like this (snippet):

BaseFileName="udpbat.exe" InternalName="wscript.exe" CommandLine="C:\Users\tupac\AppData\Roaming\avseda\udpbat.exe  C:\Users\tupac\AppData\Roaming\avseda\vnyqxluw.js"

Process Overview 
1. Splunk alert finds execution of 'wscript.exe' when BaseFileName is not 'wscript.exe'.
2. Splunk alert launches '' which then launches ''. 
3. '' sends GRR an API request to acquire files in question.
4. Profit.

Splunk uses its own python version which doesn't have modules like 'requests'. Rather than installing modules into Splunk's python, we can just use a wrapper which will use the system default version of python. 

Creating the Splunk Alert
Run the search and when you're satisfied your search has minimal false positives, save it as an Alert.
When going through the alert wizard check the 'enable' box under Run a script and enter (Mashed together from a few examples on the Splunk forums):
 import gzip, os, sys, csv   
 from subprocess import call   
 python_executable = "/usr/bin/python"   
 real_script = "/opt/splunk/bin/scripts/"   
 for envvar in ("PYTHONPATH", "LD_LIBRARY_PATH"):   
  if envvar in os.environ:   
   del os.environ[envvar]   
 def openany(p):   
  if p.endswith(".gz"):   
   return open(p)   
 results_file = sys.argv[8]  
 for row in csv.DictReader(openany(results_file)):   
  my_command = [ python_executable, real_script, row["host"], ]   

The wrapper script does the following: 
  • Remove environment path and LD_LIBRARY_PATH
  • Opens the splunk search results (unzip's and reads csv for host value) .
  • Execute '' with the systems default python along with the hostname that triggered the alert.

Splunk will pass 9 variables to the script when it executes. Variable 8 contains the path to the gzip'd search results in csv format. The other variables are documented here. 
 import sys, json, urllib2, base64, requests   
 from requests.auth import HTTPBasicAuth   
 hostname = sys.argv[2]  
 grrserver = 'https://grrserver:8000'   
 username = 'Tupac'   
 password = 'isAlive'   
 base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')   
 authheader = "Basic %s" % base64string   
 index_response = requests.get(grrserver, auth=HTTPBasicAuth(username, password))   
 csrf_token = index_response.cookies.get("csrftoken")   
 headers = {   
  "Authorization": authheader,   
  "x-csrftoken": csrf_token,   
  "x-requested-with": "XMLHttpRequest"   
 data = {   
  "hostname": hostname,   
  "paths": ["%%users.appdata%%\Roaming\*\*.{js,exe}",   
            "%%users.appdata%%\Roaming\Microsoft\Windows\Start Menu\Programs\Startup\*.lnk"],    
  "pathtype": "OS"    
 response = + "/api/clients/" + hostname + "/flows/remotegetfile",   
        headers=headers, data=json.dumps(data),   
        cookies=cookies, auth=HTTPBasicAuth(username, password))   

'' will start a FileFinder flow on the 'hostname' variable passed to it by the '' script. When IR staff is able to review the ticket, the files will be available in GRR to download and review.

In Summary...
This is just a basic example to demonstrate how all the pieces fit together. There are some really cool things you can do with these tools to automate stuff. Some things I've been playing with:

Automatically launch Incident Response Collector ($MFT, Registry, Browser History, etc.) and full memory image when a known bad MD5 or static indicator is seen in Splunk.

Utilize WLS's hash tracking to automatically submit new binaries to internal malware analysis tools. Splunk alert would be:
`wlslogs` (EventID=4688 OR EventID=592) NewHash=True
In the, you would add row["NewProcessName"] and pass it to to download (instead of the static path in the example above).

Get any executable downloaded from the internet and send it to internal malware analysis tools.
`wlslogs` (EventID=4688 OR EventID=592) Zone=3
Get any compressed file attachment opened from Outlook email and send to internal malware analysis tools.
`wlslogs` (EventID=4688 OR EventID=592) CreatorProcessName=OUTLOOK BaseFileName=winzip*

If you have any examples/suggestions on automation with GRR and WLS/Splunk, share them on the GRR user group, I'm really interested to hear what other folks have done.

1 comment:

  1. I really appreciate the information shared above. It’s of great help. If someone wants to learn Online (Virtual) instructor lead live training in Splunk TECHNOLOGY, kindly contact us
    MaxMunus Offer World Class Virtual Instructor-led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ pieces of training in India, USA, UK, Australia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Pratik Shekhar
    Ph:(0) +91 9066268701