Thursday, December 22, 2016

So somebody is throwing HTML at your sshd. What to do?

Yes, it's exactly as wrong as it sounds. Here's a distraction with bizarre twists for the true log file junkies among you. Happy reading for the holidays!

As will probably not surprise any of my regular readers, I've spent a bit of time recently browsing and processing SSH authentication logs for some of the systems in my care. As usual I browse logs with a view to extracting insights and hopefully at some future date I will be able to present useful material based on analyses of that material.

But sometimes something stands out as just too wrong. Today while browsing archived logs I came across this entry from July:

Jul 8 12:53:17 skapet sshd[88344]: Invalid user <!DOCTYPE from 187.50.71.54 port 57999

That string is the start of an SGML-style document declaration, basically what you would expect to find at the very start of an SGML-ish file such as an HTML document.

Instruct your browser to 'Display source' for this article, and that exact string is the first thing in the HTML source display buffer. But in the context of an authentication log for an SSH service, it's distinctly odd.

And what's more a little later in the same log I found:

Jul  8 20:59:08 skapet sshd[11083]: Invalid user content="text/html; from 175.143.54.193 port 26240

Again, somebody throwing HTML at my sshd, but this time from a different IP address.

This piqued my interest enough that I decided to take a look at whatever else those jokers had been up to:

[Thu Dec 22 19:40:06] peter@skapet:~$ zgrep 187.50.71.54 /var/log/authlog.23.gz
Jul 8 12:53:17 skapet sshd[88344]: Invalid user <!DOCTYPE from 187.50.71.54 port 57999
Jul 8 12:53:17 skapet sshd[88344]: Failed password for invalid user <!DOCTYPE from 187.50.71.54 port 57999 ssh2
Jul 8 12:53:17 skapet sshd[88344]: Connection closed by 187.50.71.54 port 57999 [preauth]
Jul 8 13:02:15 skapet sshd[85203]: Invalid user PUBLIC from 187.50.71.54 port 58123
Jul 8 13:02:15 skapet sshd[85203]: Failed password for invalid user PUBLIC from 187.50.71.54 port 58123 ssh2
Jul 8 13:02:15 skapet sshd[85203]: Connection closed by 187.50.71.54 port 58123 [preauth]
Jul 8 13:11:13 skapet sshd[25261]: Invalid user XHTML from 187.50.71.54 port 57227
Jul 8 13:11:13 skapet sshd[25261]: Failed password for invalid user XHTML from 187.50.71.54 port 57227 ssh2
Jul 8 13:11:13 skapet sshd[25261]: Connection closed by 187.50.71.54 port 57227 [preauth]
Jul 8 13:20:10 skapet sshd[68619]: Invalid user Strict//EN" from 187.50.71.54 port 25941
Jul 8 13:20:10 skapet sshd[68619]: Failed password for invalid user Strict//EN" from 187.50.71.54 port 25941 ssh2
Jul 8 13:20:10 skapet sshd[68619]: Connection closed by 187.50.71.54 port 25941 [preauth]
Jul 8 13:28:58 skapet sshd[96899]: Invalid user <html from 187.50.71.54 port 48462
Jul 8 13:28:58 skapet sshd[96899]: Failed password for invalid user <html from 187.50.71.54 port 48462 ssh2
Jul 8 13:28:58 skapet sshd[96899]: Connection closed by 187.50.71.54 port 48462 [preauth]
Jul 8 13:37:48 skapet sshd[59363]: Invalid user <meta from 187.50.71.54 port 46496
Jul 8 13:37:48 skapet sshd[59363]: Failed password for invalid user <meta from 187.50.71.54 port 46496 ssh2
Jul 8 13:37:48 skapet sshd[59363]: Connection closed by 187.50.71.54 port 46496 [preauth]
Jul 8 13:46:43 skapet sshd[81970]: Invalid user content="text/html; from 187.50.71.54 port 29652
Jul 8 13:46:43 skapet sshd[81970]: Failed password for invalid user content="text/html; from 187.50.71.54 port 29652 ssh2
Jul 8 13:46:43 skapet sshd[81970]: Connection closed by 187.50.71.54 port 29652 [preauth]
Jul 8 13:55:37 skapet sshd[39952]: Invalid user <title>403 from 187.50.71.54 port 45706
Jul 8 13:55:37 skapet sshd[39952]: Failed password for invalid user <title>403 from 187.50.71.54 port 45706 ssh2
Jul 8 13:55:37 skapet sshd[39952]: Connection closed by 187.50.71.54 port 45706 [preauth]
Jul 8 14:04:33 skapet sshd[68947]: Invalid user Forbidden from 187.50.71.54 port 8465
Jul 8 14:04:33 skapet sshd[68947]: Failed password for invalid user Forbidden from 187.50.71.54 port 8465 ssh2
Jul 8 14:04:34 skapet sshd[68947]: Connection closed by 187.50.71.54 port 8465 [preauth]
Jul 8 14:13:29 skapet sshd[42324]: Invalid user is from 187.50.71.54 port 54112
Jul 8 14:13:29 skapet sshd[42324]: Failed password for invalid user is from 187.50.71.54 port 54112 ssh2
Jul 8 14:13:29 skapet sshd[42324]: Connection closed by 187.50.71.54 port 54112 [preauth]
Jul 8 14:22:20 skapet sshd[83537]: Invalid user <style from 187.50.71.54 port 41269
Jul 8 14:22:20 skapet sshd[83537]: Failed password for invalid user <style from 187.50.71.54 port 41269 ssh2
Jul 8 14:22:21 skapet sshd[83537]: Connection closed by 187.50.71.54 port 41269 [preauth]
Jul 8 14:31:06 skapet sshd[53939]: Invalid user body{margin from 187.50.71.54 port 10587
Jul 8 14:31:06 skapet sshd[53939]: Failed password for invalid user body{margin from 187.50.71.54 port 10587 ssh2
Jul 8 14:31:07 skapet sshd[53939]: Connection closed by 187.50.71.54 port 10587 [preauth]
Jul 8 14:40:08 skapet sshd[24320]: Connection closed by 187.50.71.54 port 58537 [preauth]
Jul 8 14:48:57 skapet sshd[97150]: Invalid user fieldset{padding from 187.50.71.54 port 11375
Jul 8 14:48:57 skapet sshd[97150]: Failed password for invalid user fieldset{padding from 187.50.71.54 port 11375 ssh2
Jul 8 14:48:58 skapet sshd[97150]: Connection closed by 187.50.71.54 port 11375 [preauth]
Jul 8 14:57:55 skapet sshd[38951]: Invalid user 10px from 187.50.71.54 port 43776
Jul 8 14:57:55 skapet sshd[38951]: Failed password for invalid user 10px from 187.50.71.54 port 43776 ssh2
Jul 8 14:57:55 skapet sshd[38951]: Connection closed by 187.50.71.54 port 43776 [preauth]
Jul 8 15:07:53 skapet sshd[72492]: Invalid user \^M from 187.50.71.54 port 58382
Jul 8 15:07:53 skapet sshd[72492]: Failed password for invalid user \^M from 187.50.71.54 port 58382 ssh2
Jul 8 15:07:53 skapet sshd[72492]: Failed password for invalid user \^M from 187.50.71.54 port 58382 ssh2
Jul 8 15:07:54 skapet sshd[72492]: Connection closed by 187.50.71.54 port 58382 [preauth]
Jul 8 15:17:05 skapet sshd[68616]: Invalid user 0 from 187.50.71.54 port 3795
Jul 8 15:17:05 skapet sshd[68616]: Failed password for invalid user 0 from 187.50.71.54 port 3795 ssh2
Jul 8 15:17:05 skapet sshd[68616]: Connection closed by 187.50.71.54 port 3795 [preauth]
Jul 8 15:26:09 skapet sshd[14795]: Connection closed by 187.50.71.54 port 59139 [preauth]
Jul 8 15:35:04 skapet sshd[8499]: Invalid user #header{width from 187.50.71.54 port 16030
Jul 8 15:35:04 skapet sshd[8499]: Failed password for invalid user #header{width from 187.50.71.54 port 16030 ssh2
Jul 8 15:44:12 skapet sshd[17233]: Invalid user 2% from 187.50.71.54 port 2551
Jul 8 15:44:12 skapet sshd[17233]: Failed password for invalid user 2% from 187.50.71.54 port 2551 ssh2
Jul 8 15:44:13 skapet sshd[17233]: Connection closed by 187.50.71.54 port 2551 [preauth]
Jul 8 15:53:05 skapet sshd[36380]: Invalid user 2%;font-family from 187.50.71.54 port 35369
Jul 8 15:53:05 skapet sshd[36380]: Failed password for invalid user 2%;font-family from 187.50.71.54 port 35369 ssh2
Jul 8 15:53:05 skapet sshd[36380]: Connection closed by 187.50.71.54 port 35369 [preauth]
Jul 8 16:02:05 skapet sshd[5384]: Invalid user Verdana, from 187.50.71.54 port 10140
Jul 8 16:02:05 skapet sshd[5384]: Failed password for invalid user Verdana, from 187.50.71.54 port 10140 ssh2
Jul 8 16:02:06 skapet sshd[5384]: Connection closed by 187.50.71.54 port 10140 [preauth]
Jul 8 16:11:27 skapet sshd[80640]: Invalid user #content{margin from 187.50.71.54 port 27941
Jul 8 16:11:27 skapet sshd[80640]: Failed password for invalid user #content{margin from 187.50.71.54 port 27941 ssh2
Jul 8 16:20:24 skapet sshd[71772]: Invalid user <div from 187.50.71.54 port 5467
Jul 8 16:20:24 skapet sshd[71772]: Failed password for invalid user <div from 187.50.71.54 port 5467 ssh2
Jul 8 16:20:25 skapet sshd[71772]: Connection closed by 187.50.71.54 port 5467 [preauth]
Jul 8 16:29:31 skapet sshd[22288]: Invalid user Error</h1></div>\^M from 187.50.71.54 port 2932
Jul 8 16:29:31 skapet sshd[22288]: Failed password for invalid user Error</h1></div>\^M from 187.50.71.54 port 2932 ssh2
Jul 8 16:29:31 skapet sshd[22288]: Connection closed by 187.50.71.54 port 2932 [preauth]
Jul 8 16:38:32 skapet sshd[64659]: Invalid user id="content">\^M from 187.50.71.54 port 44037
Jul 8 16:38:32 skapet sshd[64659]: Failed password for invalid user id="content">\^M from 187.50.71.54 port 44037 ssh2
Jul 8 16:38:33 skapet sshd[64659]: Connection closed by 187.50.71.54 port 44037 [preauth]
Jul 8 16:47:47 skapet sshd[60396]: Invalid user class="content-container"><fieldset>\^M from 187.50.71.54 port 50741
Jul 8 16:47:47 skapet sshd[60396]: Failed password for invalid user class="content-container"><fieldset>\^M from 187.50.71.54 port 50741 ssh2
Jul 8 16:56:46 skapet sshd[84720]: Invalid user Access from 187.50.71.54 port 56868
Jul 8 16:56:46 skapet sshd[84720]: Failed password for invalid user Access from 187.50.71.54 port 56868 ssh2
Jul 8 16:56:46 skapet sshd[84720]: Connection closed by 187.50.71.54 port 56868 [preauth]
Jul 8 17:05:47 skapet sshd[39792]: Invalid user denied.</h2>\^M from 187.50.71.54 port 55262
Jul 8 17:05:47 skapet sshd[39792]: Failed password for invalid user denied.</h2>\^M from 187.50.71.54 port 55262 ssh2
Jul 8 17:05:47 skapet sshd[39792]: Connection closed by 187.50.71.54 port 55262 [preauth]
Jul 8 17:14:42 skapet sshd[2165]: Invalid user do from 187.50.71.54 port 16650
Jul 8 17:14:43 skapet sshd[2165]: Failed password for invalid user do from 187.50.71.54 port 16650 ssh2
Jul 8 17:14:43 skapet sshd[2165]: Connection closed by 187.50.71.54 port 16650 [preauth]
Jul 8 17:23:39 skapet sshd[45938]: Invalid user have from 187.50.71.54 port 15855
Jul 8 17:23:39 skapet sshd[45938]: Failed password for invalid user have from 187.50.71.54 port 15855 ssh2
Jul 8 17:23:39 skapet sshd[45938]: Connection closed by 187.50.71.54 port 15855 [preauth]
Jul 8 17:32:35 skapet sshd[64595]: Invalid user to from 187.50.71.54 port 64962
Jul 8 17:32:35 skapet sshd[64595]: Failed password for invalid user to from 187.50.71.54 port 64962 ssh2
Jul 8 17:32:35 skapet sshd[64595]: Connection closed by 187.50.71.54 port 64962 [preauth]
Jul 8 17:41:30 skapet sshd[99157]: Invalid user this from 187.50.71.54 port 63460
Jul 8 17:41:30 skapet sshd[99157]: Failed password for invalid user this from 187.50.71.54 port 63460 ssh2 

Jul 8 17:41:30 skapet sshd[99157]: Connection closed by 187.50.71.54 port 63460 [preauth]
Jul 8 17:50:27 skapet sshd[60500]: Invalid user or from 187.50.71.54 port 47364
Jul 8 17:50:27 skapet sshd[60500]: Failed password for invalid user or from 187.50.71.54 port 47364 ssh2
Jul 8 17:50:27 skapet sshd[60500]: Connection closed by 187.50.71.54 port 47364 [preauth]
Jul 8 17:59:26 skapet sshd[57379]: Invalid user using from 187.50.71.54 port 60084
Jul 8 17:59:26 skapet sshd[57379]: Failed password for invalid user using from 187.50.71.54 port 60084 ssh2
Jul 8 17:59:26 skapet sshd[57379]: Connection closed by 187.50.71.54 port 60084 [preauth]
Jul 8 18:08:22 skapet sshd[64892]: Invalid user credentials from 187.50.71.54 port 18558
Jul 8 18:08:22 skapet sshd[64892]: Failed password for invalid user credentials from 187.50.71.54 port 18558 ssh2
Jul 8 18:08:22 skapet sshd[64892]: Connection closed by 187.50.71.54 port 18558 [preauth]
Jul 8 18:17:19 skapet sshd[22377]: Invalid user you from 187.50.71.54 port 46996
Jul 8 18:17:19 skapet sshd[22377]: Failed password for invalid user you from 187.50.71.54 port 46996 ssh2
Jul 8 18:17:19 skapet sshd[22377]: Connection closed by 187.50.71.54 port 46996 [preauth]
Jul 8 18:24:50 skapet sshd[98670]: Connection closed by 187.50.71.54 port 40682 [preauth]


The other IP address offered up:

[Thu Dec 22 19:39:24] peter@skapet:~$ zgrep 175.143.54.193 /var/log/authlog.23.gz
Jul 8 16:10:42 skapet sshd[79062]: Connection closed by 175.143.54.193 port 61453 [preauth]
Jul 8 17:01:03 skapet sshd[28839]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 17:49:47 skapet sshd[1472]: Connection closed by 175.143.54.193 port 39552 [preauth]
Jul 8 18:34:12 skapet sshd[58208]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 19:19:12 skapet sshd[93151]: Connection closed by 175.143.54.193 port 6465 [preauth]
Jul 8 20:07:33 skapet sshd[84813]: Connection closed by 175.143.54.193 port 39552 [preauth]
Jul 8 20:59:08 skapet sshd[11083]: Invalid user content="text/html; from 175.143.54.193 port 26240
Jul 8 20:59:08 skapet sshd[11083]: Failed password for invalid user content="text/html; from 175.143.54.193 port 26240 ssh2
Jul 8 20:59:08 skapet sshd[11083]: Connection closed by 175.143.54.193 port 26240 [preauth]
Jul 8 21:47:54 skapet sshd[50641]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 22:38:16 skapet sshd[33990]: Invalid user Forbidden from 175.143.54.193 port 64640
Jul 8 22:38:16 skapet sshd[33990]: Failed password for invalid user Forbidden from 175.143.54.193 port 64640 ssh2
Jul 8 22:38:16 skapet sshd[33990]: Connection closed by 175.143.54.193 port 64640 [preauth]
Jul 8 23:29:47 skapet sshd[84765]: Invalid user is from 175.143.54.193 port 49280
Jul 8 23:29:47 skapet sshd[84765]: Failed password for invalid user is from 175.143.54.193 port 49280 ssh2
Jul 8 23:29:47 skapet sshd[84765]: Connection closed by 175.143.54.193 port 49280 [preauth]
Jul 9 00:18:32 skapet sshd[75290]: Connection closed by 175.143.54.193 port 13952 [preauth]
Jul 9 01:07:11 skapet sshd[15889]: Connection closed by 175.143.54.193 port 16000 [preauth]
Jul 9 01:54:19 skapet sshd[3570]: Connection closed by 175.143.54.193 port 22144 [preauth]
Jul 9 02:38:44 skapet sshd[212]: Connection closed by 175.143.54.193 port 57472 [preauth]
Jul 9 03:24:00 skapet sshd[38938]: Connection closed by 175.143.54.193 port 50304 [preauth]
Jul 9 04:08:22 skapet sshd[60530]: Connection closed by 175.143.54.193 port 21481 [preauth]
Jul 9 04:55:14 skapet sshd[77880]: Connection closed by 175.143.54.193 port 40064 [preauth]
Jul 9 05:45:26 skapet sshd[65360]: Invalid user 0;color from 175.143.54.193 port 20096
Jul 9 05:45:26 skapet sshd[65360]: Failed password for invalid user 0;color from 175.143.54.193 port 20096 ssh2
Jul 9 05:45:26 skapet sshd[65360]: Connection closed by 175.143.54.193 port 20096 [preauth]
Jul 9 06:35:50 skapet sshd[49775]: Invalid user #header{width from 175.143.54.193 port 8320
Jul 9 06:35:50 skapet sshd[49775]: Failed password for invalid user #header{width from 175.143.54.193 port 8320 ssh2
Jul 9 07:24:21 skapet sshd[88261]: Invalid user 2% from 175.143.54.193 port 57472
Jul 9 07:24:21 skapet sshd[88261]: Failed password for invalid user 2% from 175.143.54.193 port 57472 ssh2
Jul 9 07:24:22 skapet sshd[88261]: Connection closed by 175.143.54.193 port 57472 [preauth]
Jul 9 08:16:55 skapet sshd[79482]: Invalid user 2%;font-family from 175.143.54.193 port 57984
Jul 9 08:16:55 skapet sshd[79482]: Failed password for invalid user 2%;font-family from 175.143.54.193 port 57984 ssh2
Jul 9 08:16:55 skapet sshd[79482]: Connection closed by 175.143.54.193 port 57984 [preauth]
Jul 9 09:05:58 skapet sshd[67909]: Connection closed by 175.143.54.193 port 38016 [preauth]
Jul 9 09:57:24 skapet sshd[51227]: Connection closed by 175.143.54.193 port 22144 [preauth]
Jul 9 10:47:35 skapet sshd[89081]: Invalid user 



The sequences become a little easier to read if we extract the user field from the "Invalid user ..." messages:

[Thu Dec 22 20:23:23] peter@skapet:~$ zgrep 187.50.71.54 /var/log/authlog.23.gz | grep Invalid | awk '{print $8}'
<!DOCTYPE
PUBLIC
XHTML
Strict//EN"
<html <meta content="text/html;
<title>403
Forbidden
is
<style
body{margin
fieldset{padding
10px \^M
0
#header{width
2%
2%;font-family
Verdana,
#content{margin
<div
Error</h1></div>\^M
id="content">\^M
class="content-container"><fieldset>\^M
Access
denied.</h2>\^M
do
have
to
this
or
using
credentials
you
 


Now looking at what came from the the other IP address we get

[Fri Dec 23 00:51:28] peter@skapet:~$ zgrep 175.143.54.193 /var/log/authlog.23.gz | grep Invalid | awk '{print $8}'
content="text/html;
Forbidden
is
0;color
#header{width
2%
2%;font-family
<div
Error</h1></div>\^M
id="content">\^M
Access
have
using
you


Well, definitely HTML-ish, but no proper document start. Distinctly odd.

The two machines involved are apparently far apart geographically (one in Brazil and the other in Malaysia if the data from whois is anything to go by). It is of course possible that they're still connected somehow, perhaps compromised by the same group of cybercriminals and run by the same operators.

These operators then fatfingered some command or other, and their charges started pushing bizarre streams of HTML at unsuspecting SSH servers in the great elsewhere of the Internet. What they were actually trying to achieve I suspect we'll never know, but HTML was involved.

HTML is also part of the problem in one of the other bizarre phenomena I find at semi-random intervals in the SSH authentication logs.

Here are some samples from the same preserved log file:

[Thu Dec 22 20:1
6:36] peter@skapet:~$ zgrep "Bad protocol" /var/log/authlog.23.gz                                 
Jul  6 19:03:17 skapet sshd[28549]: Bad protocol version identification 'GET / HTTP/1.1' from 107.179.242.130 port 36147
Jul  8 16:14:07 skapet sshd[89181]: Bad protocol version identification 'GET / HTTP/1.1' from 204.9.214.98 port 49385
Jul  8 20:15:15 skapet sshd[28469]: Bad protocol version identification 'GET / HTTP/1.1' from 189.90.20.100 port 55039
Jul  9 10:56:40 skapet sshd[67430]: Bad protocol version identification 'GET / HTTP/1.1' from 195.88.41.10 port 52504


Again, it's not clear what these operations were attempting to do, but to me this looks like they were expecting to find either a web server or perhaps a web proxy listening on port 22.

Just like the first kind of web-to-ssh stupidity this won't actually get the requester anything and you can safely ignore both kinds of activity if you see traces of them in your own logs.

That is, if you have seen something similar in your own logs and you would care to share, I would like to hear from you, via email or in the comments below.

Even more so if you have any input on the question 'what were these clowns trying to achieve?'.

If the log analyses or related activities turn up any useful insights, you won't need to go far from here to check the results.

Good night and good luck.


Update 2016-12-23: Among the various comments that the initial version of this piece generated, two stand out as particularly useful.

The first came in a Facebook comment on my post about the story there, from my former colleague Egil Mõller, who wrote:

"Maybe they read password guesses to try from a central REST service, but that service has somehow broken, and is serving a default error message (which, since it's REST, is most likely in html)?"

The other came from OpenBSD developer Stuart Henderson, who tweeted:



If the link Twitter gave me doesn't work, here's the plaintext of Stuart's tweet:

"@pitrh the GETs aren't all that odd - could easily be scanning via proxy. I see similar on SMTP too. Not sure about the html though.
1:00 PM - 23 Dec 2016 "

I must admit I had not noticed any GETs in any SMTP related logs on my systems, but now I an honor bound to check. Stuart has given me a task, and I must finish it separately.

Also, I really like Egil's input here, because it fits so well with the data we have. In the meantime I discovered more data from a second host. Unfortunately the actual logs had been rotated out of existence, but it was still possible to piece together data on failed logins from the summaries logsentry sends me.

It appears that one machine apparently located in Hong Kong that had been trying a few logins earlier that month, with no apparent succcess, started spitting HTML at roughly the same time the other two did.

Here is all the activity in early July 2016 from that host:

Jul  4 01:08:19 delilah sshd[13635]: Failed password for invalid user bob from 218.188.213.5 port 38728 ssh2
Jul  4 01:18:28 delilah sshd[26798]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul  4 01:18:29 delilah sshd[26798]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul  4 01:27:13 delilah sshd[24543]: Failed password for invalid user ts from 218.188.213.5 port 9224 ssh2
Jul  4 01:35:54 delilah sshd[16906]: Failed password for invalid user pi from 218.188.213.5 port 9224 ssh2
Jul  7 18:30:48 skapet sshd[89165]: Failed password for invalid user admin from 218.188.213.5 port 43498 ssh2
Jul  7 18:40:33 skapet sshd[35698]: Failed password for invalid user lp from 218.188.213.5 port 9224 ssh2
Jul  7 18:50:21 skapet sshd[21112]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul  9 11:33:25 delilah sshd[10234]: Failed password for invalid user <!DOCTYPE from 218.188.213.5 port 46959 ssh2
Jul  9 11:42:28 delilah sshd[10230]: Failed password for invalid user PUBLIC from 218.188.213.5 port 9224 ssh2
Jul  9 11:51:36 delilah sshd[25489]: Failed password for invalid user XHTML from 218.188.213.5 port 9224 ssh2
Jul  9 12:09:47 delilah sshd[28023]: Failed password for invalid user <html from 218.188.213.5 port 9224 ssh2
Jul  9 12:18:51 delilah sshd[9873]: Failed password for invalid user <meta from 218.188.213.5 port 9224 ssh2
Jul  9 12:28:01 delilah sshd[13890]: Failed password for invalid user content="text/html; from 218.188.213.5 port 9224 ssh2
Jul  9 12:37:01 delilah sshd[10856]: Failed password for invalid user <title>403 from 218.188.213.5 port 9224 ssh2
Jul  9 12:46:04 delilah sshd[19947]: Failed password for invalid user Forbidden from 218.188.213.5 port 9224 ssh2
Jul  9 13:04:32 delilah sshd[22444]: Failed password for invalid user <style from 218.188.213.5 port 9224 ssh2
Jul  9 13:13:14 delilah sshd[15268]: Failed password for invalid user body{margin from 218.188.213.5 port 9224 ssh2
Jul  9 13:22:31 delilah sshd[19611]: Failed password for invalid user Helvetica, from 218.188.213.5 port 9224 ssh2
Jul  9 13:31:35 delilah sshd[15652]: Failed password for invalid user fieldset{padding from 218.188.213.5 port 59101 ssh2
Jul  9 13:40:39 delilah sshd[15607]: Failed password for invalid user 10px from 218.188.213.5 port 9224 ssh2
Jul  9 13:51:00 delilah sshd[18900]: Failed password for invalid user \^M from 218.188.213.5 port 9224 ssh2
Jul  9 13:51:00 delilah sshd[18900]: Failed password for invalid user \^M from 218.188.213.5 port 9224 ssh2
Jul  9 14:00:09 delilah sshd[24794]: Failed password for invalid user 0 from 218.188.213.5 port 9224 ssh2
Jul  9 14:18:23 delilah sshd[22897]: Failed password for invalid user #header{width from 218.188.213.5 port 9224 ssh2
Jul  9 14:27:11 delilah sshd[12713]: Failed password for invalid user 2% from 218.188.213.5 port 9224 ssh2
Jul  9 14:35:56 delilah sshd[32320]: Failed password for invalid user 2%;font-family from 218.188.213.5 port 9224 ssh2
Jul  9 14:44:46 delilah sshd[30676]: Failed password for invalid user Verdana, from 218.188.213.5 port 9224 ssh2
Jul  9 14:53:46 delilah sshd[12799]: Failed password for invalid user #content{margin from 218.188.213.5 port 9224 ssh2
Jul  9 15:02:29 delilah sshd[19535]: Failed password for invalid user <div from 218.188.213.5 port 9224 ssh2
Jul  9 15:11:17 delilah sshd[6404]: Failed password for invalid user Error</h1></div>\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:20:08 delilah sshd[2837]: Failed password for invalid user id="content">\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:29:13 delilah sshd[7831]: Failed password for invalid user class="content-container"><fieldset>\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:38:05 delilah sshd[12172]: Failed password for invalid user Access from 218.188.213.5 port 9224 ssh2
Jul  9 15:46:56 delilah sshd[23460]: Failed password for invalid user denied.</h2>\^M from 218.188.213.5 port 9224 ssh2
Jul  9 15:55:48 delilah sshd[19891]: Failed password for invalid user do from 218.188.213.5 port 9224 ssh2
Jul  9 16:13:29 delilah sshd[5999]: Failed password for invalid user to from 218.188.213.5 port 9224 ssh2
Jul  9 16:22:20 delilah sshd[17535]: Failed password for invalid user this from 218.188.213.5 port 9224 ssh2
Jul  9 16:31:11 delilah sshd[8428]: Failed password for invalid user or from 218.188.213.5 port 9224 ssh2
Jul  9 16:40:00 delilah sshd[2522]: Failed password for invalid user using from 218.188.213.5 port 9224 ssh2
Jul  9 16:48:52 delilah sshd[15034]: Failed password for invalid user credentials from 218.188.213.5 port 9224 ssh2
Jul  9 16:57:43 delilah sshd[14515]: Failed password for invalid user you from 218.188.213.5 port 53073 ssh2


If we again extract only the user name field, we get:

bob
ts
pi
admin
lp
<!DOCTYPE
PUBLIC
XHTML
<html
<meta
content="text/html;
<title>403
Forbidden
<style
body{margin
Helvetica,
fieldset{padding
10px
\^M
\^M
0
#header{width
2%
2%;font-family
Verdana,
#content{margin
<div
Error</h1></div>\^M
id="content">\^M
class="content-container"><fieldset>\^M
Access
denied.</h2>\^M
do
to
this
or
using
credentials
you


From this we see that this host was already busy poking us at long intervals with 'likely' user names, then suddenly started spewing HTML instead of likely user names in roughly the same time frame as the other two.

Seen from my perch here, this serves to validate Egil's suggestion that a common back end system started misbehaving is what caused the odd activity we're seeing.

And the apparent coordination brings to mind the Hail Mary Cloud incidents that we reported on earlier.

I suppose further digging in the logs is warranted.

If you would like to join me in the hunt for more of this, please let me know.

Tuesday, November 1, 2016

The Possibly Russian Fingerprints on the Shadow Brokers' Trick or Treat Package


Whether a result of clumsiness of a bored operator or deliberate subterfuge, there are clues that the supposed NSA front Equation Group operated out of Russia. The question remains:  What were they doing that for?

As you've probably heard already, an outfit calling themselves the Shadow Brokers published a list of what is supposedly hosts that had been compromised by the NSA and subsequently used as staging servers for whatever the NSA wanted to throw at their adversaries, with some other shady outfit known as the Equation Group actually doing the cyber-cyber thing.

The Shadow Brokers' message with links to their material is available in a number of places, among them this Medium article, which seems to be simply the text of the file message5.txt.asc, which is one of three items at both download locations.

The "list" is actually a compressed and encrypted tar archive (familiar to Unix users everywhere), which expands to a directory structure where the first level is the directory trickortreat, with two subdirectories intonation and pitchimpair, both of which in turn have numerous subdirectories with names that follow the convention

    host.domain.tld___n.m.o.p

that is, a fully qualified domain name for a host, three underscore characters and the corresponding IP version four address in the common four octet decimal notation. Each of these directories then have small text files that appear to be data about a specific program or feature.

For example, the directory intonation/hakuba.janis.or.jp___210.232.42.3 contains only the file jackladder, with the content

INTONATION___hakuba.janis.or.jp___210.232.42.3___20000822-135045() {
    ## JACKLADDER Version:2.0 OS:sparc-sun-solaris2.6
}


I take this to mean that INTONATION, whatever that is, contacted the host hakuba.janis.or.jp at the IP address 210.232.42.3, and peeking at the next line which gives us the OS version, it's my educated guess that the last string of the first line is the date in YYYYMMDD and the patch level for the operating system recorded.

The second line, or the body of the curly braces ({}) part if you prefer, tells us the JACKLADDER version and the operating system.

Other subdirectories have several files that appear to follow roughly the same format, and some record other parameters such as trickortreat/intonation/msgstore2.pldtprv.net___192.168.120.3, where the file orangutan

INTONATION___msgstore2.pldtprv.net___192.168.120.3___20021114-120148() {
    ## ORANGUTAN Version:1.4 OS:sparc-sun-solaris2.8
    export CONFIG_KEYS="81733968 69bb0b91 8b6400d6"
}


also appears to record the content of an environment variable, possibly one that the ORANGUTAN software needs to see exported to its environment in order to work as intended on that host.

Basically, this looks like what a fairly well automated system would leave behind while performing a number of operations targeted at various hosts, in a directory structure where humans will be able to find what they need to look at quickly. In my line of work, it's fairly common for such things as system logs to be collected in file system structures much like what we see here.

For your convenience if you want to study the material yourself and can't really be bothered to figure out how to extract the clear text from the gpg encrypted archive, I've put the plaintext tar archive here and the extracted files here for you to browse at your own pace.

My initial plan when I downloaded and decrypted the material was to check whether any of the hostnames or IP addresses in this material matched any of the entries in my records, such as the Hail Mary Cloud cycle that targeted SSH servers or the more recent password guessing efforts aimed at POP3 mail servers.

But then I noticed while reading the analyses of other geeks who had gotten around to doing their thing that the lists of IP addresses all had in them some addresses that should not have been there at all.

The entire 10.0.0.0/8 network was set aside way back in RFC1918 (February 1996) as one of several non-routeable ranges for private use in local area or campus networks. The way the world and IP version four works, if you have a local network at home or at work (even in large multinational enterprises), more likely than not your machines have addresses in one of these ranges:

     10.0.0.0        -   10.255.255.255  (10/8 prefix)
     172.16.0.0      -   172.31.255.255  (172.16/12 prefix)
     192.168.0.0     -   192.168.255.255 (192.168/16 prefix)

and something between those hosts and the Internet that does the network address translation (NAT) so all traffic from your network appears to come from a routable address.

Even on machines that have routeable addresses, it is not uncommon that one or more other interfaces are configured with RFC1918 addresses to pass internal but necessary traffic such as administrator logons, backups and other administrative tasks somewhere that does not interfere with the internet-facing production traffic.

Still, the Shadow Brokers' Trick or Treat package contains a handful of directories for hosts that only give internal, non-routable (RFC1918) addresses:

    intonation/butt-head.mos.ru___10.30.1.130
    intonation/mcd-su-2.mos.ru___10.34.100.2
    intonation/postbox.mos.ru___10.30.10.32
    intonation/webserv.mos.ru___10.30.10.2
    intonation/msgstore2.pldtprv.net___192.168.120.3


If your most convenient route to a machine to a specific machine is to an interface on that machine that has a local area network address, that is a strong indicator that you are working from that local network.

The first four hosts are in a Russian domain which is still operating, apparently out of Russia. The last domain, pldtprv.net, appears to be no longer active but I assume a diligent search will turn up clues about where they were based and when they were operating.
This could mean that the supposed NSA front Equation Group was actually operating from inside Russia, or at the very least had establised a 'forward base' there which was so well connected (via virtual private networks (VPNs) or other means) to their home ground that the least costly route from wherever those scripts ran to one or more Russian hosts was via those machines' internal administrative or backup interfaces.

I repeat, this, that private, nonrouteable IP addresses appear in a place you would expect to find public and routeable addresses, is a strong indicator that the Equation Group ran their activities from a local network in Russia, likely rubbing shoulders with whoever operates the mos.ru domain.

This is also a sign that whoever ran those scripts was a little careless about their routing at some point. But it could equally well mean that somebody, somewhere, is very adept at inserting clues that may in fact be false and misleading.

I probably will go forward with the comparison of this IP address and hostname hoard with my accumulated logs of less than desirable activity at some point. In the meantime I welcome your feedback on this story in comments or (if you don't want your name known and really only want me to paraphrase you in public) via email.

Good night and good luck.



Update 2016-11-02: Added the 'I repeat ...' paragraph to emphasize that finding private and nonrouteable addresses where you would otherwise expect find public and routeable ones is a strong clue that the perpetrators were operating from that local network. A surprising number of security professionals apparently miss that point.

Thursday, October 20, 2016

Is SPF Simply Too Hard For Application Developers?

The Sender Policy Framework (SPF) is unloved by some, because it conflicts with some long-established SMTP email use cases. But is it also just too hard to understand and to use correctly for application developers?

Here is a story involving a web-based contact form that indicates that this may be the case.


A wise man once noted that only two things in life are inevitable: death and taxes.

Just how taxes and interactions with the tax authorities are handled vary widely from jurisdiction to jurisdiction, but in Norway, where I live, the default mode of contact with the tax authorities for most of us is via web forms protected by sometimes cumbersome authentication procedures and the occasional alert via SMS text message to your phone.

And we're used to that the things just work with only occasional and very minor technical embarrassments for the people who operate the thing.

Then in August 2016, I tried to report a bug via the contact form at Altinn.no, the main tax authorities web site.

The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics.

The next time I checked my mail at bsdly.net, I found this bounce.

The meat of the message is

support@altinn.no
    SMTP error from remote mail server after RCPT TO::
    host mx.isp.as2116.net [193.75.104.7]: 550 5.7.23 SPF validation failed   


which means that somebody, somewhere tried to send a message to support@altinn.no, but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain.

SPF expands to "Sender Policy Framework", which is one of the early and still valid ways for a domain to publish which hosts are supposed to be sending mail on the domain's behalf.

There is no requirement to refuse delivery of mail from a host that is not in a domain's SPF record, and emphatically so when no such record exists. On the other hand, when a domain does publish SPF data, rejecting mail from hosts that are not in the set published is a valid and accepted action.

What happened is actually quite clear even from the part quoted above: the host mx.isp.as2116.net [193.75.104.7] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in bsdly.net's SPF record either, and the delivery fails.

The bounce message contained one Received: header which tells part of the story, and after decoding the MIME-encoded part it becomes clear that it contains my bug report with some slightly odd markup.

So it's clear that what produced the bounce was that the contact form had tried to deliver mail using the address I had given the contact form.

I can hear your groans all the way from there to here.

My original bug report had not been delivered, but I thought perhaps I could help them with the other problem, so I sent off a new message to the addresses that were given as contacts in the altinn.no domain's whois info, taking care to send it from a machine that was properly authorized to send mail for my domain.

The full text of the message is available here, in Norwegian. The message includes the bounce with a short introduction that said essentially "this is the result of trying to submit a bug report via your web contact form. This is clearly an SPF problem, please look into that".

Then that message bounced, too.

The exact reason is likely buried in the configuration files for altinn.no's MX, but it could be that it has some reservations against accepting mail from what it sees as a subdomain that doesn't have an MX record of its own.

Anyway, by then I had spent far too much time on this rather trivial matter while I was supposed to be doing other things, so I quickly wrote up a summary and sent it off to the same contact addresses, this time from my gmail.com account, with the various accumulated data as attachments.

Then, as often happens when dealing with the authorities, nothing happened. For quite a while.

It was only on October 18, 2016 that my gmail.com account received a reply from the support address, which quoted my last message in full, and added their resolution text saying:

"Løsning:
Det kan se ut som du har Sender Policy Framework (SPF) aktivert på din mailserver, Det er en kjent svakhet ved vårt kontaktskjema at mailservere med SPF ikke støttes.
"

Translation:

"Solution:
It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported.
"

Once again, I can hear and fully sympathize with your groans.

This is as perfectly wrong as can be, in fact the exact opposite of right.

The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem.

So I put it to you: Is SPF, the Sender Policy Framework, simply too hard for application developers to understand?

Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification?

I anticipate your feedback in comments, or if you have things you really want me to only paraphrase in public, email.



Update 2016-12-02: While looking for something else entirely, I stumbled across the DMARC FAQ, where the next to final entry describes the exact problem described in this column. Developers should perhaps learn to read FAQs.

To wit,


Why are messages I send on behalf of visitors to my website being blocked?

This depends on how you are sending these messages. If you are simply taking the website visitor’s email address and inserting it into the “From:” header of the message, and sending that message from your own servers, then you are impersonating the domain in their email address – in a way that is indistinguishable from spammers. These practices may have worked previously – in many cases for decades – because before spam became a literally overwhelming problem, nobody checked. The most successful initial mechanisms to combat such spam were IP address-based blocklists, and so your site may have been allowed to continue because it did not appear on such a list.
For the past decade, however email authentication has been introduced as a filtering mechanism, and is increasingly being used to detect and block such messages.As a best practice, you should instead be using a domain you control in the address of the From: header, and use mechanisms like SPF, DKIM, and DMARC to show that this message is authorized to use your domain. In the example below, the site visitor’s name is shown in the descriptive part of the From: header, and the Reply-To: header is set to the website visitor’s address, but the actual address used in the From: header clearly indicates that your website is the origin of the message.
From: "John Doe via the Example Website" <service@website.example.com>
Reply-To: "John Doe" <john@firstmailboxprovider.com>
To: "Bob Smith" <bob@secondmailboxprovider.com>
Subject: "An article I thought you would find interesting"

Taken from the DMARC.ORG FAQ: 4.23 Why are messages I send on behalf of visitors to my website being blocked? as of December 2. 2016.

Monday, August 29, 2016

The Voicemail Scammers Never Got Past Our OpenBSD Greylisting

We usually don't see much of the scammy spam and malware. But that one time we went looking for them, we found a campaign where our OpenBSD greylisting setup was 100% effective in stopping the miscreants' messages.

During August 23rd to August 24th 2016, a spam campaign was executed with what appears to have been a ransomware payload. I had not noticed anything particularly unusual about the bsdly.net and friends setup that morning, but then Xavier Mertens' post at isc.sans.edu Voice Message Notifications Deliver Ransomware caught my attention in the tweetstream, and I decided to have a look.

The first step was, as always, to grep the spamd logs, and sure, there were entries with from: addresses of voicemail@ in several of the domains my rigs are somehow involved in handling mail for.

But no message from voicemail@bsdly.net had yet reached any mailbox within my reach at that point. However, a colleague checked the quarantine at one of his private mail servers, and found several messsages from voicemail@ aimed at users in his domains.

Dissecting a random sample confirmed that the message came with an attachment with a .wav.zip filename that was actually a somewhat obfuscated bit of javascript, and I take others at their word that this code, if executed on your Microsoft system, would wreak havoc of some sort.

At this point, before I start presenting actual log file evidence, it is probably useful to sketch how the systems here work and interact. The three machines skapet, deliah and portal are all OpenBSD systems that run spamd in greylisting mode, and they sync their spamd data with each other via spamd's own synchronization mechanism.

All of those machines do greytrapping based on the bsdly.net list of spamtraps, and skapet has the additional duty of dumping the contents of its greytrapping generated blacklist to a downloadable text file once per hour. Any message that makes it past spamd is then fed to a real mail server that performs content filtering before handing the messages over a user's mailbox or, in the case of domains we only do the filtering for, forwards the message to the target domain's mail server.

The results of several rounds of 'grep voicemail $logfile' over the three spamd machines are collected here, or with the relatively uninteresting "queueing deletion of ..." messages removed, here.

From those sources we can see that there were a total of 386 hosts that attempted delivery, to a total of 396 host and target email pairs (annotated here in a .csv file with geographic origin according to whois).

The interesting part came when I started looking at the mail server logs to see how many had reached the content filtering or had even been passed on in the direction of users' mailboxes.

There were none.

The number of messages purportedly from voicemail@ in any of the domains we handle that made it even to the content filtering stage was 0.

Zero. Not a single one made it through even to content filtering.

That shouldn't have been a surprise.

After all I've spent significant time over the years telling people how effective greylisting is, and that the OpenBSD spamd version is the best of the breed.

You could take this episode as a recent data point that you are free to refer to in your own marketing pushes if you're doing serious business involving OpenBSD.

And if you're into those things, you will probably be delighted to learn, if you hadn't figured that out already, that a largish subset of the attempted deliveries were to addresses that were already in our published list of spamtrap addresses.

That means our miscreants automatically had themselves added to the list of trapped spammer IP addresses as intended.

If you're interested in how this works and why, I would suggest taking a peek at the OpenBSD web site, and of course I have a book out (available at that link and via better bookstores everywhere) that explains those things as well.

Relevant blog posts of mine include Keep smiling, waste spammers' time, Maintaining A Publicly Available Blacklist - Mechanisms And Principles, In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe and a few others, including the somewhat lengty Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools . To fully enjoy the experience of what these articles describe, you may want to get hold of your own CD set from the OpenBSD store.

And again, if you're doing business involving OpenBSD, please head over to the project's donations page and use one or more of the methods there to send the developers some much needed cash.

In addition to the files directly referenced in this article, some related files are  available from this directory. I'll be happy to answer any reasonable queries related to this material.

Good night and good luck.


Update 2016-08-30: I've been getting questions about the currently active campaign that has document@ as its sender. The same story there: I see them in the greylist and spamd logs, no trace whatsoever in later steps. Which means they're not getting anyhwere.

Update 2016-09-13: A quick glance at a tail -f'ed spamd log file reveals that today's fake sender of choice is CreditControl@. Otherwise same story as before, no variations. And of course, there may have been dozens I haven't noticed in the meantime.

Update 2016-11-25: Apparently another round of voicemail@ messages is in progress. The first entry in my spamd logs in this round is

Nov 25 12:39:14 skapet spamd[18359]: new entry 117.211.125.18 from <voicemail@skolelinux.no> to <axelb@skolelinux.no>, helo [117.211.125.18]

and the rest so far are listed here. Time and other factors allowing, refreshed data may appear later, possibly along with further analysis.

Monday, August 8, 2016

Chinese Hunting Chinese Over POP3 In Fjord Country

Yes, you read that right: There is a coordinated effort in progress to steal Chinese-sounding users' mail, targeting machines at the opposite end of the Eurasian landmass (and probably elsewhere).

More specifically, here at bsdly.net we've been seeing attempts at logging in to the pop3 mail retrieval service using usernames that sound distinctively like Chinese names, and the attempts originate almost exclusively from Chinese networks.

This table lists the user names and corresponding real life names attempted so far:

Name Username
Chen Qiang chenqiang
Fa Dum fadum
Gao Dang gaodang
Gao Di gaodi
Gao Guan gaoguan
Gao Hei gaohei
Gao Hua gaohua
Gao Liu gaoliu
Gao Yang gaoyang
Gao Zhang gaozhang
He An hean
He Biao hebiao
He Bing hebing
He Chang hechuang
He Chao hechao
He Chen hechen
He Cheng hecheng
He Chun hechun
He Cong hecong
He Da heda
He Di hedi
He Die hedie
He Ding heding
He Dong hedong
He Duo heduo
He Fa hefa
He Ging heqing
He Guo heguo
He Han hehan
He Hao hehao
He Heng heheng
He Hui hehui
He Jia hejia
He Jian hejian
He Jiang hejiang
He Jie hejie
He Jin hejin
He Juan hejuan
He Kai hekai
He Kan hekan
He Kong hekong
He La hela
He Le hele
He Leng heleng
He Li heli
He Lian helian
He Lie helie
He Mu hemu
He Niang heniang
He Quan hequan
He Ran heran
He Sha hesha
He Shan heshan
He Shi heshi
He Si hesi
He Song hesong
He Xiao hexiao
He Yao heyao
He Yi heyi
He Yin heyin
He Yu heyu
He Yun heyun
He Zeng hezeng
He Zeng hezhan
He Zhang hezhangxxxx
He Zhe hezhe
He Zheng hezheng
He Zhi hezhi
He Zhong hezhong
He Zhuang hezhuang
Li An lian
Li Biao libiao
Li Bin libin
Li Bo libo
Li Cheng licheng
Li Chi lichi
Li Chong lichong
Li Chuang lichuang
Li Chun lichun
Li Da lida
Li Deng lideng
Li Di lidi
Li Die lidie
Li Ding liding
Li Dong lidong
Li Duo liduo
Li Fa lifa
Li Fang lifang
Li Fen lifen
Li Feng lifeng
Li Gang ligang
Li Gao ligao
Li Guan liguan
Li Guang liguang
Li Hai lihai
Li Ka lika
Li Kai likai
Li La lila
Li Le lile
Li Lei lilei
Li Lin lilin
Li Ling liling
Li Liu liliu
Li Long lilong
Li Man liman
Li Mei limei
Li Mu limu
Li Neng lineng
Li Niang liniang
Li Peng lipeng
Li Pian lipian
Li Qian liqian
Li Qu liqu
Li Rang lirang
Li Ren liren
Li Ru liru
Li Sha lisha
Li Shi lishi
Li Shuai lishuai
Li Shun lishun
Li Si lisi
Li Song lisong
Li Tao litao
Li Teng liteng
Li Tian litian
Li Ting liting
Li Wang liwang
Li Wei liwei
Li Wen liwen
Li Xiang lixiang
Li Xing lixing
Li Xiu lixiu
Li Ying liying
Li You liyou
Li Ze lize
Li Zeng lizeng
Li Zheng lizheng
Li Zhong lizhong
Li Zhu lizhu
Li Zhuang lizhuang
Li Zhuo lizhuo
Liang Min liangmin
Liang Ming liangming
Liang Qiang liangqiang
Liang Rui liangrui
Lin Chen linchen
Lin Cheng lincheng
Lin He linhe
Lin Hua linhua
Lin Huang linhuang
Lin Neng linneng
Lin Pian linpian
Lin Qu linqu
Lin Ru linru
Lin Zhang linzhang
Liu Bin liubin
Liu Duo liuduo
Liu Fang liufang
Liu Han liuhan
Liu Hao liuhao
Liu Heng liuheng
Liu Hong liuhong
Liu Hui liuhui
Liu Jia liujia
Liu Jiang liujiang
Liu Jiao liujiao
Liu Ju liuju
Liu Juan liujuan
Liu Kai liukai
Liu Kan liukan
Liu Kang liukang
Liu Ke liuke
Liu Kong liukong
Liu Lang liulang
Liu Long liulong
Liu Mu liumu
Liu Nuo liunuo
Liu Qin liuqin
Liu Qing liuqing
Liu Qiong liuqiong
Liu Rong liurong
Liu Sen liusen
Liu Sha liusha
Liu Shun liushun
Liu Si liusi
Liu Tian liutian
Liu Wang liuwang
Liu Wei liuwei
Liu Xia liuxia
Liu Xiu liuxiu
Liu Yao liuyao
Liu Yi liuyi
Liu Ying liuying
Liu Yu liuyu
Liu Yuan liuyuan
Liu Yun liuyun
Liu Zhen liuzhen
Liu Zheng liuzheng
Liu Zhi liuzhi
Liu Zun liuzun
Lou Liu luoliu
Lu Huang lihuang
Luo Chang luochuang
Luo Chen luochen
Luo Cheng luocheng
Luo Deng luochi
Luo Deng luodeng
Luo Di luodi
Luo Dian luodian
Luo Gao luogao
Luo Guai luoguai
Luo Hang luohuang
Luo Hua luohua
Luo Lie luolie
Luo Neng luoneng
Luo Pian luopian
Luo Qi luoqi
Luo Qin luoqin
Luo Qing luoqing
Luo Qu luoqu
Luo Rong luorong
Luo Ru luoru
Luo Rui luorui
Luo Shuang luoshuang
Luo Ting luoting
Luo Tong luotong
Luo Wang luowang
Luo Wei luowei
Luo Yang luoyang
Luo Ze luoze
Song Chen songchen
Song Cheng songcheng
Song Chuang songchuang
Song Da songda
Song Deng songdeng
Song Dian songdian
Song Die songdie
Song Fei songfei
Song Fen songfen
Song Gang songgang
Song Gao songgao
Song Guai songguai
Song Guan songguan
Song Guo songguo
Song Hai songhai
Song Han songhan
Song Hang songhang
Song He songhe
Song Hei songhei
Song Heng songheng
Song Hu songhu
Song Hua songhua
Song Jia songjia
Song Jiao songjiao
Song Jie songjie
Song Jin songjin
Song Jing songjing
Song Ka songka
Song Kan songkan
Song Kang songkang
Song Kong songkong
Song Lan songlan
Song Le songle
Song Lei songlei
Song Lian songlian
Song Liang songliang
Song Liang songliao
Song Liang songliang
Song Liao songliao
Song Lin songlin
Song Liu songliu
Song Meng songmeng
Song Ming songming
Song Mu songmu
Song Nan songnan
Song Neng songneng
Song Ning songning
Song Pian songpian
Song Pin songpin
Song Qi songqi
Song Qiang songqiang
Song Qing songqing
Song Qiu songqiu
Song Ran songran
Song Rong songrong
Song Rui songrui
Song Sha songsha
Song Shuai songshuai
Song Shuang songshuang
Song Song songsong
Song Song Jun songsongjun
Song Tao songtao
Song Teng songteng
Song Wang songwang
Song Wei songwei
Song Xi songxi
Song Xia songxia
Song Xiu songxiu
Song Ya songya
Song Yang songyang
Song Yong songyong
Song You songyou
Song Yuan songyuan
Song Yue songyue
Song Yun songyun
Song Zhe songzhe
Song Zhen songzhen
Song Zheng songzheng
Song Zhuang songzhuang
Tan Qian tangqian
Tang Bing tangbing
Tang Chi tangchi
Tang Chong tangchong
Tang Chuang tangchuang
Tang Cong tangcong
Tang Di tangdi
Tang Dian tangdian
Tang Duo tangduo
Tang Fa tangfa
Tang Fan tangfan
Tang Fang tangfang
Tang Fei tangfei
Tang Fen tangfen
Tang Feng tangfeng
Tang Gang tanggang
Tang Guai tangguai
Tang Guan tangguan
Tang Guang tangguang
Tang Guo tangguo
Tang Han tanghan
Tang Hao tanghao
Tang Hei tanghei
Tang Heng tangheng
Tang Hong tanghong
Tang Hu tanghu
Tang Hui tanghui
Tang Jie tangjie
Tang Jin tangjin
Tang Jing tangjing
Tang Ju tangju
Tang Ka tangka
Tang Kai tangkai
Tang Kan tangkan
Tang Kang tangkang
Tang Ke tangke
Tang Kong tangkong
Tang La tangla
Tang Lang tanglang
Tang Le tangle
Tang Leng tangleng
Tang Li tangli
Tang Lian tanglian
Tang Lie tanglie
Tang Lin tanglin
Tang Ling tangling
Tang Liu tangliu
Tang Long tanglong
Tang Mei tangmei
Tang Mo tangmo
Tang Mu tangmu
Tang Neng tangneng
Tang Niang tangniang
Tang Nuo tangnuo
Tang Peng tangpeng
Tang Pian tangpian
Tang Ping tangping
Tang Qian tangqian
Tang Qin tangqin
Tang Qu tangqu
Tang Quan tangquan
Tang Quing tangqing
Tang Rang tangrang
Tang Ren tangren
Tang Ru tangru
Tang Ruan tangruan
Tang Rui tangrui
Tang Sen tangsen
Tang Sha tangsha
Tang Shan tangshan
Tang Shi tangshi
Tang Shun tangshun
Tang Song tangsong
Tang Tang Jun tangtangjun
Tang Tao tangtao
Tang Tian tangtian
Tang Tian tangyan
Tang Wei tangwei
Tang Xi tangxi
Tang Xia tangxia
Tang Xing tangxing
Tang Xiong tangxiong
Tang Yan tangyan
Tang Yang tangyang
Tang Yao tangyao
Tang Yi tangyi
Tang Ying tangying
Tang Yong tangyong
Tang You tangyou
Tang Yue tangyue
Tang Yun tangyun
Tang Ze tangze
Tang Zeng tangzeng
Tang Zhang tangzhang
Tang Zhe tangzhe
Tang Zhen tangzhen
Tang Zun tangzun
Xie An xiean
Xie Bin xiebin
Xie Bo xiebo
Xie Chao xiechao
Xie Cong xiecong
Xie Da xieda
Xie Di xiedi
Xie Dian xiedian
Xie Die xiedie
Xie Ding xieding
Xie Dong xiedong
Xie Duo xieduo
Xie Fang xiefang
Xie Fei xiefei
Xie Feng xiefeng
Xie Gang xiegang
Xie Gao xiegao
Xie Guai xieguai
Xie Guan xieguan
Xie Hai xiehai
Xie Hang xiehang
Xie Heng xieheng
Xie Heng xieneng
Xie Heng xieheng
Xie Heng xieneng
Xie Hong xiehong
Xie Hu xiehu
Xie Hui xiehui
Xie Jia xiejia
Xie Jian xiejian
Xie Jiang xiejiang
Xie Jiao xiejiao
Xie Jie xiejie
Xie Jing xiejing
Xie Ju xieju
Xie Kai xiekai
Xie La xiela
Xie Leng xieleng
Xie Liang xieliang
Xie Lie xielie
Xie Lin xielin
Xie Ling xieling
Xie Long xielong
Xie Man xieman
Xie Meng xiemeng
Xie Min xiemin
Xie Ming xieming
Xie Na xiena
Xie Niang xieniang
Xie Peng xiepeng
Xie Pian xiepian
Xie Pin xiepin
Xie Qi xieqi
Xie Qing xieqing
Xie Qiong xieqiong
Xie Qiu xieqiu
Xie Qu xiequ
Xie Quan xiequan
Xie Ran xieran
Xie Ruan xieruan
Xie Rui xierui
Xie Sha xiesha
Xie Shuang xieshuang
Xie Si xiesi
Xie Tao xietao
Xie Ting xieting
Xie Tong xietong
Xie Wei xiewei
Xie Wen xiewen
Xie Xi xiexi
Xie Xiang xiexiang
Xie Xin xiexin
Xie Xing xiexing
Xie Xiu xiexiu
Xie Ya xieya
Xie Yi xieyi
Xie Yin xieyin
Xie Ying xieying
Xie Yong xieyong
Xie Yu xieyu
Xie Yue xieyue
Xie Zeng xiezeng
Xie Zhan xiezhan
Xie Zhang xiezhang
Xie Zhe xiezhe
Xie Zhuo xiezhuo
Zheng Nan zhengnan

That list of some 493 names is up to date as of this writing, 2016-08-23 early evening CEST. A few more turn up with the bursts of activity we have seen every day since June 19th, 2016.

A possibly more up to date list is available here. That's a .csv file, if that sounds unfamiliar, think of it as a platform neutral text representation (to wit, "Comma Separated Values") of a spreadsheet or database -- take a peek with Notepad.exe or similar if you're not sure. I'll be updating that second list along with other related data at quasi-random intervals as time allows and as long as interesting entries keep turning up in my logs.

If your name or username is on either of those lists, you would be well advised to change your passwords right now and to check breach notification sites such as Troy Hunt's haveibeenpwned.com or breachalarm.com for clues to where your accounts could have been compromised.

That's your scoop for now. If you're interested in some more background and data, keep reading.

If you are a regular or returning reader of this column, you are most likely aware that I am a Unix sysadmin. In addition to operating and maintaining variuos systems in my employers' care, I run a small set of servers of my own that run a few Internet-facing services for myself and a small circle of friends and family.

For the most part those systems are roundly ignored by the world at large, but when they are not, funny, bizarre or interesting things happen. And mundane activities like these sometimes have interesting byproducts. When you run a mail service, you are bound to find a way to handle the spam people will try to send, and about ten years ago I started publishing a blacklist of known spamming hosts, generated from attempts to deliver mail to a slowly expanding list of known bad, invalid, never to be deliverable addresses in the domains we handle mail for.

After a while, I discovered that the list of spamtrap addresses (once again, invalid and destined never to be deliverable, ever) had been hilariously repurposed: The local parts (the string before the @ or 'at sign') started turning up as usernames in failed attempts to log on to our pop3 mail retrieval service. That was enough fun to watch that I wrote that article, and for reasons known only to the operators of the machines at the other end, those attempts have never stopped entirely.

These attempts to log in as our imaginary friends is a strong contender for the most bizarre and useless activity ever, but when those attempts were no longer news, there was nothing to write about. The spamtrap login attempts make up sort of a background noise in the authentication logs, and whenever there is an attempt to log in as a valid user from somewhere that user is clearly not, the result is usually that an entire network (whatever I could figure out from whois output) would be blocked from any communication with our site for 24 hours.

There are of course also attempts to log in as postmaster, webmaster and other IDs, some RFC mandated, that most sites including this one would handle as aliases to make up the rest of the background noise.

Then recently, something new happened. The first burst looked like this in my logs (times given in local timezone, CEST at the time):

Jun 19 06:14:58 skapet spop3d[37601]: authentication failed: no such user: lilei - 59.54.197.34
Jun 19 06:15:01 skapet spop3d[46539]: authentication failed: no such user: lilei - 59.54.197.34
Jun 19 06:15:03 skapet spop3d[8180]: authentication failed: no such user: lilei - 59.54.197.34


-- and so on, for a total of 78 attempts to log in as the non-existing user lilei, in the space of about five minutes. A little later, a similar burst of activity came for the user name lika:

Jun 19 14:11:30 skapet spop3d[68573]: authentication failed: no such user: lika - 182.87.253.48
Jun 19 14:12:22 skapet spop3d[22421]: authentication failed: no such user: lika - 182.87.253.28
Jun 19 14:12:26 skapet spop3d[7587]: authentication failed: no such user: lika - 182.87.253.28
Jun 19 14:12:30 skapet spop3d[16753]: authentication failed: no such user: lika - 182.87.253.28


and so on, for a total of 76 attempts. Over the next few days I noticed an uptick in failed pop3 access attempts that were not for valid users and did not match any entry on our spamtraps list. Still, those attempts were for users that do not exist, and would produce no useful result so I did not do anything much about them.

It was only during the early weeks of July that it struck me that the user name attempted here

Jul  8 12:19:08 skapet spop3d[54818]: authentication failed: no such user: lixing - 49.87.78.12
Jul  8 12:19:28 skapet spop3d[1987]: authentication failed: no such user: lixing - 49.87.78.12
Jul  8 12:19:37 skapet spop3d[70622]: authentication failed: no such user: lixing - 49.87.78.12
Jul  8 12:19:49 skapet spop3d[31208]: authentication failed: no such user: lixing - 49.87.78.12


(a total of 54 attempts for that user name) might actually be based on the name of a Chinese person. "Li Xing" sounded plausible enough as a possible real person. It's perhaps worth noting that at the time I had just finished reading the first two volumes of Cixin Liu's The Three Body Problem, so I was a bit more in tune than usual with what could be plausible Chinese names than I had been. (And yes, the books are very much to my taste and I have the yet unpublished translation of the third volume on pre-order.)

Unsurprisingly, a quick whois lookup revealed that the machines that tried reading the hypothetical person Li Xing's mail all had IP addresses that belonged to Chinese networks.

Once I realized I might be on to a new pattern, I went back over a few days' worth of failed pop3 login attempts and found more than a handful of usernames that looked like they could be based on Chinese names. Checking the whois data for the IP addresses in those attempts, all turned out to be from Chinese networks.

That was in itself an interesting realization, but a small, random sample does not make for proof. In order to establish an actual data set, it was back to collecting data and analysing the content.

First, collect all log data on failed pop3 attempts for a long enough period that we have a reasonable baseline and can distinguish between the background noise and new, exciting developements.

The file bigauthlog is that collection of data. Digging through my archives going back in time, I stopped at January 16, 2016 for no other reason than this would be roughly six months' worth of data, probably enough to give a reasonable baseline and to spot anomalies.

If you've read the previous columns, you will be familiar with the scripts that produce various text and CSV reports from log data input: A text report of user names by number of access attempts, a CSV dump of the same, with first and last spotted, a text report of hosts attempting access, sorted by number of attempts, a CSV dump of the same, with first and last seen dates as for the user names.

But what I wanted to see was where the login attempts were coming from for which usernames, so I started extracting the unique host to username mappings. For each entry in this CSV file, there is a host and a user name it has tried at least once (if you import that somewhere, make sure you mark the Username column as text -- LibreOffice Calc at least becomes confused when trying to parse some of those strings). The data also records whether that particular username was part of the spamtrap database at the time. If you want to do that particular check on your own greytrapping database, any matching output from

$ doas spamdb | grep -i username@

on your greytrapper box will mean it is in your list. And then finally for each entry there is the expected extract from available whois info: network address range, the network name and the country.

The most useful thing to do with that little database is to play with sorting on various fields and field combinations. If you sort on the "In spamtraps" field, the supposed Chinese names turn up with "No"s, along with a few more random-seeming combinations.

While I was building the data set I decided to add those new usernames with @bsdly.net appended to the spamtraps, and this is what finally pushed the number of spamtraps past the 30,000 mark.

Just browsing the data or perhaps sorting by IP address will show you that the pop3 gropers are spread across a large number of networks in a number of countries and territories with numbers roughly in proportion to the size of that country or territory's economy. Some, such as a particular Mexican ISP and cable TV operator stand out as being slightly over-represented, and as expected networks in the US and China stand for a large number of the total.

If you sort on the In spamtraps field, you will see that a large number of the entries that were not in the spamtraps are the ones identified as Chinese personal names, but not all. Some of the No entries are the RFC mandated mailboxes, some are aliases that are in use here for other reasons, and finally more than a handful that would fit the general description of the rest of the spamtraps: Strings superficially resembling personal names or simply random strings. These may be parts of the potential spamtraps I missed while fishing  spamtrap candidates out of logfiles some time over the decade of weirdness that has gone into maintaining the spamtraps list.

But if you sort the data primarily on the fields Name, Country, and if you like IP address and User name, you will see that as anticipated the attempts on Chinese-sounding user names come exclusively from Chinese networks, except only the "Fa Dum" (fadum) user, which appears to have been attempted only twice (on June 6th) from an IP address registered in the USA and may very well be a misclassification on my part. That particular sorting, with duplicates removed, is the origin of the list of names and usernames given earlier in this article and this CSV file.

Now that we have established that the attempts at Chinese user names come exclusively from Chinese networks, the next questions become: Who are the cyber criminals behind this activity, and what are their motivations? And why are they bothering with hosts in faraway Europe to begin with?

For the first question, it is hard to tell from this perch, but whoever runs those attempts apparently have the run of large swathes of network real estate and seem to not take any special care not to be detected, other than of course distributing the attempts widely across the network ranges and coming in only in short bursts.

So are those attempts by, let us say the public sector, to steal political dissidents' email? Or perhaps, still with a public sector slant, simply hunting for any and all overseas assets belonging to Chinese nationals? Or are we simply seeing the activities of Chinese private sector cyber criminals who are trying out likely user names wherever they can find a service that listens?

Any of all of these things could be true, but in any case it's not unlikely that what we are seeing somebody trying to find new places where username and password combinations from a recent breach might work. After all, username and password combinations that have been verified to work somewhere are likely worth more on the market than the unverified ones.

Looking at the log entries, there are sequences there that could plausibly have been produced by humans typing at keyboards. Imagine if you please vast, badly lit and insufficiently ventilated Asian cyber-sweatshops, but I would not be too surprised to find that this is actually a highly automated operation, with timing tuned to avoid detection.

Security professionals have been recommending that people stop using the pop3 protocol since as long as I care to remember, but typing "pop3" into shodan.io still produces a whopping 684,291 results, meaning that the pop3 service is nowhere near as extinct as some would have preferred.

The large number of possible targets is a likely explanation for the burstiness of the activity we are seeing: with that many hosts to cover, the groping hosts will need to set up some sort of rotation, and in addition there is the need to stay below some volume of traffic per host in order to avoid detection. This means that what any one site sees is only a very small part of the total activity. The pop3 hunt for Chinese users is most likely not exclusive to the fjord country.

If you run a pop3 service, please do yourself a favor and check your setup for any weaknesses including any not yet applied updates, as you were about to do anyway. Once you've done that, take some moments to browse your logs for strange looking login attempts.

If you find something similar to what I've reported here, I would like to hear from you. Please note that at least one of the pop3 deaemons out there by default does not report the username for failed authentication attempts but notes that the username was unknown instead. Anyway, your war stories will be appreciated in email or comments.

If your name or username appears in the table at the start of this article or in this CSV file, please start checking for unusual activity involving your accounts and start changing passwords right away. Ask your service providers if they offer more secure alternatives, and if they do, consider using these alternatives. And as I mentioned earlier, do check breach notification sites such as haveibeenpwned.com or breachalarm.com for clues to help find out whether your data could be at risk in any of the services you do use. And of course, feedback in comments or email is welcome.

And finally, if you have information on one or more breaches that may have been the source of this list of likely Chinese user names, I'd like to hear from you too.

Good night and good luck.


Update 2016-10-15: The attempts at logging in with Chinese-sounding user names from hosts in Chinese networks became incrementally less frequent over time, and seem to have stopped entirely in early October 2016.

The final entry is this one, from October 6:

Oct  6 18:11:23 skapet spop3d[97769]: authentication failed: no such user: maxiang - 114.99.9.152

That is, an attempt from the IP address range assigned to the Chinanet Anhui province network, for the user name maxiang which may very well map to Ma Xiang (or Xiang Ma) as a person's name.

During the months they were active, the robots or sweatshops in the Chinese networks tried a total of 957 distinct user names, from 3794 distinct hosts for a total of 3998 host-username combinations.

Although the number of failed pop3 attempts have now fallen to almost none (bar a treesome of persistent miscreants in the Quasi Networks, Seychelles IP address range), I will make an effort to publish updates to the data at not too infrequent intervals. You are of course free to use the data in your own analyses, as long as reasonable credit is given for the data collection. If you're unsure what that means, please contact me directly (the address in the whois information works).

Update 2016-12-07: Even though the campaign that prompted me to write this article has ended or moved its attention elsewhere, I do update the data occasionally. Returning readers may be happy to hear about a slight enhancement in presentation of the data: Startiing with today's edition, I've added an 'Attempts'  column to the main .csv file, denoting the number of attempts for each host-username pair.

Update 2017-02-08: Another round of attempts at usernames that are likely Chinese user names started on February 8th, 2017.

The first few hours brought the following user names, with the likely corresponding real life name in the second column:

Name Username
Luo Chun luochun
Luo Fa luofa
Luo Feng luofeng
Luo Hai luohai

These names have been added to the full data as well as the 2017-only portion. The log file (2016 and 2017 version or 2017-only data) contains the entries starting at Feb  8 15:26:45 (times are CET local time). It will be interesting to see how long this cycle lasts. Look for updates to the data at irregular but hopefully frequent intervals.

If you are seeing similar activity, I would like to hear from you, in comments or (these most recent attempts all originate in the 49.64.0.0/11 network (range 49.64.0.0 - 49.95.255.255, also known as  CHINANET-JS or the CHINANET jiangsu province network). The previous cycle involved several distinct Chinese networks, and as we all know, stretched over several months of low intensity activity.


I would like to thank Tore Nordstrand and Øystein Alsaker for valuable input on various aspects of this article.
The data referenced in this article will likely be updated on a roughly daily basis while the Chinese episode lasts. You can fetch them from the links in the article or from this directory, which also contains some trivial data extraction and data massaging scripts I use. If you find any errors or have any concerns, please let me know.