Log Files: You Don’t Know What You’ve Got til it’s Gone

Log files. That’s a whole lot of information most people have no idea even exists. But it’s the chronological capture of system events that you are going to need one day, and trust me – you will be so damn glad you have them.

So, two points right now.

  1. Enable logging. Make sure all your devices that have this feature are putting it to work for you. This is how you know what went wrong when something goes wrong. How you find the elephant’s footprints in the peanut butter after there has been an unfortunate incident.
  2. Where possible, make sure logs are backed up and not accessible to everyone. Because bad people happen to good logs. Sorry, I cannot say more. You’ll have to take my word for it.

 

In my talks on Threat Intel, I reference log files as having a story to tell, if you are listening. Knowing how to use your logs is key to assuming proactive defense posture.

so many logs

Logs are generated by a multitude of sources which can be overwhelming. What do you look at? Where do you start? Automation. There are log viewers and scripts by those who have come before you that will enable you to access and utilize what’s in your log data.

To help you get started, Nasruminallah Zeeshan has written a very good piece for Peerlyst, “How to Build a List of Log Files That You Need to be Inspecting Regularly” that presents the main log files you should know and be inspecting regularly for Windows and Linux. Let me share that here.

Log files in Windows systems

Windows manages and provides an assessment of log files with the help of Event Viewer. The Windows Event Viewer shows logs about application and system messages, errors, information messages, and warnings. You can run the windows event viewer by entering eventvwr.msc into Run box. In the following lines, we are going to list down the necessary log files in windows. You may need to check the following files for improved security, on a daily basis.

  • The %WINDIR%\System32\config or %WINDIR%\System32\winevt\Logs folders contain most of the log files you can see with Event Viewer.
  • The folder %WINDIR%\Logs contains various log files in text format.
  • Microsoft Security Essential stores its Runtime log files in the %PROGRAMDATA%\Microsoft\Microsoft Antimalware\Support folder and Installation log files in the %PROGRAMDATA%\Microsoft\Microsoft Security Client\Support folder.
  • Microsoft Windows system stores temporary installation and Windows defender log files in the %WINDIR%\Temp\*.log and %AppData%\Local\Temp\*.log folders. The first one contains information about MSI installations and Windows Defender scanning log files, and the second folder contains information about MSI installations run by the current user.
  • The %WINDIR%\INF\setupapi.dev.log includes information on plug and play devices and their installation.
  • The %WINDIR%\INF\setupapi.app.log file holds information about application installations.
  • The file %WINDIR%\Performance\Winsat\winsat.log file is composed of information about test results regarding performance.
  • To read Windows update information, the %WINDIR%\WindowsUpdate.log holds information about all events related to Windows Update.
  • To know about software related events and update status reports, focus on the %WINDIR%\SoftwareDistribution\ReportingEvents.log file.
  • To find out changes being made to Windows components and features, you can access the information in the %WINDIR%\Logs\CBS\CBS.log file.

Log files in Linux systems

To keep an eye on log files in Linux, carry out checking activities on a daily basis. As Linux systems contain multiple users, system administrators are advised to keep track of important log files actively. If possible, make a list of log files based on criticality level, and check them accordingly on a routine basis. In the Linux, most log files are stored in /var/log/ directory. To help you make a list of important log files in Linux, considering on picking the ones listed below.

  • The /var/log/messages file contains information about general system activities. The information stored in this file helps you troubleshoot general system errors and messages.
  • The Linux systems use /var/log/auth.log file to save information about authentication matters. This file helps you track activity regarding user authentication, such as failed logins attempts, brute force attacks and other security attack vectors related to user authentication. For the same purpose, the Red Hat and CentOS based systems use /var/log/secure file to track information. It also logs information about sudo and SSH logins.
  • To find out information about system incidents related to shutdown or restarting routines, you can use the /var/log/boot.log file.
  • The Linux systems log hardware devices and their driver information into /var/log/dmesg file. The system logs information to this file during startup, by writing data about device status, hardware errors and other generic messages. If a hardware device is not functioning properly, you can see the file for relevant information.
  • The Kernel information is important to know the system status. To investigate about troubleshooting Kernel level errors, use the /var/log/kern.log file. This file can help you cover the gap between stable system statuses, especially in case of a custom built Kernel.
  • Similar to /var/log/auth.log, the /var/log/faillog contains information on failed login attempts. The auth.log and faillogfiles are used to fingerprint security breaches related to usernames and passwords. These files also play a vital role in gathering information about a brute force attack.
  • In Linux and UNIX systems, Cron allows you to run commands or scripts on a given, pre-scheduled time. The file /var/log/cron holds information about Cron jobs. With reviewing this file, you can find information about Cron jobstatuses such as successful execution or errors in case of failure job execution.
  • The application installation information is logged into /var/log/yum.log file, if the package is installed with the Yum tool. If you have to see for information related to package installation, or you want to look for errors occurred by recent installation activities, focus on yum.log file. In this file, you can find a complete status of the installation of any package.
  • The mail server related logs are stored in Linux /var/log/maillog or /var/log/mail.log files. These files help you track the information about all incoming and outgoing emails, along with failed email delivery information. You can also find information about blocked spam emails within these files.
  • The /var/log/httpd location holds information about Apache server. The Apache server keeps logging information in error_log and access_log files. To track information related to Apache system performance, you can have a look at the error_log, while on the other hand, the access_log file is used to store information about all access requests received over HTTP.
Advertisements

Book Club: Defensive Security Handbook Chapter 2

My apologies. I am overdue on our next chapter review and this is a good one. Asset management.  The best offence is a good defence. Let’s start here.

“You don’t know what you’ve got til it’s gone.” Ain’t that the truth, especially in light of the growing blight of the Equifax breach: all that data, all those victims. Simply put, you can’t secure what you don’t know.  This applies to both tangible and intangible assets, specifically data. While this seems like common sense, for what is a basic fundamental, people do a terrible job or don’t do it at all.

tarahquote

We are told to remember these two things: “ensure there is one source of truth, and that it is a process, not a project.” In addition, classification and ownership play key roles in the success of this process. One source of truth means that whatever software or system you use to keep track of things, there are no conflicts or discrepancies with anything else. This is understood to be the single, definitive source of truth about assets.  Engage a sense of responsibility throughout the company to detect when “one of these things is not like the others”. BYOD is a thing, and unmanaged, it’s why we can’t have nice things. Ideally, get some executives involved to champion the ongoing cause. Because this is a process, not a one-time project.

Let’s talk about classification.  We live in the age of big data. As we keep learning breach after breach, it’s harrrrd to safeguard the ephemeral. Data is our most valuable asset, in digital form.  You need to know what you have, and ensure that this is understood by everyone inside and outside your organization. Most importantly, know what your crown jewels are and where they are. Your critical assets should be as prized by you as they are by attackers. Just ask the guys at Equifax and OPM about that.

Steps to classify data:

  1. Identify the sources to be protected: what they are, where they live, who are the owners.
  2. Identify the information classes: make sure the labels assigned have the same meaning for everyone. There should be no questions around critical or sensitive.
  3. Map protections to set information classification levels: Authentication, authorization, security controls, encryption.
  4. Classify and protect information
  5. Repeat as a necessary part of a yearly audit: Nothing stays the same. That’s why this is a process, and not a project.

Let’s talk about the 4 steps in the asset management process:

  1. Define the lifecycle: easier said than done. There are a lot of stages between delivery and death. It’s new, it’s old; it’s mine, now it’s yours; repair or replace it. Here is a simple set of stages: Procure, deploy; manage; decommission. And that does not mean it just gets thrown out. You need to permanently and responsibly remove all data and its traces.
  2. Gather information: how do you collect all the details on all the stuff? You could use:
    ARP cache or Address Resolution Protocol from routers and switches for a list of all the IP and MAC addresses connecting to the network.
    DHCP or Dynamic Host Configuration Protocol has all IP address reservations and may even have hostnames.
    NMAP is a comprehensive scanning tool of networks that can yield a lot of results.
    SNMP is Simple Network Management Protocol and can provide a lot of information on networked devices. Netdisco is a free automated scanning tool to help you do this.
    WMI or Windows Management Interface can get most the information from a device.
  3. Powershell is a powerhouse command line solution to get information about AD users.3. Track changes: How do you manage all the changes, the additions and deletions that affect your hardware and software inventories, and your personnel? When someone leaves, does something leave with them?
  4. Monitor and report:  You need to track updates and license renewals, or warranty expiration. It can also alert you to the addition of new and potentially unauthorized devices.

Automation: this is your helper. It works for you, with your supervision.  And ensures that routine tasks and monitoring get done consistently. Find ways to put it to work, like barcodes on items.

 

 

 

 

Getting Things Done

Dedication. Vision. Accomplishment. Passion. These are the forces of change within cyber security, and just some of the distinctive qualities about the guests Dr. Gary McGraw featured for an entire year on his Silver Bullet podcast.

We know there is a shortage of women, of diversity, in science and technology careers, particularly in cyber security.  Rather than make that the focus, this series and these women tell stories that resonate. They share their experiences, and their passion for what they do enfuses each conversation.  There are no rockstars or grandstanders here because there is no room for ego when there is work to be done.

These are my role models, my teachers, my heroes. They illuminate the darkness of our own ignorance about medical device security; making security meaningful to those outside our security enclave; understanding the power of digital forensics; crafting not just secure code but a security mindset within development.

This series is so much more than just an homage to women in tech. There is tremendous strength to be realized in our diversity; within our differences are the tools and solutions we seek for what lies ahead. I am so honoured to have been included. Thank you!

Avast AV & CCleaner Massive Malware Download: How to Help the End users

ccleaner

Screenshot of CCleaner from Talos Blog

Computers are hard. Ask the average user. They expect technology to serve their needs, not the other way around. Computers are supposed to be instant gratification, entertainment, making life easier, solving problems. They are not supposed to require much more effort than pressing the “on” key and typing. Anything else is our problem – we we were supposed to build security in, right?

We talk increasingly about “the human condition” in tech and security, because more often than not, it is that path of least resistance. Attackers know how we succumb – hence phishing. We opt for free – but you really do only get what you pay for, and buyer beware. Convenience, immediacy, lowest price – these drive the standard of quality in our connected world. It explains the current abysmal state of the IoT. And as we know, we cannot keep doing what we have been doing because – say it with me – it just doesn’t work anymore.

So when things go wrong, which they have been on an almost daily basis it seems, we who are tech reach out to the end users and let them know that they have to do more: remove software, delete files, check for files, run scans. As anyone who has ever worked helpdesk or worked with end users knows, this is not an easy ask. Most people struggle with just setting up their ISP modem/routers. Never mind removing default passwords or enabling controls. People tend to be afraid of technology, because as humans, we are afraid of what we don’t know. So we are afraid of breaking things, just as we are afraid to ask for help. And face it, tech support has earned its reputation for good reason.  People know when they are being made fun of, talked down to. We don’t make it easy for people to ask for help.

It doesn’t help that mega breaches and global ransomware outbreaks have been consistently in the headlines this past year. It’s enough to give anyone breach fatigue. And that’s what brings me to this. The talented team at Cisco Talos have issued a warning in their blog about a massive malware infection being spread by a tool, CCleaner 5.33, that has been shipping with a popular, often free, antivirus product, Avast. This is the statement according to Piriform, who owns CCleaner:

“An unauthorized modification of the CCleaner.exe binary resulted in an insertion of a two-stage backdoor capable of running code received from a remote IP address on affected systems.”

There are excellent technical write ups on this latest event and mine is not one of them. Initially, I saw the threat of securing third parties – we all know the perils of supply chain. But then, as I read through it, I realized I could read through it only after months of immersing myself, by choice, in infosec. Choosing to look up and learn what I did not already know (which is still a lot). The average user – that ain’t happening. They may read some of the articles that are more mainstream, but don’t bank on that either. Increasingly, end users are hitting the bar. Some are defeatist, saying they don’t care anymore, it’s pointless, what can they do anyway. Others believe in the power of the megacorps to protect them, so they follow whatever advice is given, like buying credit monitoring. Because that is easier than having to piece together a solution themselves on something they really know nothing about. And others prefer the head in the sand approach – Hear no evil, see no evil. I kid you not.

Some are lucky enough to have the money to pay a tech to fix the problem. Some have tech friends/family who can fix it for them. Most, however, are cast adrift on a sea of increasing peril, without life preservers. And even if we threw them a lifeline, we can’t expect they would be willing to take it. Trust goes both ways.

Before you make fun of the folks who chose Avast because it was free, here’s how I rationalized it years ago, before I arrived in InfoSec. I knew I needed to do something to secure my computer, and a free AV was better then nothing at all. Plus I could use it. And understand enough to use it, to scan. To pay attention if it alerted me. Maybe I even read a bit more to see that it suggested things I could do to clean up my computer and be safer. So, I would have downloaded CCleaner, which I have seen recommended in other places as a safe and free solution to optimizing my performance. And here’s the thing – I would have expected a known AV product, like Avast, would not be endorsing something harmful. Hence, I could trust CCleaner because I could trust Avast.

certsAnd Avast trusted CCleaner enough to promote and bundle them. To download them. So let’s look at that breakdown of trust. The researchers at Cisco Talos flagged a malicious executable file while doing some beta testing for their new product. That file happened to be the installer file for CCleaner v5.33. Now, that file was being delivered as downloads in good faith by legit CCleaner servers to millions of customers. It was legit because the appropriate digital certification was issued and signed to the main company, Piriform.

Enter the attackers. They had managed to intrude this trust worthy process and include a free, unwelcome gift with download.  This was malware, a malicious payload containing the ability to call back to the attackers command and control server, as well as being equipped with a DGA or Domain Generating Algorithm – definitely not a good thing. Obfuscation is a thing. If you can’t find someone was there, how do you know? And, without evidence or proof, trying to analyze this after the fact is problematic. The good news is there was a short window of release between August 15 til the latest version, 5.34 was issued on September 12. In previous attacks I’ve seen, manipulation of digital certificates is often an indicator that compromise is deep, systemic even, and trust in the signing authority may have been misplaced. In this case, Cisco cites:

 “the fact that the binary was digitally signed using a valid certificate issued to the software developer, it is likely that an external attacker compromised a portion of their development or build environment and leveraged that access to insert malware into the CCleaner build that was released and hosted by the organization. It is also possible that an insider with access to either the development or build environments within the organization intentionally included the malicious code or could have had an account (or similar) compromised which allowed an attacker to include the code”

Looking through the malware, Cisco found clues that the attacker tried to cover their tracks. Once the infection was in place, the program worked to erase its source data and the memory regions it inhabited. With the legit program now installed, the attacker has the ability to do as they wish in the machine they now occupy. Which means they can gather system information on the machine and send it back to their command + control server. With this link established, other malware could be sent to infect the compromised machines. Here is a high level view of what happens, as written by the Talos crew:talos pic2

As for the DGA, if the key C+C server for the malware failed to respond, the program had a failback to generate some other IP addresses using the DGA and dns lookups. Here’s the good news. Talos used the algorithm and found that the domains it generated had not been registered. Moving on it,  they registered them instead and sinkholed them to keep the attackers out. As well, the malicious version of CCleaner had been removed from the download servers.

talos pic3

What is of concern is how many people around the world apparently use CCleaner.  As of today, Piriform is somewhat ambivalent in its claims of the number of users affected. Are they limited to only 32 bit windows machines? If you go back to Aug 15, would almost 4 million users have downloaded the malware?

cleaner

Talos advises that users need to either rollback to the previous version or install the new one. Which brings me to my earlier point about the human condition:

“according to the CCleaner download page, the free version of CCleaner does not provide automated updates, so this might be a manual process for affected users.”

The team at Talos is seeing a lot of DNS activity around machines trying to connnect with those suspect domains that are no longer available. And the only reason can be those machines are being controlled by malware. Worse, the malware is not being detected using current methods. So far as fixing things goes: if you currently are a Cisco customer then you are covered. As for the rest of us, sigh. We have work to do. Uninstalling will not remove the malware. That is left to you.  If you have a full backup of your system, (and in this age of ransomware you really, really need one)  you can restore from that. Otherwise, I suggest using Malwarebytes.

http://blog.talosintelligence.com/2017/09/avast-distributes-malware.html

https://techcrunch.com/2017/09/18/avast-reckons-ccleaner-malware-infected-2-27m-users/

Equifax: WTF

Sorry. I waited to weigh in on the “dumpster fire” (credit to Brian Krebs) that is the Equifax breach because I wanted to see if those impacted expand beyond the US. They do.  If it was Apache Struts. It was. And if things got worse. Don’t cry for me Argentina but they just did.

How do you say I’m sorry for losing the confidential data of 143 million people who are your customers? You don’t. Certainly not if you are Equifax, one of the three largest bureaus for credit reports on consumers globally. You make them wait. And then, you sell them a half-baked service to fix the problem you made.  The site known as equifaxsecurity2017.com (sorry – not linking it here) is, in the words of Brian Krebs, “completely broken at best, and little more than a stalling tactic or sham at worst”.  It was flagged as a phishing site, and provided inconsistent responses.

And help comes with big strings. The offer for a year of free credit monitoring by the same firm that f*cked up in the first place has some dual-edged fine print to absolve Equifax of their responsibilities, originally stating that those who consent forfeit their rights to participate or launch a class action suit, or receive any benefits from a suit. They have since amended the injurious clause (see – I can speak legal too!) to say it “does not apply to this cybersecurity incident.” Insult to injury is that victims would have to pay for all the subsequent years of credit monitoring.  Freezing your credit is far cheaper, and effective.

We should be worried. Over 200K Visa and Mastercard holders are at risk of fraudulent purchases at the least because attackers have their account numbers, expiration dates and cardholder names.

Now, let’s talk about “Apache Struts”. Which has been flagged three times this year. Struts is hard to patch because it requires more migration and a lot more testing, which is impact and cost to business, but it happens to be used in over 60% of corporations on their major web server applications. There was a massive critical patch alert issued back around March for a zero day being actively exploited. Zero day means you’re not ready to fix it but attackers are ready to move. Guess what? The Struts flaw was unpatched back in May, when the attackers hit.

Jeff Williams is the co-founder and CTO of Contrast Security and explained the severity of this flaw which allows attackers to take over a Web host with just one HTTP request.

“This vulnerability was scored CVSS 10/10 – the highest rating. Within hours of the disclosure, we started seeing widespread automated attacks attempting to exploit this vulnerability. Those attacks are still ongoing…Essentially, an attacker could send a single HTTP request – just like the ones your browser sends – except with a specially crafted header that contains the attack.”

And then there is what happened in Argentina. Earlier this week,  it was reported by investigators who were looking into the risk to Argentina that “an online portal designed to let Equifax employees in Argentina manage credit report disputes from consumers in that country was wide open, protected by perhaps the most easy-to-guess password combination ever: “admin/admin.” I can’t even. The good news is that they took the portal down after Krebs gave them a call.

Do I sound bitter? Sorry not sorry. And so far, I am not one of the confirmed compromised. But oh, I am waiting for that shoe to drop. It has taken a ridiculous length of time for anyone in authority in Canada to address this. I get that we are polite to the point of complacency but come on! Thursday our privacy commissioner, Daniel Therrien, finally stepped in, claiming he had learned via complaints and the press, not from the source. The US has more regulations on credit reporting agencies than we have in Canada, where they are regulated by individual provinces and territories. According to Tamir Israel, who is a staff lawyer with the Canadian Internet Policy and Public Interest Clinic in Ottawa, “because of that mismatch, it falls through the cracks a little”. Per an article by Nestor Arellano in IT Canada Online:

“We have advised Equifax to provide information to affected Canadians as soon as possible and we expect the company to adopt measures to help affected Canadians,” Therrien said. “…Our office is urging Equifax to find a solution to permit Canadians to find out if they are affected as soon as possible.”

Now there is full on call for investigation. Meanwhile, the Canadian Automobile Association has informed 10,000 of its members they are at risk. Per Ian Jack, CAA managing director of communications and government relations, the information of those Canadian members who signed up for the identity protection program was stored with – wait for it – Equifax USA. That would be the sound of the other shoe dropping.

But wait – there is a happy-ish ending. News is just being released that both the CIO, David Webb, and CSO, Susan Mauldin, of Equifax are retiring. Immediately. That’s the first good news we’ve had.

https://krebsonsecurity.com/2017/09/equifax-hackers-stole-200k-credit-card-accounts-in-one-fell-swoop/

https://krebsonsecurity.com/2017/09/ayuda-help-equifax-has-my-data/

https://krebsonsecurity.com/2017/09/equifax-breach-response-turns-dumpster-fire/

http://itincanadaonline.ca/index.php/security/2273-equifax-blames-apache-vulnerability-canada-s-privacy-chief-weighs-in-on-breach

https://www.programmableweb.com/news/how-not-to-be-next-equifax/analysis/2017/09/08

http://www.ctvnews.ca/business/caa-says-10-000-consumers-could-be-equifax-hack-victims-1.3589848

https://www.darkreading.com/threat-intelligence/equifax-cio-cso-step-down/d/d-id/1329907

https://www.darkreading.com/attacks-breaches/ftc-opens-probe-into-equifax-data-breach/d/d-id/1329889?piddl_msgid=329384#msg_329384

 

A Hunting We Will Go

This weekend, in my midnight forays on Twitter (I do sleep, just not when you think I do), I discovered these graphs. As they say, a picture is worth a thousand words. These are worth far more because they visually represent high-level concepts on attackers and hunting. All credit goes to Jack Crook @jackr on Twitter, whose site is findingbad.blogspot.com.  We know how this game is played, that the attackers have been living in our networks far longer than we realized. Defence isn’t passive. It can’t be. We need to be actively monitoring all the things. We need to be expanding the Cyber Kill Chain past the perimeter and into the depths of our realm, to play this game of cat and mouse.

I’ve been pursuing my love of threat intel over these past months, and shared my learnings via talks at my local DC416 chapter, and then – fireworks and music – at Wall of Sheep at Defcon this year. OMG!  Reading Jack’s work just fires up my urge to learn more, and these depictions show what I want to say so very well.

“Enumeration”. Per Jack

Enumeration is an attacker need. They need to know where they are, where they can go, where’s the data they’re after.

“Credentials”. Jack says

Attackers need credentials if they’re going to move laterally within your network. Here’s some ideas to go digging for.

“Powershell”. Jack adds

Here are some additional things to think about when looking at Powershell

And I saved the best for last! How will they execute?

Process execution is an attacker need. There’s opportunities for developing creative ways to find when malicious.

Thank you, Jack, for sharing this wisdom. And thank you for reading!

Gone

Someone I love is gone. Depression claimed another star from our infosec universe. He was funny, brilliant, so very special. There was so much more to him than most will ever know. I will forever be glad for what we shared. But now there is only grief, loss and pain increasing with each moment as the reality takes hold. Please make it not be true.