Log Files: You Don’t Know What You’ve Got til it’s Gone

Log files. That’s a whole lot of information most people have no idea even exists. But it’s the chronological capture of system events that you are going to need one day, and trust me – you will be so damn glad you have them.

So, two points right now.

  1. Enable logging. Make sure all your devices that have this feature are putting it to work for you. This is how you know what went wrong when something goes wrong. How you find the elephant’s footprints in the peanut butter after there has been an unfortunate incident.
  2. Where possible, make sure logs are backed up and not accessible to everyone. Because bad people happen to good logs. Sorry, I cannot say more. You’ll have to take my word for it.

 

In my talks on Threat Intel, I reference log files as having a story to tell, if you are listening. Knowing how to use your logs is key to assuming proactive defense posture.

so many logs

Logs are generated by a multitude of sources which can be overwhelming. What do you look at? Where do you start? Automation. There are log viewers and scripts by those who have come before you that will enable you to access and utilize what’s in your log data.

To help you get started, Nasruminallah Zeeshan has written a very good piece for Peerlyst, “How to Build a List of Log Files That You Need to be Inspecting Regularly” that presents the main log files you should know and be inspecting regularly for Windows and Linux. Let me share that here.

Log files in Windows systems

Windows manages and provides an assessment of log files with the help of Event Viewer. The Windows Event Viewer shows logs about application and system messages, errors, information messages, and warnings. You can run the windows event viewer by entering eventvwr.msc into Run box. In the following lines, we are going to list down the necessary log files in windows. You may need to check the following files for improved security, on a daily basis.

  • The %WINDIR%\System32\config or %WINDIR%\System32\winevt\Logs folders contain most of the log files you can see with Event Viewer.
  • The folder %WINDIR%\Logs contains various log files in text format.
  • Microsoft Security Essential stores its Runtime log files in the %PROGRAMDATA%\Microsoft\Microsoft Antimalware\Support folder and Installation log files in the %PROGRAMDATA%\Microsoft\Microsoft Security Client\Support folder.
  • Microsoft Windows system stores temporary installation and Windows defender log files in the %WINDIR%\Temp\*.log and %AppData%\Local\Temp\*.log folders. The first one contains information about MSI installations and Windows Defender scanning log files, and the second folder contains information about MSI installations run by the current user.
  • The %WINDIR%\INF\setupapi.dev.log includes information on plug and play devices and their installation.
  • The %WINDIR%\INF\setupapi.app.log file holds information about application installations.
  • The file %WINDIR%\Performance\Winsat\winsat.log file is composed of information about test results regarding performance.
  • To read Windows update information, the %WINDIR%\WindowsUpdate.log holds information about all events related to Windows Update.
  • To know about software related events and update status reports, focus on the %WINDIR%\SoftwareDistribution\ReportingEvents.log file.
  • To find out changes being made to Windows components and features, you can access the information in the %WINDIR%\Logs\CBS\CBS.log file.

Log files in Linux systems

To keep an eye on log files in Linux, carry out checking activities on a daily basis. As Linux systems contain multiple users, system administrators are advised to keep track of important log files actively. If possible, make a list of log files based on criticality level, and check them accordingly on a routine basis. In the Linux, most log files are stored in /var/log/ directory. To help you make a list of important log files in Linux, considering on picking the ones listed below.

  • The /var/log/messages file contains information about general system activities. The information stored in this file helps you troubleshoot general system errors and messages.
  • The Linux systems use /var/log/auth.log file to save information about authentication matters. This file helps you track activity regarding user authentication, such as failed logins attempts, brute force attacks and other security attack vectors related to user authentication. For the same purpose, the Red Hat and CentOS based systems use /var/log/secure file to track information. It also logs information about sudo and SSH logins.
  • To find out information about system incidents related to shutdown or restarting routines, you can use the /var/log/boot.log file.
  • The Linux systems log hardware devices and their driver information into /var/log/dmesg file. The system logs information to this file during startup, by writing data about device status, hardware errors and other generic messages. If a hardware device is not functioning properly, you can see the file for relevant information.
  • The Kernel information is important to know the system status. To investigate about troubleshooting Kernel level errors, use the /var/log/kern.log file. This file can help you cover the gap between stable system statuses, especially in case of a custom built Kernel.
  • Similar to /var/log/auth.log, the /var/log/faillog contains information on failed login attempts. The auth.log and faillogfiles are used to fingerprint security breaches related to usernames and passwords. These files also play a vital role in gathering information about a brute force attack.
  • In Linux and UNIX systems, Cron allows you to run commands or scripts on a given, pre-scheduled time. The file /var/log/cron holds information about Cron jobs. With reviewing this file, you can find information about Cron jobstatuses such as successful execution or errors in case of failure job execution.
  • The application installation information is logged into /var/log/yum.log file, if the package is installed with the Yum tool. If you have to see for information related to package installation, or you want to look for errors occurred by recent installation activities, focus on yum.log file. In this file, you can find a complete status of the installation of any package.
  • The mail server related logs are stored in Linux /var/log/maillog or /var/log/mail.log files. These files help you track the information about all incoming and outgoing emails, along with failed email delivery information. You can also find information about blocked spam emails within these files.
  • The /var/log/httpd location holds information about Apache server. The Apache server keeps logging information in error_log and access_log files. To track information related to Apache system performance, you can have a look at the error_log, while on the other hand, the access_log file is used to store information about all access requests received over HTTP.
Advertisements

Book Club: Defensive Security Handbook Chapter 2

My apologies. I am overdue on our next chapter review and this is a good one. Asset management.  The best offence is a good defence. Let’s start here.

“You don’t know what you’ve got til it’s gone.” Ain’t that the truth, especially in light of the growing blight of the Equifax breach: all that data, all those victims. Simply put, you can’t secure what you don’t know.  This applies to both tangible and intangible assets, specifically data. While this seems like common sense, for what is a basic fundamental, people do a terrible job or don’t do it at all.

tarahquote

We are told to remember these two things: “ensure there is one source of truth, and that it is a process, not a project.” In addition, classification and ownership play key roles in the success of this process. One source of truth means that whatever software or system you use to keep track of things, there are no conflicts or discrepancies with anything else. This is understood to be the single, definitive source of truth about assets.  Engage a sense of responsibility throughout the company to detect when “one of these things is not like the others”. BYOD is a thing, and unmanaged, it’s why we can’t have nice things. Ideally, get some executives involved to champion the ongoing cause. Because this is a process, not a one-time project.

Let’s talk about classification.  We live in the age of big data. As we keep learning breach after breach, it’s harrrrd to safeguard the ephemeral. Data is our most valuable asset, in digital form.  You need to know what you have, and ensure that this is understood by everyone inside and outside your organization. Most importantly, know what your crown jewels are and where they are. Your critical assets should be as prized by you as they are by attackers. Just ask the guys at Equifax and OPM about that.

Steps to classify data:

  1. Identify the sources to be protected: what they are, where they live, who are the owners.
  2. Identify the information classes: make sure the labels assigned have the same meaning for everyone. There should be no questions around critical or sensitive.
  3. Map protections to set information classification levels: Authentication, authorization, security controls, encryption.
  4. Classify and protect information
  5. Repeat as a necessary part of a yearly audit: Nothing stays the same. That’s why this is a process, and not a project.

Let’s talk about the 4 steps in the asset management process:

  1. Define the lifecycle: easier said than done. There are a lot of stages between delivery and death. It’s new, it’s old; it’s mine, now it’s yours; repair or replace it. Here is a simple set of stages: Procure, deploy; manage; decommission. And that does not mean it just gets thrown out. You need to permanently and responsibly remove all data and its traces.
  2. Gather information: how do you collect all the details on all the stuff? You could use:
    ARP cache or Address Resolution Protocol from routers and switches for a list of all the IP and MAC addresses connecting to the network.
    DHCP or Dynamic Host Configuration Protocol has all IP address reservations and may even have hostnames.
    NMAP is a comprehensive scanning tool of networks that can yield a lot of results.
    SNMP is Simple Network Management Protocol and can provide a lot of information on networked devices. Netdisco is a free automated scanning tool to help you do this.
    WMI or Windows Management Interface can get most the information from a device.
  3. Powershell is a powerhouse command line solution to get information about AD users.3. Track changes: How do you manage all the changes, the additions and deletions that affect your hardware and software inventories, and your personnel? When someone leaves, does something leave with them?
  4. Monitor and report:  You need to track updates and license renewals, or warranty expiration. It can also alert you to the addition of new and potentially unauthorized devices.

Automation: this is your helper. It works for you, with your supervision.  And ensures that routine tasks and monitoring get done consistently. Find ways to put it to work, like barcodes on items.

 

 

 

 

Equifax: WTF

Sorry. I waited to weigh in on the “dumpster fire” (credit to Brian Krebs) that is the Equifax breach because I wanted to see if those impacted expand beyond the US. They do.  If it was Apache Struts. It was. And if things got worse. Don’t cry for me Argentina but they just did.

How do you say I’m sorry for losing the confidential data of 143 million people who are your customers? You don’t. Certainly not if you are Equifax, one of the three largest bureaus for credit reports on consumers globally. You make them wait. And then, you sell them a half-baked service to fix the problem you made.  The site known as equifaxsecurity2017.com (sorry – not linking it here) is, in the words of Brian Krebs, “completely broken at best, and little more than a stalling tactic or sham at worst”.  It was flagged as a phishing site, and provided inconsistent responses.

And help comes with big strings. The offer for a year of free credit monitoring by the same firm that f*cked up in the first place has some dual-edged fine print to absolve Equifax of their responsibilities, originally stating that those who consent forfeit their rights to participate or launch a class action suit, or receive any benefits from a suit. They have since amended the injurious clause (see – I can speak legal too!) to say it “does not apply to this cybersecurity incident.” Insult to injury is that victims would have to pay for all the subsequent years of credit monitoring.  Freezing your credit is far cheaper, and effective.

We should be worried. Over 200K Visa and Mastercard holders are at risk of fraudulent purchases at the least because attackers have their account numbers, expiration dates and cardholder names.

Now, let’s talk about “Apache Struts”. Which has been flagged three times this year. Struts is hard to patch because it requires more migration and a lot more testing, which is impact and cost to business, but it happens to be used in over 60% of corporations on their major web server applications. There was a massive critical patch alert issued back around March for a zero day being actively exploited. Zero day means you’re not ready to fix it but attackers are ready to move. Guess what? The Struts flaw was unpatched back in May, when the attackers hit.

Jeff Williams is the co-founder and CTO of Contrast Security and explained the severity of this flaw which allows attackers to take over a Web host with just one HTTP request.

“This vulnerability was scored CVSS 10/10 – the highest rating. Within hours of the disclosure, we started seeing widespread automated attacks attempting to exploit this vulnerability. Those attacks are still ongoing…Essentially, an attacker could send a single HTTP request – just like the ones your browser sends – except with a specially crafted header that contains the attack.”

And then there is what happened in Argentina. Earlier this week,  it was reported by investigators who were looking into the risk to Argentina that “an online portal designed to let Equifax employees in Argentina manage credit report disputes from consumers in that country was wide open, protected by perhaps the most easy-to-guess password combination ever: “admin/admin.” I can’t even. The good news is that they took the portal down after Krebs gave them a call.

Do I sound bitter? Sorry not sorry. And so far, I am not one of the confirmed compromised. But oh, I am waiting for that shoe to drop. It has taken a ridiculous length of time for anyone in authority in Canada to address this. I get that we are polite to the point of complacency but come on! Thursday our privacy commissioner, Daniel Therrien, finally stepped in, claiming he had learned via complaints and the press, not from the source. The US has more regulations on credit reporting agencies than we have in Canada, where they are regulated by individual provinces and territories. According to Tamir Israel, who is a staff lawyer with the Canadian Internet Policy and Public Interest Clinic in Ottawa, “because of that mismatch, it falls through the cracks a little”. Per an article by Nestor Arellano in IT Canada Online:

“We have advised Equifax to provide information to affected Canadians as soon as possible and we expect the company to adopt measures to help affected Canadians,” Therrien said. “…Our office is urging Equifax to find a solution to permit Canadians to find out if they are affected as soon as possible.”

Now there is full on call for investigation. Meanwhile, the Canadian Automobile Association has informed 10,000 of its members they are at risk. Per Ian Jack, CAA managing director of communications and government relations, the information of those Canadian members who signed up for the identity protection program was stored with – wait for it – Equifax USA. That would be the sound of the other shoe dropping.

But wait – there is a happy-ish ending. News is just being released that both the CIO, David Webb, and CSO, Susan Mauldin, of Equifax are retiring. Immediately. That’s the first good news we’ve had.

https://krebsonsecurity.com/2017/09/equifax-hackers-stole-200k-credit-card-accounts-in-one-fell-swoop/

https://krebsonsecurity.com/2017/09/ayuda-help-equifax-has-my-data/

https://krebsonsecurity.com/2017/09/equifax-breach-response-turns-dumpster-fire/

http://itincanadaonline.ca/index.php/security/2273-equifax-blames-apache-vulnerability-canada-s-privacy-chief-weighs-in-on-breach

https://www.programmableweb.com/news/how-not-to-be-next-equifax/analysis/2017/09/08

http://www.ctvnews.ca/business/caa-says-10-000-consumers-could-be-equifax-hack-victims-1.3589848

https://www.darkreading.com/threat-intelligence/equifax-cio-cso-step-down/d/d-id/1329907

https://www.darkreading.com/attacks-breaches/ftc-opens-probe-into-equifax-data-breach/d/d-id/1329889?piddl_msgid=329384#msg_329384

 

Gone

Someone I love is gone. Depression claimed another star from our infosec universe. He was funny, brilliant, so very special. There was so much more to him than most will ever know. I will forever be glad for what we shared. But now there is only grief, loss and pain increasing with each moment as the reality takes hold. Please make it not be true.

Book Club: Defensive Security Handbook Chapter 1

Welcome! To recap. We’ll be working through this book together to learn and grow our Blue team skill. Cuz the best offence can be a proactive defence. This book is a fantastic resource, especially for those who are starting out, or need a good overall reference. Based on my real-world experience,I believe it should be a desk reference, and part of any security curriculum. I am going to go on Amazon and say that infact!

Now. Chapter 1: Creating a Security Program. That does not just magically happen. And yet, we really wish it could because everyone needs a good security program in place. If you’ve ever tried to clean your kid’s room, you’ll understand how daunting this can be. Where do you begin?  Well, as our insightful authors Amanda and Lee point out, we don’t need to reinvent the wheel. They’re right. They refer to the NIST framework, which I can tell you I get to use on an almost daily basis when doing security audits (let’s not go there, ok?) You want to work from best practices, existing and proven standards that are used to hold organizations accountable ie compliance standards.  Good news! Amanda and Lee will take us through all that fun in Chapter 8.

So Point 1: Have the right team in place. You need the right people in the right role to make the right decision.  The book recommends establishing 4 main teams: Executive, Risk, Security and Audit.  I will tell you from experience that if you don’t have Exec buy in from the get go, you will find yourself spinning your wheels. How do you get that? Speak to the suits in their love language – Risk. And you need Audit to bring the flowers &b chocolates to their door. And yes – this is from my daily reality. Plus, audit lets you put everything down, and organize it, which makes it easier to track things, and reorganize things. Because you cannot secure what you don’t know.

Point 2: Set a Baseline.  I love talking about threat intel (holding back – self-control) and how to make it relevant. This is how you make it relevant. What’s your normal? That’s your baseline. Because how else will you know something went bump in the night? The attackers are wery wery quiet. And believe me, they are in your network like those darn carpenter ants are in the woodwork. So this will be a fact gathering mission, and you want to do it well, Plus set it up with automation, and updates. SInce Asset mgmt is the next chapter, so we’ll leave that alone for now.

Point 3: Threat/Risk Assessment. This is challenging, and a learning process for those starting out. The concept of risk and being able to articulate it to business is way hard, I’ll be honest, and I am very good with words. What we in security think is a threat has to be explained in terms relevant to the organization we serve. That’s the crux right there. It’s not what we think so much as what they understand. And true – unless it negatively impacts the organization’s bottom line or existence then even if we think it is a risk, it isn’t.  So, you need a parlay with the suits to know how the organization is defined in terms of threat and risk. Then, when Patch Tuesday comes, you can look at what is critical and determine if that is critical to your organization and why as you justify the need to make adjustments to your regular patching cycle (real world). 4 steps process: Assess, Mitigate, Monitor, Prioritize.

Point 4: Practice and Prepare. Are you as ready as you think?  So, I like to talk about why everyone, everyone needs a good Disaster Recovery/Business Continuity plan in place. And that means one that has been tested, so that people know how it works, and how they work with it. Let your inner kid come out for this because you need to play “What If” to do this right. There are things called Table Top Drills that are so good especially for addressing ransomware and DDoS scenarios. Or Sharknado. Lol! As stated in the book “testing of tabletop exercises and drills can serve as a proof of concept”.  Amanda and Lee are right on the money by stipulating your need participants from across the org like HR, Legal, Marketing, Finance etc. Infact, they provide such a good explanation you should be able to go do one.

Now, I love that the book has used a great tool, the Intrusion Kill Chain, to explain how to think through an event scenario. I happen to be a HUGE fan of the Cyber kill chain (Lockheed Martin),  the extended cyber kill chain, and ATT&CK matrix by MITRE.

Point 5: Learn and Grow. The chapter finishes by encouraging us to expand our knowledge and skills through home labs and projects, CTFs, conferences and mentoring. I have done all of these and YES! It’s not hard to do and so rewarding you’ll want to make time. Because my friends, learning never stops in InfoSec. To paraphrase the wise and wonderful Leslie Carhart aka @hacksforpancakes (on the July 11 Down the Security Rabbithole podcast) “It never stops. This job never stops. And if you want to be good at it, if you really want to be good at it, you can’t stop.”

Because it’s not just what we do, it’s so much about who we are. Til next time!

 

Blue Team FTW!

Time to do some learning. There are things we can be doing better. Things we can be doing right. And with the help of two very good friends, Amanda Berlin and Lee Brotherston, we are going to batten down these hatches and secure the *&$@ out of our fortresses.

As stated in the Foreward, “the red teams get all the glory.”  And it’s true. For blue teams it feels lonely and unappreciated, but there is so much truth in this:

“Doing defense is a vital, noble and worthwhile pursuit”

It’s easy to get turned around by hype. We follow the direction the noise is making, and tbh vendors make a lot of noise.  What we need to do, and have known for so long, is not to be dazzled by the shiny, blinky boxes. As so well said by Andrew Kalat:

Security Vendors will often define the problem set as the problem they can solve with their technology, not necessarily the problem an organization actually has.

So here’s to taking a more holistic view, as this excellent guide advocates, and understanding how all the pieces need to work for this particular machine. We’ll share Chapter 1 next.

 

Petnya Post-Mortem: Wiper, not Ransom

This wasn’t just another ransomware attack. It marks a pivot. Because these are the games nationstates play. With collateral damage and no impunity because attribution is hard. We left brick and mortar behind some time ago, when the battlefield moved to cyberspace, where there are no boundaries. Moreover, whatever previous rules of battle we followed do not apply.

There was a one-two punch, with the events out of the Ukraine Thursday morning.  Absolutely things were connected and we need to remember that going forward. Bigger picture. Because a lot is at play right now. From my vantage point, as a Poli Sci grad, cyber security is intrinsically tied to whatever is going on in the larger arena. National security. Global security. The whims of the powers that be dictate their machinations of technology, which has become their new and shiny arsenal. They’ve been at it for a while now, but unlike conventional physical battlefields, we don’t witness what plays out until we’re impacted.

What’s critical to me is that this attack was presented as ransomware to throw us off. As described by The Grugq:

This is definitely not designed to make money. This is designed to spread fast and cause damage, with a plausibly deniable cover of “ransomware.”

This was actually a targeted attack against Ukraine, using malware that was highly destructive. This attack was never about making money. It was all about taking down systems and taking away access to essential service, as per this illustration from the blog post by The Grugq :

 

 

 

 

 

 

 

 

 

Think CIA – confidentiality, integrity, accessibility. Ransomware and wiperware go after accessibility. And in our world, downtime can equal death, figuratively as well as literally (think hospitals and critical infrastructure).  As Leslie Carhart says:

Blood is in the Water -Not only have criminals found that ransomware is a great money-making scheme, but nation states and terrorist organizations have realized pseudo-ransomware makes a misleading and effective weapon. A weapon that can cause collateral damage, globally.

 

 

 

 

 

 

There have been some excellent reviews of what this attack was about, and how the Eternal Blue exploit released via ShadowBrokers was yet again leveraged against unpatched systems. Key takeaways were:

  • Unpatched systems will continue to be our undoing and exploited. We’re more at risk now because of that cache of exploits being lobbed at us monthly via the ShadowBrokers.
  • Lateral movement within networks works for attackers and infection spread. Segment. Segregate. Flat networks are an attacker’s dream.
  • Multiple infection vectors. There were as many as 4 ways for the target to be compromised.
  • Backup and test how those restore. Don’t assume anything. And keep backups off the main network
  • Windows.  Everyone uses it. Powershell. Sysinternals. AD. PSExec. Let’s keep learning about these because the attackers sure as heck know what they can do with them.

We know what er are not doing well. It’s catching up with us. Let me end with these words of wisdom by Leslie:

Defense in depth, including human threat hunting and effective detection and prevention at many points, is key. This will involve policy and financial buy-in from many lagging organizations at a new level.

And this sums it up:

 

 

 

 

 

 

These blog posts say everything I could ever want you to know, only better. Please read them:

The Grugq: Pnetya: Yet Another Ransomware Outbreak  .

Leslie Carhart @hacksforpancakes:  Why NotPetya Kep Me Awake (And You Should Worry Too)

Cisco Talos Blog: New Ransomware Variant Netnya Compromises Systems Worldwide