Registrations to Open-AudIT forums are now closed. To ask any new questions please visit Opmantek Community Questions.

Open-AudIT

What's on your network?
It is currently Fri Mar 29, 2024 12:03 am

All times are UTC + 10 hours




Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 60 posts ]  Go to page 1, 2, 3, 4  Next
Author Message
PostPosted: Mon Apr 06, 2009 11:29 am 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
Ok, I know this is probably beyond the scope of OAv1, but I am kind of curious how this is planned to be implemented in OAv2. At least I'm assuming that this would be a goal in the next iteration of OA, as being able to control audits and scheduling of audits (domain wide, lists of computers, nmap audits, etc) through the web interface would certainly simplify and centralize the configuration and maintenace of an OA server setup.

I'm currently trying to implement a solution for doing this with the OA server I have set at work. I'd like to be able to simply manage everything through the web interface without having to create scheduled tasks or cron jobs for all my audit schedules. It's not hard for me to manage, I just fear for the sanity of the next person that would take my position and try to figure out how I set this up...lol. My setup also assumes certain knowledge that someone in my position probably wouldn't have.

Anyways, the solution that I'm trying is to write a script called "audit.pl", written in Perl. This script sits in the "scripts" directory of a base OA setup. I have it set to parse the "include_config.php" so it can recognize and act on options set from the web interface. This also lets the script make calls to the MySQL database using the DBI module since it can get the username/pass/server/db from the include_config.php file. This way it can get additional configuration options via the database (Where I plan to store the equivalent of audit.config in a table, along with scheduling). There's also a nifty module in CPAN called [url=http://search.cpan.org/~roland/Schedule-Cron-0.9/Cron.pm]Schedule::Cron[/url] that is basically a drop in replacement for cron. The audit.pl script could then kick off a daemon like process that could read in all the scheduled audits from the database and then just sit and wait to run them.

The only drawback I'm having with the above is that if the server is rebooted, someone would still need to manually start the perl cron daemon from the web interface. Though I suppose the script could be added as part of the startup process after the MySQL and Apache daemons start.

Are there any plans to implement such functionality in the current version of OA? Also, is there a plan to have such functionality included in OAv2? If either is true, what would the implementation look like?

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Mon Apr 06, 2009 12:38 pm 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1964
Location: Brisbane, Australia
In a word - YES.
The plan is to have _everything_ configurable from the web interface. I don't wish to have to alter _anything_ by hand. This is very much apart of some of the use-cases. People may not have access to command line to edit things (think: hosted application).

How do I plan to do auditing ?
Well, at this stage, my thoughts go something like this...
Bear in mind OAv2 is designed to be hosted and run on a Linux machine. I am sure people will get it working on a Windows machine, but that is not my primary focus. Yes, a pre-configured VM _may_ be available. So....

I plan to enable the Admin user to setup "groups". Groups are defined as either Static or Dynamic. Static groups are simply a list of devices that the Admin assigns to that group. Dynamic groups are based on a field in the DB (think network subnet, OS, etc). FYI - I already have this working.
For a Static group, the plan is to create an audit script that includes this group. There is an initial "everything" default group that always exists and does contain all devices.
Now, when auditing machines, there will be two ways to do it.

One - have the server install a scheduled task on the client. The task will run a VBscript "shim" that starts and requests the audit script from the server. It provides some identifying information (UUID, MAC, PCName) and it given the correct script (assuming one needs to do this). It also downloads a small .exe to encrypt the audit results and the Public Key of the server. It then runs the audit script and submits the (encrypted) results. That way, if you need to alter the audit script, you do it on the server only, and all PCs get the new version next time they run the script. The task can be setup on whatever schedule the Admin desires.

Two - have the server connect to each machine and run the script. Same as above, except think of server initiated rather than client scheduled.

My current thoughts are to record the configured audits in the DB. That way, if an audit does not occur when scheduled, an alert can be generated. Also, we would have the option of modifying the old schedule (regardless of it being client or server based). That way, all should be controllable from the server, using the web interface.

There would be a cron job that runs on the server evey hour to check for audits that need to be executed. Having this ability also gives rise to other things that can be run as well - think reports generated and emailed daily, etc, etc. I am thinking _one_ cron job that checks all needed tasks, rather than a seperate cron job for eash task.

At the moment, these are thoughts only. I have no code to do this, other than a proof of concept using "winexe" on Linux to run a command on a Windows machine. It does work - so I think it's more a matter of banging out code...

So, in the end it's very similar to what your thoughts are. :D

_________________
Support and Development hours available from [url=https://opmantek.com]Opmantek[/url].
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Wed Apr 08, 2009 11:06 am 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
Sounds very interesting. I like the idea of installing the scheduled task on the PC as then you only have to worry about hitting the box once and it takes care of itself after that. One possible issue though is that to do even that would require connecting to the PC with winexe, which doesn't encrypt it's password it uses to make the login to the Windows box. I'm assuming you were looking at that solution for hosted type solutions, since it encrypts the data and then could send it back over an insecure link?

It all sounds very interesting though, I look forward to seeing what it grows into as you progress with it :)

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Wed Apr 08, 2009 11:55 am 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1964
Location: Brisbane, Australia
Not quite.....

Winexe would be used "internally" on private networks.

The hosted solution would allow the Admin to create a new group (along with the corresponding audit script - which will include all config items), and have that run on the client PCs. One audit script per group/client. Would not use Winexe for that. A couple of ways to do it... Email a link to the audit script and have the client manually run it (more for "one-off" type stuff), or setup the task on each system they want monitored, manually. I don't think "pushing" audits from an internet web server would be / is acceptable to most people (I know I wouldn't allow it on my network). The audit should be initiated from "inside" the client network. Maybe they could use Group Policy ? Not sure here. Worst case, create a script to set it up on each PC...

_________________
Support and Development hours available from [url=https://opmantek.com]Opmantek[/url].
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Mon Apr 20, 2009 11:32 am 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
Yeah, that makes sense as a solution. I have a habit of overlooking simple stuff...lol.

As a brief update on scheduling audits from the web interface, I have a few pages of it done now. I have a page for building audit configurations. It consists of ...

  • Give the config a name

  • Selecting a "PC Audit" or a "Port Scan"

  • Specifying an OS to assume for the audit (Windows or Linux - Not needed if Port Scan was selected).

  • Type of audit, or how to audit I guess, you can audit by Domain, a list of computers, or an IP range.

  • Specify credentials to use (can utilize an existing LDAP connection defined in the admin config area).


I also have the page for making a schedule then assigning an audit configuration to it. It's pretty simple and justs consists of ...

  • Give the schedule a name.

  • Assign an audit config to it.

  • Define the schedule type. It can currently be hourly (every X amount of hours, or only between a certain time), daily (every X amount of days at a certain time), weekly (pick the weekdays you want it to run on and the start time), or monthly (Pick the months and day of the month and the start time).


The most annoying part was getting the forms working the way you would expect, as there are lots of possibilities for errors to account for. The data is validated via AJAX to a php page and the tables are created if needed and the config/schedule is added to the needed table. I used AJAX because I didn't want to fight with page reloads and form data on submissions with errors.

I also have a page done for managing the current configurations and schedules. It currently consists of two simple tables of data displaying the configs or schedules. You can delete schedules/configs them from this page if you want. This will also be the page to activate/deactivate schedules and start/stop the main Cron process - which is just the main Perl cron job that will read all the audit schedules each minute.

I currently dont have a page to update the configurations or schedules...lol. I'm guessing that probably wont be too bad. I'll just need to initially account for the type and of config/schedule and adjust the update page accordingly with some javascript.

I'm currently back to getting the Perl script up and going now that it can read the configurations and schedules. So progress is being made. I bet in another month I may have something useful.

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Wed Apr 22, 2009 3:36 pm 
Offline
Site Admin
User avatar

Joined: Mon Jun 07, 2004 11:48 am
Posts: 1964
Location: Brisbane, Australia
Screenshots - we want screenshots !!!
ha ha - seriously though, a simple screenie would be nice. I am trying to implement similar forms for Group creation in OZv2 and am interested to have a look at your layout. My forms are also using Ajax (to retrieve table / column details from the DB and populate drop down boxes).
I'll have to sort out a video demo, so we can compare...

_________________
Support and Development hours available from [url=https://opmantek.com]Opmantek[/url].
Please consider a purchase to help make Open-AudIT better for everyone.


Top
 Profile  
Reply with quote  
PostPosted: Mon Apr 27, 2009 1:25 pm 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
Well, my forms are just functional...not much in the form of eye candy or anything :P I could probably clean them up a bit to make them at least formatted a bit better than they are now. Anyways, here's a quick video I put together showing what the forms currently do/how they function ...

http://www.youtube.com/watch?v=vnuHyWZeP5Q (I swear this isn't a link to a Rick Astley video :twisted: )

I recently added the ability to edit the configurations and schedules. I did some testing on them and squashed most of the bugs they seemed to have. So the forms to add/edit schedules and configurations seems to work fine. The page to manage them is OK, but I need I still need to add a button to activate/deactivate the schedules from that page. It should be pretty easy....as activating/deactivating is just a change for a field in that schedules entry in the DB. I also need a way to start/stop the script that monitors the schedules table and creates the audit processes on the server.

Right now the Perl script can monitor the schedules table and dynamically add/remove the schedule processes and sets them to kick off at the correct time on the server by parsing the schedule data that was entered. The script also logs important events like if it adds/removes a schedule, or if a schedule is deleted from the DB and it has to kill the audit process already created on the server. It catches a SIGINT/SIGTERM and simply kills any schedule processes on the server and exits to shutdown the script. So stopping it is as easy as simply killing the script process, which can be easily be implemented on the manage page.

So not too long hopefully and I'll post the collection of pages/scripts that make this possible if anyone else wants to monkey with it.

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Wed May 06, 2009 12:39 pm 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
For those who may be interested...

The logic for doing the audits is in place. I've ran a few tests and it seems to work good. The script simply has a sub that is passed an array of computer names/ip addresses and a hash ref of options. It then kicks off the audit by passing the host/ip address to the sub for that type of audit (Windows audit, linux audit, nmap audit, etc), forking each one as it goes. Currently I only have the Windows audit sub working. Also, it encrypts username/password information in the same way the LDAP connections are.

I have it logging specific events from the audits in a table (audit_log) in the database. This way I can easily see how the audit went by just setting up a custom view. This means I won't have to sift through a text file to see my audit info, yay :p

Building in nmap is going to take running something as suid root, which isn't a very good solution from a security stand point...but I don't really see a way around it. I want to rewrite the linux audit to use net:ssh::expect, but not too sure how I should do data collection yet. I really think the prospect of dumping the XML output of LSHW and parsing it might be a good approach. This makes it depend on a package to be installed, but might be the best way to ensure it pulls lots of information consistently.

So, that's where it is currently at. It uses quite a few Perl modules to pull it off. The current list is...

DBI : To interface with the MySQL db
Schedule::Cron : Acts as a cron service to start/fork the audit schedules and start them at the right time.
Parallel::ForkManager : Manages the logic of forking each specific audit, killing audits running past a certain time, making sure only a certain number are active, etc.
Net::LDAP : Queries the LDAP path to grab an array of computers (specific ldap paths and paging sizes can be specified via the web interface)
Net::Ping : Tests for connectivity prior to auditing.

At least those are the ones currently implemented already. I'll be adding Net::SSH::Expect for linux audits soon.

Doing this has been a fun learning experience actually...haha.

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Tue May 26, 2009 1:19 am 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
I'm to the point now where the whole audit scheduling, activating/deactivating schedules, and stopping/starting the Perl cron daemon is done through the web interface. A screenshot is attached for the curious.

For doing Nmap audits I found a Perl module that interfaces with sudo. So if an Nmap audit is specified for a configuration type, the form will just ask for a username with sudo rights and it's password and will store it encrypted in the DB. When a schedule kicks off that is an Nmap audit, it will just relaunch the script via sudo, passing the schedule and configuration ID to be executed to the script.

I have Net:SSH::Expect working in the script for remote Linux audits. It grabs the output of LSHW and parses as much of it as possible, which is quite a bit. This also means it doesn't have to copy anything to the remote machine, it just executes some commands using Net::SSH::Expect and parses the output. More work is needed to get this in a more usable state though.

So, all in all, I'm still chugging along with this. It's really close to being in a fairly easy to use state. Granted, it does require a few Perl modules to be installed.


Attachments:
File comment: Web Scheduling - Management Interface
bleh.jpg
bleh.jpg [ 226.57 KiB | Viewed 19632 times ]

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2
Top
 Profile  
Reply with quote  
PostPosted: Thu Aug 27, 2009 12:57 pm 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
Well, this has taken a little longer than I thought it would :p I work on it here and there when I have the time. I've made lots of improvements to my original design. Kind of a long update here.

Configurations can perform the following actions : Remote Commands, PC Audits (Windows only audits at the moment), and NMAP audits.
Configurations can gather Hostnames/IP addresses using : A hostname/ip list, IP range, LDAP query, or a MySQL query.

LDAP and MySQL queries can be filtered using a Perl regular expression. So if you query LDAP or MySQL but only want to get all workstations that start with 'L', you just put '^L' in the filter and it returns only those computers. You can also inverse the filter so it returns only results that don't match. Or you can set the filter to be case-insensitive.

The MySQL query is built dynamically on the form. You pick the table, the field, what data the field should contain, and how it should determine if that data is a match ( ie. if a field contains, begins with, ends with, or is equal to the value you entered). You can add as many fields as you want to the query.

I added the ability to do remote commands because of a project at work. It was easier to add than I expected. The hardest part was working out how the form should work. You add single commands on a standalone page from the web interface. Then when you create a configuration and choose "remote commands" as the action, you then check which commands you want to run and then order them by moving them up or down on the list. The configuration stores which commands you picked and executes them one at a time, in order, on the hosts you specified.

Events from audits are logged in the database. I still need to add the ability to choose to turn logging off for specific audits and not to log specific events if you don't want to. All the logs are viewable through a custom view definition file that sorts it by schedule name, then by the date a specific audit.

It cannot yet do linux audits, as I haven't finished enough of the code for that to be useful. I also want to add the ability to email audit results after a schedule is done running, which shouldn't be too bad. Adding some sort of notification system would be nice to, like having the script email me if it detects a computer with a certain name was just audited, or automatically run a command on it if it meets certain criteria.

I can also confirm that the script works on Windows and Linux :twisted: If you're using Windows it uses RemCom.exe to do remote windows audits, if you're using Linux it uses Winexe. It works best from Linux, as I still need to sort out a way for starting the main script from the web interface on windows.

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Thu Aug 27, 2009 7:21 pm 
Offline
Moderator
User avatar

Joined: Tue Jan 25, 2005 3:09 am
Posts: 2140
Location: Scotland
.... I want one..... :D .... where do I sign ...

_________________
Andrew

[size=85]OA Server: Windows XP/ XAMPP, Mandriva/Apache, Ubuntu
Auditing: 300+ Wstns, 20+ Srvrs, Thin clients, Linux boxes, Routers, etc
OS's: Windows XP , W2K Srvr, W2K3 Srvr, W2K8, Vista, Windows 7, Linuxes (and a Mac at home)
LDAP: Active Directory[/size]


Top
 Profile  
Reply with quote  
PostPosted: Fri Sep 04, 2009 12:39 pm 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
[quote="A_Hull"].... I want one..... :D .... where do I sign ...


Not too much longer and I'll put it for download somewhere. Most of what I consider neccessary has been added.

I added the ability to disable/enable logging either at the schedule level or the configuration level. I also added the ability to email the logs after the schedule completes. SMTP settings (server, port, etc) can be edited and to tested from a webpage.

The logs are not sent as attachments to an email, but rather sent with general statistics on the audit. Such as how many audits failed, how many completed successfully, how many needed to be killed, etc. The email that is sent is very customizable. To generate the email it uses a file sitting in OpenAudit's lib directory that's processed by [url=http://template-toolkit.org/]Template Toolkit[/url]. So editing it is just like editing an HTML page. This way I don't have to edit the main script to simply change the way the email appears. The email also has the OpenAudit logo attached as a header. You can change this logo to anything you want though, provided you put the picture in the 'images/headers' folder.

I still need to add the Win32::Daemon module to the main script so it can install itself as a service if it's on a Windows box to make managing it much easier.

Also, it won't be neccessary to install Perl and all the modules that this script needs in order to run it. There is a way to package it to include perl embeded along with all the modules it needs. So the end result will just be a exe/binary. The source will still be there if anyone really wants to run it the other way though.

I'm just glad I can now basically manage the whole thing without having to touch configuration files and messing with crontabs :p

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Fri Sep 18, 2009 3:00 pm 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
Well, here are two sets of files if anyone wants to try it out. They're too large to post here, so I need to host them on my site. The easiest way to test it is to just download this...

http://chadsikorra.com/dl/oa-ws/oa-webs ... 85.tar.bz2

That is the latest SVN with the Web-Scheduling/Audit component already integrated in it, along with my other modifications that help it work the way it should (needs an extra function in include_functions.php, url redirection, search with GET vars, etc)

If you want only the set of files that make up the web scheduling component, then download this ...

http://chadsikorra.com/dl/oa-ws/oa-webs ... 21.tar.bz2

I included a README with install instructions, as it requires a few relatively easy steps to get the daemon/service running. I've only tested with Apache on windows, so I'm not sure if the service control via the web interface will work on Windows with IIS without modifying the IIS user rights in some way.

I take no responsibility for any damage you may do to your network using this :p However, I've been using it for some time now as my main auditing method and it's been working good.

Btw - there are lots of things that I probably need to change/adjust with this code. I apologize in advance for my sloppy javascript, if anyone attempts to modify it...lol.

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
PostPosted: Tue Sep 22, 2009 10:27 pm 
Offline
Contributor

Joined: Fri Jul 28, 2006 6:30 am
Posts: 157
Location: London
Interesting work Chad. I see lots of references to cron. Does the code also work on Windows hosted OA servers?

_________________
Cheers, Nick.

[size=85]OA Server: Windows Server 2003 / Apache 2
Auditing: 1600 Workstations, 200 Servers
OS's: Windows XP / Windows 2000 / Windows 2003 Server / Windows Vista
LDAP: Active Directory[/size]


Top
 Profile  
Reply with quote  
PostPosted: Wed Sep 23, 2009 2:33 am 
Offline
Contributor

Joined: Fri Jul 04, 2008 6:46 am
Posts: 153
Location: USA - WI
Yeah, it will work on Windows servers as well. The cron scheduler is a pure Perl module (Schedule::Cron). I only had to change the module slightly to not use POSIX setsid so it would work on Windows.

Of course, since the perl binary (audit.exe) runs as a service, you need rights on the box to initially install it. After it's installed I ran into no issues controlling it via the web interface by calling 'net start' or 'net stop' commands. I only tested with Apache though, IIS may require tweaking the IIS user rights for that to work, I dunno.

You can specify the name of the service too, so it's possible to have multiple copies running if you run multiple openaudit sites on the same box.

_________________
OA Server: Debian Squeeze w/ Apache2
Auditing: 700 Workstations, 250 or so Retail Terminals, about 75 Servers
OS's: Windows XP/2003/2008/2008 R2/Vista/7, Debian
LDAP: Active Directory 2008 R2


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 60 posts ]  Go to page 1, 2, 3, 4  Next

All times are UTC + 10 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group